Senior Software Engineer, AI Evals Job at Sentry (San Francisco)

Senior Software Engineer and Principal Software Engineer - Power Point AI Team

The PowerPoint team is embarking on an exciting new chapter - evolving a product...

Location

United States , Redmond

Salary:

119800.00 - 234700.00 USD / Year

Microsoft Corporation

Expiration Date

Until further notice

Requirements

Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
8+ years of experience in backend service engineering, including work on high-scale infrastructures
Proficiency in one or more systems programming languages such as C#, C++
1+ years of experience in software engineering, designing and developing systems (and APIs) that deploy and integrate with AI models
2+ years of experience working with rich telemetry, making data driven decisions, and carrying out rapid experimentation
2+ years of experience building software for scale, performance, and reliability
Academic or industry experience with building, finetuning, deploying or building eval-driven systems utilizing the models (any category)

Job Responsibility

Lead design and delivery of complex, scalable AI features ensuring resilience and exceptional user experience
Drive technical strategy and architecture decisions across multiple services, influencing partner teams and aligning with compliance and security requirements
Champion modern engineering practices, including AI-driven approaches, automation, and cloud-native patterns, across the full development lifecycle
Mentor and guide engineers, fostering technical excellence and continuous improvement in security, reliability, and performance
Collaborate cross-org to solve challenging technical problems, streamline processes, and reduce operational costs while improving live-site health
Design and implement scalable backend services optimized for machine learning workflows and large language model integration
Develop and maintain evaluation-driven systems that leverage text and multimodal inputs (e.g., images) to power visual-creation experiences
Build and optimize APIs and infrastructure to support high-performance model inference and experimentation at scale
Collaborate with product, ML, and design teams to integrate models into user-facing features, ensuring seamless functionality and performance
Conduct model evaluations and experiments, analyze results, and iterate on improvements to enhance accuracy and user experience

Fulltime

Senior Software Engineer, AI Product

As a Senior Applied AI Engineer at Vanta, you will play a crucial role in shapin...

Location

United States

Salary:

207000.00 - 244000.00 USD / Year

Vanta

Expiration Date

Until further notice

Requirements

At least 7 years of industry experience as a software engineer
You’ve shipped LLM-backed products and have experience with prompting, RAG, and/or agent frameworks
You have experience designing, building, and scaling full-stack applications, including backend systems, APIs, and frontend interfaces
You have familiarity with TypeScript, React, and Node.js, or a willingness to learn
You have experience improving AI systems, creating eval sets, and driving quality hill-climbing
You have experience mentoring other engineers and collaborating with product and design
You have worked at rapidly scaling startups and large companies, especially with environments that prioritize a bias for action
You are action-driven, willing to roll up your sleeves and engage directly with users
You aren’t afraid to put on your product hat
While you bring strong opinions, you prioritize building a platform that meets users where they are

Job Responsibility

Work cross-functionally to design and implement AI-powered features to deliver customer value and integrate LLMs with Vanta’s existing products and systems
Instrument evaluations, guardrails, and monitoring, and review customer usage to continually improve quality
Collaborate with AI Platform engineers shaping foundational AI systems and tooling that accelerate product teams
Make pragmatic tradeoffs that consider business priorities, user experience, and a sustainable technical foundation
Mentor engineers, champion good technical and product instincts, and model a collaborative, high-ownership engineering culture

What we offer

Offers Equity
medical benefits
401(k) plan
other company perk programs
Comprehensive medical, dental, and vision coverage, with 100% of employee-only benefit premiums covered for most medical plans
16 weeks fully-paid Parental Leave for all new parents
Health & wellness stipend
Remote workspace, internet, and cellphone stipend
Commuter benefits for team members who report to the SF and NYC office
Family planning benefits

Fulltime

Senior Software Engineer, AI Eval

As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for ...

Location

United States , San Francisco

Salary:

240000.00 - 280000.00 USD / Year

Sentry

Expiration Date

Until further notice

Requirements

Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
Comfort writing production-quality code (we use Python and TypeScript)
Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)

Job Responsibility

Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

What we offer

incentive compensation
equity grants
paid time off
group health insurance coverage

Fulltime

Senior AI Engineer - MSC AI Innovation

MSC AI Innovation is an AI-first team that incubates, builds, and accelerates so...

Location

Israel , Tel Aviv, Herzliya

Salary:

Not provided

Microsoft Corporation

Expiration Date

Until further notice

Requirements

8+ years professional software development
4+ years of software engineering experience in the AI space (e.g., building and shipping AI/ML or GenAI features in production)
Proven experience with building AI agents
Hands-on experience with evaluation methodologies and integrating quality standards/guardrails into delivery
Proficiency in Python and/or C#, with experience using REST APIs and SDKs
Deep understanding of AI system design, including ML fundamentals, Generative AI concepts, and cloud-native architectures

Job Responsibility

Design and build AI agents that plan, use tools/APIs, manage state/memory, and reliably complete multi-step workflows
Own AI features from design through production, including deployment, monitoring, and live‑site reliability, with an eval-first development lifecycle: define success criteria, build evaluation datasets and automated harnesses, and run human-in-the-loop reviews where needed
Develop and maintain prompt, retrieval, and memory strategies (system prompts, few-shot examples, tool schemas, retrieval context) with proper versioning and evaluation coverage
Debug AI behavior using prompt analysis, data inspection, and model/tool-call traces, and translate failure patterns into targeted improvements
Establish and track AI quality metrics (e.g., accuracy, groundedness, relevance, hallucination rate) and integrate them into CI/CD release gates
Optimize runtime performance and economics (token usage, inference cost, latency, caching, model selection/routing, batching) and implement monitoring and continuous improvement loops (online signals, drift detection, structured user feedback)
Partner with product, design, and domain stakeholders to define use cases, acceptance criteria, and rollout plans for AI features
Live site responsibility

Fulltime

Senior AI Engineer - Teams Messaging AI

Are you interested in joining one of the most exciting teams and working on the ...

Location

United States , Mountain View

Salary:

119800.00 - 234700.00 USD / Year

Microsoft Corporation

Expiration Date

Until further notice

Requirements

Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter

Job Responsibility

Design, implementation, and shipping of multiple new messaging and large language models (LLM) agentic features
Building end-to-end user experiences that work across multiple devices and browsers
Writing and maintaining unit tests, large language models (LLM) eval and automated integration or end-to-end tests
Building web and AI applications in enterprise and/or consumer markets
Collaborating with partner teams to meet engineering goals
Managing individual projects or feature priorities, deadlines, and deliverables

Fulltime

New

Senior Ai Engineer - Patient Health Platform

We are looking for a Senior AI Engineer to join the Patient Health Platform team...

Location

France , Paris

Salary:

Not provided

Doctolib

Expiration Date

Until further notice

Requirements

Production deployment: ability to ship algorithms to production (ECS-based service on AWS)
Strong analytical mindset: result-oriented, patient-first approach
Significant experience as a Software or/and AI engineer shipping search or recommendation systems to production
Hands-on experience building end-to-end retrieval: ranking, reranking pipelines and familiar with nDCG, MAP, Recall@k, MRR
AI-engineering proficiency: turning foundation models and off-the-shelf components into production systems: embeddings & vector search, semantic retrieval, RAG, LLM-based or managed rerankers (e.g. Vertex AI). You can succeed without training a model from scratch
Architecture-first approach: you build the system, baselines, evals, and feedback loops with standard tooling before reaching for custom ML, and know when to partner with ML Engineers to break a ceiling
Evaluation & observability built into every stage (retrieval, ranker, reranker) — offline and online eval, A/B testing, position-bias handling, monitoring
Production deployment — ability to ship reliable, low-latency services to production (hundreds-of-ms SLAs), with care for data quality and long-term maintainability

Job Responsibility

Design and build the production search & recommendation architecture: full retrieval, ranking, reranking pipeline with standard and off-the-shelf components (vector search, semantic retrieval, LLM/managed rerankers)
Establish strong baselines first (prompts, RAG, model selection) before reaching for custom ML
Build evaluation and observability into every stage, with offline and online evaluation
Set up the data/event feedback loops that drive iteration and feed deeper ML later
Improve search relevance and ranking on Patient facing products, raising result quality
Own production quality: latency reliability, monitoring, and maintainability
Partner with ML Engineers and collaborate closely with PMs and SWEs to define, build, and ship AI-powered features that deliver measurable value to users and the business.

What we offer

Free comprehensive health insurance (basic package) for you and your children
25 days of paid vacation per year, plus up to 14 days of RTT
Free mental health and coaching services through our partner Moka.care
Work from abroad for up to 10 days per year thanks to our flexibility days policy
Lunch vouchers (Swile card) worth €8.50 per working day, with €4.50 covered by Doctolib
A subsidy from the work council to refund part of the membership to a sport club or a creative class
50% reimbursement of your public transport subscription
Enrollment in Doctolib's long-term employee value sharing plan called DoctoGrowth
Parent Care Program: receive one additional month of leave on top of the legal parental leave
For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support

Fulltime

Senior AI Engineer

We're looking for a Senior AI Engineer to help us build the next generation of A...

Location

Netherlands; Germany; Romania; United Kingdom; Spain; Italy; Poland , Amsterdam; Berlin; Hannover; Iași; London; Madrid; Milano; München; Warsaw

Salary:

Not provided

Awin Global

Expiration Date

Until further notice

Requirements

8+ years of professional software engineering experience, with a track record of shipping production systems
Strong Python and backend system design
Real, hands-on experience building and running production LLM-based systems not just prototypes
Deep experience with agent-based architectures using frameworks such as LangGraph, the Deep Agents or equivalents with multi-step reasoning, tool use, sub-agent orchestration, state and streaming
Strong working knowledge of retrieval and context construction (RAG, embeddings, chunking, ranking) and good judgement about when to use those patterns inside an agentic system
A solid track record of evaluating and debugging AI systems: structured evals, regression tests and tracing or observability with tools such as LangSmith or similar
Familiarity with emerging agent orchestration standards (MCP or similar)
Experience with vector databases and hybrid retrieval
A clear understanding of LLM failure modes like prompt injection, content safety, context leakage and how to mitigate them in real systems
Strong fundamentals in distributed systems, API design and cloud-native architectures on AWS (ECS, Lambda, S3, API Gateway)

Job Responsibility

Design, build and operate customer-facing AI features end-to-end, from conversational experiences to intelligent automation
Architect agent-based systems: multi-step reasoning, tool execution, sub-agent orchestration, state management and streaming, structured outputs, persistent memory and human-in-the-loop approval flows
Decide how the system grounds, retrieves and constructs context
Pick the right pattern (retrieval, structured tools, agentic flows) for each problem and be able to defend the choice
Treat prompt and context engineering as first-class engineering work
Own AI quality
Build evaluation datasets, regression tests for prompts and agents and the debugging discipline needed to get to root cause across model outputs, retrieval and tool calls
Make AI production-ready: observability, tracing, reliability, cost, latency and the safety controls that protect against hallucination, prompt injection and context leakage
Make and explain the trade-offs between accuracy, latency, cost and safety within engineering and product partners
Partner within Product Trio to shape what we build, not just how we build it

What we offer

Flexi-Week and Work-Life Balance: We prioritise your mental health and wellbeing, offering you a flexible four-day Flexi-Week at full pay and with no reduction to your annual holiday allowance. We also offer a variety of different paid special leaves
Remote Working Allowance: You will receive a monthly allowance to cover part of your running costs. In addition, we will support you in setting up your remote workspace appropriately
Flexi-Office: We offer an international culture and flexibility through our Flexi-Office and hybrid/remote work possibilities to work across Awin regions
Meal Vouchers: You will be supported with a certain net sum to spend it on a variety of lunches
Health & Wellbeing: The insurance covers several types of health, vision and / or dental treatments for you and for up to one additional family member
Remote Working Furniture Package: After 3 months of employment, you will be eligible for a furniture package, which should enable you to set up a proper workplace at your remote working location
Appreciation: Thank and reward colleagues by sending them a voucher through our peer-to-peer program

Fulltime

Senior Software Engineer

We are looking for a Senior Software Engineer to join our team to drive all aspe...

Location

Canada , Vancouver

Salary:

114400.00 - 203900.00 CAD / Year

Microsoft Corporation

Expiration Date

Until further notice

Requirements

Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
OR equivalent experience
Understanding of building engineering tools on the server side for scale
2+ years of experience on engineering tooling or eval development
Prior experience in working on services at scale
Prior experience in driving fundamentals for AI features within web apps
Prior experience in working closely with AI feature teams and improving fundamentals like performance and reliability is a major plus
Experience solving challenging problems and cross team/organization collaboration skills

Job Responsibility

Build and evolve the Real-Time Intelligence evaluations platform: implement offline and online eval pipelines, including golden datasets, human review workflows, and LLM-as-judge / auto-raters for agents, anomaly detectors, and decisioning systems
Instrument agentic solutions for observability by wiring up telemetry, tracing, structured logging, and dashboards so quality, safety, latency, and cost are easy to monitor and debug
Integrate evals into the development lifecycle by connecting pipelines to CI/CD, canary and A/B experiments, and phased rollouts, making it simple for partner teams to run and interpret evaluations
Collaborate and mentor across product, research, and engineering teams, sharing best practices on eval design, LLM-as-judge usage, and Responsible AI, and providing code reviews and guidance that raise the bar for the AI features

Fulltime

Select Country

Senior Software Engineer, AI Evals

Job Description

Job Responsibility

Requirements

Nice to have

What we offer

Looking for more opportunities?

Senior Software Engineer, AI Evals

Senior Software Engineer and Principal Software Engineer - Power Point AI Team

Senior Software Engineer, AI Product

Senior Software Engineer, AI Eval

Senior AI Engineer - MSC AI Innovation

Senior AI Engineer - Teams Messaging AI

Senior Ai Engineer - Patient Health Platform

Senior AI Engineer

Senior Software Engineer

Our AI answers in your language