CrawlJobs Logo

Applied ML Validation Manager

United States, Sunnyvale 218000.00 - 335000.00 USD / Year · Job Posted April 16, 2026
Apply Position
Job Link Share

Job Description

As an Applied ML Validation Manager on the Software Validation team within the AV organization, you will lead a team focused on building and operating behavior critics and human benchmarking capabilities for ML-driven autonomy systems. Your team will turn subjective human expectations about safe, comfortable, and intuitive driving into rigorous, scalable evaluation frameworks that directly inform model development and release decisions. You will partner closely with autonomy, simulation, safety, and product teams to define how behavior is judged against human drivers and integrate behavior critic signals into validation pipelines, continuous release, and long-term performance monitoring.

Job Responsibility

  • Lead and grow an Applied ML validation team focused on behavior evaluation and human benchmarking for autonomy ML systems
  • Define the strategy and roadmap for evaluating ML behavior against human-like driving expectations across simulation, replay, and on-road environments
  • Design, implement, and operate behavior critic frameworks that assess model actions and trajectories, turning qualitative human feedback into structured labels, metrics, and scorecards
  • Develop and scale human benchmarking programs, including rater guidelines, calibration, and quality controls, to compare ML system performance against expert and typical human drivers
  • Partner closely with autonomy, simulation, safety, and product teams to integrate behavior critic and human benchmarking outputs into training, offline validation, release gating, and reporting

Requirements

  • 8+ years of experience and MS/PhD in Computer Science, Machine Learning, Robotics, Software Engineering, Data Science, or a related field
  • 2+ years of people management experience leading engineering, validation, or applied ML teams
  • Strong programming and data skills in Python and common analysis/ML tooling (e.g., PyTorch)
  • Demonstrated experience designing and operating evaluation/validation pipelines for complex ML systems
  • Proven ability to define, implement, and track metrics that capture system quality, reliability, safety, or user experience

Nice to have

  • Experience with autonomous driving, robotics, or other safety-critical domains, especially in validation, safety, or systems engineering roles
  • Demonstrated background with simulation-based validation, including VLM critics, human benchmarking, and scalable evaluation for ML or autonomy systems
  • Hands-on experience with agentic workflows used to accelerate analyses, automate documentation, or orchestrate complex data and metric pipelines
  • Track record of building or scaling technical teams and tooling in fast-evolving domains, especially focused on evaluation, automation, and ML observability

What we offer

  • medical
  • dental
  • vision
  • Health Savings Account
  • Flexible Spending Accounts
  • retirement savings plan
  • sickness and accident benefits
  • life insurance
  • paid vacation & holidays
  • tuition assistance programs
  • employee assistance program
  • GM vehicle discounts

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Applied ML Validation Manager

8 matching positions

Lead Applied ML Engineer

Lead Applied ML Engineer, Technology and Digital, FT, 09A-5:30P
Location
Location
United States , Remote
Salary
Salary:
144000.00 - 186000.00 USD / Year
baptisthealth.net Logo
Baptist Health
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Masters degree in Computer Science/Machine Learning or a minimum of 10 years equivalent professional experience
  • Must have experience in GCP
  • Proven team leadership background in machine learning and artificial intelligence with expertise in one or more of: computer vision, NLP, speech, optimization, deep learning, reinforcement learning, time series, generative models, signals, and distributed systems
  • Strong proficiency in ML modeling frameworks
  • Strong expertise in overall software development approach
  • Significant leadership experience in building end to end data systems
  • Advanced software engineering skills with proven experience crafting, prototyping, and delivering advanced algorithmic solutions
  • Proficiency in one or multiple machine learning languages (ex: Python) & development environments such as AWS Sagemaker
  • Minimum Required Experience: 10 years
Job Responsibility
Job Responsibility
  • Lead AI Implementation: Drive the end-to-end development of production-grade AI solutions, from LLM orchestration and backend APIs to interactive UI prototypes and automated deployment pipelines
  • Full-Stack Ownership: Take accountability for the technical lifecycle of AI products, ensuring they are scalable, secure, and seamlessly integrated into healthcare workflows
  • GenAI & Advanced Modeling: Develop and deploy advanced Generative AI applications using RAG patterns and model fine-tuning
  • architect orchestration layers and agentic workflows to ensure vendor-agnostic, autonomous problem-solving
  • Full-Stack Development & Prototyping: Build robust Python-based backends and scalable APIs
  • create interactive user interfaces (POCs) to visualize AI reasoning and gather clinical stakeholder feedback
  • Data & Infrastructure Integration: Integrate AI solutions with cloud data warehouses (e.g., Snowflake) and manage containerized deployments (Docker) via automated CI/CD and GitOps pipelines (GitLab, ArgoCD) on GCP
  • Governance, Security, & Monitoring: Engineer automated guardrails for PII/PHI masking and risk mitigation
  • implement observability tools to monitor model drift, hallucination rates, and token-based cost metrics (FinOps)
  • Safety & Interoperability: Validate clinical logic using advanced evaluation frameworks (e.g., RAGAS) and ensure seamless EHR integration through healthcare data standards like FHIR and HL7
  • Fulltime
Read More
Arrow Right

Lead Applied ML Engineer

Role Summary: Lead AI Implementation: Drive the end-to-end development of produc...
Location
Location
United States , Remote
Salary
Salary:
144000.00 - 186000.00 USD / Year
baptisthealth.net Logo
Baptist Health
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Masters degree in Computer Science/Machine Learning or a minimum of 10 years equivalent professional experience
  • Must have experience in GCP
  • Proven team leadership background in machine learning and artificial intelligence with expertise in one or more of: computer vision, NLP, speech, optimization, deep learning, reinforcement learning, time series, generative models, signals, and distributed systems
  • Strong proficiency in ML modeling frameworks
  • Strong expertise in overall software development approach
  • Significant leadership experience in building end to end data systems
  • Advanced software engineering skills with proven experience crafting, prototyping, and delivering advanced algorithmic solutions
  • Proficiency in one or multiple machine learning languages (ex: Python) & development environments such as AWS Sagemaker
  • Minimum Required Experience: 10 years
Job Responsibility
Job Responsibility
  • Lead AI Implementation: Drive the end-to-end development of production-grade AI solutions, from LLM orchestration and backend APIs to interactive UI prototypes and automated deployment pipelines
  • Full-Stack Ownership: Take accountability for the technical lifecycle of AI products, ensuring they are scalable, secure, and seamlessly integrated into healthcare workflows
  • GenAI & Advanced Modeling: Develop and deploy advanced Generative AI applications using RAG patterns and model fine-tuning
  • architect orchestration layers and agentic workflows to ensure vendor-agnostic, autonomous problem-solving
  • Full-Stack Development & Prototyping: Build robust Python-based backends and scalable APIs
  • create interactive user interfaces (POCs) to visualize AI reasoning and gather clinical stakeholder feedback
  • Data & Infrastructure Integration: Integrate AI solutions with cloud data warehouses (e.g., Snowflake) and manage containerized deployments (Docker) via automated CI/CD and GitOps pipelines (GitLab, ArgoCD) on GCP
  • Governance, Security, & Monitoring: Engineer automated guardrails for PII/PHI masking and risk mitigation
  • implement observability tools to monitor model drift, hallucination rates, and token-based cost metrics (FinOps)
  • Safety & Interoperability: Validate clinical logic using advanced evaluation frameworks (e.g., RAGAS) and ensure seamless EHR integration through healthcare data standards like FHIR and HL7
  • Fulltime
Read More
Arrow Right

Senior Staff Engineering Manager - Applied AI

GEICO is in the midst of an exciting transformation as a product and tech powere...
Location
Location
United States , Chevy Chase; Palo Alto; Dallas
Salary
Salary:
140000.00 - 300000.00 USD / Year
geico.com Logo
Geico
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years hands-on software or solutions engineering
  • recent experience architecting/building production systems
  • 5+ years leading Agile teams delivering customer-impacting products
  • proven track record shipping AI/ML-enabled features to production with measurable impact
  • strong cloud experience (AWS or Azure)
  • CI/CD, SDLC, and modern engineering practices
  • practical MLOps/LLMOps experience (model lifecycle, telemetry, A/B testing, online and batch inference)
  • excellent stakeholder management and communication
  • Bachelor’s in CS/IS or related field (or equivalent experience)
Job Responsibility
Job Responsibility
  • Partner with product, data science, and AI Platform teams to define and deliver an Applied AI roadmap for claims tied to measurable business outcomes
  • Lead end-to-end delivery of AI features in the claims domain (e.g., claims triage/routing, adjuster assistance, subrogation/recovery opportunities, workload optimization)
  • Establish robust MLOps/LLMOps (model/prompt/version management, evaluation, feature/vector stores, CI/CD for ML, online/batch inference, cost/performance tuning)
  • Rapidly prototype using AI-assisted development tools (e.g., Cursor, Claude Code) to validate concepts and accelerate delivery
  • Ensure reliability, security, and Responsible AI (privacy-by-design, guardrails, human-in-the-loop, auditability)
  • Drive engineering excellence (clean architecture, scalable APIs, event-driven integration, observability, automated testing, SLOs/runbooks/on-call)
  • Recruit, coach, and grow a high-performing team
  • champion Agile practices and operational excellence
What we offer
What we offer
  • Comprehensive Total Rewards program that offers personalized coverage tailor-made for you and your family’s overall well-being
  • Financial benefits including market-competitive compensation
  • a 401K savings plan vested from day one that offers a 6% match
  • performance and recognition-based incentives
  • and tuition assistance
  • Access to additional benefits like mental healthcare as well as fertility and adoption assistance
  • Supports flexibility- We provide workplace flexibility as well as our GEICO Flex program, which offers the ability to work from anywhere in the US for up to four weeks per year
  • Fulltime
Read More
Arrow Right

Data Scientist

We are looking for a Data Scientist to support AI and machine learning initiativ...
Location
Location
United States , Palo Alto
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • PhD or master's degree in statistics, informatics, data science, or a closely related quantitative field
  • For candidates with a master's degree, at least 4 years of relevant experience applying data science methods to business, research, or healthcare problems
  • Demonstrated experience building, evaluating, and implementing machine learning models in applied environments
  • Proficiency in Python or R, with the ability to process and analyze large datasets using SQL and cloud technologies
  • Strong grounding in statistics, machine learning, and artificial intelligence concepts at the graduate level
  • Experience collaborating with clinicians, researchers, or other cross-functional subject matter experts on complex analytical initiatives
  • Solid communication, prioritization, and problem-solving skills, with the ability to manage multiple deliverables effectively
Job Responsibility
Job Responsibility
  • Create, implement, and support AI- and ML-driven workflows that improve clinical, research, and administrative processes
  • Partner with cross-functional stakeholders to define analytical needs and deliver data science solutions aligned with healthcare use cases
  • Assess and refine tools, platforms, and methods used to manage model development, deployment, and ongoing lifecycle activities
  • Train, test, and validate internally developed or externally sourced machine learning models using hospital data and established quality controls
  • Perform bias reviews and model performance checks to help ensure responsible and reliable use of predictive algorithms
  • Analyze large-scale healthcare datasets using Python, R, SQL, and cloud-based or distributed computing environments
  • Work alongside clinicians and researchers to adapt analytical methods for real-world use in care delivery and related settings
What we offer
What we offer
  • medical
  • vision
  • dental
  • life and disability insurance
  • 401(k) plan
  • Fulltime
Read More
Arrow Right

Fp&A - Database Developer, Assistant Vice President

We are seeking a Senior Oracle Developer with strong technical depth in SQL/PLSQ...
Location
Location
India , Mumbai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong proficiency in Oracle SQL and PL/SQL
  • Experience with Oracle Cloud services
  • Solid knowledge of Python programming
  • Understanding of Big Data technologies (e.g., Hadoop, Spark)
  • Experience with ETL tools and processes
  • Familiarity with Data Warehouse design and optimization
  • Excellent communication and interpersonal skills to collaborate across teams
  • 7+ years of progressive experience in software development, Strong hands-on experience with enterprise-level Java development, including Spring Framework (Spring Boot, Spring Cloud) and related ecosystems
  • and Deep understanding and hands-on experience with data architecture principles, including relational databases (e.g., PostgreSQL, Oracle), NoSQL databases (e.g., MongoDB, Cassandra), data warehousing, and big data technologies (e.g., Spark, Kafka)
  • Proven hands-on experience with Python for data manipulation, scripting, machine learning, and backend development
Job Responsibility
Job Responsibility
  • Design, develop, and optimize Oracle database solutions using SQL and PL/SQL
  • Implement and manage applications on Oracle Cloud Infrastructure (OCI)
  • Write Python scripts for data automation, transformation, and validation
  • Apply Big Data principles to support large-scale data processing needs
  • Design and maintain robust ETL workflows and data pipelines
  • Build and support Data Warehouse solutions ensuring performance and reliability
  • Collaborate with technical and business teams, especially data stakeholders, to understand and fulfill data requirements
  • Leverage SAS knowledge (if applicable) to support data analysis and reporting initiatives
  • Act as a bridge between business users and technical teams
  • Directly interact with clients/stakeholders to gather, analyze, and document business requirements
  • Fulltime
Read More
Arrow Right
New

Ai Product Owner

Beazley’s data reflects the complexity of our business. We are committed to buil...
Location
Location
United Kingdom , Birmingham
Salary
Salary:
Not provided
socialvalueportal.com Logo
Social Value Portal Ltd
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Stakeholder influence, structured problem-solving, commercial acumen, clear communication, and change leadership
  • Domain Expertise: Strong understanding of the specialist area (Ops/Reg, Dist/Marketing, or Policy/Claims) and adjacent functions (pricing, risk, finance)
  • Detailed knowledge of business and technology issues / opportunities related to insurance, data and AI
  • Strong understanding of data management principles, data governance, and analytics
  • Experience in Agile product development and backlog management
  • Degree in Data Science, Computer Science, Business Analytics, or related field
  • Demonstrable experience of AI product delivery / ownership within insurance or financial services
  • Strong understanding of AI/ML concepts, model lifecycle, and MLOps
  • Knowledge of ethical AI principles and regulatory frameworks
  • Stakeholder influence, structured problem-solving, commercial acumen, clear communication, and change leadership
Job Responsibility
Job Responsibility
  • Define and communicate the vision, goals, and roadmap for AI products
  • Align AI initiatives with business strategy and regulatory compliance
  • Work with Head of AI to establish AI CoE and drive platform / product capability in collaboration with federated analytics teams in Finance, Underwriting, Actuarial and Claims
  • Own and prioritize the AI product backlog, balancing short-term delivery with long-term strategic goals
  • Translate business problems into AI use cases with clear actionable user stories and acceptance criteria
  • Ensure delivery practices (Agile/Scrum/Kanban), NFRs (performance, governance, resilience, privacy-by-design), and control frameworks (quality, metadata, lineage) are embedded
  • Drive delivery of AI products through agile ceremonies (sprint planning, reviews, retrospectives)
  • Monitor KPIs and value metrics to ensure AI products deliver measurable outcomes (e.g., loss ratio improvement, leakage reduction, NB premium, campaign ROI uplift, cycle-time reduction)
  • Maintain product-level scorecards and dashboards tracking usage, quality, SLA performance, and business impact
  • Manage seamless AI product integration into Value streams (martech/Claims, etc) and Change initiatives (Modernisation, Underwriting Transformation, etc.)
What we offer
What we offer
  • Attractive base compensation and discretionary performance related annual bonus
  • Private medical insurance coverage for both you and your dependents
  • Company paid life assurance and long term disability insurance
  • Pension with 10% employer contribution, plus a further matched contribution up to 2.5%
  • 25 days of annual leave (prorated for 1st calendar year of employment), plus public holidays, with the ability to flex the religious bank holidays to suit your religious beliefs, subject to approval. Additional annual leave purchase is also available up to a maximum of five additional days per year
  • Up to 2.5 days matched paid leave for volunteering at a charity of your choice and 5 days paid leave to provide or arrange care for a dependent with a long term care need annually
  • Up to £550 reimbursement towards home office setup
  • Free in-office lunch and monthly lifestyle allowance
  • Monthly travel allowance or annual season ticket
  • The opportunity to save for, and purchase, shares of Beazley stock
  • Fulltime
Read More
Arrow Right
New

Data Scientist

As part of our Client’s high-performing AI Innovation team, you’ll help design, ...
Location
Location
United States , New York
Salary
Salary:
200000.00 - 225000.00 USD / Year
solomonpage.com Logo
Solomon Page
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field (or equivalent practical experience)
  • 2+ years of experience as a Data Scientist, Machine Learning Engineer, or applied AI practitioner, with a strong foundation in computer science, algorithms, and software development
  • Advanced programming skills in Python, with experience building production-grade systems beyond research or experimentation
  • Solid understanding of machine learning and applied AI concepts, with experience taking solutions from prototype to production
  • Hands-on experience designing, building, and deploying LLM-driven or GenAI applications, including familiarity with vector databases, embeddings pipelines, or semantic search systems
  • Practical experience with cloud-based deployments and infrastructure tools (e.g., AWS, Docker, GitHub) and an understanding of modern DevOps practices, containerization, orchestration, caching strategies, and cost-aware design
  • Strong problem-solving skills and systems thinking, with the ability to balance trade-offs across model quality, scalability, inference latency, cost, and operational complexity
  • Ability to interpret and implement research ideas and algorithms, actively contributing to research and development initiatives while translating them into production solutions
  • Excellent communication and collaboration skills, with experience working closely with product managers, engineers, and domain experts to deliver actionable technical solutions
  • Passion for learning and staying current with the rapidly evolving AI/ML landscape, including emerging best practices for GenAI applications
Job Responsibility
Job Responsibility
  • Apply strong problem-solving and critical thinking skills to break down complex, ambiguous requirements into clear, implementable technical components and system designs
  • Design, build, and maintain AI-powered and data-driven systems with a focus on modern language and multimodal models, including LLM-driven applications, RAG pipelines, and agentic workflows
  • Evaluate and productionize commercial and open-source LLMs, choosing appropriate models, tools, and techniques for each use case
  • Develop multi-step agentic workflows that incorporate tools, external data sources, memory, and control logic
  • Manage the orchestration of production LLM workflows and agentic systems, ensuring reliability and efficiency through prompt routing, state management, retries, fallbacks, and error handling
  • Design, test, and iteratively refine prompts and system instructions using prompt engineering and tuning techniques to improve model reliability, accuracy, and task performance
  • Maintain production-grade code and services with automated monitoring and performance tracking, using metrics and alerts to guide continuous improvements in models, prompts, and pipelines
  • Apply systems thinking to design and optimize AI and LLM systems, balancing quality, scalability, latency, cost, and operational complexity, while implementing efficiency improvements using model selection, prompt design, batching, caching, and retrieval strategies
  • Define and implement evaluation and observability frameworks for AI systems, including automated testing, task-specific benchmarks, regression testing for prompts, human-in-the-loop validation, and performance monitoring
  • Build and integrate AI models into backend systems and APIs to support both real-time and batch inference, ensuring solutions are production-ready, scalable, and efficient
  • Fulltime
Read More
Arrow Right
New

Product Manager, Risk Automation

Block Risk Product builds the systems that power our automated, intelligent, and...
Location
Location
United States , Bay Area
Salary
Salary:
Not provided
block.xyz Logo
Block
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6-9 years of product management experience including a proven track record launching successful AI or ML driven products
  • Strong technical intuition about ML models, training loops, evaluation, prompt design, data pipelines, and real-time inference constraints
  • Experience designing and running structured experiments, including A/B tests and offline evaluations
  • An ownership mentality with the ability to rapidly drive complex, cross-functional initiatives from strategy through execution at scale
  • Excellent communicator, capable of distilling complex technical concepts into clear narratives for executive, operational, and technical audiences. Ability to work across scaled risk, compliance, and operational teams in a regulated industry
  • Familiarity with AI quality, safety, and responsible-use frameworks
  • Experience with LLMs, multimodal models, or applied generative AI
  • Experience building operational tooling that drives cost savings and quality
Job Responsibility
Job Responsibility
  • Own the strategy and roadmap for AI-powered products and capabilities
  • Identify AI opportunities grounded in user needs and measurable business value
  • Translate ambiguous problem spaces into clear product requirements
  • Work closely with ML engineers, data scientists, and researchers on model goals, data needs, evaluation methods, and quality gates
  • Build and run experiments to validate hypotheses and iterate quickly
  • Define how the product should behave across success, failure, and edge cases
  • Establish metrics for accuracy, safety, latency, reliability, and UX satisfaction
  • Partner with design on intuitive, transparent, and safe human-AI interactions
  • Collaborate with legal, security, and compliance teams to ensure responsible AI practices
  • Lead cross-functional execution from concept through launch and iteration
What we offer
What we offer
  • Remote work
  • medical insurance
  • flexible time off
  • retirement savings plans
  • modern family planning
  • Fulltime
Read More
Arrow Right