CrawlJobs Logo
Scale Logo Scale · IT - Software Development

Senior Machine Learning Engineer - Model Evaluations

United States, San Francisco 216300.00 - 300300.00 USD / Year · Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

The Public Sector ML team at Scale deploys advanced AI systems—including LLMs, agentic models, and multimodal pipelines—into mission-critical government environments. We build evaluation frameworks that ensure these models operate reliably, safely, and effectively under real-world constraints. As an ML Engineer, you will design, implement, and scale automated evaluation pipelines that help customers trust and operationalize advanced AI systems across defense, intelligence, and federal missions.

Job Responsibility

  • Develop and maintain automated evaluation pipelines for ML models across functional, performance, robustness, and safety metrics, including LLM-judge–based evaluations
  • Design test datasets and benchmarks to measure generalization, bias, explainability, and failure modes
  • Build evaluation frameworks for LLM agents, including infrastructure for scenario-based and environment-based testing
  • Conduct comparative analyses of model architectures, training procedures, and evaluation outcomes
  • Implement tools for continuous monitoring, regression testing, and quality assurance for ML systems
  • Design and execute stress tests and red-teaming workflows to uncover vulnerabilities and edge cases
  • Collaborate with operations teams and subject matter experts to produce high-quality evaluation datasets

Requirements

  • Experience in computer vision, deep learning, reinforcement learning, or NLP in production settings
  • Strong programming skills in Python
  • experience with TensorFlow or PyTorch
  • Background in algorithms, data structures, and object-oriented programming
  • Experience with LLM pipelines, simulation environments, or automated evaluation systems
  • Ability to convert research insights into measurable evaluation criteria
  • This role will require an active security clearance or the ability to obtain a security clearance

Nice to have

  • Graduate degree in CS, ML, or AI
  • Cloud experience (AWS, GCP) and model deployment experience
  • Experience with LLM evaluation, CV robustness, or RL validation
  • Knowledge of interpretability, adversarial robustness, or AI safety frameworks
  • Familiarity with ML evaluation frameworks and agentic model design
  • Experience in regulated, classified, or mission-critical ML domains

What we offer

  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
Scale - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Senior Machine Learning Engineer - Model Evaluations

8 matching positions

Senior Machine Learning Engineer, Model Risk Management

Block lends, moves money, and screens for financial crime at enormous scale, and...
Location
Location
United States , New York
Salary
Salary:
194500.00 - 343100.00 USD / Year
cash.app Logo
Cash App
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A quantitative degree or equivalent experience, and senior-IC depth building or validating models in a high-stakes domain such as credit, fraud, or financial crime
  • Command of effective-challenge methodology: reproduction, conceptual-soundness review, benchmarking, stress testing, and outcomes analysis, with an eye for how a model holds up after launch and where its assumptions break
  • Deep applied ML and statistics across model families, from regression and tree ensembles to deep learning, with sound judgment about evaluation, calibration, and generalization
  • Experimentation and statistical rigor: holdout and experiment design, reasoning about uncertainty, and evaluating a model beyond aggregate accuracy
  • Solid software and data engineering: production-quality Python, SQL on large datasets, and reproducible, tested code
  • Fluency with modern AI: building with LLMs and agentic tools, and the judgment to know when their output can be trusted
  • Familiarity with model risk management frameworks and fair-lending standards, with the specifics learnable on the job
  • The communication to explain and defend your conclusions to model owners and senior stakeholders, and the independence to operate under ambiguity
Job Responsibility
Job Responsibility
  • Independently challenge model owners across lending, fraud, and AML: reproduce their results, set and defend the acceptance thresholds, and own the call on whether a model is sound
  • Hunt the silent errors that make metrics lie, and prove them out before they reach production
  • Choose evaluation that holds up under real conditions: rare events, shifting populations, and drift that only shows up after launch
  • Work hands-on in codebases you did not write, learning the data, configs, and conventions, and ship production code in the tooling you build to validate them
  • Build the agentic validation tooling the team depends on, orchestrating agents that run in parallel
  • Reason about ML systems end to end — how features, training, serving, monitoring, and scale fit together — to evaluate and challenge an owner's design
  • Tie explainability and fair-lending findings on consumer credit models back to the model and product decisions that follow
  • Help define how Block validates the systems at the frontier of production AI, setting standards where none exist yet
What we offer
What we offer
  • Remote work
  • Medical insurance
  • Flexible time off
  • Retirement savings plans
  • Modern family planning
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer, Reinforcement Learning

About the Role: We are looking for a Senior Machine Learning Engineer with stron...
Location
Location
United States , Beverly Hills, CA
Salary
Salary:
150000.00 - 185000.00 USD / Year
snail.com Logo
Snail Games
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong foundation in machine learning, with hands-on experience building, training, and iterating on applied ML systems
  • Professional or substantial project experience with reinforcement learning, agent-based systems, sequential decision-making, or closely related areas
  • Strong Python skills and experience with modern ML frameworks such as PyTorch
  • Experience designing experiments, evaluating model behavior, and improving results through systematic iteration
  • T-shaped capability: deep machine learning expertise plus practical range across one or more adjacent areas such as simulation, evaluation, model integration, systems collaboration, or robotics-adjacent machine learning
  • Strong problem-solving ability, sound judgment, and comfort working in ambiguous, fast-changing environments
  • Respectful, low-ego collaborative style and willingness to work beyond a narrow specialty when the work requires it
Job Responsibility
Job Responsibility
  • Design, train, and iterate on machine learning models for intelligent agents and decision-making systems, with an emphasis on reinforcement learning and related approaches
  • Define and refine state representations, action spaces, reward structures, and evaluation criteria to improve agent behavior
  • Build and improve practical experimentation and training workflows, including data generation, experiment tracking, and reproducibility
  • Analyze results, debug model behavior, and make pragmatic tradeoffs between model performance, iteration speed, and system complexity
  • Work closely with engineers and other partners to help integrate successful ML work into usable product systems
  • Contribute thoughtful technical input on next-step experiments, tooling, and ML direction as Egofold continues to evolve
What we offer
What we offer
  • True focus on work/life balance
  • Paid company holidays, vacation, and separate sick leave
  • Medical, dental, vision, and Life/LTD
  • 401k with company match
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer

We’re hiring a Senior Machine Learning Engineer to help build the core AI system...
Location
Location
United States , San Francisco
Salary
Salary:
230000.00 - 300000.00 USD / Year
signifytechnology.com Logo
Signify Technology
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong experience building production AI systems around LLMs, OCR, and unstructured data workflows
  • Proven track record shipping applied AI products, not just prototyping models offline
  • Deep familiarity with modern LLM workflows including prompting, structured outputs, tool use, retries, fallbacks, guardrails, and model evaluation
  • Experience with document intelligence systems such as OCR pipelines, document extraction, classification, post-processing, and confidence-based review flows
  • Experience with voice or conversational AI, or adjacent systems involving transcripts, call automation, and conversational extraction
  • Strong proficiency in Python and comfort working in production codebases with APIs, queues, and backend services
  • Experience deploying and operating AI systems in AWS or similar cloud environments, including serverless or event-driven architectures
  • Strong instincts around evaluation, benchmarking, monitoring, and quality assurance for real-world AI systems
  • Ability to work across structured and unstructured data and design systems that are robust to noisy, incomplete, and ambiguous inputs
Job Responsibility
Job Responsibility
  • Build the core AI systems behind a next-generation healthcare platform that turns smartphone video into clinically accurate 3D models of human anatomy
  • Own the pipeline that bridges raw computer vision data and physical 3D-printed medical solutions, transforming noisy real-world scans into precise, CAD-compatible models used to improve patient outcomes
  • Work closely with engineers, researchers, and product leaders to design systems that translate cutting-edge ML research into reliable production technology used in healthcare
What we offer
What we offer
  • Equity
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer

Reddit is a community of communities. It’s built on shared interests, passion, a...
Location
Location
United States , Remote
Salary
Salary:
216700.00 - 303400.00 USD / Year
Reddit
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3-5+ years of experience building, deploying, and operating machine learning systems in production
  • Strong programming skills in Python, Java, Go, or similar languages, with solid software engineering fundamentals
  • ML Fundamentals: a strong grasp of algorithms, from classic statistical learning (XGBoost, Random Forests, regressions) to DL architectures (Transformers, CNNs, GNNs)
  • Hands-on experience with modern ML frameworks (e.g., PyTorch, TensorFlow)
  • Experience designing scalable ML pipelines, data processing systems, and model serving infrastructure
  • Ability to work cross-functionally and translate ambiguous product or business problems into technical solutions
  • Experience improving measurable metrics through applied machine learning
Job Responsibility
Job Responsibility
  • Design, build, and deploy production-grade machine learning models and systems at scale
  • Own the full ML lifecycle: from problem definition and feature engineering to training, evaluation, deployment, and monitoring
  • Build scalable data and model pipelines with strong reliability, observability, and automated retraining
  • Work with large-scale datasets to improve ranking, recommendations, search relevance, prediction, content/user understanding, and optimization systems
  • Partner cross-functionally with Product, Data Science, Infrastructure, and Engineering teams to translate complex problems into ML solutions
  • Improve system performance across latency, throughput, and model quality metrics
  • Research and apply state-of-the-art machine learning and AI techniques, including deep learning, graph & transformers based, and LLM evaluation/alignment
  • Contribute to technical strategy, architecture, and long-term ML roadmap
What we offer
What we offer
  • Comprehensive Healthcare Benefits and Income Replacement Programs
  • 401k with Employer Match
  • Global Benefit programs that fit your lifestyle, from workspace to professional development to caregiving support
  • Family Planning Support
  • Gender-Affirming Care
  • Mental Health & Coaching Benefits
  • Flexible Vacation & Paid Volunteer Time Off
  • Generous Paid Parental Leave
  • Medical, dental, and vision insurance
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer, Simulation Evaluation

Waymo is an autonomous driving technology company with the mission to be the wor...
Location
Location
United States , Mountain View
Salary
Salary:
213000.00 - 263000.00 USD / Year
a16z
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's, Master's, or PhD in computer science, machine learning, robotics, or a related field
  • Five or more years of experience in machine learning engineering or applied deep learning, supported by a portfolio of shipped products or peer-reviewed publications
  • Proficient programming skills in Python and hands-on experience with modern machine learning frameworks such as Jax, Flax, or PyTorch
  • Experience designing and implementing evaluation frameworks for complex systems or machine learning models.
Job Responsibility
Job Responsibility
  • Lead the design, development and deployment of cutting-edge evaluation approaches to assess realism of state-of-the-art multimodel world models and generative systems for simulation use cases at Waymo
  • Architect and implement robust and scalable machine learning pipelines for tuning, evaluating, and deploying large-scale discriminator models for the purposes of simulator realism evaluation
  • Evaluate open-source and production-ready video generation techniques that measure realism (e.g. temporal stability, multi-modal consistency, geometric discrepancy, condition following, etc.)
  • Apply vision language models to evaluate semantic understanding and controllability across our world simulation products
  • Collaborate with research teams across Waymo and Alphabet to integrate advancements in 4D world modeling and generative AI into production systems
  • Mentor engineers on the team and provide technical guidance on architecture and execution.
What we offer
What we offer
  • Waymo’s discretionary annual bonus program
  • equity incentive plan
  • generous Company benefits program
  • Health and wellness
  • Financial wellness
  • Flexibility and time off
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer

Location
Location
Greece , Athens
Salary
Salary:
60000.00 - 65000.00 EUR / Year
signifytechnology.com Logo
Signify Technology
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience designing, building, and deploying production-grade machine learning solutions
  • Strong Python development skills
  • Hands-on experience with Large Language Models (LLMs), LangChain, and Hugging Face Transformers
  • Experience with deep learning frameworks such as PyTorch
  • Strong knowledge of machine learning libraries including Scikit-learn and XGBoost
  • Experience working with NLP frameworks such as spaCy and NLTK
  • Strong understanding of the end-to-end machine learning lifecycle, including model training, validation, deployment, and monitoring
  • Experience collaborating across engineering, product, and business teams to deliver AI-driven solutions
  • Passion for staying current with emerging AI technologies and applying them to real-world business challenges
  • Strong communication skills with the ability to explain complex technical concepts to a range of stakeholders
Job Responsibility
Job Responsibility
  • Design, develop, and deploy advanced AI and machine learning solutions that support strategic business objectives
  • Lead the research, development, and optimisation of machine learning models for production environments
  • Build, fine-tune, and evaluate Large Language Models (LLMs) and modern AI applications using tools such as LangChain and Hugging Face
  • Develop solutions using Python alongside deep learning, machine learning, and natural language processing frameworks
  • Collaborate with software engineering, data, and product teams to identify opportunities for AI-driven innovation
  • Research and evaluate emerging AI technologies, ensuring the business remains at the forefront of industry developments
  • Monitor, maintain, and optimise model performance to ensure reliability and scalability in production
  • Translate business challenges into practical AI solutions, balancing speed of delivery with technical excellence
  • Mentor junior engineers and provide technical leadership across AI and machine learning initiatives
  • Champion best practices for model development, experimentation, deployment, and ongoing optimisation
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer

Roku is the #1 TV streaming platform in the U.S., Canada, and Mexico, and we've ...
Location
Location
United Kingdom , Cambridge
Salary
Salary:
Not provided
roku.com Logo
Roku
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelors, Masters, or PhD in Computer Science, Statistics, or a related field
  • Demonstrated production experience in applied machine learning, with PhD-level research and implementation experience considered a strong accelerator
  • Great coding skills and strong software development experience (we use Spark, Python, Java)
  • Familiarity with real-time evaluation of models with low latency constraints
  • Familiarity with distributed ML frameworks such as Spark-MLlib, TensorFlow, etc
  • Ability to work with large scale computing frameworks, data analysis systems, and modeling environments. Examples include Spark, Hive, NoSQL stores such as Aerospike and ScyllaDB
Job Responsibility
Job Responsibility
  • ML infrastructure: Help build a first-class machine learning platform from the ground up which manages the entire model lifecycle - feature engineering, model training, versioning, deployment, online serving/evaluation, and monitoring prediction quality
  • Data analysis and feature engineering: Apply your expertise to identify and generate features that can be leveraged by multiple use cases and models
  • Model training with batch and real-time prediction scenarios: Use machine learning and statistical modeling techniques such as Decision Trees, Logistic Regression, Neural Networks, Bayesian Analysis and others to develop and evaluate algorithms for improving product/system performance, quality, and accuracy
  • Production operations: Low-level systems debugging, performance measurement, and optimization on large production clusters
  • Collaboration with cross-functional teams: Partner with product managers, data scientists, and other engineers to deliver impactful solutions
  • Staying ahead of the curve: Continuously learn and adapt to emerging technologies and industry trends
What we offer
What we offer
  • Global access to mental health and financial wellness support and resources
  • Local benefits include statutory and voluntary benefits which may include healthcare (medical, dental, and vision), life, accident, disability, commuter, and retirement options (401(k)/pension)
  • Employees are supported in taking time off, in accordance with local leave policies and other personal needs
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer

Senior Machine Learning Engineer at Amgen. Join Amgen’s Mission of Serving Patie...
Location
Location
United States , Thousand Oaks; Jacksonville
Salary
Salary:
156190.05 - 211315.95 USD / Year
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Solid foundation in machine learning algorithms and techniques
  • Experience in MLOps practices and tools (e.g., MLflow, Kubeflow, Airflow)
  • Experience in DevOps tools (e.g., Docker, Kubernetes, CI/CD)
  • Proficiency in Python and relevant ML libraries (e.g., TensorFlow, PyTorch, Scikit-learn)
  • Outstanding analytical and problem-solving skills
  • Ability to learn quickly
  • Good communication and interpersonal skills
  • Doctorate degree OR Master’s degree and 2 years of Computer Science experience OR Bachelor’s degree and 4 years of Computer Science experience OR Associate’s degree and 8 years of Computer Science experience OR High school diploma / GED and 10 years of Computer Science experience
  • Excellent analytical and troubleshooting skills
  • Strong verbal and written communication skills
Job Responsibility
Job Responsibility
  • Collaborate with data scientists to develop, train, and evaluate machine learning models
  • Build and maintain MLOps pipelines, including data ingestion, feature engineering, model training, deployment, and monitoring
  • Leverage cloud platforms (AWS, GCP, Azure) for ML model development, training, and deployment
  • Implement DevOps/MLOps best practices to automate ML workflows and improve efficiency
  • Develop and implement monitoring systems to track model performance and identify issues
  • Conduct A/B testing and experimentation to optimize model performance
  • Work closely with data scientists, engineers, and product teams to deliver ML solutions
  • Stay updated with the latest trends and advancements
What we offer
What we offer
  • A comprehensive employee benefits package, including a Retirement and Savings Plan with generous company contributions, group medical, dental and vision coverage, life and disability insurance, and flexible spending accounts
  • A discretionary annual bonus program, or for field sales representatives, a sales-based incentive plan
  • Stock-based long-term incentives
  • Award-winning time-off plans
  • Flexible work models, including remote and hybrid work arrangements, where possible
  • Fulltime
Read More
Arrow Right