CrawlJobs Logo

Training: ML Framework Engineer

United States, San Francisco 205000.00 - 445000.00 USD / Year · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

Training Runtime designs the core distributed machine-learning training runtime that powers everything from early research experiments to frontier-scale model runs. With a dual mandate to accelerate researchers and enable frontier scale, we’re building a unified, modular runtime that meets researchers where they are and moves with them up the scaling curve. Our work focuses on three pillars: high-performance, asynchronous, zero-copy tensor and optimizer-state-aware data movement; performant, high-uptime, fault-tolerant training frameworks (training loop, state management, resilient checkpointing, deterministic orchestration, and observability); and distributed process management for long-lived, job-specific and user-provided processes. We integrate proven large-scale capabilities into a composable, developer-facing runtime so teams can iterate quickly and run reliably at any scale, partnering closely with model-stack, research, and platform teams. Success for us is measured by raising both training throughput (how fast models train) and researcher throughput (how fast ideas become experiments and products).

Job Responsibility

  • Apply the latest techniques in our internal training framework to achieve impressive hardware efficiency for our training runs
  • Profile and optimize our training framework
  • Work with researchers to enable them to develop the next generation of models

Requirements

  • Have run small scale ML experiments
  • Love figuring out how systems work and continuously come up with ideas for how to make them faster while minimizing complexity and maintenance burden
  • Have strong software engineering skills and are proficient in Python

What we offer

  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Relocation support for eligible employees
  • Additional taxable fringe benefits, such as charitable donation matching and wellness stipends, may also be provided
  • Offers Equity
  • Performance-related bonus(es) for eligible employees

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Training: ML Framework Engineer

8 matching positions

Staff Machine Learning Engineer - ML Training Infrastructure

The Role:   We are seeking an experienced, technically strong, impact-driven ex...
Location
Location
United States , Austin; Mountain View
Salary
Salary:
185000.00 - 335300.00 USD / Year
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree or higher in Computer Science or a related field, or equivalent practical experience
  • 8+ years of professional software engineering experience
  • 5+ years of specialized experience in AI/ML infrastructure, such as enabling distributed training for large-scale ML models
  • Strong programming skills in Python, with deep proficiency in frameworks such as PyTorch (preferred), TensorFlow, or similar ML systems
  • Proven experience designing and operating distributed systems for ML training, including distributed computing, GPU computing, and cloud environments (AWS, GCP, Azure)
  • Demonstrated track record of leading technically ambiguous, cross-team infrastructure initiatives and driving them to measurable impact
  • Strong architectural judgment and ability to make sound technical tradeoffs across performance, reliability, usability, and cost
  • Willingness to travel to Sunnyvale, CA as needed
  • Comfortable operating in highly ambiguous and dynamic environments
Job Responsibility
Job Responsibility
  • Define and drive the architecture, design, and development of scalable, reliable, and high-performance ML frameworks and platform capabilities to support model training at scale
  • Lead model training performance analysis and optimization efforts across distributed training workflows, improving scalability, efficiency, and cost across heterogeneous hardware environments
  • Raise the bar on system observability, debuggability, operational excellence, and developer experience across the ML training stack
  • Own large, ambiguous, cross-functional technical initiatives from strategy through execution, including technical roadmap definition, tradeoff analysis, and delivery
  • Influence platform direction by identifying long-term infrastructure investments, setting engineering standards, and driving adoption of best practices across teams
  • Collaborate across organizational boundaries to align requirements, resolve technical disagreements, and integrate new capabilities into the platform ecosystem
  • Mentor engineers through design reviews, technical guidance, and hands-on partnership, while elevating engineering quality across the team
What we offer
What we offer
  • medical
  • dental
  • vision
  • Health Savings Account
  • Flexible Spending Accounts
  • retirement savings plan
  • sickness and accident benefits
  • life insurance
  • paid vacation & holidays
  • tuition assistance programs
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer - ML Training Infrastructure

We are seeking an experienced, technical oriented, impact delivering-driven expe...
Location
Location
United States , Mountain View
Salary
Salary:
170000.00 - 240000.00 USD / Year
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelors degree or higher in Computer Science or equivalent major OR equivalent relevant experience
  • 3+ years professional software engineering experience
  • 2+ years specialized experience in AI/ML infrastructure, e.g., enabling distributed training for scaling large ML models
  • Strong programming skills in Python, with proficiency in frameworks such as, PyTorch (preferred), TensorFlow, or similar
  • Experience with distributed computing, GPU computing, and cloud environments (AWS, GCP, Azure)
  • Willingness to travel to Sunnyvale, CA as needed
  • Comfortable working in highly ambiguous and dynamic environments
Job Responsibility
Job Responsibility
  • Design and development of scalable, reliable, high-performance ML framework to support model training at scale
  • Model training performance analysis and optimization solutions to scale distributed training workflows and maximize resource utilization across heterogeneous hardware environments, and save cost
  • Raise the bar on system observability, debuggability, and operational excellence, and user experience
  • Collaborate with cross-functional teams to integrate new features and technologies into the platform
What we offer
What we offer
  • medical
  • dental
  • vision
  • Health Savings Account
  • Flexible Spending Accounts
  • retirement savings plan
  • sickness and accident benefits
  • life insurance
  • paid vacation & holidays
  • tuition assistance programs
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer, ML Training Platform

Location
Location
United States
Salary
Salary:
216700.00 - 303400.00 USD / Year
Reddit
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of software engineering experience, with a focus on Platform Engineering, ML Infrastructure, or Backend Systems
  • Deep Kubernetes Expertise: You know K8s beyond just 'deploying pods.' You understand CRDs, Controllers and the Operator pattern
  • Jupyter Ecosystem Knowledge: Experience customizing JupyterHub, JupyterLab extensions, or building similar interactive computing platforms
  • Strong Coding Skills: Proficiency in Python (for the ML ecosystem) and Go (for Kubernetes controllers/infrastructure tooling)
  • GPU Experience: Hands-on practice with CUDA environments, GPU virtualization/containerization, and doing it all within Kubernetes
  • Cloud Provider Experience: Familiarity with both managed ML offerings (Vertex AI, Sagemaker, etc) and building custom ML components in AWS and/or GCP
  • Experience working with distributed training frameworks, including Ray and Kubernetes
  • Comfortable with distributed systems, big data (Petabyte scale) and data-intensive systems
  • Strong focus on scalability, reliability, performance, and ease of use. You are an undying advocate for platform users and have a deep intuition for the machine learning development lifecycle
  • Strong organizational & communication skills
Job Responsibility
Job Responsibility
  • Lead the building, testing, and maintenance of ML training infrastructure at Reddit
  • Play a pivotal role in designing, building, and optimizing the infrastructure and tooling required to support large-scale machine learning workflows
  • Evolve the MLE experience, from provisioning interactive GPU environments through large-scale training, supporting on-demand and self-service workflows
  • Kubernetes Automation: Write custom Kubernetes Controllers and Operators to manage the lifecycle of interactive Jupyter workspaces and long-running ML training jobs, handle auto-idling, and ensure fault tolerance
  • GPU Orchestration: Work with the underlying compute team to ensure MLEs have efficient access to training hardware resources and handle resource contention gracefully
  • Developer Experience (DevX): Treat internal MLEs as your customers. Conduct user research, reduce friction in the 'Idea-to-Prototype' loop, and standardize software environments (Docker images, Python dependency management)
What we offer
What we offer
  • Comprehensive Healthcare Benefits and Income Replacement Programs
  • 401k Match
  • Family Planning Support
  • Gender-Affirming Care
  • Mental Health & Coaching Benefits
  • Flexible Vacation & Reddit Global Days off
  • Generous paid Parental Leave
  • Paid Volunteer time off
  • Fulltime
Read More
Arrow Right

Principal ML Engineer - Large Scale Training Performance Optimization

We are looking for a Principal Machine Learning Engineer to join our Models and ...
Location
Location
United States , San Jose; Bellevue
Salary
Salary:
226400.00 - 339600.00 USD / Year
amd.com Logo
AMD
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience with ML/DL frameworks such as PyTorch, JAX, or TensorFlow
  • Experience with distributed training and distributed training frameworks, such as Megatron-LM, MaxText, TorchTitan
  • Experience with LLMs or computer vision, especially large models
  • Experience with GPU kernel optimization
  • Excellent Python or C++ programming skills, including debugging, profiling, and performance analysis at scale
  • Experience with ML infra at kernel, framework, or system level
  • Strong communication and problem-solving skills
  • A master's degree or PhD degree in Computer Science, Artificial Intelligence, Machine Learning, or a related field
Job Responsibility
Job Responsibility
  • Train large models to convergence on AMD GPUs at scale
  • Improve the end-to-end training pipeline performance
  • Optimize the distributed training pipeline and algorithm to scale out
  • Contribute your changes to open source
  • Stay up-to-date with the latest training algorithms
  • Influence the direction of AMD AI platform
  • Collaborate across teams with various groups and stakeholders
  • Fulltime
Read More
Arrow Right

ML Research Engineer, ML Systems

Scale’s ML platform (RLXF) team builds our internal distributed framework for la...
Location
Location
United States , San Francisco; Seattle; New York
Salary
Salary:
218400.00 - 273000.00 USD / Year
scale.com Logo
Scale
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong excitement about system optimization
  • Experience with multi-node LLM training and inference
  • Experience with developing large-scale distributed ML systems
  • Strong software engineering skills, proficient in frameworks and tools such as CUDA, Pytorch, transformers, flash attention, etc.
  • Strong written and verbal communication skills and the ability to operate in a cross functional team environment
Job Responsibility
Job Responsibility
  • Build, profile and optimize our training and inference framework
  • Collaborate with ML teams to accelerate their research and development and enable them to develop the next generation of models and data curation
  • Research and integrate state-of-the-art technologies to optimize our ML system
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
  • additional benefits such as a commuter stipend
  • equity based compensation
  • Fulltime
Read More
Arrow Right

Senior Ml Engineer

Location
Location
Colombia , Medellín, Antioquia;Bogotá, Capital District;Cali, Valle del Cauca;Barranquilla;Bucaramanga, Santander
Salary
Salary:
Not provided
provectus.com Logo
Provectus
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • ML Fundamentals: supervised, unsupervised, and reinforcement learning
  • Model Development: feature engineering, model training, evaluation, hyperparameter tuning, and validation
  • ML Frameworks: classical ML libraries, TensorFlow, PyTorch, or similar frameworks
  • Deep Learning: CNNs, RNNs, Transformers
  • LLM Applications: Experience building production LLM-based applications
  • Prompt Engineering: Ability to design effective prompts and chain-of-thought strategies
  • RAG Systems: Experience building retrieval-augmented generation architectures
  • Vector Databases: Familiarity with embedding models and vector search
  • LLM Evaluation: Experience with evaluation metrics and techniques for LLM outputs
  • Python: Advanced proficiency in Python for ML applications
Job Responsibility
Job Responsibility
  • Design and implement end-to-end ML solutions from experimentation to production
  • Build scalable ML pipelines and infrastructure
  • Optimize model performance, efficiency, and reliability
  • Write clean, maintainable, production-quality code
  • Conduct rigorous experimentation and model evaluation
  • Troubleshoot and resolve complex technical challenges
  • Mentor junior and mid-level ML engineers
  • Conduct code reviews and provide constructive feedback
  • Share knowledge through documentation, presentations, and workshops
  • Collaborate with cross-functional teams (DevOps, Data Engineering, SAs)
What we offer
What we offer
  • Long-term B2B collaboration
  • Fully remote setup
  • A budget for your medical insurance
  • Paid sick leave, vacation, public holidays
  • Continuous learning support, including unlimited AWS certification sponsorship
  • Fulltime
Read More
Arrow Right

Ml Engineer, End-to-End Autonomy

We're Blue River, a team of innovators driven to create intelligent machinery th...
Location
Location
United States , Santa Clara
Salary
Salary:
135000.00 USD / Year
bluerivertechnology.com Logo
Blue River Technology
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven track record in developing ML models. Prefer experience deploying to production for end-to-end robotics and boosting key metrics
  • Expertise in designing and developing software for complex systems
  • Comfortable working on new hardware systems and working on new ML/software problems
  • Strong Python coding skills and proficiency with deep learning frameworks like PyTorch
  • Comfortable working across traditional team boundaries to deliver results
  • Excellent brainstorming, creative thinking, mathematical analysis, and communication skills
  • Track record of regularly anticipating technical issues and making architectural and design decisions to avoid them
Job Responsibility
Job Responsibility
  • Design and implement end-to-end policies for complex navigation and manipulation tasks with long horizons
  • Enhance policy robustness to challenges, including environmental variability, machine wear, and deployment across machine forms
  • Drive the full data + modeling life cycle, from data collection requirements, experimental design, evaluation frameworks, model training, and deployment
  • Develop pretraining strategies leveraging multimodal data at scale
  • Edge deployment: help deliver performance with constrained compute
  • Design and implement systems for active learning on the edge
  • Collaborate with infrastructure engineers on scalable data pipelines, including ingest, curation, and processing training artifacts
  • Collaborate with infrastructure engineers on scalable distributed training pipelines
  • Collaborate with reinforcement learning, robotics, and hardware engineers to integrate ML seamlessly into the full E2E autonomy stack
  • Work with product management to learn from customers, including how they want to interface with the product
  • Fulltime
Read More
Arrow Right

Senior ML Engineer (GenAI, AWS)

Provectus helps companies adopt ML/AI to transform the ways they operate, compet...
Location
Location
Colombia , Medellín; Bogotá; Cali; Barranquilla; Bucaramanga
Salary
Salary:
Not provided
provectus.com Logo
Provectus
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • ML Fundamentals: supervised, unsupervised, and reinforcement learning
  • Model Development: feature engineering, model training, evaluation, hyperparameter tuning, and validation
  • ML Frameworks: classical ML libraries, TensorFlow, PyTorch, or similar frameworks
  • Deep Learning: CNNs, RNNs, Transformers
  • LLM Applications: Experience building production LLM-based applications
  • Prompt Engineering: Ability to design effective prompts and chain-of-thought strategies
  • RAG Systems: Experience building retrieval-augmented generation architectures
  • Vector Databases: Familiarity with embedding models and vector search
  • LLM Evaluation: Experience with evaluation metrics and techniques for LLM outputs
  • Python: Advanced proficiency in Python for ML applications
Job Responsibility
Job Responsibility
  • Design and implement end-to-end ML solutions from experimentation to production
  • Build scalable ML pipelines and infrastructure
  • Optimize model performance, efficiency, and reliability
  • Write clean, maintainable, production-quality code
  • Conduct rigorous experimentation and model evaluation
  • Troubleshoot and resolve complex technical challenges
  • Mentor junior and mid-level ML engineers
  • Conduct code reviews and provide constructive feedback
  • Share knowledge through documentation, presentations, and workshops
  • Collaborate with cross-functional teams (DevOps, Data Engineering, SAs)
What we offer
What we offer
  • Long-term B2B collaboration
  • Fully remote setup
  • A budget for your medical insurance
  • Paid sick leave, vacation, public holidays
  • Continuous learning support, including unlimited AWS certification sponsorship
  • Fulltime
Read More
Arrow Right