CrawlJobs Logo
Prior Labs Logo Prior Labs · IT - Software Development

ML Engineer, Training Infrastructure

Germany; United States, Berlin · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

You’ll take on challenging engineering tasks crucial to the development of tabular foundation models. You’ll work on building and maintaining best-in-class training infrastructure, while maintaining our developer productivity tooling and open source projects. You’ll work closely with researchers to ensure that we can iterate quickly and scale our models.

Job Responsibility

  • Training & research compute infrastructure: Own our cloud GPU cluster (operations, reliability, and cost/performance) currently based on Slurm. Design and implement future versions as our compute needs scale and we expand across multiple cloud/HPC providers
  • Training & inference performance: Work closely with researchers to identify and resolve performance bottlenecks in distributed training and inference. Support high hardware utilization and efficient memory usage through systems-level debugging, profiling, and infrastructure improvements
  • Developer productivity: Manage our internal repositories on GitHub and keep their CI and other pipelines speedy. Ensure our experiment tracking, model registry, data processing pipelines are working smoothly
  • Try out your own ideas! We operate an open environment. If you’ve got the next SOTA tabular architecture up your sleeve, go ahead and train it

Requirements

  • Exceptional software engineering fundamentals and expert-level Python proficiency, with 5+ years of hands-on industry experience building and operating production systems
  • Proven track record of designing and building complex, scalable software, preferably for data processing or distributed systems
  • Deep, practical knowledge of the modern ML ecosystem (PyTorch, scikit-learn, etc.) and a genuine interest in applying systems thinking to solve hard problems in AI
  • Core MLOps Concepts: Strong understanding of the entire machine learning lifecycle (MLLC) from data ingestion and preparation to model deployment, monitoring, and retraining. Familiarity with MLOps principles and best practices (e.g., reproducibility, versioning, automation, continuous integration/delivery for ML)

What we offer

  • Competitive compensation package with meaningful equity
  • 30 days of paid vacation + public holidays
  • Comprehensive benefits including healthcare, transportation, and fitness
  • Work with state-of-the-art ML architecture, substantial compute resources and with a world-class team
Prior Labs - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

ML Engineer, Training Infrastructure

8 matching positions

Staff Machine Learning Engineer - ML Training Infrastructure

The Role:   We are seeking an experienced, technically strong, impact-driven ex...
Location
Location
United States , Austin; Mountain View
Salary
Salary:
185000.00 - 335300.00 USD / Year
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree or higher in Computer Science or a related field, or equivalent practical experience
  • 8+ years of professional software engineering experience
  • 5+ years of specialized experience in AI/ML infrastructure, such as enabling distributed training for large-scale ML models
  • Strong programming skills in Python, with deep proficiency in frameworks such as PyTorch (preferred), TensorFlow, or similar ML systems
  • Proven experience designing and operating distributed systems for ML training, including distributed computing, GPU computing, and cloud environments (AWS, GCP, Azure)
  • Demonstrated track record of leading technically ambiguous, cross-team infrastructure initiatives and driving them to measurable impact
  • Strong architectural judgment and ability to make sound technical tradeoffs across performance, reliability, usability, and cost
  • Willingness to travel to Sunnyvale, CA as needed
  • Comfortable operating in highly ambiguous and dynamic environments
Job Responsibility
Job Responsibility
  • Define and drive the architecture, design, and development of scalable, reliable, and high-performance ML frameworks and platform capabilities to support model training at scale
  • Lead model training performance analysis and optimization efforts across distributed training workflows, improving scalability, efficiency, and cost across heterogeneous hardware environments
  • Raise the bar on system observability, debuggability, operational excellence, and developer experience across the ML training stack
  • Own large, ambiguous, cross-functional technical initiatives from strategy through execution, including technical roadmap definition, tradeoff analysis, and delivery
  • Influence platform direction by identifying long-term infrastructure investments, setting engineering standards, and driving adoption of best practices across teams
  • Collaborate across organizational boundaries to align requirements, resolve technical disagreements, and integrate new capabilities into the platform ecosystem
  • Mentor engineers through design reviews, technical guidance, and hands-on partnership, while elevating engineering quality across the team
What we offer
What we offer
  • medical
  • dental
  • vision
  • Health Savings Account
  • Flexible Spending Accounts
  • retirement savings plan
  • sickness and accident benefits
  • life insurance
  • paid vacation & holidays
  • tuition assistance programs
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer - ML Training Infrastructure

We are seeking an experienced, technical oriented, impact delivering-driven expe...
Location
Location
United States , Mountain View
Salary
Salary:
170000.00 - 240000.00 USD / Year
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelors degree or higher in Computer Science or equivalent major OR equivalent relevant experience
  • 3+ years professional software engineering experience
  • 2+ years specialized experience in AI/ML infrastructure, e.g., enabling distributed training for scaling large ML models
  • Strong programming skills in Python, with proficiency in frameworks such as, PyTorch (preferred), TensorFlow, or similar
  • Experience with distributed computing, GPU computing, and cloud environments (AWS, GCP, Azure)
  • Willingness to travel to Sunnyvale, CA as needed
  • Comfortable working in highly ambiguous and dynamic environments
Job Responsibility
Job Responsibility
  • Design and development of scalable, reliable, high-performance ML framework to support model training at scale
  • Model training performance analysis and optimization solutions to scale distributed training workflows and maximize resource utilization across heterogeneous hardware environments, and save cost
  • Raise the bar on system observability, debuggability, and operational excellence, and user experience
  • Collaborate with cross-functional teams to integrate new features and technologies into the platform
What we offer
What we offer
  • medical
  • dental
  • vision
  • Health Savings Account
  • Flexible Spending Accounts
  • retirement savings plan
  • sickness and accident benefits
  • life insurance
  • paid vacation & holidays
  • tuition assistance programs
  • Fulltime
Read More
Arrow Right

Senior AI Infrastructure Engineer - Training Platform

As a Software Engineer on the Machine Learning Infrastructure team, you will bui...
Location
Location
United States , San Francisco; Seattle; New York
Salary
Salary:
216000.00 - 270000.00 USD / Year
scale.com Logo
Scale
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of experience in backend or infrastructure engineering, with at least 2 years focused on orchestrating ML workloads at scale (100+ GPU nodes)
  • Strong programming skills in one or more languages (e.g. Python, Go, Rust, C++)
  • Experience with complex compute management systems that cover queueing, quotas, preemption, and gang scheduling
  • Experience with distributed training infrastructure, such as EFA, Infiniband, and topology-aware scheduling
  • Experience with distributed storage systems (e.g. Lustre, S3) as they relate to training throughput
  • Expert-level knowledge of Kubernetes internals (Custom Resources, Operators, Admission Controllers) and how they interact with device plugins for specialized hardware
  • Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform)
  • Proven ability to solve complex problems and work independently in fast-moving environments
Job Responsibility
Job Responsibility
  • Architect and scale a multi-tenant orchestration layer that abstracts away the complexity of GPU clusters, ensuring high utilization and seamless job recovery
  • Design and implement scheduling primitives to optimize the lifecycle of training jobs
  • Develop deep observability and automated health-checking into the training stack to proactively identify and isolate hardware failures
  • Evaluate and integrate emerging technologies in the CNCF and AI ecosystem (e.g. Ray, Kueue), making data-driven build vs. buy decisions that balance velocity with long-term maintainability
  • Work closely with Finance and Procurement teams to drive our capacity planning process
  • Participate in our team's on call process to ensure the availability of our services
  • Own projects end-to-end, from requirements, scoping, design, to implementation, in a highly collaborative and cross-functional environment
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
  • commuter stipend (may be eligible)
  • Fulltime
Read More
Arrow Right

Senior Machine Learning Engineer, ML Training Platform

Location
Location
United States
Salary
Salary:
216700.00 - 303400.00 USD / Year
Reddit
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of software engineering experience, with a focus on Platform Engineering, ML Infrastructure, or Backend Systems
  • Deep Kubernetes Expertise: You know K8s beyond just 'deploying pods.' You understand CRDs, Controllers and the Operator pattern
  • Jupyter Ecosystem Knowledge: Experience customizing JupyterHub, JupyterLab extensions, or building similar interactive computing platforms
  • Strong Coding Skills: Proficiency in Python (for the ML ecosystem) and Go (for Kubernetes controllers/infrastructure tooling)
  • GPU Experience: Hands-on practice with CUDA environments, GPU virtualization/containerization, and doing it all within Kubernetes
  • Cloud Provider Experience: Familiarity with both managed ML offerings (Vertex AI, Sagemaker, etc) and building custom ML components in AWS and/or GCP
  • Experience working with distributed training frameworks, including Ray and Kubernetes
  • Comfortable with distributed systems, big data (Petabyte scale) and data-intensive systems
  • Strong focus on scalability, reliability, performance, and ease of use. You are an undying advocate for platform users and have a deep intuition for the machine learning development lifecycle
  • Strong organizational & communication skills
Job Responsibility
Job Responsibility
  • Lead the building, testing, and maintenance of ML training infrastructure at Reddit
  • Play a pivotal role in designing, building, and optimizing the infrastructure and tooling required to support large-scale machine learning workflows
  • Evolve the MLE experience, from provisioning interactive GPU environments through large-scale training, supporting on-demand and self-service workflows
  • Kubernetes Automation: Write custom Kubernetes Controllers and Operators to manage the lifecycle of interactive Jupyter workspaces and long-running ML training jobs, handle auto-idling, and ensure fault tolerance
  • GPU Orchestration: Work with the underlying compute team to ensure MLEs have efficient access to training hardware resources and handle resource contention gracefully
  • Developer Experience (DevX): Treat internal MLEs as your customers. Conduct user research, reduce friction in the 'Idea-to-Prototype' loop, and standardize software environments (Docker images, Python dependency management)
What we offer
What we offer
  • Comprehensive Healthcare Benefits and Income Replacement Programs
  • 401k Match
  • Family Planning Support
  • Gender-Affirming Care
  • Mental Health & Coaching Benefits
  • Flexible Vacation & Reddit Global Days off
  • Generous paid Parental Leave
  • Paid Volunteer time off
  • Fulltime
Read More
Arrow Right

Senior ML Infrastructure Engineer - Embodied AI

At General Motors, our product teams are redefining mobility. Through a human-ce...
Location
Location
United States , Sunnyvale
Salary
Salary:
153200.00 - 234100.00 USD / Year
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years of experience working on large-scale distributed systems, applications, or ML infrastructure
  • Experience designing robust services or frameworks with durable, well-designed APIs
  • Solid understanding of machine learning workflows and hands-on experience applying ML systems in production environments
  • Experience building reliable, high-performance, and cost-efficient systems on modern cloud infrastructure
  • Practical experience across the ML development lifecycle, including model training, deployment, and MLOps practices
  • Strong cross-functional collaboration skills across teams and organizations
  • Strong coding skills in Python or C++
  • Interest in autonomous driving and large-scale ML systems
  • BS, MS, or PhD in Computer Science, Mathematics, or equivalent practical experience
Job Responsibility
Job Responsibility
  • Design, implement, and deploy scalable platforms and tools supporting machine learning training and evaluation workflows across GM
  • Drive complex technical projects with strong ownership of implementation, code quality, and system reliability
  • Contribute to technical design discussions and architectural decisions while collaborating with senior engineers and technical leads
  • Work closely with partner teams to ensure platforms meet real-world ML development needs and maximize adoption
  • Identify technical improvements and help prioritize platform investments to improve performance, reliability, and developer productivity
  • Contribute to a strong engineering culture through high-quality code reviews, documentation, and operational excellence
  • Support onboarding and mentoring of junior engineers and interns
What we offer
What we offer
  • medical
  • dental
  • vision
  • Health Savings Account
  • Flexible Spending Accounts
  • retirement savings plan
  • sickness and accident benefits
  • life insurance
  • paid vacation & holidays
  • tuition assistance programs
  • Fulltime
Read More
Arrow Right

Staff ML Infrastructure Engineer - Embodied AI

At General Motors, our product teams are redefining mobility. Through a human-ce...
Location
Location
United States , Sunnyvale
Salary
Salary:
189300.00 - 290700.00 USD / Year
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of experience building large-scale distributed systems, applications, or advanced ML systems
  • Proven track record of designing robust frameworks with high-quality, durable APIs
  • Deep understanding of machine learning algorithms with hands‑on application
  • Expertise in building reliable, high-performance, and cost-efficient systems on modern cloud infrastructure
  • End-to-end experience across the ML development lifecycle, including MLOps practices
  • Strong cross functional collaboration skills across teams and organizations
  • Exceptional coding skills in Python or C++
  • Strong interest in autonomous driving and its transformative potential
  • BS, MS, or PhD in Computer Science, Mathematics, or equivalent practical experience
Job Responsibility
Job Responsibility
  • Lead the design, implementation, and deployment of scalable platforms and tools that drive machine learning model training and evaluation workflows across GM
  • Own complex technical projects end-to-end, making key architectural decisions and technical trade-offs
  • Take a holistic view of projects, considering their impact across multiple teams, and across a longer timeline
  • Proactively drive technical prioritization
  • Collaborate closely with partner teams to ensure maximum benefit from the systems we build
  • Help shape our team through technical interviewing with high, well-calibrated standards, and play an essential role in recruiting
  • Mentor and onboard junior engineers and interns, helping them grow their careers
What we offer
What we offer
  • Medical
  • Dental
  • Vision
  • Health Savings Account
  • Flexible Spending Accounts
  • Retirement savings plan
  • Sickness and accident benefits
  • Life insurance
  • Paid vacation & holidays
  • Tuition assistance programs
  • Fulltime
Read More
Arrow Right

Senior Software Engineer - ML Infrastructure

We build simple yet innovative consumer products and developer APIs that shape h...
Location
Location
United States , New York
Salary
Salary:
190800.00 - 286800.00 USD / Year
plaid.com Logo
Plaid
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of industry experience as a software engineer, with strong focus on ML/AI infrastructure or large-scale distributed systems
  • Hands-on expertise in building and operating ML platforms (e.g., feature stores, data pipelines, training/inference frameworks)
  • Proven experience delivering reliable and scalable infrastructure in production
  • Solid understanding of ML Ops concepts and tooling, as well as best practices for observability, security, and reliability
  • Strong communication skills and ability to collaborate across teams
Job Responsibility
Job Responsibility
  • Design and implement large-scale ML infrastructure, including feature stores, pipelines, deployment tooling, and inference systems
  • Drive the rollout of Plaid’s next-generation feature store to improve reliability and velocity of model development
  • Help define and evangelize an ML Ops “golden path” for secure, scalable model training, deployment, and monitoring
  • Ensure operational excellence of ML pipelines and services, including reliability, scalability, performance, and cost efficiency
  • Collaborate with ML product teams to understand requirements and deliver solutions that accelerate experimentation and iteration
  • Contribute to technical strategy and architecture discussions within the team
  • Mentor and support other engineers through code reviews, design discussions, and technical guidance
  • Fulltime
Read More
Arrow Right

Senior ML Infrastructure Engineer

Parametric is building robots to reliably automate physical labor in the real wo...
Location
Location
United States , San Francisco
Salary
Salary:
150000.00 - 210000.00 USD / Year
workatastartup.com Logo
YC Work at a Startup
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Three or more years (or equivalent) working in devops, ML infrastructure, or platform engineering roles
  • Experience designing and implementing production-grade AI infrastructure
  • Deep understanding of the ML lifecycle: data pipelines, distributed training, model evaluation, and deployment
  • Strong proficiency with cloud platforms (AWS, GCP, or Azure) and infrastructure-as-code tools
  • Experience building CI/CD pipelines with tools like GitHub Actions, Jenkins, or similar
  • Comfortable with Python, bash, and infrastructure scripting
Job Responsibility
Job Responsibility
  • Design and implement robust ML infrastructure for dataset management, model training and evaluation, and deployment
  • Collaborate with ML engineers to gather requirements and develop plans
  • Build and operate cloud infrastructure (e.g. AWS, GCP) for machine learning workloads for experiments and production
  • Automate model evaluation, selection, and deployment
  • Fulltime
Read More
Arrow Right