ML Engineer - Inference Serving Job at Luma AI (Palo Alto)

Senior ML Inference Engineer - Platform

The Model Deployment & Inference Solutions team in GM AV deploys machine learnin...

Location

United States , Austin; Mountain View

Salary:

128700.00 - 261300.00 USD / Year

General Motors

Expiration Date

Until further notice

Requirements

BS, MS, or PhD in Computer Science or a related technical field
3+ years of relevant industry experience
Strong fundamentals and excellent coding ability in Python
Experience building or operating production platform or infrastructure systems where reliability, observability, and extensibility matter
Experience with ML model deployment, inference integration, model optimization workflows, or model serving infrastructure, with at least one prior context where you owned the path from a trained model to a running inference workload
Experience using coding agents (Cursor, Claude Code, GitHub Copilot, or equivalent) as part of your engineering workflow
Experience designing clean, well-tested software with clear interfaces and good abstractions
Strong cross-team collaboration skills

Job Responsibility

Design, build, and operate the ML deployment platform that automates the path from trained model to on-vehicle inference
Drive cross-organization model deployments to the autonomous vehicle stack, partnering with model development teams to take high-value models from training to production on-vehicle
Build agentic tools that diagnose and fix deployment-blocking issues, automating workflows currently performed manually by engineers
Build the developer experience that ML model development teams use day to day: tooling, dashboards, automation, and observability
Drive shift-left validation that surfaces deployment risk (compile, runtime, parity, latency) early in the model development cycle
Build platform tools that integrate the work of our sister teams (kernels, compiler, reduced precision and parity) so their optimization wins land directly in the deployment workflow
Partner with the team's Performance pillar and model development teams across the AV organization

What we offer

Medical
Dental
Vision
Health Savings Account
Flexible Spending Accounts
Retirement savings plan
Sickness and accident benefits
Life insurance
Paid vacation & holidays
Tuition assistance programs

Fulltime

Staff ML Engineer, Inference Platform

The ML Inference Platform is part of the AI Compute Platforms organization withi...

Location

United States , Sunnyvale

Salary:

185500.00 - 270000.00 USD / Year

General Motors

Expiration Date

Until further notice

Requirements

8+ years of industry experience, with focus on machine learning systems or high performance backend services
Expertise in either Go, Python, C++ or other relevant coding languages
Expertise in ML inference, model serving frameworks (triton, rayserve, vLLM etc)
Strong communication skills and a proven ability to drive cross-functional initiatives
Experience working with cloud platforms such as GCP, Azure, or AWS
Ability to thrive in a dynamic, multi-tasking environment with ever-evolving priorities

Job Responsibility

Design and implement core platform backend software components
Collaborate with ML engineers and researchers to understand critical workflows, parse them to platform requirements, and deliver incremental value
Lead technical decision-making on model serving strategies, orchestration, caching, model versioning, and auto-scaling mechanisms
Drive the development of monitoring, observability, and metrics to ensure reliability, performance, and resource optimization of inference services
Proactively research and integrate state-of-the-art model serving frameworks, hardware accelerators, and distributed computing techniques
Lead large-scale technical initiatives across GM’s ML ecosystem
Raise the engineering bar through technical leadership, establishing best practices
Contribute to open source projects
represent GM in relevant communities

What we offer

medical
dental
vision
Health Savings Account
Flexible Spending Accounts
retirement savings plan
sickness and accident benefits
life insurance
paid vacation & holidays
tuition assistance programs

Fulltime

Senior ML Infrastructure Engineer, Inference Platform

About the Team: The ML Inference Platform is part of the AV ML Infrastructure or...

Location

United States , Austin, Texas; Mountain View, California; Sunnyvale, California

Salary:

155420.00 USD / Year

General Motors

Expiration Date

Until further notice

Requirements

5+ years of industry experience, with focus on machine learning systems or high performance backend services
Expertise in either Python, C++ or other relevant coding languages
Expertise in ML inference, model serving frameworks (triton, rayserve, vLLM etc)
Strong communication skills and a proven ability to drive cross-functional initiatives
Ability to thrive in a dynamic, multi-tasking environment with ever-evolving priorities

Job Responsibility

Design and implement core platform backend software components
Collaborate with ML engineers and researchers to understand critical workflows, parse them to platform requirements, and deliver incremental value
Lead technical decision-making on model serving strategies, orchestration, caching, model versioning, and auto-scaling mechanisms for highly optimized use of accelerators
Drive the development of monitoring, observability, and metrics to ensure reliability, performance, and resource optimization of inference services
Proactively research and integrate state-of-the-art model serving frameworks, hardware accelerators, and distributed computing techniques
Lead technical initiatives across GM’s ML ecosystem
Raise the engineering bar through technical leadership, establishing best practices
Contribute to open source projects
represent GM in relevant communities

What we offer

medical
dental
vision
Health Savings Account
Flexible Spending Accounts
retirement savings plan
sickness and accident benefits
life insurance
paid vacation & holidays
tuition assistance programs

Fulltime

ML Model Serving Engineer

Sesame believes in a future where computers are lifelike - with the ability to s...

Location

United States , San Francisco; New York; Bellevue

Salary:

175000.00 - 280000.00 USD / Year

Sesame

Expiration Date

Until further notice

Requirements

Expert in some differentiable array computing framework, preferably PyTorch
Expert in optimizing machine learning models for serving reliably at high throughput, with low latency
Significant systems programming experience
ex. Experience working on high-performance server systems—you’d be just as comfortable with the internals of VLLM as you would with a complex PyTorch codebase
Significant performance engineering experience
ex. Bottleneck analysis in high-scale server systems or profiling low-level systems code
Always up to date on the latest techniques for model serving optimization

Job Responsibility

Turbocharge our serving layer, consisting of a variety of LLM, speech, and vision models
Partner with ML infrastructure and training engineers to build a fast, cost-effective, accurate, and reliable serving layer to power a new consumer product category
Modify and extend LLM serving frameworks like VLLM and SGLang to take advantage of the latest techniques in high-performance model serving
Work with the training team to identify opportunities to produce faster models without sacrificing quality
Use techniques like in-flight batching, caching, and custom kernels to speed up inference
Find ways to reduce model initialization times without sacrificing quality

What we offer

401k matching
100% employer-paid health, vision, and dental benefits
Unlimited PTO and sick time
Flexible spending account matching (medical FSA)

Fulltime

Senior Inference ML Runtime Engineer

The Inference ML Engineering team at Cerebras Systems is dedicated to enabling o...

Location

United States; Canada , Sunnyvale; Toronto

Salary:

Not provided

Cerebras Systems

Expiration Date

Until further notice

Requirements

Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Mathematics, or a related field
8+ years of experience in large-scale software engineering, with a focus on deep learning or related domains
Proficiency in Python for building and maintaining scalable systems
Advanced proficiency in C++, with an emphasis on multi-threaded programming, performance optimization, and system-level development
Demonstrated experience driving cross-functional projects
Experience building and scaling large-scale inference systems for LLMs or multimodal models
Familiarity with LLM serving frameworks, such as vLLM, SGLang, and TensorRT-LLM
Solid understanding of software architectural patterns for large-scale, high-performance applications
Hands-on experience with ML frameworks, such as PyTorch, and a strong understanding of their underlying architectures
Strong problem-solving skills, with the ability to balance technical depth with practical implementation constraints

Job Responsibility

Drive and provide technical guidance to a team of software engineers working on complex machine learning integration projects
Design and implement ML features (e.g., structured outputs, biased sampling, predicted outputs) that improve performance of generative AI models at inference time
Design and implement high-throughput, low-latency multimodal inference models that support delivery of image, audio, and video inputs and outputs
Maintain our scalable serving backend for handling many concurrent requests per minute
Scale our inference service by implementing detailed observability throughout the entire stack
Analyze and improve latency, throughput, memory usage, and compute efficiency on the service and the implementation of various features
Optimize software to accelerate generative LLM inference by achieving high throughput and low latency
Stay up-to-date with advancements in machine learning and deep learning, and apply state-of-the-art techniques to enhance our solutions
Evaluate trade-offs between different approaches, clearly articulate design choices, and develop detailed proposals for implementing new features
Uncover, scope, and prioritize significant areas of technical debt across the software stack to ensure continued high quality of the inference service

What we offer

Build a breakthrough AI platform beyond the constraints of the GPU
Publish and open source their cutting-edge AI research
Work on one of the fastest AI supercomputers in the world
Enjoy job stability with startup vitality
Our simple, non-corporate work culture that respects individual beliefs

New

Lead Machine Learning Inference Engineer, Advertising

Roku is changing how the world watches TV. Roku is the #1 TV streaming platform ...

Location

United States , Austin

Salary:

Not provided

Roku

Expiration Date

Until further notice

Requirements

M.S. or above in CS, ECE, or a related field
10+ years of experience in developing and deploying large-scale, distributed systems, with at least 5 years in a leadership or technical lead role
Strong programming skills in high-performance languages
Deep understanding of inference frameworks and ML system deployment
Proven experience optimizing performance for large-scale machine learning systems, including a deep knowledge of SOTA model optimizations, hardware-software co-design, GPU acceleration, and HPC techniques
Excellent communication and collaboration skills
Experience leading teams working on high-throughput, low-latency ML serving systems
Experience collaborating with and leading global, cross-functional teams
Contributions to open-source ML or systems projects

Job Responsibility

Lead the design and development of a SOTA Inference platform
Oversee the development of monitoring, observability, and other tooling to ensure system and model performance, reliability, and scalability of online inference services
Identify and resolve system inefficiencies, performance bottlenecks, and reliability issues, ensuring optimized end-to-end performance
Stay at the forefront of advancements in inference frameworks, ML hardware acceleration, and distributed systems, and incorporate innovations where and when they are impactful

What we offer

Global access to mental health and financial wellness support and resources
Healthcare (medical, dental, and vision) where applicable
Life, accident, disability, commuter, and retirement options (401(k)/pension) where applicable
Time off in accordance with local leave policies

Fulltime

Senior / Lead Machine Learning Engineer, Serving

Inworld is a product-oriented research lab of top AI researchers and engineers, ...

Location

Germany

Salary:

Not provided

Inworld AI

Expiration Date

Until further notice

Requirements

Inference Optimization. Deep understanding of modern serving frameworks and techniques like vLLM or TRT-LLM
Model Acceleration. Hands-on experience with quantization, distillation, caching strategies, continuous batching, paged attention, and speculative decoding
High-Performance Systems. Proficiency in C++, CUDA, Rust, or highly optimized Python. You know how to profile code and squeeze every ounce of performance out of NVIDIA GPUs
Distributed Systems & Scaling. Experience with Kubernetes, Ray, custom load balancing, multi-GPU/multi-node inference, and reliably handling thousands of concurrent connections
Public work. Non-trivial systems programming projects, open-source contributions to major inference engines, or deep-dive technical write-ups
Full-cycle ownership. You can take a model from the research team, containerize it, optimize its serving, and ensure it runs reliably in production
Background. PhD in CS, Physics, Math, or equivalent practical experience building backend or ML systems
Professional fluency in English (written and spoken) is required, as you will be collaborating daily with our US-based leadership and engineering teams

Staff / Principal Machine Learning Engineer, Serving

A year ago, reliably working agentic systems and sub-second multimodal inference...

Location

United States , Mountain View

Salary:

270000.00 - 500000.00 USD / Year

Inworld AI

Expiration Date

Until further notice

Requirements

Inference Optimization. Deep understanding of modern serving frameworks and techniques like vLLM or TRT-LLM
Model Acceleration. Hands-on experience with quantization, distillation, caching strategies , continuous batching, paged attention, and speculative decoding
High-Performance Systems. Proficiency in C++, CUDA, Rust, or highly optimized Python. You know how to profile code and squeeze every ounce of performance out of NVIDIA GPUs
Distributed Systems & Scaling. Experience with Kubernetes, Ray, custom load balancing, multi-GPU/multi-node inference, and reliably handling thousands of concurrent connections
Public work. Non-trivial systems programming projects, open-source contributions to major inference engines, or deep-dive technical write-ups
Full-cycle ownership. You can take a model from the research team, containerize it, optimize its serving, and ensure it runs reliably in production
Background. PhD in CS, Physics, Math, or equivalent practical experience building backend or ML systems

Job Responsibility

We hand you unclear problems and expect you to make them clear
We value engineers who say 'I don't know yet' and then design the benchmark or prototype that finds out
We treat performance, latency, and reliability as first-class product features, not a box to check before launch
Impact comes before everything else, though we support sharing work and open-source contributions that move the field forward
Your work should be visible

What we offer

bonus
equity
benefits
relocation assistance

Fulltime

Select Country

ML Engineer - Inference Serving

Job Description

Job Responsibility

Requirements

Nice to have

Looking for more opportunities?