CrawlJobs Logo

Training Performance Engineer

United States, San Francisco 250000.00 - 445000.00 USD / Year · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

As a Training Performance Engineer, you’ll drive efficiency improvements across our distributed training stack. You’ll analyze large-scale training runs, identify utilization gaps, and design optimizations that push the boundaries of throughput and uptime. This role blends deep systems understanding with practical performance engineering — analyzing GPU kernel performance, collective communication throughput, investigating I/O bottlenecks, and sharding our models so we can train them at massive scale. You’ll help ensure that our clusters are running at peak performance, enabling OpenAI to train larger, more capable models with the same compute budget.

Job Responsibility

  • Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage
  • Optimize GPU utilization and throughput for large-scale distributed model training
  • Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance
  • Implement model graph transforms to improve end to end throughput
  • Build tooling to monitor and visualize MFU, throughput, and uptime across clusters
  • Partner with researchers to ensure new model architectures scale efficiently during pre-training
  • Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs

Requirements

  • Love optimizing performance and digging into systems to understand how every layer interacts
  • Have strong programming skills in Python and C++ (Rust or CUDA a plus)
  • Have experience running distributed training jobs on multi-GPU systems or HPC clusters
  • Enjoy debugging complex distributed systems and measuring efficiency rigorously
  • Have exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built
  • Are comfortable collaborating across teams and translating raw profiling data into practical engineering improvements

Nice to have

  • Familiarity with NCCL, MPI, or UCX communication libraries
  • Experience with large-scale data loading and checkpointing systems
  • Prior work on training runtime, distributed scheduling, or ML compiler optimization

What we offer

  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Relocation support for eligible employees
  • Additional taxable fringe benefits, such as charitable donation matching and wellness stipends, may also be provided
  • Offers Equity

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Training Performance Engineer

8 matching positions

Member of Technical Staff, Training Performance Engineer

As a Performance Engineer in the Pre-Training team you will be responsible for o...
Location
Location
Salary
Salary:
Not provided
cohere.com Logo
Cohere
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Extremely strong software engineering skills
  • Proficiency in Python and related ML frameworks such as JAX, Pytorch and XLA/MLIR
  • Experience writing kernels for GPUs using CUDA, triton, etc
  • Experience using large-scale distributed training strategies
  • Familiarity with autoregressive sequence models, such as Transformers
Job Responsibility
Job Responsibility
  • Design and write high-performant and scalable software for training
  • Understand architectural modifications and design choices and their effects on training throughput and quality
  • Write low-level CUDA, triton kernels to squeeze every last bit of performance from our accelerators
  • Research, implement, and experiment with ideas on our supercompute and data infrastructure
  • Learn from and work with the best researchers in the field
What we offer
What we offer
  • An open and inclusive culture and work environment
  • Work closely with a team on the cutting edge of AI research
  • Weekly lunch stipend, in-office lunches & snacks
  • Full health and dental benefits, including a separate budget to take care of your mental health
  • 100% Parental Leave top-up for up to 6 months
  • Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • 6 weeks of vacation (30 working days!)
  • Fulltime
Read More
Arrow Right

Senior GPU Software Performance Engineer — Post‑Training

Drive the performance of post‑training workloads on AMD Instinct™ GPUs. You’ll w...
Location
Location
United States , San Jose
Salary
Salary:
204000.00 - 306000.00 USD / Year
amd.com Logo
AMD
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven GPU performance engineering for deep learning (ROCm/HIP, Triton, or similar)
  • Hands-on with SFT. LoRA and RL-based training at scale
  • Strong PyTorch experience (torch.distributed, FSDP/ZeRO or equivalent)
  • Proficient in Python and C++
  • comfortable reading/writing kernels when needed
  • Experience with distributed systems and collective communication libraries
  • Track record of turning profiles into fixes, upstreaming changes, and documenting results
Job Responsibility
Job Responsibility
  • Lead performance for finetuning and RL training solutions on AMD GPUs
  • Improve throughput, memory efficiency, and stability across data, model, and optimizer steps
  • Optimize multi-GPU/multi-node training and communication patterns
  • Contribute efficient kernels/ops and targeted graph-level optimizations
  • Profile, diagnose, and resolve bottlenecks using standard tooling
  • prevent regressions in CI
  • Ship reproducible pipelines and documentation adopted by internal teams and external developers
  • Collaborate with framework, compiler, and model teams to land durable improvements
  • Fulltime
Read More
Arrow Right

Principal ML Engineer - Large Scale Training Performance Optimization

We are looking for a Principal Machine Learning Engineer to join our Models and ...
Location
Location
United States , San Jose; Bellevue
Salary
Salary:
226400.00 - 339600.00 USD / Year
amd.com Logo
AMD
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience with ML/DL frameworks such as PyTorch, JAX, or TensorFlow
  • Experience with distributed training and distributed training frameworks, such as Megatron-LM, MaxText, TorchTitan
  • Experience with LLMs or computer vision, especially large models
  • Experience with GPU kernel optimization
  • Excellent Python or C++ programming skills, including debugging, profiling, and performance analysis at scale
  • Experience with ML infra at kernel, framework, or system level
  • Strong communication and problem-solving skills
  • A master's degree or PhD degree in Computer Science, Artificial Intelligence, Machine Learning, or a related field
Job Responsibility
Job Responsibility
  • Train large models to convergence on AMD GPUs at scale
  • Improve the end-to-end training pipeline performance
  • Optimize the distributed training pipeline and algorithm to scale out
  • Contribute your changes to open source
  • Stay up-to-date with the latest training algorithms
  • Influence the direction of AMD AI platform
  • Collaborate across teams with various groups and stakeholders
  • Fulltime
Read More
Arrow Right

Field Engineer - Engineer in Training

Provide engineering and consulting services for a broad array of projects and cl...
Location
Location
United States , Greenville
Salary
Salary:
Not provided
terracon.com Logo
Terracon Consultants, Inc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Engineering
  • Valid driver’s license with acceptable violation history
Job Responsibility
Job Responsibility
  • Follow safety rules, guidelines and standards for all projects
  • Participate in pre-task planning
  • Report any safety issues or concerns to management
  • Understand and practice quality acumen
  • Support consistent quality standards on proposal and project delivery
  • Assist in research, data collection, field observation, inspection, and field and laboratory testing
  • Prepare associated reports or sections of larger reports
  • Evaluate mostly routine laboratory and field data for inclusion in reports
  • Consult with other technical people and supervisor on routine and less complex projects
  • May perform basic calculations and engineering analysis
What we offer
What we offer
  • Medical
  • dental
  • vision
  • life insurance
  • 401(k) plan
  • paid time off and holidays
  • education reimbursement
  • various bonus programs
  • Fulltime
Read More
Arrow Right

Chief Engineer (2nd Class Stationary Engineer) In Training

The Chief Engineer in Training will work with our Chief Stationary Engineer to d...
Location
Location
Canada , Welland
Salary
Salary:
80000.00 - 90000.00 CAD / Year
valbrunaasw.ca Logo
Valbruna ASW Inc.
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Must hold a valid Ontario 3rd Class Operating Engineer Certificate and have working experience as a 3rd Class Stationary Engineer
  • Must be working towards obtaining the 2nd Class Stationary Certification
  • Willingness and ability to work shifts, overtime and some weekend work when required
  • Proven troubleshooting and diagnostic skills, mechanical aptitude and strong mathematical skills
  • Exceptional organizational, time management and problem-solving skills are key to success in this position
  • Demonstrates the ability to prioritize tasks, working independently or as part of a team
  • Excellent written and verbal communication skills, with strong interpersonal skills
  • Strong safety track record and demonstrated willingness to participate in workplace safety and ensure compliance with all Valbruna ASW Inc. Health and Safety policies and procedures
  • A strong willingness to work in a fast-paced, lean environment where adaptability, innovation and hard work are valued above all else
Job Responsibility
Job Responsibility
  • Complete all duties and responsibilities in compliance with the Technical Standards & Safety Authority (TSSA) Code
  • Perform and record operational checks and servicing of pump house, water reclaims, cooling towers, baghouses, air conditioning, boilers, compressors and auxiliaries
  • Inspect equipment, adjust pump and fan sequences, maintain supply of chemicals, manually operate filter backwash as required
  • Responsible for operational checks and servicing of the compressor room, allied equipment and auxiliaries
  • Read meters, record readings, change charts, take water samples/test with chemicals, add chemicals to water system, adjust blow downs for proper boiler water concentrations as required by operating conditions
  • Perform daily routine maintenance such as packing pumps, lubricating equipment etc. and participate in major overhauls and repairs to equipment in the maintenance department
  • Plan details of working procedure and perform work requiring a thorough knowledge of mechanical principles, hydraulics, pneumatics, and the application of the theory of operation of mechanical equipment
  • Inspect all mechanical equipment to determine material replacement needs and service requirements and report all abnormal conditions and hazards as identified through preventive maintenance and inspection programs
  • Analyze and interpret blueprints, drawings, manufacturers’ specifications and instructions and install all equipment to locate faults, determine repair or installation procedure and ascertain material needs
What we offer
What we offer
  • health insurance
  • pension plan
  • Fulltime
Read More
Arrow Right

Qa Performance Engineer (Iot Sphere)

The project focuses on performance testing of the middleware service layer power...
Location
Location
Salary
Salary:
Not provided
coherentsolutions.com Logo
Coherent Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 1+ year of experience in QA, manual or automation
  • Basic understanding of performance testing concepts, including common test types and key metrics
  • Hands-on exposure to JMeter, including basic test plan structure, script recording, parameterization, and assertions
  • Understanding of REST APIs and HTTP basics, including methods, status codes, headers, and JSON payloads
  • Basic knowledge of Java and object-oriented programming
  • Ability to read and interpret graphs, dashboards, and basic system metrics
  • Strong analytical thinking, attention to detail, and problem-solving skills
  • Good communication skills and ability to work in a team
  • Familiarity with Agile/Scrum software development process
  • English level: B2 or higher
Job Responsibility
Job Responsibility
  • Execute performance, load, stress, and soak tests for backend services, REST APIs, and microservices using JMeter
  • Develop and maintain JMeter test scripts under the guidance of senior performance engineers
  • Contribute to the performance testing framework by adding new scenarios, reusable components, and test data sets
  • Analyze test results, prepare reports, and help identify performance bottlenecks together with Developers and DevOps Engineers
  • Use observability tools such as Grafana and Datadog to review dashboards and logs and support root cause analysis
  • Document test plans, scenarios, execution results, and findings in a clear and reproducible way
  • Collaborate with Developers, DevOps Engineers, and QA peers to ensure performance criteria are met before release
  • Participate in Agile ceremonies, including planning, refinement, retrospectives, and daily team communication
  • Continuously improve skills in performance engineering, JMeter, and cloud-based system performance through self-learning, training, and knowledge sharing
What we offer
What we offer
  • Technical and non-technical training for professional and personal growth
  • Internal conferences and meetups to learn from industry experts
  • Support and mentorship from an experienced employee to help you professional grow and development
  • Health insurance
  • English courses
  • Sports activities to promote a healthy lifestyle
  • Flexible work options, including remote and hybrid opportunities
  • Referral program for bringing in new talent
  • Work anniversary program and additional vacation days
  • Fulltime
Read More
Arrow Right

Senior LLM Inference Performance Engineer

WHAT YOU DO AT AMD CHANGES EVERYTHING At AMD, our mission is to build great prod...
Location
Location
Finland , Helsinki
Salary
Salary:
Not provided
amd.com Logo
AMD
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Seasoned in deploying LLMs and other AI model types in production using frameworks like vLLM, SGLang, or similar tooling
  • Deep knowledge about LLM serving and performance metric evaluation
  • Comfortable with Python software development and bash scripting
  • Experience with Docker, Kubernetes and Helm
  • Desire and ability to continuously learn in a fast-changing environment
  • Initiative, pragmatic problem solving, and great collaboration skills
  • Bachelor's or master's degree in computer science, computer engineering, electrical engineering, or an equivalent field
Job Responsibility
Job Responsibility
  • LLM and AI Performance: Measure, analyze, and optimize LLM and AI service performance across metrics like latency and throughput for various training and inference use cases
  • Design and implement methodologies for measuring model performance, and automating optimization strategies to identify optimal configurations
  • Stay on top of current advances in AI, models, APIs, and open-source ecosystems, and translate them into scalable solutions
  • LLM and AI Tooling: Design and develop tooling to measure and analyze the performance of AI model deployments and the effect of different configurations and infrastructure, standalone and Kubernetes clusters
  • Develop and maintain tooling for interacting with different ecosystem functions to improve developer and user experience
  • Develop and maintain internal tooling to support LLM and AI performance tuning at scale
  • Fulltime
Read More
Arrow Right

AI/HPC System Performance Engineer

Meta is building some of the world's largest AI and high-performance computing i...
Location
Location
United States , Menlo Park
Salary
Salary:
154000.00 - 217000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience profiling and optimizing distributed AI or HPC workloads, including familiarity with GPU interconnects, RDMA networking, and collective communication frameworks such as NCCL or MPI
  • Experience debugging complex, non-reproducible performance issues across multi-layer systems including network fabric, operating system, and application layers
  • Experience designing and implementing performance monitoring systems, including instrumentation, telemetry pipelines, and alerting for large-scale infrastructure
  • Experience driving cross-functional technical projects from requirements definition through production deployment, including communicating performance findings and trade-offs to diverse stakeholders
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • 6+ years of experience in system performance engineering, network infrastructure engineering, or a related field within large-scale distributed computing or HPC environments
Job Responsibility
Job Responsibility
  • Profile and benchmark AI training and inference workloads across large-scale HPC clusters to identify network, compute, and memory bottlenecks
  • Develop and maintain performance analysis frameworks and dashboards to track system-level metrics including GPU utilization, network bandwidth, latency, and collective communication efficiency
  • Investigate and resolve performance regressions in distributed AI training environments, including issues related to RDMA fabrics, collective communication libraries, and job scheduling
  • Collaborate with network infrastructure, hardware, and AI research teams to define performance requirements and validate new HPC cluster configurations
  • Design and execute capacity and scalability experiments to inform network topology decisions for AI supercomputing infrastructure
  • Build tooling and automation to continuously monitor HPC system health, detect anomalies, and reduce mean time to mitigation during performance incidents
  • Establish service level objectives for AI cluster network performance and drive cross-functional alignment on reliability and efficiency targets
  • Lead technical design reviews for network and system architecture changes affecting AI workload performance, communicating trade-offs clearly to engineering and product stakeholders
  • Mentor other engineers on HPC performance methodologies, debugging techniques, and instrumentation best practices
  • Leverage AI-assisted workflows to accelerate root cause analysis, automate routine performance reporting, and expand coverage across the HPC stack
What we offer
What we offer
  • bonus + equity + benefits
  • Fulltime
Read More
Arrow Right