CrawlJobs Logo

Research Engineer - Distributed Training

United States, San Francisco · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

Building Open Superintelligence Infrastructure. Prime Intellect is building the open superintelligence stack - from frontier agentic models to the infra that enables anyone to create, train, and deploy them. We aggregate and orchestrate global compute into a single control plane and pair it with the full rl post-training stack: environments, secure sandboxes, verifiable evals, and our async RL trainer. We enable researchers, startups and enterprises to run end-to-end reinforcement learning at frontier scale, adapting models to real tools, workflows, and deployment contexts. As a Research Engineer working on Distributed Training, you'll play a crucial role in shaping our technological direction, focusing on our decentralizing AI training stack. If you love scaling things and maximizing training efficiency, this role is for you.

Job Responsibility

  • Lead and participate in novel research to build a massive scale, highly reliable and secure decentralized training orchestration solution
  • Optimize the performance, cost, and resource utilization of AI workloads by leveraging the most recent advances for compute & memory optimization techniques
  • Contribute to the development of our open-source libraries and frameworks for distributed model training
  • Publish research in top-tier AI conferences such as ICML & NeurIPS
  • Distill highly technical project outcomes in layman approachable technical blogs to our customers and developers
  • Stay up-to-date with the latest advancements in AI/ML infrastructure and tools, decentralized training research and proactively identify opportunities to enhance our platform's capabilities and user experience

Requirements

  • Strong background in AI/ML engineering, with extensive experience in designing and implementing end-to-end pipelines for training and deploying large-scale AI models
  • Deep expertise in distributed training techniques, frameworks (e.g., PyTorch Distributed, DeepSpeed, MosaicML’s LLM Foundry), and tools (e.g. Ray) for optimizing the performance and scalability of AI workloads
  • Experience in large-scale model training incl. distributed training techniques such as data, tensor & pipeline parallelism
  • Solid understanding of MLOps best practices, including model versioning, experiment tracking, and continuous integration/deployment (CI/CD) pipelines
  • Passion for advancing the state-of-the-art in decentralized AI model training and democratizing access to AI capabilities for researchers, developers, and businesses worldwide

What we offer

  • Competitive compensation, including equity incentives, aligning your success with the growth and impact of Prime Intellect
  • Flexible work arrangements, with the option to work remotely or in-person at our offices in San Francisco
  • Visa sponsorship and relocation assistance for international candidates
  • Quarterly team off-sites, hackathons, conferences and learning opportunities
  • Opportunity to work with a talented, hard-working and mission-driven team, united by a shared passion for leveraging technology to accelerate science and AI

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Research Engineer - Distributed Training

8 matching positions

Distributed Training Engineer

As a Distributed Systems/ML engineer, you will work on improving the training th...
Location
Location
United States , San Francisco
Salary
Salary:
293000.00 - 490000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience working with multi-modal ML pipelines
  • Strong software engineering skills and proficiency in Python
  • Experience understanding and optimizing training kernels
  • Passionate about understanding stable training dynamics
Job Responsibility
Job Responsibility
  • Collaborate with researchers to enable them to develop systems-efficient video models and architectures
  • Apply the latest techniques to our internal training framework to achieve impressive hardware efficiency for our training runs
  • Profile and optimize our training framework
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right

Research Scientist / Engineer – Training Infrastructure

Luma’s mission is to build multimodal AI to expand human imagination and capabil...
Location
Location
United States , Palo Alto
Salary
Salary:
187500.00 - 395000.00 USD / Year
lumalabs.ai Logo
Luma AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Extensive experience with distributed PyTorch training and parallelisms in foundation model training
  • Deep understanding of GPU clusters, networking, and storage systems
  • Familiarity with communication libraries (NCCL, MPI) and distributed system optimization
Job Responsibility
Job Responsibility
  • Design, implement, and optimize efficient distributed training systems for models with thousands of GPUs
  • Research and implement advanced parallelization techniques (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel)
  • Build monitoring, visualization, and debugging tools for large-scale training runs
  • Optimize training stability, convergence, and resource utilization across massive clusters
  • Fulltime
Read More
Arrow Right

Research Scientist / Engineer – Pre-training / Scaling

At Luma, the Pre-Training / Scaling team is responsible for building the core mu...
Location
Location
United States , Palo Alto
Salary
Salary:
187500.00 - 395000.00 USD / Year
lumalabs.ai Logo
Luma AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Expertise in Python and PyTorch with experience building ML models from scratch
  • Deep understanding of multimodal generative models and deep learning architectures
  • (Preferred) Strong research track record in generative AI with published work in top-tier venues preferred
  • (Preferred) Experience with large-scale distributed training systems
Job Responsibility
Job Responsibility
  • Lead cutting-edge research in multimodal foundation models spanning video, image, text, and audio
  • Design and implement novel algorithms, architectures, and techniques for large-scale generative AI models
  • Develop training methodologies for foundation models across thousands of GPUs
  • Research and implement state-of-the-art techniques in Autoregressive LLMs, Vision Language Models, and / or Diffusion Models
  • Collaborate with cross-functional teams to transition research into production systems
  • Fulltime
Read More
Arrow Right

Research Engineer, Language Model Pre-Training

As a Research Engineer, Language Model Pre-training, you'll shape our language m...
Location
Location
United States , Palo Alto
Salary
Salary:
Not provided
zyphra.com Logo
Zyphra
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong engineering aptitude for rapidly implementing reliable and robust systems
  • Can rapidly learn new fields and are excited to implement new ideas
  • Excellent communication and collaboration skills, and can work effectively on both research and engineering implementation at scale
  • Deep expertise and intuition for solving machine learning problems and training models
  • Experience with training on large-scale (multi-node) GPU clusters
  • Deep understanding of model training pipelines – including model/data parallelism, distributed optimizers, etc.
  • Strong grasp of proper experimental methodology for running rigorous ablations and other hypothesis testing
  • Understanding of large-scale, highly parallel data processing pipelines
  • High proficiency with PyTorch and Python
  • Strong ability to dive into large pre-existing codebases and rapidly get up to speed
Job Responsibility
Job Responsibility
  • Shape our language model roadmap through end-to-end pretraining development
  • Work across: Large-scale training runs and model parallelization
  • Performance optimization of our pretraining stack
  • Dataset collection, processing, and evaluation
  • Architecture and methodology research, including optimizer ablations
What we offer
What we offer
  • Comprehensive medical, dental, vision, and FSA plans
  • Competitive compensation and 401(k)
  • Relocation and immigration support on a case-by-case basis
  • On-site meals prepared by a dedicated culinary team
  • Thursday Happy Hours
  • In-person team in Palo Alto, CA, with a collaborative, high-energy environment
  • Fulltime
Read More
Arrow Right

Research Scientist / Engineer – Foundation Model: Core Research

This is a rare and foundational opportunity to define the future of multimodal A...
Location
Location
United States , Palo Alto
Salary
Salary:
250000.00 - 450000.00 USD / Year
lumalabs.ai Logo
Luma AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A Bachelor's, Master's, or PhD degree in Computer Science, Machine Learning, Physics, or Mathematics is essential
  • A 'first-principles' intuition for scaling
  • Fluent in the language of frontier AI
  • Proven ability to design and rigorously analyze experiments and to articulate complex technical concepts effectively
  • Practical experience with distributed or high-performance computing environments, particularly managing and optimizing training runs on large-scale GPU clusters
Job Responsibility
Job Responsibility
  • Unified Modeling & Efficiency Drive the core research that powers all of Luma's products — co-designing multimodal representations, advancing core algorithms for long-context training, and establishing rigorous scaling laws to predict performance across compute budgets
  • Alignment & Evaluation Close the gap between training loss and user experience. Develop proxy tasks and automated metrics that serve as the compass for research decisions — ensuring our models optimize for what actually matters to users, not just benchmarks
  • Research Infrastructure Build the engine for high-velocity research. Maintain production-research parity, ensure reproducibility, and design systems for rapid experimentation — so that novel ideas go from hypothesis to validated result as fast as possible
  • Fulltime
Read More
Arrow Right

Staff Research Engineer, MRS AI

Meta is seeking a Staff Software Engineer to join the MRS AI Knowledge team, whe...
Location
Location
United States , Bellevue
Salary
Salary:
183997.00 - 257000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • 8+ years of experience in software engineering with a focus on natural language processing, computational linguistics, or language model systems
  • Experience designing and shipping production NLP systems such as text classification, named entity recognition, semantic parsing, machine translation, or language generation at scale
  • Experience leading major technical initiatives end-to-end, including architecture design, cross-team coordination, staged rollout using feature flagging and A/B testing frameworks, and post-launch monitoring
  • Experience with distributed systems and large-scale data processing pipelines for training or serving language models
  • Experience communicating technical decisions and trade-offs in writing to both technical and non-technical stakeholders, including design documents and cross-functional reviews
Job Responsibility
Job Responsibility
  • Lead the design and implementation of recommendation system models and AI infrastructure based on state-of-the-art ML technologies including AI Agents, LLMs, Graph Learning (GNN, GCN, Random Walk), and CNN/SNN/TTSN architectures
  • Build and optimize ML solutions for video, photo, audio, user, and text understanding and matching across Meta's recommendation products
  • Drive data optimization initiatives including reweighting, sampling, denoising, and augmentation to improve model performance
  • Engage in and contribute meaningfully to projects' direction and roadmap, helping shape the technical vision for the team
  • Build deep understanding of Meta's recommendation products and content ecosystem, and effectively incorporate that knowledge into solution designs
  • Hold yourself and others accountable for engineering, ML, and data excellence across all deliverables
  • Communicate effectively with team and cross-functional partners to unblock work streams and deliver product impact
  • Mentor other engineers on ML system design, recommendation system architecture, and engineering best practices
  • Drive experimentation frameworks for evaluating model quality, including designing evaluation metrics and making data-informed decisions on model selection and deployment
  • Champion privacy-by-design and responsible AI principles in the development of ML models and recommendation systems
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right

AI Research Engineer

Domyn is a company specializing in the research and development of Responsible A...
Location
Location
Italy , Milan
Salary
Salary:
50000.00 - 80000.00 EUR / Year
igenius.ai Logo
iGenius
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • PhD in Computer Science, Artificial Intelligence or a related field, or equivalent practical experience
  • At least 5 years of proven experience as an AI research engineer or more than 2 years of experience and a PhD
  • Expertise in modern machine learning frameworks such as PyTorch, TensorFlow, and JAX, with deep knowledge of distributed training (Pytorch Distributed, Ray, DeepSpeed)
  • Strong background in parallel computing and high-performance systems, including CUDA programming and compiler optimizations
  • Hands-on experience with ML model debugging and performance profiling tools (TensorBoard, Weights & Biases, NVIDIA Nsight)
  • Proficiency in Python and C++ or Rust, particularly for high-performance inference and AI accelerators
  • Solid understanding of mathematics behind deep learning (linear algebra, probability, optimization)
  • Experience deploying models in production, optimizing for latency, throughput, and memory efficiency
  • Fluent in English
Job Responsibility
Job Responsibility
  • Build the next generation of large language models, across the full life cycle: pretraining on trillions of tokens, state-of-the-art post-training, and releases spanning from a few billion to hundreds of billions of parameters
  • Collaborate directly with leading industry labs like NVIDIA
  • Work will ship into mission-critical deployments with some of the largest players in banking, defense, manufacturing, and the public sector
  • Design and optimize the infrastructure that trains and serves these models
  • Contribute to open-source projects shaping the European AI ecosystem
What we offer
What we offer
  • Learning Friday
  • Training budget for books, online courses or other training materials
  • Smart Working (option to work from home)
  • Salary topped up with other bonuses
  • Opportunity to receive company equity
  • Stock options
  • Fulltime
Read More
Arrow Right

Data Research Engineer

Fundamental is an AI company pioneering the future of enterprise decision-making...
Location
Location
Spain , Barcelona
Salary
Salary:
Not provided
Fundamental
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience with: Identifying good data sources to train and evaluate ML models, including real-world and realistic synthetic data sources
  • Bringing data from structured and unstructured sources, as well as simulators and causal models, into formats accessible by ML models
  • Strong fundamentals of software engineering
  • Strong knowledge of: Python
  • Python data processing stack (numpy, pandas, …)
  • Familiarity with: distributed processing (e.g. Ray, Dask Spark, Beam)
  • data storage solutions
  • Basic ML knowledge
Job Responsibility
Job Responsibility
  • Helping to identify, characterize and evaluate data sources, including realistic synthetic data generated from Structured Causal Models and physical / systems-based simulators
  • Building and maintaining ETL pipelines
  • Designing and implementing scalable, reliable data storage solutions
  • Collaborating with the rest of the research team to maintain a reliable, efficient training pipeline where data is a critical component
  • Collaborating with the wider engineering and infrastructure team
What we offer
What we offer
  • Competitive compensation with salary and equity
  • Comprehensive health coverage for you and your dependents
  • Paid parental leave for all new parents, inclusive of adoptive and surrogate journeys
  • Relocation support for employees moving to join the team in one of our office locations
  • A mission-driven, low-ego culture that values diversity of thought, ownership, and bias toward action
  • Fulltime
Read More
Arrow Right