CrawlJobs Logo
Physical Intelligence Logo Physical Intelligence · IT - Software Development

ML Infra Engineer (Data Systems)

United States, San Francisco · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

As an ML Infra Engineer (Data Systems), you’ll build and operate the data infrastructure that powers large-scale robot learning. Your systems will sit directly between raw data sources and training/evaluation, enabling us to move faster while maintaining performance, correctness, and reliability at scale. This is a systems role at the intersection of distributed systems, storage, and machine learning infrastructure.

Job Responsibility

  • Data Ingestion & Processing: Design and build high-throughput pipelines that validate, transform, and featurize raw multimodal data
  • Batch & Streaming Systems: Operate large-scale batch and streaming workflows over massive datasets
  • Storage Systems: Design object storage layouts, metadata systems, and efficient access patterns
  • choose file formats with performance and scalability in mind
  • Data Lifecycle Management: Build systems for backfills, dataset rebuilds, garbage collection, and large-scale transformations
  • Training-Time Performance: Optimize dataloaders, sharding, prefetching, caching, and throughput to reduce time from data arrival → model training
  • Metadata & Indexing: Build scalable metadata stores for datasets, annotations, and training artifacts
  • Data Movement: Move hundreds of terabytes to petabytes efficiently across clusters and environments
  • Operational Correctness: Implement observability, validation, and guardrails to prevent silent data regressions
  • Cross-Functional Collaboration: Work closely with cross-functional teams of researchers, engineers and roboticists to translate evolving data needs into robust systems

Requirements

  • Strong software engineering fundamentals
  • Experience building distributed systems or large-scale data pipelines
  • Comfort reasoning about performance, memory, I/O, and storage efficiency
  • Familiarity with batch and/or streaming processing systems
  • Experience with object storage systems and data format tradeoffs
  • Ownership mindset: design, build, operate, and iterate on systems end-to-end
  • Enjoy working closely with researchers and unblocking fast-moving projects

Nice to have

  • Experience with large ML training pipelines or dataloading systems
  • Knowledge of columnar or custom data formats
  • Experience with systems like ClickHouse, Ray, Flink, Spark, or similar
  • Hands-on experience operating petabyte-scale datasets
  • Debugging and fixing performance bottlenecks in data-heavy systems
Physical Intelligence - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

ML Infra Engineer (Data Systems)

8 matching positions

ML Infra Engineer

In this role you will help scale and optimize our training systems and core mode...
Location
Location
United States , San Francisco
Salary
Salary:
Not provided
physicalintelligence.company Logo
Physical Intelligence
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong software engineering fundamentals and experience building ML training infrastructure or internal platforms
  • Hands-on large-scale training experience in JAX (preferred), PyTorch
  • Familiarity with distributed training, multi-host setups, data loaders, and evaluation pipelines
  • Experience managing training workloads on cloud platforms (e.g., SLURM, Kubernetes, GCP TPU/GKE, AWS)
  • Ability to debug and optimize performance bottlenecks across the training stack
  • Strong cross-functional communication and ownership mindset
Job Responsibility
Job Responsibility
  • Own training/inference infrastructure: Design, implement, and maintain systems for large-scale model training, including scheduling, job management, checkpointing, and metrics/logging
  • Scale distributed training: Work with researchers to scale JAX-based training across TPU and GPU clusters with minimal friction
  • Optimize performance: Profile and improve memory usage, device utilization, throughput, and distributed synchronization
  • Enable rapid iteration: Build abstractions for launching, monitoring, debugging, and reproducing experiments
  • Manage compute resources: Ensure efficient allocation and utilization of cloud-based GPU/TPU compute while controlling cost
  • Partner with researchers: Translate research needs into infra capabilities and guide best practices for training at scale
  • Contribute to core training code: Evolve JAX model and training code to support new architectures, modalities, and evaluation metrics
  • Fulltime
Read More
Arrow Right

Software Engineer, Systems ML - SW/HW Co-design

Meta is seeking an AI Software Engineer to join our Research & Development teams...
Location
Location
United States , Sunnyvale
Salary
Salary:
181000.00 USD / Year ▼
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • Specialized experience in one or more of the following machine learning/deep learning domains: Hardware accelerators architecture, GPU architecture, machine learning compilers, or ML systems, AI infrastructure, high performance computing, performance optimizations, or Machine learning frameworks (e.g. PyTorch), numerics and SW/HW co-design
  • Experience developing AI-System infrastructure or AI algorithms in C/C++ or Python
  • Currently has, or is in the process of obtaining a Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience. Degree must be completed prior to joining Meta
Job Responsibility
Job Responsibility
  • Apply relevant AI infrastructure and hardware acceleration techniques to build & optimize our intelligent ML systems that improve Meta’s products and experiences
  • Goal setting related to project impact, AI system design, and infrastructure/developer efficiency
  • Directly or influencing partners to deliver impact through deep, thorough data-driven analysis
  • Drive large efforts across multiple teams
  • Define use cases, and develop methodology & benchmarks to evaluate different approaches
  • Apply in depth knowledge of how the ML infra interacts with the other systems around it
  • Mentor other engineers / research scientists & improve the quality of engineering work in the broader team
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

Senior ML Systems Engineer, Frameworks & Tooling

We’re looking for a senior engineer to help build, maintain and evolve the train...
Location
Location
Salary
Salary:
Not provided
cohere.com Logo
Cohere
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong engineering experience in large-scale distributed training or HPC systems
  • Deep familiarity with JAX internals, distributed training libraries, or custom kernels/fused ops
  • Experience with multi-node cluster orchestration (Slurm, Ray, Kubernetes, or similar)
  • Comfort debugging performance issues across CUDA/NCCL, networking, IO, and data pipelines
  • Experience working with containerized environments (Docker, Singularity/Apptainer)
  • A track record of building tools that increase developer velocity for ML teams
  • Excellent judgment around trade-offs: performance vs complexity, research velocity vs maintainability
  • Strong collaboration skills — you’ll work closely with infra, research, and deployment teams
Job Responsibility
Job Responsibility
  • Build and own the training framework responsible for large-scale LLM training
  • Design distributed training abstractions (data/tensor/pipeline parallelism, FSDP/ZeRO strategies, memory management, checkpointing)
  • Improve training throughput and stability on multi-node clusters (e.g., GB200/300, AMD, H200/100)
  • Develop and maintain tooling for monitoring, logging, debugging, and developer ergonomics
  • Collaborate closely with infra teams to ensure our cluster, container environments, and hardware configurations support high-performance training
  • Investigate and resolve performance bottlenecks across the ML systems stack
  • Build robust systems that ensure reproducible, debuggable, large-scale runs
What we offer
What we offer
  • An open and inclusive culture and work environment
  • Work closely with a team on the cutting edge of AI research
  • Weekly lunch stipend, in-office lunches & snacks
  • Full health and dental benefits, including a separate budget to take care of your mental health
  • 100% Parental Leave top-up for up to 6 months
  • Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • 6 weeks of vacation (30 working days!)
  • Fulltime
Read More
Arrow Right

Software Engineer: ML Infra

Generalist trains very large robot foundation models. This requires utilizing ve...
Location
Location
United States , San Mateo; Somerville
Salary
Salary:
200000.00 - 350000.00 USD / Year
generalistai.com Logo
Generalist AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Have managed large fleets of GPUs doing large-scale, long-term, highly distributed training runs or inference
  • Deep experience in Slurm or Kubernetes for ML workload orchestration
  • Have build high-scale ML data loaders and preparation systems
  • Deeply understand every layer of the ML hardware, storage, and networking stacks
  • Have experience in the NVidia GPU ecosystem
Job Responsibility
Job Responsibility
  • Owning our GPU compute fleets
  • Ensure our GPUs are easy for researchers to use and maximally utilized
  • Optimizing and improving ML data loading transport and storage in highly distributed fully utilized environments
  • Orchestration of robot inference fleets
What we offer
What we offer
  • Offers Equity
  • Fulltime
Read More
Arrow Right

Software Engineer, Systems ML - SW/HW Co-design

Meta is seeking an AI Software Engineer to join our Research & Development teams...
Location
Location
United States , Sunnyvale
Salary
Salary:
217000.00 USD / Year ▼
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • Specialized experience in one or more of the following machine learning/deep learning domains: Hardware accelerators architecture, GPU architecture, machine learning compilers, or ML systems, AI infrastructure, high performance computing, performance optimizations, or Machine learning frameworks (e.g. PyTorch), numerics and SW/HW co-design
  • Experience developing AI-System infrastructure or AI algorithms in C/C++ or Python
Job Responsibility
Job Responsibility
  • Apply relevant AI infrastructure and hardware acceleration techniques to build & optimize our intelligent ML systems that improve Meta’s products and experiences
  • Goal setting related to project impact, AI system design, and infrastructure/developer efficiency
  • Directly or influencing partners to deliver impact through deep, thorough data-driven analysis
  • Drive large efforts across multiple teams
  • Define use cases, and develop methodology & benchmarks to evaluate different approaches
  • Apply in depth knowledge of how the ML infra interacts with the other systems around it
  • Mentor other engineers / research scientists & improve the quality of engineering work in the broader team
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

Software Engineer, Systems ML - SW/HW Co-design

Meta is seeking an AI Software Engineer to join our Research & Development teams...
Location
Location
United States , Sunnyvale
Salary
Salary:
257000.00 USD / Year ▼
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • Specialized experience in one or more of the following machine learning/deep learning domains: Hardware accelerators architecture, GPU architecture, machine learning compilers, or ML systems, AI infrastructure, high performance computing, performance optimizations, or Machine learning frameworks (e.g. PyTorch), numerics and SW/HW co-design
  • Experience developing AI-System infrastructure or AI algorithms in C/C++ or Python
Job Responsibility
Job Responsibility
  • Apply relevant AI infrastructure and hardware acceleration techniques to build & optimize our intelligent ML systems that improve Meta’s products and experiences
  • Goal setting related to project impact, AI system design, and infrastructure/developer efficiency
  • Directly or influencing partners to deliver impact through deep, thorough data-driven analysis
  • Drive large efforts across multiple teams
  • Define use cases, and develop methodology & benchmarks to evaluate different approaches
  • Apply in depth knowledge of how the ML infra interacts with the other systems around it
  • Mentor other engineers / research scientists & improve the quality of engineering work in the broader team
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

Senior AI Data Engineer

We are looking for a Senior AI Data Engineer to join a high-impact AI product in...
Location
Location
United States
Salary
Salary:
Not provided
velvetech.com Logo
Velvetech
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of experience in Data Engineering / ML Engineering / AI Engineering
  • Strong programming skills in Python
  • Hands-on experience with PyTorch (training and deploying deep learning models)
  • Experience working with Vertex AI or similar ML platforms (GCP preferred)
  • Proven experience with vector databases (Milvus, Pinecone, or similar)
  • Strong knowledge of: Feature engineering techniques, Model evaluation and validation frameworks, Predictive inference systems
  • Experience with multiple database paradigms: Relational (PostgreSQL), Time-series (InfluxDB), Graph (Neo4j)
  • Solid understanding of embeddings and semantic/vector search systems
  • Experience implementing model lifecycle management, including: Drift detection, Monitoring, Governance
  • Strong understanding of scalable system design and performance optimization
Job Responsibility
Job Responsibility
  • Own and develop the biometric extraction model lifecycle (training, validation, deployment)
  • Design and maintain a vector memory layer using tools such as Milvus or Pinecone
  • Build and optimize predictive inference services for real-time and batch use cases
  • Develop and maintain data pipelines for PFM (Personal Financial Management) data preparation
  • Implement advanced feature engineering frameworks and model evaluation pipelines
  • Work with Vertex AI for model training, deployment, and orchestration
  • Manage and integrate heterogeneous data storage systems: InfluxDB (time-series data), PostgreSQL (relational data), Neo4j (graph data)
  • Develop vector embeddings pipelines and similarity search logic
  • Implement model governance processes: Drift detection and monitoring, Shadow-mode validation, Performance tracking and reporting
  • Design and apply optimization policies for inference latency, cost, and accuracy
What we offer
What we offer
  • FLEXIBLE working conditions
  • COOPERATIVE environment
  • Competitive salary
  • Many CHALLENGING and exciting projects with new opportunities and learning
  • GROWTH opportunities, skills and competencies improvement, and professional certification
  • In-company TRAINING (English, Software / DevOps / Project management / Design / Business)
  • Fulltime
Read More
Arrow Right

Senior Big Data Engineer

Start.io is a mobile marketing and audience platform. Start.io empowers the mobi...
Location
Location
Poland , Warsaw
Salary
Salary:
Not provided
start.io Logo
Start.io
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • B.Sc. or M.Sc. in Computer Science, Software Engineering, or other equivalent fields.
  • 5+ years of hands-on experience in backend or ML engineering
  • Strong Python skills and experience working with distributed systems and parallel data processing frameworks such as Spark (using PySpark or Scala), Dask, or similar technologies. Familiarity with Scala is a strong advantage, especially in performance-critical environments.
  • Proven track record in designing and scaling ML infrastructure
  • Deep understanding of ML workflows and lifecycle management
  • Experience in cloud environments (AWS, GCP, OCI) and containerized deployment (Kubernetes)
  • Understanding databases and SQL for data retrieval.
  • Strong communication skills and ability to drive initiatives independently
  • A passion for clean code, elegant architecture, and measurable impact
  • Monitoring and alerting tools (e.g. Grafana, Kibana)
Job Responsibility
Job Responsibility
  • Design and implement large-scale, distributed ML training pipelines
  • Build scalable infrastructure for data preprocessing, feature engineering, and model evaluation
  • Lead the technical design and development of new ML systems: from architecture to production
  • Collaborate cross-functionally with DS, infra teams, Product, BA and Engineering teams to define and deliver impactful solutions
  • Own the full lifecycle of ML infra: tooling, versioning, monitoring, automation, measuring results and quickly responding to critical issues.
  • Continuously research and adopt best-in-class practices in MLOps, performance tuning, and distributed systems
Read More
Arrow Right