CrawlJobs Logo
Scale Logo Scale · IT - Software Development

Ai Infrastructure Engineer, Core Infrastructure

United States, San Francisco 179400.00 - 310500.00 USD / Year · Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

As a Software Engineer on the ML Infrastructure team, you will design and build the next generation of foundational systems that power all ML Infrastructure compute at Scale - from model training and evaluation to large-scale inference and experimentation. Our platform is responsible for orchestrating workloads across heterogeneous compute environments (GPU, CPU, on-prem, and cloud), optimizing for reliability, cost efficiency, and developer velocity.

Job Responsibility

  • Design and maintain fault-tolerant, cost-efficient systems that manage compute allocation, scheduling, and autoscaling across clusters and clouds
  • Build common abstractions and APIs that unify job submission, telemetry, and observability across serving and training workloads
  • Develop systems for usage metering, cost attribution, and quota management, enabling transparency and control over compute budgets
  • Improve reliability and efficiency of large-scale GPU workloads through better scheduling, bin-packing, preemption, and resource sharing
  • Partner with ML engineers and API teams to identify bottlenecks and define long-term architectural standards
  • Lead projects end-to-end — from requirements gathering and design to rollout and monitoring — in a cross-functional environment

Requirements

  • 4+ years of experience building large-scale backend or distributed systems
  • Strong programming skills in Python, Go, or Rust, and familiarity with modern cloud-native architecture
  • Experience with containers and orchestration tools (Kubernetes, Docker) and Infrastructure as Code (Terraform)
  • Familiarity with schedulers or workload management systems (e.g., Kubernetes controllers, Slurm, Ray, internal job queues)
  • Understanding of observability and reliability practices (metrics, tracing, alerting, SLOs)
  • A track record of improving system efficiency, reliability, or developer velocity in production environments

Nice to have

  • Experience with multi-tenant compute platforms or internal PaaS
  • Knowledge of GPU scheduling, cost modeling, or hybrid cloud orchestration
  • Familiarity with LLM or ML training workloads, though deep ML expertise is not required

What we offer

  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
  • equity based compensation
Scale - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Ai Infrastructure Engineer, Core Infrastructure

8 matching positions

Sr Staff Engineer - Core Infrastructure

We are seeking a Senior Staff Engineer (L6) to lead the technical strategy and e...
Location
Location
United States , New York; Seattle; San Francisco; Sunnyvale
Salary
Salary:
267000.00 - 297000.00 USD / Year
uber.com Logo
Uber
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 12+ years of software engineering experience, with a focus on massive-scale distributed systems or infrastructure
  • Proven Track Record at Scale: Experience managing infrastructure that supports millions of concurrent users or petabyte-scale data processing
  • Deep Systems Expertise: Mastery of Kubernetes internals, container runtimes, and the Linux kernel, with the ability to debug impossible performance bottlenecks
  • Cloud-Native Fluency: Deep experience with cloud-native networking (Envoy, CNI, Service Mesh) and multi-cloud (AWS/GCP) architecture
  • Coding Proficiency: Expert-level proficiency in Go, Java, or C++
  • Leadership: Demonstrated ability to lead 40+ person technical initiatives and influence VPs and GMs on infrastructure investment
Job Responsibility
Job Responsibility
  • Architect Strategic Efficiency: Own the technical vision to drive fleet-wide CPU utilization and unit-cost optimization through ARM adoption (targeting XM+ cores) and silicon diversity
  • Scale AI & ML Infrastructure: Define the architecture for shared GPU pools and high-performance clusters to support 300x larger ranking models and Autonomous Vehicle data ingestion
  • Modernize the Data Plane: Drive the convergence of Uber’s networking stack toward industry standards (Kubernetes, Envoy, CNI) while enhancing SkyEdge for active-active multi-cloud resilience
  • Enforce Foundations & Reliability: Lead the 100% Done-Done initiative, ensuring every service follows standardized safe-deployment (Starship) and reaches 100% zero-trust authorization
  • Agentic Augmentation: Integrate AI-driven Minions and AIOps into the infrastructure to automate 80% of alerts and unlock thousands of years of developer productivity
  • Cross-Org Influence: Partner with Delivery, Rides, and AV teams to ensure the infrastructure isn't just a container, but a competitive advantage that accelerates their time-to-market
  • Mentor Staff+ Engineers: Act as a force multiplier by coaching the next generation of technical leaders and influencing company-wide engineering standards
What we offer
What we offer
  • Eligible to participate in Uber's bonus program
  • May be offered an equity award & other types of comp
  • Eligible to participate in a 401(k) plan
  • Various benefits
  • Fulltime
Read More
Arrow Right

Senior Software Engineer - AI Infrastructure (Scheduler) - CoreAI

The AI Platform organization builds the end-to-end Azure AI stack, from the infr...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C++, C#, Java, Scala, Rust, Go, TypeScript | OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Work on the design and development of the core AI Infrastructure distributed and in-cluster services that support large scale AI training and inferencing
  • Develop, test, and maintain control plane services written in C#, hosted on Service Fabric or Kubernetes (AKS) clusters
  • Enhance systems and applications to ensure high stability, efficiency and maintainability, low latency, tight cloud security
  • Provide operational support and DRI (on-call) responsibilities for the service
  • Develop and foster a deep understanding of the machine learning concepts, use cases, and relevant services used by our customers
  • Collaborate closely with service engineers, product managers, and internal applied research and data science teams within Microsoft to build better solutions together
  • Provide vision, expertise, and technical leadership to other team members
  • Help to grow talent in these areas
  • Embody our culture and values
  • Fulltime
Read More
Arrow Right

Senior AI Infrastructure Engineer

Together AI is building the AI Acceleration Cloud, an end-to-end platform for th...
Location
Location
Netherlands , Amsterdam
Salary
Salary:
Not provided
together.ai Logo
Together AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of professional software development experience and proficiency in at least one backend programming language (Golang desired)
  • 5+ years experience writing high-performance, well-tested, production quality code
  • Demonstrated experience with building and operating high-performance and/or globally distributed micro-service architectures across one or more cloud providers (AWS, Azure, GCP)
  • Excellent communication skills – able to write clear design docs and work effectively with both technical and non-technical team members
  • Strong systems knowledge across compute, networking, and storage, including concurrency, memory management, performant I/O, and scale
  • Experience with infrastructure automation tools (Terraform, Ansible), monitoring/observability stacks (Prometheus, Grafana), and CI/CD pipelines (GitHub Actions, ArgoCD)
Job Responsibility
Job Responsibility
  • Design, build, and maintain performant, secure, and highly-available backend services/operators that run in our data centers and automate hardware management, such as Infiniband partitioning, in-DC parallel storage provisioning, and VM provisioning
  • Design and build out the IaaS software layer for a new GB200 data center with thousands of GPUs
  • Work on a global multi-exabyte high-performance object store, serving massive datasets for pretraining
  • Build advanced observability stacks for our customers with automated node lifecycle management for fault-tolerant distributed pretraining
  • Perform architecture and research work for decentralized AI workloads
  • Work on the core, open-source Together AI platform
  • Create services, tools, and developer documentation
  • Create testing frameworks for robustness and fault-tolerance
  • Fulltime
Read More
Arrow Right

Staff Software Engineer, Core Infrastructure

At Harvey, we’re transforming how legal and professional services operate — not ...
Location
Location
United States , San Francisco
Salary
Salary:
236000.00 - 290000.00 USD / Year
harvey.ai Logo
Harvey
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of experience in Infrastructure Engineering or Platform Engineering in a production environment
  • Long track record building and scaling complex, large-scale distributed systems
  • Deep proficiency with cloud infrastructure platforms (Azure preferred
  • GCP or AWS experience transfers well)
  • Strong fluency in Infrastructure as Code (IaC) tools — Terraform, Pulumi, or CloudFormation
  • Solid understanding of Kubernetes, container orchestration, networking, and cloud security at scale
  • Experience with observability tools (Datadog, Sentry) and incident response practices (PagerDuty, Incident.io)
  • Strong programming skills in Python, Go, or similar languages
  • Excellent problem-solving skills, a "spidey sense" of where things could go wrong, and a commitment to operational excellence
Job Responsibility
Job Responsibility
  • Design and build scalable, fault-tolerant infrastructure systems that power Harvey's AI platform across multiple cloud regions
  • Own and evolve our multi-cloud infrastructure (Azure, GCP), including Kubernetes orchestration, networking, and container management
  • Lead technical initiatives around observability, incident response, and operational excellence — building systems that enable rapid detection and resolution of issues
  • Architect and optimize our distributed systems for reliability, including load balancing, quota management, and failover mechanisms
  • Partner with Product Engineering and Security teams to ensure our infrastructure is an accelerant, not a constraint
  • Drive infrastructure-as-code practices using tools like Terraform and Pulumi to enable reproducible, auditable deployments
  • Mentor engineers and raise the technical bar across the organization through code reviews, design reviews, and technical leadership
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits (401k match up to 4%)
  • flexible PTO
  • Offers Equity
  • Offers Bonus
  • relocation assistance
  • Fulltime
Read More
Arrow Right

Staff Software Engineer, Core Infrastructure

At Harvey, we’re transforming how legal and professional services operate — not ...
Location
Location
United States , New York
Salary
Salary:
201000.00 - 264000.00 USD / Year
harvey.ai Logo
Harvey
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of experience in Infrastructure Engineering or Platform Engineering in a production environment
  • Long track record building and scaling complex, large-scale distributed systems
  • Deep proficiency with cloud infrastructure platforms (Azure preferred
  • GCP or AWS experience transfers well)
  • Strong fluency in Infrastructure as Code (IaC) tools — Terraform, Pulumi, or CloudFormation
  • Solid understanding of Kubernetes, container orchestration, networking, and cloud security at scale
  • Experience with observability tools (Datadog, Sentry) and incident response practices (PagerDuty, Incident.io)
  • Strong programming skills in Python, Go, or similar languages
  • Excellent problem-solving skills, a "spidey sense" of where things could go wrong, and a commitment to operational excellence
Job Responsibility
Job Responsibility
  • Design and build scalable, fault-tolerant infrastructure systems that power Harvey's AI platform across multiple cloud regions
  • Own and evolve our multi-cloud infrastructure (Azure, GCP), including Kubernetes orchestration, networking, and container management
  • Lead technical initiatives around observability, incident response, and operational excellence — building systems that enable rapid detection and resolution of issues
  • Architect and optimize our distributed systems for reliability, including load balancing, quota management, and failover mechanisms
  • Partner with Product Engineering and Security teams to ensure our infrastructure is an accelerant, not a constraint
  • Drive infrastructure-as-code practices using tools like Terraform and Pulumi to enable reproducible, auditable deployments
  • Mentor engineers and raise the technical bar across the organization through code reviews, design reviews, and technical leadership
What we offer
What we offer
  • Equity plan
  • Bonus
  • Comprehensive health, dental and vision coverage
  • Retirement benefits (401k match up to 4%)
  • Flexible PTO
  • Relocation assistance
  • Fulltime
Read More
Arrow Right

Senior Software Engineer, Core Infrastructure

As a Software Engineer on the Core Infrastructure team at Harvey, you'll play a ...
Location
Location
United States , San Francisco
Salary
Salary:
200000.00 - 250000.00 USD / Year
harvey.ai Logo
Harvey
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years of experience in Infrastructure Engineering or Platform Engineering in a production environment
  • Long track record building and scaling complex, large-scale distributed systems
  • Deep proficiency with cloud infrastructure platforms (Azure preferred
  • GCP or AWS experience transfers well)
  • Strong fluency in Infrastructure as Code (IaC) tools — Terraform, Pulumi, or CloudFormation
  • Solid understanding of Kubernetes, container orchestration, networking, and cloud security at scale
  • Experience with observability tools (Datadog, Sentry) and incident response practices (PagerDuty, Incident.io)
  • Strong programming skills in Python, Go, or similar languages
  • Excellent problem-solving skills, a "spidey sense" of where things could go wrong, and a commitment to operational excellence
Job Responsibility
Job Responsibility
  • Design and build scalable, fault-tolerant infrastructure systems that power Harvey's AI platform across multiple cloud regions
  • Own and evolve our multi-cloud infrastructure (Azure, GCP), including Kubernetes orchestration, networking, and container management
  • Lead technical initiatives around observability, incident response, and operational excellence — building systems that enable rapid detection and resolution of issues
  • Architect and optimize our distributed systems for reliability, including load balancing, quota management, and failover mechanisms
  • Partner with Product Engineering and Security teams to ensure our infrastructure is an accelerant, not a constraint
  • Drive infrastructure-as-code practices using tools like Terraform and Pulumi to enable reproducible, auditable deployments
  • Mentor junior / intern engineers and raise the technical bar across the organization through code reviews, design reviews, and technical leadership
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits (401k match up to 4%)
  • flexible PTO
  • equity plan
  • bonus
  • Fulltime
Read More
Arrow Right

Senior DevOps Engineer (AI & Cloud Infrastructure)

We are seeking a Senior DevOps Engineer to design, deploy, and operate the next ...
Location
Location
United States , Palo Alto
Salary
Salary:
175000.00 - 250000.00 USD / Year
inflection.ai Logo
Inflection AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of hands-on experience in DevOps, Site Reliability Engineering, or ML Infrastructure supporting high-scale, production systems
  • Deep expertise in Azure and AWS, including storage, compute, networking, databases, and cloud-native monitoring services
  • Strong Kubernetes administration experience, including GPU scheduling, operator deployment, and management of core infrastructure components
  • experience with Slurm is highly desirable
  • Proven experience deploying, scaling, and operating Large Language Models (LLMs) and inference engines such as vLLM, TGI, or Triton
  • Strong experience with modern DevOps tooling: Terraform, Helm, Kustomize, ArgoCD, GitHub Actions or GitLab CI, Prometheus, Grafana, and Clickhouse
  • Advanced scripting and automation skills in Python and Bash, with the ability to debug complex distributed systems and optimize performance at scale
  • Demonstrated ability to troubleshoot LLM servers, Kubernetes workloads, GPU utilization, and cloud infrastructure bottlenecks
  • Have a bachelor’s degree or equivalent in a related field to the offered position requirements.
Job Responsibility
Job Responsibility
  • Architect, deploy, and operate large-scale LLM inference servers and AI applications with a focus on low latency, high availability, and production reliability
  • Design, provision, and maintain complex cloud architectures across Azure and AWS, including storage, compute, networking, databases, and native LLM services
  • Manage GPU-enabled Kubernetes clusters and Slurm-based HPC environments, optimizing resource allocation for AI training and inference workloads
  • Deploy and operate core Kubernetes infrastructure components and operators (GPU operators, ingress controllers, service meshes, CNIs, CSIs, and storage drivers)
  • Build scalable infrastructure-as-code and deployment workflows using Terraform, Helm, Kustomize, ArgoCD, and GitOps best practices
  • Design and maintain centralized observability systems using Prometheus, Grafana, Clickhouse, and cloud-native monitoring tools
  • Participate in on-call rotations, lead incident response, perform post-mortems, and continuously improve system reliability and SLAs.
What we offer
What we offer
  • Diverse medical, dental and vision options
  • 401k matching program
  • Unlimited paid time off
  • Parental leave and flexibility for all parents and caregivers
  • Support of country-specific visa needs for international employees living in the Bay Area
  • Meaningful equity component.
  • Fulltime
Read More
Arrow Right

Senior Software Engineer - Data Platform, AI Infrastructure

We are building a large-scale, productized data platform that powers critical in...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
  • Strong programming experience in Python
  • Experience building and operating large-scale distributed systems
  • Hands-on experience with: Backend services or APIs (e.g., FastAPI, Flask, or similar)
  • Cloud-based infrastructure (Azure, AWS, or GCP)
  • Monitoring and observability systems (metrics, logging, alerting)
  • Experience designing systems with reliability, scalability, and operational clarity in mind
  • Proven ability to own and deliver production systems end-to-end
  • Ability to break down ambiguous problems, ask the right questions, and execute effectively
Job Responsibility
Job Responsibility
  • Design, build, and operate core components of a distributed data platform, including: Orchestration systems (e.g., Airflow or equivalent)
  • Backend services and APIs (Python/FastAPI or similar)
  • Monitoring, alerting, and reliability systems
  • Own the end-to-end lifecycle of platform components - from design through deployment, scaling, and maintenance
  • Ensure systems meet requirements for availability, performance, and data reliability at large scale
  • Define and enforce standardized patterns for infrastructure, deployment, and observability across the platform
  • Partner with data engineering teams to enable efficient, reliable data processing workflows
  • Diagnose and resolve complex issues in distributed systems, including performance bottlenecks and failure modes
  • Contribute to infrastructure-as-code and deployment systems to support reproducibility and operational excellence
  • Drive continuous improvements in system robustness, cost efficiency, and operational clarity
What we offer
What we offer
  • Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
  • Fulltime
Read More
Arrow Right