CrawlJobs Logo

Engineering Manager - Inference

United States, San Francisco 300000.00 - 385000.00 USD / Year · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

We are looking for an Inference Engineering Manager to lead our AI Inference team. This is a unique opportunity to build and scale the infrastructure that powers Perplexity's products and APIs, serving millions of users with state-of-the-art AI capabilities. You will own the technical direction and execution of our inference systems while building and leading a world-class team of inference engineers. Our current stack includes Python, PyTorch, Rust, C++, and Kubernetes. You will help architect and scale the large-scale deployment of machine learning models behind Perplexity's Comet, Sonar, Search, Deep Research products.

Job Responsibility

  • Lead and grow a high-performing team of AI inference engineers
  • Develop APIs for AI inference used by both internal and external customers
  • Architect and scale our inference infrastructure for reliability and efficiency
  • Benchmark and eliminate bottlenecks throughout our inference stack
  • Drive large sparse/MoE model inference at rack scale, including sharding strategies for massive models
  • Push the frontier with building inference systems to support sparse attention, disaggregated pre-fill/decoding serving, etc.
  • Improve the reliability and observability of our systems and lead incident response
  • Own technical decisions around batching, throughput, latency, and GPU utilization
  • Partner with ML research teams on model optimization and deployment
  • Recruit, mentor, and develop engineering talent
  • Establish team processes, engineering standards, and operational excellence

Requirements

  • 5+ years of engineering experience with 2+ years in a technical leadership or management role
  • Deep experience with ML systems and inference frameworks (PyTorch, TensorFlow, ONNX, TensorRT, vLLM)
  • Strong understanding of LLM architecture: Multi-Head Attention, Multi/Grouped-Query Attention, and common layers
  • Experience with inference optimizations: batching, quantization, kernel fusion, FlashAttention
  • Familiarity with GPU characteristics, roofline models, and performance analysis
  • Experience deploying reliable, distributed, real-time systems at scale
  • Track record of building and leading high-performing engineering teams
  • Experience with parallelism strategies: tensor parallelism, pipeline parallelism, expert parallelism
  • Strong technical communication and cross-functional collaboration skills

Nice to have

  • Experience with CUDA, Triton, or custom kernel development
  • Background in training infrastructure and RL workloads
  • Experience with Kubernetes and container orchestration at scale
  • Published work or contributions to inference optimization research

What we offer

  • Equity
  • Health
  • Dental
  • Vision
  • Retirement
  • Fitness
  • Commuter and dependent care accounts

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Engineering Manager - Inference

8 matching positions

Engineering Manager, Inference Platform

Cerebras Systems builds the world's largest AI chip, 56 times larger than GPUs. ...
Location
Location
United States; Canada , Sunnyvale; Toronto
Salary
Salary:
Not provided
cerebras.net Logo
Cerebras Systems
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years in high-scale software engineering
  • 3+ years leading distributed systems or ML infra teams
  • strong coding and review skills
  • Proven track record scaling LLM inference: optimizing latency (<100ms P99), throughput, batching, memory/IO efficiency and resources utilization
  • Expertise in distributed inference/training for modern LLMs
  • understanding of AI/ML ecosystems, including public clouds (AWS/GCP/Azure)
  • Hands-on with model-serving frameworks (e.g. vLLM, TensorRT-LLM, Triton or similar) and ML stacks (PyTorch, Hugging Face, SageMaker)
  • Deep experience with orchestration (Kubernetes/EKS, Slurm), large clusters, and low-latency networking
  • Strong background in monitoring and reliability engineering (Prometheus/Grafana, incident response, post-mortems)
  • Demonstrated ability to recruit and retain high-performing teams, mentor engineers, and partner cross-functionally to deliver customer-facing products
Job Responsibility
Job Responsibility
  • Provide hands-on technical leadership, owning the technical vision and roadmap for the Cerebras Inference Platform, from internal scaling to on-prem customer solutions
  • Lead the end-to-end development of distributed inference systems, including request routing, autoscaling, and resource orchestration on Cerebras' unique hardware
  • Drive a culture of operational excellence, guaranteeing platform reliability (>99.9% uptime), performance, and efficiency
  • Lead, mentor, and grow a high-caliber team of engineers, fostering a culture of technical excellence and rapid execution
  • Productize the platform into an enterprise-ready, on-prem solution, collaborating closely with product, ops, and customer teams to ensure successful deployments
What we offer
What we offer
  • Build a breakthrough AI platform beyond the constraints of the GPU
  • Publish and open source their cutting-edge AI research
  • Work on one of the fastest AI supercomputers in the world
  • Enjoy job stability with startup vitality
  • Our simple, non-corporate work culture that respects individual beliefs
  • Fulltime
Read More
Arrow Right

Principal Software Engineering Manager - Substrate Efficiency

M365 Copilot inference is a high-impact engineering team advancing applied AI an...
Location
Location
United States , Redmond
Salary
Salary:
142800.00 - 274800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Job Responsibility
Job Responsibility
  • Build and lead a high-performing engineering team focused on inference runtime efficiency and model execution performance
  • Define and drive strategy to improve throughput per GPU through runtime optimizations
  • Increase engineering agility, enabling faster experimentation, iteration, and rollout of performance improvements
  • Partner across M365 Core, AI Core, Azure, and Microsoft Research to co-design and productionize advanced inference optimizations
  • Establish metrics, telemetry, and experimentation frameworks to measure efficiency gains and guide investment decisions
  • Own live-site performance, reliability, and operational excellence for inference engines at scale
  • Drive alignment across partner teams on engine interfaces, performance goals, and optimization priorities.
  • Fulltime
Read More
Arrow Right

Principal Software Engineering Manager - AI Frameworks

As a Principal Software Engineering Manager - AI Frameworks on the team, you wil...
Location
Location
United States , Redmond
Salary
Salary:
139900.00 - 304200.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Master’s Degree in Computer Science or related technical field AND 10+ years of software engineering experience, including 6+ years in engineering management, OR Bachelor’s Degree in Computer Science or related technical field AND 12+ years of software engineering experience, including 6+ years in engineering management, or equivalent experience
  • Strong technical foundation in software engineering principles, computer architecture, GPU architecture, and hardware acceleration for neural networks, with the ability to guide teams working in these areas
  • Experience leading teams responsible for end-to-end performance analysis and optimization of LLMs, AI systems, or HPC workloads, including use of GPU profiling and performance analysis tools
  • Demonstrated ability to lead cross-team initiatives, align stakeholders, and translate research or platform capabilities into scalable, production-ready solutions
  • Proven people leadership skills, including hiring, coaching, performance management, and career development, with a track record of building high-performing, inclusive teams
  • Exposure to AI / ML infrastructure, including DNN or LLM training and/or inference systems, and experience with at least one modern deep learning framework (e.g., PyTorch, TensorFlow, ONNX Runtime)
  • Familiarity with GPU software stacks and acceleration technologies such as CUDA, ROCm, Triton, or equivalent, sufficient to guide technical direction and evaluate tradeoffs
Job Responsibility
Job Responsibility
  • Lead and develop a team of engineers working across multiple layers of the AI software stack to enable large-scale training and inference
  • Set technical vision and execution strategy for model performance benchmarking, optimization, and deployment across GPUs and Microsoft hardware
  • Drive performance outcomes by prioritizing and overseeing efforts to benchmark, profile, debug, and optimize training and inference workloads
  • Own performance health by establishing mechanisms to monitor regressions, measure impact, and continuously improve time-to-deploy and hardware efficiency
  • Partner cross-functionally with research, product, infrastructure, and hardware teams to deliver scalable, production-ready AI performance improvements
  • Balance short-term delivery and long-term investments, ensuring the team’s work aligns with organizational goals, platform roadmaps, and Azure capex objectives
  • Build a strong engineering culture through coaching, feedback, hiring, and career development, enabling the team to operate with increasing autonomy and impact
  • Fulltime
Read More
Arrow Right

Principal Software Engineering Manager, Simulation Platform - CoreAI

The AI Frameworks team at Microsoft develops AI software that enables running AI...
Location
Location
United States , Multiple Locations
Salary
Salary:
163000.00 - 296400.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science, or related technical discipline
  • 5+ years of experience building/managing team of software engineers
  • 5+ years of experience managing a software project
  • 10+ years of experience in computer architecture and/or embedded systems/software
  • 10+ years of software development experience
  • 10+ years of experience with C++ based object-oriented programming and design
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • 1+ years’ experience with Python
  • A deep technical background and solid foundation in computer architecture, system/hardware simulation, embedded software development and/or firmware
  • Experience designing and/or managing large C++ OOP, scalable, multi-threaded and multi-process software
Job Responsibility
Job Responsibility
  • Managing a team of software engineers
  • Managing development of AI chip simulator, which involves writing requirements, scoping and planning solutions, estimating and assigning work, scheduling and tracking deliverables, integration and releases to partner team, documentation
  • Technical contribution to design, code quality reviews, and capable to step in with hands-on code development when necessary (C++ and Python)
  • Collaborate broadly across multiple disciplines and with various partner teams from hardware designers to AI models developers
  • Fulltime
Read More
Arrow Right

Principal Software Engineering Manager

Join the Microsoft Location Service Team and Lead the Future of Location-Based E...
Location
Location
Spain , Barcelona
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Demonstrated professional experience in software engineering, including designing, building, and operating large‑scale, complex, and data‑intensive systems
  • Demonstrated experience in an engineering management role, including leading, coaching, and scaling engineering teams responsible for delivering high‑quality, reliable software
  • Solid technical background in distributed systems, data platforms, or machine learning–enabled services, with experience participating in architecture reviews and technical decision‑making
  • Proven ability to lead multi‑team initiatives, operate effectively in ambiguous problem spaces, and influence outcomes across organizational boundaries
  • Demonstrated track record of developing engineers and leaders, and of fostering inclusive, high‑performing engineering cultures focused on quality, ownership, and continuous improvement
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Lead, coach, and grow a team of engineers and scientists, fostering a culture of technical excellence, accountability, and continuous learning
  • Set and communicate a clear technical vision and execution strategy for core location services and inference platforms, aligning team priorities with broader organizational goals
  • Enable teams to adopt AI‑assisted and agentic coding practices to improve engineering velocity, code quality, and long‑term maintainability
  • Drive delivery of large-scale, highly reliable systems, balancing innovation with operational excellence across availability, reliability, performance, and cost
  • Partner with product, applied science, data, and partner teams to translate ambiguous problem spaces into clear, high-impact solutions
  • Lead identification of cross-team dependencies and drive the creation and review of design documents for complex products, services, and platforms
  • Influence org-level architecture, quality standards, and investment decisions across Microsoft’s location ecosystem
  • Serve as a role model for Microsoft values by creating a psychologically safe and inclusive environment where diverse perspectives thrive
  • Fulltime
Read More
Arrow Right

Engineering Manager - Machine Learning Infrastructure

We build simple yet innovative consumer products and developer APIs that shape h...
Location
Location
United States , New York
Salary
Salary:
216000.00 - 367200.00 USD / Year
plaid.com Logo
Plaid
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8–10 years of experience in ML infrastructure, including direct hands-on expertise as an engineer, IC/TL
  • 2+ years of experience managing infrastructure or ML platform engineers
  • Proven experience delivering and operating ML or AI infrastructure at scale
  • Solid technical depth across ML/AI infrastructure domains (e.g., feature stores, pipelines, deployment, inference, observability)
  • Demonstrated ability to drive execution on complex technical projects with cross-team stakeholders
  • Strong communication and stakeholder management skills
Job Responsibility
Job Responsibility
  • Lead and support the ML Infra team, driving project execution and ensuring delivery on key commitments
  • Build and launch Plaid’s next-generation feature store to improve reliability and velocity of model development
  • Define and drive adoption of an ML Ops “golden path” for secure, scalable model training, deployment, and monitoring
  • Ensure operational excellence of ML pipelines, deployment tooling, and inference systems
  • Partner with ML product teams to understand requirements and deliver solutions that accelerate model development and iteration
  • Recruit, mentor, and develop engineers, fostering a collaborative and high-performing team culture
  • Fulltime
Read More
Arrow Right

Technical Support Engineering Manager - Azure AI & Governance

As a Support Engineering Manager, you will lead a team of support engineering ro...
Location
Location
Romania , Bucharest
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s Degree in Computer Science, Information Technology (IT), Data Science, Engineering, or a related field AND 3+ years of experience in operational excellence, service delivery, customer-facing roles, technical operations, or program management OR 5+ years of equivalent professional experience in lieu of a degree
  • 1+ year of people management or team leadership experience, including coaching, mentoring, or guiding early-career professionals
  • Foundational understanding of Artificial Intelligence concepts, including: a) Basic machine learning principles (e.g., supervised vs. unsupervised learning, model training, inference
  • b) High-level understanding of how Generative AI systems work (e.g., large language models, prompts, tokens, limitations)
  • Awareness of Responsible AI principles, including topics such as fairness, reliability, privacy, transparency, and ethical use of AI systems
  • Strong communication skills in English (written and verbal), with the ability to explain technical or AI-related concepts to non-technical audiences
  • Demonstrated customer empathy and problem-solving mindset, with the ability to balance business impact, technical constraints, and user trust
  • Ability to meet Microsoft, customer and / or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire / transfer and every two years thereafter.
Job Responsibility
Job Responsibility
  • Identifies opportunities to engage with high value or area customers to provide solutions proactively
  • Identifies resources needed to resolve bugs. Drives visibility of the product bug to ensure timely engagement and/or action. Tracks pending cases due to bugs and influences product team to improve resolution and prevent customers from being impacted
  • Ensures that his or her team understands the product feedback cycle and/or proper coding of customer cases for compliance
  • Ensures team member participation in case triage meetings and/or case discussions to share knowledge with other engineers and develop efficient customer solutions by prioritizing issues, identifying resources, reviewing processes, or providing additional customer support
  • Identifies resources needed to implement automatization or tools
  • Implements processes for responding to and resolving issues
  • Analyzes group readiness strategy proactively to ensure the team has the correct readiness plan and executes readiness plan for new and existing technologies and feature releases. Provides feedback to global readiness team to ensure they have they appropriate readiness strategy and build material. Creates Microsoft role guides and ensures Technical Support Engineers have attended compliance trainings
  • Collaborates on cross-team and cross-product technical issues by working with resources from other groups including support engineering groups, product groups, services team, and account team as needed to resolve complex customer issues
  • Tracks resolution speed and removes roadblocks (e.g., skill gaps, handle difficult customers) preventing issues from being resolved. Acts as a management escalation point in case resolution as necessary. Analyzes the business impact and utilizes this information to involve the right resources proactively
  • Ensures customers stay informed as to the status/solution of their issue and manages customer relationship. Monitors tickets of direct reports and delivery units to ensure resolution. Uses tools and strategy to monitor the performance of a team. Serves as the first line of management escalation
  • Fulltime
Read More
Arrow Right

Senior Staff Engineering Manager - Applied AI

GEICO is in the midst of an exciting transformation as a product and tech powere...
Location
Location
United States , Chevy Chase; Palo Alto; Dallas
Salary
Salary:
140000.00 - 300000.00 USD / Year
geico.com Logo
Geico
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years hands-on software or solutions engineering
  • recent experience architecting/building production systems
  • 5+ years leading Agile teams delivering customer-impacting products
  • proven track record shipping AI/ML-enabled features to production with measurable impact
  • strong cloud experience (AWS or Azure)
  • CI/CD, SDLC, and modern engineering practices
  • practical MLOps/LLMOps experience (model lifecycle, telemetry, A/B testing, online and batch inference)
  • excellent stakeholder management and communication
  • Bachelor’s in CS/IS or related field (or equivalent experience)
Job Responsibility
Job Responsibility
  • Partner with product, data science, and AI Platform teams to define and deliver an Applied AI roadmap for claims tied to measurable business outcomes
  • Lead end-to-end delivery of AI features in the claims domain (e.g., claims triage/routing, adjuster assistance, subrogation/recovery opportunities, workload optimization)
  • Establish robust MLOps/LLMOps (model/prompt/version management, evaluation, feature/vector stores, CI/CD for ML, online/batch inference, cost/performance tuning)
  • Rapidly prototype using AI-assisted development tools (e.g., Cursor, Claude Code) to validate concepts and accelerate delivery
  • Ensure reliability, security, and Responsible AI (privacy-by-design, guardrails, human-in-the-loop, auditability)
  • Drive engineering excellence (clean architecture, scalable APIs, event-driven integration, observability, automated testing, SLOs/runbooks/on-call)
  • Recruit, coach, and grow a high-performing team
  • champion Agile practices and operational excellence
What we offer
What we offer
  • Comprehensive Total Rewards program that offers personalized coverage tailor-made for you and your family’s overall well-being
  • Financial benefits including market-competitive compensation
  • a 401K savings plan vested from day one that offers a 6% match
  • performance and recognition-based incentives
  • and tuition assistance
  • Access to additional benefits like mental healthcare as well as fertility and adoption assistance
  • Supports flexibility- We provide workplace flexibility as well as our GEICO Flex program, which offers the ability to work from anywhere in the US for up to four weeks per year
  • Fulltime
Read More
Arrow Right