Software Engineer, Networking - Inference Job at OpenAI (San Francisco)

Software Engineer, Inference – AMD GPU Enablement

We’re hiring engineers to scale and optimize OpenAI’s inference infrastructure a...

Location

United States , San Francisco

Salary:

295000.00 - 555000.00 USD / Year

OpenAI

Expiration Date

Until further notice

Requirements

Experience writing or porting GPU kernels using HIP, CUDA, or Triton
Familiarity with communication libraries like NCCL/RCCL
Experience working on distributed inference systems
Ability to solve end-to-end performance challenges across hardware, system libraries, and orchestration layers
Ability to thrive in a small, fast-moving team building new infrastructure from first principles

Job Responsibility

Own bring-up, correctness and performance of the OpenAI inference stack on AMD hardware
Integrate internal model-serving infrastructure (e.g., vLLM, Triton) into a variety of GPU-backed systems
Debug and optimize distributed inference workloads across memory, network, and compute layers
Validate correctness, performance, and scalability of model execution on large GPU clusters
Collaborate with partner teams to design and optimize high-performance GPU kernels for accelerators using HIP, Triton, or other performance-focused frameworks
Collaborate with partner teams to build, integrate and tune collective communication libraries (e.g., RCCL) used to parallelize model execution across many GPUs

What we offer

Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
401(k) retirement plan with employer match
Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
Mental health and wellness support
Employer-paid basic life and disability coverage
Annual learning and development stipend to fuel your professional growth
Daily meals in our offices, and meal delivery credits as eligible

Fulltime

Software Engineer, Inference - Multi Modal

OpenAI’s Inference team powers the deployment of our most advanced models - incl...

Location

United States , San Francisco

Salary:

295000.00 - 555000.00 USD / Year

OpenAI

Expiration Date

Until further notice

Requirements

Experience building and scaling inference systems for LLMs or multimodal models
Worked with GPU-based ML workloads and understand the performance dynamics of large models, especially with complex data like images or audio
Enjoy experimental, fast-evolving work and collaborating closely with research
Comfortable dealing with systems that span networking, distributed compute, and high-throughput data handling
Familiarity with inference tooling like vLLM, TensorRT-LLM, or custom model parallel systems
Own problems end-to-end and are excited to operate in ambiguous, fast-moving spaces

Job Responsibility

Design and implement inference infrastructure for large-scale multimodal models
Optimize systems for high-throughput, low-latency delivery of image and audio inputs and outputs
Enable experimental research workflows to transition into reliable production services
Collaborate closely with researchers, infra teams, and product engineers to deploy state-of-the-art capabilities
Contribute to system-level improvements including GPU utilization, tensor parallelism, and hardware abstraction layers

What we offer

Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
401(k) retirement plan with employer match
Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
Mental health and wellness support
Employer-paid basic life and disability coverage
Annual learning and development stipend to fuel your professional growth
Daily meals in our offices, and meal delivery credits as eligible

Fulltime

Software Engineer, Load Balancing - Inference

We’re looking for a senior engineer to design and build the load balancer that w...

Location

United States , San Francisco

Salary:

293000.00 - 490000.00 USD / Year

OpenAI

Expiration Date

Until further notice

Requirements

Deep experience designing and operating large-scale distributed systems, particularly load balancers, service gateways, or traffic routing layers
5+ years of experience designing in theory for and debugging in practice for the algorithmic and systems challenges of consistent hashing, sticky routing, and low-latency connection management
5+ years of experience as a software engineer and systems architect working on high-scale, high-reliability infrastructure
Strong debugging mindset and enjoy spending time in tracing, logs, and metrics to untangle distributed failures
Comfortable writing and reviewing production code in Rust or similar systems languages (C/C++, Java, Go, Zig, etc)
Operated in big tech or high-growth environments and are excited to apply that experience in a faster-moving setting
Take ownership of problems end-to-end and are excited to build something foundational to how our models interact with the world

Job Responsibility

Architect and build the gateway / network load balancer that fronts all research jobs, ensuring long-lived connections remain consistent and performant
Design traffic stickiness and routing strategies that optimize for both reliability and throughput
Instrument and debug complex distributed systems — with a focus on building world-class observability and debuggability tools (distributed tracing, logging, metrics)
Collaborate closely with researchers and ML engineers to understand how infrastructure decisions impact model performance and training dynamics
Own the end-to-end system lifecycle: from design and code to deploy, operate, and scale
Work in an outcome-oriented environment where everyone contributes across layers of the stack, from infra plumbing to performance tuning

What we offer

Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
401(k) retirement plan with employer match
Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
Mental health and wellness support
Employer-paid basic life and disability coverage
Annual learning and development stipend to fuel your professional growth
Daily meals in our offices, and meal delivery credits as eligible

Fulltime

Senior Software Development Engineer in Test (SDET) - AI Cluster Networking and Security

In AI infrastructure organization, simplifying large hardware deployments with p...

Location

India , Bengaluru

Salary:

Not provided

Cerebras Systems

Expiration Date

Until further notice

Requirements

Bachelor's or master's degree in engineering in computer science, electrical, AI, data science of related field
10+ years of experience in testing one of areas like enterprise software, distributed systems, datacenter hardware and software
Experience working in large enterprise or cloud networking infrastructure, high speed switches, routers, firewalls
Experience in qualifying networking vendor platforms like Juniper, Arista or Cisco and network test equipment like Ixia/Spirent
Experience in Datacenter technology like BGP, ECN, PFC
Experience testing networking security, compliance and firewalls
Strong coding skills in one of the programming languages like python, golang or C/C++
Strong debugging skills to debug issues in large distributed systems, hardware, and software. Experience with debugging tools like gdb, strace, networking monitors
Strong understanding of operating systems internals like memory management, file system working, security basics and performance
Strong understanding of datacenter layout, device performance characteristics like PCIe, networking and storage

Job Responsibility

Innovate and execute tests on cutting edge AI infrastructure
Define optimized test strategies and methodologies
Be a quick learner, adapt to new technologies
Build a strong understanding of how to break these large distributed systems challenge into smaller components that can be unit tested
Automate first approach - Aim for 100% automated tests to test all cluster features in areas of high availability, failure scenarios, performance, stress and security
Champion cluster security, reliability for uptime of 99.9999% and ease of use with observability
Test all components of AI cluster including but not limited to cluster software involving kubernetes, prometheus and grafana. Cluster hardware components like ML wafer scale accelerators, CPU runtime nodes, High speed swarmx interconnect, High speed data transfer of weights through memoryx interconnect
Qualify cluster networking solutions which consists of high-speed switches, routers and optics from various vendors
Qualify cluster security features including OS security, network security, cloud compliance user access and security certifications

What we offer

Build a breakthrough AI platform beyond the constraints of the GPU
Publish and open source their cutting-edge AI research
Work on one of the fastest AI supercomputers in the world
Enjoy job stability with startup vitality
Our simple, non-corporate work culture that respects individual beliefs

New

Software Engineer 2

The Artificial Intelligence (AI) Frameworks team at Microsoft develops the AI so...

Location

India , Noida

Salary:

Not provided

Microsoft Corporation

Expiration Date

Until further notice

Requirements

B.S. or advanced degree in computer engineering, computer science, or related fields
4+ years of software development and experience of leading at least two commercialized features from requirement to deployment
4+ years' experience with C/C++ , python low levels
Exposure to NVIDIA/AMD development environment comprising CUDA / ROCm for Deep Neural Network training and inference development

Job Responsibility

Identify requirements, scope solutions, estimate work, schedule deliverables
Apply strong engineering principles for defining robust and maintainable architectures and designs
Collaborate broadly across multiple disciplines from hardware designers, system engineers, tool developers, performance analysis and with ML application developers
Perform software development in C/C++, C#, Python, and other languages

Fulltime

Staff Software Engineer

You'll help build Confluent Cloud's AI capabilities — the layer that lets custom...

Location

Canada

Salary:

225100.00 - 260500.00 CAD / Year

Confluent

Expiration Date

Until further notice

Requirements

10+ years of significant experience designing, building, and operating distributed systems or cloud-native backend infrastructure in production
Strong working knowledge of Kubernetes and distributed-systems patterns (control loops, API servers, high-scale control planes), plus the fundamentals — containerization, networking, resource isolation
Proficiency in at least one of Go, Java, or Python, and the willingness to work across all three
A track record of leading cross-team technical work: turning ambiguous requirements into designs others can rally behind
Excellent written and verbal communication — you can write a design doc that aligns people who don't report to you

Job Responsibility

Design and build the backend services (primarily Go, Java, and Python) that run AI and model inference on real-time data
Own features end to end — drafting the design, aligning stakeholders inside and outside the team, and driving the decision to a conclusion
Make the technical calls on systems that span teams: model lifecycle, inference routing, and agent execution
Own the quality of what you ship — code, test coverage, documentation, operability, and rollout safety
Make the engineers around you better through code review, design feedback, and being someone the team trusts with ambiguous, cross-cutting work
Participate in on-call for the services your team owns, and help keep the team's processes and rituals healthy

What we offer

Remote-First Work
Robust Insurance Benefits
Flexible Time Away
The Best Teammates
Experience Ambassadors
Open and Honest Culture
Well-Being and Growth
Offers Equity

Fulltime

Backend Software Engineer - Engine Team (Voice Agent)

Deepgram is looking for a backend software engineer to lead the design and imple...

Location

United States

Salary:

150000.00 - 220000.00 USD / Year

Deepgram

Expiration Date

Until further notice

Requirements

3+ years of experience in an industry role
Programming experience in Rust (or C, C++), with competence in Python
Excellent communication and organizational skills, both written and verbal
A high level of experience and understanding of version control
preferably git
Comprehensive experience with UNIX-style systems

Job Responsibility

Improve Deepgram’s core inference services including areas in networking, speech processing, model orchestration, and observability
Develop integrations with cutting edge in-house, third-party, and open-source AI models for perception and managing conversational dynamics
Debug complex system issues that include networking, scheduling, and highly concurrent workloads
Rapidly customize backend services to support our customer needs
Partner with Product to design and implement new services, features, and/or products end to end

What we offer

Medical, dental, vision benefits
Annual wellness stipend
Mental health support
Life, STD, LTD Income Insurance Plans
Unlimited PTO
Generous paid parental leave
Flexible schedule
12 Paid US company holidays
Quarterly personal productivity stipend
One-time stipend for home office upgrades

Fulltime

Software Engineer - Machine Learning

Roku is changing how the world watches TV. Roku is the #1 TV streaming platform ...

Location

India , Bengaluru

Salary:

Not provided

Roku

Expiration Date

Until further notice

Requirements

5+ years of experience applying Machine Learning to concrete problems at large scale
Strong CS fundamentals
Solid understanding of Data Science and ML fundamentals – Regression, Classification, Tree-based approach, Neural network, and sequence-based models
Understanding of NLP approaches like W2V or Bert
Good understanding of system architecture
Experience in big data technologies – streaming architecture, data pipelines, etc.
Bachelors in Computer Science, Statistics, or related field

Job Responsibility

Apply state of the art ML on recommendations using techniques in deep learning, bandits, transformers, LLMs, causal inference, and optimizations
Run online AB tests and analyze them against critical business KPIs
Collaborate with US engineering teams and cross-functional teams to translate business requirements into technical specifications
Nurture ML ecosystem to withstand scale, developer velocity and future business shifts
Help in training and mentoring new team members

What we offer

Global access to mental health and financial wellness support and resources
Healthcare (medical, dental, and vision)
Life, accident, disability, commuter, and retirement options (401(k)/pension)
Time off in accordance with local leave policies

Fulltime

Select Country

Software Engineer, Networking - Inference

Job Description

Job Responsibility

Requirements

Nice to have

What we offer

Looking for more opportunities?