Software Engineer, Load Balancing - Inference Job at OpenAI (San Francisco)

Software Engineer, Networking - Inference

We’re looking for a senior engineer to design and build the load balancer that w...

Location

United States , San Francisco

Salary:

325000.00 - 490000.00 USD / Year

OpenAI

Expiration Date

Until further notice

Requirements

Deep experience designing and operating large-scale distributed systems, particularly load balancers, service gateways, or traffic routing layers
5+ years of experience designing in theory for and debugging in practice for the algorithmic and systems challenges of consistent hashing, sticky routing, and low-latency connection management
5+ years of experience as a software engineer and systems architect working on high-scale, high-reliability infrastructure
Strong debugging mindset and enjoy spending time in tracing, logs, and metrics to untangle distributed failures
Comfortable writing and reviewing production code in Rust or similar systems languages (C/C++, Java, Go, Zig, etc)
Operated in big tech or high-growth environments and are excited to apply that experience in a faster-moving setting
Take ownership of problems end-to-end and are excited to build something foundational to how our models interact with the world

Job Responsibility

Architect and build the gateway / network load balancer that fronts all research jobs, ensuring long-lived connections remain consistent and performant
Design traffic stickiness and routing strategies that optimize for both reliability and throughput
Instrument and debug complex distributed systems — with a focus on building world-class observability and debuggability tools (distributed tracing, logging, metrics)
Collaborate closely with researchers and ML engineers to understand how infrastructure decisions impact model performance and training dynamics
Own the end-to-end system lifecycle: from design and code to deploy, operate, and scale
Work in an outcome-oriented environment where everyone contributes across layers of the stack, from infra plumbing to performance tuning

What we offer

Offers Equity
Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
401(k) retirement plan with employer match
Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
Mental health and wellness support
Employer-paid basic life and disability coverage
Annual learning and development stipend to fuel your professional growth

Fulltime

New

Software Engineer 2

Microsoft Azure AI Inference platform is the next generation cloud business posi...

Location

United States , Redmond

Salary:

100600.00 - 199000.00 USD / Year

Microsoft Corporation

Expiration Date

Until further notice

Requirements

Bachelor’s degree in Computer Science or a related technical field AND 2+ years of technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, or Golang, OR equivalent experience
Ability to meet Microsoft, customer, and/or government security screening requirements for this role
Technical background with a solid foundation in software engineering principles, distributed computing, and system architecture
Experience working on high-scale, reliable online systems
Experience with real-time online services requiring low latency and high throughput
Experience working with Layer 7 (L7) network proxies and gateways
Knowledge of network architecture and concepts, including HTTP and TCP protocols, authentication, and session management
Knowledge and experience with OSS, Docker, Kubernetes, C++, Golang, or equivalent programming languages
Cross-team collaboration skills and the desire to collaborate in a team of researchers and developers
Ability to independently lead projects

Job Responsibility

Design and implement core inference infrastructure for serving frontier AI models in production
Identify and drive improvements to end-to-end inference performance and efficiency of state-of-the-art LLMs and GenAI models from OpenAI, Anthropic and xAI hosted on AI Foundary
Design and implement efficient load scheduling and balancing strategies, by leveraging key insights and features of the model and workload
Scale the platform to support the growing inferencing demand and maintain high availability
Deliver critical capabilities required to serve the latest and greatest Gen AI models such as GPT5, Realtime audio, Sora, and enable fast time to market for them
Drive generic features to cater to the needs of customers such as GitHub, M365, Microsoft AI and third-party companies
Collaborate with our partners both internal and external
Embody Microsoft's Culture and Values

Fulltime

Software Engineer, Caching Infrastructure

The Caching Infrastructure team is responsible for building a caching layer that...

Location

United States , San Francisco

Salary:

230000.00 - 385000.00 USD / Year

OpenAI

Expiration Date

Until further notice

Requirements

5+ years of experience building and scaling distributed systems, with a strong focus on caching, load balancing, or storage systems
Deep expertise with Redis, Memcached, or similar solutions, including clustering, durability configurations, client-side connection patterns, and performance tuning
Production experience with Kubernetes, service meshes (e.g., Envoy), and autoscaling systems
Think rigorously about latency, reliability, throughput, and cost in designing platform capabilities
Thrive in a fast-paced environment and enjoy balancing pragmatic engineering with long-term technical excellence

Job Responsibility

Design, build, and operate OpenAI’s multi-tenant caching platform used across inference, identity, quota, and product experiences
Define the long-term vision and roadmap for caching as a core infra capability, balancing performance, durability, and cost
Collaborate with other infra teams (e.g., networking, observability, databases) and product teams to ensure our caching platform meets their needs

What we offer

Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
401(k) retirement plan with employer match
Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
Mental health and wellness support
Employer-paid basic life and disability coverage
Annual learning and development stipend to fuel your professional growth
Daily meals in our offices, and meal delivery credits as eligible

Fulltime

Sr Staff Engineer Software, Fullstack (Prisma AIRS) - NetSec

Join our team building a cutting-edge multi-tenanted GenAI Security Platform tha...

Location

India , Bengaluru

Salary:

Not provided

Palo Alto Networks

Expiration Date

Until further notice

Requirements

Proven experience building and scaling multi-tenant SaaS platforms with strict data isolation
Strong knowledge of API design, RESTful principles, and OpenAPI specifications
Proficiency in modern JavaScript frameworks (React, Vue, or Svelte) with TypeScript
Experience building data-intensive dashboards with complex visualisations and real-time data
Strong CSS/styling skills and responsive design principles
Demonstrated experience working with production AI/ML systems at scale
Practical experience integrating LLM APIs and managing inference at scale
Understanding of LLM operational challenges: rate limiting, cost optimisation, latency management, fallback strategies
Familiarity with AI agent frameworks (LangChain, AutoGen, MCP, or similar)
Knowledge of prompt engineering, semantic search, and vector databases

Job Responsibility

Design and implement high-performance REST APIs with enterprise-grade multi-tenant isolation and strict security boundaries
Work on distributed systems architecture handling high-throughput workloads with mission-critical uptime requirements
Build responsive dashboards and administrative interfaces for platform management, data visualisation, and system configuration
Integrate multiple LLM providers, implement semantic search capabilities, and build intelligent agent workflows
Architect complex, multi-step AI evaluation pipelines for asynchronous job execution and large-scale data processing
Design and implement database schemas with proper indexing, query optimisation, and data isolation strategies
Build and maintain scalable micro-services with async/await patterns and type-safe code
Develop data-intensive UIs with real-time updates, complex state management, and intuitive user experiences
Deploy and manage containerised applications on Kubernetes with comprehensive observability
Write thorough tests (frontend and backend) and maintain high code quality standards with automated tooling

Fulltime

Site Reliability Engineer

As a Site Reliability Engineer (SRE), you will be a key player in ensuring our p...

Location

Portugal , Lisboa

Salary:

Not provided

Tekever

Expiration Date

Until further notice

Requirements

Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field
3+ years of experience in Site Reliability Engineering, DevOps, or a related software/systems engineering role
Proficiency in one or more programming languages such as Python, Go, or Bash for automation and tooling
Deep understanding of Linux/Unix operating systems and networking fundamentals (TCP/IP, DNS, HTTP, load balancing)
Experience with cloud platforms such as AWS, Azure, or Google Cloud, with a focus on Google Cloud
Strong knowledge of CI/CD tools like Jenkins, GitLab CI, or CircleCI
Strong hands-on experience operating Kubernetes in production, including troubleshooting of networking, storage, scheduling, autoscaling, and stateful workloads
Experience with Infrastructure as Code (IaC) tools such as Terraform and Ansible
Understanding of version control systems (e.g., Git) and with CI/CD principles and tools (e.g., GitLab CI, Jenkins)
Knowledge of monitoring, logging and tracing tools (e.g., Prometheus, Grafana, ELK stack)

Job Responsibility

Design, build, and maintain highly available, scalable infrastructure for distributed and stateful workloads, supporting real-time data ingestion, AI inference pipelines, and hybrid cloud/edge deployment
Automate repetitive manual tasks, infrastructure provisioning, and operational workflows to reduce toil and improve system efficiency
Implement and manage robust monitoring, logging, and alerting solutions to proactively detect and address issues
Define and track Service Level Indicators (SLIs) and Service Level Objectives (SLOs)
Participate in an on-call rotation to respond to production incidents
Lead blameless post-mortem analyses for incidents in complex distributed systems, identifying root causes, systemic weaknesses, and implementing long-term preventative measures
Manage and provision cloud and on-premise infrastructure using IaC principles and tools like Terraform and Ansible
Conduct performance analysis, system tuning, and capacity planning to ensure our services meet performance and cost-efficiency goals
Develop, test, and maintain disaster recovery plans and business continuity strategies to ensure service resilience
Work closely with software development teams to consult on system design, platform choices, and reliability best practices for new features and services

What we offer

An excellent work environment and an opportunity to create a real impact in the world
A truly high-tech, state-of-the-art engineering company with flat structure and no politics
Working with the very latest technologies in Data & AI, including Edge AI, Swarming - both within our software platforms and within our embedded on-board systems
Flexible work arrangements
Professional development opportunities
Collaborative and inclusive work environment
Salary compatible with the level of proven experience

Fulltime

New

Finance ERP Consultant - Oracle EBS/Fusion

This role sits within the internal IT team, responsible for the comprehensive co...

Location

China , Shenzhen

Salary:

400000.00 - 700000.00 CNY / Year

Randstad

Expiration Date

June 14, 2026

Requirements

5+ years of Oracle ERP (EBS/Fusion) experience with a proven track record in financial system implementation
Familiarity with the Oracle EPM (Enterprise Performance Management) module
Strong software engineering skills, with proficiency in tools like Axure and Xmind
Excellent communication, a strong sense of service, and high learning agility
Bilingual proficiency in Chinese and English

Job Responsibility

Gathering and documenting business requirements
Proposing effective solutions and validating functional specifications
Acting as a primary liaison between technical teams and users
Developing and executing test plans, and conducting ERP system training

What we offer

Attractive welfare benefits and mentoring

New

Parts Supervisor

We are seeking a motivated Parts Supervisor to lead our parts team, ensuring eff...

Location

United Kingdom , Liverpool

Salary:

31015.00 - 34615.00 GBP / Year

360 Resourcing Solutions

Expiration Date

Until further notice

Requirements

Experience in an automotive parts environment
Previous supervisory or team leadership experience
Strong organisational and communication skills
Good understanding of stock control and parts processes
Customer-focused with a proactive, team-oriented approach

Job Responsibility

Lead and support the parts team, promoting strong teamwork and collaboration
Ensure smooth, efficient running of all parts operations and processes
Deliver high levels of customer service and meet customer requirements
Drive parts and accessories sales to achieve departmental targets
Manage stock control, ordering, and returns in line with business needs
Monitor transactions, outstanding payments, and supplier credits
Support stock checks and ensure accurate reporting and administration
Deputise for the Parts Manager when required

What we offer

Up to 27 days Holiday
Employee assistance programmes
Wellbeing courses
Medicash Healthcare
Pension scheme
Life assurance cover
Employee reward & recognition schemes
Annual children’s Christmas party & staff party
Staff discounts
£1,000 referral award

Fulltime

New

HR Advisor

This is an exciting opportunity to join Hyde a purpose-driven organisation where...

Location

United Kingdom , London Bridge

Salary:

47000.00 - 48000.00 GBP / Year

360 Resourcing Solutions

Expiration Date

Until further notice

Requirements

Proven experience in an HR Advisory role within a fast-paced unionised environment
Strong Employee Relations case management experience
Good working knowledge of UK employment law
Confident communicator with the ability to influence and challenge constructively
Strong judgement and ability to manage risk effectively

Job Responsibility

Manage a broad range of Employee Relations casework, providing clear, pragmatic advice and solutions
Support and guide managers through grievances, disciplinaries, capability, and absence management
Coach managers to build confidence and capability in handling people matters
Support organisational change activity, including restructures, TUPE transfers, redundancy, and redeployment
Build strong, credible relationships with stakeholders across the business
Identify trends and insights from ER activity to inform decision-making
Contribute to the continuous improvement of HR policies, processes, and ways of working

What we offer

Generous annual leave allowance plus bank holidays
Competitive pension scheme
Life assurance
Flexible benefits package tailored to you
Volunteering days to support causes that matter to you
Agile working approach (typically 40% office-based)
Wellbeing support and employee assistance programmes

Fulltime

Software Engineer, Load Balancing - Inference

OpenAI

Location:
United States , San Francisco

Category:
IT - Software Development

Contract Type:
Not provided

Salary:

Job Description:

Job Responsibility:

Requirements:

Nice to have:

Additional Information:

Job Posted:
February 21, 2026

Looking for more opportunities? Search for other job offers that match your skills and interests.

Similar Jobs for Software Engineer, Load Balancing - Inference