CrawlJobs Logo
AssemblyAI Logo AssemblyAI · IT - Software Development

Senior Software Engineer, Go - LLM Team

United States 180000.00 - 240000.00 USD / Year · Job Posted February 17, 2026
Apply Position
Job Link Share

Job Description

We're seeking an exceptional Senior Software Engineer to join our LLM team. This role is focused on building and maintaining our LLM gateway service—a unified API platform that connects customers to multiple LLM providers. You'll work on high-impact projects that directly solve customer problems, improve their AI and agentic workflows, and ensure reliable access to the best models for their use cases.

Job Responsibility

  • Partner closely with the customer success team to understand customer use cases, challenges, and integration needs
  • Translate customer pain points surfaced by the CS team into technical solutions and product improvements
  • Build features and tooling that directly address customer needs and improve their workflows
  • Provide technical guidance and expertise to the customer success team to help them support customers effectively
  • Own and deliver complete features and integrations within our LLM gateway service
  • Build and maintain integrations with multiple LLM providers and AI services (OpenAI, Anthropic, Google Vertex, AWS Bedrock etc.)
  • Write clean, maintainable, well-tested code following best practices
  • Design and implement scalable, fault-tolerant solutions with appropriate abstractions
  • Proactively identify and address technical debt, reliability issues, and code quality concerns
  • Participate in on-call rotation to ensure service reliability and rapid incident response
  • Conduct thorough code reviews focused on maintainability, testing, reliability, and architectural concerns
  • Ensure proper test coverage across unit, integration, and end-to-end testing levels
  • Improve code maintainability and extensibility through targeted refactoring
  • Contribute to runbooks, incident postmortems, customer-facing documentation, and knowledge sharing
  • Identify bottlenecks and propose solutions to improve team velocity and system reliability
  • Work closely with product, customer success, and engineering teams to deliver solutions
  • Collaborate with external API providers to troubleshoot integration issues and improve reliability
  • Translate ambiguous customer requirements into well-scoped technical work
  • Proactively communicate status, risks, and blockers
  • Mentor junior engineers on coding standards, reliability practices, and customer-centric thinking
  • Participate in architectural discussions and contribute to technical decisions

Requirements

  • 5+ years of professional software engineering experience
  • Strong proficiency in Go with demonstrated ability to write production-quality code
  • Hands-on experience with LLM APIs (OpenAI, Anthropic, etc.) and understanding of prompt engineering concepts
  • Experience building or working with API gateways, proxies, and rate limiting systems
  • Deep experience with prompt engineering, chain-of-thought, few-shot learning, and other LLM optimization techniques
  • Experience building applications with llms or ai agents with cascading architectures
  • Solid understanding of software engineering fundamentals: Data structures and algorithms, System design and architectural patterns, Testing strategies (unit, integration, end-to-end), Code review practices and technical collaboration
  • Experience with: RESTful APIs and distributed systems concepts, Building or integrating with third-party APIs, SQL and database design, Version control (Git) and CI/CD pipelines, Containerization (Docker) and cloud infrastructure
  • Strong communication skills and ability to collaborate effectively with cross-functional teams
  • Track record of delivering high-quality, reliable software in a fast-paced, team environment
  • Ability to thrive in a startup environment with changing priorities and rapid iteration

Nice to have

  • Experience with AWS services (ECS, Lambda, ElastiCache/Redis, RDS, S3, CloudWatch, API Gateway)
  • Experience with monitoring and observability tools (Datadog, CloudWatch)
  • Experience with request routing, load balancing, and traffic management
  • Experience with circuit breakers, retry logic, fallback strategies, and fault tolerance patterns
  • Understanding of API design, versioning, and backward compatibility
  • Experience working closely with customer success or support teams

What we offer

  • Competitive equity grants
  • 100% employer-paid benefits
  • Flexibility of being fully remote
  • 401k match up to 4% for US-based full time team members
AssemblyAI - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Senior Software Engineer, Go - LLM Team

8 matching positions

Senior Lead Software Engineer (Golang + EKS, Kubernetes, LLM's + Agentic flows + control/data planes)

Do you love building and pioneering in the technology space? Do you enjoy solvin...
Location
Location
United States , San Francisco, California; McLean, Virginia; New York, New York; San Jose, California
Salary
Salary:
229900.00 - 286200.00 USD / Year
capitalone.com Logo
Capital One
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree
  • At least 6 years of experience in software engineering
  • At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)
Job Responsibility
Job Responsibility
  • Lead a portfolio of diverse technology projects and a team of developers with deep experience in distributed microservices, and full stack systems to create solutions that help meet regulatory needs for the company
  • Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, mentoring other members of the engineering community, and from time to time, be asked to code or evaluate code
  • Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment
  • Utilize programming languages like JavaScript, Java, HTML/CSS, TypeScript, SQL, Python, and Go, Open Source RDBMS and NoSQL databases, Container Orchestration services including Docker and Kubernetes, and a variety of AWS tools and services
What we offer
What we offer
  • performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
  • comprehensive, competitive, and inclusive set of health, financial and other benefits
  • Fulltime
Read More
Arrow Right
New

Senior Software Engineer

CoreAI is at the forefront of Microsoft’s mission to redefine how software is bu...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or master’s degree in computer science or a related field
  • 3+ years designing, developing, and shipping high quality software
  • 2+ years of experience with distributed systems and cloud-based infrastructure
  • 1+ year of experience with DevOps practices (CI/CD, automated testing, deployment, etc.)
  • Passionate and self-motivated
  • Strong ability in self-learning, entering new domain, managing through uncertainty in an innovative team environment
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
Job Responsibility
Job Responsibility
  • Architect, design, and develop core AI Infrastructure services developed in Go, Rust, Python, C++, and C# deployed on large-scale Kubernetes clusters to support state-of-the-art LLM training and inference
  • Design, build, and manage large-scale GPU clusters to support LLM training, and inference workloads
  • Enhance systems and applications to deliver high stability, low latency, strong security, and maintainability in large-scale complex training environments in Azure and in partner clouds
  • Provide operational support, technical leadership, and vision while contributing to the deployment, monitoring, and continuous improvement of engineering systems and practices
  • Support development and troubleshooting from the frontline, resolving complex issues impacting large-scale services
  • Fulltime
Read More
Arrow Right

Senior Software Engineer

CoreAI is at the forefront of Microsoft’s mission to redefine how software is bu...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or master’s degree in computer science or a related field
  • 3+ years designing, developing, and shipping high quality software
  • 2+ years of experience with distributed systems and cloud-based infrastructure
  • 1+ year of experience with DevOps practices (CI/CD, automated testing, deployment, etc.)
  • Passionate and self-motivated
  • Strong ability in self-learning, entering new domain, managing through uncertainty in an innovative team environment
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
Job Responsibility
Job Responsibility
  • Architect, design, and develop core AI Infrastructure services developed in Go, Rust, Python, C++, and C# deployed on large-scale Kubernetes clusters to support state-of-the-art LLM training and inference
  • Design, build, and manage large-scale GPU clusters to support LLM training, and inference workloads
  • Enhance systems and applications to deliver high stability, low latency, strong security, and maintainability in large-scale complex training environments in Azure and in partner clouds
  • Provide operational support, technical leadership, and vision while contributing to the deployment, monitoring, and continuous improvement of engineering systems and practices
  • Support development and troubleshooting from the frontline, resolving complex issues impacting large-scale services
  • Fulltime
Read More
Arrow Right

Senior Software Engineer, Data Infrastructure & AI

Fullstory Anywhere is one of Fullstory's three primary product verticals, and it...
Location
Location
United States , Atlanta
Salary
Salary:
160000.00 - 170000.00 USD / Year
fullstory.com Logo
Fullstory
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Significant experience building and operating high-throughput data pipelines (batch and/or streaming) in a major cloud platform, including work with cloud data warehouses like BigQuery, Snowflake, or Databricks.
  • Proficiency in Go, Python, Java or a similar language.
  • Hands-on experience with data transformation tooling such as dbt, with a strong understanding of data modeling and pipeline observability.
  • Familiarity with LLM integration patterns and evaluation approaches (e.g., LangSmith, Vertex AI, or comparable frameworks), or demonstrated ability to ramp quickly in applied AI.
  • A track record of owning major system areas end-to-end: driving architectural decisions, maintaining production health, and improving reliability over time.
Job Responsibility
Job Responsibility
  • Maintain, extend, and scale Go microservices that transform and deliver Fullstory session data into customer warehouses and power the team's MCP server that enables AI agent integrations.
  • Develop and maintain dbt models and pipeline orchestration to ensure timely, fault-tolerant data migrations across hundreds of customer destinations.
  • Define evaluation frameworks for LLM outputs using tools like Langsmith and Vertex AI, ensuring AI-powered customer agents produce accurate, useful results.
  • Investigate and resolve production incidents across the data pipeline, implementing systemic fixes that prevent entire classes of failure from recurring.
  • Write technical design documents that drive consensus on architectural changes, proactively surfacing scaling bottlenecks, edge cases, and cross-team dependencies.
  • Demonstrate sound technical judgment by de-risking work through spikes, taking on tech debt deliberately, and knowing when to escalate versus dig in.
What we offer
What we offer
  • Flexibility and Connection
  • flexible PTO policy
  • annual company-wide closure
  • Benefits
  • paid parental leave
  • Bereavement leave, including miscarriage/pregnancy loss
  • Learning opportunities
  • annual learning subsidy
  • Productivity support
  • monthly productivity stipend
  • Fulltime
Read More
Arrow Right

Senior Software Engineer - Application Security

Uber is seeking a Senior Software Engineer to join our Application Security team...
Location
Location
United States , Seattle
Salary
Salary:
202000.00 - 224000.00 USD / Year
uber.com Logo
Uber
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Understanding of Agents, MCP servers, and LLMs
  • BS/BE degree in Computer Science or related fields
  • 5+ years of experience in software engineering
  • Experience building applications from end to end
  • Hands-on experience with Go, Java, C, or Python (Uber heavily utilizes Go)
  • Deep knowledge of datastore technologies, including RDBMS and NoSQL systems
  • Experience implementing REST or gRPC APIs
  • Service design and architecture experience
  • Implementation of secure system design, including authentication, authorization, and encryption
Job Responsibility
Job Responsibility
  • Design end-to-end features and autonomous systems for Agent & MCP Security, Code Scanning, and Vulnerability Remediation applications
  • Develop security automation capable of identifying vulnerabilities and performing automated exploit validation across Uber's microrepository landscape
  • Collaborate and consult with multiple engineering teams and stakeholders from Privacy, Security, Compliance, Infrastructure, and Product for integration into our security platforms
  • Design and develop new systems, such as automated security scoring and repository-to-service mapping, to empower fast, data-driven security decisions
  • Build distributed backend systems, including workflows, that serve real-time analytics and security features at Uber scale
  • Mentor junior engineers and lead technical design reviews for high-impact security projects
What we offer
What we offer
  • Bonus program
  • Equity award
  • 401(k) plan
  • Various benefits
  • Fulltime
Read More
Arrow Right

Senior Software Engineer

The R&D of Search Ads aims to build an online advertising ecosystem of users, ad...
Location
Location
China , Beijing
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Python, CUDA, or ROCm OR equivalent experience
  • 3+ years' practical experience working on applications that use GPUs, experience in optimizing their performance
  • Practical Experience writing new GPU kernels, going beyond experience of GPU workloads with existing library kernels
  • Cross-team collaboration skills and the desire to collaborate in a team of researchers and developers
  • Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C/C++, CUDA, or ROCm OR Master's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C/C++, CUDA, or ROCm OR equivalent experience
  • Experience in low-level performance analysis and optimization, including proficiency using GPU profiling tools such as NVIDIA Visual Profiler, and NVIDIA Nsight Compute
  • Technical background and solid foundation in software engineering principles and architecture design
  • Familiar with inference optimization, experience in developing popular inference framework such as TensorRT-LLM, SGLang, vLLM
  • Exposure to Deep Neural Network inference and experience in one or more deep learning frameworks such as PyTorch, Tensorflow, or ONNX Runtime
Job Responsibility
Job Responsibility
  • Design, develop, and maintain high-performance software in C/C++ and Python, including GPU programming with CUDA, ROCm, or Triton
  • Optimize model inference and training pipelines for speed, throughput, memory efficiency, and cost across GPU platforms
  • Collaborate with platform teams to integrate and tune solutions on emerging accelerator stacks and rapidly evolving toolchains
  • Profile workloads end-to-end, identify bottlenecks, and implement kernel-level and system-level performance improvements
  • Partner with internal and external stakeholders to translate requirements into scalable performance features and optimizations for state-of-the-art models
  • Validate performance, stability, and correctness through benchmarking, automated testing, and production readiness reviews
  • Fulltime
Read More
Arrow Right

Senior Software Engineer - Build AI Tools

This role sits within the newly formed GenAI Security team, which is responsible...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Highly motivated self-starter with excellent interpersonal and problem-solving skills
  • Bachelor’s degree or equivalent work experience
  • Good oral and written communication skills
  • Significant relevant industry work experience
  • Experience of the full lifecycle of design, implementation and running of enterprise software solutions involving cross functional team collaboration
  • Expertise in a major programming language such as Python and/or Go, and associated tooling (Git, Maven, IDEs, Jenkins, Bitbucket etc)
  • Expertise in designing and implementing secure APIs and libraries
  • Experience with Cyber engineering and Operations, which could include DevSecOps or MLSecOps
  • Experience contributing to the architecture and design (architecture, design patterns, reliability, scaling) of new and current systems
  • Experience in containerized deployment (Kubernetes, OpenShift etc)
Job Responsibility
Job Responsibility
  • Designing, developing, optimizing, and enhancing a GenAI prompt security platform to protect firm AI/LLM-based applications from adversarial attacks and prompt injections
  • Building and automating a security testing framework to validate protection mechanisms for various LLM use cases
  • Owning solutions that are expected to operate and perform at scale across the organisation
  • Collaboration with multiple stakeholders and partners across Engineering and Operations as well as partner teams within the wider Citi organisation, across different time zones
What we offer
What we offer
  • 27 days annual leave (plus bank holidays)
  • A discretional annual performance related bonus
  • Private Medical Care & Life Insurance
  • Employee Assistance Program
  • Pension Plan
  • Paid Parental Leave
  • Special discounts for employees, family, and friends
  • Access to an array of learning and development resources
  • Fulltime
Read More
Arrow Right

Senior Software Engineer- AI

Are you looking for an opportunity to work with the latest Azure offerings and p...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in Software Development
  • Strong programming expertise in one or more languages such as Python, Go, Java, or C#, with experience designing production-grade services and APIs
  • Experience building AI-powered applications, including integrating LLMs, implementing agent or Copilot workflows, and orchestrating multi-step AI interactions
  • Hands-on experience with LLM application frameworks and orchestration tools such as Semantic Kernel, LangChain, or similar agent frameworks
  • Familiarity with retrieval-augmented generation (RAG) architectures, vector databases, embeddings, and semantic search systems
  • Experience evaluating and improving model performance through prompt design, evaluation frameworks, fine-tuning, or feedback loops
  • Solid understanding of distributed systems concepts including scalability, reliability, observability, caching, and asynchronous processing
  • Experience deploying and operating AI workloads in cloud environments (preferably Azure), including containerized services and GPU-enabled infrastructure
  • Understanding of Responsible AI practices, including model governance, safety, privacy, and evaluation of AI behaviour in production systems
  • Ability to work across product, research, and engineering teams to translate product scenarios into scalable AI system architectures
Job Responsibility
Job Responsibility
  • Design, build, and operate scalable AI systems that power intelligent product experiences, including Copilot and agent-driven workflows
  • Architect and implement backend services that support multi-step AI interactions, including orchestration pipelines, context management, memory/state persistence, and tool execution
  • Integrate large language models (LLMs), APIs, and internal services to enable context-aware, human-in-the-loop experiences across customer scenarios
  • Build and maintain data and inference pipelines that support model training, fine-tuning, evaluation, and real-time inference across diverse data sources
  • Evaluate, benchmark, and tune AI/ML models (LLMs and traditional models) to meet product requirements for accuracy, latency, reliability, and safety
  • Implement robust retrieval, grounding, and knowledge integration mechanisms (e.g., RAG systems, semantic indexing, vector search) to power intelligent applications
  • Collaborate with product managers, software engineers, and researchers to translate product vision into production-ready AI capabilities and measurable outcomes
  • Ensure reliability, observability, and governance of AI systems, including monitoring model performance, data quality, and responsible AI practices
  • Build reusable platforms, APIs, and tools that enable teams to rapidly develop AI-powered features and self-service intelligent applications
  • Fulltime
Read More
Arrow Right