CrawlJobs Logo
Apollo.io Logo Apollo.io · IT - Software Development

Principal AI Engineer

United States 200000.00 - 300000.00 USD / Year · Job Posted December 06, 2025
Apply Position
Job Link Share

Job Description

We are looking for a Principal AI Engineer to lead the design and deployment of cutting-edge agentic systems, AI assistants, and LLM-powered features. This role is pivotal to Apollo’s AI roadmap, owning the architecture and productionization of intelligent systems that power the workflows of thousands of GTM teams globally. You’ll be a technical thought leader on the AI Engineering team and work alongside product, backend, and platform leaders to drive forward Apollo’s competitive edge in applied AI.

Job Responsibility

  • Architect and lead the development of multi-agent systems capable of long-horizon planning, reasoning, and API orchestration
  • build reusable agentic components that integrate deeply into sales and marketing processes
  • own and evolve our in-house platform for scalable, low-latency, and cost-efficient LLM and agent deployments
  • lead design of interfaces powered by natural language understanding and retrieval-augmented generation (RAG)
  • build embedding-based, intent-aware search and personalization systems tuned to business user needs
  • drive innovation in personalized outreach generation using context-aware generation pipelines
  • tune inference pipelines, caching layers, and model selection logic for high-scale, cost-aware performance
  • define and drive robust offline and online testing methodologies (A/B, sandboxing, human evals) across agents and LLM flows
  • architect human-in-the-loop systems and telemetry to improve accuracy, UX, and explainability over time

Requirements

  • 10+ years of software engineering experience
  • at least 3 years in applied LLM or agentic AI systems (2023–present)
  • proven success in deploying LLM-powered products used by real users at scale
  • deep backend & systems engineering expertise with Python, distributed systems, and scalable APIs
  • familiarity with LangChain, LlamaIndex, or similar orchestration frameworks
  • experience with RAG pipelines, vector DBs, embedding models, and semantic search tuning
  • experience managing performance across cloud providers (e.g., AWS Bedrock, OpenAI, Anthropic, etc.)
  • demonstrated experience building multi-step agents, planning workflows, chaining reasoning steps, and integrating APIs with agent memory/state
  • comfort with advanced prompting strategies, few-shot and chain-of-thought reasoning, and embedding retrieval setups
  • strong understanding of AI system evaluation, human ratings, A/B experimentation, and feedback loop pipelines
  • experience designing safety-aware, reliable LLM systems in production environments
  • experience owning logging, monitoring, and observability for live AI systems
  • principal-level ownership
  • AI-native mentality
  • applied focus

What we offer

  • equity
  • company bonus or sales commissions/bonuses
  • 401(k) plan
  • at least 10 paid holidays per year
  • flex PTO
  • parental leave
  • employee assistance program
  • wellbeing benefits
  • global travel coverage
  • life/AD&D/STD/LTD insurance
  • FSA/HSA
  • medical benefits
  • dental benefits
  • vision benefits
Apollo.io - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Principal AI Engineer

8 matching positions

New

Principal AI Engineer

We are seeking a highly accomplished Principal AI Engineer to define and drive t...
Location
Location
Ireland , Dublin 18
Salary
Salary:
Not provided
mastercard.com Logo
Mastercard
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Demonstrated experience designing and building AI/ML systems in production at scale, ideally across multiple problem domains
  • Expert-level proficiency in Python and deep experience with modern AI frameworks such as PyTorch and TensorFlow
  • Strong experience with cloud-native architectures and AI infrastructure on platforms such as AWS, Azure, or GCP
  • Deep understanding of machine learning, deep learning, NLP, generative AI, and transformer-based architectures (e.g., BERT, GPT-style models, ViTs)
  • Proven expertise in MLOps, including model versioning, deployment strategies, monitoring, evaluation, and lifecycle management
  • Strong systems-thinking mindset, with experience designing resilient, scalable, and cost-efficient AI services
  • Experience working with large-scale data architectures, streaming and batch processing, and model inference optimization
  • Excellent communication skills with the ability to explain complex technical concepts to both technical and non-technical stakeholders
  • Track record of technical mentorship and influence without relying on formal line management
  • Comfortable operating in high-ambiguity environments and making sound technical judgments with incomplete information
Job Responsibility
Job Responsibility
  • Define and drive the technical direction of our AI platforms and solutions
  • Architect, build, and scale production-grade AI systems that deliver durable business impact
  • Lead through deep hands-on expertise, influence technical strategy across teams, and raise the engineering bar for AI development across the organization
  • Design, implement, and operate advanced AI systems that support critical business and client needs in a scalable, secure, and reliable manner
  • Partner closely with product, engineering, and data leaders to translate business intent into robust AI architectures and platforms
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

We are seeking a Senior/Principal AI Engineer to join our Stellar AdTech Busines...
Location
Location
Salary
Salary:
Not provided
sigma.software Logo
Sigma Software Group
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Designing and building AI-powered systems in production / expert
  • Agentic frameworks, multi-agent collaboration, orchestrator/worker models / strong
  • RAG pipelines, relevance tuning / strong
  • Python and/or TypeScript, API design, microservices, cloud-native architectures / good
  • Presales consulting, translating business problems into AI solutions / good
What we offer
What we offer
  • Diversity of Domains & Businesses
  • Variety of technology
  • Health & Legal support
  • Active professional community
  • Continuous education and growing
  • Flexible schedule
  • Remote work
  • Outstanding offices (if you choose it)
  • Sports and community activities
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

Location
Location
Canada , Mississauga
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience: Extensive experience in designing and building AI/ML solutions, with a significant focus on generative AI and Large Language Models (LLMs)
  • Gen AI Expertise: Deep understanding of modern AI architectures and techniques, including Retrieval-Augmented Generation (RAG), fine-tuning, function calling, and AI agentic workflows
  • Programming Proficiency: Expert-level skills in Python and extensive experience with core AI/ML libraries such as PyTorch, TensorFlow
  • System Design: Proven ability to architect and develop large-scale, distributed, multi-tier applications. Strong knowledge of microservices, API design, and system integration
  • MLOps: Solid understanding of MLOps principles and experience with tools for model versioning, deployment, monitoring, and lifecycle management
  • Leadership: Demonstrated experience serving as a technical lead, architect, or principal engineer, with a track record of mentoring team members and driving projects to completion
Job Responsibility
Job Responsibility
  • Architectural Leadership: Design and architect end-to-end generative AI solutions, from proof-of-concept to production, ensuring scalability, performance, and reliability
  • Technical Strategy: Develop and maintain a comprehensive strategic roadmap for generative AI adoption, evaluating new models, techniques, and platforms to keep our capabilities at the forefront of the industry
  • Solution Development: Lead the hands-on development of complex AI systems, including Retrieval-Augmented Generation (RAG) pipelines, autonomous AI agents, fine-tuning workflows, and custom model integrations
  • Best Practices & Standards: Establish and govern best practices for the full AI development lifecycle, including prompt engineering, model evaluation, MLOps, and data management
  • Cross-Functional Partnership: Collaborate closely with multiple management teams and business units to identify high-impact use cases and ensure the successful integration of AI solutions to meet business goals
  • Mentorship & Guidance: Serve as a senior advisor and coach to other engineers and analysts, fostering a culture of innovation and technical excellence. Allocate work and provide technical direction to the team
  • Risk & Compliance: Appropriately assess risk when business decisions are made, demonstrating consideration for the firm's reputation and safeguarding its clients and assets. Drive compliance with all applicable laws, rules, and regulations, particularly those related to AI ethics, data privacy, and model bias
  • Innovation and Research: Stay abreast of the latest advancements in generative AI research, and translate state-of-the-art developments into practical, innovative solutions
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

The Principal AI Engineer is part of our AI Centre of Excellence team at the for...
Location
Location
United States , Boston, MA; Austin, TX
Salary
Salary:
236500.00 - 320000.00 USD / Year
rapid7.com Logo
Rapid7
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Significant hands-on experience in applied AI, able to independently design and build AI models and systems
  • Strong Python skills and practical expertise with: scikit-learn for conventional ML and data science
  • PyTorch, TensorFlow, or Keras for deep learning
  • AWS Bedrock, SageMaker, HuggingFace, or Transformers for LLM-based solutions
  • CrewAI (or similar frameworks) for agentic AI projects
  • Pandas/NumPy for data processing and transformation
  • Intuition for selecting the right models and approaches for different use cases
  • Cybersecurity expertise is a plus, but experience from other domains (e.g., finance, healthcare, marketplaces, insurance, legal, social media) is also highly valuable
  • Positive, can-do, solution-oriented mindset with persistence in tackling complex challenges
  • Bias for action and delivery, balancing speed with care
Job Responsibility
Job Responsibility
  • Develop and ship AI systems with a focus on delivering customer value
  • Tackle a diverse range of complex problems, often reframing existing solutions with new perspectives
  • Take ownership of multiple projects as the principal go-to expert
  • Act as a first-class stakeholder throughout the AI R&D process
  • Build strong cross-functional relationships with engineering, UX, product management, and customer research teams
  • Mentor colleagues with less experience, acting as a trusted technical resource
Read More
Arrow Right

Principal AI Engineer

As a Principal AI Engineer on the AI Foundations team, you are an established su...
Location
Location
Singapore , Singapore
Salary
Salary:
Not provided
mastercard.com Logo
Mastercard
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Engineering, Data Science, Applied Mathematics, or related technical field
  • advanced degree preferred
  • Strong foundation in software engineering, distributed systems, and applied machine learning relevant to production AI systems
  • Demonstrated understanding of responsible AI, model/system risk, privacy/security considerations, and governance requirements for enterprise deployments
  • Demonstrated, sustained ownership of production AI/ML systems, including design, build, deployment, and ongoing lifecycle operations
  • Real-world experience shipping complex agentic systems into production, including multi-agent coordination and multi-tool integration with safe action policies
  • Hands-on experience building production pipelines for evaluation, monitoring, versioning, and continuous improvement (including retraining or policy/guardrail updates)
  • Proven ability to define and operationalize observability and reliability practices for agentic systems (traceability, telemetry, SLOs, incident management)
  • Track record of influencing architecture and standards across multiple teams or programs, and mentoring engineers to raise overall engineering rigor
Job Responsibility
Job Responsibility
  • Serve as an established subject matter expert in AI Engineering, influencing stakeholders and shaping technical direction across multiple initiatives
  • Architect, design, develop, and maintain advanced AI/ML systems, with emphasis on complex agentic solutions (multi-agent orchestration, tool/function-calling, memory, reflection/self-correction, and autonomy policies)
  • Lead production implementation of agentic AI systems, including scalable training and evaluation pipelines, deployment frameworks, and runtime orchestration patterns
  • Define and implement safe tool-use patterns: structured outputs, robust error handling, permissioning and auditability, human-in-the-loop (HITL) approval steps for sensitive actions, and guardrail enforcement
  • Establish end-to-end AgentOps/LLMOps practices for agentic systems: release pipelines for prompts/tools/policies, canary strategies, safe rollback mechanisms, and continuous regression/safety evaluations as release gates
  • Build and optimize data ingestion, preprocessing, feature/embedding engineering, and retrieval/memory workflows to improve grounding quality and reduce failure modes
  • Own production observability for agentic systems: trace capture, cost/token telemetry, latency and reliability SLOs, and incident response practices for agent failures
  • Implement drift detection and performance decay monitoring (data drift, concept drift), and automate model/agent retraining, policy updates, and redeployment to maintain output quality over time
  • Drive measurable improvements in system effectiveness, safety, and efficiency by defining success metrics (task success, intervention rate, policy violations, cost and latency per task) and continuously improving evaluation coverage
  • Mentor and grow senior and junior engineers through design reviews, code reviews, hands-on coaching, and the creation of reusable patterns, playbooks, and standards for agentic delivery
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

Lead the evolution of T-Mobile’s Agentic Next Best Action (ANBA) capabilities—po...
Location
Location
United States , Overland Park; Frisco; Bellevue
Salary
Salary:
150700.00 - 271900.00 USD / Year
https://www.t-mobile.com Logo
T-Mobile
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Deep expertise in LLM fine-tuning and prompt engineering (e.g., OpenAI APIs, Hugging Face, Anthropic Claude, Google Gemini)
  • Strong experience with AI orchestration tools (e.g., LangChain, LlamaIndex, vector databases for retrieval augmentation)
  • Hands-on knowledge of function calling and API-based reasoning models (e.g., using structured outputs to drive automated workflows)
  • Proficiency in Python and AI development frameworks for building scalable AI applications
  • Understanding of multi-agent architectures and best practices in agentic AI design
  • Experience with real-world AI evaluation techniques, including golden sets, synthetic data generation, and interactive testing
  • Ability to collaborate across teams, working with engineers, product managers, and conversational designers to refine AI-driven solutions
  • Bachelor's degree in Computer Science, Artificial Intelligence, or equivalent
  • At least 18 years of age
  • Legally authorized to work in the United States
Job Responsibility
Job Responsibility
  • Architects agentic AI systems that accomplish sophisticated tasks by invoking AI models as well as internal and third-party tools using APIs, ensuring flawless data flow in production environments
  • Optimizes performance of agentic AI systems through prompt engineering, fine-tuning and reinforcement learning using T-Mobile’s customer interaction data
  • Develops and maintains supporting software components and scripts to enable AI model deployment, testing, evaluation and monitoring
  • Implement retrieval-augmented generation (RAG) techniques to ensure AI responses are contextually accurate and grounded in real-time data
  • Stay at the forefront of LLM advancements, incorporating the latest techniques in prompt design, few-shot learning, Tool integration protocols like MCP and AI orchestration frameworks like Agent SDK
  • Collaborates with backend engineers, business experts and conversation designers ensuring AI-driven enhancements are optimally integrated into production environments
  • Defining success metrics that aligns with business requirements and continuously evaluate and improve model quality based on those metrics
  • Participates in other duties or projects as assigned by business management as needed
What we offer
What we offer
  • Annual stock grant
  • Employee stock purchase plan
  • 401(k)
  • Access to free, year-round money coaches
  • Medical, dental and vision insurance
  • Flexible spending account
  • Paid time off and up to 12 paid holidays
  • Paid parental and family leave
  • Family building benefits
  • Back-up care
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

The Principal AI Engineer will serve as a technical cornerstone of VideoAmp's AI...
Location
Location
United States , Los Angeles; Boulder; New York; St. Petersburg
Salary
Salary:
175000.00 - 200000.00 USD / Year
videoamp.com Logo
VideoAmp
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Engineering, or a related field preferred
  • equivalent practical experience considered
  • 8+ years of software engineering experience
  • 3+ years in AI/ML infrastructure, LLM platform engineering, or agentic systems
  • Deep hands-on experience with LLM APIs (Anthropic, OpenAI, or equivalent)
  • Familiarity with prompt engineering, tool use / function calling, and multi-step agent orchestration
  • Strong background in resource-based API design
  • Experience building or consuming developer-facing platform APIs at scale
  • Experience with MCP or equivalent tool-layer abstractions for exposing platform capabilities to AI agents
  • Proficiency in Golang, Python, and SQL
Job Responsibility
Job Responsibility
  • Design, build, and operate VideoAmp's AI infrastructure and its universal tool layers
  • Lead the development of scenario evaluation frameworks
  • Architect and implement efficient tool discovery systems
  • Partner with internal engineering teams to negotiate and promote API-first designs
  • Own full SDLC of new Agent APIs from design through production, testing, releases, and enhancement
  • Facilitate weekly AI office hours
  • Contribute to multi-provider LLM abstraction layers
  • Author, review, and drive clear, technical requirements documentation for new solutions
What we offer
What we offer
  • Equity participation included
  • Discretionary & flexible PTO + Spring, Summer & Winter company breaks
  • Inclusive and comprehensive medical, dental & vision
  • 401(k) with matching
  • HSA & FSA
  • Paid Maternity & Parental Leave for all family additions
  • Cell phone & wifi reimbursement
  • Commuter benefits
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

Join OutSystems, a global leader in software innovation, and contribute to a tru...
Location
Location
United States , Menlo Park, California
Salary
Salary:
229500.00 - 270000.00 USD / Year
outsystems.com Logo
OutSystems
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • M.Sc. or Ph.D. in Computer Science, Artificial Intelligence, Machine Learning, or a related quantitative field (e.g., Electrical Engineering, Physics, Statistics)
  • 10+ years of hands-on professional experience in applied AI/ML roles with a proven record of delivering impactful solutions
  • Demonstrable experience deploying and managing Large Language Models (LLMs) in production environments
  • Proven ability to design, build, and ship complex, data-intensive, and scalable systems in a real-world setting
  • Strong problem-solving skills and programming proficiency, including the ability to architect and implement end-to-end solutions (from data processing and model development to deployment and monitoring)
  • Expertise in Python is required
  • experience with C# is valued
  • Exhibit creativity, ambition, curiosity, resourcefulness, and innovation
Job Responsibility
Job Responsibility
  • Design and architect ML/AI systems, collaborating cross-functionally with multiple stakeholders
  • Contribute to the architecture of OutSystems' AI platform and services
  • Partner with Research Scientists to design, prototype, and implement state-of-the-art AI models and algorithms, ensuring production optimization for scale and performance
  • Develop and implement highly available, fault-tolerant systems for serving sophisticated AI models, with a focus on reliability, efficiency, and MLOps best practices
  • Define data collection strategies and feedback loops to enable continuous learning and improvement of our AI capabilities and product offerings
  • Guide, mentor, and elevate the technical capabilities of other AI Engineers, championing best practices in design, implementation, and AI development
  • Maintain deep knowledge of the state-of-the-art in AI/ML models, agents, and developer tools, driving the adoption of relevant advancements across the team and company
  • Collaborate cross-functionally to identify opportunities and enhance our AI products and data infrastructure
What we offer
What we offer
  • A company that is always growing, changing, and innovating
  • Real career opportunities
  • Work colleagues that are as smart, hard-working, and driven as you
  • Disrupting the status quo is in our DNA
  • We ask “why” a lot
  • Inclusive culture of diversity
  • Equal opportunity employer
  • Fulltime
Read More
Arrow Right