CrawlJobs Logo

Principal Engineer, AI Model Lifecycle

United States, San Francisco 260000.00 - 326000.00 USD / Year · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

The Principal Software Engineer for the Model LifeCycle team will play a crucial role in building a comprehensive managed platform for the entire application development lifecycle, with a specific focus on leveraging Machine Learning models, including Large Language Models (LLMs). This role offers significant 0 → 1 ownership — you'll be designing and building core systems from first principles.

Job Responsibility

  • Manage fine-tuning systems for large foundation models (SFT, PEFT, LoRA, adapters), including multi-node orchestration, checkpointing, failure recovery, and cost-efficient scaling
  • Implement and maintain end-to-end training pipelines for Large Language Models
  • Distillation and reinforcement learning pipelines (e.g., preference optimization, policy optimization, reward modeling)
  • Agent execution infrastructure
  • Dataset, model, and experiment management: versioning, lineage, evaluation, and reproducible fine-tuning at scale
  • Work closely with product, business, and platform teams to shape the core abstractions and APIs of the system
  • Influence long-term architectural decisions around training runtimes, scheduling, storage, and model lifecycle management
  • Contribute to and engage with the open-source LLM ecosystem

Requirements

  • Advanced degree in Computer Science, Engineering, or a related field
  • 10-15+ years of industry experience driving impactful projects in the AI Space
  • Proven track record of delivering early-stage projects under tight deadlines
  • Expertise in using cloud-based services, such as, elastic compute, object storage, virtual private networks, managed database, etc.
  • Experience in Generative AI (Large Language Models, Multimodal)
  • Deep experience with AI infrastructure, including training, inference

Nice to have

  • Proficiency in Golang or Python for large-scale, production-level services
  • Contributions to open-source AI projects such as vLLM or similar frameworks
  • Performance optimizations on GPU systems and inference frameworks
  • Experience working with PyTorch
  • Experience with training and fine-tuning LLMs

What we offer

  • Restricted Stock Units
  • Health insurance package options that include HDHP and PPO, vision, and dental for you and your dependents
  • Employer contributions to HSA accounts
  • Paid Parental Leave
  • Paid life insurance, short-term and long-term disability
  • Teladoc
  • 401(k) with a 100% match up to 4% of salary
  • Generous paid time off and holiday schedule
  • Cell phone reimbursement
  • Tuition reimbursement
  • Subscription to the Calm app
  • MetLife Legal
  • Company paid commuter benefit
  • $300/month

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Principal Engineer, AI Model Lifecycle

8 matching positions

Senior Software Engineer, Managed AI - AI model LifeCycle

The Senior Software Engineer for the Model LifeCycle team will contribute to bui...
Location
Location
United States , San Francisco
Salary
Salary:
172425.00 - 209000.00 USD / Year
crusoe.ai Logo
Crusoe
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Engineering, or a related field
  • Experience delivering production-ready features
  • Familiarity with essential cloud-based services (e.g., compute, storage, networking)
  • Familiarity with Generative AI (Large Language Models, Multimodal)
  • Experience with AI infrastructure components (training, inference)
  • 4-5+ years of industry experience with demonstrated history of consistent success leading a varied portfolio of initiatives across your function
Job Responsibility
Job Responsibility
  • Implement and maintain systems for fine-tuning large foundation models (SFT, PEFT, LoRA, adapters), including multi-node orchestration, checkpointing, failure recovery, and cost-efficient scaling
  • Implement and maintain end-to-end training pipelines for Large Language Models
  • Implement components for distillation and reinforcement learning pipelines (e.g., preference optimization, policy optimization, reward modeling)
  • Develop and maintain core agent execution infrastructure
  • Implement features for dataset, model, and experiment management, focusing on versioning, lineage, evaluation, and reproducible fine-tuning
  • Work closely with Senior Engineers and Principal Engineers, as well as product and platform teams, to implement system abstractions and APIs
  • Contribute to technical discussions on training runtimes, scheduling, storage, and model lifecycle management
  • Engage with the open-source LLM ecosystem
What we offer
What we offer
  • Restricted Stock Units
  • Health insurance package options that include HDHP and PPO, vision, and dental for you and your dependents
  • Employer contributions to HSA accounts
  • Paid Parental Leave
  • Paid life insurance, short-term and long-term disability
  • Teladoc
  • 401(k) with a 100% match up to 4% of salary
  • Generous paid time off and holiday schedule
  • Cell phone reimbursement
  • Tuition reimbursement
  • Fulltime
Read More
Arrow Right

Senior Software Engineer and Principal Software Engineer - Power Point AI Team

The PowerPoint team is embarking on an exciting new chapter - evolving a product...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
  • Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • 8+ years of experience in backend service engineering, including work on high-scale infrastructures
  • Proficiency in one or more systems programming languages such as C#, C++
  • 1+ years of experience in software engineering, designing and developing systems (and APIs) that deploy and integrate with AI models
  • 2+ years of experience working with rich telemetry, making data driven decisions, and carrying out rapid experimentation
  • 2+ years of experience building software for scale, performance, and reliability
  • Academic or industry experience with building, finetuning, deploying or building eval-driven systems utilizing the models (any category)
Job Responsibility
Job Responsibility
  • Lead design and delivery of complex, scalable AI features ensuring resilience and exceptional user experience
  • Drive technical strategy and architecture decisions across multiple services, influencing partner teams and aligning with compliance and security requirements
  • Champion modern engineering practices, including AI-driven approaches, automation, and cloud-native patterns, across the full development lifecycle
  • Mentor and guide engineers, fostering technical excellence and continuous improvement in security, reliability, and performance
  • Collaborate cross-org to solve challenging technical problems, streamline processes, and reduce operational costs while improving live-site health
  • Design and implement scalable backend services optimized for machine learning workflows and large language model integration
  • Develop and maintain evaluation-driven systems that leverage text and multimodal inputs (e.g., images) to power visual-creation experiences
  • Build and optimize APIs and infrastructure to support high-performance model inference and experimentation at scale
  • Collaborate with product, ML, and design teams to integrate models into user-facing features, ensuring seamless functionality and performance
  • Conduct model evaluations and experiments, analyze results, and iterate on improvements to enhance accuracy and user experience
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

Location
Location
Canada , Mississauga
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience: Extensive experience in designing and building AI/ML solutions, with a significant focus on generative AI and Large Language Models (LLMs)
  • Gen AI Expertise: Deep understanding of modern AI architectures and techniques, including Retrieval-Augmented Generation (RAG), fine-tuning, function calling, and AI agentic workflows
  • Programming Proficiency: Expert-level skills in Python and extensive experience with core AI/ML libraries such as PyTorch, TensorFlow
  • System Design: Proven ability to architect and develop large-scale, distributed, multi-tier applications. Strong knowledge of microservices, API design, and system integration
  • MLOps: Solid understanding of MLOps principles and experience with tools for model versioning, deployment, monitoring, and lifecycle management
  • Leadership: Demonstrated experience serving as a technical lead, architect, or principal engineer, with a track record of mentoring team members and driving projects to completion
Job Responsibility
Job Responsibility
  • Architectural Leadership: Design and architect end-to-end generative AI solutions, from proof-of-concept to production, ensuring scalability, performance, and reliability
  • Technical Strategy: Develop and maintain a comprehensive strategic roadmap for generative AI adoption, evaluating new models, techniques, and platforms to keep our capabilities at the forefront of the industry
  • Solution Development: Lead the hands-on development of complex AI systems, including Retrieval-Augmented Generation (RAG) pipelines, autonomous AI agents, fine-tuning workflows, and custom model integrations
  • Best Practices & Standards: Establish and govern best practices for the full AI development lifecycle, including prompt engineering, model evaluation, MLOps, and data management
  • Cross-Functional Partnership: Collaborate closely with multiple management teams and business units to identify high-impact use cases and ensure the successful integration of AI solutions to meet business goals
  • Mentorship & Guidance: Serve as a senior advisor and coach to other engineers and analysts, fostering a culture of innovation and technical excellence. Allocate work and provide technical direction to the team
  • Risk & Compliance: Appropriately assess risk when business decisions are made, demonstrating consideration for the firm's reputation and safeguarding its clients and assets. Drive compliance with all applicable laws, rules, and regulations, particularly those related to AI ethics, data privacy, and model bias
  • Innovation and Research: Stay abreast of the latest advancements in generative AI research, and translate state-of-the-art developments into practical, innovative solutions
  • Fulltime
Read More
Arrow Right

Principal AI Engineer – Vivado EDA Tools

We are seeking a skilled AI expert to join our new team focused on integrating A...
Location
Location
United States , San Jose
Salary
Salary:
240000.00 - 360000.00 USD / Year
amd.com Logo
AMD
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Extensive experience with AI/ML initiatives and a proven track record of deploying production AI systems
  • Deep expertise in Large Language Models, fine-tuning, and prompt engineering
  • Experience designing agentic AI solutions and multi-agent systems
  • Strong background in EDA tools and workflows, preferably with Synopsys experience
  • Proficiency in MLOps, model deployment, and lifecycle management
  • Advanced skills in Python, C++, and AI/ML orchestration frameworks
  • In-depth knowledge of FPGA design methodologies and tool development
Job Responsibility
Job Responsibility
  • Design and implement Large Language Model integrations for FPGA design assistance
  • Develop Agentic AI systems for automated design optimization
  • Conduct research and development of novel AI applications in EDA tools
What we offer
What we offer
  • Benefits offered are described: AMD benefits at a glance
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

We are seeking a highly accomplished Principal AI Engineer to define and drive t...
Location
Location
Ireland , Dublin 18
Salary
Salary:
Not provided
mastercard.com Logo
Mastercard
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Demonstrated experience designing and building AI/ML systems in production at scale, ideally across multiple problem domains
  • Expert-level proficiency in Python and deep experience with modern AI frameworks such as PyTorch and TensorFlow
  • Strong experience with cloud-native architectures and AI infrastructure on platforms such as AWS, Azure, or GCP
  • Deep understanding of machine learning, deep learning, NLP, generative AI, and transformer-based architectures (e.g., BERT, GPT-style models, ViTs)
  • Proven expertise in MLOps, including model versioning, deployment strategies, monitoring, evaluation, and lifecycle management
  • Strong systems-thinking mindset, with experience designing resilient, scalable, and cost-efficient AI services
  • Experience working with large-scale data architectures, streaming and batch processing, and model inference optimization
  • Excellent communication skills with the ability to explain complex technical concepts to both technical and non-technical stakeholders
  • Track record of technical mentorship and influence without relying on formal line management
  • Comfortable operating in high-ambiguity environments and making sound technical judgments with incomplete information
Job Responsibility
Job Responsibility
  • Define and drive the technical direction of our AI platforms and solutions
  • Architect, build, and scale production-grade AI systems that deliver durable business impact
  • Lead through deep hands-on expertise, influence technical strategy across teams, and raise the engineering bar for AI development across the organization
  • Design, implement, and operate advanced AI systems that support critical business and client needs in a scalable, secure, and reliable manner
  • Partner closely with product, engineering, and data leaders to translate business intent into robust AI architectures and platforms
  • Fulltime
Read More
Arrow Right

Principal AI Engineer – Vivado EDA Tools

We are seeking a skilled AI expert to join our new team focused on integrating A...
Location
Location
United States , Boxborough
Salary
Salary:
212000.00 - 318000.00 USD / Year
amd.com Logo
AMD
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Extensive experience with AI/ML initiatives and a proven track record of deploying production AI systems
  • Deep expertise in Large Language Models, fine-tuning, and prompt engineering
  • Experience designing agentic AI solutions and multi-agent systems
  • Strong background in EDA tools and workflows, preferably with Synopsys experience
  • Proficiency in MLOps, model deployment, and lifecycle management
  • Advanced skills in Python, C++, and AI/ML orchestration frameworks
  • In-depth knowledge of FPGA design methodologies and tool development
Job Responsibility
Job Responsibility
  • Design and implement Large Language Model integrations for FPGA design assistance
  • Develop Agentic AI systems for automated design optimization
  • Conduct research and development of novel AI applications in EDA tools
  • Fulltime
Read More
Arrow Right

Staff Software Engineer, Model LifeCycle

The Staff Software Engineer for the Model LifeCycle team will play a key role in...
Location
Location
United States , San Francisco
Salary
Salary:
208725.00 - 253000.00 USD / Year
crusoe.ai Logo
Crusoe
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field
  • 8-10+ years of industry experience with demonstrated history of consistent success leading a varied portfolio of initiatives across your function
  • Proven track record of delivering production features on time
  • Experience in using cloud-based services, such as, elastic compute, object storage, virtual private networks, managed database, etc.
  • Experience with Generative AI (Large Language Models, Multimodal)
  • Experience with AI infrastructure, including training, inference
Job Responsibility
Job Responsibility
  • Contribute to fine-tuning systems for large foundation models (SFT, PEFT, LoRA, adapters), including multi-node orchestration, checkpointing, failure recovery, and cost-efficient scaling
  • Implement and maintain end-to-end training pipelines for Large Language Models
  • Contribute to distillation and reinforcement learning pipelines (e.g., preference optimization, policy optimization, reward modeling)
  • Develop and maintain agent execution infrastructure
  • Implement features for dataset, model, and experiment management: versioning, lineage, evaluation, and reproducible fine-tuning at scale
  • Work closely with Principal Engineers, product, business, and platform teams to implement the core abstractions and APIs of the system
  • Contribute to architectural decisions around training runtimes, scheduling, storage, and model lifecycle management
  • Engage with the open-source LLM ecosystem
What we offer
What we offer
  • Restricted Stock Units in a fast growing, well-funded technology company
  • Health insurance package options that include HDHP and PPO, vision, and dental for you and your dependents
  • Employer contributions to HSA accounts
  • Paid Parental Leave
  • Paid life insurance, short-term and long-term disability
  • Teladoc
  • 401(k) with a 100% match up to 4% of salary
  • Generous paid time off and holiday schedule
  • Cell phone reimbursement
  • Tuition reimbursement
  • Fulltime
Read More
Arrow Right

Principal AI Software Engineer, Senior Vice President

Are you looking for a career move that will put you at the heart of a global fin...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Exceptional Python Expertise: Demonstrated mastery of core Python, including advanced features, performance optimization, and a deep understanding of the FastAPI framework
  • Prior hands-on experience with Generative AI, Large Language Model (LLM) frameworks (e.g. LangChain, LlamaIndex), and their application in enterprise environments is a must. This must be underpinned by a profound understanding of core machine learning principles, algorithms, and data science methodologies
  • Full Lifecycle Ownership: Extensive hands-on experience and technical authority throughout the entire software development lifecycle, from conceptualization and design to implementation, deployment, and operational ownership of enterprise software solutions, involving significant cross-functional collaboration
  • Strategic System Design: Significant hands-on experience in architecting and designing (architecture, design patterns, reliability, scaling) highly complex new and current systems with broad technical impact
  • Hands-on expertise with containerized deployment technologies (e.g. Kubernetes, OpenShift, Docker) and orchestration strategies
  • Hands-on experience and in-depth understanding of C++ is a significant bonus, particularly for complex code analysis, parsing, and integration into knowledge graph structures
Job Responsibility
Job Responsibility
  • Architect and implement cutting-edge software systems, defining the technical design for our AI solutions to ensure scalability, performance, and reliability
  • Drive the hands-on design, implementation, and deployment of sophisticated systems that automate the analysis of data, code, and documentation
  • Apply deep expertise to structure extracted knowledge within a Credit Risk Domain-aware knowledge graph, including advanced strategies for effectively modelling complex codebases, particularly C++, within this graph
  • Act as a critical technical partner with data scientists, business analysts, and other engineering teams to translate challenging business requirements into robust technical solutions and ensure successful, high-quality project delivery
  • Tackle the most complex technical challenges within our AI initiatives, providing solutions that set the standard for engineering excellence
What we offer
What we offer
  • Generous holiday allowance starting at 27 days plus bank holidays
  • increasing with tenure
  • A discretional annual performance related bonus
  • Private medical insurance packages to suit your personal circumstances
  • Employee Assistance Program
  • Pension Plan
  • Paid Parental Leave
  • Special discounts for employees, family, and friends
  • Access to an array of learning and development resources
  • Fulltime
Read More
Arrow Right