CrawlJobs Logo

AI Research Engineer, Enterprise Evaluations

United States, San Francisco 179400.00 - 224250.00 USD / Year · Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

Scale AI is seeking a technically rigorous and driven AI Research Engineer to join our Enterprise Evaluations team. This high-impact role is critical to our mission of delivering the industry's leading GenAI Evaluation Suite. You will be a hands-on contributor to the core systems that ensure the safety, reliability, and continuous improvement of LLM-powered workflows and agents for the enterprise.

Job Responsibility

  • Partner with Scale’s Operations team and enterprise customers to translate ambiguity into structured evaluation data, guiding the creation and maintenance of gold-standard human-rated datasets and expert rubrics that anchor AI evaluation systems
  • Analyze feedback and collected data to identify patterns, refine evaluation frameworks, and establish iterative improvement loops that enhance the quality and relevance of human-curated assessments
  • Design, research, and develop LLM-as-a-Judge autorater frameworks and AI-assisted evaluation systems. This includes creating models that critique, grade, and explain agent outputs (e.g., RLAIF, model-judging-model setups), along with scalable evaluation pipelines and diagnostic tools
  • Pursue research initiatives that explore new methodologies for automatically analyzing, evaluating, and improving the behavior of enterprise agents, pushing the boundaries of how AI systems are assessed and optimized in real-world contexts

Requirements

  • Bachelor’s degree in Computer Science, Electrical Engineering, a related field, or equivalent practical experience
  • 2+ years of experience in Machine Learning or Applied Research, focused on applied ML systems or evaluation infrastructure
  • Hands-on experience with Large Language Models (LLMs) and Generative AI in professional or research environments
  • Strong understanding of frontier model evaluation methodologies and the current research landscape
  • Proficiency in Python and major ML frameworks (e.g., PyTorch, TensorFlow)
  • Solid engineering and statistical analysis foundation, with experience developing data-driven methods for assessing model quality

Nice to have

  • Advanced degree (Master’s or Ph.D.) in Computer Science, Machine Learning, or a related quantitative field
  • Published research in leading ML or AI conferences such as NeurIPS, ICML, ICLR, or KDD
  • Experience designing, building, or deploying LLM-as-a-Judge frameworks or other automated evaluation systems for complex models
  • Experience collaborating with operations or external teams to define high-quality human annotator guidelines
  • Expertise in ML research engineering, stochastic systems, observability, or LLM-powered applications for model evaluation and analysis
  • Experience contributing to scalable pipelines that automate the evaluation and monitoring of large-scale models and agents
  • Familiarity with distributed computing frameworks and modern cloud infrastructure

What we offer

  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
  • commuter stipend
  • equity grant

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

AI Research Engineer, Enterprise Evaluations

8 matching positions

Principal Ai Engineer - Enterprise Ai Solutions

Our Mission: At Palo Alto Networks®, we’re united by a shared mission—to protect...
Location
Location
United States , Santa Clara
Salary
Salary:
185200.00 - 299475.00 USD / Year
paloaltonetworks.com Logo
Palo Alto Networks
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in a related field or equivalent military experience required
  • 12+ years of related experience, or a Master's degree with 8 years of experience, or a PhD with 5 years of experience
  • Proven expertise in designing and building complex, enterprise-grade AI/ML platforms and applications
  • Direct hands-on experience with Generative AI technologies, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and agentic AI systems
  • Strong programming skills in languages such as Python, Java, or Go and proficiency in modern AI/ML frameworks (e.g., TensorFlow, PyTorch)
  • Extensive hands-on experience with distributed systems architecture, streaming data platforms, and cloud AI/ML platforms (e.g., GCP Vertex AI, AWS SageMaker)
Job Responsibility
Job Responsibility
  • Lead the applied AI solution design and architecture, breaking down ambiguous business problems into concrete, actionable AI solution designs
  • Drive the hands-on development and implementation of key AI components, supporting both traditional and Generative AI model development and deployment
  • Contribute significantly to the detailed design of large-scale, distributed AI/ML systems, ensuring performance, reliability, and security
  • Proactively collaborate with executive leadership, data science, engineering, and product stakeholders to translate business use-cases into scalable AI solutions
  • Provide technical leadership and mentorship to other AI/ML engineers, fostering a culture of engineering excellence and hands-on experimentation
  • Lead the implementation and continuous improvement of MLOps pipelines, including automated model training, versioning, deployment, and monitoring
  • Champion and enforce design standards, patterns, and best practices for scalable and secure development of AI applications across various teams
  • Actively research and evaluate cutting-edge AI/ML techniques, algorithms, and models to identify opportunities for platform enhancement and new solution development
What we offer
What we offer
  • Restricted stock units
  • Bonus
  • Fulltime
Read More
Arrow Right

Enterprise AI Engineer - Modernization

We’re hiring a Generative AI Engineer for our AI-powered modernization platform ...
Location
Location
India , Chennai, Madurai
Salary
Salary:
Not provided
optisolbusiness.com Logo
OptiSol Business Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands-on experience working with foundation models such as GPT-4, Claude, Llama 3, or Mistral to build real-world AI applications
  • Strong prompt engineering skills including Chain-of-Thought, Few-Shot Prompting, and Tree-of-Thought reasoning techniques
  • Ability to design and optimize prompts to improve response accuracy, reduce latency, and control token cost
  • Experience creating system prompts with structured outputs (JSON or schema-based) and implementing safety guardrails
  • Understanding of LLM data preparation including dataset cleaning, formatting & tokenization
  • Exposure to model adaptation or fine-tuning workflows using modern ML pipelines
  • Working knowledge of machine learning frameworks such as PyTorch or TensorFlow
  • Hands-on experience designing and implementing end-to-end Retrieval-Augmented Generation (RAG) pipelines
  • Exposure to vector databases such as Pinecone, Weaviate, Milvus, or Qdrant
  • Strong understanding of embeddings, semantic search, and re-ranking strategies
Job Responsibility
Job Responsibility
  • Design, develop, and deploy Generative AI–powered software solutions
  • Build robust, scalable Python services integrating Large Language Models (LLMs)
  • Craft and optimize advanced prompts to enforce accuracy, tone, formatting, and guardrails
  • Fine-tune and adapt LLMs for domain-specific and business-critical use cases
  • Architect and implement end-to-end Retrieval-Augmented Generation (RAG) pipelines
  • Develop autonomous and semi-autonomous AI agents
  • Integrate LLMs with external APIs, databases, and computation engines
  • Evaluate and monitor model behavior to detect hallucinations, drift, and quality regressions
  • Write production-grade, modular Python code and participate in code reviews
  • Stay current with emerging research, tools, and best practices in Generative AI
  • Fulltime
Read More
Arrow Right

Distinguished AI Engineer (Agentic Ai Platform)

Distinguished AI Engineer (Agentic AI Platform). At Capital One, we are creating...
Location
Location
United States , San Francisco, California; McLean, Virginia; New York, New York; San Jose, California; Cambridge, Massachusetts
Salary
Salary:
Not provided
capitalone.com Logo
Capital One
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Engineering, or AI plus at least 8 years of experience developing AI and ML algorithms or technologies, or Master's degree plus at least 6 years of experience developing AI and ML algorithms or technologies
  • At least 8 years of experience programming with Python, Go, Scala, or Java
Job Responsibility
Job Responsibility
  • Partner with a cross-functional team of engineers, research scientists, technical program managers, and product managers to deliver AI-powered products that change how our associates work and how our customers interact with Capital One
  • Contribute to the north star platform architecture, continuously publishing and refining living diagrams and canonical APIs that cover agent orchestration, RAG pipelines, prompt libraries and multi-tenant policy enforcement
  • Evaluate agentic frameworks such LangGraph, AutoGen, Semantic Kernal, CrewAI and LlamaIndex and then harden / blend patterns that best meet enterprise SLAs
  • Contribute to crafting an end to end GenAI SDK, CLI and starter kits that let AI engineers spin up secure, observable agentic workflows in under minutes
  • Help bring together a vision of central guardrail services - prompt firewalls, content-filter hooks, red team harnesses and audit APIs
  • Collaborate with cross organization architects to drive end to end performance by optimizing orchestration - level batching, retrieval caching, heuristic tuning to achieve reductions in per token spend
  • Accelerate innovation by incubating proof of concepts and driving RFCs such as hierarchical agent memory, multimodal guardrails, multimodal RAG
  • Own central Helm charts, operators and CRDs that auto scale agents to hit tenant SLAs
  • Coach and evangelize - hosting architecture office hours, mentoring Staff, Principal and Senior engineers, authoring technical design documents and blogs and representing Capital One at Tier1 AI conferences
What we offer
What we offer
  • Performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
  • comprehensive, competitive, and inclusive set of health, financial and other benefits
  • Healthy Body, Healthy Mind
  • Save Money, Make Money
  • Time, Family and Advice
  • Fulltime
Read More
Arrow Right

Senior Data Scientist – Gen AI Engineer - Assistant Vice President

Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Data Science, Artificial Intelligence, or a related quantitative field.
  • 8–12 years of experience as a Data Scientist or equivalent role, with at least 2 years of specialized, hands-on experience in Generative AI, including leading technical development and mentoring teams.
  • Demonstrable experience across the full lifecycle of production-level GenAI projects — from ideation and prototyping through deployment, monitoring, and ongoing maintenance in live environments.
  • Expert-level Python proficiency
  • Scikit-learn, XGBoost, LightGBM
  • PyTorch
  • Hugging Face Transformers, LangChain, LlamaIndex, Semantic Kernel
  • LangGraph, CrewAI, AutoGen
  • Transformer architectures
  • RAG patterns
Job Responsibility
Job Responsibility
  • Working with financial and enterprise data, applying modern NLP and GenAI techniques to solve business problems.
  • Designing, refining, and systematizing prompt engineering strategies for large language models (LLMs), including structured prompting, chain-of-thought, and few-shot/zero-shot approaches.
  • Collaborating with business stakeholders to translate requirements into GenAI-powered solutions.
  • Developing, testing, and maintaining production-grade Python code for GenAI applications.
  • Integrating with vector databases (e.g., Pinecone, Weaviate, Milvus, pgvector, Qdrant) for retrieval-augmented generation (RAG) pipelines.
  • Building, monitoring, and optimizing MLOps/LLMOps pipelines for continuous model deployment and observability.
  • Researching and evaluating emerging GenAI technologies, frameworks, and best practices to maintain competitive advantage.
  • Troubleshooting and debugging GenAI models and agentic systems in production, including rapid identification and resolution of issues in real-world deployments.
  • Communicating complex AI/ML concepts clearly to non-technical stakeholders, translating technical jargon into actionable business terms.
  • Participating in and leading team meetings, design reviews, and architecture discussions.
  • Fulltime
Read More
Arrow Right

Ai Engineer

We are building a new AI / Machine Learning team within an industrial digital or...
Location
Location
Czech Republic , Prague
Salary
Salary:
110000.00 - 140000.00 CZK / Year
apollo-solutions.com Logo
Apollo Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Degree in Computer Science, AI, Data Science, or related field
  • 5+ years experience in software engineering and AI/ML
  • 2+ years hands-on experience with generative AI
  • Proven experience deploying and fine-tuning ML models in production
  • Strong skills in Python (Java / .NET a plus)
  • Solid understanding of ML, deep learning, and statistical modeling
  • Experience with cloud platforms and containerization (Docker, Kubernetes)
  • Familiarity with Agile environments and version control (Git)
  • Strong collaboration and problem-solving skills
Job Responsibility
Job Responsibility
  • Act as a technical lead between business, product, and engineering teams
  • Translate business needs into scalable AI / ML solutions
  • Design, fine-tune, deploy, and optimize generative AI models (primarily LLMs)
  • Develop production-grade machine learning pipelines and algorithms
  • Work with structured, unstructured, time-series, and text data
  • Integrate AI/ML models into enterprise digital platforms
  • Evaluate model performance and continuously improve accuracy and reliability
  • Stay current with AI/ML research, tooling, and best practices
  • Document architectures, models, and processes for reuse and scalability
  • Fulltime
Read More
Arrow Right

Azure Data Science Platform Engineer (AI)

WFH flexibility! Up to 4 days/week! Global Environment! Competitive salary! We a...
Location
Location
Japan , Tokyo 23 wards
Salary
Salary:
7000000.00 - 12000000.00 JPY / Year
https://www.randstad.com Logo
Randstad
Expiration Date
February 29, 2028
Flip Icon
Requirements
Requirements
  • Bilingual proficiency in Japanese and English is preferred (English is a MUST)
  • Day-to-day communication will primarily be in English, with occasional interaction with Japanese-speaking stakeholders
  • 6+ years of experience in data science, machine learning, advanced analytics, or applied AI, with demonstrated business results
  • Strong experience taking solutions from development into production and supporting them in live environments
  • Strong Python programming skills and solid engineering discipline
  • Experience with GenAI / LLM use cases or solution delivery
  • Hands-on experience with Azure for deploying, supporting, or operating production workloads
  • Strong experience with Terraform and Infrastructure as Code in enterprise cloud environments
  • Experience with CI/CD, GitHub Actions, deployment automation, and DevOps practices
  • Experience with SRE, MLOps, or LLMOps, particularly in monitoring, incident handling, reliability, and operational support
Job Responsibility
Job Responsibility
  • We are looking for a candidate who combines strong data science delivery capability with practical production and operational ownership
  • This is not a pure research role and not a pure platform role. It is intended for someone who can build business-facing AI/ML solutions and also help ensure those solutions are deployable, stable, and maintainable
  • The ideal candidate is proactive, technically hands-on, comfortable working independently, and effective in cross-functional enterprise environments.
What we offer
What we offer
  • WFH flexibility! Up to 4 days/week!
  • Global Environment!
  • Competitive salary!
  • 健康保険
  • 厚生年金保険
  • 雇用保険
  • 土曜日
  • 日曜日
  • 祝日
  • Fulltime
Read More
Arrow Right

Sr AI Engineer - Agentic Systems

As a core technical leader within our Agentic AI initiatives, you will shape the...
Location
Location
United States , Anywhere
Salary
Salary:
Not provided
dialpad.com Logo
Dialpad
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of relevant software engineering experience
  • Proven track record of technical leadership (as a Senior, Staff, or Principal Engineer) shipping complex, large-scale systems
  • Strong foundations in scaling distributed systems and production-grade infrastructure
  • Experience with LLM Platforms: Inference optimization and fine-tuning strategies
  • Experience with Data & Retrieval: Advanced retrieval systems and memory architectures
  • Experience with Agent Frameworks like LangChain/LangGraph, CrewAI, or AWS/Google Agent ecosystems
  • Experience with AI Ops: Evaluation, observability, and safety frameworks for production AI systems
  • Experience with Real-Time Infrastructure: Streaming infrastructure and voice/conversational AI
  • Experience with Tool Integration: Tool use, API execution frameworks, and human-in-the-loop validation systems
  • Operational Excellence: Experience setting clear technical goals, identifying architectural risks, and systematically clearing tech-debt gaps
Job Responsibility
Job Responsibility
  • Drive Technical Strategy: Own the architectural roadmap and delivery of Dialpad’s Agentic infrastructure, core orchestration layers, memory architectures, and evaluation/observability systems
  • Build & Scale: Design and deploy scalable, multi-modal AI agents capable of autonomous support, real-time voice reasoning, and secure API tool execution across complex enterprise workflows
  • Mentor & Influence: Act as a technical anchor for the organization, raising the engineering bar, mentoring senior peers, and defining technical standards for an AI-native SDLC
  • Partner Cross-Functionally: Collaborate with leadership across Product, Engineering, and Applied Research to align technical execution with Dialpad’s long-term business strategy
  • Push the Frontier: Research and implement emerging agent frameworks, LLM inference optimization, advanced retrieval systems, and cutting-edge safety/policy guardrails to keep Dialpad at the absolute forefront of the era of the agent
What we offer
What we offer
  • Competitive salary
  • Comprehensive benefits
  • Real opportunities for growth
  • Cutting-edge AI tools
  • Robust training program
  • Inclusive office environment
  • Fulltime
Read More
Arrow Right

Ai Engineer (Corp Functions & Supply Chain)

We are currently seeking a AI Engineer (Corp Functions & Supply Chain) to join o...
Location
Location
United States , Dallas
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Education: Bachelor's or Master's degree in Computer Science, Data Science, Operations Research, or a related technical field
  • Programming & Tech Stack: 7+ years of high proficiency in languages like Python or Java. Experience with frameworks and libraries such as TensorFlow, PyTorch, LangChain, and LlamaIndex
  • Enterprise Platforms: 5+ years of familiarity with modern data and ML platforms (e.g., Databricks, Oracle Cloud SCM, AWS, or Azure)
  • Supply Chain Domain: 3 to 5 years of understanding of Procure-to-Pay, Order-to-Cash, and warehouse/inventory logistics
  • Compliance with Client's responsible AI principles and Acceptable Use policy
  • Adherence to data residency, privacy (GDPR, HIPAA where applicable), and 21 CFR Part 11 controls where in scope
  • Third-party risk assessment and SOC 2 Type II (or equivalent) certification
  • Disclosure of subcontractors and offshore delivery locations
  • Disclosure of model providers, training data practices, and any use of client data for model improvement (opt-out required)
Job Responsibility
Job Responsibility
  • AI Model Development & Deployment: Build and productionize machine learning algorithms for demand forecasting, inventory optimization, and logistics routing
  • Corporate & Procurement Optimization: Design GenAI assistants and agents to automate supplier evaluation, negotiation, contract review, and vendor management
  • Workflow Automation: Deploy Agentic AI to enable autonomous decision-making—such as auto-rerouting shipments during a supply disruption or updating your Enterprise Resource Planning (ERP) without human intervention
  • System Integration & Architecture: Create RAG pipelines and integrate AI models into existing enterprise systems (like Oracle Cloud, SAP, or AWS/Azure) so that AI recommendations flow directly into business execution
  • Monitoring & Guardrails: Establish continuous performance metrics and guardrails to ensure AI models remain accurate, cost-effective, and aligned with enterprise safety standards
  • Fulltime
Read More
Arrow Right