CrawlJobs Logo

Manager, Agent Evaluation

comcastadvertising.com Logo

Comcast Advertising

Location Icon

Location:
United States , Washington D.C.

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

183063.62 - 274595.42 USD / Year

Job Description:

The Agent Evaluation team is responsible for testing whether AI agents return the correct and expected responses. We build the framework, metrics, and test cases that validate agent behavior, accuracy, and reliability before release. Our goal is to ensure agents perform consistently and meet product and user expectations. The Manager, Agent Evaluation will lead the team responsible for building and scaling the evaluation framework that tests whether AI agents return accurate, reliable, and expected responses across real-world scenarios.

Job Responsibility:

  • Lead and grow a team focused on agent and model evaluation
  • Define the strategy, roadmap, and standards for agent testing and validation
  • Oversee development of metrics, benchmarks, and testing frameworks to measure response quality, accuracy, safety, and performance
  • Ensure evaluation coverage aligns with product, UX, and business requirements
  • Partner closely with Product, Engineering, Research, and Platform teams to integrate evaluation into the development lifecycle
  • Drive experimentation and continuous improvement of evaluation methodologies
  • Establish reporting mechanisms to clearly communicate evaluation results and trade-offs to leadership
  • Implement best practices for model versioning, monitoring, and release validation
  • Stay current with advancements in LLMs, AI agents, and evaluation techniques

Requirements:

  • Strong foundation in machine learning fundamentals and applied ML systems
  • Hands-on experience with model and agent evaluation methodologies
  • Familiarity with LLMs, AI agents, and prompt-driven systems
  • Proficiency in Python and modern ML frameworks (e.g., PyTorch, TensorFlow)
  • Experience defining metrics, benchmarks, and experimentation frameworks
  • Solid understanding of MLOps practices, including model versioning, monitoring, and CI/CD
  • Ability to collaborate effectively with product, platform, and research teams
  • Clear communicator of technical trade-offs, evaluation insights, and results
  • Master's Degree
  • 5-7 Years Relevant Work Experience
What we offer:
  • Paid Time off
  • Physical Wellbeing benefits
  • Financial Wellbeing benefits
  • Emotional Wellbeing benefits
  • Life Events + Family Support benefits

Additional Information:

Job Posted:
February 13, 2026

Employment Type:
Fulltime
Work Type:
On-site work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Manager, Agent Evaluation

Senior Product Manager, AI Agents

This role owns AI research, messaging, and context—spanning both the user experi...
Location
Location
United States
Salary
Salary:
187000.00 - 250000.00 USD / Year
apollo.io Logo
Apollo.io
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years in product management
  • 2+ years experience launching AI/ML new products and scaling existing products
  • Track record of shipping AI features that drove measurable business outcomes
  • Experience with LLM-powered applications, prompt engineering, evaluation frameworks, and model selection tradeoffs
  • Comfortable working in Python/SQL to analyze data, prototype prompts, and evaluate outputs
  • Understanding of LLM architectures, RAG pipelines, agent frameworks, and inference optimization
  • Obsession with quality over speed
  • GTM or sales tech experience (strongly preferred)
  • Familiarity with sales workflows, prospecting tools, or CRM systems
  • Understanding of why sales teams are skeptical of AI tools and what it takes to earn their trust
Job Responsibility
Job Responsibility
  • Develop and execute a strategic roadmap for AI research, messaging, and context capabilities
  • Enhance Apollo's AI research agents to surface actionable insights from the web
  • Define how AI understands each user's business
  • Own AI-powered messaging tools that create personalized, context-aware emails at scale
  • Build and scale evaluation infrastructure across accuracy, relevance, clarity, and tone
  • Partner with engineering, design, prompt writers, and sales to deliver cohesive AI experiences
What we offer
What we offer
  • Equity
  • Company bonus or sales commissions/bonuses
  • 401(k) plan
  • At least 10 paid holidays per year
  • Flex PTO
  • Parental leave
  • Employee assistance program and wellbeing benefits
  • Global travel coverage
  • Life/AD&D/STD/LTD insurance
  • FSA/HSA and medical, dental, and vision benefits
  • Fulltime
Read More
Arrow Right

AI Engineering Manager - Internal AI Agent

We are looking for an AI Engineering Manager to drive Mirakl's internal AI trans...
Location
Location
France , Paris
Salary
Salary:
Not provided
mirakl.com Logo
Mirakl
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in AI/ML or software engineering
  • Proven track record building AI agents using LLMs, RAG, MCP and related technologies
  • Strong technical proficiency in Python and multiple programming languages, with architectural design experience
  • Production deployment expertise - you've shipped AI solutions to real users
  • Technical pragmatism - ability to match the right technology to the use case
  • Curiosity and continuous learning - you stay current with AI/ML trends
  • 1+ years experience as a Lead or management roles (team management or technical leadership)
  • Strong leadership skills - you inspire and develop high-performing engineering teams
  • Cross-functional stakeholder management - you build relationships and excel at working with all organizational levels & functions
  • Strong communication & presentation skills - in both English and French
Job Responsibility
Job Responsibility
  • Partner closely with Mirakl teams & leadership to identify & prioritize opportunities, redesign workflows around AI agents, and drive adoption at scale
  • Lead and mentor a team of cross-functional AI engineers, defining your team’s roadmap to support strategic AI initiatives
  • Build advanced Mirakl-specific AI agents centrally, owning the complete delivery cycle from discovery to production deployment and operations
  • Foster organization-wide AI adoption by animating internal communities, providing self-service tools, training & support to empower teams as autonomous AI builders
  • Establish & scale technical standards & stack to ensure secure, compliant & high-quality deliverables across all internal AI projects
  • Explore emerging AI paradigms, evaluate new tools and technologies, and maintain active technology watch
Read More
Arrow Right

AI Engineering Manager - Internal AI Agent

We are looking for an AI Engineering Manager to drive Mirakl's internal AI trans...
Location
Location
France
Salary
Salary:
Not provided
mirakl.com Logo
Mirakl
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in AI/ML or software engineering
  • Proven track record building AI agents using LLMs, RAG, MCP and related technologies
  • Strong technical proficiency in Python and multiple programming languages, with architectural design experience
  • Production deployment expertise - you've shipped AI solutions to real users
  • Technical pragmatism - ability to match the right technology to the use case
  • Curiosity and continuous learning
  • 1+ years experience as a Lead or management roles (team management or technical leadership)
  • Strong leadership skills
  • Cross-functional stakeholder management
  • Strong communication & presentation skills - in both English and French
Job Responsibility
Job Responsibility
  • Partner closely with Mirakl teams & leadership to identify & prioritize opportunities, redesign workflows around AI agents, and drive adoption at scale
  • Lead and mentor a team of cross-functional AI engineers, defining your team’s roadmap to support strategic AI initiatives
  • Build advanced Mirakl-specific AI agents centrally, owning the complete delivery cycle from discovery to production deployment and operations
  • Foster organization-wide AI adoption by animating internal communities, providing self-service tools, training & support to empower teams as autonomous AI builders
  • Establish & scale technical standards & stack to ensure secure, compliant & high-quality deliverables across all internal AI projects
  • Explore emerging AI paradigms, evaluate new tools and technologies, and maintain active technology watch
Read More
Arrow Right

AI Engineering Manager - Internal AI Agent

We are looking for an AI Engineering Manager to drive Mirakl's internal AI trans...
Location
Location
France , Bordeaux
Salary
Salary:
Not provided
mirakl.com Logo
Mirakl
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in AI/ML or software engineering
  • Proven track record building AI agents using LLMs, RAG, MCP and related technologies
  • Strong technical proficiency in Python and multiple programming languages, with architectural design experience
  • Production deployment expertise - you've shipped AI solutions to real users
  • Technical pragmatism - ability to match the right technology to the use case
  • Curiosity and continuous learning
  • 1+ years experience as a Lead or management roles (team management or technical leadership)
  • Strong leadership skills
  • Cross-functional stakeholder management
  • Strong communication & presentation skills - in both English and French
Job Responsibility
Job Responsibility
  • Partner closely with Mirakl teams & leadership to identify & prioritize opportunities, redesign workflows around AI agents, and drive adoption at scale
  • Lead and mentor a team of cross-functional AI engineers, defining your team’s roadmap to support strategic AI initiatives
  • Build advanced Mirakl-specific AI agents centrally, owning the complete delivery cycle from discovery to production deployment and operations
  • Foster organization-wide AI adoption by animating internal communities, providing self-service tools, training & support to empower teams as autonomous AI builders
  • Establish & scale technical standards & stack to ensure secure, compliant & high-quality deliverables across all internal AI projects
  • Explore emerging AI paradigms, evaluate new tools and technologies, and maintain active technology watch
Read More
Arrow Right

Assistant Construction Project Manager

This role focuses on managing a range of Private Residential projects, from priv...
Location
Location
United Kingdom , London
Salary
Salary:
28000.00 - 38000.00 GBP / Year
https://brandonjames.co.uk Logo
Brandon James
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Holds a degree in Project Management or an equivalent qualification
  • Aspiring to achieve chartership in the future
  • Strong communication skills, both written and verbal
  • Keen interest in the field of high-end private residential construction
  • Able to effectively support senior team members in project management tasks
Job Responsibility
Job Responsibility
  • Assist in the setup and governance of high-end private residential projects
  • Monitor project processes, ensuring compliance and efficiency
  • Conduct due diligence and quality assurance checks
  • Assist in financial monitoring and progress reporting of projects
  • Participate in project audits and post-project evaluations
What we offer
What we offer
  • 25 Days holiday + Bank holidays
  • Hybrid working
  • Pension contribution
  • APC Support
  • Clear progression pathway
  • Supportive culture
  • Internal training programmes
  • Flexible working conditions
  • Birthday off
  • Company phone and laptop
  • Fulltime
Read More
Arrow Right

Unit Business Risk & Compliance Agent

You could think that we have supernatural powers, but the truth is that our team...
Location
Location
Canada , Richmond
Salary
Salary:
19.37 CAD / Hour
https://www.ikea.com Logo
IKEA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • You have previous experience in the Health and Safety and Security sector and/or Safety and Security experience within a Retail environment
  • You’re knowledgeable of relevant safety standards and regulations, security processes, tools and working methods
  • You’re energized by the implementation of safeguards that bring value to the business and protect the financial and moral position
  • you can ensure the integrity of safety and security systems, guidelines and documentation
  • You know how to conduct a risk assessment and implement the hierarchy of controls
  • You have good communication and documentation skills in dealings with various levels of management
  • You think and work in a risk-based way (i.e. Evaluate trade-offs between potential costs and benefits and acts accordingly)
  • You have good analytical and numerical skills
Job Responsibility
Job Responsibility
  • Promote risk management in the unit, informing and sharing expertise in order to develop risk-aware decision taking in relation to unit goals and unit business plan
  • Support co-workers, by providing expertise, in acting in accordance with Ingka Risk & Compliance Rules and Local legislation on Health Safety and Security to secure a safe environment for customers and co-workers
  • Promote and ensure completion of trainings needed and facilitate for unit employees
  • Support a Risk & Compliance culture by utilizing systems to detect, analyze and reduce business loss and financial impact
  • Ensure the reporting of relevant figures for co-workers, customer and visitor incidents to establish progress and areas for improvement
What we offer
What we offer
  • Wellness days (in addition to your vacation days!)
  • Extended health, dental, and vision coverage (for you and your family)
  • RRSP with IKEA contribution matching options
  • Eligibility for our annual IKEA bonus incentive plan
  • Flexible spending account
  • Life insurance
  • Merchandise and restaurant discounts (plus free drinks and different healthy meal options in the co-worker restaurant, where available)
  • Parental leave
  • Bereavement leave
  • Employee assistance program (that helps you support your mental, physical, and financial wellbeing)
  • Fulltime
Read More
Arrow Right

Senior Staff Machine Learning Engineer

Help design our AI platform and develop our next generation of machine learning ...
Location
Location
United States , San Francisco
Salary
Salary:
216500.00 - 324500.00 USD / Year
gofundme.com Logo
GoFundMe
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 9+ years of hands-on experience in machine learning engineering, AI development, software engineering, or related fields
  • Experience emphasizing secure, large-scale, distributed system design, AI/ML pipeline development, and implementation
  • Extensive experience designing, developing, and operating scalable backend systems
  • Experience applying software engineering best practices such as domain-driven design, event-driven architectures, and microservices
  • Deep expertise in agentic workflows, AI evaluation solutions, prompt management, and secure AI development and testing practices
  • Strong knowledge of relational and document-based databases, data storage paradigms, and efficient RESTful API design
  • Experience establishing robust CI/CD pipelines, automated testing (unit and integration), and deployment practices
  • Strong leadership skills, including effective planning and management of complex projects, mentoring of team members, and fostering a collaborative, high-performing engineering culture
  • Excellent communicator, able to articulate complex technical concepts clearly to both technical and non-technical stakeholders
  • Bachelor's degree in Computer Science, Software Engineering, or a related technical field (preferred)
Job Responsibility
Job Responsibility
  • Design and implement AI platforms to enable scalable and secure access to LLMs from multiple model providers for diverse use cases
  • Design and implement agentic workflows, agentic tool ecosystems, and LLM prompt management solutions
  • Design, build, and optimize scalable model training, fine tuning, and inference pipelines, ensuring robust integration with production systems
  • Influence technical strategy and approach to developing embedding stores, vector databases, and other reusable assets
  • Lead initiatives to streamline ML and AI workflows, improve operational efficiency, and establish standardized procedures to achieve consistent, high-quality results across our AI systems
  • Design and develop backend services and RESTful APIs using Python and FastAPI, integrating seamlessly with ML pipelines and services
  • Take operational responsibility for team-owned services, including performance monitoring, optimization, troubleshooting, and participation in an on-call rotation
  • Collaborate with both technical and non-technical colleagues, including data and applied scientists, software engineers, product managers, and business stakeholders, to deliver reliable and scalable ML-driven products
  • Coach and mentor fellow ML engineers, promoting a culture of collaboration, continuous improvement, and engineering excellence within the team
  • Employ a diverse set of tools and platforms including Python, AWS, Databricks, Docker, Kubernetes, FastAPI, Terraform, Snowflake, Coralogix, and GitHub to build, deploy, and maintain scalable, highly available machine learning infrastructure
What we offer
What we offer
  • Competitive pay
  • Comprehensive healthcare benefits
  • Financial assistance for things like hybrid work, family planning
  • Generous parental leave
  • Flexible time-off policies
  • Mental health and wellness resources
  • Learning, development, and recognition programs
  • Fulltime
Read More
Arrow Right

Quality Assurance Specialist for Contact Centre

We are looking for our next QA specialist for Contact Centre, main activities in...
Location
Location
Mexico , Ciudad de México
Salary
Salary:
Not provided
https://www.ikea.com Logo
IKEA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience on Contact Centre: Quality Assurance supervisor or similar
  • 1-3+ years of experience in a Contact Center environment, preferably in a Quality Assurance (QA) or Coaching role
  • Demonstrable prior experience monitoring and evaluating customer interactions (calls, emails, chat)
  • Experience conducting calibration sessions with leadership and training teams is highly desirable
  • Exceptional Attention to Detail: Must be meticulous in evaluating interactions against complex criteria
  • Strong Analytical and Critical Thinking Skills: The ability to be objective and data-driven in assessments
  • Outstanding Communication Skills (Written and Verbal): Essential for delivering clear, constructive, and professional feedback to agents and creating comprehensive reports for management
  • Coaching and Interpersonal Skills: The capacity to deliver feedback empathetically and motivate agents to improve their performance
  • Time Management and Organization: Ability to manage a high volume of monitoring tasks and meet deadlines
  • Problem-Solving: Proactive approach to identifying and proposing solutions for quality or process gaps
Job Responsibility
Job Responsibility
  • Monitor and evaluate agent-customer interactions (calls, emails, chat, etc.) to assess adherence to quality standards, policies, and procedures
  • Conduct call monitoring and scoring using established quality scorecards and evaluation criteria
  • Provide constructive and actionable feedback to contact center agents based on evaluation findings to drive performance improvement
  • Participate in calibration sessions with team leaders and trainers to ensure consistency and objectivity in the evaluation process
  • Analyze quality metrics (e.g., quality scores, Customer Satisfaction (CSAT), First Contact Resolution (FCR)) to identify trends, root causes of issues, and areas for improvement
  • Prepare comprehensive quality reports and performance analyses for management and relevant stakeholders
  • Collaborate with the Training and Coaching teams to develop and recommend targeted training programs based on identified skill and knowledge gaps
  • Assist in the development and maintenance of quality assurance standards, scorecards, and process documentation (SOPs)
  • Ensure agents comply with regulatory requirements and company policies related to customer interactions and data privacy
  • Identify and report systemic operational or process issues that impact customer experience or agent performance
  • Fulltime
Read More
Arrow Right