CrawlJobs Logo

Manager, Agent Evaluation

United States, Washington D.C. 183063.62 - 274595.42 USD / Year · Job Posted February 13, 2026
Apply Position
Job Link Share

Job Description

The Agent Evaluation team is responsible for testing whether AI agents return the correct and expected responses. We build the framework, metrics, and test cases that validate agent behavior, accuracy, and reliability before release. Our goal is to ensure agents perform consistently and meet product and user expectations. The Manager, Agent Evaluation will lead the team responsible for building and scaling the evaluation framework that tests whether AI agents return accurate, reliable, and expected responses across real-world scenarios.

Job Responsibility

  • Lead and grow a team focused on agent and model evaluation
  • Define the strategy, roadmap, and standards for agent testing and validation
  • Oversee development of metrics, benchmarks, and testing frameworks to measure response quality, accuracy, safety, and performance
  • Ensure evaluation coverage aligns with product, UX, and business requirements
  • Partner closely with Product, Engineering, Research, and Platform teams to integrate evaluation into the development lifecycle
  • Drive experimentation and continuous improvement of evaluation methodologies
  • Establish reporting mechanisms to clearly communicate evaluation results and trade-offs to leadership
  • Implement best practices for model versioning, monitoring, and release validation
  • Stay current with advancements in LLMs, AI agents, and evaluation techniques

Requirements

  • Strong foundation in machine learning fundamentals and applied ML systems
  • Hands-on experience with model and agent evaluation methodologies
  • Familiarity with LLMs, AI agents, and prompt-driven systems
  • Proficiency in Python and modern ML frameworks (e.g., PyTorch, TensorFlow)
  • Experience defining metrics, benchmarks, and experimentation frameworks
  • Solid understanding of MLOps practices, including model versioning, monitoring, and CI/CD
  • Ability to collaborate effectively with product, platform, and research teams
  • Clear communicator of technical trade-offs, evaluation insights, and results
  • Master's Degree
  • 5-7 Years Relevant Work Experience

What we offer

  • Paid Time off
  • Physical Wellbeing benefits
  • Financial Wellbeing benefits
  • Emotional Wellbeing benefits
  • Life Events + Family Support benefits

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Manager, Agent Evaluation

8 matching positions

Agent Product Manager

Plaud is building the world's most trusted AI work companion for professionals t...
Location
Location
Singapore , Singapore
Salary
Salary:
Not provided
plaud.ai Logo
Plaud
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience in LLM Agent architectures and API Platform products
  • Proficient in defining JSON Schemas and OpenAPI specifications
  • Capability to transform abstract business logic into standardized, executable AI tools
  • Strong ability to define 'Ground Truth' and build quantifiable evaluation frameworks
  • Experience in creating 'Golden Datasets' and analyzing production logs
  • Professional English proficiency
  • Deep expertise in independently iterating context management strategies and tool descriptions
Job Responsibility
Job Responsibility
  • Architect the Plaud Agent Tooling Ecosystem: Lead the end-to-end strategic planning and execution of the Plaud Intelligence Agent's skill ecosystem
  • Hold full ownership of the ecosystem's lifecycle—driving tool quantity, execution quality, and user penetration rates—transforming it from a 0-to-1 concept into a scalable, high-utility production platform
  • Define 'Ground Truth' for User Scenarios: Transform complex product scenarios into quantifiable, high-bar evaluation metrics
  • Drive the collaboration across Product, Evaluation, and Engineering teams to operationalize these standards
  • Push Prompt Performance to SOTA Standards: Spearhead the advanced iteration of tool-related prompt strategies and context engineering infrastructure
  • Fine-tune the AI’s interaction logic and reasoning capabilities
What we offer
What we offer
  • Employee Stock Ownership Plan (ESOP)
  • Work in a fast-moving, product-driven environment
  • Access to best-in-class AI tools (Cursor, GPT models, Gemini, Claude, and other frontier AI systems)
  • Choice of top-spec laptops, high-performance workstation setups, and cutting-edge Plaud devices
  • Annual company offsites, team events
  • Comprehensive benefits to protect employees' financial security, physical health, and long-term growth
  • Medical insurance and WICA coverage for all full-time employees
  • Fulltime
Read More
Arrow Right

Agent Product Manager

Plaud is building the world's most trusted AI work companion for professionals t...
Location
Location
United States , San Francisco
Salary
Salary:
Not provided
plaud.ai Logo
Plaud
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Technical PM & Agent Experience: Proven experience in LLM Agent architectures and API Platform products
  • Proficiency in defining JSON Schemas and OpenAPI specifications
  • Capability to transform abstract business logic into standardized, executable AI tools
  • Data-Driven Evaluation Capabilities: Strong ability to define "Ground Truth" and build quantifiable evaluation frameworks
  • Experience in creating "Golden Datasets" and analyzing production logs
  • Advanced Prompt Engineering & English Proficiency: Professional English proficiency
  • Deep expertise in independently iterating context management strategies and tool descriptions
Job Responsibility
Job Responsibility
  • Architect the Plaud Agent Tooling Ecosystem: Lead the end-to-end strategic planning and execution of the Plaud Intelligence Agent's skill ecosystem
  • Hold full ownership of the ecosystem's lifecycle—driving tool quantity, execution quality, and user penetration rates
  • Define "Ground Truth" for User Scenarios: Transform complex product scenarios into quantifiable, high-bar evaluation metrics
  • Drive collaboration across Product, Evaluation, and Engineering teams to operationalize these standards
  • Push Prompt Performance to SOTA Standards: Spearhead the advanced iteration of tool-related prompt strategies and context engineering infrastructure
  • Fine-tune the AI’s interaction logic and reasoning capabilities
What we offer
What we offer
  • Meaningful Ownership: Employee Stock Ownership Plan (ESOP)
  • High-Impact Environment
  • Comprehensive Health & Retirement Benefits: Top-tier medical, dental, and vision insurance
  • 401(k) retirement plan with company matching
  • Time Off & Workplace Benefits: Unlimited PTO
  • 13 paid holidays
  • 12 weeks of fully paid parental leave
  • Hybrid work model with a minimum of three in-office days per week
  • Access to high-quality office snacks, drinks, and equipment
  • Cutting-Edge AI Tools for Productivity: Access to best-in-class AI tools
  • Fulltime
Read More
Arrow Right

Agent Assist Product Manager

As a Product Manager for the Agent Assist product suite at Cresta, you will play...
Location
Location
United States
Salary
Salary:
130000.00 - 190000.00 USD / Year
cresta.com Logo
Cresta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3-5+ years of experience in product management, with a proven track record of driving product decisions, delivery and customer impact
  • Strong AI/ML systems and process thinking, with the ability to turn ambiguous problems into clear and structured requirements and drive alignment across multiple teams
  • Understand how to evaluate AI/ML solutions and defining the bar for what “good” looks like
  • Experience partnering cross functionally with stakeholders to deliver product launches
  • Analytical and data-driven mindset: skilled at working with product metrics, user feedback, and operational data to inform decisions
  • Ability to create early prototypes using various AI tools (eg: v0, Proto) for fast iterations
  • Excellent written and verbal communication skills
  • you can turn complex problems into clear narratives
  • Hands-on operator who thrives in fast-paced, evolving environments and is comfortable with taking on new technical challenges in an unfamiliar domain
  • Passion for AI, automation, and improving how teams build and deliver products
Job Responsibility
Job Responsibility
  • Work with customers and their proxies (our Customer Success and go-to-market teams) to identify needs and translate them into requirements for your product areas
  • Define and maintain the roadmap and backlogs for your ownership areas, balancing immediate customer needs with longer-term product goals
  • Partner with the engineering team and engage in technical discussions on architectural decisions, model evaluations, system design, and the feasibility of ML/AI capabilities to ensure solutions are scalable, reliable, and secure
  • Collaborate with the Professional Services team to implement playbooks and best practices for customer deployments in your product ownership areas in order to scale
What we offer
What we offer
  • Comprehensive medical, dental, and vision coverage with plans to fit you and your family
  • Flexible PTO to take the time you need, when you need it
  • Paid parental leave for all new parents welcoming a new child
  • Retirement savings plan to help you plan for the future
  • Remote work setup budget to help you create a productive home office
  • Monthly wellness and communication stipend to keep you connected and balanced
  • In-office meal program and commuter benefits provided for onsite employees
  • Equity
  • Fulltime
Read More
Arrow Right

Technical Program Manager II - Copilot Agent & Platform Ecosystem

The Copilot and Platform Ecosystem (CAPE) team at Microsoft empowers organizatio...
Location
Location
Ireland , Dublin
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree AND inclusive years experience in engineering, product/technical program management, data analysis, or product development OR equivalent experience
  • Proven years of experience managing cross-functional and/or cross-team projects
  • Hands-on experience with Microsoft 365 Copilot, Copilot Studio, Power platform Graph Connectors, and Teams extensibility, including grounding strategy choices, content ingestion patterns, and connector-based retrieval behaviors
  • Practical experience integrating enterprise systems such as ServiceNow, Workday, SharePoint, or Dataverse, with an understanding of common authentication, schema, and permission-model challenges
  • Experience creating or operationalizing readiness frameworks, deployment governance models, and scalable delivery playbooks for AI or enterprise platform rollouts
  • Ability to distill customer feedback, telemetry, and escalation patterns into clear product insights, influencing roadmap priorities across engineering and platform teams
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Serve as the primary driver for resolving the most advanced enterprise AI issues, including grounding inconsistencies, retrieval errors, hallucination reduction, connector authentication issues, and policy conflicts
  • Lead technical escalation across engineering, support, partners, and customer teams
  • Reproduce and isolate AI agent behavior issues using logs, snapshots, payload inspection, and evaluation frameworks
  • Lead complex multi-team programs for Copilot extensibility, deployments, and enterprise agent rollouts
  • Build readiness assessments, deployment playbooks, troubleshooting flows, and best-practice frameworks
  • Identify cross-customer themes and influence engineering roadmaps
  • Run predictable operating rhythms—status updates, risk reviews, and exec-readouts
  • Fulltime
Read More
Arrow Right

Staff Engineer

As a Staff Software Engineer, you will play a crucial role in building the next ...
Location
Location
United States , Pleasanton
Salary
Salary:
193000.00 - 242000.00 USD / Year
blackline.com Logo
BlackLine
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS degree in Computer Science, or equivalent experience
  • 6+ years of professional experience in building Web scale highly available multi-tenant SaaS with focus on backend platform, frameworks, RESTful APIs and microservices
  • Expert in one or more object-oriented programming languages - C#, Java or C++
  • A solid computer science foundation including data structures, algorithms, and design patterns, with a proven track record of writing high concurrency, multi-threaded, secure, scalable code
  • Proven experience delivering and extending highly extensible, modular applications with various models of extensibility such as white-box, glass-box, gray-box, and black-box
  • Experience as a technical lead for design, architecture and code review of highly scalable distributed systems and event driven architecture
  • Fluent in SQL, No-SQL, data modeling and transactional flows
  • Superior analytical, problem-solving and system level performance analysis abilities
  • Excellent written and verbal communication skills
  • Adaptable team player with strong collaboration skills and a focus on results and value delivery
Job Responsibility
Job Responsibility
  • Provide technical expertise and leadership in requirements analysis, design, effort estimation, development, testing and delivery of highly scalable and secure distributed backend services
  • Work with product management, architects and other engineering teams to understand stated and unstated needs and turn them into functional and technical requirements
  • Maintain a strong sense of business value and return on investment in planning, design, and communication
  • Lead technical design and architecture discussions and help drive technical decisions while making appropriate trade-offs on technology, functionality, robustness, performance and extensibility
  • Estimate the project scope and timelines, and consistently deliver on those commitments
  • Implement, refine, and enforce software development techniques to ensure that the delivered features meet software integration, performance, security, and maintainability expectations
  • Research, test, benchmark, and evaluate new tools and technologies, and recommend ways to implement them in product development
  • Maintain high standards of software quality and technical excellence within the team by establishing good practices and writing clean, testable, maintainable, and secure code
  • Inspire a forward-thinking team of developers, acting as an agent of change and evangelist for a quality-first culture within the organization
  • Mentor and coach team members, guide them to solutions on complex design issues and do peer code reviews
What we offer
What we offer
  • short-term and long-term incentive programs
  • robust offering of benefit and wellness plans
  • Fulltime
Read More
Arrow Right

Human Resources Business Partner

HR Business Partner – Commercial Functions (Trade Marketing, Commercial Excellen...
Location
Location
Romania , Bucuresti
Salary
Salary:
Not provided
coca-colahellenic.com Logo
Coca-Cola HBC
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum experience of 5 years in HR (and in at least two different areas) or 5 years HR Business Partner
  • Experience in retail or FMCG
  • Stakeholders management (Board , Board -1)
  • High business acumen and understanding
  • Bachelor’s degree
  • Proficient English level
  • General knowledge on the most important HR processes (Talent Acquisition, Talent Management, Engagement, Rewards, HR Services)
  • Strong business awareness combined with functional expertise
  • Multi-tasker that can manage several projects at once under tight deadlines and accommodate rapid changes and ad-hoc requests from the business partners
  • Problem solving with the ability to take responsibility and accountability on the results
Job Responsibility
Job Responsibility
  • Business Partnering: Partner with People & Culture Manager to design and implement the people strategy that support business goals
  • Ensure the entire Employee lifecycle in the assigned organization offering support, information, and guidance to managers: Talent Acquisition, Onboarding, Talent Development, Performance Management etc
  • Execute people plans aligned with business strategy and manage complex projects cross-functionally to ensure the implementation in collaborates with key stakeholders
  • Coach and advise managers, supporting them in implementing the people strategy
  • Acts as a point of contact for line managers and their teams for all related inquiries and grievances
  • Analyze, prepare and provide actionable insights to leadership, as well as initiatives on employee engagement, turnover, and other key performance indicators
  • Build and run the proper routines with line managers and their teams: 1 to 1 routines, team leadership routines
  • Talent Acquisition and Onboarding: Partner with Talent Acquisition to support recruitment strategy, including interviewing and candidate selection for selected roles
  • Supports On-boarding and induction for all positions up to Function Head -1 in collaboration
  • Engagement and Retention: Drive initiatives to improve employee engagement and retention, building accountability to line managers for the action plan
Read More
Arrow Right

Product Manager, Applied Research

Luma's mission is to build unified general intelligence that can generate, under...
Location
Location
United States , SF Bay Area
Salary
Salary:
Not provided
lumalabs.ai Logo
Luma AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Principal or Staff-level product management experience (L6 seniority or above) with a track record of bridging pure research and ambitious product development at an AI lab or applied AI company
  • You are genuinely comfortable in a research environment — you can read evaluations, understand what they mean, ask the right questions, and help researchers see the product implications of their work
  • You've shipped products that originated in research — not just optimized existing products, but taken emergent capabilities and turned them into new product surfaces
  • You can build prototypes yourself — you have the technical depth to work directly with models, write prompts, test hypotheses, and validate ideas before involving a full engineering team
  • You operate with extremely high agency and a strong self-starter mentality — research environments are inherently ambiguous and this role has no existing playbook
  • You have a general manager mindset — you can evaluate research directions not just for technical novelty but for commercial viability and customer impact
  • You bring a mix of large-company and earlier-stage experience — you understand both the pace of a research lab and the urgency of a startup building a commercial business
  • You can influence without authority — researchers don't report to you, but your input on priorities needs to carry weight because it's well-informed and well-reasoned
Job Responsibility
Job Responsibility
  • Own the product strategy for translating Luma's frontier research capabilities into shippable features across Canvas, Agents, and the model platform
  • Work directly within the research team to shape priorities, ensuring model development and evaluation are informed by real enterprise needs in marketing, advertising, and entertainment
  • Establish and operate the feedback loops between research, product, go-to-market, and forward-deployed teams — so that market signals flow into research prioritization and research breakthroughs flow into product planning
  • Interpret evaluation results and research findings to identify which emerging capabilities have the highest product and commercial leverage — and make bets accordingly
  • Build prototypes yourself to validate product ideas before committing engineering resources — using the models directly to test hypotheses about what's possible and what customers will value
  • Define the product implications of research advances in multimodal understanding, generation, tool use, and agent capabilities — translating "the model can now do X" into "customers can now solve Y"
  • Partner with the Agent PM and Enterprise PM to ensure research priorities align with the capabilities that enterprise products and customer deployments actually need
  • Drive the cadence and process for research-to-product handoffs — defining what "ready to ship" means, managing the transition from research prototype to production feature, and owning the quality bar
  • Fulltime
Read More
Arrow Right

Technical Architect - Business Applications

Roku is changing how the world watches TV. Roku is the #1 TV streaming platform ...
Location
Location
United States , Boston
Salary
Salary:
250000.00 - 450000.00 USD / Year
roku.com Logo
Roku
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master's or PhD in Computer Science, Mathematics, Statistics, or a related technical field, or equivalent practical experience.
  • 10+ years of hands-on engineering experience, with a track record of technically leading and delivering large-scale assistant or autonomous systems
  • Strong foundation in machine learning and statistical modeling, including clustering, classification, regression, decision trees, neural networks, SVMs, and anomaly detection, with deep understanding of supervised and unsupervised learning, feature engineering, model evaluation, bias-variance tradeoffs, and offline vs. online metrics.
  • Proven experience building and scaling production ML and AI systems, including LLMs, RAG architectures, embeddings, retrieval-based systems, and multi-agent or agent-based system design.
  • Hands-on experience designing, training, tuning, and deploying models for ranking, prediction, recommendation, forecasting, classification, or NLP use cases.
  • Experience with cloud platforms (AWS, Azure, Google Cloud), microservices, containerization (Docker, Kubernetes), and DevOps
  • Demonstrated technical leadership, including setting direction, influencing across teams, and elevating the work of other engineers.
  • Experience in advertising, marketplaces, e-commerce, travel, or similar data-rich, decision-driven platforms is a strong plus.
Job Responsibility
Job Responsibility
  • Develop and oversee technical strategy for Roku's platform that support diverse business applications.
  • Drive the AI-native transformation of Roku's advertising business operations, identifying high-leverage opportunities and influencing roadmap, architecture, and system design across multiple teams.
  • Start from the needs of internal engineers and business operators, bringing a creative and strategic approach to simplifying systems and introducing practical, high-impact solutions.
  • Work closely with advertising business stakeholders to identify needs, surface high-impact opportunities, and translate operational pain points into ML and agentic AI solutions.
  • Design and ship production ML and agentic systems end-to-end, including recommendation, personalization, ranking, forecasting, anomaly detection, and multi-agent or agent-to-agent workflows, embedded directly within product and operational systems where they drive decisions.
  • Apply GenAI and LLMs to create measurable value, building and scaling Retrieval Augmented Generation (RAG) pipelines, prompt orchestration, evaluation frameworks, and guardrails, with a focus on reliability and outcomes over hype.
  • Translate ambiguous business problems into production solutions, partnering with engineers, data scientists, product managers, and business teams to ground ML and AI work in real operational impact.
  • Build and operationalize evaluation loops covering precision and recall, calibration, drift detection, and human-in-the-loop systems, and define the dashboards and Service Level Objectives (SLO) that tie model performance to business outcomes.
  • Provide technical leadership across the broader ML and AI surface area, raising the bar for engineering rigor, mentoring engineers, and shaping how the organization thinks about AI-native systems.
What we offer
What we offer
  • Health insurance
  • Equity awards
  • Life insurance
  • Disability benefits
  • Parental leave
  • Wellness benefits
  • Paid time off
  • Global access to mental health and financial wellness support and resources
  • Local benefits including healthcare (medical, dental, and vision), life, accident, disability, commuter, and retirement options (401(k)/pension)
  • Fulltime
Read More
Arrow Right