CrawlJobs Logo
Plaud Logo Plaud · IT - Software Development

AI Evaluation Product Manager

Singapore · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

Plaud is building the future of AI-native productivity — where meetings, conversations, and knowledge become an extension of your memory. As the first dedicated owner of Memory Evaluation, you will shape how AI understands, stores and recalls human knowledge for millions of users worldwide.

Job Responsibility

  • Define how 'AI memory quality' should be measured for the industry
  • Turn product goals and real user scenarios into measurable evaluation criteria, evaluation frameworks and reproducible test cases
  • Build the global memory evaluation system from the ground up
  • Create evaluation standards, test pipelines, and datasets across markets (US, EU, Japan, China), working with real multilingual and multimodal data
  • Run continuous quality evaluations across Summary, Ask Plaud and other memory-enabled experiences
  • Become the source of truth for ‘is memory working?’
  • Benchmark against global AI meeting and knowledge products (Otter, Notion AI, NotebookLM, etc.) to extract best practices in memory accuracy, retrieval reliability, and user trust — and convert these insights into scientific evaluation methodology

Requirements

  • Master's degree or above, preferably in Computer Science, Data Science, AI, or related fields
  • 3+ years of product experience, with at least 1 year working on AI or LLM-related products or features
  • Solid analytical skills and familiarity with data processing (SQL/Python preferred)
  • Ability to communicate clearly in English and collaborate across product, engineering, and operations
  • Strong ownership, structured problem solving, and willingness to learn quickly
  • Understanding of English-speaking user scenarios, product expectations, or cultural differences

Nice to have

  • Hands-on experience designing test plans, evaluation standards, or datasets for LLM or AI features
  • Familiarity with evaluation of memory-related behaviors (extraction, retrieval, contextual usage), or prior work on summary/Ask AI products
  • Experience reviewing real user data and translating findings into actionable product improvements

What we offer

  • Market-competitive compensation
  • Global exposure
  • Vibrant, creativity-fueled work atmosphere
Plaud - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

AI Evaluation Product Manager

8 matching positions

Principal Product Manager, AI Product

We are looking for a Principal Product Manager, AI Products to help define and d...
Location
Location
United States
Salary
Salary:
Not provided
themuse.com Logo
The Muse
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 12+ years in enterprise SaaS, platform products, AI-enabled products, data products, or similarly complex technical environments
  • bonus if experience includes healthcare environments
  • Experience building and scaling enterprise-grade AI, machine learning, or generative AI products
  • Experience delivering products powered by LLMs, agentic systems, retrieval-augmented generation (RAG), intelligent automation, predictive analytics, or similar AI technologies
  • Proven track record shipping products used by enterprise customers at scale
  • Experience defining and delivering products tied to measurable business outcomes and revenue impact
  • Experience owning products with measurable commercial outcomes, including adoption, retention, growth, or revenue objectives
  • Demonstrable understanding of how product decisions connect to business performance and customer value realization
  • Experience partnering closely with Engineering and GTM organizations to successfully bring products to market
  • Strong technical fluency across modern AI systems, including generative AI, LLMs, retrieval architectures (RAG), intelligent automation, model evaluation, and AI governance
Job Responsibility
Job Responsibility
  • Own the end-to-end lifecycle of AI-powered product capabilities from concept through customer adoption and measurable business impact
  • Deliver enterprise-grade generative AI products and workflow automation capabilities into production environments at scale
  • Work directly with customers to understand operational challenges, validate product direction, and ensure solutions address real-world healthcare workflows
  • Translate advances in AI, LLMs, agentic systems, retrieval architectures, and analytics into practical customer solutions
  • Partner closely with Engineering, Data Science, Customer Success, and GTM teams to ensure products are successfully adopted and deliver measurable value
  • Drive roadmap decisions using customer needs, technical realities, business objectives, and market opportunities
  • Own product success metrics tied to customer adoption, revenue growth, operational efficiency, and business outcomes
  • Act as the accountable owner for shipping high-quality products on a predictable cadence
  • Mentor and elevate product management practices across the broader Product organization
What we offer
What we offer
  • Pet Insurance
  • Health Insurance
  • Dental Insurance
  • Vision Insurance
  • FSA
  • HSA
  • HSA With Employer Contribution
  • Life Insurance
  • Short-Term Disability
  • Long-Term Disability
  • Fulltime
Read More
Arrow Right

Principal Product Manager - AI Governance (Azure AI Foundry)

AI is transforming how organizations operate, and with it comes a new frontier o...
Location
Location
United States , Redmond
Salary
Salary:
139900.00 - 274800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s Degree AND 8+ years in product management, program management, or technical leadership roles OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
  • Deep familiarity with AI/ML systems, model governance, or enterprise compliance frameworks
  • Proven experience driving cross-functional initiatives with engineering, security, and policy teams
  • Excellent communication skills with demonstrated ability to influence across organizational boundaries
  • 10+ years of product management or applied AI experience, ideally in enterprise cloud or responsible AI domains
  • Knowledge of AI governance frameworks (EU AI Act, NIST AI RMF, ISO 42001, SOC 2, etc.)
  • Experience building or integrating security, compliance, or observability products
  • Familiarity with agentic AI systems and associated risk classes (e.g., sensitive data leakage, prohibited actions, task drift, jailbreaks)
Job Responsibility
Job Responsibility
  • Lead the AI Governance product area within Azure AI Foundry — defining the long-term vision, strategy, and roadmap for policy management, compliance automation, and regulatory readiness
  • Design and deliver core governance experiences, including agent-level policies, data sensitivity signals, prohibited action controls, and AI system compliance dashboards
  • Integrate Foundry governance with Microsoft’s broader security and compliance stack — Entra (identity and A2A policies), Purview (data classification and DLP), and Defender (threat insights)
  • Translate emerging AI regulations (EU AI Act, ISO 42001, NIST AI RMF) into actionable platform capabilities and customer experiences
  • Partner with Responsible AI researchers and engineering teams to operationalize ethical principles into measurable safeguards and evaluators
  • Collaborate with customers, industry bodies, and policymakers to help shape standards for trustworthy AI deployment
  • Establish and track success metrics (governance coverage, compliance posture, customer adoption), ensuring measurable impact and clarity across engineering and partner orgs
  • Represent the product in executive and customer forums, evangelizing Microsoft’s approach to responsible, governed AI systems
  • Fulltime
Read More
Arrow Right

Senior Engineering Manager, AI Product (Thunderbolt)

Thunderbolt is an open-source, cross-platform AI product initiative from the tea...
Location
Location
Canada; United States
Salary
Salary:
190000.00 - 240000.00 USD; CAD / Year
mozilla.org Logo
Mozilla
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 15+ years of professional software development experience, including significant experience in senior engineering leadership or complex product delivery roles
  • 5+ years of experience managing technical teams, including senior engineers or senior individual contributors
  • A track record of leading a team that shipped polished, top-tier products held to a genuine standard of craft
  • Strong ability to translate ambiguous product and technical direction into clear plans, ownership models, and delivery momentum
  • Technical fluency with AI-enabled products, including the ability to evaluate tradeoffs across areas such as LLM applications, RAG workflows, agents, enterprise search, automation, on-device AI, or AI evaluation
  • Demonstrated experience delivering complex applications across multiple platforms, such as desktop, web, and mobile, using technologies such as Tauri, Electron, React Native, or comparable frameworks
  • Experience with enterprise software, regulated environments, or similarly complex technical products, including security, privacy, compliance readiness, administration, integrations, and customer-controlled deployment models
  • Practical experience with local-first, offline-capable, self-hosted, hybrid, on-premise, air-gapped, or other customer-controlled product architectures, including tradeoffs related to data storage, sync, privacy, reliability, and user experience
  • Experience improving engineering practices related to technical planning, code review, release readiness, testing, documentation, CI/CD, maintainability, or technical debt management
  • Excellent communication skills around technical prioritization, release risk, ownership, quality expectations, and tradeoffs
Job Responsibility
Job Responsibility
  • Lead engineering execution by clarifying priorities, ownership, timelines, risks, dependencies, and quality expectations
  • Manage engineers and contractors, including coaching, feedback, prioritization, performance support, and team development
  • Stay close to the codebase by contributing code, reviewing technical work, evaluating tradeoffs, and helping the team maintain momentum and quality
  • Partner with technical and product leadership to turn product direction, AI strategy, user experience goals, and technical priorities into coordinated engineering plans
  • Establish planning, review, release, and follow-through practices that improve predictability and reduce bottlenecks
  • Maintain high standards for code quality, testing, documentation, reliability, security awareness, maintainability, and release readiness
  • Build strong product and architectural context across AI-enabled workflows, integrations, local-first or customer-controlled architectures, and release requirements
  • Partner across product, design, enterprise-facing teams, and technical leadership to balance speed, user experience, privacy, quality, enterprise readiness, and long-term sustainability
  • Lead hiring, onboarding, and resource planning as the Thunderbolt engineering team grows
  • Help Thunderbolt scale from an incubation-stage product into a reliable, enterprise-ready open-source AI product
What we offer
What we offer
  • Fully remote work & schedule flexibility
  • Company-provided laptop
  • Annual bonus program
  • Monthly remote work stipend
  • Annual professional development stipend
  • Access to Coursera learning platform
  • Industry conferences
  • Company all-hands and team gatherings
  • 24 days PTO per year (prorated)
  • Your birthday
  • Fulltime
Read More
Arrow Right
New

Senior Ai Product Manager

Amazon Global Selling (AGS) has been helping individuals and businesses increase...
Location
Location
China , Shanghai
Salary
Salary:
Not provided
amazon.de Logo
Amazon Pforzheim GmbH
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Outstanding verbal and written communication skills in Chinese and English
  • Experience owning/driving roadmap strategy and definition
  • Experience with feature delivery and tradeoffs of a product
  • Experience as a product manager or owner
  • Experience owning technology products
Job Responsibility
Job Responsibility
  • Define AI product vision and strategy across Global Selling seller segments and lifecycle stages
  • identify high-impact opportunities to apply AI/ML to empower business teams
  • Own the end-to-end AI product lifecycle
  • Develop and communicate product requirements including model objectives, data specifications, evaluation metrics, and responsible AI guardrails
  • Lead cross-functional and cross-regional collaboration with scientists, engineering, and data teams to build, evaluate, and ship AI-powered products
  • Drive measurable business impact post-launch through continuous monitoring, data-driven iteration, and feedback loops
  • Manage stakeholder and leadership communications across multiple geographies
  • Fulltime
Read More
Arrow Right

Senior Product Manager - AI Content Creation

As Senior Product Manager for AI Content Creation, you will lead the evolution o...
Location
Location
Germany , Berlin
Salary
Salary:
Not provided
babbel.com Logo
Babbel
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A demonstrated track record of owning complex digital products end-to-end, from discovery through delivery
  • Strong product judgment and the ability to operate in ambiguous, fast-evolving problem spaces
  • Solid understanding of AI-powered systems and how to productize them. Experience building AI products is a plus
  • Experience working closely with engineers and data teams
  • A high bar for quality and a strong focus on user outcomes
  • Excellent communication and stakeholder alignment skills
  • Able to drive progress in ambiguous environments and communicate decisions clearly to technical and non-technical stakeholders
  • Note that our company's operating language is English, so you will need to be able to work in English.
Job Responsibility
Job Responsibility
  • Own the product strategy for AI-driven content creation, from discovery to execution
  • Create, improve and scale AI generation workflows, including prompting, evaluation, and quality loops
  • Enable faster creation of new languages, levels, and learning experiences, including more complex language systems
  • Balance automation and quality, evolving human-in-the-loop processes where needed
  • Define success metrics and use learner data and feedback to guide decisions
  • Collaborate across Product, Engineering, Data, Design and Learning Content to deliver reliable, learner-ready solutions
What we offer
What we offer
  • Flexible working models
  • Learning budgets
  • Free language learning
  • An inclusive, supportive environment that values curiosity and growth
  • Fulltime
Read More
Arrow Right

Senior Product Manager, AI & ML Platform

Reporting to the Senior Director of Product, Customer Value, you will be instrum...
Location
Location
United States , New York; San Francisco
Salary
Salary:
150000.00 - 202400.00 USD / Year
springhealth.com Logo
Spring Health
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years of Product Management experience, including ownership of highly technical platforms, infrastructure, or developer-facing products
  • Demonstrated experience partnering deeply with Engineering to ship scalable, reliable systems in production environments
  • Strong understanding of AI/ML systems and workflows (e.g., model lifecycle, evaluation, deployment, observability), with the ability to translate complex technical concepts into clear product direction
  • Track record of driving roadmap clarity in ambiguous, fast-moving spaces and delivering measurable outcomes
  • Experience running continuous discovery with internal users and translating feedback into high-impact platform improvements
  • Ability to define clear success metrics, service standards, and rollout strategies that drive adoption
  • Excellent cross-functional communication skills, with the ability to influence senior technical stakeholders without formal authority
  • Prior machine learning engineering experience is preferred but not required
Job Responsibility
Job Responsibility
  • Shape and drive the AI/ML platform strategy and roadmap
  • Lead continuous discovery with feature teams
  • Align platform investments to business impact
  • Define platform solutions in close partnership with Engineering leadership
What we offer
What we offer
  • Health, Dental, Vision benefits start on your first day
  • Access to One Medical accounts
  • HSA and FSA plans with Spring contributing up to $1K for HSAs
  • Employer sponsored 401(k) match of up to 2%
  • Yearly allotment of no cost visits to Spring Health network
  • Competitive paid time off policies including vacation, sick leave and company holidays
  • Parental leave of 18 weeks for birthing parents and 16 weeks for non-birthing parents
  • Access to Noom
  • Access to fertility care support through Carrot
  • $4,000 reimbursement for related fertility expenses
  • Fulltime
Read More
Arrow Right

Principal AI Product Manager, Technical

Principal Innovation Product Manager, Technical is a visionary, strategist, anal...
Location
Location
United States , Bellevue
Salary
Salary:
139700.00 - 252000.00 USD / Year
https://www.t-mobile.com Logo
T-Mobile
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science, Engineering, IT or equivalent experience
  • 8+ years’ experience in hands on technical role writing production code, solution engineering, or technical archtecture in large scale company / appplication / product environment
  • More than 10 years Relevant Product Management experience in an agile software product development environment
  • At least 18 years of age
  • Legally authorized to work in the United States
Job Responsibility
Job Responsibility
  • Vision & Strategy for AI Products
  • Owns strategy, roadmap, and lifecycle for enterprise AI products and reusable AI capabilities serving multiple customer journeys, business units, or platforms
  • Accountable for measurable business outcomes including revenue growth, cost reduction, adoption, retention, automation impact, and customer experience improvement
  • Drives multi-year product investment strategy, including business cases, funding requests, and prioritization of AI initiatives across a portfolio of opportunities
  • Own the product lifecycle for complex technical solutions, ensuring alignment with business objectives and customer needs
  • Conducts deep customer research and market analysis to identify breakthrough AI opportunities
  • Defines success metrics and tracks progress against ambitious growth targets
  • Anticipates AI trends and competitive threats, incorporating insights into product strategy
  • Conducts analysis of quantitative and qualitative data to identify product innovation opportunities or root cause of issues, and assess opportunity size and impact
  • May work with data scientists to answer complex questions or identify meaningful insights from data
What we offer
What we offer
  • competitive base salary and compensation package
  • annual stock grant
  • employee stock purchase plan
  • 401(k)
  • free, year-round money coaches
  • medical, dental and vision insurance
  • flexible spending account
  • paid time off
  • up to 12 paid holidays
  • paid parental and family leave
  • Fulltime
Read More
Arrow Right

AI product manager

At HubX we build mobile apps, used and loved by millions all around the world. W...
Location
Location
Turkey , Istanbul
Salary
Salary:
Not provided
hubx.co Logo
HubX
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS in Software/Computer/Electronics/Industrial Engineering or related disciplines
  • Experience using and applying Large Language Models. You should know how to prompt-engineer them, how to evaluate them, and understand how they are tuned
  • Hands-on experience building or using LLM-powered agentic solutions
  • A healthy blend of general technical knowledge with a customer-focused mindset and empathy
  • Great communication skills and background knowledge for customers, stakeholders, and technical team
  • General knowledge of the ins and outs of development, UI/UX design and flows, agile framework, software development, scrum approach, and IT infrastructure
  • A desire to work in a fast-paced, results-driven environment
Job Responsibility
Job Responsibility
  • Creating business analysis documents, test cases, mockups, and prototypes to assist with development, including leveraging AI tools (e.g., LLMs) to accelerate documentation, generate scenarios, and validate edge cases
  • Analyzing metrics to understand customer and product performance, incorporating AI-driven insights (e.g., model outputs, user interactions with AI features, prompt success rates) to guide product decisions
  • Staying in sync with team members using agile approaches, while adapting workflows to support rapid experimentation and iteration of AI-powered features
  • Attending grooming sessions with the development team to clarify requirements, including defining AI-specific behaviors (e.g., prompts, expected outputs, fallback scenarios) and ensuring the team understands both product and model-related objectives
  • Partnering closely with technical teams by bringing clear, structured requirements to engineers, and actively contributing to discussions around LLM capabilities, constraints, and implementation trade-offs
  • Running qualitative and quantitative research to define ambiguous problems, including evaluating where AI/LLMs can provide value and validating hypotheses through data and experimentation
  • Designing, testing, and iterating on prompt strategies
  • understanding prompt engineering techniques and how they impact model performance and user experience
  • Evaluating LLM outputs using structured approaches (e.g., accuracy, consistency, safety, and relevance), and working with teams to improve model performance through fine-tuning or prompt adjustments
  • Collaborating on or contributing to the development of LLM-powered or agentic solutions, understanding how multi-step AI workflows operate and how they integrate into the product experience
What we offer
What we offer
  • Huge impact
  • Ownership, opportunity to take responsibility from day one
  • A competitive compensation package
  • A brand new Macbook and welcome kit
  • Private Medical Insurance & HPV Vaccine & Critical Women’s Health Coverage
  • Gym Reimbursement
  • A unique and top-notch office
  • Unlimited coffee from XPresso
  • Limitless Snacks & breakfast
  • Continuous education
  • Fulltime
Read More
Arrow Right