This list contains only the countries for which job offers have been published in the selected language (e.g., in the French version, only job offers written in French are displayed, and in the English version, only those in English).
Senior Product Manager – Online Metrics & Experimentation. Microsoft Bing and Copilot Search power AI-driven consumer experiences used by hundreds of millions of people worldwide. As these experiences evolve from classic search to AI-integrated experiences, defining how we measure success becomes one of the most critical product problems we face. The Bing Metrics & Experimentation team sits at the center of this transformation. We build the metrics, experimentation frameworks, and decision systems that guide product investment, ship decisions, and long-term strategy across Bing and Copilot Search. We are looking for a Senior Product Manager to lead online metrics and experimentation strategy for Bing, with scope spanning metric definition, experimentation, and the telemetry foundations that make high-quality measurement possible.
Job Responsibility:
Own the vision for online metrics, experimentation, and telemetry
Define and evolve Bing’s core online success metrics (e.g., session success, satisfaction, utility-based measures) for AI-powered search and Copilot experiences
Own the product direction for experimentation and metrics systems, including scorecards, score consolidation, triage workflows, and decision frameworks
Inform and influence telemetry strategy: what we log, how we instrument user interactions, and how data flows from product experience → logs → metrics → decisions
Lead A/B experimentation at scale
Drive the strategy for Bing’s A/B experimentation ecosystem, from experiment design to decision-ready scorecards
Help teams navigate tradeoffs across short-term gains, quality, user trust, and long-term product health
Ensure experimentation and metrics evolve to handle complex AI scenarios (multi-intent sessions, long-running interactions, generative outputs)
Partner deeply with engineering and data science
Work closely with engineering to ensure logging and instrumentation are designed to support accurate, debuggable, and scalable metrics
Partner with data science to turn raw telemetry into trusted signals, causal insights, and executive-ready narratives
Identify gaps in today’s instrumentation and metrics ecosystem and drive investments to close them
Influence product decisions and senior leadership
Act as a thought partner to senior leaders across Bing and Copilot Search on how to interpret experiments, quality signals, and tradeoffs
Represent the metrics and experimentation perspective in executive reviews, planning discussions, and ship decisions
Set the bar for rigor, clarity, and storytelling in data-driven decision-making
Requirements:
Bachelor's Degree AND 5+ years experience in product/service/program management or software development OR equivalent experience
Deep experience with A/B experimentation, online metrics, and causal thinking
Proven ability to influence complex, cross-org decisions and communicate data clearly to senior stakeholders
Solid collaboration experience with engineering and data science teams on technically sophisticated products
Solid intuition for measuring quality in AI/ML-driven products
Familiarity with telemetry design, logging strategies, and data pipelines (product ownership perspective)
Background in search, recommendations, AI assistants, or large-scale consumer platforms
Nice to have:
Bachelor's Degree AND 8+ years experience in product/service/program management or software development OR equivalent experience
2+ years experience taking a product, feature, or experience to market (e.g., design, addressing product market fit, and launch, internal tool/framework)
4+ years experience improving product metrics for a product, feature, or experience in a market (e.g., growing customer base, expanding customer usage, avoiding customer churn)
4+ years experience disrupting a market for a product, feature, or experience (e.g., competitive disruption, taking the place of an established competing product)