CrawlJobs Logo

Principal Product Manager - Observability & Evaluations

https://www.microsoft.com/ Logo

Microsoft Corporation

Location Icon

Location:
United States , New York

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

163000.00 - 296400.00 USD / Year

Job Description:

Microsoft Foundry is the platform that empowers developers to build, deploy, and scale AIpowered applications and agents across Microsoft’s ecosystem. As part of the CoreAI organization, Foundry brings together Azure AI, orchestration patterns, agent tooling, data connectivity, and the emerging stack that transforms how applications are created. As a Principal Product Manager on the Foundry team, you will shape the strategy and roadmap for foundational developer and AI capabilities that serve millions of developers globally. You will partner closely with engineering, design, and research to define experiences that help teams modernize legacy applications, build agentic workflows, and leverage state-of-the-art LLMs and SLMs. This role requires deep technical curiosity, strong customer empathy, and the ability to drive clarity and alignment across complex engineering environments. You will contribute to Microsoft’s mission to empower every developer and organization to do more with AI.

Job Responsibility:

  • Define and evolve the longterm vision for the Foundry platform, including agent frameworks, orchestration capabilities, and developer tooling used across CoreAI products
  • Translate complex enterprise and developer requirements into clear product roadmaps and prioritized investment plans
  • Identify market opportunities and validate product direction through research, experimentation, and customer insights
  • Engage directly with strategic AI customers to deeply understand their application modernizations, agent use cases, and infrastructure requirements
  • Synthesize customer feedback into actionable insights that inform backlog prioritization and ongoing product improvements
  • Collaborate with engineering teams to deliver scalable, reliable, and secure platform features
  • Lead execution of multiquarter initiatives, ensuring clarity on acceptance criteria, technical dependencies, and timelines
  • Partner with design, research, marketing, and PM peers across Azure AI, GitHub, and broader CoreAI to craft worldclass developer experiences
  • Drive capabilities for agenttoagent interactions, skill orchestration, prompt management, AI workflow tooling, and hybrid cloud/local inference patterns
  • Influence the platform infrastructure underpinning Foundry, ensuring that developer scenarios—from app migration to AIfirst application creation—are firstclass and intuitive
  • Measure product performance, collect feedback, and iterate quickly to optimize reliability, usability, and developer satisfaction
  • Maintain clear, executiveready insights on product performance and strategic direction

Requirements:

  • Bachelor’s Degree in Computer Science, Engineering, or related technical discipline AND 8+ years of product management, program management, or software development experience, OR equivalent experience
  • Proven experience leading technical product areas from concept through launch
  • Experience building developer platforms, cloud services, AI orchestration frameworks, or agentcentric architectures
  • Demonstrated ability to partner across engineering, design, research, and business teams
  • Ability to analyze complex technical and business requirements and drive alignment across stakeholders
  • Ability to pass Microsoft Cloud Background Check requirements
  • Background working with enterprise customers on technical adoption or migration efforts
  • Experience driving product strategy in global or complex markets
  • Strong analytical and communication skills, including the ability to synthesize complex technical concepts for executive audiences

Additional Information:

Job Posted:
February 20, 2026

Employment Type:
Fulltime
Work Type:
Hybrid work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Principal Product Manager - Observability & Evaluations

AI Product Manager

We’re scaling AI and machine learning across our products, devices, and operatio...
Location
Location
United States , Boston
Salary
Salary:
121300.00 - 177900.00 USD / Year
simplisafe.com Logo
SimpliSafe
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of product management experience, including significant ownership of AI/ML or data-intensive products
  • Clear track record of shipping production ML systems (not just integrating third-party AI APIs), in close partnership with data science, ML engineering, and MLOps
  • Principal-level impact: leading cross-team initiatives, shaping strategy, and influencing senior stakeholders
  • Strong understanding of core ML concepts and lifecycle: data, labeling, training/validation, evaluation metrics, deployment, monitoring, and retraining
  • ML experience with at least one of following: computer vision or sensor data, LLM-powered applications (prompting, RAG, fine-tuning, evaluation) and/or hardware or edge products (e.g., on-device models, connectivity/latency trade-offs)
  • Familiarity with modern ML infrastructure (cloud platforms, model serving, CI/CD for ML, monitoring/alerting)
  • Comfortable going deep into data, metrics, and model behavior—not just the UX layer
  • Excellent communicator who can make complex AI topics clear to diverse audiences
  • Strong alignment with our values: customer-obsessed, low ego, highly collaborative, comfortable with ambiguity, and biased toward learning and iteration.
Job Responsibility
Job Responsibility
  • Define and communicate the multi-year roadmap for key AI/ML capabilities across SimpliSafe
  • Identify and prioritize AI opportunities where models and data can materially improve safety, customer experience, or efficiency—on both devices and cloud services
  • Make build-vs-buy decisions for AI capabilities in partnership with data science and engineering
  • Partner with data scientists, ML engineers, and MLOps to design and deliver end-to-end ML solutions—from problem framing through data, training, evaluation, deployment, and monitoring
  • Work with hardware and embedded teams to shape edge AI/ML experiences (e.g., on-device detection, low-latency decisions, bandwidth-aware designs)
  • Define model-level requirements (metrics, latency, cost, guardrails) and connect them to business outcomes (e.g., false alarm reduction, detection accuracy, handle time, CSAT)
  • Translate product needs into requirements for ML platform capabilities (model serving, observability, experiment tracking, human-in-the-loop tools)
  • Lead product direction for LLM and multimodal use cases (e.g., text, vision, sensor data)
  • Decide when to use prompt engineering, RAG, fine-tuning, or traditional ML—and how to evaluate quality, safety, and hallucinations
  • Design workflows that incorporate human review and escalation where needed
What we offer
What we offer
  • A mission- and values-driven culture and a safe, inclusive environment where you can build, grow, and thrive
  • A comprehensive total rewards package that supports your wellness and provides security for SimpliSafers and their families
  • Free SimpliSafe system and professional monitoring for your home
  • Employee Resource Groups (ERGs) that bring people together, give opportunities to network, mentor and develop, and advocate for change
  • Participation in our annual bonus program, equity, and other forms of compensation, in addition to a full range of medical, retirement, and lifestyle benefits.
  • Fulltime
Read More
Arrow Right

Principal Product Manager, AI

LogicMonitor® is the AI-first hybrid observability platform powering the next ge...
Location
Location
India , Pune
Salary
Salary:
Not provided
logicmonitor.com Logo
LogicMonitor
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8 to 9 years of product management experience, ideally in B2B software
  • Strong hands on fluency with AI powered products and how modern AI systems work
  • Proven experience owning products end to end in ambiguous environments
  • Strong customer instincts and comfort being customer facing
  • Ability to think strategically while executing tactically
  • Clear communicator who can align teams without relying on authority
  • Bias for action and comfort with scrappy execution
Job Responsibility
Job Responsibility
  • End to end ownership of major Edwin AI product areas from discovery to delivery to adoption
  • Deep customer engagement to uncover real problems and validate solutions
  • Clear product strategy and roadmap grounded in customer value, data, and business outcomes
  • Hands on collaboration with engineering, design, GTM, and applied AI teams
  • Product decisions across AI capabilities including agents, workflows, and intelligent experiences
  • Success metrics like adoption, usage, customer impact, and business outcomes
  • Spend real time with customers through calls, demos, feedback sessions, and follow ups
  • Translate messy customer problems into crisp product bets and shipped features
  • Work closely with engineers on scope, tradeoffs, and delivery details
  • Get hands on with AI concepts like prompting, evaluations, agent behavior, and failure modes
Read More
Arrow Right

Principal Product Manager

AgentX Support is Workato’s flagship AI-native engine designed to transform cust...
Location
Location
United States , Palo Alto
Salary
Salary:
Not provided
workato.com Logo
Workato
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of product management experience in enterprise software
  • Demonstrated success building 0-to-1 products or defining new market categories
  • Track record shipping ML/AI products to production (not just prototyping)
  • Experience with customer support platforms or CRM systems
  • Experience with real-time systems or high-throughput environments
  • Strong systems thinking and comfort designing across APIs, data, models, and workflows
  • Sufficient technical depth to partner effectively with engineering and AI research teams
  • Ability to operate effectively in ambiguous, fast-moving environments
  • Exceptional written and verbal communication skills, with executive-level presence
Job Responsibility
Job Responsibility
  • Define the vision and long-term strategy for AI-native, agentic customer support
  • Establish the product narrative and positioning that differentiates Workato in the customer service market
  • Represent Workato’s point of view with customers, partners, and industry stakeholders
  • Partner with design and marketing to translate product capabilities into clear customer value
  • Define how AI agents interpret intent, reason through complex customer issues, and determine when to act autonomously
  • Design orchestration frameworks that connect large language models with enterprise systems of record
  • Lead the development of workflows that enable AI agents to execute actions such as refunds, order modifications, and escalations
  • Ensure appropriate controls for safety, observability, compliance, and human-in-the-loop oversight at enterprise scale
  • Define the next-generation support agent workspace augmented by real-time AI guidance
  • Deliver systems that surface relevant context, recommendations, and next-best actions to human support agents during live customer interactions
What we offer
What we offer
  • Vibrant and dynamic work environment
  • Multitude of benefits they can enjoy inside and outside of their work lives
Read More
Arrow Right

Architectural Project Designer - Interior Specialist

Location
Location
United States
Salary
Salary:
Not provided
adcmaui.com Logo
Architectural Design & Construction, Inc.
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Learn to create realistic project schedule: estimate time frame for design development, structural design, construction documents
  • Manage client expectations: schedule, budget, and project scope
  • Establish due dates for all finish selections for client
  • Educate client concerning the process and importance of obtaining selections on schedule
  • Obtain selections on schedule and before construction starts
  • Record all selections
  • Obtain client approval in writing before construction starts
  • Provide client with regular project status reports throughout the design phase
  • Coordinate with consultants to meet project requirements
  • Meet project schedule and budget for design services to maintain profitability
Job Responsibility
Job Responsibility
  • Reports to the Principal/Construction Project Manager
  • Properly communicate information regarding selections to the Principal/Construction Project Manager, Project Architect, Project Manager, and client as appropriate and in a timely manner
  • Prior to all client meetings, share all information that will be presented to a client in the meeting with the Project Team members
  • Promote, discuss and offer ADC’s Interior Design services when clients have not selected an Interior Designer for their project
  • Maintain and enhance client relationship with ADC
  • Walk client through the selections process
  • Ensure that they understand the phases of construction
  • Notify suppliers when ADC clients will visit them, if possible
  • Provide Construction Manager with selections prior to client sign-off on Final Budget Estimate
  • Assist the Construction Manager putting the selections out to bid with trade contractors and suppliers
Read More
Arrow Right

Principal Product Manager - AI Governance (Azure AI Foundry)

AI is transforming how organizations operate, and with it comes a new frontier o...
Location
Location
United States , Redmond
Salary
Salary:
139900.00 - 274800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s Degree AND 8+ years in product management, program management, or technical leadership roles OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
  • Deep familiarity with AI/ML systems, model governance, or enterprise compliance frameworks
  • Proven experience driving cross-functional initiatives with engineering, security, and policy teams
  • Excellent communication skills with demonstrated ability to influence across organizational boundaries
  • 10+ years of product management or applied AI experience, ideally in enterprise cloud or responsible AI domains
  • Knowledge of AI governance frameworks (EU AI Act, NIST AI RMF, ISO 42001, SOC 2, etc.)
  • Experience building or integrating security, compliance, or observability products
  • Familiarity with agentic AI systems and associated risk classes (e.g., sensitive data leakage, prohibited actions, task drift, jailbreaks)
Job Responsibility
Job Responsibility
  • Lead the AI Governance product area within Azure AI Foundry — defining the long-term vision, strategy, and roadmap for policy management, compliance automation, and regulatory readiness
  • Design and deliver core governance experiences, including agent-level policies, data sensitivity signals, prohibited action controls, and AI system compliance dashboards
  • Integrate Foundry governance with Microsoft’s broader security and compliance stack — Entra (identity and A2A policies), Purview (data classification and DLP), and Defender (threat insights)
  • Translate emerging AI regulations (EU AI Act, ISO 42001, NIST AI RMF) into actionable platform capabilities and customer experiences
  • Partner with Responsible AI researchers and engineering teams to operationalize ethical principles into measurable safeguards and evaluators
  • Collaborate with customers, industry bodies, and policymakers to help shape standards for trustworthy AI deployment
  • Establish and track success metrics (governance coverage, compliance posture, customer adoption), ensuring measurable impact and clarity across engineering and partner orgs
  • Represent the product in executive and customer forums, evangelizing Microsoft’s approach to responsible, governed AI systems
  • Fulltime
Read More
Arrow Right

Principal Software Engineer

Are you looking for an opportunity to work with the latest Azure offerings and p...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10–12+ years of experience in software engineering, with significant experience building scalable backend or distributed systems
  • Strong programming expertise in one or more languages such as Python, Go, Java, or C#, with experience designing production-grade services and APIs
  • Experience building AI-powered applications, including integrating LLMs, implementing agent or Copilot workflows, and orchestrating multi-step AI interactions
  • Hands-on experience with LLM application frameworks and orchestration tools such as Semantic Kernel, LangChain, or similar agent frameworks
  • Familiarity with retrieval-augmented generation (RAG) architectures, vector databases, embeddings, and semantic search systems
  • Experience evaluating and improving model performance through prompt design, evaluation frameworks, fine-tuning, or feedback loops
  • Solid understanding of distributed systems concepts including scalability, reliability, observability, caching, and asynchronous processing
  • Experience deploying and operating AI workloads in cloud environments (preferably Azure), including containerized services and GPU-enabled infrastructure
  • Understanding of Responsible AI practices, including model governance, safety, privacy, and evaluation of AI behaviour in production systems
  • Ability to work across product, research, and engineering teams to translate product scenarios into scalable AI system architectures
Job Responsibility
Job Responsibility
  • Design, build, and operate scalable AI systems that power intelligent product experiences, including Copilot and agent-driven workflows
  • Architect and implement backend services that support multi-step AI interactions, including orchestration pipelines, context management, memory/state persistence, and tool execution
  • Integrate large language models (LLMs), APIs, and internal services to enable context-aware, human-in-the-loop experiences across customer scenarios
  • Build and maintain data and inference pipelines that support model training, fine-tuning, evaluation, and real-time inference across diverse data sources
  • Evaluate, benchmark, and tune AI/ML models (LLMs and traditional models) to meet product requirements for accuracy, latency, reliability, and safety
  • Implement robust retrieval, grounding, and knowledge integration mechanisms (e.g., RAG systems, semantic indexing, vector search) to power intelligent applications
  • Collaborate with product managers, software engineers, and researchers to translate product vision into production-ready AI capabilities and measurable outcomes
  • Ensure reliability, observability, and governance of AI systems, including monitoring model performance, data quality, and responsible AI practices
  • Build reusable platforms, APIs, and tools that enable teams to rapidly develop AI-powered features and self-service intelligent applications
  • Fulltime
Read More
Arrow Right

Principal AI Engineer

We are looking for a Principal AI Engineer to lead the design and deployment of ...
Location
Location
United States
Salary
Salary:
200000.00 - 300000.00 USD / Year
apollo.io Logo
Apollo.io
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of software engineering experience
  • at least 3 years in applied LLM or agentic AI systems (2023–present)
  • proven success in deploying LLM-powered products used by real users at scale
  • deep backend & systems engineering expertise with Python, distributed systems, and scalable APIs
  • familiarity with LangChain, LlamaIndex, or similar orchestration frameworks
  • experience with RAG pipelines, vector DBs, embedding models, and semantic search tuning
  • experience managing performance across cloud providers (e.g., AWS Bedrock, OpenAI, Anthropic, etc.)
  • demonstrated experience building multi-step agents, planning workflows, chaining reasoning steps, and integrating APIs with agent memory/state
  • comfort with advanced prompting strategies, few-shot and chain-of-thought reasoning, and embedding retrieval setups
  • strong understanding of AI system evaluation, human ratings, A/B experimentation, and feedback loop pipelines
Job Responsibility
Job Responsibility
  • Architect and lead the development of multi-agent systems capable of long-horizon planning, reasoning, and API orchestration
  • build reusable agentic components that integrate deeply into sales and marketing processes
  • own and evolve our in-house platform for scalable, low-latency, and cost-efficient LLM and agent deployments
  • lead design of interfaces powered by natural language understanding and retrieval-augmented generation (RAG)
  • build embedding-based, intent-aware search and personalization systems tuned to business user needs
  • drive innovation in personalized outreach generation using context-aware generation pipelines
  • tune inference pipelines, caching layers, and model selection logic for high-scale, cost-aware performance
  • define and drive robust offline and online testing methodologies (A/B, sandboxing, human evals) across agents and LLM flows
  • architect human-in-the-loop systems and telemetry to improve accuracy, UX, and explainability over time
What we offer
What we offer
  • equity
  • company bonus or sales commissions/bonuses
  • 401(k) plan
  • at least 10 paid holidays per year
  • flex PTO
  • parental leave
  • employee assistance program
  • wellbeing benefits
  • global travel coverage
  • life/AD&D/STD/LTD insurance
  • Fulltime
Read More
Arrow Right

Principal Engineer

To lead and manage engineering teams, providing technical guidance, mentorship, ...
Location
Location
India , Pune
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 20+ years experience
  • Owns the team’s mission, roadmap, and outcomes
  • Translates business goals into a clear technical strategy and measurable OKRs spanning research, prototyping, and production AI systems
  • Builds and scales a high‑performing engineering culture (quality, reliability, security, cost discipline), embedding modern SDLC practices, SRE, and CI/CD with AI‑assist tools and agentic workflows
  • Guides system architecture end‑to‑end (data pipelines, model serving, APIs/UI, observability), ensuring privacy, compliance, model governance, and safe/ethical AI
  • Coaches engineers through design reviews, code reviews, experiment design, and productionization
  • Removes blockers and grows talent through clear career paths
  • Partners cross‑functionally with Product, Design, Risk/Controls, and Platform teams to prioritize work, manage dependencies, and ship iteratively with fast feedback loops
  • Owns delivery metrics (lead time, deployment frequency, MTTR) and uses data to improve flow, reduce toil, and standardize reusable components and platforms
  • Manages vendor/platform selection for AI toolchains (IDE assistants, agent platforms), balancing ROI, reliability, and governance
Job Responsibility
Job Responsibility
  • Lead engineering teams effectively, fostering a collaborative and high-performance culture to achieve project goals and meet organizational objectives
  • Oversee timelines, team allocation, risk management and task prioritization to ensure the successful delivery of solutions within scope, time, and budget
  • Mentor and support team members' professional growth, conduct performance reviews, provide actionable feedback, and identify opportunities for improvement
  • Evaluation and enhancement of engineering processes, tools, and methodologies to increase efficiency, streamline workflows, and optimize team productivity
  • Collaboration with business partners, product managers, designers, and other stakeholders to translate business requirements into technical solutions and ensure a cohesive approach to product development
  • Enforcement of technology standards, facilitate peer reviews, and implement robust testing practices to ensure the delivery of high-quality solutions
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Fulltime
Read More
Arrow Right