Job Description:
Microsoft’s Azure Data engineering team is leading the transformation of analytics in the world of data with products like databases, data integration, big data analytics, messaging & real-time analytics, and business intelligence. The products our portfolio include Microsoft Fabric, Azure SQL DB, Azure Cosmos DB, Azure PostgreSQL, Azure Data Factory, Azure Synapse Analytics, Azure Service Bus, Azure Event Grid, and Power BI. Our mission is to build the data platform for the age of AI, powering a new class of data-first applications and driving a data culture. Within Azure Data, the messaging and real-time analytics team provides comprehensive solutions and a robust platform that enables users to ingest high granularity signals (real-time & observability) and complex data, converting those into a competitive advantage in real-time for both end users and modern applications. The Real-Time Intelligence team is hiring a Senior AI Engineer to build and scale the evals platform that underpins our real-time, agentic solutions on live operational data. You will focus on creating human and LLM-as-judge pipelines, and telemetry that measure how well our autonomous agents, anomaly detection, and decisioning systems perform in the real world. In this role, you’ll own end-to-end implementation of eval pipelines and tooling, integrate them into CI/CD and production workflows, and partner closely with product and platform teams so that every AI feature is measurable, debuggable, and continuously improving. Rigorous evaluation is the most critical ingredient in building reliable agents, this is a highly leveraged, key role that will directly shape how we design, ship, and operate real-time AI agentic products.
