CrawlJobs Logo

Senior Software Engineer - CoreAI Model Inference & Serving

https://www.microsoft.com/ Logo

Microsoft Corporation

Location Icon

Location:
United States , Redmond

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

119800.00 - 234700.00 USD / Year
Save Job
Save Icon
Apply Position

Job Description:

Join our team within CoreAI, where we are building the AI data-plane that powers all LLM inferencing workloads across Microsoft and Azure customers—from cutting-edge startups to Fortune 500 enterprises. Our converged AI fabric delivers inference capabilities for all LLMs in Microsoft catalog, including OpenAI, Anthropic, Mistral, Cohere, Llama, and more. As a Senior Software Engineer, you will shape the future of one of the largest and fastest-growing services in Azure, foundational to Microsoft’s AI strategy. Our mission is to serve models at scale—reliably, efficiently, and with ultra-low latency—enabling a rich set of AI-powered product experiences. This is a rapidly evolving space with immense opportunities to learn, innovate, and drive industry-wide impact!

Job Responsibility:

  • Be a hands-on technical leader, designing, coding, and shipping core serving systems, smart routing, and request distribution for a broad portfolio of LLMs
  • Build large-scale AI services and platform capabilities that power new products and customer experiences
  • Drive cutting-edge innovation in AI systems alongside world-class engineers and cross-functional partners
  • Lead through architecture, code reviews, mentorship, and technical excellence while staying close to implementation
  • Improve reliability, scalability, observability, efficiency, and performance across mission-critical services

Requirements:

  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, or Java
  • OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
  • 4+ years of design and problem-solving experience, with understanding of system performance, scalability, and engineering best practices
  • Understanding of distributed systems specifically in request serving at scale
  • (e.g. inferencing, L7 gateways, high-performance storage, distributed databases across global-scale infrastructure)
  • Demonstrated experience in building high-quality, reliable systems at scale
  • Experience using modern AI-assisted development tools and workflows to move faster, improve quality, and amplify engineering impact
  • Customer-obsessed approach to problem solving, with empathy and a drive to deliver impactful solutions

Additional Information:

Job Posted:
April 03, 2026

Employment Type:
Fulltime
Work Type:
Hybrid work
Icon
Microsoft Corporation - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon
PREMIUM
More languages and countries
+ Unlock 31694 hidden job offers
Languages
English Čeština Deutsch Ελληνικά Español Français +15
Countries
United States United Kingdom India Canada Australia +
See plans
Plans from $2.99 / month

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Senior Software Engineer - CoreAI Model Inference & Serving

Senior Software Engineer, CoreAI Workload Engines

The CoreAI Workloads team builds the foundational inference engines and APIs tha...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field and 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Python, or equivalent experience.
  • Proven ability to design and operate large-scale, production inference services with high reliability and performance requirements, and to ship performance improvements safely via disciplined experimentation.
  • Strong skills in performance analysis: benchmarking, profiling, diagnosing regressions, and turning results into concrete engine/runtime changes.
  • Strong problem-solving skills and the ability to debug complex, cross layer systems issues.
  • Demonstrated technical leadership, including mentoring engineers, driving cross-team architectural alignment, and leveraging AI tools and AI-assisted workflows to accelerate engineering velocity and quality.
  • Hands-on experience with Kubernetes (building and operating services on k8s), including debugging production issues and designing platform abstractions (e.g., custom resources/controllers) and scheduling-aware deployments (e.g., node affinity, taints/tolerations, resource requests/limits).
  • Strong collaboration and communication skills, with the ability to work across organizational boundaries.
Job Responsibility
Job Responsibility
  • Optimize inference engines for OpenAI and open-source models by implementing and shipping performance/efficiency improvements across runtime, scheduling, and serving paths (latency, throughput, utilization, availability, and cost).
  • Run experiments end-to-end: formulate hypotheses, implement engine changes (including Python/PyTorch integration points where relevant), analyze results, and ship improvements behind guardrails.
  • Build and use experimentation capabilities for large-scale AI inference (experiment lifecycle, tracking, metric modeling, comparability standards, automated analysis) so the team can iterate quickly and safely.
  • Own serving availability and efficiency for Azure OpenAI Service workloads through tiered experimentation, lean segmentation, and multi-modal utilization across heterogeneous fleets—turning findings into shipped engine improvements.
  • Design and evolve inference serving architectures to improve utilization and latency using techniques such as disaggregated serving, multi-token prediction, KV offload/retrieval, and quantization—validated via staged rollouts and production guardrails.
  • Extend AI infrastructure abstractions to support elastic, heterogeneous inference engines reliably at scale (e.g., dynamic scaling across model families, modalities, and workload classes while maintaining isolation and SLOs).
  • Tune and scale inference engines across NVIDIA GPU generations (A100, H100, H200) for state-of-the-art OpenAI models, focusing on serving efficiency, utilization, and reliability (not hardware bring-up).
  • Partner with networking and storage teams to leverage high-performance interconnects (e.g., RDMA/InfiniBand-class fabrics such as RoCE over IB) for distributed inference, without owning low-level kernel/driver enablement.
  • Drive end-to-end features from design through production: observability, diagnostics, performance regression detection, and operational excellence for inference serving.
  • Influence platform architecture and technical direction across teams through design reviews, clear metrics, and technical leadership focused on experimentation velocity and production reliability.
What we offer
What we offer
  • Benefits and other compensation
  • Fulltime
Read More
Arrow Right

Principal Product Manager/Architect - Foundry Inference Platform (CoreAI)

We are seeking a Principal Product Manager/Architect to define and guide the tec...
Location
Location
United States , Redmond
Salary
Salary:
163000.00 - 296400.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree AND 10+ years experience in product/service/program management or software development OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
Job Responsibility
Job Responsibility
  • 1. Product Reliability: Own the product direction for Microsoft Foundry inference, with a primary mandate to make the platform the most reliable enterprise inferencing service available. This includes defining architectural standards for global serving, multi-region resiliency, automated failover, and platform-managed disaster recovery
  • Drive architectural alignment across global routing, capacity pooling, observability, and control plane abstractions to ensure consistent availability, predictable recovery behavior, and simplified customer operations at scale
  • Partner with engineering, infrastructure, and security leaders to ensure reliability targets, SLAs, SLOs and recovery objectives are designed into the platform by default
  • 2. GPU Fleet Efficiency & Capacity: Set the product direction for GPU fleet efficiency and capacity management, guiding platform-level design decisions that maximize utilization, minimize fragmentation, and accelerate timetomonetization of new hardware and models
  • This includes shaping the architecture for global capacity pooling, intelligent scheduling, fungibility across workloads, automated demand forecasting, and softwaredefined allocation
  • The Product Manager/Architect is expected to influence architectural investments across inference utilization, model serving, and hardware/system performance
  • 3. Strategic Customer & Innovation Engagement: Act as a senior technical advisor and architect for Foundry’s most innovative and strategic customers
  • Engage directly with customers on deep technical challenges, including largescale model migrations, reliabilitysensitive production deployments, and advanced serving architectures
  • Support competitive and strategic initiatives by articulating Foundry’s architectural advantages, turning bespoke requests into scalable features
  • 4. Cross-Company Technical Leadership: Serve as a unifying architectural voice across product management, engineering, infrastructure, and partner teams
  • Fulltime
Read More
Arrow Right
New

Senior Lecturer/Associate Professor in Literacy

As a Senior Lecturer / Associate Professor in Literacy, you will play a key role...
Location
Location
Australia , Albury-Wodonga, Bathurst, Port Macquarie, Wagga Wagga
Salary
Salary:
Not provided
csu.edu.au Logo
Charles Sturt University
Expiration Date
June 08, 2026
Flip Icon
Requirements
Requirements
  • A doctoral qualification relevant to literacy or education, with a recognised teaching qualification
  • A strong record of high-quality teaching and student-centred learning
  • An established or emerging research profile aligned to literacy, curriculum or pedagogy
  • The ability to build productive partnerships and contribute to academic leadership
Job Responsibility
Job Responsibility
  • Lead impactful literacy teaching and research
  • Teach across online and on-campus environments
  • Shape future teachers and education practice
  • Contribute to curriculum innovation
  • Build strong relationships with students and partners
  • Provide academic leadership in literacy education
  • Contribute to the School's research profile
  • Supervise higher degree research students
  • Actively engage with professional, community and government stakeholders
  • At Associate Professor level: significant academic leadership, research impact, and contribution to the broader discipline at national/international level
What we offer
What we offer
  • 17% superannuation
  • Fulltime
Read More
Arrow Right
New

Program Manager - Controls and Avionics Solutions

This position is based in Endicott, New York. New York and on-site work will be ...
Location
Location
United States , Endicott
Salary
Salary:
120874.00 - 205486.00 USD / Year
baesystems.com Logo
Baesystems
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in engineering, engineering or manufacturing management, or other discipline
  • Demonstrated ability for building strong customer/ stakeholder relationships
  • Strong communication, negotiation, and presentation skills
  • Ability to interpret data and make data-driven decisions
  • Highly adaptable with strong initiative
  • Demonstrated ability to lead and motivate cross-functional teams
  • Knowledge of the global aviation market and regulatory requirements and/ or military aviation market
Job Responsibility
Job Responsibility
  • Maintaining strong customer relationships and leading a multidisciplinary team to execute complex development programs within schedule and budget
  • Leadership and management oversight of a project team assuring that project’s financials, schedule, and technical objectives are met and that the highest level of customer satisfaction is achieved while meeting all contractual commitments
  • Work effectively and collaboratively with Engineering, Operations, and all Program Office functional leadership to assure deliveries continue to exceed customer commitments and achievement of financial commitments to the company
  • Manages, coordinates, plans, organizes, controls, integrates, and executes projects within the Military Aircraft Systems portfolio
  • Participates in the support of new business and in the development of proposals
What we offer
What we offer
  • Health insurance
  • Dental insurance
  • Vision insurance
  • Health savings accounts
  • 401(k) savings plan
  • Disability coverage
  • Life and accident insurance
  • Employee assistance program
  • Legal plan
  • Discounts on home, auto, and pet insurance
  • Fulltime
Read More
Arrow Right
New

Finance Business Partner (Research)

Full Time, Fixed Term (12 months). Level 7 - $101,421 to $110,819 p.a. (plus 17%...
Location
Location
Australia , Wagga Wagga
Salary
Salary:
101421.00 - 110819.00 AUD / Year
csu.edu.au Logo
Charles Sturt University
Expiration Date
June 02, 2026
Flip Icon
Requirements
Requirements
  • A degree in Accounting or Finance (professional accounting body membership is desirable)
  • Experience in project budgeting, forecasting and financial analysis
  • Background in management accounting or business partnering within complex environments
  • exposure to government funding or higher education is advantageous
  • Excellent stakeholder engagement skills, with the ability to work effectively with academics and researchers
  • Familiarity with business intelligence systems and dashboard reporting
Job Responsibility
Job Responsibility
  • Partner with academics to deliver strategic financial insights that enable research success
  • Directly influence world-class projects and decisions shaping the future of education and innovation
  • Lead initiatives that enhance financial governance, deliver accurate and timely reporting, and support key projects such as cost-pricing systems and research budgeting
  • Help build financial capability across the University, fostering collaboration and continuous improvement
What we offer
What we offer
  • Flexibility with a 35-hour work week
  • Access to hybrid work arrangements
  • 17% superannuation
  • Fulltime
Read More
Arrow Right
New

Associate Lecturer/ Lecturer in Oral Health

Make a real impact by educating future oral health professionals to serve the ur...
Location
Location
Australia , Wagga Wagga
Salary
Salary:
80046.00 - 134965.00 AUD / Year
csu.edu.au Logo
Charles Sturt University
Expiration Date
June 16, 2026
Flip Icon
Requirements
Requirements
  • A qualification relevant to the discipline and appropriate to the level being applied for
  • Full registration (for teaching/research) as a Dentist or Oral Health Therapist with the Australian Health Practitioner Regulation Agency (Ahpra)
  • Excellent understanding of the clinical practice of oral health therapy, supported by a record of teaching and subject coordination relevant to the discipline and appropriate to the level being applied for
  • Evidence of the delivery of high quality student-centred learning and teaching in oral health therapy and/or general dentistry
  • A record of research activity or capability relevant to the discipline and appropriate to the level being applied for, as outlined in the position descriptions, may facilitate the progression of research opportunities
Job Responsibility
Job Responsibility
  • deliver high-quality teaching, clinical supervision and learning experiences in Oral Health
  • work with students in both clinical and preclinical settings while contributing to curriculum development, industry engagement and community partnerships
What we offer
What we offer
  • Generous support provided to assist with relocating to Riverina’s beautiful Wagga Wagga or surrounds
  • 17% superannuation
  • Fulltime
Read More
Arrow Right
New

Change Analyst

As Change Analyst you will provide specialist change management expertise to sup...
Location
Location
Australia , Albury-Wodonga, Bathurst, Dubbo, Orange, Wagga Wagga
Salary
Salary:
101421.00 - 110819.00 AUD / Year
csu.edu.au Logo
Charles Sturt University
Expiration Date
June 03, 2026
Flip Icon
Requirements
Requirements
  • Relevant qualifications and/or equivalent experience in organisational change and transformation
  • Experienced in applying change management frameworks and methodologies to large-scale/complex organisational initiatives
  • Skilled in analysing change impacts and shaping clear, targeted responses in policy-driven environments
  • Strong communication and interpersonal skills
Job Responsibility
Job Responsibility
  • Provide specialist change management expertise to support the successful planning and implementation of the Models of Engagement and Assessment initiative
  • Lead change analysis, stakeholder engagement planning and adoption activities to enable a sustainable transition to new models of course delivery and assessment.
What we offer
What we offer
  • Competitive salary and benefits including 17% super
  • Flexible working arrangements that support a healthy work-life balance
  • Fulltime
Read More
Arrow Right
New

Postdoc / Research Fellow in Digital Agricultural Futures

We are seeking a Research Associate / Postdoctoral Research Fellow (Level A) or ...
Location
Location
Australia , Mildura
Salary
Salary:
80046.00 - 134965.00 AUD / Year
csu.edu.au Logo
Charles Sturt University
Expiration Date
June 10, 2026
Flip Icon
Requirements
Requirements
  • Level A: A relevant postgraduate qualification (Masters or PhD) or equivalent experience in digital agriculture, irrigation, spatial science or related fields
  • Level B: A completed PhD (or equivalent standing) with demonstrated independent research capability
  • Experience in applied or multidisciplinary research environments, ideally connected to agriculture, education, or regional systems
  • Knowledge of, or interest in, education and training frameworks, workforce development or professional learning
  • Strong communication and relationship-building skills, with the ability to work effectively with researchers, industry and community stakeholders
  • The ability to manage priorities, work independently and collaboratively, and contribute to impactful research outcomes
Job Responsibility
Job Responsibility
  • Preparing the future workforce for digital irrigated agriculture
  • Working closely with academics, industry partners, education providers and communities across the Murray–Darling Basin
  • Contributing to research that explores digital literacy, education frameworks and innovative delivery models for contemporary agriculture
  • At Level A: contributing to research delivery under the guidance of senior researchers, supporting data collection, analysis, stakeholder engagement and co-authored outputs
  • At Level B: taking a more independent and substantive leadership position, leading defined research components, cultivating partnerships, and producing high-quality scholarly and industry-focused outputs
  • Regular interstate travel is required, along with strong collaboration across multidisciplinary and industry-linked projects
What we offer
What we offer
  • 17% superannuation
  • Relocation opportunity to Mildura, VIC
  • Flexible/hybrid arrangements considered
  • Fulltime
Read More
Arrow Right