CrawlJobs Logo
Arize Logo Arize · IT - Software Development

UX Engineer, LLM Experimentation Platform

United States 175000.00 - 200000.00 USD / Year · Job Posted December 20, 2025
Apply Position
Job Link Share

Job Description

You'll work on our LLM evaluation and prompt experimentation platform, taking functional features and refining them into intuitive, polished experiences. We're a technical team building for technical users (ML engineers, researchers) who are drowning in complexity. Your job is to make that complexity manageable and maybe even enjoyable.

Job Responsibility

  • Refine and iterate on UX for existing features
  • Watch real users struggle in Fullstory, identify friction points, and fix them
  • Prototype interaction alternatives rapidly (in Figma or code) to test different approaches
  • Work closely with our Senior engineers who own feature architecture
  • Own the 'feel' of the product: transitions, feedback, error handling, progressive disclosure
  • Build reusable interaction patterns that the team can leverage
  • Run lightweight usability tests to validate improvements

Requirements

  • You've made products noticeably better through interaction refinement
  • You're comfortable with modern frontend frameworks and animation libraries
  • You've used session replay tools and incorporated user feedback into designs
  • You can work in Figma to prototype before committing to code
  • You think about edge cases, loading states, and empty states automatically
  • You've worked on complex tools (dev tools, data tools, B2B SaaS) where UX really matters

Nice to have

You've built or contributed to design systems

What we offer

  • competitive equity package
  • comprehensive benefits package, including medical, dental, vision
  • a 401(k) plan
  • unlimited paid time off
  • a generous parental leave plan
  • additional support for mental health and wellness
  • WFH monthly stipend to pay for co-working spaces
Arize - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

UX Engineer, LLM Experimentation Platform

8 matching positions

Senior Product Manager, Mobile Conversational AI

As a Senior Product Manager, Mobile Conversational AI, you will define and lead ...
Location
Location
United States , Austin
Salary
Salary:
Not provided
gm.com Logo
General Motors
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of digital/software product management experience
  • 2+ years of hands-on product experience with conversational AI, virtual assistants, chatbots, voice interfaces, or LLM-powered products
  • Proven ability to own and ship a customer-facing experience end-to-end in a complex environment with many upstream dependencies
  • Strong understanding of LLMs, prompt design, retrieval, evaluation methods, and the trade-offs between model quality, latency, and cost, sufficient to make informed product decisions and partner credibly with AI/ML teams
  • Demonstrated success partnering with platform, backend, and agent teams to compose a single coherent product experience from independently delivered services
  • Excellent UX intuition for mobile, with experience working with Design on novel interaction patterns where best practices are still emerging
  • Expertise in defining metrics and using analytics, experimentation, and user research to drive product decisions
  • comfortable with tools such as Adobe, Mixpanel, Firebase, SQL, or equivalent
  • Strong written communication and narrative skills, able to articulate vision, strategy, and trade-offs clearly to engineers, designers, and senior executives
  • Demonstrated success leading cross-functional initiatives in fast-paced, matrixed organizations
Job Responsibility
Job Responsibility
  • Own the long-term vision, roadmap, and KPIs for the mobile conversational AI experience across iOS and Android, defining what 'great' looks like for customers interacting with GM via natural language
  • Lead the end-to-end customer experience, including entry points, conversation UI, voice and text modalities, response formatting, error handling, hand-offs to humans or deeper app flows, and continuity across sessions
  • Partner closely with agent, platform, and backend services teams to integrate new conversational use cases into the mobile experience, defining clear contracts, latency and quality expectations, and a consistent customer experience across agents built by different teams
  • Develop the interaction design system for conversational AI in collaboration with Design and Research, covering tone of voice, prompt patterns, suggested actions, citations/disclosures, and fallback behaviors
  • Define and drive the trust, safety, and quality bar for the experience, including policies for accuracy, transparency, escalation, and customer feedback loops
  • Establish success metrics (task completion, containment, CSAT, latency, deflection, engagement, retention) and build dashboards and review rituals to track and improve them over time
  • Run a continuous experimentation and evaluation program, A/B tests, prompt and model evaluations, and qualitative research, to iteratively improve customer outcomes
  • Work with Legal, Privacy, Cybersecurity, and Responsible AI partners to ensure the experience meets GM's standards for data handling, consent, and responsible AI
  • Collaborate with Customer Care, Marketing, and Brand teams to align the conversational experience with broader customer journeys and to surface insights from real conversations back into the product organization
  • Be the voice of the customer for conversational AI inside GM, communicating strategy, progress, and trade-offs clearly to senior stakeholders
  • Fulltime
Read More
Arrow Right

Lead Product Manager - Services Experience

Millions of people come to Yelp to find and hire trusted local professionals—fro...
Location
Location
United States , San Francisco
Salary
Salary:
149000.00 - 269000.00 USD / Year
Yelp
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A strong foundational knowledge of what it takes to succeed as a Product Manager, with demonstrated expertise in consumer journeys, conversion optimization, or marketplace dynamics
  • Proven ability to analyze user funnels and translate insights into product changes that measurably move key metrics
  • Strong appreciation for UX design and an ability to reason about how information presentation and interaction patterns drive user behavior
  • Experience with or strong familiarity with LLM/chat product experiences, including conversational design, evaluation frameworks, or chat-based interfaces
  • Track record of setting product vision and roadmap for 12+ months, and driving execution in partnership with engineering teams
  • Comfortable operating across multiple surfaces (iOS, Android, Web) and working with experimentation platforms to validate hypotheses
  • Excellent communication and stakeholder management skills
  • able to align cross-functional partners around a shared strategy
  • A Bachelor’s Degree or an equivalent work experience is required
Job Responsibility
Job Responsibility
  • Own the upper-funnel user experience, identifying the information and interactions that move consumers from exploration toward submitting a project (request-a-quote)
  • Define and execute a monetization strategy outside of Yelp Assistant, including features like shareable “Request a Quote” links and enhanced project workspace capabilities
  • Establish and communicate a 12–18 month product roadmap for your lane, aligning stakeholders across Engineering, Design, Ads, and Marketing
  • Partner with design and analytics to analyze consumer journey data, identify high-leverage conversion opportunities, and run rapid experiments
  • Contribute to Yelp Assistant feature development as needs emerge post-launch, including defining chat UX patterns, instrumentation, and evaluation frameworks
  • Define success metrics (project submissions, funnel conversion rates, connected user need, revenue proxies) and build dashboards to track progress
  • Collaborate cross-functionally with peer PMs, Engineering Managers, and commercial stakeholders to ensure monetization efforts complement the broader Yelp Assistant strategy
What we offer
What we offer
  • Bonus
  • Restricted stock units
  • Benefits
  • Fulltime
Read More
Arrow Right

Principal Software Engineer- AI

Project Sophia is a new generation business application, built ground up from ma...
Location
Location
United States , Redmond
Salary
Salary:
139900.00 - 274800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  • OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Design, implement, and ship AI-first product capabilities end-to-end from rapid prototype to production, spanning LLM-powered services, retrieval/grounding pipelines, and intelligent UX experiences
  • Shape and evolve full-stack architecture, integrating front-end experiences, back-end services, and AI orchestration layers
  • Collaborate with design, research, and platform teams to adapt or fine-tune LLMs/SLMs and multimodal models for real-world customer scenarios
  • Build agentic, tool-using, and multimodal workflows that reason across data and services
  • Drive engineering excellence, including secure-by-design principles, accessibility compliance, automated testing, and high-quality code craftsmanship
  • Define and apply evaluation frameworks for AI systems, using telemetry, experimentation, and continuous feedback loops
  • Drive live-site reliability and operational excellence, participating in On-Call rotations
  • Fulltime
Read More
Arrow Right

Senior Product Manager - AI

As a Senior Product Manager in the Bentley Infrastructure Cloud group, you will ...
Location
Location
Canada , Quebec
Salary
Salary:
Not provided
bentley.com Logo
Bentley Systems
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A minimum of 8 years of professional experience in product management
  • PM experience with AI-enhanced products, including generative AI, is highly desirable
  • Track record of managing cross-functional initiatives with a high level of organizational visibility
  • Extensive knowledge of product management practices
  • Proven ability to build and lead high-performing, empowered teams in dynamic environments
  • Advanced degree in Business, Computer Sciences, or a related field
  • Technical proficiency for effective collaboration with engineering teams and defining product specifications
  • Passion for understanding and meeting the needs of developers and internal customers
  • Comfortable with ambiguity and adept at evolving ideas from conception to successful launch
  • Skill in balancing strategic product vision with hands-on operational management
Job Responsibility
Job Responsibility
  • Develop strategies and roadmaps to enhance the Bentley Infrastructure Cloud portfolio, focusing on integrating AI functionalities in Bentley’s Cloud solutions
  • Survey existing APIs available in Bentley products, identify opportunities for new APIs and tools, and plan how to instrument them, govern their usage by AI Agents, and monetize them
  • Develop strategies for interactions through the Model Context Protocol between Bentley’s cloud solutions and embedded copilots and with user-developed generative AI solutions
  • Perform internal and external research to understand developer and user needs, along with industry trends, to improve our products and AI strategy
  • Define strategic business outcomes and establish quarterly Objectives and Key Results (OKRs) for aligning development with Bentley's product and technology goals
  • Proactively manage and communicate the OKR and plans with stakeholders, leadership, and senior management transparently
  • Lead discoveries for developing innovative platform capabilities through experimentation to exceed developer and user expectations
  • Work closely with UX designers, developers, and other product team members to execute on the Bentley Infrastructure Cloud strategy
  • Monitor the Bentley Infrastructure Cloud performance using Key Performance Indicators (KPIs) to ensure ongoing enhancement
  • Build and sustain relationships with teams inside and outside the Bentley Infrastructure Cloud group for relevant capability development
Read More
Arrow Right

Software Engineer II

The Follow-on Suggestions team in STCI, in the Microsoft AI (MAI) organization d...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience, and experience with generative AI
  • Drive experimentation through A/B testing and offline evaluation to evaluate system performance
  • Comfortable driving complex server and client architecture across large product teams
  • Hands-on experience with modern LLM evaluation techniques, including LLM-as-a-Judge, agentic evaluations and RAG assessments
  • A track record of delivering successful, large scale applied ML projects in an industry setting
  • Experience with MLOps practices, model versioning, automated testing, monitoring and CI/CD for machine learning
  • Experience with proficient coding, debugging, and problem-solving skills
  • Outstanding communication and collaboration skills
Job Responsibility
Job Responsibility
  • Design, implement, and ship AI-first product capabilities end-to-end from rapid prototype to production, spanning LLM-powered services, retrieval/grounding pipelines, and intelligent UX experiences
  • Own implementation across the full stack integrating front-end experiences, back-end services, and AI orchestration layers that connect models, context, and tools to deliver cohesive, extensible, high-performance recommendation systems
  • Collaborate with design, research, and platform teams to adapt or fine-tune LLMs/SLMs for follow-on scenarios
  • Build agentic, tool-using workflows that reason across data and services
  • optimize for security, safety, latency, reliability, and cost efficiency
  • Contribute to engineering excellence secure-by-design, accessibility compliance, automated testing, and code craftsmanship across the product lifecycle
  • Instrument and evaluate AI features with telemetry, experimentation, and continuous feedback loops to improve user experience
  • Drive live-site reliability and operational excellence, participating in On-Call rotations while maintaining a sustainable, high-ownership engineering culture
  • Fulltime
Read More
Arrow Right

Senior Frontend Engineer

Copilot Studio is Microsoft’s enterprise platform for building and interacting w...
Location
Location
Ireland , Dublin
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND proven years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Strong proficiency in modern front-end technologies and architectures, including React, JavaScript, TypeScript, and state management solutions such as Redux, Recoil, MobX, or similar
  • Experience building and shipping AI-powered product features using large language models (LLMs), with hands-on experience integrating AI capabilities into user-facing or developer-facing workflows in production systems
  • Strong computer science fundamentals with demonstrated experience making architectural and design tradeoffs across platforms, environments, and complex systems
  • Demonstrated experience leading technical design and implementation of medium-to-large features or systems, including ownership of architecture, performance, reliability, and maintainability
  • Experience working with AI tools or AI-assisted development workflows (e.g., using coding models to generate, refactor, analyze, or validate code) as part of modern software development practices
  • Experience designing, building, and maintaining frontend build, testing, and quality practices, including unit, integration, and end-to-end testing, for scalable applications
  • Ability to collaborate across disciplines and teams, working effectively with product managers, designers, applied scientists, and other engineering partners to deliver cohesive end-to-end experiences
  • A track record of delivering customer-focused solutions with measurable user or business impact, and growing ownership in ambiguous or evolving problem spaces
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
Job Responsibility
Job Responsibility
  • Design and build complex, high-impact user experiences that simplify advanced AI capabilities for diverse user personas
  • Lead technical design for features spanning multiple components, influencing frontend architecture, UX quality, and system behavior
  • Drive projects from ambiguous problem statements through delivery, balancing speed, quality, and long-term maintainability
  • Partner deeply with design, product, and research to shape experience direction and execution
  • Leverage telemetry, experimentation, and user feedback to guide technical and experience decisions
  • Mentor peers and raise the bar for frontend engineering and experience quality
  • Define and drive the technical and experience architecture for major areas of Copilot Studio, with a strong emphasis on front-end frameworks, authoring experiences, and experience quality at scale
  • Design of complex, cross-team solutions spanning multiple components and services, demonstrating excellence in frontend architecture, performance, accessibility, and maintainability
  • Partner with senior product, design, and research leaders to on the long-term experience strategy, helping translate ambiguous product and business goals into durable technical solutions
  • Influence and align engineering decisions across teams and organizations, ensuring cohesive, consistent user experiences across Copilot Studio and the broader Microsoft Copilot ecosystem
  • Fulltime
Read More
Arrow Right

Applied AI Engineer II

Project Sophia is a new generation business application, built ground up from ma...
Location
Location
United States , Redmond
Salary
Salary:
100600.00 - 199000.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 2+ years related experience (e.g., statistics, predictive analytics, research) OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, OR related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
  • 2+ years of extensive experience with one or more modern web technologies such as .NET / Node / React / Angular, building RESTful APIs
  • 2+ years of solid experience in an OO Language like C# or Java or Python
  • 1+ years of professional experience working with generative artificial intelligence, large language models, or agent-based systems
  • BS/MS in Computer Science or equivalent or 5+ years of industry experience
  • Excellence in one or more general programming languages including but not limited to: Python, C#
  • JavaScript
  • TypeScript
Job Responsibility
Job Responsibility
  • Design, implement, and ship AI-first product capabilities end-to-end from rapid prototype to production, spanning LLM-powered services, retrieval/grounding pipelines, and intelligent UX experiences that delight users through Sophia’s AI canvas
  • Own implementation across the full stack integrating front-end experiences, back-end services, and AI orchestration layers that connect models, context, and tools to deliver cohesive, extensible, high-performance systems
  • Collaborate with design, research, and platform teams to adapt or fine-tune LLMs/SLMs and multimodal models for real-world customer scenarios, ensuring outcomes are contextual, transparent, and human-centered
  • Build agentic, tool-using, and multimodal workflows that reason across data and services
  • optimize for safety, latency, reliability, and cost efficiency
  • Contribute to engineering excellence secure-by-design, accessibility compliance, automated testing, and code craftsmanship across the product lifecycle
  • Instrument and evaluate AI features with telemetry, experimentation, and continuous feedback loops to refine reasoning quality and user experience
  • Drive live-site reliability and operational excellence, participating in On-Call rotations while maintaining a sustainable, high-ownership engineering culture
  • Fulltime
Read More
Arrow Right

Senior Design Technologist

At Capgemini Invent, we believe difference drives change. As inventive transform...
Location
Location
United States , San Francisco
Salary
Salary:
100000.00 - 150000.00 USD / Year
frog.co Logo
frog
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Professional experience designing and developing user interfaces, UX concepts, web sites and applications on a variety of platforms, frameworks, and methodologies
  • Ability to collaborate with a multi-disciplinary team of designers and strategists
  • A background in data management and manipulation to support data specific needs and concepts
  • Ability to communicate with clients directly and complete program and workstream tasks autonomously in some cases
  • Exceptional written and verbal communications skills along with experience in preparing and delivering documentation, proposals, presentations, and structured working sessions to all levels in a client organization
  • Bachelor’s Degree in the sciences, engineering, or creative discipline or equivalent experience
  • Proficiency in: Modern web front-end application development
  • Working within Cloud platforms (e.g., AWS, GCP, Azure, etc.)
  • Source control (e.g., GitHub, GitLab) and toolchain integrations
  • Third-party API integrations
Job Responsibility
Job Responsibility
  • Design Technologists at frog blend futurist vision and practical thinking, serving as hands-on organizers of technical insights, experiments, proofs, and models
  • Bridge the gap between creative design services, technology strategy, and software engineering
  • Drive technology platform research, experimentation, concept exploration, prototyping, and ideation within the creative process of designing user experiences
  • Leverage and manage data to support technical needs ranging from AI applications to impact assessments
  • Gather, ingest, and utilize data from diverse modalities to advance project goals
  • Identify and analyze current and emerging UX technology patterns and trends
  • Conduct in-depth investigations and experiments with new platforms and frameworks
  • Spot new opportunities for business value and UX innovation through technology enablers
  • Work closely with the creative team to translate design assets into user experience simulations, interactive prototypes, technical proofs-of-concept, and UI/presentation layers
  • Embrace a maker and tinkerer mindset—technically adept, passionate about design, and eager to tackle challenges hands-on
What we offer
What we offer
  • Flexible work
  • Healthcare including dental, vision, mental health, and well-being programs
  • Financial well-being programs such as 401(k) and Employee Share Ownership Plan
  • Paid time off and paid holidays
  • Paid parental leave
  • Family building benefits like adoption assistance, surrogacy, and cryopreservation
  • Social well-being benefits like subsidized back-up child/elder care and tutoring
  • Mentoring, coaching and learning programs
  • Employee Resource Groups
  • Disaster Relief
  • Fulltime
Read More
Arrow Right