CrawlJobs Logo
Marriott Bonvoy Logo Marriott Bonvoy · IT - Software Development

Ai Qa Engineer (Agents)

Ireland, Cork · Job Posted May 27, 2026
Apply Position
Job Link Share

Job Description

An AI QA Engineer (Agents) is responsible for ensuring the quality, reliability, and performance of AI agents and agentic experiences. This role involves designing and executing test strategies, identifying defects, and working closely with engineering teams to ensure high‑quality releases of AI agent solutions for business use cases. This role requires a strong attention to detail, analytical thinking, and the ability to think like both a user and a developer. The ideal candidate is passionate about quality and enjoys finding creative ways to test AI agents and ensure they work correctly across various scenarios and edge cases. The ideal candidate has experience testing AI/ML applications, with particular strength in testing conversational interfaces, LLM integrations, and AI agent workflows. This person should also have the desire to grow and learn every day, which will be essential for success in this role. The landscape changes daily, and we are changing with it.

Job Responsibility

  • Design and execute test plans for AI agents and agentic experiences
  • Write and maintain automated test suites for agent functionality (unit tests, evals integration tests, end‑to‑end tests)
  • Perform (minimal)manual testing of agent interactions, workflows, and business logic
  • Test agent responses, accuracy, and behavior across various scenarios and edge cases
  • Identify, document, and track bugs through resolution
  • Collaborate with engineers, product managers, and business stakeholders to understand requirements and acceptance criteria
  • Participate in test planning, test case design, and test strategy discussions
  • Create and maintain test data, test scenarios, and test environments for agents
  • Participate in feature design sessions, highlighting key testing scenarios and fault zones
  • Execute performance and load testing to ensure agent scalability and response times
  • Validate agent integrations with business systems, APIs, and data sources
  • Test agent security features and validate compliance with security requirements
  • Participate in release planning and ensure quality gates are met before releases
  • Contribute to improving testing processes and test automation infrastructure for AI agents

Requirements

  • 4+ years' total experience, including 1+ year testing AI/ML applications, LLM integrations, or conversational interfaces
  • Hands-on experience with end-to-end testing and automation for AI/agentic products
  • 3+ years of experience in software quality assurance or testing
  • 1+ years of experience testing AI/ML applications, LLM integrations, or conversational interfaces
  • Strong understanding of software testing principles, methodologies, and best practices
  • Experience writing and maintaining automated tests (unit, integration, or end‑to‑end)
  • Proficiency in at least one programming language (Python, TypeScript, JavaScript, Java, etc.)
  • Experience with API testing tools (Postman, REST Assured, etc.) or frameworks
  • Strong analytical and problem‑solving skills
  • Excellent attention to detail and ability to identify edge cases
  • Good written and verbal communication skills
  • Experience with bug tracking systems and test management tools
  • Ability to work collaboratively with engineering and product teams
  • Understanding of CI/CD pipelines and test automation in continuous integration
  • Interest in AI/ML concepts and understanding of how to test AI systems

Nice to have

  • Experience with AI Eval tools or frameworks
  • Experience testing AI agents, chatbots, or virtual assistants
  • Background in testing LLM integrations and prompt‑based systems
  • Experience with agent testing frameworks and tools
  • Knowledge of testing RAG (Retrieval Augmented Generation) systems
  • Experience with performance testing tools (JMeter, k6, Locust, etc.)
  • Experience with test automation frameworks (Playwright, Cypress, Selenium, pytest, etc.)
  • Familiarity with cloud platforms and testing cloud‑native applications
  • Experience with observability tools and using metrics/logs for test validation
  • Knowledge of security testing and vulnerability assessment for AI applications
  • Experience with contract testing and API mocking
  • Familiarity with prompt testing and LLM response validation
  • ISTQB or similar testing certification
  • Chaos and Resilience Testing
Marriott Bonvoy - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Ai Qa Engineer (Agents)

8 matching positions

Qa Engineer – Agents & Ai Platform

QA Engineer – Agents & AI Platform. We are looking for a QA Engineer – Agents & ...
Location
Location
India , Chennai
Salary
Salary:
Not provided
optisolbusiness.com Logo
OptiSol Business Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • QA experience testing complex web platforms, workflow systems, or enterprise applications
  • Strong knowledge of QA methodologies, test case design, and defect lifecycle management
  • Hands-on experience with API testing tools such as Postman and exploratory testing techniques
  • Ability to build light automation scripts using Python, JavaScript, or testing frameworks for regression and API testing
  • Understanding of workflow-driven platforms or enterprise process systems
  • Awareness of security and privacy considerations, including RBAC, PHI/PII protection, and audit logging
  • Strong communication skills and a proactive mindset toward owning product quality
Job Responsibility
Job Responsibility
  • Agent Platform Testing & System Validation Understand product requirements and workflows across vertical agents and design end-to end test scenarios for agents, platform components, observability systems, and prompt management
  • Create and maintain comprehensive test plans, test cases, and test data for agent orchestration, prompts, and integrations
  • Build sample agents, mock tools, and demo workflows to simulate real-world scenarios and identify edge cases
  • Execute manual and automated tests including API, regression, and smoke testing for web UI, APIs, and background workflows
  • Perform basic security testing such as access control validation, data privacy checks, input validation, and misuse scenarios
  • Use logs, metrics, and traces to investigate defects and validate system behavior
  • Ensure platform reliability, performance, and quality across the testing lifecycle
  • Collaboration, Automation & Release Support Collaborate with engineers and product managers to triage issues and improve product test ability
  • Build light automation scripts for API and regression testing where applicable
  • Participate in release validation and quality sign-offs before deployments
What we offer
What we offer
  • Opportunity to work on next-generation AI agent platforms and intelligent automation systems
  • Hands-on experience testing LLM-driven products and AI-powered workflows
  • A collaborative engineering culture focused on innovation, experimentation, and continuous learning
  • Competitive compensation and strong opportunities for career growth in AI and platform engineering
  • Fulltime
Read More
Arrow Right

QA Engineer - Agents & AI Platform

We are looking for a skilled and innovative AI / Machine Learning Engineer to de...
Location
Location
India , Chennai
Salary
Salary:
Not provided
optisolbusiness.com Logo
OptiSol Business Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong understanding of Machine Learning and Deep Learning fundamentals
  • Knowledge of Transformer architectures and language models
  • Experience working with Small Language Models (SLMs) or fine-tuned models
  • Hands-on experience with LangChain, LangGraph, CrewAI, AutoGen, OpenAI Agent SDK, or similar frameworks
  • Strong Python programming skills
  • Experience with FastAPI or similar backend frameworks
  • Knowledge of Relational and NoSQL databases
  • Familiarity with Git, Docker, and CI/CD pipelines
  • Experience with testing frameworks such as pytest or unittest
Job Responsibility
Job Responsibility
  • Design, develop, and deploy AI applications powered by Small Language Models (SLMs) and fine-tuned language models
  • Build and manage agent orchestration workflows using frameworks such as LangGraph,CrewAI, AutoGen, or OpenAI Agent SDK
  • Develop multi-agent systems that coordinate tasks through planning, reasoning, and tool interaction
  • Build Retrieval-Augmented Generation (RAG) pipelines integrating vector databases, APIs and enterprise data sources
  • Apply strong Python programming and data structure knowledge to build scalable AI systems
  • Develop backend services and APIs using Python frameworks such as FastAPI
  • Integrate AI solutions with relational or NoSQL databases and external services
  • Write clean, maintainable, and testable code following software engineering best practices
What we offer
What we offer
  • Supportive and professional work environment
  • Competitive salary as per market standards
  • Opportunity to work on advanced AI and multi-agent technologies
  • Career growth and learning opportunities in AI engineering
  • Fulltime
Read More
Arrow Right

Senior QA Engineer – AI & Conversational Products

We're looking for a Senior QA Engineer who's passionate about quality, thrives i...
Location
Location
Israel
Salary
Salary:
Not provided
radancy.com Logo
Radancy
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of QA engineering experience, with at least 2 years in a senior or lead capacity
  • Hands-on experience testing AI/ML features, LLM-based products, or conversational systems (a plus)
  • Strong proficiency with test automation frameworks (e.g., Playwright, Selenium, Cypress, Pytest)
  • Solid understanding of API testing (REST/GraphQL) and tools like Postman or similar
  • Experience with CI/CD pipelines (GitHub Actions, Jenkins, or equivalent)
  • Ability to think like a user and an engineer — identifying failure modes others miss
  • Excellent communication skills and a collaborative approach to cross-functional teamwork
Job Responsibility
Job Responsibility
  • Own the end-to-end quality assurance process for features across web, API, and backend services within Radancy's talent platform
  • Design and implement test strategies for LLM-based features and AI agents — ensuring accuracy, consistency, and robustness at scale
  • Build and maintain automated test frameworks covering UI, API, performance, and regression testing
  • Develop comprehensive test plans and test cases that address both functional requirements and edge scenarios unique to AI-driven hiring experiences
  • Partner closely with Product, Engineering, and Data teams to surface quality gaps early in the development cycle
  • Analyze and report bugs with precision, advocating for high-impact fixes that protect candidate and employer experience
  • Participate in release processes, CI/CD pipeline validation, and production monitoring
Read More
Arrow Right

Middle AI QA Engineer

We are looking for an AI QA Engineer to ruthlessly test our digital employees ab...
Location
Location
Salary
Salary:
Not provided
maddevs.io Logo
Mad Devs
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2+ years of hands-on experience in manual testing
  • Ability to read code (Python, Jinja2) and work confidently with JSON data payloads to better understand test scenarios
  • Understanding of STT (Speech-to-Text), TTS (Text-to-Speech), and VAD (Voice Activity Detection) concepts, including the importance of silence detection
  • Strong exploratory testing mindset with the ability to intentionally stress and challenge AI agents (e.g., interruptions, rapid speech, topic switching, unclear pronunciation)
  • Russian - Native or Fluent
  • English - Upper-Intermediate (B2+)
Job Responsibility
Job Responsibility
  • Voice UX Testing: Measure and optimize Latency (P95/P99). Verify Barge-In functionality: ensure the agent stops speaking immediately when interrupted, even in noisy environments
  • Logic & Flow Validation: Read and understand NSL (Jinja2) scripts to identify potential logical loops or dead ends. Verify that the agent handles state transitions correctly (e.g., booking created -> confirmation sent)
  • Hallucination Control: Detect when the AI drifts from its instructions or invents facts. Help build "Golden Datasets" of correct conversations for regression testing
  • Integration Testing: Verify that the agent correctly triggers external tools (e.g., CRM updates) during a call. What happens if the API fails? Does the agent handle it gracefully?
What we offer
What we offer
  • Flexible working hours
  • Remote-first culture
  • Long-term projects
  • Salary in dollars
  • Professional communities
  • Onsite business trips
  • Training budget
  • Paid conferences
Read More
Arrow Right
New

Tech Lead QA Engineer with AI experience

N-iX is looking for a Tech Lead QA Engineer with AI experience to join our team....
Location
Location
Ukraine
Salary
Salary:
Not provided
n-ix.com Logo
N-iX
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years of experience in manual QA, with exposure to frontend, backend and AI testing
  • Solid experience with API testing using tools such as Postman, REST Client, or similar
  • Experience testing web UIs and understanding of cross-browser/cross-environment considerations
  • Ability to read and interpret API documentation, data schemas, and system architecture descriptions
  • Hands-on experience working on projects that included AI, ML, or NLP components — particularly validating outputs that are probabilistic or context-dependent
  • Familiarity with the concept of RAG (Retrieval-Augmented Generation) or LLM-based systems
  • Strong analytical thinking — especially the ability to assess whether an AI response is contextually correct, not just technically non-null
  • Good understanding of test documentation practices: test plans, test cases, bug reports, traceability
  • English level at least Upper-Intermediate
Job Responsibility
Job Responsibility
  • Follow a phased QA approach — begin with CMS and backend testing to establish a reliable baseline of expected system behavior, then apply those insights to validate AI agent outputs effectively
  • Design and execute test cases for REST APIs, covering functional correctness, edge cases, error handling, authentication, and data integrity
  • Perform UI testing across core user journeys, validating layout, behavior, and integration with backend services
  • Transition into AI output validation once the deterministic layers are stable — using your knowledge of business rules to identify inconsistencies, hallucinations, or degraded outputs in agent responses
  • Document and maintain test cases, test plans, and bug reports in a structured and traceable way
  • Participate in requirement reviews and technical discussions to identify testability gaps early
  • Collaborate with the Lead Big Data/AI Engineer and AI team to understand RAG pipeline behavior, document ingestion flows, and output quality expectations
  • Contribute to building reusable test assets and QA processes as the project scales
What we offer
What we offer
  • Flexible working format - remote, office-based or flexible
  • A competitive salary and good compensation package
  • Personalized career growth
  • Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
  • Active tech communities with regular knowledge sharing
  • Education reimbursement
  • Memorable anniversary presents
  • Corporate events and team buildings
  • Other location-specific benefits
  • Fulltime
Read More
Arrow Right

Ai Automation Qa Engineer

Intellias helps retailers deliver consistent, omnichannel experiences through te...
Location
Location
Bulgaria; Croatia; Poland; Portugal; Ukraine
Salary
Salary:
Not provided
Intellias
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands-on work with LLM pipelines / agents (not just tools)
  • Prompt tuning, structured outputs (JSON/schema)
  • Understanding of MCP tools and data retrieval
  • Ability to work with APIs, JSON data models, async flows (queues, events)
  • Understanding of PHP / Laravel architecture
  • DB concepts (Postgres / configuration / tenants)
  • Deep Playwright usage: fixtures, storage state, network interception, async/debugging
  • Ability to build reusable flows (no duplication)
  • Ability to debug across AI output → test → API → application
  • Handling of authentication / cookies / environment issues
Job Responsibility
Job Responsibility
  • Build and maintain AI-driven test automation using Playwright + TypeScript
  • Work with LLM-based automation pipelines and AI-generated test cases
  • Use MCP tools locally as part of daily automation workflow (Copilot + AI tools included)
  • Extend and maintain existing automation framework (reusable, no duplication)
  • Validate and work with AI-generated structured outputs (JSON / schema)
  • Participate in peer code reviews (automation / TypeScript)
  • Debug issues across test automation: failing tests, API mismatches, environment/auth issues, AI-generated incorrect scenarios
  • Work with REST APIs and JSON-based data flows (in context of testing)
  • Contribute to an AI-first QA environment transitioning toward LLM/MCP-driven automation
Read More
Arrow Right
New

Sdet Ai Qa / Evaluation Engineer – Agentic Applications

職種: 品質管理・テスト・評価 業務内容: WFH flexibility! Global Environment! Competitive salary!
Location
Location
Japan , 東京23区
Salary
Salary:
7000000.00 - 12000000.00 JPY / Year
https://www.randstad.com Logo
Randstad
Expiration Date
February 29, 2028
Flip Icon
Requirements
Requirements
  • Min of 4-5 years of work experience
  • Core experience in Software QA, test automation, data validation, or AI/ML application quality engineering
  • Solid conceptual understanding of LLMs, prompt engineering, RAG, OCR, tool calling, and agentic system behaviors
  • Hands-on automation scripting experience using Python or JavaScript/TypeScript
  • Working familiarity with modern CI/CD pipelines, Azure DevOps, and GitHub Enterprise
  • Business-level Japanese (or above) is mandatory for this role
What we offer
What we offer
  • WFH flexibility
  • Global Environment
  • Competitive salary
  • 健康保険
  • 厚生年金保険
  • 雇用保険
  • Fulltime
Read More
Arrow Right

Senior QA Engineer (AI-based)

We are seeking a detail-oriented Senior QA Engineer to keep the quality assuranc...
Location
Location
United Arab Emirates , Dubai
Salary
Salary:
Not provided
parserdigital.com Logo
Parser Limited
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of experience in QA engineering, with proven experience in testing data systems or AI applications
  • Strong analytical skills and attention to detail, capable of spotting subtle inconsistencies in data and agent behavior
  • Proficiency in Python and SQL for data validation, automation scripting, and test preparation
  • Experience with API testing (REST/GraphQL automation and validation)
  • Familiarity with chatbot frameworks, LLMs, or conversational testing
  • (Desirable) Experience or strong interest in using LLM-based assistants (e.g., GitHub Copilot, ChatGPT) in test and execution
  • (Desirable) Exposure to QA automation tools (e.g., PyTest, Postman, dbt tests)
Job Responsibility
Job Responsibility
  • Expertise in quality assurance efforts for data pipelines, APIs, and conversational agents
  • Design and execute generative AI testing scenarios, including exploratory testing for chatbot interactions, intent recognition, tone, and edge-case behavior
  • Experience in LLM test designs, including prompt and grounding validation, golden sets, and non-deterministic assertions
  • Perform crucial safety and quality checks, such as testing for hallucination, bias, toxicity, and PII leakage
  • Validate structured and unstructured data outputs, ensuring consistency, accuracy, and compliance
  • Establish and drive real-time testing approaches, including streaming data validation and API monitoring
  • Collaborate with ML and NLP teams to define comprehensive evaluation metrics and criteria for agent performance
  • Integrate AI-driven tools (like Copilot) into the QA lifecycle to accelerate test design, documentation, and defect analysis
  • Fulltime
Read More
Arrow Right