CrawlJobs Logo
Marriott Bonvoy Logo Marriott Bonvoy · IT - Software Development

Ai Qa Engineer (Agents)

Ireland, Cork · Job Posted May 27, 2026
Apply Position
Job Link Share

Job Description

An AI QA Engineer (Agents) is responsible for ensuring the quality, reliability, and performance of AI agents and agentic experiences. This role involves designing and executing test strategies, identifying defects, and working closely with engineering teams to ensure high‑quality releases of AI agent solutions for business use cases. This role requires a strong attention to detail, analytical thinking, and the ability to think like both a user and a developer. The ideal candidate is passionate about quality and enjoys finding creative ways to test AI agents and ensure they work correctly across various scenarios and edge cases. The ideal candidate has experience testing AI/ML applications, with particular strength in testing conversational interfaces, LLM integrations, and AI agent workflows. This person should also have the desire to grow and learn every day, which will be essential for success in this role. The landscape changes daily, and we are changing with it.

Job Responsibility

  • Design and execute test plans for AI agents and agentic experiences
  • Write and maintain automated test suites for agent functionality (unit tests, evals integration tests, end‑to‑end tests)
  • Perform (minimal)manual testing of agent interactions, workflows, and business logic
  • Test agent responses, accuracy, and behavior across various scenarios and edge cases
  • Identify, document, and track bugs through resolution
  • Collaborate with engineers, product managers, and business stakeholders to understand requirements and acceptance criteria
  • Participate in test planning, test case design, and test strategy discussions
  • Create and maintain test data, test scenarios, and test environments for agents
  • Participate in feature design sessions, highlighting key testing scenarios and fault zones
  • Execute performance and load testing to ensure agent scalability and response times
  • Validate agent integrations with business systems, APIs, and data sources
  • Test agent security features and validate compliance with security requirements
  • Participate in release planning and ensure quality gates are met before releases
  • Contribute to improving testing processes and test automation infrastructure for AI agents

Requirements

  • 4+ years' total experience, including 1+ year testing AI/ML applications, LLM integrations, or conversational interfaces
  • Hands-on experience with end-to-end testing and automation for AI/agentic products
  • 3+ years of experience in software quality assurance or testing
  • 1+ years of experience testing AI/ML applications, LLM integrations, or conversational interfaces
  • Strong understanding of software testing principles, methodologies, and best practices
  • Experience writing and maintaining automated tests (unit, integration, or end‑to‑end)
  • Proficiency in at least one programming language (Python, TypeScript, JavaScript, Java, etc.)
  • Experience with API testing tools (Postman, REST Assured, etc.) or frameworks
  • Strong analytical and problem‑solving skills
  • Excellent attention to detail and ability to identify edge cases
  • Good written and verbal communication skills
  • Experience with bug tracking systems and test management tools
  • Ability to work collaboratively with engineering and product teams
  • Understanding of CI/CD pipelines and test automation in continuous integration
  • Interest in AI/ML concepts and understanding of how to test AI systems

Nice to have

  • Experience with AI Eval tools or frameworks
  • Experience testing AI agents, chatbots, or virtual assistants
  • Background in testing LLM integrations and prompt‑based systems
  • Experience with agent testing frameworks and tools
  • Knowledge of testing RAG (Retrieval Augmented Generation) systems
  • Experience with performance testing tools (JMeter, k6, Locust, etc.)
  • Experience with test automation frameworks (Playwright, Cypress, Selenium, pytest, etc.)
  • Familiarity with cloud platforms and testing cloud‑native applications
  • Experience with observability tools and using metrics/logs for test validation
  • Knowledge of security testing and vulnerability assessment for AI applications
  • Experience with contract testing and API mocking
  • Familiarity with prompt testing and LLM response validation
  • ISTQB or similar testing certification
  • Chaos and Resilience Testing
Marriott Bonvoy - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Ai Qa Engineer (Agents)

8 matching positions

AI Research Engineer - Reinforcement Learning

At Helsing we deliver AI-based capabilities and the enabling infrastructure that...
Location
Location
Germany , Munich
Salary
Salary:
Not provided
helsing.ai Logo
Helsing
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hold MSc in machine learning with a speciality in either reinforcement learning, multi-agent systems, automation and control, or robotics
  • Have excellent communication skills and the ability to report and present research findings clearly and efficiently both internally and externally
  • Are passionate about keeping up-to-date with current research and enjoy reimplementing / extending papers on state-of-the-art Deep Learning-based approaches
  • Possess solid software engineering skills, writing clean and well-structured code in Python and/or languages like Rust, Java, or modern C++, and experience deploying AI software to production including testing, QA, and monitoring
Job Responsibility
Job Responsibility
  • Design, train and deploy agents in complex multi-agent environments
  • Contribute to our reinforcement learning stack by implementing, improving and extending the current state of the art in multi-agent reinforcement learning
  • Be a part of impactful projects and will collaborate with people across several teams and backgrounds to integrate cutting edge ML/AI in our production systems
What we offer
What we offer
  • Competitive compensation and stock options
  • Relocation support
  • Social and education allowances
  • Regular company events and all-hands to bring together employees as one team across Europe
  • A hands-on onboarding program (affectionately labelled “AI-duction”), in which you will be familiarising yourself with our tools and ML pipelines used across the company
  • Fulltime
Read More
Arrow Right

AI Engineer

Reporting to the AI & Technology Oversight Manager, the AI Engineer is responsib...
Location
Location
India , Mumbai
Salary
Salary:
Not provided
waystone.com Logo
Waystone Governance Ltd.
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Deep understanding of the distinction between Generative AI and Agentic AI, including their foundations, capabilities, and appropriate use cases
  • Strong understanding of AI, ML and LLM concepts, including prompt engineering, prompt grounding, iterative loop techniques, context windows, embeddings, RAG, agentic workflows
  • Proven ability to integrate AI capabilities both into low-code automation flows and high-code stacks, including, applications, APIs, microservices, distributed systems, and development or testing tools
  • Solid software development background with hands-on coding experience in one or more engineering ecosystem such as .NET (C#), Python, or TypeScript
  • Excellent communication skills, with the ability to translate complex AI concepts for non‑experts and to effectively influence and collaborate with stakeholders at all levels, both technical and non‑technical
  • Strong writing skills, with the ability to contribute to AI literacy and AI fluency documentation
  • Strong understanding of responsible AI principles, including governance, bias mitigation, compliance, and risk-based decision-making
  • Analytical thinking with excellent problem‑solving ability and keen attention to details
  • Ability to mentor developers and testers, and to drive innovation across engineering, QA, and architecture
  • Ability to assess AI‑enabled capabilities in third‑party SaaS platforms (e.g., Appian, Salesforce,etc) and provide guidance on responsible, effective adoption
Job Responsibility
Job Responsibility
  • Hands-on contributor to the design and development of AI-enabled solutions, capable of writing both production-quality code and rapid experimental prototypes
  • Develop and implement AI‑enabled microservices, APIs, applications, and internal tools
  • Integrate AI capabilities following secure, scalable engineering best practices
  • Design, build and validate AI‑driven solutions leveraging providers such as OpenAI and Anthropic
  • Enhance low‑code/no‑code automation platforms (e.g., Power Automate, n8n, Workato) by embedding intelligent processing and applying agentic patterns where relevant
  • Implement Model Context Protocol (MCP) servers for secure AI‑to‑system connectivity
  • Lead AI‑based document parsing and intelligent data extraction initiatives
  • Contribute to educating and enabling Enterprise Capabilities areas, including Integration and Automation, by providing guidance, training, and best practices, e.g., on effective use of n8n agents
  • Engage with business stakeholders to understand requirements, constraints, and key drivers, identifying and implementing high‑value AI opportunities across Waystone
  • Prototype AI features and iterate towards production‑ready capabilities
  • Fulltime
Read More
Arrow Right

AI Engineer

Reporting to the AI & Technology Oversight Manager, the AI Engineer is responsib...
Location
Location
United Kingdom , Leeds
Salary
Salary:
Not provided
waystone.com Logo
Waystone Governance Ltd.
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Deep understanding of the distinction between Generative AI and Agentic AI, including their foundations, capabilities, and appropriate use cases
  • Strong understanding of AI, ML and LLM concepts, including prompt engineering, prompt grounding, iterative loop techniques, context windows, embeddings, RAG, agentic workflows
  • Proven ability to integrate AI capabilities both into low-code automation flows and high-code stacks, including, applications, APIs, microservices, distributed systems, and development or testing tools
  • Solid software development background with hands-on coding experience in one or more engineering ecosystem such as .NET (C#), Python, or TypeScript
  • Excellent communication skills, with the ability to translate complex AI concepts for non‑experts and to effectively influence and collaborate with stakeholders at all levels, both technical and non‑technical
  • Strong writing skills, with the ability to contribute to AI literacy and AI fluency documentation
  • Strong understanding of responsible AI principles, including governance, bias mitigation, compliance, and risk-based decision-making
  • Analytical thinking with excellent problem‑solving ability and keen attention to details
  • Ability to mentor developers and testers, and to drive innovation across engineering, QA, and architecture
  • Ability to assess AI‑enabled capabilities in third‑party SaaS platforms (e.g., Appian, Salesforce,etc) and provide guidance on responsible, effective adoption
Job Responsibility
Job Responsibility
  • Hands-on contributor to the design and development of AI-enabled solutions, capable of writing both production-quality code and rapid experimental prototypes
  • Develop and implement AI‑enabled microservices, APIs, applications, and internal tools
  • Integrate AI capabilities following secure, scalable engineering best practices
  • Design, build and validate AI‑driven solutions leveraging providers such as OpenAI and Anthropic
  • Enhance low‑code/no‑code automation platforms (e.g., Power Automate, n8n, Workato) by embedding intelligent processing and applying agentic patterns where relevant
  • Implement Model Context Protocol (MCP) servers for secure AI‑to‑system connectivity
  • Lead AI‑based document parsing and intelligent data extraction initiatives
  • Contribute to educating and enabling Enterprise Capabilities areas, including Integration and Automation, by providing guidance, training, and best practices, e.g., on effective use of n8n agents
  • Engage with business stakeholders to understand requirements, constraints, and key drivers, identifying and implementing high‑value AI opportunities across Waystone
  • Prototype AI features and iterate towards production‑ready capabilities
  • Fulltime
Read More
Arrow Right

Principal Software Consultant - AI/ML Engineer

As an ML Team Lead, you will be responsible for leading the technical direction ...
Location
Location
Pakistan , Lahore, Karachi, Islamabad
Salary
Salary:
Not provided
10pearls.com Logo
10Pearls
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in computer science, Artificial Intelligence, Data Science, Software Engineering, or a related field
  • 7+ years of professional software engineering experience with at least 5 years of hands-on experience building and deploying ML systems into production
  • Prior experience as a Tech Lead, Staff Engineer, or hands-on lead for AI/ML engineering teams
  • Strong expertise in classical machine learning domains such as forecasting, ranking, classification, and optimization
  • Hands-on experience building modern LLM and agentic AI systems including RAG pipelines, tool-using agents, multi-step workflows, and evaluation systems
  • Strong proficiency in Python and backend system development
  • Experience with ML frameworks such as PyTorch or TensorFlow
  • Strong understanding of scalable distributed systems, APIs, system integration, architecture design, and production engineering practices
  • Experience operating ML services at scale, including SLO management, monitoring, on-call practices, and incident response
  • Experience working with Kubernetes-based deployments, CI/CD pipelines, and modern cloud-native engineering practices
Job Responsibility
Job Responsibility
  • Lead the technical direction for the team’s ML and LLM systems, including architecture patterns, platform choices, evaluation frameworks, and engineering standards
  • Stay hands-on by designing and implementing complex ML and agentic AI systems, writing production-grade code, and leading through technical execution
  • Design, develop, and deploy scalable ML and LLM-powered applications and services in production environments
  • Build and optimize AI-powered solutions such as RAG systems, multi-step agents, AI assistants, chatbots, forecasting systems, ranking models, classification models, and optimization systems
  • Drive architecture and design reviews to ensure scalability, reliability, security, and maintainability of AI/ML systems
  • Own the technical roadmap for ML/LLM initiatives and translate business objectives into execution plans and scalable solutions
  • Collaborate closely with Product Managers, Engineers, Data Engineers, MLOps Engineers, QA Engineers, and cross-functional stakeholders to deliver business-aligned AI solutions
  • Establish engineering best practices for prompt engineering, model evaluation, regression testing, observability, and production readiness
  • Define and implement quality standards, evaluation suites, acceptance metrics, and regression plans for all AI/ML features
  • Ensure high availability, scalability, and resilience of tier-1 ML services through SLOs, monitoring, incident response, failover strategies, circuit breakers, and multi-zone deployments
  • Fulltime
Read More
Arrow Right

Senior Application Engineer (Salesforce Agentforce AI)

We’re looking for creative thinkers with hands-on experience in Agentforce, AI a...
Location
Location
United States
Salary
Salary:
121000.00 - 135000.00 USD / Year
pointclickcare.com Logo
PointClickCare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands-on experience building Agentforce AI Agents, AI workflows, custom agent development, and integration of external data sources
  • Experience creating intelligent process automations using tools like Power Automate or others
  • Solid experience creating test automations for Salesforce and other platforms
  • Solid experience writing clean code that performs well at scale using JavaScript, Apex, Python, etc.
  • Solid experience with Salesforce integrations and external system integrations (Conga, Docusign, Adobe, Marketo, Gainsight, Gong, Qualtrics, Kantata, TaskRay, etc.)
  • Proficient in DevOps processes and tools (e.g., Copado, AutoRabit)
  • Proficient in GITHUB and troubleshooting standard versioning and DevOps issues
  • Proficient in Agile methodologies (Scrum/Kanban) to iteratively deliver standard solutions
  • Solid experience with automated testing frameworks (e.g. PMD, Selenium, Copado CRT)
  • In-depth understanding of Salesforce data models, security, and governor limits
Job Responsibility
Job Responsibility
  • Work at the forefront of AI-driven Salesforce solutions and build AI Agents using Agentforce, and other AI platforms like Microsoft Studio for CoPilot Agents
  • Pioneer intelligent automations in enterprise environments using IPA tools like Power Automate, Alteryx, and others
  • Build test automations to improve QA efficiencies in QA for Salesforce and other applications
  • Collaborate with product designers, end-users, and other team members to refine and understand outcomes and AI use cases
  • Collaborate with other team members to brainstorm ideas for prototyping to test and solution for use cases identified and prioritized
  • Execute standard to complex tasks in the application development life cycle using languages like Apex, JavaScript, Python, etc., etc., relevant to the core applications (Salesforce, Billing Platform, etc.) and connected or integrated apps (Conga, Docusign, Adobe, Marketo, Gainsight, Gong, Qualtrics, Kantata, TaskRay, etc.)
  • Perform unit tests for your builds and collaborate with Product Designers and other team members to create accurate and quality test scripts for automated and manual testing
  • Analyze, troubleshoot, and debug systems for standard issues
  • Be an AI change agent in your team to gain team buy-in for adopting use of AI agents to save time, reduce waste and improve efficiency to deliver value to customers faster
  • Create and maintain technical documentation for all builds and AI Agents produced
What we offer
What we offer
  • Benefits starting from Day 1
  • Retirement Plan Matching
  • Flexible Paid Time Off
  • Wellness Support Programs and Resources
  • Parental & Caregiver Leaves
  • Fertility & Adoption Support
  • Continuous Development Support Program
  • Employee Assistance Program
  • Allyship and Inclusion Communities
  • Employee Recognition
  • Fulltime
Read More
Arrow Right

Senior Application Engineer (Salesforce Agentforce AI)

We’re looking for creative thinkers with hands-on experience in Agentforce, AI a...
Location
Location
Canada , Mississauga
Salary
Salary:
112000.00 - 125000.00 USD / Year
pointclickcare.com Logo
PointClickCare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands-on experience building Agentforce AI Agents, AI workflows, custom agent development, and integration of external data sources
  • Experience creating intelligent process automations using tools like Power Automate or others
  • Solid experience creating test automations for Salesforce and other platforms
  • Solid experience writing clean code that performs well at scale using JavaScript, Apex, Python, etc.
  • Solid experience with Salesforce integrations and external system integrations (Conga, Docusign, Adobe, Marketo, Gainsight, Gong, Qualtrics, Kantata, TaskRay, etc.)
  • Proficient in DevOps processes and tools (e.g., Copado, AutoRabit)
  • Proficient in GITHUB and troubleshooting standard versioning and DevOps issues
  • Proficient in Agile methodologies (Scrum/Kanban) to iteratively deliver standard solutions
  • Solid experience with automated testing frameworks (e.g. PMD, Selenium, Copado CRT)
  • In-depth understanding of Salesforce data models, security, and governor limits
Job Responsibility
Job Responsibility
  • Work at the forefront of AI-driven Salesforce solutions and build AI Agents using Agentforce, and other AI platforms like Microsoft Studio for CoPilot Agents
  • Pioneer intelligent automations in enterprise environments using IPA tools like Power Automate, Alteryx, and others
  • Build test automations to improve QA efficiencies in QA for Salesforce and other applications
  • Collaborate with product designers, end-users, and other team members to refine and understand outcomes and AI use cases
  • Collaborate with other team members to brainstorm ideas for prototyping to test and solution for use cases identified and prioritized
  • Execute standard to complex tasks in the application development life cycle using languages like Apex, JavaScript, Python, etc., etc., relevant to the core applications (Salesforce, Billing Platform, etc.) and connected or integrated apps (Conga, Docusign, Adobe, Marketo, Gainsight, Gong, Qualtrics, Kantata, TaskRay, etc.)
  • Perform unit tests for your builds and collaborate with Product Designers and other team members to create accurate and quality test scripts for automated and manual testing
  • Analyze, troubleshoot, and debug systems for standard issues
  • Be an AI change agent in your team to gain team buy-in for adopting use of AI agents to save time, reduce waste and improve efficiency to deliver value to customers faster
  • Create and maintain technical documentation for all builds and AI Agents produced
What we offer
What we offer
  • Benefits starting from Day 1
  • Retirement Plan Matching
  • Flexible Paid Time Off
  • Wellness Support Programs and Resources
  • Parental & Caregiver Leaves
  • Fertility & Adoption Support
  • Continuous Development Support Program
  • Employee Assistance Program
  • Allyship and Inclusion Communities
  • Employee Recognition
  • Fulltime
Read More
Arrow Right

QA Engineer - Agents & AI Platform

We are looking for a skilled and innovative AI / Machine Learning Engineer to de...
Location
Location
India , Chennai
Salary
Salary:
Not provided
optisolbusiness.com Logo
OptiSol Business Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong understanding of Machine Learning and Deep Learning fundamentals
  • Knowledge of Transformer architectures and language models
  • Experience working with Small Language Models (SLMs) or fine-tuned models
  • Hands-on experience with LangChain, LangGraph, CrewAI, AutoGen, OpenAI Agent SDK, or similar frameworks
  • Strong Python programming skills
  • Experience with FastAPI or similar backend frameworks
  • Knowledge of Relational and NoSQL databases
  • Familiarity with Git, Docker, and CI/CD pipelines
  • Experience with testing frameworks such as pytest or unittest
Job Responsibility
Job Responsibility
  • Design, develop, and deploy AI applications powered by Small Language Models (SLMs) and fine-tuned language models
  • Build and manage agent orchestration workflows using frameworks such as LangGraph,CrewAI, AutoGen, or OpenAI Agent SDK
  • Develop multi-agent systems that coordinate tasks through planning, reasoning, and tool interaction
  • Build Retrieval-Augmented Generation (RAG) pipelines integrating vector databases, APIs and enterprise data sources
  • Apply strong Python programming and data structure knowledge to build scalable AI systems
  • Develop backend services and APIs using Python frameworks such as FastAPI
  • Integrate AI solutions with relational or NoSQL databases and external services
  • Write clean, maintainable, and testable code following software engineering best practices
What we offer
What we offer
  • Supportive and professional work environment
  • Competitive salary as per market standards
  • Opportunity to work on advanced AI and multi-agent technologies
  • Career growth and learning opportunities in AI engineering
  • Fulltime
Read More
Arrow Right

Sr. Software Development Engineer

You will safeguard the quality of our AI and GenAI features by evaluating model ...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
highspot.com Logo
Highspot
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years of experience as a Software Development Engineer in AI/ML systems
  • Strong coding skills in Python (evaluation pipelines, data processing, metrics computation)
  • Hands-on experience with evaluation frameworks (Ragas or equivalent)
  • Knowledge of vector embeddings, similarity search, and RAG evaluation
  • Familiarity with evaluation metrics (precision, recall, F1, relevance, hallucination detection)
  • Understanding of LLM-as-a-judge evaluation approaches
  • Strong analytical and problem-solving skills
  • ability to combine human judgment with automated evaluations
  • Bachelor’s or Master’s degree in Computer Science, Data Science, or related field
  • Strong English written and verbal communication skills
Job Responsibility
Job Responsibility
  • Evaluation Frameworks – Develop reusable, automated evaluation pipelines using frameworks such as Raagas
  • integrate LLM-as-a-judge methods for scalable assessments
  • Golden Datasets – Build and maintain high-quality benchmark datasets in collaboration with subject matter experts
  • AI Output Validation – Evaluate results across text, documents, audio, and video, using both automated metrics and human-in-the-loop judgment
  • Metric Evaluation – Implement and track metrics such as precision, recall, F1 score, relevance scoring, and hallucination penalties
  • RAG & Embeddings – Design and evaluate retrieval-augmented generation (RAG) pipelines, vector embedding similarity, and semantic search quality
  • Error & Bias Analysis – Investigate recurring errors, biases, and inconsistencies in model outputs
  • propose solutions
  • Framework & Tooling Development – Build tools that enable large-scale model evaluation across hundreds of AI agents
  • Cross-Functional Collaboration – Partner with ML engineers, product managers, and QA peers to integrate evaluation frameworks into product pipelines
  • Fulltime
Read More
Arrow Right