Quality Assurance Automation Engineer - AI Job at OnePlan Solutions

Job Description

As a QA Automation Engineer - AI at OnePlan, you will play a critical role in ensuring the quality, reliability, and performance of our AI-powered features and capabilities. This role goes beyond writing test scripts. It’s about pioneering quality assurance practices for intelligent systems, defining how we validate non-deterministic AI outputs, and shaping the future of our product through rigorous automation and deep technical collaboration. You will work closely with AI engineers, product managers, and UX designers in a dynamic, agile environment to design, build, and scale automation frameworks that keep pace with our rapidly evolving AI capabilities. This is a deeply AI-focused role: the majority of your work will center on testing, evaluating, and improving AI systems, not general software, and requires both a quality engineering mindset and a genuine curiosity about how large language models and intelligent features behave in production.

Job Responsibility

Design, build, and maintain robust automation frameworks specifically tailored for testing AI features, including LLM-powered outputs, recommendation engines, and intelligent workflows
Develop and execute test strategies for validating AI model outputs, including accuracy, relevance, consistency, and bias evaluation across diverse input scenarios
Perform functional, regression, performance, and scalability testing on AI-driven features, including API-level testing of AI service integrations and end-to-end automated test suites
Collaborate with AI engineers, data scientists, product managers, and UX designers to define acceptance criteria and quality standards for AI features throughout the development lifecycle
Identify, document, and track defects in AI behavior, including edge cases, hallucinations, and unexpected model responses through resolution using appropriate tools and workflows
Champion quality and responsible AI practices throughout the development lifecycle, contributing to prompt engineering reviews, model evaluation rubrics, and AI safety checklists
Build and maintain CI/CD-integrated automation pipelines that continuously validate AI feature quality, using tools such as Playwright, Claude, or equivalent frameworks
Participate in team stand-ups, sprint planning, retrospectives, and continuous improvement activities, bringing an AI-quality-first mindset to every discussion and helping establish best practices for testing intelligent systems

Requirements

2-4 years of experience in QA automation engineering, with demonstrated hands-on work building and maintaining automated test frameworks
Proficiency with automation tools and frameworks (e.g., Playwright, Claude, or similar) and API testing tools (e.g., Postman, REST-assured)
Demonstrated experience testing AI/ML-integrated features or services, with a solid understanding of the unique challenges of validating non-deterministic outputs, including LLM responses, probabilistic recommendations, and generative content
Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience
Strong analytical and critical-thinking skills, with the ability to design test cases for complex, probabilistic AI behaviors
Excellent verbal and written communication skills, with the ability to clearly articulate AI quality issues and their business impact to both technical and non-technical stakeholders

Nice to have

Hands-on experience with LLM evaluation techniques, including prompt testing, output scoring, and benchmarking AI model responses against defined quality criteria
Familiarity with Microsoft Azure AI services, Azure OpenAI, and cloud-based AI infrastructure (e.g., Azure Machine Learning, Cognitive Services)
Experience with data-driven testing approaches, including generating and managing synthetic test datasets for training and validating AI model behavior at scale
Understanding of responsible AI principles, including fairness, transparency, and explainability, and experience incorporating these into QA testing practices
Proficiency JavaScript for scripting test utilities, data processing pipelines, and custom AI evaluation harnesses
Background in integrating automated QA into CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) with experience in shift-left testing and continuous quality monitoring for AI-driven products

What we offer

Comprehensive health, dental, and vision benefits, with additional insurance options
Employer RRSP and 401K matching programs
A fun, collaborative, and diverse environment with regular health and team challenges

OnePlan Solutions - All Job Offers

Select Country

Quality Assurance Automation Engineer - AI

Job Description

Job Responsibility

Requirements

Nice to have

What we offer

Looking for more opportunities?

Quality Assurance Automation Engineer - AI

Senior Test Automation & AI Quality Engineering Lead - VOIS

Quality Assurance Engineer II

Intern Quality Assurance Engineer

Quality Assurance Engineer II

Quality Assurance Engineer II

Quality Engineer – AI & Data Platforms

Senior Product Manager – Conversational AI Servicing Quality Assurance

Senior Quality Assurance / SDET Engineer, GenAI/AI Platform

Our AI answers in your language