CrawlJobs Logo

QA Engineer, Automation and Evals

aciinfotech.com Logo

ACI Infotech

Location Icon

Location:

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided
Save Job
Save Icon
Apply Position

Job Description:

The QA Engineer, Automation and Evals will design, implement, and maintain automated test suites across APIs, UIs, and workflows to ensure product reliability and faster release cycles. This role involves building evaluation pipelines for AI and RAG systems, integrating tests into CI/CD pipelines, and driving higher test coverage in multi-tenant environments. The ideal candidate will combine strong technical testing skills with the ability to communicate risks and quality tradeoffs effectively.

Job Responsibility:

  • Design and maintain automated test suites across services and UI
  • Build evaluation pipelines for ML and RAG systems
  • Integrate quality gates into CI pipelines and monitor flake rate
  • Create and manage test data strategies for multi-tenant environments
  • Drive high automation coverage on APIs and UI
  • Reduce regression rates and improve release confidence

Requirements:

  • 3+ years of QA automation experience in SaaS environments
  • Strong skills with testing frameworks such as Playwright, Cypress, or PyTest
  • Familiarity with CI/CD test integration
  • Proven ability to design test strategies for APIs, UIs, and workflows
  • Experience building reliable test harnesses and CI integrations
  • Strong communication skills for articulating risk and quality tradeoffs
  • Ability to analyze logs and traces to triage failures quickly

Nice to have:

  • Knowledge of evals for AI and LLM applications
  • Experience in security or load testing
  • Background in regression testing for RAG pipelines
  • Test data management in multi-tenant environments

Additional Information:

Job Posted:
December 14, 2025

Employment Type:
Fulltime
Work Type:
Remote work
Icon
ACI Infotech - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for QA Engineer, Automation and Evals

Sr. SDET

As a Sr. SDET in Agentic QA, you will own the test automation and quality framew...
Location
Location
Canada , Vancouver
Salary
Salary:
150500.00 - 175250.00 CAD / Year
dialpad.com Logo
Dialpad
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of experience in software engineering or SDET roles with an emphasis on software development
  • Strong programming skills in Python (preferred), Java, or JavaScript
  • Experience testing distributed, cloud-native SaaS systems and APIs
  • Demonstrated proficiency in coding with AI agents to accelerate development and improve code quality
  • Hands-on exposure to LLMs or AI/ML systems (e.g., OpenAI, Claude, Gemini, or similar platforms)
  • Understanding of non-deterministic systems and probabilistic testing approaches
  • Experience building test frameworks and scalable automation systems
  • Familiarity with AI evaluation techniques (benchmarking, golden datasets, human-in-the-loop validation)
  • Experience with CI/CD pipelines (e.g., Jenkins, GitHub Actions)
  • Strong collaboration skills with the ability to work across distributed teams and time zones
Job Responsibility
Job Responsibility
  • Own end-to-end quality for agentic features and workflows, including strategy, development, execution, and release qualification
  • Design and build automation tooling and frameworks for AI/LLM-driven systems, including prompt flows, agent orchestration, and tool integrations
  • Develop and maintain evaluation frameworks (evals) to measure response quality, accuracy, and hallucination rates
  • Drive automation coverage (80%+ for critical AI workflows) using deterministic + probabilistic validation approaches
  • Integrate AI quality checks into CI/CD pipelines with fast feedback cycles (<15 minutes for PR validation)
  • Build tooling for LLM observability and debugging, including prompt tracing and response analysis
  • Partner with Applied AI teams on prompt engineering, model selection, and evaluation strategies
  • Design and execute performance and load tests for AI services (latency, throughput, cost efficiency)
  • Identify and mitigate risks related to hallucinations, bias, safety, and edge cases
  • Define and track AI quality KPIs (task success rates, precision/recall, latency, etc.)
What we offer
What we offer
  • Competitive salary, comprehensive benefits, and real opportunities for growth
  • Fulltime
Read More
Arrow Right

Senior AI Software Developer

The Senior AI Engineer owns end-to-end delivery of AI features—from design to pr...
Location
Location
Salary
Salary:
Not provided
https://www.hpe.com/ Logo
Hewlett Packard Enterprise
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or master’s degree in computer science, engineering, data science, machine learning, artificial intelligence, or closely related quantitative discipline
  • Typically, 7-10 years’ experience
  • LLMs & Agents: Prompt engineering, function/tool calling, orchestration frameworks, RAG
  • ML/DS: Evaluation metrics (precision/recall, BLEU/ROUGE where relevant), error analysis
  • Data/RAG: Embeddings, similarity (cosine/IP), chunking, rerankers, vector DB operations
  • Backend: Python (FastAPI/Flask), microservices patterns
  • MLOps/Infra: Docker, Kubernetes, CI/CD, artifact management, GPU scheduling
  • Observability: Metrics/logging/tracing, dashboards, automated evaluation pipelines
  • Frameworks: PyTorch/TensorFlow, Hugging Face, LangChain/LlamaIndex
  • Data: Pandas, SQL/NoSQL, Parquet/Arrow, Kafka/queues
Job Responsibility
Job Responsibility
  • Translate high-level designs into clear component contracts, APIs, and service boundaries
  • Implement LLM integrations, RAG pipelines, agents, tool/function calling, and prompt strategies
  • Own feature delivery for sprints/releases
  • maintain high code quality and documentation
  • Fine-tune models when needed
  • design evaluation harnesses and metrics
  • Build A/B testing setups
  • track accuracy, latency, robustness, and task success rates
  • Conduct error analysis
  • iterate using feedback efficacy loops and prompt refinement
What we offer
What we offer
  • Health & Wellbeing
  • Personal & Professional Development
  • Unconditional Inclusion
Read More
Arrow Right
New

Manager, Content Engineering — AI Content Understanding

Product Content Engineering is a horizontal function supporting initiatives acro...
Location
Location
United States , Menlo Park
Salary
Salary:
162000.00 - 227000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in content strategy, content operations, AI evaluation, or a related field
  • 2+ years of people management experience, including hiring, developing, and performance-managing direct reports
  • Experience managing cross-functional programs with engineering, data science, and product partners in fast-paced environments
  • 1+ years working with generative AI products, AI evaluation, prompt engineering, annotation, and/or content labeling and analysis
  • Experience designing or operationalizing evaluation frameworks, annotation guidelines, or quality rubrics for AI/ML systems
  • Demonstrated ability to manage multiple concurrent workstreams with competing priorities and tight deadlines
  • Proven analytical skills with experience interpreting evaluation data and communicating findings to technical and non-technical audiences
  • Track record of building team operational processes and quality standards from the ground up or during periods of significant change
Job Responsibility
Job Responsibility
  • Manage and develop a team of Content Engineers and contingent workers, setting clear goals, providing regular feedback, and supporting career growth
  • Own the execution of continuous CU model evaluations — coordinating sprint planning, reviewer assignments, QA processes, and delivery timelines across multiple concurrent workstreams
  • Drive the creation and maintenance of golden datasets that serve as ground truth for model benchmarking and auto-eval calibration
  • Partner with engineering, data science, and product teams to translate evaluation insights into actionable recommendations for model improvement and prompt optimization
  • Lead the team's contribution to LLM-as-a-Judge (auto-eval) initiatives — ensuring human evaluation data is used to calibrate, validate, and improve automated evaluation systems
  • Define and maintain evaluation guidelines, rubrics, and quality standards in partnership with Lead Content Engineers, ensuring consistency across reviewers and use cases
  • Build repeatable operational processes for evaluation sprints, including reviewer training, calibration sessions, and escalation workflows
  • Manage CW workforce planning — hiring, onboarding, allocation across workstreams, and performance management
  • Synthesize evaluation results into structured reports and present findings to cross-functional leadership, including engineering leads and lead product stakeholders
  • Identify and mitigate operational risks — staffing gaps, timeline conflicts, quality regressions — before they impact delivery
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right

LLM Inference Performance & Evals Engineer

Join the inference model team dedicated to bring up the state-of-the-art models,...
Location
Location
Canada , Toronto
Salary
Salary:
Not provided
cerebras.net Logo
Cerebras Systems
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years building high-performance ML or systems software
  • Solid grounding in Transformer math—attention scaling, KV-cache, quantisation—or clear evidence you learn this material rapidly
  • Comfort navigating the full AI toolchain: Python modeling code, compiler IRs, performance profiling, etc.
  • Strong debugging skills across performance, numerical accuracy, and runtime integration
  • Prior experience in modeling, compilers or crafting benchmarks or performance studies
  • not just black-box QA tests
  • Strong passion to leverage AI agents or workflow orchestration tools to boost personal productivity
Job Responsibility
Job Responsibility
  • Prototype and benchmark cutting-edge ideas: new attentions, MoE, speculative decoding, and many more innovations as they emerge
  • Develop agent-driven automation that designs experiments, schedules runs, triages regressions, and drafts pull-requests
  • Work closely with compiler, runtime, and silicon teams: unique opportunity to experience the full stack of software/hardware innovation
  • Keep pace with the latest open- and closed-source models
  • run them first on wafer scale to expose new optimization opportunities
What we offer
What we offer
  • Build a breakthrough AI platform beyond the constraints of the GPU
  • Publish and open source their cutting-edge AI research
  • Work on one of the fastest AI supercomputers in the world
  • Enjoy job stability with startup vitality
  • Our simple, non-corporate work culture that respects individual beliefs
Read More
Arrow Right

Provider Operations & Support

OpenRouter is an LLM marketplace that lets developers use frontier models in one...
Location
Location
United States
Salary
Salary:
Not provided
openrouter.ai Logo
OpenRouter
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2-3 years in a startup, solutions engineering, product ops, or similar
  • Comfortable building demos and scripts
  • can read API docs and troubleshoot with logs/cURL/Postman
  • Experience with TypeScript/JavaScript and/or Python
  • Git-literate
  • CS/Engineering degree is a plus, not required with demonstrated technical aptitude
Job Responsibility
Job Responsibility
  • Run end-to-end launch playbooks: scoping, test plans, latency/quality checks, pricing & quotas, docs, and announcement assets
  • Coordinate with model providers to integrate, QA, and hit ship dates
  • Maintain clear versioning and release notes
  • manage deprecations and migrations
  • Build internal tools to help speed up the model onboarding process
  • Build internal evals to evaluate models and endpoints quickly
  • Document internal processes and prime them for automation
What we offer
What we offer
  • Competitive compensation, benefits, and equity
  • Fulltime
Read More
Arrow Right
New

Pharmacy Technician

We’re building a world of health around every individual — shaping a more connec...
Location
Location
United States , Drexel Hill
Salary
Salary:
Not provided
https://www.cvshealth.com/ Logo
CVS Health
Expiration Date
June 22, 2026
Flip Icon
Requirements
Requirements
  • Must comply with any state board of pharmacy requirements or laws governing the practice of pharmacy, which includes but is not limited to, age, education, and licensure/certification
  • If the state board of pharmacy does not address or mandate a minimum age requirement, must be at least 16 years of age
  • If the state board of pharmacy does not address or mandate a minimum educational requirement, must have a high school diploma or equivalent, or be actively enrolled in high school or high school equivalency program
  • State-level licensure and national certification requirements vary by state, click here to learn more
  • Regular and predictable attendance, including nights and weekends
  • Ability to complete required training within designated timeframe
  • Attention and Focus
  • Customer Service and Team Orientation
  • Communication Skills
  • Mathematical Reasoning
Job Responsibility
Job Responsibility
  • Living our purpose by following all company SOPs at each workstation to help our Pharmacists manage and improve patient health
  • Following pharmacy workflow procedures at each pharmacy workstation (i.e., production, pick-up, drive-thru, and drop-off) for safe and accurate prescription fulfillment
  • Contributing to positive patient experiences by showing empathy and genuine care
  • Completing basic inventory activities, as permitted by law, and as directed by the pharmacy leadership team
  • Contributing to a high-performing team, embracing a growth mindset, and being receptive to feedback
  • Remaining flexible for both scheduling and business needs, while contributing to a safe, inclusive, and engaging team dynamic
  • Understanding and complying with all relevant federal, state, and local laws, regulations, professional standards, and ethical principles
  • Delivering additional patient health care services (e.g., immunizations, point-of-care testing, and voluntarily staffing offsite clinics), where allowable by law and supported by required training and certification
  • Where permissible, the Pharmacy Technician may also support immunizations, which includes the following responsibilities: Completing additional licensure and training requirements, in compliance with state Board of Pharmacy regulations, to obtain Technician Immunizer status to support preparing and administering vaccines
  • Educating patients about the importance of vaccines and referring patients to the Pharmacist-on-duty for vaccination questions
What we offer
What we offer
  • medical, dental, and vision coverage
  • paid time off
  • retirement savings options
  • wellness programs
  • and other resources, based on eligibility
  • Fulltime
Read More
Arrow Right
New

Mri Technologist

MedPro Healthcare Staffing, a Joint Commission-certified staffing agency, is see...
Location
Location
United States , Springfield
Salary
Salary:
Not provided
medprostaffing.com Logo
MedPro Healthcare Staffing
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Completion of a two year approved School of Radiologic Technology
  • Valid state radiology registration as required by state law
  • Registry by the American Registry of Radiologic Technology.
Job Responsibility
Job Responsibility
  • MRI technologists utilize their knowledge of anatomy, physiology and the principles of MRI to safely and efficiently operate MRI scanners, assisting in the diagnosis of disease and injury.
  • Ensure the safety of patients, staff and visitors who come in contact with the powerful magnetic field of a MRI scanner.
  • Position patients and coils on a table that slides inside the MRI scanner.
  • Inject contrast media as required.
  • Set appropriate technical parameters, operate MRI scanners and related equipment, and observe image data on computer monitors during scans.
  • Be familiar with the differences from a normal image and an abnormal image.
  • Recognize and respond to life threatening situations.
  • Assure compliance with federal, state, and local technical and professional regulations and accepted practiced guidelines.
  • Delivers quality, cost effective patient care in a professional manner.
  • Works effectively to maintain an environment of excellence, which is patient focused, providing timely, compassionate, quality patient care.
What we offer
What we offer
  • Weekly pay and direct deposit
  • Full coverage of all credentialing fees
  • Private housing or housing allowance
  • Group Health insurance for you and your family
  • Company-paid life and disability insurance
  • Travel reimbursement
  • 401(k) matching
  • Unlimited Referral Bonuses up to $1,000
  • Fulltime
Read More
Arrow Right
New

Pharmacy Technician

We're building a world of health around every individual — shaping a more connec...
Location
Location
United States , Oakville
Salary
Salary:
18.94 - 28.94 USD / Hour
https://www.cvshealth.com/ Logo
CVS Health
Expiration Date
June 23, 2026
Flip Icon
Requirements
Requirements
  • Must be at least 16 years of age
  • Licensure requirements vary by state
  • Attention and Focus
  • Customer Service Orientation
  • Communication Skills
  • Mathematical Reasoning
  • Problem Resolution
  • Physical Demands
  • Visual Acuity
  • High School diploma or equivalent preferred but not required.
Job Responsibility
Job Responsibility
  • manage all assigned pharmacy workstations and tasks to support the team's ability to promptly, safely and accurately fill patient prescriptions
  • provide caring service that exceeds customer expectations
What we offer
What we offer
  • dental
  • vision
  • wellness resources
  • employee discounts
  • access to certain voluntary benefits
  • Parttime
Read More
Arrow Right