CrawlJobs Logo

QA Engineer, Automation and Evals

aciinfotech.com Logo

ACI Infotech

Location Icon

Location:

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided

Job Description:

The QA Engineer, Automation and Evals will design, implement, and maintain automated test suites across APIs, UIs, and workflows to ensure product reliability and faster release cycles. This role involves building evaluation pipelines for AI and RAG systems, integrating tests into CI/CD pipelines, and driving higher test coverage in multi-tenant environments. The ideal candidate will combine strong technical testing skills with the ability to communicate risks and quality tradeoffs effectively.

Job Responsibility:

  • Design and maintain automated test suites across services and UI
  • Build evaluation pipelines for ML and RAG systems
  • Integrate quality gates into CI pipelines and monitor flake rate
  • Create and manage test data strategies for multi-tenant environments
  • Drive high automation coverage on APIs and UI
  • Reduce regression rates and improve release confidence

Requirements:

  • 3+ years of QA automation experience in SaaS environments
  • Strong skills with testing frameworks such as Playwright, Cypress, or PyTest
  • Familiarity with CI/CD test integration
  • Proven ability to design test strategies for APIs, UIs, and workflows
  • Experience building reliable test harnesses and CI integrations
  • Strong communication skills for articulating risk and quality tradeoffs
  • Ability to analyze logs and traces to triage failures quickly

Nice to have:

  • Knowledge of evals for AI and LLM applications
  • Experience in security or load testing
  • Background in regression testing for RAG pipelines
  • Test data management in multi-tenant environments

Additional Information:

Job Posted:
December 14, 2025

Employment Type:
Fulltime
Work Type:
Remote work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for QA Engineer, Automation and Evals

Senior AI Software Developer

The Senior AI Engineer owns end-to-end delivery of AI features—from design to pr...
Location
Location
Salary
Salary:
Not provided
https://www.hpe.com/ Logo
Hewlett Packard Enterprise
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or master’s degree in computer science, engineering, data science, machine learning, artificial intelligence, or closely related quantitative discipline
  • Typically, 7-10 years’ experience
  • LLMs & Agents: Prompt engineering, function/tool calling, orchestration frameworks, RAG
  • ML/DS: Evaluation metrics (precision/recall, BLEU/ROUGE where relevant), error analysis
  • Data/RAG: Embeddings, similarity (cosine/IP), chunking, rerankers, vector DB operations
  • Backend: Python (FastAPI/Flask), microservices patterns
  • MLOps/Infra: Docker, Kubernetes, CI/CD, artifact management, GPU scheduling
  • Observability: Metrics/logging/tracing, dashboards, automated evaluation pipelines
  • Frameworks: PyTorch/TensorFlow, Hugging Face, LangChain/LlamaIndex
  • Data: Pandas, SQL/NoSQL, Parquet/Arrow, Kafka/queues
Job Responsibility
Job Responsibility
  • Translate high-level designs into clear component contracts, APIs, and service boundaries
  • Implement LLM integrations, RAG pipelines, agents, tool/function calling, and prompt strategies
  • Own feature delivery for sprints/releases
  • maintain high code quality and documentation
  • Fine-tune models when needed
  • design evaluation harnesses and metrics
  • Build A/B testing setups
  • track accuracy, latency, robustness, and task success rates
  • Conduct error analysis
  • iterate using feedback efficacy loops and prompt refinement
What we offer
What we offer
  • Health & Wellbeing
  • Personal & Professional Development
  • Unconditional Inclusion
Read More
Arrow Right

LLM Inference Performance & Evals Engineer

Join the inference model team dedicated to bring up the state-of-the-art models,...
Location
Location
Canada , Toronto
Salary
Salary:
Not provided
cerebras.net Logo
Cerebras Systems
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years building high-performance ML or systems software
  • Solid grounding in Transformer math—attention scaling, KV-cache, quantisation—or clear evidence you learn this material rapidly
  • Comfort navigating the full AI toolchain: Python modeling code, compiler IRs, performance profiling, etc.
  • Strong debugging skills across performance, numerical accuracy, and runtime integration
  • Prior experience in modeling, compilers or crafting benchmarks or performance studies
  • not just black-box QA tests
  • Strong passion to leverage AI agents or workflow orchestration tools to boost personal productivity
Job Responsibility
Job Responsibility
  • Prototype and benchmark cutting-edge ideas: new attentions, MoE, speculative decoding, and many more innovations as they emerge
  • Develop agent-driven automation that designs experiments, schedules runs, triages regressions, and drafts pull-requests
  • Work closely with compiler, runtime, and silicon teams: unique opportunity to experience the full stack of software/hardware innovation
  • Keep pace with the latest open- and closed-source models
  • run them first on wafer scale to expose new optimization opportunities
What we offer
What we offer
  • Build a breakthrough AI platform beyond the constraints of the GPU
  • Publish and open source their cutting-edge AI research
  • Work on one of the fastest AI supercomputers in the world
  • Enjoy job stability with startup vitality
  • Our simple, non-corporate work culture that respects individual beliefs
Read More
Arrow Right

Provider Operations & Support

OpenRouter is an LLM marketplace that lets developers use frontier models in one...
Location
Location
United States
Salary
Salary:
Not provided
openrouter.ai Logo
OpenRouter
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2-3 years in a startup, solutions engineering, product ops, or similar
  • Comfortable building demos and scripts
  • can read API docs and troubleshoot with logs/cURL/Postman
  • Experience with TypeScript/JavaScript and/or Python
  • Git-literate
  • CS/Engineering degree is a plus, not required with demonstrated technical aptitude
Job Responsibility
Job Responsibility
  • Run end-to-end launch playbooks: scoping, test plans, latency/quality checks, pricing & quotas, docs, and announcement assets
  • Coordinate with model providers to integrate, QA, and hit ship dates
  • Maintain clear versioning and release notes
  • manage deprecations and migrations
  • Build internal tools to help speed up the model onboarding process
  • Build internal evals to evaluate models and endpoints quickly
  • Document internal processes and prime them for automation
What we offer
What we offer
  • Competitive compensation, benefits, and equity
  • Fulltime
Read More
Arrow Right
New

Healthcare Assistant & Lead Healthcare Assistant

Are you passionate about supporting older people and making a meaningful differe...
Location
Location
Ireland , Cork
Salary
Salary:
Not provided
jobs.360resourcing.co.uk Logo
360 Resourcing Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience in elderly care is desirable but not essential (training may be provided)
  • QQI Level 5 in Healthcare Support or equivalent is an advantage
  • A caring, patient and respectful approach to supporting older people
  • Ability to work flexible shifts including days, nights, and weekends
  • Strong commitment to resident safety, dignity and wellbeing
  • Applicants must have legal entitlement to work in Ireland
Job Responsibility
Job Responsibility
  • Deliver person-centred care to residents with dignity and respect
  • Support residents with activities of daily living where required
  • Encourage independence and social engagement
  • Maintain a clean, safe and comfortable living environment
  • Build positive relationships with residents, families, and colleagues
  • Follow all health, safety and safeguarding procedures
  • Assist with compliance processes, inspections, and quality standards
  • Lead Healthcare Assistants will also assist in guiding care teams, supporting best practice, and helping maintain high standards of care delivery
What we offer
What we offer
  • Flexible working hours to support work–life balance
  • Local employment opportunities within the Douglas, Cork area
  • Competitive rates of pay in line with experience and role
  • Supportive team environment with ongoing training and development
  • Employee wellbeing initiatives
  • Uniform provided
  • Free on-site parking
  • Meals provided while on duty
Read More
Arrow Right
New

Bakery Team Member

Join us to "Change Hospitality For Good". Here at the Cornish Bakery, we want to...
Location
Location
United Kingdom , Falmouth
Salary
Salary:
8.05 - 12.80 GBP / Hour
jobs.360resourcing.co.uk Logo
360 Resourcing Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Eligible to live and work in the UK
Job Responsibility
Job Responsibility
  • Make the World’s Best Coffee
  • Deliver Outstanding Customer Service
  • Prepare Award Winning Food
  • Maintain High standards within the Bakery
  • Be part of a Team who strive to Aim Higher at all times
What we offer
What we offer
  • 50% discount on food and drink
  • Pasty Perks – bespoke reward system offering discounts and savings on favourite brands
  • Financial well-being benefit through Stream – save directly from wages into a high interest savings account or access wages before pay-day
  • Employer funded health and wellbeing services with access to a 24/7 GP line, Employee Assistance Program and discounted gym memberships
  • Paid birthday day off
  • Member of the shareholder fund (service dependent)
  • Team social events with in-bakery incentives
  • Up to 28 days of holiday (including bank holidays), pro rata if necessary
Read More
Arrow Right
New

Housekeeping Assistant

In Glasgow's prestigious West End, Kelvinside Manor provides luxurious accommoda...
Location
Location
United Kingdom , Glasgow
Salary
Salary:
12.27 GBP / Hour
jobs.360resourcing.co.uk Logo
360 Resourcing Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience from a housekeeping or domestic cleaning role
  • Basic understanding of infection control, COSHH, and compliance standards
  • Ability to follow instructions and cleaning schedules
  • Ability to report issues
  • Ability to work well with others in a professional and courteous manner
  • Pride in your work, attention to detail, time management, and respectful communication
  • Ability to work well as part of a team or on your own
  • Demonstrating a kind and caring nature to all within the care home
  • Satisfactory references
  • Disclosure Scotland Scheme Membership (PVG)
Job Responsibility
Job Responsibility
  • Maintaining the highest levels of cleanliness in public areas and bedrooms within the home
  • Responsible for the wash, dry, iron and put away of laundry
  • Ensuring that our residents' care home facilities, public areas and rooms are maintained to the highest cleaning standards
  • Replenish and maintain supplies in both resident rooms and common areas
  • Communicate effectively with residents and colleagues to address needs and concerns
What we offer
What we offer
  • Competitive benefits
  • Overtime rates
  • Flexible working hours considered
  • Parttime
Read More
Arrow Right
New

Product Manager - Observability AIOps

We are looking for a Product Manager - Observability AIOps to lead the strategy,...
Location
Location
Salary
Salary:
Not provided
coca-colahellenic.com Logo
Coca-Cola HBC
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or Master’s in Computer Science, IT, or related field
  • 5+ years in Observability and AIOps within hybrid, complex environments
  • Strong knowledge of FSO (APM, logs, traces, RUM), SLI/SLOs, and service topology
  • Hands on with Dynatrace or similar smart observability (i.e. Datadog, Splunk, New Relic) and AIOps engines (correlation, deduplication, anomaly detection, auto remediation)
  • Experience integrating with ServiceNow, SAP, CI/CD pipelines, cloud services (Azure, AWS, GCP), and automation platforms
  • Solid understanding of modern application architectures, networks, databases, and infrastructure
  • Excellent communication and stakeholder management skills
  • Fluent in English
Job Responsibility
Job Responsibility
  • Define and execute the product vision and roadmap for Observability and AIOps, aligned with company objectives
  • Develop capabilities across metrics, logs, traces, RUM/synthetics, topology, and AI driven insights (anomaly detection, forecasting, correlation)
  • Prioritize features that enhance reliability, reduce MTTR, automate L1/L2 tasks, and improve customer experience
  • Collaborate with engineering, platform teams, and partners to ensure effective instrumentation, intelligent alerting, and automated workflows
  • Integrate Observability with ServiceNow, CI/CD, cloud platforms, and business processes to streamline operations
  • Leverage analytics to detect issues proactively, reduce noise, and optimize performance and resource usage
  • Champion an observability first and automation driven culture across teams
What we offer
What we offer
  • Coaching and mentoring programs
  • Development opportunities
  • Equal opportunity employer
  • IT Equipment
  • Work with iconic brands
  • Supportive team
Read More
Arrow Right
New

Reactive Plumbing Engineer

Metro Plumb is a national plumbing business with more than 40 local depots acros...
Location
Location
United Kingdom , Croydon
Salary
Salary:
33000.00 - 38000.00 GBP / Year
jobs.360resourcing.co.uk Logo
360 Resourcing Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience as a Plumbing/ Plumbing & Heating Engineer for both domestic and commercial customers
  • NVQ Level 2
  • Full UK driving license
  • Flexibility to work overtime
  • DBS checked (or willingness for this to be carried out)
Job Responsibility
Job Responsibility
  • Carry out all aspects of plumbing work, including but not limited to, installation and maintenance of toilets, taps, ball valves, siphons, flexible hoses, pipework, radiator valves, cylinders, heating systems etc across a range of domestic and commercial customers
  • Attend primarily emergency plumbing jobs relating to installations/leaks/issues
What we offer
What we offer
  • Personal development plan with timelines to achieve good OTE and growth
  • Auto enrolment into pension scheme
  • Company van and fuel card
  • Branded uniform
  • Company mobile
  • Personal development and genuine career opportunity
  • Induction and training
  • On-call rota in addition to salary
  • Optional overtime
  • Accommodating hours to support personal life
  • Fulltime
Read More
Arrow Right