CrawlJobs Logo

QA Engineer, Automation and Evals

aciinfotech.com Logo

ACI Infotech

Location Icon

Location:

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided
Save Job
Save Icon
Apply Position

Job Description:

The QA Engineer, Automation and Evals will design, implement, and maintain automated test suites across APIs, UIs, and workflows to ensure product reliability and faster release cycles. This role involves building evaluation pipelines for AI and RAG systems, integrating tests into CI/CD pipelines, and driving higher test coverage in multi-tenant environments. The ideal candidate will combine strong technical testing skills with the ability to communicate risks and quality tradeoffs effectively.

Job Responsibility:

  • Design and maintain automated test suites across services and UI
  • Build evaluation pipelines for ML and RAG systems
  • Integrate quality gates into CI pipelines and monitor flake rate
  • Create and manage test data strategies for multi-tenant environments
  • Drive high automation coverage on APIs and UI
  • Reduce regression rates and improve release confidence

Requirements:

  • 3+ years of QA automation experience in SaaS environments
  • Strong skills with testing frameworks such as Playwright, Cypress, or PyTest
  • Familiarity with CI/CD test integration
  • Proven ability to design test strategies for APIs, UIs, and workflows
  • Experience building reliable test harnesses and CI integrations
  • Strong communication skills for articulating risk and quality tradeoffs
  • Ability to analyze logs and traces to triage failures quickly

Nice to have:

  • Knowledge of evals for AI and LLM applications
  • Experience in security or load testing
  • Background in regression testing for RAG pipelines
  • Test data management in multi-tenant environments

Additional Information:

Job Posted:
December 14, 2025

Employment Type:
Fulltime
Work Type:
Remote work
Icon
View All Jobs In This Company
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for QA Engineer, Automation and Evals

New

Senior Pricing Analyst

An exciting opportunity has arisen to join Veygo, part of Admiral Pioneer, as a ...
Location
Location
United Kingdom , Cardiff
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2+ years of experience in a General Insurance or Financial Services Pricing department, or a strong background in data and analytics
  • Solid understanding of pricing methodologies and insurance data analytics
  • Proficiency in analytical tools such as SAS, SQL, or similar
  • Strong communication skills, with the ability to present confidently to a range of stakeholders
  • Proactive and self-motivated, with a drive to move projects forward
  • Comfortable working in a dynamic, fast-paced environment that values innovation
Job Responsibility
Job Responsibility
  • Develop and implement innovative pricing strategies that drive profitability, competitiveness, and customer value
  • Translate complex data into actionable insights, identifying trends and opportunities to inform strategic decisions
  • Support the technical rollout, testing, and validation of pricing changes
  • Create impactful reports and dashboards to monitor performance and guide business actions
  • Partner with cross-functional teams and effectively communicate insights to influence decision-making
  • Monitor industry trends, competitor activity, and market developments to ensure pricing remains competitive and compliant
  • Take ownership of projects and develop leadership capabilities with opportunities for progression
What we offer
What we offer
  • Attractive salary, bonus opportunities, and comprehensive benefits
  • Up to £3,600 of free shares each year after one year of service
  • 33 days holiday (including bank holidays) when they join us, increasing the longer you stay with us, up to a maximum of 38 days (including bank holidays)
  • Option to buy or sell up to an additional five days of annual leave
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Fulltime
Read More
Arrow Right
New

Pricing Analyst

At Veygo our world is learner and temporary pay-as-you-go car insurance. We know...
Location
Location
United Kingdom , Cardiff
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Ideally a degree of 2:1 (or equivalent) in a STEM or strong numeric discipline
  • A numerate individual ready to roll up their sleeves and dive into data analysis
  • A problem solver who thrives on challenging projects and isn't afraid to get stuck in
  • A willingness to learn and master programming and data manipulation to handle large datasets effectively
  • Strong numerical skills
  • Eager to learn programming with tools like SQL, Python, and other data analytics software
  • Adaptable
  • Proactive
  • Ready to own your work from day one
  • Thrives in a fast-paced, dynamic environment
Job Responsibility
Job Responsibility
  • Competitor benchmarking
  • Pricing reviews
  • Customer segmentation analysis to uncover actionable insights
  • Contribute to developing pricing strategies
  • Managing Veygo’s customer portfolio
  • Delivering pricing reports
  • Interpreting complex data and recommending pricing adjustments
  • Tackle complex pricing challenges while keeping the customer at the heart of every decision
  • Helping redefine the profile of traditional pricing teams
What we offer
What we offer
  • 33 days holiday (including bank holidays) when they join us, increasing the longer you stay with us, up to a maximum of 38 days (including bank holidays)
  • Option to buy or sell up to an additional five days of annual leave
  • Eligible for up to £3,600 of free shares each year after one year of service
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Annual Holiday Allowance
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Fulltime
Read More
Arrow Right
New

Mortgage Underwriter Team Leader

Admiral Mortgages are looking for a Mortgage Underwriter Team Leader to join the...
Location
Location
United Kingdom , Cardiff
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Preferably CEMAP qualified
  • Proven experience as a mortgage underwriter with in-depth knowledge of Second Charge or Specialist lending
  • Previous experience in a leadership or supervisory role within financial services or homeowner loan underwriting
  • Strong understanding of credit risk assessment, affordability, and regulatory requirements (FCA, PRA, AML, GDPR, etc.)
  • Excellent analytical and decision-making skills with strong attention to detail
  • Effective communicator with the ability to influence and support team members at all levels
  • Highly organised with the ability to manage workloads, prioritise tasks, and deliver against deadlines
Job Responsibility
Job Responsibility
  • Lead, motivate, and develop a team of mortgage underwriters through coaching, mentoring, regular one-to-ones, and ongoing training
  • Review and authorise complex homeowner loan applications in line with credit policy, risk frameworks, and regulatory standards
  • Act as a referral point for escalated or challenging cases, providing clear guidance and sound decision-making
  • Support the Underwriting Manager in shaping and improving underwriting policies, procedures, and risk controls
  • Collaborate with colleagues across Sales, Risk, Compliance, and Servicing to resolve queries, enhance processes, and deliver an excellent customer experience
  • Ensure compliance with FCA, PRA, and other relevant regulatory requirements, maintaining up-to-date knowledge of industry trends and changes
  • Foster a positive team culture that values integrity, accountability, and continuous improvement
What we offer
What we offer
  • Up to £3,600 of free shares each year after one year of service
  • 33 days holiday (including bank holidays) when they join us, increasing the longer you stay with us, up to a maximum of 38 days (including bank holidays)
  • Option to buy or sell up to an additional five days of annual leave
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Annual Holiday Allowance
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Fulltime
Read More
Arrow Right
New

Business Development Manager

This position serves as a pivotal role in sales and relationship management, wor...
Location
Location
United Kingdom , Cardiff
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong knowledge of UK specialist lending landscape and regulation
  • Experience of the homeowner loans market desirable but not essential
  • Commercially astute with a data-driven, results-oriented mindset
  • Willingness to travel within a designated region of the UK and to Cardiff HQ as required
Job Responsibility
Job Responsibility
  • Develop strong, lasting partnerships with brokers and key stakeholders
  • Understanding brokers individual business needs, addressing their queries promptly, and proactively offering guidance
  • Attend broker and industry events to promote Admiral’s offerings and stay abreast of market developments
  • Monitor and enforce compliance with all Broker Oversight requirements
  • Collaborate with Admiral’s operational, marketing and product teams to align on pipeline management, campaign execution, and new product launches
  • Ensure that feedback from brokers is channelled back to the relevant teams
  • Gathering and monitor market trends, competitor activity, and regulatory changes to inform strategy and participate in trading meetings
  • Work closely with the Admiral’s operational teams to assess their requirements and understand their insights
  • Leverage data and insights from Sales Ops to optimise sales processes
  • Set clear goals, KPIs, and frameworks to drive accountability and performance within brokers
What we offer
What we offer
  • Up to £3,600 of free shares each year after one year of service
  • 33 days holiday (including bank holidays) when they join us, increasing the longer you stay with us, up to a maximum of 38 days (including bank holidays)
  • Option to buy or sell up to an additional five days of annual leave
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Annual Holiday Allowance
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Fulltime
Read More
Arrow Right
New

Qualified Solicitor

An exciting opportunity for qualified solicitors to join a growing team within a...
Location
Location
United Kingdom , Cardiff
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Qualified Solicitor
  • Managing a case load
  • Achieving Performance Targets (KPIs)
  • Setting strategy for assistant litigators
  • Mentoring junior case handlers
  • Supporting the wider firm/other departments
  • Excellent time management skills
  • Ability to manage and prioritise tasks
  • Ability to meet clients’ expectations
Job Responsibility
Job Responsibility
  • Manage a case load
  • Support other litigation teams in ensuring compliance with regulatory obligations and the Legal Service Act 2007
  • Set strategy for assistant litigators
  • Mentor junior case handlers
  • Support the wider firm/other departments
What we offer
What we offer
  • 33 days holiday (including bank holidays) when joining, increasing up to 38 days
  • Option to buy or sell up to an additional five days of annual leave
  • Eligible for up to £3,600 of free shares each year after one year of service
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Critical Illness Cover
  • Fulltime
Read More
Arrow Right
New

Solution Architect - Gen AI & Integration

The Solution Architects are part of the Architecture practice within the CTO org...
Location
Location
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience with anyone API Management & Gateway platform
  • Experience with one or more major Integration Platform such as TIBCO, MuleSoft, Dell Boomi, Azure Integration Services etc.
  • Experience in designing data retrieval pipelines (RAG), implementing autonomous LLM orchestration frameworks, and Model Evaluation Services (MES) frameworks
  • Experience in Prompt Management - Strategic approach to prompt versioning, testing, and lifecycle management within production environment
  • Strong knowledge of integration security best practices
  • Experience with data exchange formats and transformation techniques
  • Experience in designing RESTFul APIs and Event-Driven Architectures
  • Experience of applying integration pattern and strong knowledge of architecture styles
  • Understanding of software delivery pipelines and how solution fits into CI/CD practices
  • Experience designing and deploying integration solutions within atleast one major cloud environments
Job Responsibility
Job Responsibility
  • Together with Lead Solution Architects, provide strong technical solutions for value streams, own those solutions from ideation to delivery, and ensure that business value can be realised through the correct implementation of the solutions
  • Lead and contribute to governance forums, ensuring solutions adhere to established standards and patterns, identifying and controlling divergences, and escalating where necessary
  • Build productive relationships with technical and business stakeholders to maintain alignment of technology and business strategies
  • Contribute to and help maintain a detailed understanding of the architecture for the value stream
  • Produce documentation of a high-quality and to a detailed level, ensuring that all decision points, building blocks, principles, rationale, roadmaps, and all interdependencies are understood, documented, and communicated across value streams
  • Contribute to the architecture repository using the Admiral architecture modelling tools and language
  • Write, or contributes to, articles, and papers and speak at conferences, user groups, or specialist subject groups on topics involved in Architecture. Play a leading role in special interest groups concerned with specialist field
  • Promote Architecture within Admiral and closely associated organisations
  • Support innovation by research, create and communicate reports, strategies and industry insights about emerging technologies, products, services, methods and techniques and their potential impacts, threats, and opportunities
  • Keep the Head of Solution Architecture and the CTO management team informed and escalate where appropriate
What we offer
What we offer
  • Everyone receives 33 days holiday (including bank holidays) when they join us, increasing the longer you stay with us, up to a maximum of 38 days (including bank holidays). You also have the option to buy or sell up to an additional five days of annual leave
  • eligible for up to £3,600 of free shares each year after one year of service
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Annual Holiday Allowance
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Critical Illness Cover
  • Fulltime
Read More
Arrow Right
New

Solution Architect - Claims

The Solution Architects are part of the Architecture practice within the CTO org...
Location
Location
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Azure Experience (BLOB storage, APIM, Key Vault, AKS, Event Grid, Logic Apps etc.)
  • Recognised expertise in Architecture, the technology involved, current developments, the most commonly available products, and the application of Architecture within a technical or business environment
  • Take innovative approaches to problem solving and/or devising inventive and creative solutions
  • Stakeholder management, ability to communicate at all levels to ensure business and technical objectives are met
  • Keep organisational objectives and strategies in mind and ensuring courses of action are aligned with the strategic context
  • Understand commercial considerations and ensure solutions reflect them when making decisions or recommending actions
  • Proactive approach, anticipating opportunities for improvement or development and taking appropriate action(s)
Job Responsibility
Job Responsibility
  • Together with Lead Solution Architects, provide strong technical solutions for value streams, own those solutions from ideation to delivery, and ensure that business value can be realised through the correct implementation of the solutions
  • Lead and contribute to governance forums, ensuring solutions adhere to established standards and patterns, identifying and controlling divergences, and escalating where necessary
  • Build productive relationships with technical and business stakeholders to maintain alignment of technology and business strategies
  • Contribute to and help maintain a detailed understanding of the architecture for the value stream
  • Produce documentation of a high-quality and to a detailed level, ensuring that all decision points, building blocks, principles, rationale, roadmaps, and all interdependencies are understood, documented, and communicated across value streams
  • Contribute to the architecture repository using the Admiral architecture modelling tools and language
  • Write, or contributes to, articles, and papers and speak at conferences, user groups, or specialist subject groups on topics involved in Architecture. Play a leading role in special interest groups concerned with specialist field
  • Promote Architecture within Admiral and closely associated organisations
  • Support innovation by research, create and communicate reports, strategies and industry insights about emerging technologies, products, services, methods and techniques and their potential impacts, threats, and opportunities
  • Keep the Head of Solution Architecture and the CTO management team informed and escalate where appropriate
What we offer
What we offer
  • Everyone receives 33 days holiday (including bank holidays) when they join us, increasing the longer you stay with us, up to a maximum of 38 days (including bank holidays). You also have the option to buy or sell up to an additional five days of annual leave
  • eligible for up to £3,600 of free shares each year after one year of service
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Annual Holiday Allowance
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Critical Illness Cover
  • Fulltime
Read More
Arrow Right
New

Claims Handler

We have an exciting opportunity to join our Travel Claims team as a Claims Handl...
Location
Location
United Kingdom , Cardiff
Salary
Salary:
Not provided
admiralgroup.co.uk Logo
Admiral Group Plc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Ability to work independently and as part of a team
  • Claims experience is desirable but not essential
  • Be accountable and take responsibility
  • Adhering to core competencies
  • Sharing experiences and knowledge
  • Positive attitude, approachable, helpful and friendly
  • Effective time management
  • Excellent customer service skills, exceeding expectations
Job Responsibility
Job Responsibility
  • Registering new claims and making decisions on whether a claim will be covered
  • Managing existing case load
  • Maintaining SLA 5-7 days settlement
  • Thorough investigation of claim details and documentation
  • Keeping the customer updated throughout the claims lifecycle
  • Maintain excellent communication with customers and colleagues
  • Put forward ideas for suggestions for improvement for the team and wider department
  • Working in line with department KPI’s
  • Correctly identifying prospects for recovery and pro-active pursuit (dual insurers or liable parties)
  • Correctly identifying potentially fraudulent claims
What we offer
What we offer
  • Everyone receives 33 days holiday (including bank holidays) when they join us, increasing the longer you stay with us, up to a maximum of 38 days (including bank holidays). You also have the option to buy or sell up to an additional five days of annual leave
  • Eligible for up to £3,600 of free shares each year after one year of service
  • Financial & Mortgage Advice
  • 24-Hour Ecare
  • Cycle to Work Scheme
  • Annual Holiday Allowance
  • Flexible Working
  • Simply Health
  • Private Health Cover
  • Critical Illness Cover
  • Fulltime
Read More
Arrow Right
Welcome to CrawlJobs.com –
Your Global Job Discovery Platform
At CrawlJobs.com, we simplify finding your next career opportunity by bringing job listings directly to you from all corners of the web. Using cutting-edge AI and web-crawling technologies, we gather and curate job offers from various sources across the globe, ensuring you have access to the most up-to-date job listings in one place.
Jobs by Category
Art and Entertainment Jobs Banking Jobs Call Center Jobs Construction Jobs Consulting Jobs Customer Service Jobs Education Jobs Energy Jobs Finance Jobs Gastronomy Jobs Health and Beauty Jobs Hospitality and Tourism Jobs Human Resources Jobs Insurance Jobs IT - Administration Jobs IT - Software Development Jobs Legal Jobs Logistics Jobs Manufacturing Jobs Marketing, Digital Marketing, SEO, SEM Jobs Media Jobs Nursing Jobs Office Administration Jobs Pharmacy Jobs Physical Work Jobs Public Relations Jobs Public Sector Jobs Purchasing Jobs Quality Control Jobs Real Estate Jobs Research and Development Jobs Sales Jobs
Job Positions Jobs by Country
United States Jobs United Kingdom Jobs India Jobs Australia Jobs Canada Jobs Poland Jobs Ireland Jobs Spain Jobs
Company
Blog About Us Contact
Follow Us
Facebook
X (Twitter)
LinkedIn
Instagram
© 2025 CrawlJobs Ltd. All Rights Reserved
Terms & Conditions Privacy Policy