CrawlJobs Logo

Quality Assurance Analyst, AI Delivery

cresta.com Logo

Cresta

Location Icon

Location:
United States

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

50000.00 - 70000.00 USD / Year
Save Job
Save Icon
Job offer has expired

Job Description:

We’re looking for a Quality Assurance Analyst to ensure Cresta’s AI systems perform with accuracy, fairness, and compliance. You’ll audit AI-generated conversations, evaluate model behavior, and deliver feedback that improves model performance and customer trust.

Job Responsibility:

  • Review AI-generated summaries for accuracy and completeness
  • provide feedback to improve models
  • Conduct behavioral audits to verify agent triggers and responses match expected outcomes
  • Perform redaction audits to protect sensitive customer data
  • Run bias audits to detect and document fairness issues
  • Compare audio and text for transcription accuracy
  • Support Opera platform maintenance and rule configuration to keep AI outputs accurate

Requirements:

  • 1–3 years in QA or data labeling
  • Exceptional attention to detail and analytical rigor
  • Strong documentation and communication skills
  • Comfort with accuracy-driven tasks

Nice to have:

  • Familiarity with conversational AI or annotation tools
  • Experience with bias / redaction audits or speech-to-text QA
What we offer:
  • Competitive pay and equity
  • Medical, dental, vision coverage
  • Flexible PTO, remote-first culture
  • Monthly wellness stipend & home-office support

Additional Information:

Job Posted:
December 07, 2025

Employment Type:
Fulltime
Work Type:
Remote work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Quality Assurance Analyst, AI Delivery

Quality Analyst, AI Business

As a Quality Analyst within Prolific AI Data Services, you will play a critical ...
Location
Location
Mexico , Mexico City
Salary
Salary:
Not provided
prolific.com Logo
Prolific
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 1–3 years of experience in analysis, quality, operations, or data-focused roles, or equivalent academic projects or internships
  • A background in computer science, data science, statistics, engineering, or a related quantitative discipline
  • Strong SQL skills (required), with the ability to independently query, join, and summarise datasets to answer quality and operational questions
  • Working proficiency in Python and/or R, with experience writing scripts or notebooks to clean data, compute metrics, and automate checks
  • Strong analytical rigour, with a high bar for precision, consistency, and clear documentation of decisions
  • Solid statistical fundamentals, including intuition for sampling, distributions, confidence, and agreement concepts
  • Clear and structured communication skills, with the ability to explain issues using concrete examples and propose practical fixes
  • A collaborative mindset and willingness to partner cross-functionally to improve quality outcomes and delivery reliability
Job Responsibility
Job Responsibility
  • Run day-to-day quality measurement for managed service studies, including sampling, rubric-based review, and structured reporting
  • Apply rubrics and guidelines consistently, flag ambiguities, and propose clear, actionable clarifications as task types evolve
  • Maintain and update golden sets, and support calibration efforts by preparing materials, tracking outcomes, and documenting decisions
  • Track defects and failure modes by categorising issues, assessing severity, and keeping defect taxonomies current
  • Run automated quality checks using SQL and Python or R, including completeness checks, schema and format validation, anomaly detection, and label distribution monitoring
  • Support launch gates by preparing readiness artefacts such as rubric versioning, gold status, sampling plans, and early pilot findings
  • Investigate quality drops by pulling examples, quantifying scope, and supporting root-cause analysis and corrective and preventive actions (CAPAs)
  • Build dashboards and weekly updates covering defect rates, rework, agreement, throughput versus quality trade-offs, and SLA risk
  • Partner with Operations, reviewers, and—when needed—Product and Engineering to improve task design, reviewer workflows, and escalation paths
What we offer
What we offer
  • competitive salary, benefits, and hybrid working within our impactful, mission-driven culture
  • equity
  • opportunity to earn a cash variable element, such as a bonus or commission
Read More
Arrow Right

DevOps Quality Engineer

Engineer reliability at scale. We’re looking for a DevOps Quality Engineer who t...
Location
Location
Bulgaria , Sofia
Salary
Salary:
Not provided
ebrd.com Logo
European Bank for Reconstruction and Development
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Holds ISTQB Foundation as a minimum
  • Advanced Test Analyst or equivalent certifications desirable
  • Qualification in IT Service Management (ITIL v3 or v4 Foundation) or demonstrable experience integrating QA practices into ITSM processes
  • Familiar with the NIST Cybersecurity Framework (CSF) and Digital Operational Resilience Act (DORA), with practical awareness of how they influence quality standards and assurance planning
  • Demonstrates solid understanding of automation and non-functional testing concepts, including performance, accessibility, and shift-left/shift-right practices
  • Experience working within Agile, DevOps, and product-aligned teams, contributing to sprint-based delivery and continuous integration testing strategies
  • Proficient in test tooling and CI/CD frameworks including Azure DevOps, Selenium, Cypress, Jenkins, Git, and test management platforms such as TestRail or Zephyr
  • Familiarity with AI/ML use cases in quality engineering, including AI-assisted test case generation, defect clustering, and predictive analytics
  • Strong communication and collaboration skills, with the ability to explain test scenarios, defects, and coverage to technical and non-technical stakeholders
  • Awareness of security, compliance, and resilience considerations such as OWASP Top 10, ISO 27001, GDPR, and DORA, with practical experience embedding these into quality practices
Job Responsibility
Job Responsibility
  • Plans and performs testing of infrastructure changes, configuration updates, and cloud deployments across data centres, branch offices, and Azure environments, ensuring functional and non-functional validations
  • Collaborates with DevOps, Infrastructure, and Cloud Platform teams to verify IaC deployments, perform smoke testing post-deployment, and document any issues in platform reliability
  • Writes and maintains basic automation scripts or checklists for network, AV, and platform components, contributing to regression detection and provisioning assurance
  • Investigates environment or deployment defects, escalates issues with supporting diagnostics, and retests fixes with attention to uptime, latency, and failover coverage
  • Engages in sprint reviews and planning sessions, raising risks tied to infrastructure changes, recommending testing scope for AV, cloud services, or network resilience improvements
What we offer
What we offer
  • Varied, stimulating and engaging work that gives you an opportunity to interact with a wide range of experts in the financial, political, public and private sectors across the regions we invest in
  • A working culture that embraces inclusion and celebrates diversity
  • An environment that places sustainability, equality and digital transformation at the heart of what we do
  • Fulltime
Read More
Arrow Right

Associate Product Manager

A Tyfone Associate Product Manager will be working on cutting edge digital banki...
Location
Location
United States
Salary
Salary:
Not provided
tyfone.com Logo
Tyfone
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Ability to use AI-driven tools, workflows, and frameworks to build rapid prototypes, test concepts, communicate requirements, create documentation, and optimize processes
  • An understanding of the banking ecosystem in the US
  • Bachelor’s degree in computer science, engineering or related fields
  • Business master’s degree preferred
  • Insatiable curiosity
  • High agency to get things done
  • Energy to solve problems exploring multiple angles
  • Disdain for the status quo
  • Communication skills to listen more than speak. And speak with purpose and clarity
Job Responsibility
Job Responsibility
  • Carry out market research to establish competitive landscape and market definition and engage with customers, vendors, analysts, and other players in the fintech ecosystem for the same
  • Research potential product features, in terms of — usage, best practices, relevant regulations, and fraud potential
  • Support decision-making with data: Build dashboards, create reports, and provide actionable recommendations to help product managers make informed decisions
  • Analyze product performance and user behavior: Gather and interpret data from various sources to uncover insights about how users interact with our products and identify areas for Improvement
  • Assist in product experimentation: Design, execute, and analyze A/B tests and other experiments to validate hypotheses and guide product improvements
  • Participate in product planning, backlog grooming, and roadmap definition
  • Help manage the product lifecycle from market/customer requirements to tactical activities leading to delivery
  • Drive product development across development teams (primarily Customer Support / Sales Development/Engineering and Quality Assurance) through requirements clarifications and finalization, scope definition and coordination between teams
  • Devise go-to market assets: Define, plan and automate creation of market-facing product collaterals that will be consumed by end users as well as financial institutions
  • Coordinate and build strong working relations with various internal stakeholders, including – Executive Management team, Engineering, Sales, Marketing, Quality Assurance to drive outcomes
Read More
Arrow Right

Engineering Excellence Testing Developer

The Applications Development Intermediate Programmer Analyst is an intermediate ...
Location
Location
India , Pune; Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proficiency in programming languages such as Java, Python, VB Script or JavaScript
  • Good Programming experience with Pega
  • Extensive experience with test automation tools like Selenium, Playwright or UFT
  • Proficient Experience with tools such as ALM, JIRA, Zephyr, advanced excel features
  • Experience in Oracle and SQL queries for test data mining, data generation & conditioning
  • Strong understanding of banking workflows, financial products, and regulatory requirements
  • Experience testing banking applications such as loan management systems, payment gateways, or risk management platforms
  • Experience in using mainframe for testing and automation
  • Experience in BDD/TDD, Cucumber framework development
  • Experience in building and maintaining CI/CD pipelines with tools like Jenkins
Job Responsibility
Job Responsibility
  • Test Automation Development: Design and implement robust, scalable, and maintainable test automation frameworks using AI-driven tools and technologies
  • Develop automated test scripts for functional, regression testing of banking applications
  • Ability to identify the scope and E2E delivery of Automation
  • Maintenance of Automation framework and repository
  • AI Integration: Utilize AI and machine learning models to optimize test case generation, execution, and defect prediction
  • Implement AI-based solutions for intelligent test data generation and anomaly detection
  • Banking Domain Expertise: Apply knowledge of banking processes, regulations, and workflows to ensure comprehensive test coverage especially with Disputes workflow
  • Collaborate with business analysts and product owners to understand banking requirements and translate them into effective test strategies
  • Quality Assurance: Analyze requirements of projects, develop test plans, design test cases for broad range of systems and applications and ensure they meet specified standards and end-user requirements
  • Ensure end-to-end quality of banking applications by identifying risks, creating test plans, and executing test cases
  • Fulltime
Read More
Arrow Right

Senior Trust Analyst

The Trust Engagement team is a globally distributed group of Security and Risk p...
Location
Location
Poland , Gdańsk
Salary
Salary:
Not provided
https://www.atlassian.com Logo
Atlassian
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years experience working with customers across Security, Governance, Risk and Compliance domain areas
  • Expertise to lead discussions with decision makers about security and risk trade-offs
  • Direct customer-facing or consulting engagement experience
  • Success with customer, executive, or board member briefings on strategy, including impactful and effective presentations
  • Knowledge of security, compliance, and privacy frameworks & standards (e.g., GDPR, HIPAA, SOC2, ISO27001, NIST 800-53, PCI-DSS, C5, DORA, NIS2)
  • Experience working with Regulatory Compliance, Global Privacy Programs, Cloud Security or related functions
  • Familiarity with large-scale enterprise SaaS platforms and challenges across security, privacy and compliance related to customer trust
  • Experience with cybersecurity principles and controls, including their application in a SaaS context
  • Understanding infrastructure and delivery models of SaaS products
  • Proficient knowledge of cloud platforms (e.g., AWS, Azure, Google Cloud) and their services
Job Responsibility
Job Responsibility
  • Represent Atlassian with our Strategic & Enterprise customers to manage trust-related questions and concerns for customer acquisition, customer migration, and long term customer retention
  • Be accountable for engaging in a variety of Atlassian customer facing initiatives. Your primary focus will involve helping our customers gain an understanding of Atlassian’s commitment to Trust by articulating our Security practices, Compliance posture, Privacy measures, Resilience efforts and Anti-Abuse program
  • Manage complex customer security reviews & audits
  • Support Atlassian’s legal teams in negotiating security and compliance terms & exhibits for Atlassian Customer Agreements
  • Represent the Voice of the Customer by identifying deal blockers and friction points to influence Security, Product & Engineering strategy in meeting customer expectations
  • Drive customer facing Trust collateral by publishing publicly facing materials, presentations, white papers
  • Provide security, compliance, and privacy education and training to internal partner teams in Sales, Product Marketing, Legal, etc.
  • Mentor team through technical leadership and knowledge sharing.
What we offer
What we offer
  • Health and wellbeing resources
  • Paid volunteer days
  • Wide range of perks and benefits designed to support you and your family.
Read More
Arrow Right
New

Patient Safety and Pharmacovigilance Intern

The Patient Safety and Pharmacovigilance Intern (PSPV Intern) will report direct...
Location
Location
United States , Waltham
Salary
Salary:
25.00 - 30.00 USD / Hour
kailera.com Logo
Kailera Therapeutics
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proactive, engaged, eager and willing to learn approach to work
  • Support and be part of the Kailera core values: Courageous, Collaborative, Compassionate, and Committed
  • Excellent oral and written communication skills
  • Interest and/or experience in drug safety and pharmacovigilance a plus
  • Ability to work well in a fast-paced team environment
  • Bachelor’s degree candidates who have completed two or more years and are pursuing a degree in life sciences (nursing, neuroscience, psychology, psychiatry, biology, pharmacy)
Job Responsibility
Job Responsibility
  • Research and understand the disease, symptoms, and treatment under investigation
  • Participate in safety data analysis/signal detection (ICSR and aggregate data review)
  • Understand the process of vendor management
  • Review applicable Health Authority regulations and ICH guidelines to develop working knowledge of PSPV reporting obligations to ensure patient safety
  • Lead a defined project (e.g. process optimization, data visualization, or data analytics) and prepare a final presentation for department and company leadership
  • Opportunity to support PSPV projects
What we offer
What we offer
  • comprehensive health coverage
  • monthly wellness stipend
  • Fulltime
Read More
Arrow Right
New

Materials Engineer

Join Meta Reality Labs and help define the next generation of Wearables and Mixe...
Location
Location
United States , Sunnyvale
Salary
Salary:
173000.00 - 245000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Optical Engineering, Electrical Engineering, Physics, or a related technical discipline
  • 8+ years of industry experience with a proven track record of full product development cycle in consumer electronics or similar related field
  • Experience and expertise in these areas or similar related technologies: optical design software, geometrical optics, wave optics, and polarization optics, optical materials, coatings, and manufacturing processes, Imaging optics design and manufacturing, display optics, optical sensing, optical system integration and testing
  • Technical leadership, communication, and project management skills
  • Problem-solving, analytical, and critical thinking skills
Job Responsibility
Job Responsibility
  • Lead cross-functional teams in the design, development, and implementation of cutting-edge optical solutions for new optics architectures, utilizing novel materials and processes
  • Identify, incubate, and productize optical technologies that have the potential to revolutionize the field of wearable optics
  • Specify key performance metrics, test methods, and test requirements for optical components and systems
  • Develop and maintain technical roadmaps
  • Interface with stakeholders and leadership to influence product roadmaps
  • Stay updated on industry trends, emerging technologies, and competitor activity
  • Manage budgets, resources, and collaborations with global vendors
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right
New

Foreman

Position available for utility line clearance crew Foreman. The crew Foreman wil...
Location
Location
United States , Harrisburg
Salary
Salary:
Not provided
pennline.com Logo
Penn Line Energy T&D, Inc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience in the tree/line clearance industry is necessary
  • 3 to 5 years’ verifiable line clearance experience or related tree work experience
  • CDL License is a plus
  • Previous Foreman experience required
  • Excellent oral and written communication skills are necessary
  • Herbicide or ISA licenses are a plus
Job Responsibility
Job Responsibility
  • Assure that the work is completed in the most productive and efficient manner
  • Maintains work site safety for all persons and property throughout the assigned area(s)
  • Provides supervision and oversight of vegetation management crew activity
  • Assures that the utility customer needs are met within the scope of work
  • Supports the needs of employees
What we offer
What we offer
  • Medical, Dental, & Vision Insurance
  • Short Term Disability
  • Life Insurance
  • Travel Pay Reimbursement
  • Tuition Reimbursement
  • Retirement Plan
  • 401k
  • Paid Time Off
  • Fulltime
Read More
Arrow Right