CrawlJobs Logo

Member of Technical Staff, Data Analysis and Evaluation

cohere.com Logo

Cohere

Location Icon

Location:

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided

Job Description:

As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability, and performance of our large language models (LLMs). Your primary focus will be on designing and conducting data collection tasks, assessing and evaluating dataset quality, and analysing the robustness and generalisability of our models. You will work closely with cross-functional teams, including researchers, engineers, and data annotators, to conduct data-driven decision-making and improve the overall effectiveness of our AI systems. This role combines expertise in statistics, experimental design incl. human annotators, and machine learning to ensure that our models are trained on high-quality data and perform reliably across diverse scenarios. You will contribute to Cohere’s mission of advancing AI by ensuring our systems are robust, scalable, and impactful.

Job Responsibility:

  • Design and oversee data collection tasks, including supporting human annotators and ensuring data quality
  • Develop and apply statistical methods to evaluate the quality and reliability of datasets
  • Analyse and assess the generalisability and robustness of ML systems across diverse use cases
  • Collaborate with teams to improve dataset quality and model performance
  • Train and fine-tune large language models (LLMs) on distributed training infrastructures
  • Conduct experiments to evaluate model performance and identify areas for improvement

Requirements:

  • Extremely strong software engineering skills
  • Strong expertise in designing and conducting data collection tasks, including working with human annotators
  • Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance
  • Experience analysing datasets with respect to their quality, biases, and suitability for training ML models
  • Hands-on experience training large language models (LLMs) on distributed training infrastructures
  • Familiarity with evaluating and improving the generalisability and robustness of ML systems
  • Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX)
  • Excellent communication skills to collaborate effectively with cross-functional teams and present findings
  • One or more papers at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP)
What we offer:
  • An open and inclusive culture and work environment
  • Work closely with a team on the cutting edge of AI research
  • Weekly lunch stipend, in-office lunches & snacks
  • Full health and dental benefits, including a separate budget to take care of your mental health
  • 100% Parental Leave top-up for up to 6 months
  • Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • 6 weeks of vacation (30 working days!)

Additional Information:

Job Posted:
February 20, 2026

Employment Type:
Fulltime
Work Type:
Remote work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Member of Technical Staff, Data Analysis and Evaluation

Salesforce Technical Analyst

Develops technical documentation, designs, analyzes, tests, provides end user su...
Location
Location
United States , Phoenix
Salary
Salary:
Not provided
azblue.com Logo
Blue Cross Blue Shield of Arizona
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 1 year of college-level coursework in system requirements design and interpretation and/or support of production systems (Lev 1)
  • 3 years of experience in computer technology and/or information systems (Lev 2 – 4)
  • 2 years of experience in data analysis, data management and integration, extracting and analysis reporting. (Lev 2-4)
  • High-School Diploma or GED in general field of study (All Levels)
Job Responsibility
Job Responsibility
  • Develops technical documentation, designs, analyzes, tests, provides end user support and training for new applications and data structures
  • and provides post-implementation review for systems and reporting to support customers and management.
  • Track and resolve basic or routine processing problems, coordinating with all areas.
  • Research, analyze, track and resolve basic production problems.
  • Develop business objectives, processing requirements, and technical specifications.
  • Analyze files and map data from one system to another.
  • Responsible for production cycles, solving production incidents and notification of problems.
  • Write SQL queries needed for production support.
  • Develop basic test data.
  • Process test files and analyze test results.
  • Fulltime
Read More
Arrow Right

Data Quality Lead Analyst

The Data Quality Lead Analyst is a strategic professional who stays abreast of d...
Location
Location
Canada , Mississauga
Salary
Salary:
94300.00 - 141500.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years software experience preferred
  • High proficiency in MS Excel, Python, SQL for Data analysis
  • Ability to create BI tools such as Qlikview and/or Tableau views for analysis
  • Problem Solving - Evaluates moderately complex and variable issues with substantial potential impact
  • Good analytical skills to filter, prioritize and validate potentially complex and dynamic material from multiple sources
  • Outstanding written and oral communication, influencing and presentation skills
  • A self-starter with ability to independently manage work and drive multiple deliverables concurrently
  • Strong subject matter expertise regarding technology application control disciplines and technology infrastructure knowledge
  • Demonstrated capability and maturity to work in an individual capacity
  • Demonstrates experience in managing teams and managing integrated internal audit and assurance delivery
Job Responsibility
Job Responsibility
  • Devises methods for identifying data patterns and trends in available data sources
  • Defines strategies to drive data quality measurement, produce data quality dashboards and reports, and implement data quality strategies to effectively govern data and improve data quality
  • Works with data quality partners and Technology teams to identify and implement data quality tools
  • Identify critical data elements, data quality rules, thresholds, and other business requirements
  • Mentors junior analysts
  • Applies in-depth understanding and knowledge of how business integrates within the sub function
  • Contributes to the development of new techniques and processes and for the aligned business
  • Integrates subject matter and industry expertise within a defined area
  • Responsible for own work to support business teams in assigned area
  • Build data quality assessments and measurement of critical data utilized by Citi strategic systems and processes
  • Fulltime
Read More
Arrow Right

Member of Technical Staff, Data Research Engineer

We are seeking Data Research Engineers to join our Multimodal team, where we are...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in AI, Computer Science, Data Science, Statistics, Physics, Engineering, or a related technical field AND technical engineering experience with coding in languages including, but not limited to, Python and common data libraries (Pandas, NumPy, etc.)
  • OR equivalent experience
  • Experience in data analysis or data engineering
  • Proficiency in statistics and exploratory data analysis methods
  • Ability to communicate technical findings effectively to research and product teams
Job Responsibility
Job Responsibility
  • Create high-quality datasets for training and evaluation
  • run experiments on new datasets (data ablations) to assess their impact and determine the most effective data
  • Develop and maintain scalable data pipelines for multimodal ingestion, pre-processing, filtering, and annotation
  • Analyse real-world multimodal datasets to assess quality, diversity, relevance, and identify areas for improvement
  • Build lightweight tools and workflows for dataset auditing, visualization, and versioning
  • Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices
  • Fulltime
Read More
Arrow Right
New

Data Quality Lead Analyst

The Data Quality Lead Analyst is a strategic professional who stays abreast of d...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years software experience preferred
  • High proficiency in MS Excel, Python, SQL for Data analysis
  • Ability to create BI tools such as, Qlikview and/or Tableau views for analysis
  • Problem Solving - Evaluates moderately complex and variable issues with substantial potential impact
  • Requires good analytical skills in order to filter, prioritize and validate potentially complex and dynamic material from multiple sources
  • Outstanding written and oral communication, influencing and presentation skills
  • A self-starter with ability to independently manage work and drive multiple deliverables concurrently
  • Strong subject matter expertise regarding technology application control disciplines and technology infrastructure knowledge, with a sound understanding of the financial services provided by Citi
  • Demonstrated capability and maturity to work in an individual capacity and must proactively seek out ways to enhance the mission and identify new data analysis needs
  • Demonstrates experience in managing teams and managing integrated internal audit and assurance delivery within a matrix reporting environment
Job Responsibility
Job Responsibility
  • Devises methods for identifying data patterns and trends in available data sources
  • Defines strategies to drive data quality measurement, produce data quality dashboards and reports, and implement data quality strategies to effectively govern data and improve data quality
  • Works with data quality partners and Technology teams to identify and implement data quality tools. Identify critical data elements, data quality rules, thresholds, and other business requirements
  • Mentors junior analysts
  • Applies in-depth understanding and knowledge of how business integrates within the sub function
  • as well as coordinates and contributes to the objectives of the function and overall business
  • Contributes to the development of new techniques and processes and for the aligned business
  • Integrates subject matter and industry expertise within a defined area
  • Responsible for own work to support business teams in assigned area
  • Build data quality assessments and measurement of critical data utilized by Citi strategic systems and processes supporting Citi Global Functions and Sectors
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Pretraining Text Data

We are seeking engineers and researchers to join our Pretraining Text Data team,...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in AI, Computer Science, Data Science, Statistics, Physics, Engineering, or related technical discipline AND technical engineering experience with coding in languages including, but not limited to, Python and common data libraries (Pandas, NumPy, etc.) OR equivalent experience.
  • 2+ years of experience in data analysis or data engineering, including work with large-scale datasets that are unstructured or semi-structured.
  • Proficiency in statistics and exploratory data analysis methods.
Job Responsibility
Job Responsibility
  • Create high-quality datasets for training and evaluation
  • run experiments on new datasets (data ablations) to assess their impact and determine the most effective data.
  • Develop and maintain scalable data pipelines for text data ingestion, preprocessing, filtering, and annotation.
  • Analyze real-world text datasets to assess quality, diversity, relevance, and identify areas for improvement.
  • Build lightweight tools and workflows for dataset auditing, visualization, and versioning.
  • Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices.
  • Embody our culture and values.
  • Fulltime
Read More
Arrow Right

Senior ASP Dotnet Developer

PA State Police (PSP) requires the services of a Senior Application Developer st...
Location
Location
United States , Philadelphia
Salary
Salary:
Not provided
oceanbluecorp.com Logo
Ocean Blue Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • IT expertise in development and support of large, complex, mission critical, enterprise software systems
  • Provide a highly stable and supportable environment improving all aspects of the Enterprise system
  • Provide ongoing assessment of the operational state of the network, compute, and security operations and drive continuous improvements.
  • Continuously evaluate the evolving needs of the Enterprise systems and ensure sufficient capacity is planned and in place to meet the needs of the agencies and customers.
  • Lead the identification and mitigation of risks, troubleshoot, and provide resolution to complex end to end scenarios involving multiple external agencies, products, protocols, networks and technology stacks
  • Develop and document standards, policies, processes, roles, and responsibilities to support the operations
  • Maintain and enhance documentation of architecture, infrastructure, and system trouble shooting methods
  • Develop, measure, and maintain KPI’s for the core network, compute, and security operations and supporting reports, dashboards, and executive summaries.
  • Unique combination of expertise in Solution Management applications
  • Understanding of OO concepts and design patterns
Job Responsibility
Job Responsibility
  • Defines test plans and criteria for acceptance for the technical components of an application.
  • Ensures business requirements are supported by the technical architecture.
  • Assists in leading technical direction of software
  • coordinating and presenting proposals to Group Leads, Leads, and the client.
  • Defines and evaluates logical and physical data models.
  • Assists in development of overall system technical architecture - including software and hardware.
  • Analyzes, determines, and documents technical requirements and change request impact analysis.
  • Participates in detailed design and product test execution as required.
  • Develops technical programming standards.
  • Conducts structured walk-throughs or inspections for technical areas
  • Fulltime
Read More
Arrow Right
New

Senior Data Management Analyst

Wells Fargo is seeking a Senior Data Management Analyst.
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.wellsfargo.com/ Logo
Wells Fargo
Expiration Date
February 20, 2026
Flip Icon
Requirements
Requirements
  • 4+ years of Data Management, Business Analysis, Analytics, or Project Management experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
  • Data warehousing experience
  • Microsoft Fabric capacity administration
  • Dev-Ops experience support application development teams
  • Cloud based data management experience
  • Deep understanding of Microsoft Fabric components, including One Lake, Data Engineering, and Power BI integration
  • Support Human Resources Data applications and their development teams
  • Limited production/Non Prod support activities
  • Microsoft Fabric administration
  • Support database and application access control processes
Job Responsibility
Job Responsibility
  • Lead or participate in moderately complex programs and initiatives for data quality, governance, and metadata activities
  • Design and conduct moderately complex analysis to identify and remediate data quality, data integrity, process, and control gaps
  • Analyze, assess, and test data controls and data systems to ensure quality and risk compliance standards are met and adhere to data governance standards and procedures
  • Identify data quality metrics and execute data quality audits to benchmark the state of data quality
  • Develop recommendations for optimal approaches to resolve data quality issues and implement plans for assessing the quality of new data sources leveraging domain expertise and data, business, or process analysis to inform and support solution design
  • Lead project teams and mentor less experienced staff members
  • Drive planning and coordination on moderately complex remediation efforts acting as central point of contact
  • Consult with clients to assess the current state of data and metadata quality within area of assigned responsibility
  • Participate in cross-functional groups to develop companywide data governance strategies
  • Provide input into communication routines with stakeholders, business partners, and experienced leaders
  • Fulltime
Read More
Arrow Right
New

Senior Data Management Analyst

Wells Fargo is seeking a Senior Data Management Analyst.
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.wellsfargo.com/ Logo
Wells Fargo
Expiration Date
February 20, 2026
Flip Icon
Requirements
Requirements
  • 4+ years of Data Management, Business Analysis, Analytics, or Project Management experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
  • Data warehousing experience
  • Microsoft Fabric capacity administration
  • Dev-Ops experience support application development teams
  • Cloud based data management experience
  • Deep understanding of Microsoft Fabric components, including One Lake, Data Engineering, and Power BI integration
  • Support Human Resources Data applications and their development teams
  • Limited production/Non Prod support activities
  • Microsoft Fabric administration
  • Support database and application access control processes
Job Responsibility
Job Responsibility
  • Lead or participate in moderately complex programs and initiatives for data quality, governance, and metadata activities
  • Design and conduct moderately complex analysis to identify and remediate data quality, data integrity, process, and control gaps
  • Analyze, assess, and test data controls and data systems to ensure quality and risk compliance standards are met and adhere to data governance standards and procedures
  • Identify data quality metrics and execute data quality audits to benchmark the state of data quality
  • Develop recommendations for optimal approaches to resolve data quality issues and implement plans for assessing the quality of new data sources leveraging domain expertise and data, business, or process analysis to inform and support solution design
  • Lead project teams and mentor less experienced staff members
  • Drive planning and coordination on moderately complex remediation efforts acting as central point of contact
  • Consult with clients to assess the current state of data and metadata quality within area of assigned responsibility
  • Participate in cross-functional groups to develop companywide data governance strategies
  • Provide input into communication routines with stakeholders, business partners, and experienced leaders
  • Fulltime
Read More
Arrow Right