CrawlJobs Logo

Member of Technical Staff, Data Analysis and Evaluation

cohere.com Logo

Cohere

Location Icon

Location:

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided

Job Description:

As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability, and performance of our large language models (LLMs). Your primary focus will be on designing and conducting data collection tasks, assessing and evaluating dataset quality, and analysing the robustness and generalisability of our models. You will work closely with cross-functional teams, including researchers, engineers, and data annotators, to conduct data-driven decision-making and improve the overall effectiveness of our AI systems. This role combines expertise in statistics, experimental design incl. human annotators, and machine learning to ensure that our models are trained on high-quality data and perform reliably across diverse scenarios. You will contribute to Cohere’s mission of advancing AI by ensuring our systems are robust, scalable, and impactful.

Job Responsibility:

  • Design and oversee data collection tasks, including supporting human annotators and ensuring data quality
  • Develop and apply statistical methods to evaluate the quality and reliability of datasets
  • Analyse and assess the generalisability and robustness of ML systems across diverse use cases
  • Collaborate with teams to improve dataset quality and model performance
  • Train and fine-tune large language models (LLMs) on distributed training infrastructures
  • Conduct experiments to evaluate model performance and identify areas for improvement

Requirements:

  • Extremely strong software engineering skills
  • Strong expertise in designing and conducting data collection tasks, including working with human annotators
  • Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance
  • Experience analysing datasets with respect to their quality, biases, and suitability for training ML models
  • Hands-on experience training large language models (LLMs) on distributed training infrastructures
  • Familiarity with evaluating and improving the generalisability and robustness of ML systems
  • Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX)
  • Excellent communication skills to collaborate effectively with cross-functional teams and present findings
  • One or more papers at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP)
What we offer:
  • An open and inclusive culture and work environment
  • Work closely with a team on the cutting edge of AI research
  • Weekly lunch stipend, in-office lunches & snacks
  • Full health and dental benefits, including a separate budget to take care of your mental health
  • 100% Parental Leave top-up for up to 6 months
  • Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • 6 weeks of vacation (30 working days!)

Additional Information:

Job Posted:
February 20, 2026

Employment Type:
Fulltime
Work Type:
Remote work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Member of Technical Staff, Data Analysis and Evaluation

Data Quality Lead Analyst

The Data Quality Lead Analyst is a strategic professional who stays abreast of d...
Location
Location
Canada , Mississauga
Salary
Salary:
94300.00 - 141500.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years software experience preferred
  • High proficiency in MS Excel, Python, SQL for Data analysis
  • Ability to create BI tools such as Qlikview and/or Tableau views for analysis
  • Problem Solving - Evaluates moderately complex and variable issues with substantial potential impact
  • Good analytical skills to filter, prioritize and validate potentially complex and dynamic material from multiple sources
  • Outstanding written and oral communication, influencing and presentation skills
  • A self-starter with ability to independently manage work and drive multiple deliverables concurrently
  • Strong subject matter expertise regarding technology application control disciplines and technology infrastructure knowledge
  • Demonstrated capability and maturity to work in an individual capacity
  • Demonstrates experience in managing teams and managing integrated internal audit and assurance delivery
Job Responsibility
Job Responsibility
  • Devises methods for identifying data patterns and trends in available data sources
  • Defines strategies to drive data quality measurement, produce data quality dashboards and reports, and implement data quality strategies to effectively govern data and improve data quality
  • Works with data quality partners and Technology teams to identify and implement data quality tools
  • Identify critical data elements, data quality rules, thresholds, and other business requirements
  • Mentors junior analysts
  • Applies in-depth understanding and knowledge of how business integrates within the sub function
  • Contributes to the development of new techniques and processes and for the aligned business
  • Integrates subject matter and industry expertise within a defined area
  • Responsible for own work to support business teams in assigned area
  • Build data quality assessments and measurement of critical data utilized by Citi strategic systems and processes
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Data Scientist

We’re looking for data scientists to help build the next generation of post-trai...
Location
Location
Switzerland , Zürich
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands‑on experience with large language models, including training or applying them in production (not just prompting)
  • Designing and running post‑training experiments (evals, ablations, preference tuning / RLHF‑style methods)
  • Building and owning scalable data pipelines for training and evaluation data
  • Strong Python skills for ML experimentation, data processing, and analysis
  • Solid statistical, experimental, and general engineering fundamentals
Job Responsibility
Job Responsibility
  • Design evaluations of advanced model capabilities and use them to drive rapid, high-signal iteration loops
  • Work with vendors to produce high quality evaluation and training data
  • Build data pipelines to produce high quality evaluation and training data
  • Build data flywheels to hill-climb on model weaknesses, using data from various surfaces where our models are deployed
  • Ensure optimal quality, quantity and coverage of data across our post-training stages
  • Run post-training experiments and ablations to produce models that climb our evals
  • Embody our culture and values
  • Fulltime
Read More
Arrow Right

Member of Technical Staff, Data Research Engineer

We are seeking Data Research Engineers to join our Multimodal team, where we are...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in AI, Computer Science, Data Science, Statistics, Physics, Engineering, or a related technical field AND technical engineering experience with coding in languages including, but not limited to, Python and common data libraries (Pandas, NumPy, etc.)
  • OR equivalent experience
  • Experience in data analysis or data engineering
  • Proficiency in statistics and exploratory data analysis methods
  • Ability to communicate technical findings effectively to research and product teams
Job Responsibility
Job Responsibility
  • Create high-quality datasets for training and evaluation
  • run experiments on new datasets (data ablations) to assess their impact and determine the most effective data
  • Develop and maintain scalable data pipelines for multimodal ingestion, pre-processing, filtering, and annotation
  • Analyse real-world multimodal datasets to assess quality, diversity, relevance, and identify areas for improvement
  • Build lightweight tools and workflows for dataset auditing, visualization, and versioning
  • Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices
  • Fulltime
Read More
Arrow Right

Data Quality Lead Analyst

The Data Quality Lead Analyst is a strategic professional who stays abreast of d...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years software experience preferred
  • High proficiency in MS Excel, Python, SQL for Data analysis
  • Ability to create BI tools such as, Qlikview and/or Tableau views for analysis
  • Problem Solving - Evaluates moderately complex and variable issues with substantial potential impact
  • Requires good analytical skills in order to filter, prioritize and validate potentially complex and dynamic material from multiple sources
  • Outstanding written and oral communication, influencing and presentation skills
  • A self-starter with ability to independently manage work and drive multiple deliverables concurrently
  • Strong subject matter expertise regarding technology application control disciplines and technology infrastructure knowledge, with a sound understanding of the financial services provided by Citi
  • Demonstrated capability and maturity to work in an individual capacity and must proactively seek out ways to enhance the mission and identify new data analysis needs
  • Demonstrates experience in managing teams and managing integrated internal audit and assurance delivery within a matrix reporting environment
Job Responsibility
Job Responsibility
  • Devises methods for identifying data patterns and trends in available data sources
  • Defines strategies to drive data quality measurement, produce data quality dashboards and reports, and implement data quality strategies to effectively govern data and improve data quality
  • Works with data quality partners and Technology teams to identify and implement data quality tools. Identify critical data elements, data quality rules, thresholds, and other business requirements
  • Mentors junior analysts
  • Applies in-depth understanding and knowledge of how business integrates within the sub function
  • as well as coordinates and contributes to the objectives of the function and overall business
  • Contributes to the development of new techniques and processes and for the aligned business
  • Integrates subject matter and industry expertise within a defined area
  • Responsible for own work to support business teams in assigned area
  • Build data quality assessments and measurement of critical data utilized by Citi strategic systems and processes supporting Citi Global Functions and Sectors
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Pretraining Text Data

We are seeking engineers and researchers to join our Pretraining Text Data team,...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in AI, Computer Science, Data Science, Statistics, Physics, Engineering, or related technical discipline AND technical engineering experience with coding in languages including, but not limited to, Python and common data libraries (Pandas, NumPy, etc.) OR equivalent experience.
  • 2+ years of experience in data analysis or data engineering, including work with large-scale datasets that are unstructured or semi-structured.
  • Proficiency in statistics and exploratory data analysis methods.
Job Responsibility
Job Responsibility
  • Create high-quality datasets for training and evaluation
  • run experiments on new datasets (data ablations) to assess their impact and determine the most effective data.
  • Develop and maintain scalable data pipelines for text data ingestion, preprocessing, filtering, and annotation.
  • Analyze real-world text datasets to assess quality, diversity, relevance, and identify areas for improvement.
  • Build lightweight tools and workflows for dataset auditing, visualization, and versioning.
  • Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices.
  • Embody our culture and values.
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Full Stack Software Engineer

Microsoft AI is looking for a Member of Technical Staff – Full Stack Software En...
Location
Location
United States , Redmond; Mountain View
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to C#, Java, or Python
  • OR equivalent experience
  • 2+ years’ experience with SQL, PostgreSQL or MySQL
  • 2+ years' experience building scalable services on top of public cloud infrastructure like Azure, AWS, or GCP
  • Extensive use datastores like RDBMS, key-value stores, etc.
  • Experience with Development & Debugging with dev environments like Visual Studio or Visual Studio Code
  • HTML, CSS, JavaScript, ASP.NET, REST, jQuery
  • Familiarity with browser automation tools like Selenium, Puppeteer or Playwright
  • Familiarity with mobile automation tools like Appium
  • Familiarity with LLMs and AI ChatBots
Job Responsibility
Job Responsibility
  • Expertise in experimentation methodologies, including A/B evaluation, data sampling, measurement techniques, evaluation design, and data analysis
  • Demonstrating strategic vision by understanding organizational goals, translating metrics into actionable insights, and enhancing product quality
  • Designing pipeline architecture to ensure rapid iteration and scalability
  • Conducting post-analysis of labeled data and developing dashboards to visualize insights
  • Collaborating closely with the product team to enhance quality and address gaps
  • Embody our Culture and Values
  • Fulltime
Read More
Arrow Right

Member of Technical Staff, Applied AI Engineer

We’re hiring a Applied AI Engineer to join a fast‑moving, high‑ownership team bu...
Location
Location
United States , Mountain View
Salary
Salary:
100600.00 - 199000.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) or consulting experience OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 2+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR equivalent experience.
  • 2+ years shipping production-level code, models, or data analysis.
  • 1+ years using AI-assisted coding and analysis techniques.
  • Experience working on small teams and mid-stage startup environments.
  • Experience working on AI products.
  • PhD in engineering, applied math, statistics, or related analytical field.
  • 4+ years shipping production-level code, models, or data analysis.
  • Deep experience building from zero-to-one.
  • Hands on work hillclimbing AI evaluations.
Job Responsibility
Job Responsibility
  • LLM Feature & Agent Development
  • Design and ship LLM‑powered assistant features, including conversational flows, agentic behaviors, retrieval pipelines, and multimodal interactions.
  • Build prompt architectures, system instructions, and orchestration logic that ensure reliability, grounding, and personality consistency.
  • Prototype new capabilities rapidly and iterate based on user signals and evaluation data.
  • Evaluation, Hillclimbing & Quality Systems
  • Build and maintain evaluation frameworks for correctness, safety, grounding, and UX quality.
  • Run hillclimbing loops across prompts, models, and tool‑use strategies to continuously improve assistant performance.
  • Analyze failure modes, design mitigations, and drive systematic improvements across the stack.
  • LLM Tooling & Internal Infrastructure
  • Develop internal tools for prompt experimentation, model comparison telemetry and debugging automated eval pipelines
  • Fulltime
Read More
Arrow Right

Staff Bioinformatics Scientist - Genetics

The Staff Bioinformatics Scientist is largely self-driven, working in collaborat...
Location
Location
United States , San Diego
Salary
Salary:
160000.00 - 240000.00 USD / Year
exactsciences.com Logo
Exact Sciences
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Ph.D. in bioinformatics, computational biology, computer science, mathematics, or related field
  • or Master’s degree in bioinformatics, computational biology, computer science, mathematics, or related field plus 4 years of experience in lieu of a Ph.D.
  • or Bachelor’s degree in bioinformatics, computational biology, computer science, mathematics, or related field plus 6 years of experience in lieu of Ph.D.
  • 8+ years of industry or academia experience in bioinformatics
  • 6+ years of experience in medical device/IVD, biotech, life science or pharmaceutical industry
  • Demonstrated experience in developing statistical and quantitative methods
  • Deep knowledge and thorough understanding of molecular biology and/or biochemistry and/or next-generation sequencing (NGS) and genomics
  • Demonstrated expertise in product/assay research, design, or development
  • Basic computer skills to include Internet navigation, Email usage, and word processing
  • Proficient in Microsoft Office to include Excel, Word, and PowerPoint
Job Responsibility
Job Responsibility
  • Conduct comprehensive analyses within several product or technology areas and identify problems and discrepancies
  • Independently plan and develop algorithms
  • Independently develop, plan, and analyze results of experiments within several product or technology areas within defined timelines
  • effectively present and clearly communicate findings at data meetings, group meetings and project team meetings to inform and enable project progression
  • Critically analyze research/experimental data, interpret the results, and provide insights into the next steps and the direction of the project for successful outcomes
  • Contribute towards the development of timeline and plan for new assignments/study/projects
  • Provide technical guidance to junior scientists in project teams
  • Manage bench-generated data and bioinformatics outputs within high performance computing (HPC) or cloud computing environment
  • Determine, utilize, and apply methods or technologies effectively and provide ideas for new techniques, when appropriate
  • Maintain knowledge of technological industry developments that could assist in completion of an assignment or aid in the development of new processes or procedures
What we offer
What we offer
  • paid time off (including days for vacation, holidays, volunteering, and personal time)
  • paid leave for parents and caregivers
  • a retirement savings plan
  • wellness support
  • health benefits including medical, prescription drug, dental, and vision coverage
  • bonus eligible
  • Fulltime
Read More
Arrow Right