CrawlJobs Logo

Member of Technical Staff, Data Analysis and Evaluation

· Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability, and performance of our large language models (LLMs). Your primary focus will be on designing and conducting data collection tasks, assessing and evaluating dataset quality, and analysing the robustness and generalisability of our models. You will work closely with cross-functional teams, including researchers, engineers, and data annotators, to conduct data-driven decision-making and improve the overall effectiveness of our AI systems. This role combines expertise in statistics, experimental design incl. human annotators, and machine learning to ensure that our models are trained on high-quality data and perform reliably across diverse scenarios. You will contribute to Cohere’s mission of advancing AI by ensuring our systems are robust, scalable, and impactful.

Job Responsibility

  • Design and oversee data collection tasks, including supporting human annotators and ensuring data quality
  • Develop and apply statistical methods to evaluate the quality and reliability of datasets
  • Analyse and assess the generalisability and robustness of ML systems across diverse use cases
  • Collaborate with teams to improve dataset quality and model performance
  • Train and fine-tune large language models (LLMs) on distributed training infrastructures
  • Conduct experiments to evaluate model performance and identify areas for improvement

Requirements

  • Extremely strong software engineering skills
  • Strong expertise in designing and conducting data collection tasks, including working with human annotators
  • Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance
  • Experience analysing datasets with respect to their quality, biases, and suitability for training ML models
  • Hands-on experience training large language models (LLMs) on distributed training infrastructures
  • Familiarity with evaluating and improving the generalisability and robustness of ML systems
  • Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX)
  • Excellent communication skills to collaborate effectively with cross-functional teams and present findings
  • One or more papers at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP)

What we offer

  • An open and inclusive culture and work environment
  • Work closely with a team on the cutting edge of AI research
  • Weekly lunch stipend, in-office lunches & snacks
  • Full health and dental benefits, including a separate budget to take care of your mental health
  • 100% Parental Leave top-up for up to 6 months
  • Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • 6 weeks of vacation (30 working days!)

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Member of Technical Staff, Data Analysis and Evaluation

8 matching positions

Data Quality Lead Analyst

The Data Quality Lead Analyst is a strategic professional who stays abreast of d...
Location
Location
Canada , Mississauga
Salary
Salary:
94300.00 - 141500.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years software experience preferred
  • High proficiency in MS Excel, Python, SQL for Data analysis
  • Ability to create BI tools such as Qlikview and/or Tableau views for analysis
  • Problem Solving - Evaluates moderately complex and variable issues with substantial potential impact
  • Good analytical skills to filter, prioritize and validate potentially complex and dynamic material from multiple sources
  • Outstanding written and oral communication, influencing and presentation skills
  • A self-starter with ability to independently manage work and drive multiple deliverables concurrently
  • Strong subject matter expertise regarding technology application control disciplines and technology infrastructure knowledge
  • Demonstrated capability and maturity to work in an individual capacity
  • Demonstrates experience in managing teams and managing integrated internal audit and assurance delivery
Job Responsibility
Job Responsibility
  • Devises methods for identifying data patterns and trends in available data sources
  • Defines strategies to drive data quality measurement, produce data quality dashboards and reports, and implement data quality strategies to effectively govern data and improve data quality
  • Works with data quality partners and Technology teams to identify and implement data quality tools
  • Identify critical data elements, data quality rules, thresholds, and other business requirements
  • Mentors junior analysts
  • Applies in-depth understanding and knowledge of how business integrates within the sub function
  • Contributes to the development of new techniques and processes and for the aligned business
  • Integrates subject matter and industry expertise within a defined area
  • Responsible for own work to support business teams in assigned area
  • Build data quality assessments and measurement of critical data utilized by Citi strategic systems and processes
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Data Scientist

We’re looking for data scientists to help build the next generation of post-trai...
Location
Location
Switzerland , Zürich
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands‑on experience with large language models, including training or applying them in production (not just prompting)
  • Designing and running post‑training experiments (evals, ablations, preference tuning / RLHF‑style methods)
  • Building and owning scalable data pipelines for training and evaluation data
  • Strong Python skills for ML experimentation, data processing, and analysis
  • Solid statistical, experimental, and general engineering fundamentals
Job Responsibility
Job Responsibility
  • Design evaluations of advanced model capabilities and use them to drive rapid, high-signal iteration loops
  • Work with vendors to produce high quality evaluation and training data
  • Build data pipelines to produce high quality evaluation and training data
  • Build data flywheels to hill-climb on model weaknesses, using data from various surfaces where our models are deployed
  • Ensure optimal quality, quantity and coverage of data across our post-training stages
  • Run post-training experiments and ablations to produce models that climb our evals
  • Embody our culture and values
  • Fulltime
Read More
Arrow Right

Member of Technical Staff, Data Research Engineer

We are seeking Data Research Engineers to join our Multimodal team, where we are...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in AI, Computer Science, Data Science, Statistics, Physics, Engineering, or a related technical field AND technical engineering experience with coding in languages including, but not limited to, Python and common data libraries (Pandas, NumPy, etc.)
  • OR equivalent experience
  • Experience in data analysis or data engineering
  • Proficiency in statistics and exploratory data analysis methods
  • Ability to communicate technical findings effectively to research and product teams
Job Responsibility
Job Responsibility
  • Create high-quality datasets for training and evaluation
  • run experiments on new datasets (data ablations) to assess their impact and determine the most effective data
  • Develop and maintain scalable data pipelines for multimodal ingestion, pre-processing, filtering, and annotation
  • Analyse real-world multimodal datasets to assess quality, diversity, relevance, and identify areas for improvement
  • Build lightweight tools and workflows for dataset auditing, visualization, and versioning
  • Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices
  • Fulltime
Read More
Arrow Right

Data Quality Lead Analyst

The Data Quality Lead Analyst is a strategic professional who stays abreast of d...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years software experience preferred
  • High proficiency in MS Excel, Python, SQL for Data analysis
  • Ability to create BI tools such as, Qlikview and/or Tableau views for analysis
  • Problem Solving - Evaluates moderately complex and variable issues with substantial potential impact
  • Requires good analytical skills in order to filter, prioritize and validate potentially complex and dynamic material from multiple sources
  • Outstanding written and oral communication, influencing and presentation skills
  • A self-starter with ability to independently manage work and drive multiple deliverables concurrently
  • Strong subject matter expertise regarding technology application control disciplines and technology infrastructure knowledge, with a sound understanding of the financial services provided by Citi
  • Demonstrated capability and maturity to work in an individual capacity and must proactively seek out ways to enhance the mission and identify new data analysis needs
  • Demonstrates experience in managing teams and managing integrated internal audit and assurance delivery within a matrix reporting environment
Job Responsibility
Job Responsibility
  • Devises methods for identifying data patterns and trends in available data sources
  • Defines strategies to drive data quality measurement, produce data quality dashboards and reports, and implement data quality strategies to effectively govern data and improve data quality
  • Works with data quality partners and Technology teams to identify and implement data quality tools. Identify critical data elements, data quality rules, thresholds, and other business requirements
  • Mentors junior analysts
  • Applies in-depth understanding and knowledge of how business integrates within the sub function
  • as well as coordinates and contributes to the objectives of the function and overall business
  • Contributes to the development of new techniques and processes and for the aligned business
  • Integrates subject matter and industry expertise within a defined area
  • Responsible for own work to support business teams in assigned area
  • Build data quality assessments and measurement of critical data utilized by Citi strategic systems and processes supporting Citi Global Functions and Sectors
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Pretraining Text Data

We are seeking engineers and researchers to join our Pretraining Text Data team,...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in AI, Computer Science, Data Science, Statistics, Physics, Engineering, or related technical discipline AND technical engineering experience with coding in languages including, but not limited to, Python and common data libraries (Pandas, NumPy, etc.) OR equivalent experience.
  • 2+ years of experience in data analysis or data engineering, including work with large-scale datasets that are unstructured or semi-structured.
  • Proficiency in statistics and exploratory data analysis methods.
Job Responsibility
Job Responsibility
  • Create high-quality datasets for training and evaluation
  • run experiments on new datasets (data ablations) to assess their impact and determine the most effective data.
  • Develop and maintain scalable data pipelines for text data ingestion, preprocessing, filtering, and annotation.
  • Analyze real-world text datasets to assess quality, diversity, relevance, and identify areas for improvement.
  • Build lightweight tools and workflows for dataset auditing, visualization, and versioning.
  • Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices.
  • Embody our culture and values.
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Full Stack Software Engineer

Microsoft AI is looking for a Member of Technical Staff – Full Stack Software En...
Location
Location
United States , Mountain View
Salary
Salary:
100600.00 - 199000.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to C#, Java, or Python OR equivalent experience.
  • 2+ years’ experience with SQL, PostgreSQL or MySQL
  • 2+ years' experience building scalable services on top of public cloud infrastructure like Azure, AWS, or GCP. Extensive use datastores like RDBMS, key-value stores, etc.
  • Experience with Development & Debugging with dev environments like Visual Studio or Visual Studio Code
  • HTML, CSS, JavaScript, ASP.NET, REST, jQuery
  • Familiarity with browser automation tools like Selenium, Puppeteer or Playwright
  • Familiarity with mobile automation tools like Appium
  • Familiarity with LLMs and AI ChatBots
  • Prompt EngineeringAzure DevOps, GIT
  • Azure Open AI, Azure Foundry
Job Responsibility
Job Responsibility
  • Expertise in experimentation methodologies, including A/B evaluation, data sampling, measurement techniques, evaluation design, and data analysis.
  • Demonstrating strategic vision by understanding organizational goals, translating metrics into actionable insights, and enhancing product quality.
  • Designing pipeline architecture to ensure rapid iteration and scalability.
  • Conducting post-analysis of labeled data and developing dashboards to visualize insights.
  • Collaborating closely with the product team to enhance quality and address gaps.
  • Embody our Culture and Values.
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Full Stack Software Engineer

Microsoft AI is looking for a Member of Technical Staff – Full Stack Software En...
Location
Location
United States , Redmond; Mountain View
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to C#, Java, or Python
  • OR equivalent experience
  • 2+ years’ experience with SQL, PostgreSQL or MySQL
  • 2+ years' experience building scalable services on top of public cloud infrastructure like Azure, AWS, or GCP
  • Extensive use datastores like RDBMS, key-value stores, etc.
  • Experience with Development & Debugging with dev environments like Visual Studio or Visual Studio Code
  • HTML, CSS, JavaScript, ASP.NET, REST, jQuery
  • Familiarity with browser automation tools like Selenium, Puppeteer or Playwright
  • Familiarity with mobile automation tools like Appium
  • Familiarity with LLMs and AI ChatBots
Job Responsibility
Job Responsibility
  • Expertise in experimentation methodologies, including A/B evaluation, data sampling, measurement techniques, evaluation design, and data analysis
  • Demonstrating strategic vision by understanding organizational goals, translating metrics into actionable insights, and enhancing product quality
  • Designing pipeline architecture to ensure rapid iteration and scalability
  • Conducting post-analysis of labeled data and developing dashboards to visualize insights
  • Collaborating closely with the product team to enhance quality and address gaps
  • Embody our Culture and Values
  • Fulltime
Read More
Arrow Right
New

Member Of Technical Staff, Microsoft Robotics (Robotics Data)

Microsoft’s Discovery and Quantum (MDQ) division develops and delivers advanced ...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3+ years data-science experience
  • OR Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience
  • OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
Job Responsibility
Job Responsibility
  • Define and implement data collection strategies for robot learning, including specifying demonstration coverage requirements, environmental diversity targets, task distribution plans, and quality acceptance criteria for teleoperation, egocentric, and autonomous data collection campaigns
  • Build and maintain data curation pipelines that ingest, clean, validate, label, and version robotics datasets (manipulation demonstrations, navigation trajectories, sensor logs, simulation rollouts), ensuring data integrity and provenance tracking
  • Develop data analysis frameworks that quantify dataset characteristics (coverage, diversity, balance, quality scores), identify data gaps and biases, and provide recommendations for targeted data collection to improve model performance
  • Create interactive data visualization tools and dashboards (using tools such as Power BI, Plotly, or custom web applications) that enable researchers, engineers, and leadership to explore dataset properties, model training metrics, evaluation results, and fleet operational telemetry
  • Collaborate with ML researchers and learning engineers to design and execute experiments that measure the impact of data quantity, quality, and diversity on model performance, producing statistical analyses that guide data investment decisions
  • Formulate and maintain a roadmap of data science project activity that leads to measurable improvement in model performance metrics, data pipeline efficiency, and data quality over time
  • Develop and apply statistical techniques (hypothesis testing, causal inference, regression analysis, clustering) to analyze robot performance data, identify failure modes, and uncover patterns that inform model architecture and training strategy decisions
  • Write efficient, readable, extensible code in Python (including Pandas, NumPy, scikit-learn, matplotlib) for data processing, analysis, and visualization, building professional-grade documentation for knowledge transfer
  • Adhere and contribute to ethics and privacy policies related to collecting and preparing robotics data, providing guidance on responsible data practices including bias detection, consent, and data governance
  • Present results and findings to senior stakeholders, using compelling visualizations and storytelling to influence data investment priorities and model development strategy
What we offer
What we offer
  • Benefits and other compensation may be eligible
  • Fulltime
Read More
Arrow Right