CrawlJobs Logo

Member of Technical Staff, Data Analysis and Evaluation

· Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability, and performance of our large language models (LLMs). Your primary focus will be on designing and conducting data collection tasks, assessing and evaluating dataset quality, and analysing the robustness and generalisability of our models. You will work closely with cross-functional teams, including researchers, engineers, and data annotators, to conduct data-driven decision-making and improve the overall effectiveness of our AI systems. This role combines expertise in statistics, experimental design incl. human annotators, and machine learning to ensure that our models are trained on high-quality data and perform reliably across diverse scenarios. You will contribute to Cohere’s mission of advancing AI by ensuring our systems are robust, scalable, and impactful.

Job Responsibility

  • Design and oversee data collection tasks, including supporting human annotators and ensuring data quality
  • Develop and apply statistical methods to evaluate the quality and reliability of datasets
  • Analyse and assess the generalisability and robustness of ML systems across diverse use cases
  • Collaborate with teams to improve dataset quality and model performance
  • Train and fine-tune large language models (LLMs) on distributed training infrastructures
  • Conduct experiments to evaluate model performance and identify areas for improvement

Requirements

  • Extremely strong software engineering skills
  • Strong expertise in designing and conducting data collection tasks, including working with human annotators
  • Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance
  • Experience analysing datasets with respect to their quality, biases, and suitability for training ML models
  • Hands-on experience training large language models (LLMs) on distributed training infrastructures
  • Familiarity with evaluating and improving the generalisability and robustness of ML systems
  • Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX)
  • Excellent communication skills to collaborate effectively with cross-functional teams and present findings
  • One or more papers at top-tier venues (such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP)

What we offer

  • An open and inclusive culture and work environment
  • Work closely with a team on the cutting edge of AI research
  • Weekly lunch stipend, in-office lunches & snacks
  • Full health and dental benefits, including a separate budget to take care of your mental health
  • 100% Parental Leave top-up for up to 6 months
  • Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • 6 weeks of vacation (30 working days!)

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Member of Technical Staff, Data Analysis and Evaluation

8 matching positions

Member of Technical Staff - Data Scientist

As a data scientist at Microsoft AI, you will be tasked with helping us to maxim...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR equivalent experience
Job Responsibility
Job Responsibility
  • Drive product insights, opportunity analysis, and track metrics to support efforts across Microsoft Copilot
  • Drive new ways of instrumenting and measuring impact to evaluate new feature performance through experimentation
  • Define metrics and build basic data pipelines to enable A/B experimentation for new features and mitigating abusive users
  • Hands-on analysis of large volumes of telemetry data using various algorithms and tools including your own
  • Articulate insights, storyboard with data and communicate to influence leadership and other key decision makers
  • Find a path to get things done despite roadblocks to get your work into the hands of users quickly and iteratively
  • Enjoy working in a fast-paced, design-driven, product development cycle
  • Work collaboratively with engineers, Product Managers, and marketing to take ambiguous projects that drive user growth, engagement, and retention
  • Embody our Culture and Values
  • Fulltime
Read More
Arrow Right

Member of Technical Staff, Data Scientist

As Microsoft continues to push the boundaries of AI, we are on the lookout for p...
Location
Location
United States , Mountain View
Salary
Salary:
119800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR equivalent experience.
Job Responsibility
Job Responsibility
  • Develop metrics for Copilot usage and quality
  • Drive product insights, opportunity analysis, and track metrics to support Microsoft Copilot
  • Drive new ways of instrumentation and measurement approach to evaluate new feature performance through experimentation
  • Enable A|B experimentation for new features
  • Hands-on analysis of large volumes of telemetry data using various algorithms and tools including your own
  • Articulate insights, storyboard with data and communication to influence leadership and other key decision makers
  • Find a path to get things done despite roadblocks to get your work into the hands of users quickly and iteratively
  • Enjoy working in a fast-paced, design-driven, product development cycle
  • Embody our Culture and Values
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Data Scientist

We’re looking for data scientists to help build the next generation of post-trai...
Location
Location
Switzerland , Zürich
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands‑on experience with large language models, including training or applying them in production (not just prompting)
  • Designing and running post‑training experiments (evals, ablations, preference tuning / RLHF‑style methods)
  • Building and owning scalable data pipelines for training and evaluation data
  • Strong Python skills for ML experimentation, data processing, and analysis
  • Solid statistical, experimental, and general engineering fundamentals
Job Responsibility
Job Responsibility
  • Design evaluations of advanced model capabilities and use them to drive rapid, high-signal iteration loops
  • Work with vendors to produce high quality evaluation and training data
  • Build data pipelines to produce high quality evaluation and training data
  • Build data flywheels to hill-climb on model weaknesses, using data from various surfaces where our models are deployed
  • Ensure optimal quality, quantity and coverage of data across our post-training stages
  • Run post-training experiments and ablations to produce models that climb our evals
  • Embody our culture and values
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Pretraining Text Data

We are seeking engineers and researchers to join our Pretraining Text Data team,...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in AI, Computer Science, Data Science, Statistics, Physics, Engineering, or related technical discipline AND technical engineering experience with coding in languages including, but not limited to, Python and common data libraries (Pandas, NumPy, etc.) OR equivalent experience.
  • 2+ years of experience in data analysis or data engineering, including work with large-scale datasets that are unstructured or semi-structured.
  • Proficiency in statistics and exploratory data analysis methods.
Job Responsibility
Job Responsibility
  • Create high-quality datasets for training and evaluation
  • run experiments on new datasets (data ablations) to assess their impact and determine the most effective data.
  • Develop and maintain scalable data pipelines for text data ingestion, preprocessing, filtering, and annotation.
  • Analyze real-world text datasets to assess quality, diversity, relevance, and identify areas for improvement.
  • Build lightweight tools and workflows for dataset auditing, visualization, and versioning.
  • Collaborate with Safety, Ethics, and Governance teams to ensure datasets meet standards for quality, privacy, and responsible AI practices.
  • Embody our culture and values.
  • Fulltime
Read More
Arrow Right

Member Of Technical Staff, Microsoft Robotics (Robotics Data)

Microsoft’s Discovery and Quantum (MDQ) division develops and delivers advanced ...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3+ years data-science experience
  • OR Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience
  • OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
Job Responsibility
Job Responsibility
  • Define and implement data collection strategies for robot learning, including specifying demonstration coverage requirements, environmental diversity targets, task distribution plans, and quality acceptance criteria for teleoperation, egocentric, and autonomous data collection campaigns
  • Build and maintain data curation pipelines that ingest, clean, validate, label, and version robotics datasets (manipulation demonstrations, navigation trajectories, sensor logs, simulation rollouts), ensuring data integrity and provenance tracking
  • Develop data analysis frameworks that quantify dataset characteristics (coverage, diversity, balance, quality scores), identify data gaps and biases, and provide recommendations for targeted data collection to improve model performance
  • Create interactive data visualization tools and dashboards (using tools such as Power BI, Plotly, or custom web applications) that enable researchers, engineers, and leadership to explore dataset properties, model training metrics, evaluation results, and fleet operational telemetry
  • Collaborate with ML researchers and learning engineers to design and execute experiments that measure the impact of data quantity, quality, and diversity on model performance, producing statistical analyses that guide data investment decisions
  • Formulate and maintain a roadmap of data science project activity that leads to measurable improvement in model performance metrics, data pipeline efficiency, and data quality over time
  • Develop and apply statistical techniques (hypothesis testing, causal inference, regression analysis, clustering) to analyze robot performance data, identify failure modes, and uncover patterns that inform model architecture and training strategy decisions
  • Write efficient, readable, extensible code in Python (including Pandas, NumPy, scikit-learn, matplotlib) for data processing, analysis, and visualization, building professional-grade documentation for knowledge transfer
  • Adhere and contribute to ethics and privacy policies related to collecting and preparing robotics data, providing guidance on responsible data practices including bias detection, consent, and data governance
  • Present results and findings to senior stakeholders, using compelling visualizations and storytelling to influence data investment priorities and model development strategy
What we offer
What we offer
  • Benefits and other compensation may be eligible
  • Fulltime
Read More
Arrow Right

Member of Technical Staff, Microsoft Robotics

Overview Microsoft’s Discovery and Quantum (MDQ) division develops and delivers...
Location
Location
United States , Redmond
Salary
Salary:
165600.00 - 296400.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
  • Ability to meet Microsoft, customer and/or government security screening requirements is required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
Job Responsibility
Job Responsibility
  • Own the multi-year technical architecture for the Microsoft Robotics platform, setting the direction the broader engineering organization, partners, and customers build on, and holding that architecture coherent as it scales across teams and deployments.
  • Make the foundational design decisions that determine how robotics software, AI models, and cloud and edge components fit together, defining the core interfaces and engineering contracts other teams depend on.
  • Architect systems that meet demanding real-world constraints, balancing latency, throughput, compute efficiency, reliability, and cost across cloud and on-robot edge environments.
  • Lead the evaluation and validation strategy that gates what ships, establishing the benchmarks, test approach, and quality bar for autonomy, safety, and task performance across simulation, lab, and field, including safe-autonomy boundaries and human-in-the-loop fallback.
  • Drive the hardest cross-stack technical problems personally, taking on the integration failure modes that emerge where AI models meet production robots, and resolving them through rigorous root-cause analysis of system behavior and field data.
  • Connect classical robotics engineering with modern AI, partnering across foundation-model, perception, manipulation, locomotion, and simulation teams to bring learned capabilities into a common, production-grade platform rather than one-off integrations.
  • Establish the engineering bar for the team, owning design standards and review practices, leading the consequential technical reviews, and lifting the quality, testability, and operability of the systems that matter most.
  • Influence direction across MDQ, Microsoft Research, Azure, hardware and silicon partners, and customers without relying on formal authority, building technical consensus and aligning many teams on a single coherent platform strategy, including representing the platform in the open-source and standards community.
  • Multiply the team's output through technical leadership, mentoring senior and principal engineers and raising the technical ceiling of the organization rather than only its headcount.
  • Track the state of the art across embodied AI, robot learning, robotics middleware, and simulation, and decide deliberately how and when new techniques are incorporated into the platform.
  • Fulltime
Read More
Arrow Right

Member Of Technical Staff, Microsoft Robotics (Perception)

Microsoft's Discovery and Quantum (MDQ) division develops and delivers advanced ...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
Job Responsibility
Job Responsibility
  • Lead the design, development, and integration of perception and navigation systems for robotic platforms, including SLAM, localization, mapping, sensor fusion, obstacle detection, semantic scene understanding, and motion planning interfaces
  • Develop robust state estimation and localization pipelines using data from cameras, depth sensors, LiDAR, IMUs, wheel odometry, proprioception, GPS, force/torque sensors, or other robot sensor modalities
  • Build spatial representations for navigation and planning, including occupancy maps, semantic maps, costmaps, 3D scene representations, traversability maps, object-level representations, and dynamic environment models
  • Integrate perception outputs with planning, control, simulation, and fleet operations systems to support safe and reliable robot autonomy in real-world environments
  • Architect perception-navigation pipelines that meet practical constraints for latency, bandwidth, compute efficiency, reliability, calibration drift, sensor degradation, and environmental variability
  • Develop evaluation frameworks, benchmarks, and telemetry-driven analysis methods to measure perception, localization, mapping, and navigation performance across simulation, lab, and field deployments
  • Partner with learning engineers, spatial AI engineers, simulation engineers, and platform engineers to connect classical robotics pipelines with modern AI models, including learned perception, physical world models, and data-driven navigation approaches
  • Analyze field logs, sensor data, planner behavior, and failure cases to identify root causes, improve system robustness, and drive cross-stack improvements across perception, planning, controls, and hardware integration
  • Lead technical design reviews, establish engineering patterns, and mentor other engineers on production-quality robotics autonomy development, testing, debugging, and deployment practices
  • Stay current with state-of-the-art research and industry practice in robotics perception, SLAM, VIO, sensor fusion, semantic mapping, spatial AI, navigation, and physical AI, assessing how new techniques should be incorporated into Microsoft's robotics platform.
What we offer
What we offer
  • Certain roles may be eligible for benefits and other compensation.
  • Fulltime
Read More
Arrow Right

Member Of Technical Staff, Microsoft Robotics (Field Robotics)

Microsoft’s Discovery and Quantum (MDQ) division develops and delivers advanced ...
Location
Location
United States , Redmond
Salary
Salary:
102100.00 - 202200.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Electrical Engineering, Computer Engineering, Mechanical Engineering, or related field AND 2+ years technical engineering experience
  • OR Master's Degree in Electrical Engineering, Computer Engineering, Mechanical Engineering, or related field
  • OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
Job Responsibility
Job Responsibility
  • Plan, coordinate, and execute field test campaigns for robotic systems across diverse real-world environments, including customer sites, partner facilities, and first-party scenarios, ranging from indoor semi-structured settings to unstructured outdoor terrains, that replicate operational conditions
  • Develop and maintain structured field test protocols, acceptance criteria, data collection procedures, and reporting templates that ensure repeatable, measurable evaluation of robot hardware, software, and AI model performance
  • Deploy, configure, and operate robotic platforms in the field, including hardware setup, software provisioning, network configuration, sensor calibration, and integration with site-specific infrastructure
  • Collect, organize, and analyze field test data including sensor logs, telemetry streams, performance metrics, video recordings, and environmental measurements, producing clear and actionable test reports for engineering and program leadership
  • Diagnose and troubleshoot field issues across the full robot stack (mechanical, electrical, software, networking, AI behavior), performing initial root cause analysis and resolving complex issues alongside engineering teams with well-documented reproduction steps
  • Collaborate with hardware engineers, software engineers, AI researchers, and program managers to communicate and integrate field findings, prioritize bug fixes and design improvements, and validate engineering changes through follow-up field testing
  • Identify and provide feedback on process gaps in field deployment and test procedures, sharing best practices broadly across the team and contributing to continuous improvement of field operations readiness
  • Maintain detailed documentation around field deployment procedures, site-specific configurations, inventory management, and service support workflows
  • Participate in test triage and review meetings to share knowledge, actively contribute to faster issue resolution, and identify readiness needs for upcoming field campaigns
  • Assist in the development of end-to-end readiness programs, including mentoring, knowledge sharing, technical document creation, and development of field operations training materials
What we offer
What we offer
  • Certain roles may be eligible for benefits and other compensation
  • Fulltime
Read More
Arrow Right