CrawlJobs Logo

Senior Research Scientist, Model Evaluation

United States; Canada; United Kingdom, Toronto · Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

Evaluation is critical to making progress in scaling intelligence. As models continue to become superhuman in many real-world use cases, we must continue to develop new evaluation techniques that accurately reflect what models are already capable of, as well as set the agenda for what future models should be capable of. In this role, you are responsible for creating these next-generation evaluation methods and infrastructure to measure LLM progress.

Job Responsibility

  • Create ambitious new evaluation benchmarks that push the limits of what our models can accomplish
  • Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations
  • Conduct research to advance the state-of-the-art in LLM evaluation methods, including training LLM judges
  • refining LLM-based data synthesis pipelines
  • and improving evaluation efficiency
  • Build scalable and reusable tools for digging into model performance

Requirements

  • Enjoy rapidly building prototypes that demonstrate the boundaries of what LLMs are capable of
  • Have developed resources to measure LLM capabilities
  • Have spent dozens of hours reviewing complex data and LLM outputs to ensure high data quality
  • Obsessive about rigorously measuring AI capabilities and ensuring measurements align with the capabilities you care about
  • Have strong software engineering skills

What we offer

  • An open and inclusive culture and work environment
  • Work closely with a team on the cutting edge of AI research
  • Weekly lunch stipend, in-office lunches & snacks
  • Full health and dental benefits, including a separate budget to take care of your mental health
  • 100% Parental Leave top-up for up to 6 months
  • Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • 6 weeks of vacation (30 working days!)

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Senior Research Scientist, Model Evaluation

8 matching positions

Senior Scientist – Research Computational Biology Platforms

The ARIA Computational Biology team is seeking a talented and creative computati...
Location
Location
United States , South San Francisco; Boston; Thousand Oaks
Salary
Salary:
146404.00 - 198076.00 USD / Year
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • PhD in computational biology, bioinformatics, data science, or a related discipline [and relevant post-doc where applicable]
  • Master’s degree in computational biology, bioinformatics, data science, or a related discipline and 3 years of relevant research experience
  • Bachelor’s degree in computational biology, bioinformatics, data science, or a related discipline and 5 years of relevant research experience
  • Strong programming expertise in Python and R, with proficiency in Linux/Unix environments, cloud computing, HPC, and collaborative development practices (e.g., Git)
  • Experience developing complex computational tools and visualizations that translate biological insights into actionable strategies
  • Experience developing and deploying computational tools and pipelines, including use of workflow management systems (e.g., Nextflow), CI/CD practices, and interactive visualization tools (e.g., Shiny/Streamlit)
  • Strong background in molecular and systems biology
  • Demonstrated expertise in the analysis and interpretation of single-cell omics data
  • Experience leveraging and fine-tuning transcriptional foundation models, large language models (LLMs), and biomedical knowledge graphs to advance research objectives
  • Familiarity with high-value public data resources (e.g., DepMap, Human Cell Atlas, TCGA, GTEx, and Tahoe-100M) used to complement and enhance analyses of internal datasets
Job Responsibility
Job Responsibility
  • Developing and applying methods and platforms to discover, prioritize, and validate targets across a broad range of diseases and therapeutic modalities
  • Designing and building a Precision Targeting Engine that integrates internal and high-value public data into an automated, scalable framework with custom expression analytics across diverse data types and disease areas
  • Contributing to the design and evolution of Amgen’s preclinical omics data ecosystem in partnership with the Research Informatics team
  • Evaluating and integrating emerging AI/ML technologies (e.g., biological foundation models), and developing custom research agents to accelerate early pipeline efforts
  • Partnering with data science and information systems teams to embed tools and analyses developed within ARIA Computational Biology into integrated R&D platforms
  • Training and mentoring other scientists in the use of advanced research tools and platforms
What we offer
What we offer
  • A comprehensive employee benefits package, including a Retirement and Savings Plan with generous company contributions, group medical, dental and vision coverage, life and disability insurance, and flexible spending accounts
  • A discretionary annual bonus program, or for field sales representatives, a sales-based incentive plan
  • Stock-based long-term incentives
  • Award-winning time-off plans
  • Flexible work models where possible
  • Fulltime
Read More
Arrow Right

Senior Research Scientist

The Data and AI team in Microsoft CoreAI is working on the future of AI for Deve...
Location
Location
China , Shanghai
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4-year degree in computer science or engineering AND 3+ years related-research experience OR equivalent experience
  • Strong academic work and professional experience in statistics, machine learning, including deep learning, NLP, econometrics
  • Experience in building cloud-scale systems and experience working with open-source stacks for data processing and data science is desirable
  • Experience with LLMs in natural language, AI for code or related fields
  • Experience in productizing AI and collaborating with multidisciplinary teams
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Lead the development and evaluation of state-of-the-art models for code completion and editing, pushing the boundaries of code understanding, generation, fix and review
  • Develop retrieval-augmented systems that improve a model’s awareness of large and complex codebases, enabling context-rich code assistance
  • Design and prototype efficient inference algorithms to enable fast, interactive code generation experiences at scale
  • Collaborate across disciplines with product teams across Microsoft and Github
  • Stay up to date with the research literature and product advances in AI for software engineering
  • Fulltime
Read More
Arrow Right

Senior Research Scientist

PolyAI automates customer service through lifelike voice assistants that let cus...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
poly.ai Logo
PolyAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • PhD in Machine Learning, Natural Language Processing, Computer Science, or a related field
  • 5+ years of hands-on experience in deep learning
  • Proven track record of research innovation, including published work or deployed systems
  • Strong programming skills in Python and deep learning frameworks like PyTorch
  • Demonstrated expertise in at least one domain area such as reinforcement learning, conversational AI, audio modelling, or LLM alignment
  • Experience leading projects end-to-end, from ideation to deployment
  • Excellent communication skills with the ability to write clear technical documents and explain complex concepts to diverse audiences
  • Comfortable working in ambiguity and driving clarity through experimentation and data
Job Responsibility
Job Responsibility
  • Lead and execute complex research projects with clear business impact
  • Design and implement novel post-training strategies including preference tuning, reward modeling, and synthetic supervision
  • Develop innovative model architectures and training approaches for conversational AI, including speech-aware and multimodal models
  • Conduct empirical studies to assess model performance in live deployments and iterate quickly based on real-world data
  • Generate, collect, and annotate training data - including synthetic and real-world conversational datasets - with an eye for quality and bias mitigation
  • Design robust evaluation metrics and benchmarks for LLM-based assistants in customer service domains
  • Work closely with engineering and product teams to integrate research into production environments
  • Collaborate with legal and compliance teams to ensure responsible use of data and models
  • Stay current with academic and industry advances in LLMs, ASR, TTS, RLHF, and multimodal learning
What we offer
What we offer
  • Participation in the company’s employee share options plan
  • Tenure-Based PTO: You will receive 25 holidays when you join and will gain an additional 1 day after 2 years of service, then 1 day each year until capped at 32 holidays
  • Flexible working from home policy
  • Work from outside of the UK for up to 6 months each year
  • TELUS Health EAP 24/7 - offers you and your chosen family confidential, judgment-free support for any work, health, or life challenge
  • Enhanced parental leave
  • Bike2Work scheme
  • Annual learning and development allowance
  • We’re all about making WFH work for you - that’s why we offer a one-off WFH allowance when you join. Offering perks like noise-cancelling headphones or a comfortable desk chair to boost your comfort and focus!
  • Company-funded fertility and family-forming programmes
Read More
Arrow Right

Senior Staff Research Scientist - Clinical AI Lab

At Doctolib, we're revolutionizing healthcare delivery through advanced AI syste...
Location
Location
France , Paris
Salary
Salary:
Not provided
doctolib.fr Logo
Doctolib
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • PhD or Master’s degree (plus significant experience) in Computer Science, Machine Learning, Mathematics, or a related field
  • Publications as first author in top-tier AI/ML conferences such as NeurIPS, ICLR, ICML, ACL, EMNLP, or relevant medical informatics venues are a plus
  • Deep understanding of machine learning and deep learning concepts, with hands-on experience in NLP, LLMs, or generative AI
  • Strong programming skills in Python (and ideally experience with ML frameworks such as PyTorch or JAX)
  • Familiarity with modern data tools and scalable training on large datasets
  • Track record of independently driving research and translating it into real-world applications
  • Experience designing experiments, evaluating intrinsic and extrinsic metrics, and iterating quickly
  • Curiosity and drive to learn, research, and apply new machine learning techniques
  • Comfortable working in highly cross-functional teams, with the ability to clearly communicate complex concepts to diverse audiences
  • Passionate about sharing knowledge and contributing to a culture of learning and innovation
Job Responsibility
Job Responsibility
  • Solve Real-World Healthcare Challenges: Design and build advanced machine learning models, especially in NLP and large language models, to improve healthcare workflows, enhance patient experiences, and power intelligent health solutions at scale
  • Innovate & Experiment: Research and prototype novel algorithms and architectures, staying at the forefront of AI/ML developments. Bring new ideas from concept to working prototypes, leveraging the latest advances in deep learning and generative AI
  • Collaborate Across Teams: Work closely with product, engineering, medical experts, and other stakeholders to translate business needs into impactful research projects. Communicate technical concepts and results clearly to both technical and non-technical audiences
  • Drive Impact: Analyze large, complex data sets to extract actionable insights, define key metrics, and rigorously evaluate model performance
  • Share & Learn: Advance Doctolib’s AI expertise by sharing findings within the team and with the wider ML community through publications, talks, and workshops. Mentor and support other scientists and engineers in adopting best practices
What we offer
What we offer
  • Free comprehensive health insurance for you and your children
  • 25 days of paid vacation per year, plus up to 14 days of RTT
  • Free mental health and coaching services through our partner Moka.care
  • Work from abroad for up to 10 days per year thanks to our flexibility days policy
  • Lunch vouchers (Swile card) worth €8.50 per working day, with €4.50 covered by Doctolib
  • A subsidy from the work council to refund part of the membership to a sport club or a creative class
  • 50% reimbursement of your public transport subscription
  • Parent Care Program: receive one additional month of leave on top of the legal parental leave
  • For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
  • Relocation support in case of international mobility
  • Fulltime
Read More
Arrow Right

Senior / Lead Research Scientist

Inworld is a product-oriented research lab of top AI researchers and engineers, ...
Location
Location
Germany
Salary
Salary:
Not provided
inworld.ai Logo
Inworld AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Foundation models: training, new architectures, RL, reward modeling, scaling
  • Evaluation: benchmarks, eval loops, quality measurement, LLM-as-judge, failure analysis
  • Frontier topics: multimodal models, agents, tool use, test-time compute, world models
  • Published research at ICML, ICLR, NeurIPS, EMNLP, ACL, or AAAI
  • PhD in ML/NLP — or equivalent practical experience you can point to
  • Public work: non-trivial AI side projects, interdisciplinary experiments, open-source contributions
  • Full-stack research ownership: you frame the question, run the experiments, write the paper, ship the result
  • Professional fluency in English (written and spoken)
Read More
Arrow Right

Senior / Lead Research Scientist

A year ago, reliably working agentic systems barely existed. Nobody has a decade...
Location
Location
Serbia
Salary
Salary:
Not provided
inworld.ai Logo
Inworld AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Foundation models: training, new architectures, RL, reward modeling, scaling
  • Evaluation: benchmarks, eval loops, quality measurement, LLM-as-judge, failure analysis
  • Frontier topics: multimodal models, agents, tool use, test-time compute, world models
  • Published research at ICML, ICLR, NeurIPS, EMNLP, ACL, or AAAI
  • PhD in ML/NLP — or equivalent practical experience you can point to
  • Public work: non-trivial AI side projects, interdisciplinary experiments, open-source contributions
  • Full-stack research ownership: you frame the question, run the experiments, write the paper, ship the result
  • Professional fluency in English (written and spoken)
Read More
Arrow Right

Senior AI Research Scientist

Workato transforms technology complexity into business opportunity. As the leade...
Location
Location
United States , San Francisco
Salary
Salary:
Not provided
workato.com Logo
Workato
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • MS/PhD in Computer Science, ML, or related field—or equivalent experience
  • Hands‑on with PyTorch/JAX and modern LLM frameworks
  • Strong publication record in top-tier ML venues (NeurIPS, ICML, ICLR) or demonstrable impactful experimental work
  • hands‑on experience with large‑scale model training, transformer architectures, reinforcement‑learning techniques, and CUDA
  • Curiosity‑driven and self‑directed
  • Ability to explain complex ideas to non‑experts
  • Team player who welcomes feedback and knowledge‑sharing
Job Responsibility
Job Responsibility
  • Explore workflow‑aware planning, cost‑optimised RAG, tool‑use evaluation, and multi‑agent collaboration
  • Prototype and benchmark models
  • present findings internally and externally
  • Collaborate with Product & Engineering to ship research‑backed features
  • Contribute to open‑source repos and author technical blogs
What we offer
What we offer
  • vibrant and dynamic work environment
  • multitude of benefits they can enjoy inside and outside of their work lives
Read More
Arrow Right

Senior AI Research Scientist

Join Axon and be a Force for Good. At Axon, we’re on a mission to Protect Life. ...
Location
Location
Finland , Tampere
Salary
Salary:
Not provided
axon.com Logo
Axon
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • PhD in Computer Science or a related field with a focus on LLM, MLLMs, Computer Vision, GenAI
  • +5 years for ML Scientist, +8 years for Sr. ML Scientist, +10 years for Principal ML Scientist experience
  • Proven track record of research excellence in LLM, MLLM, Computer Vision, Robotics Perception, GenAI, demonstrated through publications in top-tier conferences or journals
  • Strong proficiency in programming languages such as Python, C/C++
  • Experience with deep learning frameworks such as TensorFlow, PyTorch, or Keras
  • Experience with ROS or robotic operational system
  • Drive one or more phases of the ML development lifecycle: shape datasets, investigate modeling approaches and architectures, train/evaluate/tune models and implement the end-to-end training pipeline
  • Leverage state-of-the-art research to deliver high quality models enabling multiple AI projects at scale
  • Contribute back to the research community via academic publications, tech blogs, open-source code and contributing to internal/external AI challenges
  • Experience in developing computer vision algorithms for resource-constrained devices such as mobile phones, IoT devices, or embedded systems is highly desirable
Job Responsibility
Job Responsibility
  • Own one or more key technical areas across LLM, MLLM, CV product portfolio
  • Provide technical leadership to junior scientists, guiding the transition of R&D concepts into impactful Axon product feature
  • Research and develop cutting-edge techniques in LLM, MLLMs, GenAI, and Computer Vision across cloud, devices and sensors based data sources
  • Design and implement efficient and scalable MLLM models for inference and analysis of multimodal data
  • Explore novel approaches to address challenges in NLP, NLU, Object Detection, Object Recognition, Object Tracking, Segmentation, and Scene Understanding
  • Optimize AI models, algorithms for performance, memory footprint, and energy efficiency to meet the requirements of resource-constrained devices
  • Join force with MLEs or firmware or hardware engineers to leverage hardware accelerators and optimize algorithms for specific hardware architectures
  • Evaluate the performance of LLM, MLLM, CV models using real-world datasets and design experiments to validate their effectiveness
  • Stay up-to-date with the latest research trends and advancements in computer vision, machine learning, and deep learning, MLLMs, GenAI and integrate relevant findings into our projects
  • Contribute to patent disclosures, academic publications, and technical documentation to share insights and findings with the broader community
What we offer
What we offer
  • Competitive salary and 401k with employer match
  • Discretionary paid time off
  • Paid parental leave for all
  • Medical, Dental, Vision plans
  • Fitness Programs
  • Emotional & Mental Wellness support
  • Learning & Development programs
  • Snacks in our offices
  • Fulltime
Read More
Arrow Right