Tech Lead Manager Machine Learning Research Scientist LLM Evals Job at Scale (San Francisco)

Job Description

As the Tech Lead Manager of the LLM Evals Research team, you will lead a talented team of research scientists and research engineers focused on developing and implementing novel evaluation methodologies, metrics, and benchmarks to assess the capabilities and limitations of our cutting-edge LLMs. This role is critical for designing and executing a roadmap that defines best practices in data driven AI development and will accelerate the next generation of generative AI models in partnership with top foundational model labs.

Job Responsibility

Lead a team of highly effective research scientists and research engineers on LLM evals
Conduct research on the effectiveness and limitations of existing LLM evaluation techniques
Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness
Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects
Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols
Implement scalable and reproducible evaluation pipelines using modern ML frameworks
Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives
Remain up-to-date on ongoing research in the team, help work through technical challenges, and be involved in design decisions
Remain deeply involved in the research community, both understanding trends, and setting them
Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results

Requirements

5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development
Experience and track of recording in landing major research impacts in a fast-paced environment
Experience supporting and leading a team of research scientists and research engineers
Excellent written and verbal communication skills
Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
Previous experience in a customer facing role

What we offer

Comprehensive health, dental and vision coverage
retirement benefits
a learning and development stipend
generous PTO

Scale - All Job Offers

Select Country

Tech Lead Manager Machine Learning Research Scientist LLM Evals

Job Description

Job Responsibility

Requirements

What we offer

Looking for more opportunities?

Tech Lead Manager Machine Learning Research Scientist LLM Evals

Inside Sales Supervisor

Senior Inside Sales Representative

Site Manager

Associate Mental Health Therapist

Payroll/ HR Clerk

Asbestos Surveyor

Water Technician - Level 2

Research Technician

Our AI answers in your language