CrawlJobs Logo
Arena Intelligence, Inc. Logo Arena Intelligence, Inc. · IT - Software Development

Machine Learning Scientist - Open Source Lead

United States, Bay Area · Job Posted February 20, 2026
Apply Position
Job Link Share

Job Description

LMArena is looking for a Machine Learning Scientist to lead our open-source research, including open data set and code releases, advancing how the world evaluates and understands AI models in the open. You’ll design, run, and share new methods and experiments that reveal what makes models useful, trustworthy, and capable, grounded in human preference signals and released openly for the full ecosystem and research community to build upon.

Job Responsibility

  • Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions
  • Develop new metrics, methodologies, and evaluation protocols that go beyond traditional benchmarks
  • Analyze large-scale human voting and interaction data to uncover insights into model performance and user preferences
  • Communicate results with the broader research community via academic papers, educational content, conference talks
  • Collaborate with engineers to implement and scale research findings into production systems
  • Prototype and test research ideas rapidly, balancing rigor with iteration speed
  • Partner with model providers to shape evaluation questions and support responsible model testing
  • Contribute to the scientific integrity and transparency of the LMArena leaderboard and tools

Requirements

  • PhD or equivalent research experience in Machine Learning, Natural Language Processing, Statistics, or a related field
  • Strong understanding of LLMs and modern deep learning architectures (e.g., Transformers, diffusion models, reinforcement learning with human feedback)
  • Proficiency in Python and ML research libraries such as PyTorch, JAX, or TensorFlow
  • Demonstrated ability to design and analyze experiments with statistical rigor
  • Experience publishing research or working on open-source projects in ML, NLP, or AI evaluation
  • Comfortable working with real-world usage data and designing metrics beyond standard benchmarks
  • Ability to translate research questions into practical systems and collaborate across engineering and product teams
  • Passion for open science, reproducibility, and community-driven research

Nice to have

  • Skilled at public speaking, writing, and presenting research work to diverse audiences
  • Actively participates in conferences, panels, and online forums to foster relationships and thought leadership
  • Builds trust through transparent communication and consistent community engagement
  • Serves as a go-to contact for external researchers, journalists, and partners

What we offer

  • Comprehensive health and wellness benefits, including medical, dental, vision, and additional support programs
  • The opportunity to work on cutting-edge AI with a small, mission-driven team
  • A culture that values transparency, trust, and community impact
  • Competitive compensation and equity aligned to the markets where our team members are based
Arena Intelligence, Inc. - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Machine Learning Scientist - Open Source Lead

8 matching positions

Tech Lead Manager Machine Learning Research Scientist LLM Evals

As the Tech Lead Manager of the LLM Evals Research team, you will lead a talente...
Location
Location
United States , San Francisco; Seattle; New York
Salary
Salary:
280000.00 - 380000.00 USD / Year
scale.com Logo
Scale
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development
  • Experience and track of recording in landing major research impacts in a fast-paced environment
  • Experience supporting and leading a team of research scientists and research engineers
  • Excellent written and verbal communication skills
  • Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
  • Previous experience in a customer facing role
Job Responsibility
Job Responsibility
  • Lead a team of highly effective research scientists and research engineers on LLM evals
  • Conduct research on the effectiveness and limitations of existing LLM evaluation techniques
  • Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness
  • Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects
  • Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols
  • Implement scalable and reproducible evaluation pipelines using modern ML frameworks
  • Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives
  • Remain up-to-date on ongoing research in the team, help work through technical challenges, and be involved in design decisions
  • Remain deeply involved in the research community, both understanding trends, and setting them
  • Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
  • Fulltime
Read More
Arrow Right

Lead Machine Learning Engineer (Manager IC)

Lead Machine Learning Engineer (Manager IC) at Capital One. In Risk Tech, you wi...
Location
Location
United States , Cambridge, Massachusetts; Richmond, Virginia; McLean, Virginia
Salary
Salary:
179400.00 - 225100.00 USD / Year
capitalone.com Logo
Capital One
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree
  • At least 6 years of experience designing and building data-intensive solutions using distributed computing
  • At least 4 years of experience programming with Python, Scala, or Java
  • At least 2 years of experience building, scaling, and optimizing ML systems
Job Responsibility
Job Responsibility
  • Partner with a cross-functional team of engineers, data scientists, product managers, and designers to deliver AI-powered products
  • Design, develop, test, deploy, and support AI software components utilizing machine learning models
  • Fine-tune, develop and evaluate machine learning and foundation models
  • Collaborate as part of a cross-functional Agile team to create and enhance software that utilizes state-of-the-art AI and ML capabilities
  • Contribute thought leadership and technical vision to the long term roadmap of pioneering AI systems at Capital One
  • Leverage a broad stack of Open Source and SaaS AI technologies
  • Inform your ML infrastructure decisions using your understanding of ML modeling techniques and issues
  • Retrain, maintain, and monitor models in production
  • Construct optimized data pipelines to feed ML models
  • Ensure all code is well-managed to reduce vulnerabilities, models are well-governed, and the ML follows best practices in Responsible and Explainable AI
What we offer
What we offer
  • Performance based incentive compensation including cash bonus(es) and/or long term incentives (LTI)
  • health, financial and other benefits that support your total well-being
  • Fulltime
Read More
Arrow Right

Lead Machine Learning Engineer

Lead Machine Learning Engineer At Capital One, we are changing banking for good...
Location
Location
United States , Cambridge, Massachusetts; Richmond, Virginia; McLean, Virginia
Salary
Salary:
197300.00 - 225100.00 USD / Year
capitalone.com Logo
Capital One
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s Degree
  • At least 6 years of experience designing and building data-intensive solutions using distributed computing (Internship experience does not apply)
  • At least 4 years of experience programming with Python, Scala, or Java
  • At least 2 years of experience building, scaling, and optimizing ML systems
Job Responsibility
Job Responsibility
  • Partner with a cross-functional team of engineers, data scientists, product managers, and designers to deliver AI-powered products that change how our associates work and provide value to our customers
  • Design, develop, test, deploy, and support AI software components utilizing machine learning models, including model evaluation and experimentation, large language model inference, similarity search, guardrails, governance, observability and agentic AI
  • Fine-tune, develop and evaluate machine learning and foundation models
  • Collaborate as part of a cross-functional Agile team to create and enhance software that utilizes state-of-the-art AI and ML capabilities
  • Contribute thought leadership and technical vision to the long term roadmap of pioneering AI systems at Capital One
  • Leverage a broad stack of Open Source and SaaS AI technologies
  • Inform your ML infrastructure decisions using your understanding of ML modeling techniques and issues
  • Retrain, maintain, and monitor models in production
  • Construct optimized data pipelines to feed ML models
  • Ensure all code is well-managed to reduce vulnerabilities, models are well-governed from a risk perspective, and the ML follows best practices in Responsible and Explainable AI
What we offer
What we offer
  • Performance based incentive compensation
  • cash bonus(es) and/or long term incentives (LTI)
  • comprehensive, competitive, and inclusive set of health, financial and other benefits that support your total well-being
  • Fulltime
Read More
Arrow Right

Lead Machine Learning Engineer (Manager IC)

Lead Machine Learning Engineer (Manager IC) at Capital One. In Risk Tech, you wi...
Location
Location
United States , McLean; Richmond; Cambridge
Salary
Salary:
179400.00 - 225100.00 USD / Year
capitalone.com Logo
Capital One
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree
  • At least 6 years of experience designing and building data-intensive solutions using distributed computing (Internship experience does not apply)
  • At least 4 years of experience programming with Python, Scala, or Java
  • At least 2 years of experience building, scaling, and optimizing ML systems
Job Responsibility
Job Responsibility
  • Partner with a cross-functional team of engineers, data scientists, product managers, and designers to deliver AI-powered products that change how our associates work and provide value to our customers
  • Design, develop, test, deploy, and support AI software components utilizing machine learning models, including model evaluation and experimentation, large language model inference, similarity search, guardrails, governance, observability and agentic AI
  • Fine-tune, develop and evaluate machine learning and foundation models
  • Collaborate as part of a cross-functional Agile team to create and enhance software that utilizes state-of-the-art AI and ML capabilities
  • Contribute thought leadership and technical vision to the long term roadmap of pioneering AI systems at Capital One
  • Leverage a broad stack of Open Source and SaaS AI technologies
  • Inform your ML infrastructure decisions using your understanding of ML modeling techniques and issues
  • Retrain, maintain, and monitor models in production
  • Construct optimized data pipelines to feed ML models
  • Ensure all code is well-managed to reduce vulnerabilities, models are well-governed from a risk perspective, and the ML follows best practices in Responsible and Explainable AI
What we offer
What we offer
  • performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
  • comprehensive, competitive, and inclusive set of health, financial and other benefits
  • Fulltime
Read More
Arrow Right

Staff Machine Learning Research Scientist, LLM Evals

As a Staff Machine Learning Research Scientist on the LLM Evals team, you will l...
Location
Location
United States , San Francisco; Seattle; New York
Salary
Salary:
280000.00 - 380000.00 USD / Year
scale.com Logo
Scale
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development
  • Experience and track of recording in landing major research impacts in a fast-paced environment
  • Experience tech leading a team of research scientists and research engineers
  • Excellent written and verbal communication skills
  • Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
  • Previous experience in a customer facing role.
Job Responsibility
Job Responsibility
  • Drive research on the effectiveness and limitations of existing LLM evaluation techniques
  • Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness
  • Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects
  • Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols
  • Implement scalable and reproducible evaluation pipelines using modern ML frameworks
  • Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives
  • Mentor and guide research scientists and engineers, providing technical leadership across cross-functional projects
  • Stay deeply engaged with the ML research community, tracking emerging work and contributing to the advancement of LLM evaluation science
  • Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
  • equity based compensation
  • commuter stipend (may be eligible).
  • Fulltime
Read More
Arrow Right

Manager, Machine Learning Research Scientist, GenAI

Scale AI accelerates the development of AI systems by providing the data, infras...
Location
Location
United States , San Francisco; Seattle; New York
Salary
Salary:
239400.00 - 299300.00 USD / Year
scale.com Logo
Scale
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of hands-on research experience (PhD or equivalent preferred) in machine learning, deep learning, generative models, agent/rl systems or related domains
  • A strong track record of research excellence, including publications in top-tier ML/AI venues (NeurIPS, ICML, ICLR, ACL, etc.)
  • Experience and track of recording in landing major research impacts in a fast-paced environment
  • Experience leading or managing research teams. You’re excited to mentor, coach and develop talent
  • Excellent written and verbal communication skills. You are able to articulate research ideas and outcomes to both technical and non-technical stakeholders
Job Responsibility
Job Responsibility
  • Lead, mentor and grow a team of research scientists and engineers working on GenAI research initiatives (e.g., evaluation, post-training, agents, RL environments)
  • Define and drive a multi-year research roadmap: identify key scientific questions, set milestones, allocate resources, and ensure rigorous execution
  • Collaborate cross-functionally with engineering, product, client-facing teams and external academic or industry partners to translate research into components, insights, and actionable outcomes
  • Communicate compellingly: publish research, present at conferences, engage in open-source contributions, and represent the team externally
  • Drive an inclusive, high-performing culture: help your team through technical challenges, provide growth opportunities, and attract top talent
  • Stay deeply connected to the research community, understanding major trends, and helping set them
  • Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • a learning and development stipend
  • generous PTO
  • Fulltime
Read More
Arrow Right

Principal Machine Learning Engineer - Forecasting

We are seeking a Principal Machine Learning Engineer to join the Forecasting tea...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Degree and 12+ years of experience in machine learning engineering, software engineering, data science engineering, or a related quantitative discipline.
  • 10+ years of professional experience building, deploying, and operating production ML, AI, data, or software systems, including significant experience as a technical lead on complex, cross-functional initiatives.
  • Demonstrated track record of designing or architecting new and existing systems with emphasis on reliability, scale, security, maintainability, and operational excellence.
  • Deep hands-on experience with the full ML engineering lifecycle, including data pipelines, feature engineering, experimentation, model training, model integration, testing, deployment, monitoring, evaluation, observability, and continuous improvement.
  • Strong experience deploying forecasting, probabilistic, Bayesian, predictive, NLP, deep learning, or LLM-based systems in production environments.
  • Experience building or integrating AI systems, including LLM-powered applications, agentic workflows, retrieval or information-retrieval systems, evaluation frameworks, and human-in-the-loop review patterns.
  • Strong object-oriented programming skills in Python and SQL, with experience using modern ML and software development frameworks such as scikit-learn, PyTorch, TensorFlow/JAX, Spark, Ray, MLflow, Airflow/Prefect/Dagster, FastAPI, or equivalent technologies.
  • Experience with cloud platforms and distributed systems, including containerization, CI/CD, infrastructure-as-code, model serving, workflow orchestration, batch and streaming data processing, and production support.
  • Strong software engineering fundamentals, including system design, architecture trade-off analysis, testing strategies, code reviews, source control, build and release processes, performance optimization, and maintainability.
  • Demonstrated ability to communicate technical strategy, system tradeoffs, and delivery risks to technical and non-technical stakeholders, including senior leaders, product/program owners, scientists, and business partners.
Job Responsibility
Job Responsibility
  • Define and drive the technical strategy for enterprise forecasting and AI decision systems, aligning architecture, reusable platforms, and delivery roadmaps to Amgen's planning, supply, commercial, manufacturing, operations, and patient-focused priorities.
  • Partner with data scientists, product and program leaders, operations, commercial, manufacturing, supply chain, finance, and other business stakeholders to translate ambiguous requirements into shipped software and measurable business outcomes.
  • Architect, build, and scale production ML, LLM, and agentic AI systems that combine forecasting, predictive analytics, simulation, optimization, and autonomous or semi-autonomous workflow automation.
  • Productionize advanced statistical, Bayesian, deep learning, and machine learning models, including training, validation, inference, serving, evaluation, lifecycle management, and governed deployment.
  • Lead development of AI agent components that automate complex forecasting and operational workflows across multiple systems, decision points, datasets, and user groups while preserving appropriate human-in-the-loop review and escalation patterns.
  • Design secure integrations across enterprise APIs, databases, analytics platforms, workflow systems, cloud services, and AI orchestration patterns to enable multi-system decision support and scalable automation.
  • Establish robust MLOps and AI engineering capabilities, including model versioning, CI/CD, automated retraining, performance monitoring, observability, drift detection, service-level reliability, rollback strategies, and operational runbooks.
  • Implement guardrails, model and agent evaluation frameworks, auditability, explainability, responsible AI controls, and human-in-the-loop operating models for production AI systems in high-impact and regulated business contexts.
  • Research and evaluate state-of-the-art open-source, vendor, and internal tools related to forecasting, LLMs, AI agents, MLOps, model optimization, model serving, and scalable AI infrastructure for potential application to Amgen business problems.
  • Provide principal-level technical mentorship, design review leadership, and engineering standard-setting across teams, promoting code quality, documentation, reproducibility, testing, security, privacy, maintainability, and operational excellence.
  • Fulltime
Read More
Arrow Right

Director, Enterprise Machine Learning & Research

The Enterprise ML team works on the front lines of the AI revolution, partnering...
Location
Location
United States , San Francisco, CA; New York, NY
Salary
Salary:
289800.00 - 362250.00 USD / Year
scale.com Logo
Scale
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of hands-on research experience (PhD or equivalent preferred) in machine learning, deep learning, generative models, agent/rl systems or related domains
  • A strong track record of research excellence, including publications in top-tier ML/AI venues (NeurIPS, ICML, ICLR, ACL, etc.)
  • Experience and track of recording in landing major research impacts in a fast-paced environment
  • Experience leading or managing research teams. You’re excited to mentor, coach and develop talent.
  • Excellent written and verbal communication skills. You are able to articulate research ideas and outcomes to both technical and non-technical stakeholders.
  • Exceptional communication and stakeholder management skills, with the ability to influence executives, customers, and cross-functional partners
Job Responsibility
Job Responsibility
  • Lead, mentor and grow a team of research scientists and engineers working on GenAI research initiatives (e.g., evaluation, post-training, agents, RL environments).
  • Define and drive a multi-year research roadmap: identify key scientific questions, set milestones, allocate resources, and ensure rigorous execution.
  • Collaborate cross-functionally with engineering, product, client-facing teams and external academic or industry partners to translate research into components, insights, and actionable outcomes.
  • Communicate compellingly: publish research, present at conferences, engage in open-source contributions, and represent the team externally.
  • Drive an inclusive, high-performing culture: help your team through technical challenges, provide growth opportunities, and attract top talent.
  • Stay deeply connected to the research community, understanding major trends, and helping set them.
  • Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.
What we offer
What we offer
  • Comprehensive health, dental and vision coverage
  • retirement benefits
  • learning and development stipend
  • generous PTO
  • commuter stipend
  • Fulltime
Read More
Arrow Right