CrawlJobs Logo
Perplexity Logo Perplexity · IT - Software Development

Data Scientist, Evals

United States; United Kingdom; Serbia; Germany, San Francisco 210000.00 - 385000.00 USD / Year · Job Posted February 21, 2026
Apply Position
Job Link Share

Job Description

Perplexity serves tens of millions of users daily with reliable, high-quality answers grounded in an LLM-first search engine and our specialized data sources. We aim to use the latest models as they are released, but the intelligence frontier is a jagged one, and popular benchmarks do not effectively cover our use cases. In this role, you will build specialized evals to improve answer quality across Perplexity, covering search-based LLM answers and other scenarios popular with our users.

Job Responsibility

  • Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products, ensuring high standards for accuracy and helpfulness
  • Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on the final answer's quality
  • Develop VLM-based solutions to programmatically evaluate how final answers render visually across different platforms and devices
  • Continuously review public benchmarks and academic evaluations for their applicability to the Perplexity product, adapting and incorporating them into our regular performance measurements
  • Operate within a small, high-impact team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer Quality

Requirements

  • PhD or MS in a technical field or equivalent experience
  • 4+ years of experience in data science or machine learning
  • Strong proficiency in Python and SQL (expected to write production-grade code)
  • Experience building within a modern cloud data stack, specifically AWS and Databricks
  • Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster

Nice to have

  • 1+ years of experience working with LLMs at scale, specifically with LLM-as-a-judge setups
  • Prior experience working on customer-facing web products or consumer apps, with real user traffic at scale
  • A strong research background, with experience applying research methods to real-world ML problems
  • Experience defining evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and building ground truth datasets

What we offer

  • Offers Equity
  • Full-time U.S. employees enjoy a comprehensive benefits program including equity, health, dental, vision, retirement, fitness, commuter and dependent care accounts, and more
  • Full-time employees outside the U.S. enjoy a comprehensive benefits program tailored to their region of residence
Perplexity - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Data Scientist, Evals

8 matching positions

Senior Data Scientist AI Tooling

Fin is the AI Customer Agent company on a mission to help businesses provide per...
Location
Location
Ireland , Dublin
Salary
Salary:
Not provided
intercom.com Logo
Intercom
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven track record of applied data science with measurable GTM impact - you've shipped models or tools that moved metrics like conversion, cycle time, or retention
  • LLM/ML application experience - familiarity with RAG, prompt and tool design, vector search, evals and have leveraged AI for development
  • Excellent SQL skills and fluency in Python or R, with experience applying analytical and statistical methods to business problems
  • Experience with orchestration tools (e.g., DBT, Airflow) for deploying reliable data workflows
  • Strong communication and empathy - ability to translate complex data concepts for non-technical stakeholders
  • Collaborative product mindset - comfort working closely with Sales and Success teams to turn ambiguity into clear deliverables
Job Responsibility
Job Responsibility
  • Design, evaluate, and ship AI-powered internal tools for GTM use cases - including account research & summaries, next-best-action recommendations, renewal propensity, pipeline risk detection, QBR/autobrief generation, and post-call summarization & follow-ups
  • Work end-to-end: Own the full lifecycle, from problem definition and data modeling to building production-ready tools, including writing Python backends and React frontends
  • Prototype fast, ship to learn: Rapidly build with users, then productionize quickly to iterate and deliver impact
  • Instrument for adoption and outcomes: Define success through real usage and measurable business impact (e.g., improved win rate, conversion, expansion)
  • Evangelize and enable: Document playbooks, run enablement sessions, and help leaders operationalize new tooling across teams
What we offer
What we offer
  • Competitive salary and equity in a fast-growing start-up
  • Unlimited access to Claude Code and best-in-class AI tools
  • weekday lunch plus snack foods and fully stocked kitchen
  • Regular compensation reviews
  • Life assurance
  • Comprehensive health and dental insurance for you and your dependents
  • Open vacation policy and flexible holidays
  • Paid maternity leave
  • 6 weeks paternity leave
  • MacBooks or other equipment
  • Fulltime
Read More
Arrow Right

Senior Data Scientist AI Tooling

Fin is the AI Customer Agent company on a mission to help businesses provide per...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
intercom.com Logo
Intercom
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven track record of applied data science with measurable GTM impact
  • LLM/ML application experience - familiarity with RAG, prompt and tool design, vector search, evals and have leveraged AI for development
  • Excellent SQL skills and fluency in Python or R, with experience applying analytical and statistical methods to business problems
  • Experience with orchestration tools (e.g., DBT, Airflow) for deploying reliable data workflows
  • Strong communication and empathy - ability to translate complex data concepts for non-technical stakeholders
  • Collaborative product mindset - comfort working closely with Sales and Success teams to turn ambiguity into clear deliverables.
Job Responsibility
Job Responsibility
  • Design, evaluate, and ship AI-powered internal tools for GTM use cases - including account research & summaries, next-best-action recommendations, renewal propensity, pipeline risk detection, QBR/autobrief generation, and post-call summarization & follow-ups
  • Work end-to-end: Own the full lifecycle, from problem definition and data modeling to building production-ready tools, including writing Python backends and React frontends
  • Prototype fast, ship to learn: Rapidly build with users, then productionize quickly to iterate and deliver impact
  • Instrument for adoption and outcomes: Define success through real usage and measurable business impact (e.g., improved win rate, conversion, expansion)
  • Evangelize and enable: Document playbooks, run enablement sessions, and help leaders operationalize new tooling across teams.
What we offer
What we offer
  • Competitive salary and equity in a fast-growing start-up
  • We serve lunch every weekday, plus a variety of snack foods and a fully stocked kitchen
  • Regular compensation reviews - we reward great work
  • Peace of mind with life assurance, as well as comprehensive health and dental insurance for you and your dependents
  • Open vacation policy and flexible holidays so you can take time off when you need it
  • Paid maternity leave, as well as 6 weeks paternity leave for fathers, to let you spend valuable time with your loved ones
  • MacBooks are our standard, but we’re happy to get you whatever equipment helps you get your job done
  • Fulltime
Read More
Arrow Right

AI Data Scientist

You'll be Gamma's first Data Scientist, turning massive volumes of user data and...
Location
Location
United States , San Francisco
Salary
Salary:
180000.00 - 310000.00 USD / Year
gamma.app Logo
Gamma
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience as a data scientist at product-focused tech companies, with experience managing or mentoring other data scientists
  • Strong statistical foundations with hands-on experience designing and analyzing A/B tests and experiments at scale
  • Experience working with large-scale data and building metrics frameworks from scratch, including modern data stack tools like dbt and Snowflake and comfort analyzing unstructured or text data
  • Experience working with AI/ML products, especially LLMs or generative AI, with familiarity evaluating model performance in production settings
  • Ability to communicate complex technical concepts to non-technical stakeholders and influence product decisions with clarity and conviction
  • A clear perspective on how agentic coding is transforming the data science role, and genuine excitement about applying AI to your own work (Nice to have)
Job Responsibility
Job Responsibility
  • Build frameworks that help the team understand AI model performance, user behavior, and product health across Gamma's platform
  • Dig into AI model outputs across user cohorts to identify quality gaps and create evals and metrics to measure improvement
  • Partner with engineering and product to define quality metrics for AI-generated content and user satisfaction
  • Develop statistical models and frameworks that empower product teams to make data-informed decisions independently
  • Design and analyze large-scale A/B tests and experiments with statistical rigor to measure product impact and guide prioritization
What we offer
What we offer
  • equity
  • Fulltime
Read More
Arrow Right

Senior Data Scientist

M365 Copilot Cadets (Customer & Analytics‑Driven Eval Team) turns real‑world cus...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3+ years data-science experience
  • OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience
  • OR equivalent experience
  • Experience with building data pipelines, performing large-scale analysis, and implementing ML workflows using Python and SQL
  • Experience in developing models or designing evaluation frameworks, including A/B testing or prompt-based assessments for LLMs
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Evaluation & Feedback Analysis: Convert multi‑source feedback (dogfood, VIP customers, production traces) into a prioritized dataset of 10–100 tasks per scenario, each with prompts and golden outputs
  • maintain a living failure taxonomy prioritized by volume × impact × fixability
  • Rubrics & LLM‑as‑Judge: Author crisp, binary‑first rubrics across 7–30 dimensions (e.g., correctness/completeness, refusal calibration, tool‑use quality, formatting/contract, persona/tone, trace hygiene)
  • Build grader prompts (with few‑shots and counter‑examples) that achieve ≥80% human‑match rate, track TPR/TNR on held‑out sets, and prevent reward hacking
  • Synthetic & Human‑Labeled Data: Design structured tuples to scale high‑signal synthetic data
  • orchestrate vendor/partner annotation sprints and live calibrations to align shared judgment
  • Ensure datasets are reproducible with linked artifacts and robust metadata/trace hygiene
  • Customer‑Grounded Scenarios: Partner with PMs/solution architects to co‑develop evals with VIP customers so tasks reflect real outcomes and workflows
  • quantify lift from fixes and inform the next hill‑climb
  • Team Leadership & Ways of Working: Co‑own the Cadets “feedback flywheel” with PM/Eng (instrumentation, taxonomy, guardrails vs. evaluators) and help operationalize weekly checklists, change logs, and judge refresh cadence
  • Fulltime
Read More
Arrow Right

Member of Technical Staff - Data Scientist

We’re looking for data scientists to help build the next generation of post-trai...
Location
Location
Switzerland , Zürich
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands‑on experience with large language models, including training or applying them in production (not just prompting)
  • Designing and running post‑training experiments (evals, ablations, preference tuning / RLHF‑style methods)
  • Building and owning scalable data pipelines for training and evaluation data
  • Strong Python skills for ML experimentation, data processing, and analysis
  • Solid statistical, experimental, and general engineering fundamentals
Job Responsibility
Job Responsibility
  • Design evaluations of advanced model capabilities and use them to drive rapid, high-signal iteration loops
  • Work with vendors to produce high quality evaluation and training data
  • Build data pipelines to produce high quality evaluation and training data
  • Build data flywheels to hill-climb on model weaknesses, using data from various surfaces where our models are deployed
  • Ensure optimal quality, quantity and coverage of data across our post-training stages
  • Run post-training experiments and ablations to produce models that climb our evals
  • Embody our culture and values
  • Fulltime
Read More
Arrow Right

Data Scientist, Platform and B2B Products

As a Data Scientist on the Platform team, you will drive a data-driven culture f...
Location
Location
United States , San Francisco
Salary
Salary:
230000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years in a quantitative role in ambiguous, high-growth environments (platforms, APIs, or B2B products a plus)
  • Depth in SQL and Python, with a track record proposing, designing, and running rigorous experiments
  • Experience defining and operationalizing metrics from scratch (including reliability/latency/cost and safety)
  • Strong cross-functional communication with PMs, engineers, and executives
  • Strategic instincts beyond p-values—clear thinking about tradeoffs and business impact
Job Responsibility
Job Responsibility
  • Embed with the Platform product team as a trusted partner, uncovering ways to improve developer experience, reliability, and usage growth
  • Define north-star metrics across the developer funnel (activation, retention, growth), as well as latency/cost guardrails for new features and models
  • Design and interpret A/B tests and controlled rollouts (e.g., new model versions, pricing/limits, new API features, new B2B products)
  • Build source-of-truth dashboards and self-serve data tools for product, engineering, and go-to-market teams
  • Translate product learnings into actionable feedback for Research (e.g., failure modes, eval gaps, model response quality)
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick and safe time (1 hour per 30 hours worked)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right

Data Scientist

As a Data Scientist on Codex, you will measure and accelerate product-market fit...
Location
Location
United States , San Francisco
Salary
Salary:
230000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years in a quantitative role at a developer-facing or high-growth product
  • Fluency in SQL and Python
  • comfort with experiment design and causal inference
  • Experience defining product metrics tied to user value
  • Ability to communicate clearly with PM, Eng, and Design—and to influence product direction
Job Responsibility
Job Responsibility
  • Embed with the Codex product team to discover opportunities that improve developer outcomes and growth
  • Design and interpret A/B tests and staged rollouts of new coding models and product features
  • Define and operationalize metrics such as suggestion acceptance, edit distance, compile/test pass rates, task completion, latency, and session productivity
  • Build dashboards and analyses that help the team self-serve answers to product questions (by language, framework, repo size, task type)
  • Diagnose failure modes and partner with Research on targeted improvements (model quality signals, user feedback, evals)
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick and safe time (1 hour per 30 hours worked)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right

Lead Data Scientist - Recommendations

Scribd’s Data & Analytics team is hiring a Lead Data Scientist to own measurable...
Location
Location
United States; Canada; Mexico , Atlanta; Austin; Boston; Dallas; Denver; Chicago; Houston; Jacksonville; Los Angeles; Miami; New York City; Phoenix; Portland; Sacramento; Salt Lake City; San Diego; San Francisco; Seattle; Washington D.C.; Ottawa; Toronto; Vancouver; Mexico City
Salary
Salary:
133000.00 - 252500.00 USD / Year
scribd.com Logo
Scribd
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years experience in Data Science, preferably on recs/search/ranking with shipped impact
  • Strong Python and SQL
  • comfort with Spark
  • Fluency in ranking evaluation (NDCG@K, MAP, MRR, calibration, coverage/diversity) and awareness of exposure/selection bias
  • Fluency in experiment design and connecting offline metrics to online outcomes
  • Ability to translate product goals into loss functions, features, and specs engineers can build
Job Responsibility
Job Responsibility
  • Opportunity mapping. Size and prioritize new recs surfaces, intents, and cohorts
  • trace the funnel and analyze by slice (cold items, long-tail users, platform) to steer the roadmap
  • Own the evaluation framework. Define north star & guardrails (e.g. diversity, novelty, duplication, safety)
  • set threshold and tradeoffs, and publish the Objective & Eval Contract per surface
  • Offline/Online alignment. Quantify correlation between offline IR metrics (e.g., NDCG@K, MAP, MRR, coverage, calibration) and online KPIs by surface/cohort
  • publish error bounds and monitor metric drift
  • Create leading indicators. Create short-horizon metrics that predict long-term outcomes (e.g., trial to bill-through)
  • backtest and run post-hoc causal checks, reporting uncertainty
  • Build the measurement architecture. Set identity & attribution standards (user_id vs. device_id, qualifying events, windows) so downstream metrics (bill-through, churn) are trustworthy
  • Design and run advanced experiments such as interleaving tests, pre-register stop/go criteria, and deliver crisp readouts that drive decisions
What we offer
What we offer
  • Healthcare Insurance Coverage (Medical/Dental/Vision): 100% paid for employees
  • 12 weeks paid parental leave
  • Short-term/long-term disability plans
  • 401k/RSP matching
  • Onboarding stipend for home office peripherals + accessories
  • Learning & Development allowance
  • Learning & Development programs
  • Quarterly stipend for Wellness, WiFi, etc.
  • Mental Health support & resources
  • Free subscription to the Scribd Inc. suite of products
  • Fulltime
Read More
Arrow Right