CrawlJobs Logo

Research Scientist, Human AI Interaction

joinhandshake.com Logo

Handshake

Location Icon

Location:
United States , San Francisco, CA, New York, NY

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

350000.00 - 420000.00 USD / Year

Job Description:

As a Research Scientist, Human–AI Interaction, you will play a pivotal role in defining how AI systems support real human work by leading research at the intersection of Human–Computer Interaction (HCI), Large Language Models (LLMs), and task-level benchmarking. You will operate at the frontier of human-centered AI evaluation, with a focus on understanding what people actually do to accomplish meaningful work—and how AI systems change, accelerate, or reshape that activity. Your research will define jobs-to-be-done benchmarks, comparative evaluation frameworks, and empirical methods for measuring human effort, time, quality, and outcomes when working with AI copilots. Additionally, the Handshake AI platform is an interface used by thousands of the top subject matter experts in the world to evaluate AI systems, and offers numerous interesting HCI / HITL-AI research questions that will drive large business impact. You’ll set research direction, establish standards for measuring human activity in AI-mediated workflows, publish papers and open-source code, and lead the development of rigorous, scalable benchmarks that connect human work, AI assistance, and real economic value.

Job Responsibility:

  • Lead high-impact research on jobs-to-be-done benchmarks for AI systems, including: Defining task taxonomies grounded in real professional and economic activities
  • Identifying what constitutes meaningful task completion, quality, and success
  • Translating qualitative work understanding into measurable, repeatable benchmarks
  • Develop methods to measure human activity in AI-mediated workflows
  • Design benchmarks to assess AI-as-a-collaborator/copilot, rather than autonomous agents / basic Q&A
  • Design and run empirical studies of how people use AI to solve tasks, including: Controlled experiments and field studies measuring task performance
  • Instrumentation for capturing fine-grained interaction traces and outcomes
  • Drive strategy for professional-domain AI benchmarks, focusing on: Understanding domain-specific workflows (e.g., analysis, writing, planning, coordination)
  • Grounding benchmark design in how work is actually performed, not idealized tasks
  • Build and prototype AI systems and evaluation infrastructure to support research and Data production, including: LLM-powered copilots and experimental tools used for task-level measurement
  • Benchmark harnesses that evaluate both model behavior and human outcomes
  • Data pipelines for analyzing human–AI interaction at scale
  • The human-in-the-loop experience for Handshake fellows to produce effective evaluations and training data for frontier models, through structured UI/UX interactions with these models
  • Collaborate closely with User Experience Research (UXR) to: Leverage deep qualitative insights into real user behavior and workflows
  • Translate ethnographic and observational findings into formal research constructs
  • Publish and present research that advances the field of human-centered AI benchmarking, with an expectation of regular contributions to top-tier venues such as CHI (Conference on Human Factors in Computing Systems), and related HCI and AI conferences

Requirements:

  • PhD or equivalent experience in Human–Computer Interaction, Computer Science, Cognitive Science, or a related field, with a strong emphasis on empirical evaluation of interactive AI/LLM systems
  • 3+ years of academic or industry research experience post-PhD, including leadership on complex research initiatives and analyzing data from a real AI product
  • Strong publication record, with demonstrated impact in top-tier AI (NeurIPS, ICML, ICLR, ACL) and HCI (CHI) venues
  • Deep expertise in experimental design and measurement, particularly for: Task performance and human activity
  • Comparative evaluation frameworks
  • Mixed-methods research grounded in real-world behavior
  • Strong technical and coding skills, including: Python and data analysis / ML tooling
  • Experience building experimental systems and benchmark infrastructure
  • Familiarity working with LLM APIs, agent frameworks, or AI-assisted tooling
  • Proven ability to define and lead research agendas that connect human work, AI capability, and business or economic impact
  • Strong collaboration skills, especially working across research, engineering, product, and UXR teams

Nice to have:

  • Experience developing benchmarks or evaluation frameworks for human–AI systems or AI-assisted productivity tools
  • Prior work on copilot-style systems, agentic workflows, or automation of professional tasks
  • Familiarity with workplace studies, CSCW, or socio-technical systems research
  • Contributions to open-source tools, datasets, or benchmarks related to task-level evaluation
  • Interest in how AI reshapes labor, productivity, and the future of work
What we offer:
  • Equity in a fast-growing company
  • 401(k) match, competitive compensation, financial coaching
  • Paid parental leave, fertility benefits, parental coaching
  • Medical, dental, and vision, mental health support, $500 wellness stipend
  • $2,000 learning stipend, ongoing development
  • Internet, commuting, and free lunch/gym in our SF office
  • Flexible PTO, 15 holidays + 2 flex days, winter #ShakeBreak where our whole office closes for a week
  • Team outings & referral bonuses

Additional Information:

Job Posted:
February 20, 2026

Employment Type:
Fulltime
Work Type:
On-site work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Research Scientist, Human AI Interaction

AI Research Scientist, Robotics

The ideal Research Scientist candidate will use their skills in system design an...
Location
Location
United States , Redmond
Salary
Salary:
154000.00 - 217000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • Currently has or is in the process of obtaining a PhD degree in the field of Artificial Intelligence, Robotics, Computer Vision, Machine Learning, Language, a related field, or equivalent practical experience
  • Experience with any of the following research areas: robotics, motion planning, embodied AI, human-robot interaction, sim-to-real transfer, learning from demonstration, reinforcement learning, dexterous manipulation, digital agents, vision language models, computer vision, egocentric perception, and/or LLMs
  • Experience in relevant robotics related research areas, such as: VLM, robot learning, reinforcement learning, imitation learning, action-conditioned world models, task and motion planning, sim-to-real transfer robotic control, manipulation, navigation, or generally embodied AI
Job Responsibility
Job Responsibility
  • Perform fundamental and applied research to push the scientific and technological frontiers of embodied artificial intelligence
  • Invent/improve novel data-driven paradigms for robotics, leveraging a variety of modalities (images, video, text, audio, tactile, etc)
  • Investigate paradigms that can deliver a spectrum of embodied behaviors - from simulated characters to real robots, and from short-horizon, low-level to long-horizon, high-level intelligence
  • Develop algorithms based on state-of-the-art machine learning and neural network methodologies
  • Define, build and benchmark new functionalities needed for the next generation of AI
  • Conduct research towards long-term product goals while identifying intermediate milestones
  • Plan and execute novel research based on long-term objectives of the organization
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

AI Research Scientist, Robotics

At Meta, we’re building the future of human connection and the technology that e...
Location
Location
United States , Redmond
Salary
Salary:
219000.00 - 301000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • PhD degree in the field of Artificial Intelligence, Robotics, Computer Vision, Machine Learning, Language, a related field, or equivalent practical experience
  • Experience with any of the following research areas: robotics, motion planning, embodied AI, human-robot interaction, sim-to-real transfer, learning from demonstration, reinforcement learning, dexterous manipulation, digital agents, vision language models, computer vision, egocentric perception, and/or Large Language Models
  • 5+ years of industry experience in relevant robotics related research areas, such as: Vision Language Models robot learning, reinforcement learning, imitation learning, action-conditioned world models, task and motion planning, sim-to-real transfer robotic control, manipulation, navigation, or generally embodied AI
Job Responsibility
Job Responsibility
  • Perform fundamental and applied research to push the scientific and technological frontiers of embodied artificial intelligence
  • Invent/improve novel data-driven paradigms for robotics, leveraging a variety of modalities (images, video, text, audio, tactile, etc.)
  • Investigate paradigms that can deliver a spectrum of embodied behaviors - from simulated characters to real robots, and from short-horizon, low-level to long-horizon, high-level intelligence
  • Develop algorithms based on state-of-the-art machine learning and neural network methodologies
  • Define, build and benchmark new functionality needed for the next generation of AI
  • Conduct research towards long-term product goals while identifying intermediate milestones
  • Lead, plan, and execute novel research based on long-term objectives of the organization
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

Director, Molecule Design Products

The Onyx Research Data Tech organization represents a major investment by GSK R&...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
us.gsk.com Logo
GSK
Expiration Date
February 26, 2026
Flip Icon
Requirements
Requirements
  • Bachelors degree in a technical or scientific field, with a focus on computational science, biomedical sciences, AI, data science, software engineering, or a related discipline
  • Significant experience in product management, with a substantial amount in a leadership role and a proven track record of driving significant product modernization initiatives
  • Proven track record of defining and executing a product strategy for a portfolio of 0-to-1 products or product lines, specifically in the life science domain, including the strategic transformation of existing products
  • Demonstrated expertise in Generative AI, LLMs, and autonomous AI agents, including a deep understanding of their application in complex technical or scientific problem spaces
  • Deep technical fluency with cloud-native architectures (e.g., AWS, GCP, Azure), API design, and the infrastructure required to serve and scale advanced AI/ML applications
Job Responsibility
Job Responsibility
  • Portfolio Strategy & Vision: Define and champion the overarching product strategy, vision, and roadmap for the entire Molecule Design product portfolio, ensuring alignment with Onyx's and GSK R&D's strategic objectives and long-term scientific breakthroughs
  • Overseeing the timely delivery of product features and solutions across the portfolio to maximize scientific impact and business outcomes
  • Team Leadership & Development: Build, mentor, and lead a high-performing team of Product Managers and Senior Product Managers, fostering a culture of innovation, user-centricity, accountability, and continuous professional growth
  • AI/GenAI Portfolio Leadership & Modernization: Drive the strategic direction for leveraging Generative AI, LLMs, and autonomous AI Agents across the entire Molecule Design portfolio, identifying breakthrough opportunities to automate and enhance complex scientific research tasks
  • Set the vision and execute a comprehensive product modernization strategy, transforming existing legacy tools and solutions to be AI-ready, while meticulously ensuring business continuity and minimal disruption to ongoing scientific research
  • Oversee the strategy and development of foundational capabilities (e.g., model-ready molecule design data products, Model Context Protocol implementations) that underpin and unify our AI-powered products
  • Establish strategic guidelines for data acquisition, model fine-tuning, and the responsible deployment of proprietary Generative AI models across the portfolio
  • Human-AI Interaction & User Experience: Champion the strategic approach to human-AI collaboration and interaction design for the portfolio, ensuring that agentic systems provide intuitive, powerful, and ethical experiences for scientists
  • Executive & Cross-Functional Influence: Foster deep strategic partnerships with executive-level product, engineering, and scientific leadership. Influence key stakeholders across R&D, Tech, and partner functions to align on product vision, secure resources, and drive adoption
  • Strategic Portfolio Growth & Capability Sourcing: Identify, evaluate, and champion significant strategic market opportunities and external partnerships. Lead critical 'buy vs. build' decisions for core capabilities across the Molecule Design product portfolio, ensuring the most effective and efficient path to accelerate growth, drive impact, and optimize resource allocation
What we offer
What we offer
  • Competitive base salary
  • Annual bonus based on company performance
  • Flexible working options available for most roles
  • Learning and career development
  • Access to healthcare & wellbeing programmes
  • Employee recognition programmes
  • Fulltime
!
Read More
Arrow Right

Senior Applied Scientist

Microsoft is a company where innovators come to collaborate, envision what can b...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics predictive analytics, research)
  • OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research)
  • OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research)
  • OR equivalent experience
  • 4+ years of experience in statistics, predictive analytics and research
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Collaborate with AI researchers and audio signal processing experts to design and build end-to-end ML systems, tuned for human-human and human-AI interactions
  • Design and develop ML pipelines involving data cleaning, feature engineering, model training, and evaluation
  • Work across the product lifecycle from prototyping to shipping production-grade code optimized for performance and memory and updating the deployed models based on A/B testing
  • Remain up to date with the latest advancements, trends and research and contribute towards our IP portfolio
  • Research and develop synthetic data generation strategies
  • Proactively follow state of the art research and share latest work, write papers, attend conferences and share knowledge in the wider team
  • Fulltime
Read More
Arrow Right

Senior Applied Scientist

We are developing the Intelligent Conversation and Communications Cloud (IC3) to...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics predictive analytics, research)
  • OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research)
  • OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research)
  • OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Collaborating with AI researchers and audio signal processing experts to design and build end-to-end ML systems, tuned for human-human and human-AI interactions
  • Design and develop ML pipelines involving data cleaning, feature engineering, model training, and evaluation
  • Work across the product lifecycle from prototyping to shipping production-grade code optimized for performance and memory and updating the deployed models based on A/B testing
  • Remain up to date with latest advancements, trends and research and contribute towards our IP portfolio
  • Fulltime
Read More
Arrow Right

Principal Research Scientist

We are hiring a Principal Research Scientist to join the Wikimedia Foundation’s ...
Location
Location
United States
Salary
Salary:
164924.00 - 251398.00 USD / Year
wikimediafoundation.org Logo
Wikimedia Foundation
Expiration Date
March 08, 2026
Flip Icon
Requirements
Requirements
  • A track record of scholarly publications and service to the research and scientific communities including but not limited to human-computer interaction and computational social science communities
  • 3 or more years of strategic technical leadership in large matrixed organizations or their equivalent in academia, with a proven ability to counsel senior leadership
  • Ability to distill scientific nuance into high-impact, actionable clarity for non-technical stakeholders
  • Ability to manage a complex portfolio and mentor contributors while pivoting between high-level strategy and technical details
  • Advanced proficiency in AI, NLP or ML frameworks, with experience auditing algorithms for bias and accuracy OR mastery of mixed-method research specifically applied to online community governance and human system interaction
  • Proven ability to perform high velocity audits of research to identify methodological flaws, Movement and organizational opportunities and risks, and insights
  • PhD degree and a minimum of five years of work experience in a related field
Job Responsibility
Job Responsibility
  • Maintaining a fluent and real-time command of the global research landscape to inform the conversations on Wikipedia quality and integrity
  • Leading a strategic research portfolio on knowledge integrity
  • Mentoring individual contributors to execute high-impact scientific projects
  • Translating complex research findings into actionable strategy and recommendations for the Head of Research, Legal, and Communications teams
  • Delivering rapid, authoritative technical vetting of external research to identify organizational risks and scientific opportunities under tight deadlines
  • Driving research advocacy and public engagement in knowledge integrity research. domain, representing the Foundation’s scientific perspective to global stakeholders and the public
  • Cultivating strategic relationships within the scientific community and standards organizations, translating Wikimedia research findings into industry-wide best practices
  • Fulltime
Read More
Arrow Right
New

Applied Data Scientist

As an Applied Data Scientist on our Insights team, you will help pioneer the nex...
Location
Location
United States
Salary
Salary:
150000.00 - 200000.00 USD / Year
cresta.com Logo
Cresta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years building and shipping models for real-world business applications, ideally in NLP and LLM-based systems
  • Strong proficiency in Python and standard ML / data tooling (e.g., SQL, data pipelines, experiment frameworks)
  • World-class first principles thinking and ML intuition
  • Ability to turn ambiguous product asks into crisp problem statements, eval specs, metrics, and hypotheses
  • Experience working directly with customers or internal stakeholders to understand constraints, explain tradeoffs, and iterate on solutions
  • Comfort working with design-partner style engagements where requirements evolve rapidly and you’re expected to co-create the solution
  • Track record of building evaluation suites that go beyond single scalar metrics to capture reliability, safety, and qualitative user experience
  • Strong written and verbal communication skills
  • able to clearly explain complex technical work to both engineers and non-technical partners
Job Responsibility
Job Responsibility
  • Co-develop new capabilities with a small number of high-impact enterprise customers along with our product, engineering, and design teams
  • using their real workflows and constraints as your testbed
  • Communicate effectively across all levels of the organization
  • Plan and run short, focused design-partner engagements (days to weeks) where you ship early versions, collect structured feedback, and iterate quickly
  • Generalize learnings from each design partner into reusable, productized capabilities rather than one-off bespoke models
  • Partner with domain experts to curate high-quality eval guidelines and datasets for domains such as CSAT prediction and outcome prediction (across both human<>human and human<>AI interactions)
  • Use the best tools + models for the job (simple and interpretable where it matters, sophisticated where it can drive outsized value
  • Write clear specs and experiment reports that make tradeoffs and assumptions explicit
  • Stay close to the research frontier in ML/AI, LLMs, and evals, translating promising ideas into pragmatic, shippable improvements
  • Where applicable, help translate your solutions into publications, whitepapers, technical blogs, etc.
What we offer
What we offer
  • Comprehensive medical, dental, and vision coverage with plans to fit you and your family
  • Flexible PTO to take the time you need, when you need it
  • Paid parental leave for all new parents welcoming a new child
  • Retirement savings plan to help you plan for the future
  • Remote work setup budget to help you create a productive home office
  • Monthly wellness and communication stipend to keep you connected and balanced
  • In-office meal program and commuter benefits provided for onsite employees
  • Offers Equity
  • Fulltime
Read More
Arrow Right
New

Senior AI Engineer (Agents)

We are looking for an experienced and exceptional Senior AI Engineer (Agents) to...
Location
Location
Singapore
Salary
Salary:
Not provided
workato.com Logo
Workato
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, or a related field, or equivalent practical experience
  • 5+ years in backend software development using modern programming languages (e.g., Python (strongly preferred!), Golang or Java)
  • Demonstrated experience building production AI systems including chatbots, virtual assistants, and automated support agents using LLMs (OpenAI, Anthropic, open-source models)
  • Expertise in natural language understanding (NLU) and intent classification for customer query interpretation, entity extraction, and conversation flow management
  • Expertise in building knowledge bases and FAQ systems with dynamic content retrieval and self-learning capabilities from support interactions
  • Experience implementing multi-channel support automation across chat, email, voice, and messaging platforms with consistent context handling
  • Deep knowledge of REST API design and integration patterns
  • Experience working with PostgreSQL and ClickHouse, or similar relational and analytical databases
  • Strong understanding of software architecture, scalability, security, and system design
Job Responsibility
Job Responsibility
  • Design and implement advanced AI/ML systems with a focus on LLMs, AI Agents, and retrieval-augmented generation (RAG) architectures
  • Build conversational AI interfaces that handle multi-turn customer interactions, maintain context across sessions, and seamlessly escalate to human agents when necessary
  • Build production-grade AI pipelines for data processing, model training, fine-tuning, and serving at scale
  • Implement feedback loops and continuous learning systems that incorporate customer satisfaction metrics, agent corrections, and conversation outcomes to improve model performance over time
  • Create analytics dashboards and reporting tools to track automation effectiveness, identify common customer pain points, and measure key performance indicators like resolution time, containment rate, and customer satisfaction scores
  • Lead technical initiatives for AI system integration into existing products and services
  • Collaborate with data scientists and ML researchers to implement and productionize new AI approaches and models
Read More
Arrow Right