CrawlJobs Logo

Senior Research Engineer, LLM Evaluation and Behavioral Analysis

United States, San Francisco 220000.00 - 270000.00 USD / Year · Job Posted February 18, 2026
Apply Position
Job Link Share

Job Description

Together AI is building the fastest, most capable open-source-aligned LLMs and inference stack in the world. As part of the Turbo organization, you will be a critical bridge between cutting-edge model research and real-world behavioral reliability. This role focuses on deeply understanding model behavior — probing reasoning, tool use, function calling, multi-step interactions, and subtle failure modes — and building the evaluation systems that ensure models behave intelligently and consistently in production. You will develop robust evaluation pipelines, design high-quality behavioral test suites, and work closely with training, post-training, inference, and product teams to identify regressions, shape datasets, and influence model improvements. Your work will directly define how Together measures model quality and reliability across releases.

Job Responsibility

  • Build and iterate on evaluation frameworks that measure model performance across instruction following, function calling, long-context reasoning, multi-turn dialog, safety, and agentic behaviors
  • Develop specialized evaluation suites for: Function calling — argument correctness, schema adherence, tool selection, multi-function planning, and error recovery
  • Agentic workflows — task decomposition, multi-step planning, self-correction, and autonomous tool-use sequences
  • Tool-augmented interactions — search, retrieval, code execution, API-driven actions
  • Create CI/CD automated pipelines for A/B comparisons, regression detection, behavioral drift monitoring, and adversarial probing
  • Design and curate high-quality evaluation datasets, especially nuanced or challenging cases across domains
  • Collaborate with researchers and engineers to diagnose failures, triage regressions, and guide data selection, shaping strategies, objective design, and system improvements
  • Work with engineering teams to build dashboards, reports, and internal tools that help visualize behavior changes across releases
  • Operate in a fast-paced, high-impact environment with deep technical ownership and close partnership with world-class model researchers and infra engineers

Requirements

  • Strong engineering skills with Python, evaluation tooling, and distributed workflows
  • Experience working with LLMs or transformer-based models, particularly in model evaluation, testing, or red-teaming
  • Ability to reason clearly about qualitative behavior, edge cases, and model failure patterns
  • Experience designing experiments, building datasets, and interpreting noisy behavioral signals
  • Understanding of function calling and structured output formats
  • Familiarity with GPU or distributed compute environments
  • Hands-on experience evaluating function-calling models, agentic systems, or tool-augmented LLM pipelines
  • Experience with multi-turn or multi-step reasoning tasks
  • Familiarity with inference systems, distributed infrastructure, or post-training workflows
  • Passion for discovering subtle behaviors, surprising model gaps, or edge-case failures

What we offer

  • competitive compensation
  • startup equity
  • health insurance
  • other benefits

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Senior Research Engineer, LLM Evaluation and Behavioral Analysis

8 matching positions

Senior AI Engineer, Banking Technology – Vice President

Citi’s Banking Technology organization is seeking a highly hands-on Senior AI En...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6–10 years of progressive software engineering experience with a strong hands‑on coding background
  • 2+ years focused on AI software development, LLM‑based solutions, agentic systems, and/or machine learning
  • Demonstrated experience delivering enterprise‑scale AI capabilities into production
  • Strong proficiency in Python and/or Java (Spring Boot)
  • working knowledge of JavaScript/TypeScript (Angular, Node.js). Full‑stack experience is a plus
  • Solid understanding of core AI concepts such as knowledge representation, planning, and multi‑agent systems
  • Hands‑on experience with LLMs, RAG, prompt engineering, MCPs, and agent frameworks (e.g., Google ADK)
  • Practical experience with ML frameworks and libraries (TensorFlow, PyTorch, Scikit‑learn, NumPy, Pandas)
  • Familiarity with AI coding tools such as Devin, GitHub Copilot, and Antigravity
  • Strong grounding in modern engineering practices including Git, CI/CD, testing, code reviews, agile delivery, application resiliency, and security
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
  • Design, implement, and deploy scalable agentic AI frameworks and generative AI solutions for critical business use cases, ensuring robustness, performance, security, and reliability
  • Develop and integrate agentic AI systems leveraging multiple model providers and platforms (e.g., OpenAI, Anthropic, Google APIs)
  • Fulltime
Read More
Arrow Right

Senior Applied Scientist and Principal Applied Scientist

The Core AI Speech Group brings together talents in the areas of signal processi...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics predictive analytics, research) OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research) OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Developing novel speech algorithms to advance state-of-the-art speech technologies for real world user scenarios, especially in integrating speech with LLM for multimodal modeling
  • Helps address scalability problems by adjusting to stakeholder needs
  • Works with large-scale computing frameworks, data analysis systems, and modeling environments to improve models
  • Applies the model to real products, and then verifies effects through iterations
  • Experiments by putting multiple models in production and evaluating their performance. Continues to monitor how algorithm performs against expected behaviors and performance or accuracy guardrails
  • Fulltime
Read More
Arrow Right

Senior Applied AI Engineer

As an Senior Applied AI Engineer for CXA, you will play a pivotal role in advanc...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Statistics, Electrical/Computer Engineering, Physics, Mathematics or related field, OR Master’s degree OR PHD AND 1+ years of experience working with machine learning libraries to solve real world AI/ML problems
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Microsoft Cloud Background Check
  • Strong 7+ software engineering skills, including hands‑on development experience in C# and Python for building scalable, high‑performance, and production‑ready systems
  • Experience in working with Generative AI models and ML stacks
  • Experience across the product lifecycle from ideation to shipping
Job Responsibility
Job Responsibility
  • Build collaborative relationships with product and business groups to deliver AI-driven impact
  • Research and implement state-of-the-art using foundation models, prompt engineering, RAG, graphs, multi-agent architectures, as well as classical machine learning techniques
  • Fine-tune foundation models using domain-specific datasets
  • Evaluate model behavior on relevance, bias, hallucination, and response quality via offline evaluations, shadow experiments, online experiments, and ROI analysis
  • Apply strong software engineering skills in languages such as C# and Python to design, develop, and optimize scalable, reliable, and maintainable AI‑driven systems
  • Develop LLM prompts, agents, and query execution workflows, often with tight latency constraints
  • Build rapid AI solution prototypes, contribute to production deployment of these solutions, debug production code, support MLOps/AIOps
  • Contribute to papers, patents, and conference presentations
  • Translate research into production-ready solutions and measure their impact through A/B testing and telemetry that address customer needs
  • Ability to use data to identify gaps in AI quality, uncover insights and implement PoCs to show proof of concepts
  • Fulltime
Read More
Arrow Right

Senior Applied AI Engineer

We’re hiring a Senior Applied AI Engineer to join a fast‑moving, high‑ownership ...
Location
Location
United States , Mountain View
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Master’s Degree AND 3+ years of experience in engineering, problem solving, model building, evaluation, data analysis OR equivalent experience
  • 2+ years shipping production-level code, models, or data analysis
  • 1+ years using AI-assisted coding and analysis techniques
  • Experience working on small teams and mid-stage startup environments
  • Experience working on AI products
  • PhD in engineering, applied math, statistics, or related analytical field
  • 4+ years shipping production-level code, models, or data analysis
  • Deep experience building from zero-to-one
  • Hands on work hillclimbing AI evaluations
Job Responsibility
Job Responsibility
  • Design and ship LLM‑powered assistant features, including conversational flows, agentic behaviors, retrieval pipelines, and multimodal interactions
  • Build prompt architectures, system instructions, and orchestration logic that ensure reliability, grounding, and personality consistency
  • Prototype new capabilities rapidly and iterate based on user signals and evaluation data
  • Build and maintain evaluation frameworks for correctness, safety, grounding, and UX quality
  • Run hillclimbing loops across prompts, models, and tool‑use strategies to continuously improve assistant performance
  • Analyze failure modes, design mitigations, and drive systematic improvements across the stack
  • Develop internal tools for prompt experimentation, model comparison telemetry and debugging automated eval pipelines
  • Create reusable frameworks that accelerate the entire AI org’s ability to ship high‑quality assistant features
  • Integrate LLMs with product surfaces, APIs, and backend systems
  • Build lightweight ML components (ranking, classification, summarization, personalization) that enhance assistant intelligence
  • Fulltime
Read More
Arrow Right
New

Principal Threat Research Lead

Protecting billions of users and the world’s largest digital estates is among th...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 12+ years of experience in threat research, threat intelligence, detection engineering, or security analytics within large-scale, complex environments
  • Proven ability to lead and individually execute advanced research on emerging threats across cloud, identity, endpoint, and multi-domain attack surfaces
  • Demonstrated expertise in at least one core domain—Threat Intelligence, AI/ML for Security, or Security Analytics—with strong cross-domain proficiency
  • Hands-on experience designing and shipping high-fidelity detection strategies on SIEM/XDR platforms (Microsoft Defender / Sentinel), with a track record of managing false positives and measuring detection efficacy
  • Depth in at least one major cloud (Azure preferred) and solid working knowledge of modern multi-cloud attack vectors
  • Strong proficiency in data analysis and engineering tools (e.g., KQL, Python, ADX and notebook-driven exploration) and experience working with large-scale analytical pipelines
  • Proven ability to independently drive ambiguous, high-impact technical problems to completion
  • Ability to influence cross-functional teams and communicate complex technical concepts to diverse audiences, including leadership and customers
Job Responsibility
Job Responsibility
  • Set technical vision for advanced threat research spanning Threat Intelligence, analytics, and AI across large-scale, cross-domain telemetry platforms — and stay hands-on enough to prove it works
  • Lead deep research into emerging threats, attacker TTPs, and campaign behavior across endpoint, identity, email, cloud apps, and multi-cloud surfaces — translating insight into concrete detection and response strategy
  • Architect AI/ML-driven detection systems — behavioral analytics, anomaly detection, and agentic / LLM-powered enrichment and investigation pipelines — including the evaluation, guardrails, and abuse-resistance (e.g. prompt-injection defense, output validation) that make them production-safe
  • Operationalize intelligence-to-detection pipelines that continuously convert TI into scalable, production-grade detections, managed as detection-as-code (versioned, tested, backtested, CI-deployed)
  • Drive detection-engineering excellence across SIEM/XDR platforms (Microsoft Defender / Sentinel) — owning measurable signal quality, broad coverage, and low false-positive rates
  • Establish efficacy frameworks for detection coverage, false-negative reduction, and signal-to-noise optimization at scale, with clear metrics (precision/recall, true-alert ratio, FP/FN discovery)
  • Individually author and ship high-fidelity detections and hunts when it matters — triaging their false positives and measuring production performance
  • Drive cross-tenant signal correlation, multi-stage attack analysis, and graph-based campaign stitching as a core research capability
  • Collaborate cross-functionally with Product, Engineering, and Operations to productionize research into customer-facing protection
  • Mentor senior researchers and engineers, setting the bar for technical depth, innovation, and execution rigor
  • Fulltime
Read More
Arrow Right

Senior Analyst

As a Senior Analyst in Meta’s Competitive Intelligence organization, you will op...
Location
Location
United States , San Francisco
Salary
Salary:
177000.00 - 247000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelors degree and a minimum of 6 years of work experience (minimum of 4 years with a Ph.D.) in business intelligence, product analytics, or economic or strategy consulting in a technology environment with increasing scope and impact
  • Demonstrated skill to ethically source, validate, and synthesize high-signal insights from people (e.g., stakeholder interviews, skilled conversations, field research, and relationship-based information gathering) while maintaining high standards for privacy, consent, and integrity
  • Proficiency in AI-powered tools: Demonstrate working knowledge of Generative AI technologies (e.g., LLM and AI agents) and experience designing, prompting, and orchestrating AI systems (e.g., prompt engineering) to automate data analyses, synthesize insights, and execute multi-step analytical tasks (e.g., prompting agent to clean datasets, build visualizations)
  • Practical working understanding of data-analytics tools, and direct experience managing, analyzing, manipulating and interpreting 1P and external 3P datasets
  • Experience with data querying languages (e.g., SQL), scripting languages (e.g., Python), and/or statistical/mathematical software (e.g., R)
  • Proven experience with statistical analysis including causal inference (e.g., randomized control trials, quasi-experimentation such as synthetic control, diff-in-diff, meta-analyses), and/or bayesian aggregation (e.g., bayesian pooling, hierarchical modeling)
  • Demonstrated communication skills and experience presenting complex findings to both technical and non-technical stakeholders
  • Demonstrated experience thriving in ambiguous environments and shape new analytics organizations or products
Job Responsibility
Job Responsibility
  • Market Strategy: Influence organization-level product direction through data-driven narratives and an in depth understanding of the market landscape. Demonstrated experience operating at scale, and across ambiguous, environments with working knowledge of econometrics. As a quantitative market-strategist, you will blend practical and applied understanding with technical expertise, including pressure-testing data for quality, reliability, understanding data-biases and being solution driven
  • Analytics Leadership: Conduct advanced analyses with 3P datasets, develop statistical models and forecasts, and deliver actionable insights that informs market and business strategy. These include, but are not limited to: Data onboarding: Identify, onboard, and rigorously evaluate 3P datasets to determine their signal-to-noise ratio and predictive power
  • Data triangulation: Triangulate data from many sources of imperfect information. Synthesize multiple, low-fidelity 3rd-party signals into a single high-fidelity trend report using Bayesian aggregation or other methods
  • Data transformation: Apply quasi-experimental designs (e.g., synthetic control, diff-in-diff) to isolate the impact of exogenous market shocks and competitor actions on internal performance metrics, using 3rd-party behavioral and economic datasets
  • Insight and implications: Apply guidance from such analyses to increase the accuracy of forecasts and better understand market trends
  • Technical & Methodological Expertise: Act as a recognized professional in a technical or methodological area (e.g., causal inference, bayesian aggregation), driving the adoption of advanced methods and organization-wide best practices that raise the bar for the entire team
  • Data Governance & Quality: Ensure data privacy, security, and compliance with organizational standards. Champion data quality frameworks and documentation practices that enable credible reproducible analyses
  • Resourceful, adaptable professional with a bias for action
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right

Senior Applied Scientist

Core Search and AI team (Bing) is looking for people who want to build the next ...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics predictive analytics, research) OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research) OR equivalent experience
  • Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research) OR equivalent experience
  • 3+ year of industry experience applying Machine Learning techniques or Data Analysis Applications
  • Experience building and improving large scale Machine Learning system for search, ads, and recommendation, adopting LLM
  • Research background on Machine Learning, LLM and NLP
  • Proficient problem solver: ability to identify and solve problems that the world has not solved before
Job Responsibility
Job Responsibility
  • Master one or more subareas and build expertise across a broad research landscape, including advanced research methodologies and applied techniques
  • Develop deep knowledge of a service, platform, or domain, and identify product opportunities by sharing emerging industry trends and applied technologies
  • Review business requirements and incorporate research insights to meet strategic goals
  • Provide direction on the types of data needed to solve complex problems and apply deep subject-matter expertise to drive measurable business impact
  • Support the onboarding of junior team members and help develop academic collaborators into effective contributors within multidisciplinary teams
  • Identify promising research talent, engage with the academic community, and strengthen Microsoft’s long-term recruiting pipeline
  • Document ongoing work, experimental results, and research findings to promote transparency and innovation
  • Apply understanding of fairness and bias to help shape ethics and privacy policies related to research processes and data collection
  • Advance LLM post-training and alignment techniques by designing, implementing, and evaluating novel methods that improve reasoning quality, safety, controllability, and factual grounding across large-scale models
  • Develop next-generation search capabilities by building and optimizing retrieval, ranking, and relevance systems that integrate deeply with LLM-powered experiences
  • Fulltime
Read More
Arrow Right

Senior GenAI Specialist – Finance – Vice President

We are seeking a highly skilled and passionate Senior GenAI Specialist with 8-10...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6-10 years of relevant experience in Apps Development or systems analysis role
  • Extensive experience system analysis and in programming of software applications
  • Experience in managing and implementing successful projects
  • Subject Matter Expert (SME) in at least one area of Applications Development
  • Ability to adjust priorities quickly as circumstances dictate
  • Demonstrated leadership and project management skills
  • Consistently demonstrates clear and concise written and verbal communication
  • Master's degree or PhD in a relevant field
  • 8-10 years of experience in AI/ML development with a proven track record in GenAI
  • Deep understanding of GenAI models and architectures, including transformers, LLMs (Llama 3, Llama 4, Gemini, GPT-4), GANs, and diffusion models
Job Responsibility
Job Responsibility
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
  • Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
  • Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency
  • Design, develop, and implement sophisticated GenAI solutions for diverse financial applications, exploring advanced concepts like Agentic AI and RLHF
  • Design and implement intelligent chatbots for enhanced customer interaction and operational efficiency
  • Fulltime
Read More
Arrow Right