CrawlJobs Logo
Replit Logo Replit · IT - Software Development

Data Scientist, AI Agent

United States, Foster City 140000.00 - 280000.00 USD / Year · Job Posted February 18, 2026
Apply Position
Job Link Share

Job Description

You'll directly impact Replit's AI agent—the core of our product strategy—by defining how we measure success, designing experiments that drive improvements, and turning agent trace data into actionable insights for the AI team and company leadership.

Job Responsibility

  • Design and analyze experiments to measure agent improvements—from model changes to UX variations—with statistical rigor and practical tradeoffs
  • Define success metrics that connect agent trace data (prompts, responses, code changes, execution outcomes) to user outcomes like successful deploys, retention, and revenue
  • Build the semantic layer for agent data in partnership with data engineering—defining the tables, metrics, and models that enable self-serve analysis across the AI team
  • Surface insights from trace analysis that identify failure modes, successful patterns, and opportunities to improve agent effectiveness
  • Partner with AI engineering, product, and leadership to translate data into roadmap decisions
  • you'll have a seat at the table for critical agent strategy discussions
  • Create dashboards and reporting that surface agent performance metrics (task completion, latency, quality scores, user satisfaction) for the AI team and executives

Requirements

  • 5+ years of experience in data science, analytics, or a quantitative role with a focus on product, growth, or experimentation
  • Deep experimentation expertise: A/B testing, experiment design, power analysis, handling skewed data, interpreting results beyond p-values
  • Strong SQL skills and experience designing data models for high-volume event data
  • experience with dbt or similar transformation tools
  • Proficiency in Python and data science libraries (pandas, scipy, statsmodels, etc.)
  • Ability to translate ambiguous questions into structured analysis and communicate findings clearly to both technical and non-technical stakeholders
  • Bias toward action: you ship insights that influence decisions, not just dashboards

Nice to have

  • Experience with LLM or AI agent evaluation—understanding of prompt-response patterns, agent evaluation frameworks, or model quality measurement
  • Background in high-growth SaaS or PLG companies with large-scale event data
  • Experience with modern data stack (BigQuery, dbt, Fivetran, Segment, Hex)
  • Familiarity with experimentation platforms (LaunchDarkly, Statsig, Eppo, or similar)
  • Understanding of developer tools or software engineering workflows
  • You've built agent or LLM evaluation frameworks from scratch
  • Experience with causal inference methods (difference-in-differences, synthetic control, CUPED)
  • Familiarity with real-time data systems or operational analytics for monitoring agent performance
  • Experience working with trace data, logging systems, or observability tooling

What we offer

  • Competitive Salary & Equity
  • 401(k) Program with a 4% match
  • Health, Dental, Vision and Life Insurance
  • Short Term and Long Term Disability
  • Paid Parental, Medical, Caregiver Leave
  • Commuter Benefits
  • Monthly Wellness Stipend
  • Autonomous Work Environment
  • In Office Set-Up Reimbursement
  • Flexible Time Off (FTO) + Holidays
  • Quarterly Team Gatherings
  • In Office Amenities
Replit - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Data Scientist, AI Agent

8 matching positions

Data Scientist – Applied AI & Agent Systems Specialist

✨ About This Role: Design, build, and deploy AI-powered agents and automation sy...
Location
Location
Salary
Salary:
Not provided
hiremoters.ai Logo
HiRemoters
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Business Administration, Operations Management, Public Policy, International Development, or a related field
  • Strong proficiency in Python (production-level coding skills)
  • Minimum 5 years experience in applied AI, machine learning engineering, or AI-driven software development
  • Experience working with LLM APIs (OpenAI, Azure OpenAI, or equivalent)
  • Understanding of agentic architectures (tool use, memory, multi-step orchestration)
  • Experience with API integration and RESTful services
  • Familiarity with vector databases and embedding workflows
  • Knowledge of software version control (Git)
  • Experience with workflow automation/orchestration tools (n8n, LangChain, or equivalent frameworks)
  • Understanding of testing, logging, monitoring, and production deployment principles
Job Responsibility
Job Responsibility
  • Design, build, and deploy AI-powered agents and automation systems that enhance the operational intelligence, decision-making, and workflow efficiency
  • Act as the primary Applied AI Engineer within the Smart Systems & Operational Enablement department, translating business challenges into production-ready AI and agentic solutions
  • Develop software-driven AI capabilities that integrate securely and reliably into enterprise systems
  • Design and build AI agents using LLM APIs (e.g., Azure OpenAI, OpenAI or equivalent)
  • Develop multi-step orchestration workflows integrating AI models with internal systems and APIs
  • Implement agent memory systems (vector databases, embeddings, contextual retrieval)
  • Build and maintain AI-enabled automation using tools such as Python frameworks, n8n, or equivalent orchestration platforms
  • Develop evaluation frameworks to measure agent performance, accuracy, reliability, and drift
  • Deploy AI solutions in production environments following enterprise architecture standards
  • Collaborate with Systems Integration to embed agents within CRM/ERP and operational workflows
What we offer
What we offer
  • Flexible and Remote Working
  • Career Growth
  • Performance Recognition
  • Health and Fitness Benefits
  • Extra Paid Annual Leave
  • Special Birthday Perk
  • Fulltime
Read More
Arrow Right

Senior Data Scientist, AI & Model Risk

It all started with an idea at Block in 2013. Initially built to take the pain o...
Location
Location
United States , Bay Area
Salary
Salary:
139000.00 - 225000.00 USD / Year
cash.app Logo
Cash App
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A minimum of 5 years of related experience with a Bachelor’s degree in a quantitative field
  • or 3 years and a Master’s degree
  • or a PhD without experience
  • or equivalent work experience in risk management, model risk management, or AI risk management
  • Proficiency in Python or similar languages for evaluating AI system behavior, writing test scripts, or analyzing model outputs
  • Strong understanding of generative AI architectures
  • Including LLMs, transformer models, RAG systems, and agentic AI, plus hands-on experience interacting with and critically evaluating these systems, sufficient to assess design decisions, output quality, and limitations
  • Understanding of interagency model risk management principles, including SR 26-2
  • Knowledge of AI testing methodologies, ex. functional testing, bias testing, adversarial testing, and performance monitoring plus familiarity with data privacy and security principles (encryption, access controls, data classification)
  • Excellent written and verbal communication and the ability to translate complex technical AI concepts for non-technical stakeholders, senior management, and regulators
Job Responsibility
Job Responsibility
  • Lead end-to-end AI Risk Assessments for generative AI and LLM use cases across the Bank
  • Embedding in Block's enterprise-wide GenAI review process, coordinating cross-functional SMEs (Legal, Compliance, InfoSec, Data Governance, MRM, ERM, BRC, TPRM, Financial Crimes), and managing timelines to ensure reviews are completed within SLA
  • Review AI system design and documentation
  • Including retrieval sources, assumptions, limitations, fallback plans, guardrail configurations, and change management procedures — across banking use cases such as fraud detection, BSA/AML compliance, credit decisioning, and customer-facing applications, ensuring governance controls are commensurate with each use case's risk profile
  • Assess pre-deployment testing for adequacy inclusive of output integrity, hallucination detection, boundary and edge case testing, ethical and safety guardrails, bias testing, A/B testing, volume testing, and UAT — designing and conducting independent testing as needed
  • Evaluate ongoing monitoring plans for comprehensiveness - including accuracy, hallucination rates, drift detection, sensitive data controls, reliability metrics, CSAT, acceptable performance ranges, and documented remediation procedures
  • Develop and maintain templates, tools, and procedures to support the effectiveness and scalability of the AI Risk Governance Program
  • Monitor the evolving regulatory landscape for AI in banking — including FFIEC IT Examination Handbook standards, FDIC Financial Institution Letters, interagency statements, and the anticipated RFI on AI model risk management referenced in SR 26-2 — and incorporate emerging guidance into the AI risk governance program
  • support SFS's response to regulatory inquiries as needed
What we offer
What we offer
  • Remote work
  • medical insurance
  • flexible time off
  • retirement savings plans
  • modern family planning
  • Fulltime
Read More
Arrow Right

Senior Data Scientist, AI & Model Risk

It all started with an idea at Block in 2013. Initially built to take the pain o...
Location
Location
United States , New York
Salary
Salary:
139000.00 - 225000.00 USD / Year
cash.app Logo
Cash App
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A minimum of 5 years of related experience with a Bachelor’s degree in a quantitative field
  • or 3 years and a Master’s degree
  • or a PhD without experience
  • or equivalent work experience in risk management, model risk management, or AI risk management
  • Proficiency in Python or similar languages for evaluating AI system behavior, writing test scripts, or analyzing model outputs
  • Strong understanding of generative AI architectures
  • Including LLMs, transformer models, RAG systems, and agentic AI, plus hands-on experience interacting with and critically evaluating these systems, sufficient to assess design decisions, output quality, and limitations
  • Understanding of interagency model risk management principles, including SR 26-2
  • Knowledge of AI testing methodologies, ex. functional testing, bias testing, adversarial testing, and performance monitoring plus familiarity with data privacy and security principles (encryption, access controls, data classification)
  • Excellent written and verbal communication and the ability to translate complex technical AI concepts for non-technical stakeholders, senior management, and regulators
Job Responsibility
Job Responsibility
  • Lead end-to-end AI Risk Assessments for generative AI and LLM use cases across the Bank
  • Embedding in Block's enterprise-wide GenAI review process, coordinating cross-functional SMEs (Legal, Compliance, InfoSec, Data Governance, MRM, ERM, BRC, TPRM, Financial Crimes), and managing timelines to ensure reviews are completed within SLA
  • Review AI system design and documentation
  • Including retrieval sources, assumptions, limitations, fallback plans, guardrail configurations, and change management procedures — across banking use cases such as fraud detection, BSA/AML compliance, credit decisioning, and customer-facing applications, ensuring governance controls are commensurate with each use case's risk profile
  • Assess pre-deployment testing for adequacy inclusive of output integrity, hallucination detection, boundary and edge case testing, ethical and safety guardrails, bias testing, A/B testing, volume testing, and UAT — designing and conducting independent testing as needed
  • Evaluate ongoing monitoring plans for comprehensiveness - including accuracy, hallucination rates, drift detection, sensitive data controls, reliability metrics, CSAT, acceptable performance ranges, and documented remediation procedures
  • Develop and maintain templates, tools, and procedures to support the effectiveness and scalability of the AI Risk Governance Program
  • Monitor the evolving regulatory landscape for AI in banking — including FFIEC IT Examination Handbook standards, FDIC Financial Institution Letters, interagency statements, and the anticipated RFI on AI model risk management referenced in SR 26-2 — and incorporate emerging guidance into the AI risk governance program
  • support SFS's response to regulatory inquiries as needed
What we offer
What we offer
  • Remote work
  • medical insurance
  • flexible time off
  • retirement savings plans
  • modern family planning
  • Fulltime
Read More
Arrow Right

Senior Staff Data Scientist (AI Solutions - Machines)

As a Data Scientist at CyberArk, you’ll join a team shaping the future of identi...
Location
Location
Israel , Petah Tikva
Salary
Salary:
Not provided
paloaltonetworks.com Logo
Palo Alto Networks
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2-4 years of hands-on experience as a Data Scientist, with proven ability in designing, building, and deploying solutions utilizing LLMs, RAG, or other Agentic AI frameworks
  • Msc degree from a university in Computer Science, Statistics, Mathematics, Engineering, Data Science or related field, with a thesis on Data Science. GPA above 85
  • Solid understanding of machine learning fundamentals and a strong grasp of data structures, algorithms, and probability
  • Demonstrated independent research capabilities and a self-driven approach to learning and problem-solving
  • Passion for solving meaningful problems and eagerness to learn in a fast-paced, collaborative environment
  • Strong communication skills and ability to explain technical concepts to different audiences
Job Responsibility
Job Responsibility
  • Research, design, Evaluate and production deployment of advanced AI/ML models, specializing in LLMs, RAG systems, and Agentic AI applications
  • Explore and analyze large-scale datasets to uncover patterns and drive actionable insights
  • Collaborate closely with product managers, engineers, and analysts to support feature development and validate ML ideas
  • Stay up to date with the latest academic and industry trends in AI/ML and apply best practices to real-world challenges
  • Grow your skills in classic machine learning, deep learning, natural language processing (NLP), LLMs and cybersecurity applications
  • Fulltime
Read More
Arrow Right

Principal Data Scientist - AI Foundations, Specialist Models

Data is at the center of everything we do. As a startup, we disrupted the credit...
Location
Location
United States , McLean; San Jose; New York
Salary
Salary:
161800.00 - 201400.00 USD / Year
capitalone.com Logo
Capital One
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A Bachelor's Degree in a quantitative field (Statistics, Economics, Operations Research, Analytics, Mathematics, Computer Science, or a related quantitative field) plus 5 years of experience performing data analytics
  • A Master's Degree in a quantitative field (Statistics, Economics, Operations Research, Analytics, Mathematics, Computer Science, or a related quantitative field) or an MBA with a quantitative concentration plus 3 years of experience performing data analytics
  • A PhD in a quantitative field (Statistics, Economics, Operations Research, Analytics, Mathematics, Computer Science, or a related quantitative field)
Job Responsibility
Job Responsibility
  • Partner with a cross-functional team of data scientists, software engineers, and product managers to build a next-generation entity resolution solution
  • Leverage a broad stack of technologies — Python, AWS, Spark, modern state of the art models like transformers, graph ML, and more
  • Build machine learning models through all phases of development
  • Leverage Agentic AI tools and workflows to build and test
  • Flex your interpersonal skills to translate the complexity of your work into tangible business goals
What we offer
What we offer
  • Eligible to earn performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
  • Comprehensive, competitive, and inclusive set of health, financial and other benefits
  • Fulltime
Read More
Arrow Right

AI Data Scientist

You'll be Gamma's first Data Scientist, turning massive volumes of user data and...
Location
Location
United States , San Francisco
Salary
Salary:
180000.00 - 310000.00 USD / Year
gamma.app Logo
Gamma
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience as a data scientist at product-focused tech companies, with experience managing or mentoring other data scientists
  • Strong statistical foundations with hands-on experience designing and analyzing A/B tests and experiments at scale
  • Experience working with large-scale data and building metrics frameworks from scratch, including modern data stack tools like dbt and Snowflake and comfort analyzing unstructured or text data
  • Experience working with AI/ML products, especially LLMs or generative AI, with familiarity evaluating model performance in production settings
  • Ability to communicate complex technical concepts to non-technical stakeholders and influence product decisions with clarity and conviction
  • A clear perspective on how agentic coding is transforming the data science role, and genuine excitement about applying AI to your own work (Nice to have)
Job Responsibility
Job Responsibility
  • Build frameworks that help the team understand AI model performance, user behavior, and product health across Gamma's platform
  • Dig into AI model outputs across user cohorts to identify quality gaps and create evals and metrics to measure improvement
  • Partner with engineering and product to define quality metrics for AI-generated content and user satisfaction
  • Develop statistical models and frameworks that empower product teams to make data-informed decisions independently
  • Design and analyze large-scale A/B tests and experiments with statistical rigor to measure product impact and guide prioritization
What we offer
What we offer
  • equity
  • Fulltime
Read More
Arrow Right

Senior Data Scientist – AI & ML | MLOps Enablement

The Developer Platform Organization’s mission is to accelerate the delivery of r...
Location
Location
United States , Seattle
Salary
Salary:
166000.00 - 258000.00 USD / Year
nordstrom.com Logo
Nordstrom
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s, Master’s, or PhD in Statistics, Data Science, Computer Science, Engineering, or a related technical field required
  • 10+ years of hands-on Data Science experience with production model delivery across multiple ML (classification, ranking, NLP, time-series, recommendation) and GenAI models
  • Deep expertise in model evaluation — defining metrics, thresholds, and evaluation pipelines for real-world production models
  • Experience with Feature Store design, feature engineering, and understanding of feature freshness, reuse, and drift across different model families
  • Proficiency in Python with experience writing clean, maintainable, production-quality ML code
  • Strong understanding of ML monitoring — data drift, prediction drift, and concept drift detection
  • Experience with experiment tracking and model lifecycle management
  • Ability to translate between DS practice and platform engineering — comfortable driving design decisions, authoring DS-native documentation, and engaging in technical design reviews
  • Self-directed
  • comfortable owning POC work end-to-end without a dedicated DS team structure
Job Responsibility
Job Responsibility
  • Run end-to-end POC validation for new platform capabilities — Feature Store, Endpoints, Model Evaluation, AutoML, BigQuery ML etc. — independently, before they reach DS teams at scale
  • Attend DS team planning and design sessions as an embedded practitioner
  • surface real workflow pain points and translate them into reusable MLOps platform requirements
  • Design and own the Model Evaluation Framework — defining metrics, thresholds, and evaluation pipelines for batch, online, and streaming use cases on Vertex AI
  • Build model-type-aware Feature Store schemas, endpoint configurations, and evaluation pipelines that accommodate the fundamentally different needs of different ML models
  • Lead benchmarking of Nordstrom’s platform against industry standards — SageMaker vs. Vertex AI — across feature parity, cost, and DS practitioner ergonomics
  • Author DS-native documentation, onboarding guides, and quickstart notebooks that lower the adoption barrier for new platform features
  • Contribute DS domain expertise to the emerging Vertex AI Agentic Platform — identifying DS workflow pain points as agent use cases and defining evaluation frameworks for agentic responses
  • Own model card standards — capturing what actually matters to a practitioner, not just governance checkboxes
  • Communicate complex trade-offs and platform decisions to technical and non-technical stakeholders across DS, engineering, and leadership
What we offer
What we offer
  • Medical/Vision
  • Dental
  • Retirement
  • Paid Time Away
  • Life Insurance
  • Disability
  • Merchandise Discount
  • EAP Resources
  • 401k
  • performance-based incentives/bonuses
  • Fulltime
Read More
Arrow Right

Senior Data Scientist / AI Consultant – Security & Risk Analytics

Senior Data Scientist / AI Consultant – Security & Risk Analytics. Hybrid – 3 da...
Location
Location
Ireland , Dublin 18
Salary
Salary:
100000.00 EUR / Year
solasit.ie Logo
Solas IT Recruitment
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience delivering end-to-end ML and DL solutions in production environments
  • 8+ years commercial experience
  • Strong programming skills in Python and solid SQL expertise
  • Experience with Generative AI, including LLMs, RAG, vector databases, and agent-based frameworks
  • Strong analytical thinking with the ability to communicate insights to both technical and non-technical stakeholders
  • Hands-on experience in data preparation, feature engineering, and model evaluation
  • A proactive and solution-oriented mindset, comfortable working in fast-paced environments
Job Responsibility
Job Responsibility
  • Translate complex business challenges into data-driven solutions using ML, DL, and GenAI techniques
  • Manage the full model lifecycle including data exploration, feature engineering, modelling, validation, and deployment
  • Build classification and scoring models, effectively handling incomplete or ambiguous datasets
  • Develop explainable AI outputs to support business decision-making
  • Deploy and monitor models to ensure performance, scalability, and reliability
  • Prototype new concepts and evaluate emerging tools and frameworks
  • Collaborate with engineering and product teams to deliver scalable, production-ready systems
  • Support data pipelines, cloud platforms, and CI/CD processes for analytics workloads
What we offer
What we offer
  • Excellent salary package
  • Great benefits
  • Very good career progression
  • Fulltime
Read More
Arrow Right