Software Engineer, Product (Agents / Evals) Job at Lovable (Stockholm)

Software Engineer, Agents

At Harvey, we’re transforming how legal and professional services operate — not ...

Location

United States , San Francisco

Salary:

165000.00 - 312000.00 USD / Year

Harvey

Expiration Date

Until further notice

Requirements

Passion for building effective domain-specific agents
Iterative mindset: you develop proof of concepts, make decisions quickly, and ship v0s
Comfortable with when and how to use evaluations to drive quality
Humble and adaptable about code and frameworks. We expect you to drive adoption of new best practices as they develop
3+ years (post-BS/MS) of software engineering experience
Proficiency in Python and experience working with LLM APIs and agent frameworks
Experience with shipping user-facing products, either on the backend or full-stack

Job Responsibility

Partner with customers and PMs to understand legal workflows, design practical evaluations that capture what “excellent” means, and ship agents that get the job done
Optimize agent performance through prompt engineering, model selection, tool design, skill writing, context window management, and eval harness development
Work with our model infra team to design and implement infrastructure for low-latency agent execution, including caching strategies, parallel tool calls, or subagent patterns
Improve our observability and instrumentation to profile agent behavior, identify bottlenecks, and drive optimization decisions
Stay current on new developments in agentic systems and bring those learnings back to the products we build

What we offer

Comprehensive health, dental and vision coverage
retirement benefits (401k match up to 4%)
flexible PTO
equity plan
bonus

Fulltime

Senior Software Engineer, AI Product

As a Senior Applied AI Engineer at Vanta, you will play a crucial role in shapin...

Location

United States

Salary:

207000.00 - 244000.00 USD / Year

Vanta

Expiration Date

Until further notice

Requirements

At least 7 years of industry experience as a software engineer
You’ve shipped LLM-backed products and have experience with prompting, RAG, and/or agent frameworks
You have experience designing, building, and scaling full-stack applications, including backend systems, APIs, and frontend interfaces
You have familiarity with TypeScript, React, and Node.js, or a willingness to learn
You have experience improving AI systems, creating eval sets, and driving quality hill-climbing
You have experience mentoring other engineers and collaborating with product and design
You have worked at rapidly scaling startups and large companies, especially with environments that prioritize a bias for action
You are action-driven, willing to roll up your sleeves and engage directly with users
You aren’t afraid to put on your product hat
While you bring strong opinions, you prioritize building a platform that meets users where they are

Job Responsibility

Work cross-functionally to design and implement AI-powered features to deliver customer value and integrate LLMs with Vanta’s existing products and systems
Instrument evaluations, guardrails, and monitoring, and review customer usage to continually improve quality
Collaborate with AI Platform engineers shaping foundational AI systems and tooling that accelerate product teams
Make pragmatic tradeoffs that consider business priorities, user experience, and a sustainable technical foundation
Mentor engineers, champion good technical and product instincts, and model a collaborative, high-ownership engineering culture

What we offer

Offers Equity
medical benefits
401(k) plan
other company perk programs
Comprehensive medical, dental, and vision coverage, with 100% of employee-only benefit premiums covered for most medical plans
16 weeks fully-paid Parental Leave for all new parents
Health & wellness stipend
Remote workspace, internet, and cellphone stipend
Commuter benefits for team members who report to the SF and NYC office
Family planning benefits

Fulltime

Software Engineer, Applied Evals

Applied Evals defines what good looks like for safe, advanced AI systems. We tur...

Location

United States , San Francisco

Salary:

230000.00 - 325000.00 USD / Year

OpenAI

Expiration Date

Until further notice

Requirements

4+ years of experience in software engineering with strong fundamentals and a track record of shipping production systems end-to-end
Experience building AI agents or applications, including designing evals and improving performance through prompting or scaffolding
Familiarity with evaluation methods for LLMs and have worked with patterns like multi-agent workflows, tool use, or long context
Familiarity with deep learning concepts or prior exposure to training models
Ability to communicate clearly across technical and non-technical audiences across levels
Motivated by high-impact collaboration with research and product teams and thrive in ambiguity

Job Responsibility

Define the core evaluation signals that drive model improvement at OpenAI, turning vague product gaps into crisp, defensible measures of quality
Design agents, harnesses, and eval pipelines that are reliable, reproducible, and extendable
Prototype solutions with real workflows and convert them into scalable feedback loops
Connect evaluation signals directly to research and training systems so product improvements show up in what users experience
Shape model interaction paradigms by partnering with engineering, research, and product teams on how models are deployed and measured
Build reusable systems and tools that enable contributions from across the company and steadily raise the quality bar

What we offer

Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
401(k) retirement plan with employer match
Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
Mental health and wellness support
Employer-paid basic life and disability coverage
Annual learning and development stipend to fuel your professional growth
Daily meals in our offices, and meal delivery credits as eligible

Fulltime

Senior Software Engineer, AI Evals

As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for ...

Location

United States , San Francisco

Salary:

240000.00 - 280000.00 USD / Year

Sentry

Expiration Date

Until further notice

Requirements

Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
Comfort writing production-quality code (we use Python and TypeScript)
Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)

Job Responsibility

Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

What we offer

Offers Equity
incentive compensation
equity grants
paid time off
group health insurance coverage

Fulltime

New

Senior Software Engineer

Wells Fargo is seeking a Senior Software Engineer. In this role, you will: Lead ...

Location

India , Bengaluru

Salary:

Not provided

Wells Fargo

Expiration Date

June 09, 2026

Requirements

4+ years of Software Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
Experience in backend application software development, with ability to quickly adapt to Java, C# and Python code bases
Strong understanding of Retrieval-Augmented Generation (RAG), prompt engineering and agentic workflows
Deep knowledge of implementing guardrails and advanced techniques for query enrichment and re-writing
Expertise in test or eval driven development including data and error analysis ensuring robust and scalable AI software
Experience architecting and implementing agentic frameworks for autonomous multi-step reasoning and planning
Solid grasp of parsing, chunking, indexing and re-ranking of multiple file formats
Experience with Generative AI Operations, and enterprise-scale AI adoption strategies
Familiarity with enterprise-scale software systems and their integration within large organizations
Experience in enterprise AI model lifecycle management, AI compliance, and risk mitigation strategies

Job Responsibility

Lead moderately complex initiatives and deliverables within technical domain environments
Contribute to large scale planning of strategies
Design, code, test, debug, and document for projects and programs associated with technology domain, including upgrades and deployments
Review moderately complex technical challenges that require an in-depth evaluation of technologies and procedures
Resolve moderately complex issues and lead a team to meet existing client needs or potential new clients needs while leveraging solid understanding of the function, policies, procedures, or compliance requirements
Collaborate and consult with peers, colleagues, and mid-level managers to resolve technical challenges and achieve goals
Lead projects and act as an escalation point, provide guidance and direction to less experienced staff

Fulltime

Ai Qa Engineer (Agents)

An AI QA Engineer (Agents) is responsible for ensuring the quality, reliability,...

Location

Ireland , Cork

Salary:

Not provided

Marriott Bonvoy

Expiration Date

Until further notice

Requirements

4+ years' total experience, including 1+ year testing AI/ML applications, LLM integrations, or conversational interfaces
Hands-on experience with end-to-end testing and automation for AI/agentic products
3+ years of experience in software quality assurance or testing
1+ years of experience testing AI/ML applications, LLM integrations, or conversational interfaces
Strong understanding of software testing principles, methodologies, and best practices
Experience writing and maintaining automated tests (unit, integration, or end‑to‑end)
Proficiency in at least one programming language (Python, TypeScript, JavaScript, Java, etc.)
Experience with API testing tools (Postman, REST Assured, etc.) or frameworks
Strong analytical and problem‑solving skills
Excellent attention to detail and ability to identify edge cases

Job Responsibility

Design and execute test plans for AI agents and agentic experiences
Write and maintain automated test suites for agent functionality (unit tests, evals integration tests, end‑to‑end tests)
Perform (minimal)manual testing of agent interactions, workflows, and business logic
Test agent responses, accuracy, and behavior across various scenarios and edge cases
Identify, document, and track bugs through resolution
Collaborate with engineers, product managers, and business stakeholders to understand requirements and acceptance criteria
Participate in test planning, test case design, and test strategy discussions
Create and maintain test data, test scenarios, and test environments for agents
Participate in feature design sessions, highlighting key testing scenarios and fault zones
Execute performance and load testing to ensure agent scalability and response times

Fulltime

Agios AI Foundation Software Engineer

At Meta Reality Labs Research (RL-R), our goal is to explore, innovate and desig...

Location

United States , Redmond

Salary:

121992.00 - 181000.00 USD / Year

Software Engineer, AI

At Monarch, AI is the engine powering intelligent, personalized financial experi...

Location

United States

Salary:

Not provided

Monarch Money

Expiration Date

Until further notice

Requirements

5+ years of experience in software engineering
at least 2 years focused on building and operating production ML/AI systems
proven track record of shipping LLM-powered features
deep, hands-on expertise in prompt engineering, RAG systems, and evaluation techniques
strong fundamentals in machine learning: embeddings, similarity search, classification, and probabilistic reasoning
demonstrated experience building and using AI evaluation tooling (e.g., golden sets, rubric scoring, LLM-as-judge)
excellent Python skills
history of building production-grade AI features and services
strong collaboration and communication skills with a sharp product sensibility
strategic mindset, comfortable making build-vs-buy decisions and designing features for long-term reliability

Job Responsibility

Apply AI to Real Financial Problems: Use GenAI and ML to help users make sense of their money, understand spending patterns, surface actionable insights, or automate tedious financial tasks
Choose the Right Tool for Each Problem: Navigate the AI toolkit thoughtfully, know when a well-crafted prompt suffices, when retrieval systems add value, and when custom models are worth the investment
Ship with Confidence: Leverage and enhance our sophisticated evaluation framework to ensure AI quality, design test datasets, implement new scorers, and use our Braintrust-based eval system to validate changes before they reach users
AI feature development, agent design and orchestration, ML model improvements, evaluation datasets and scorers, prompt engineering, and feature-level quality

What we offer

Work wherever you want! As a fully remote company
Competitive cash and equity compensation
Stipend to set-up your ideal working environment
Competitive Benefit Plans for employees based on your location (e.g. in the US we offer: Medical, dental and vision benefits and the ability to contribute to a 401k plan)
Unlimited PTO
3 day weekend every month! We take off the “First Friday” every month

Fulltime

Select Country

Software Engineer, Product (Agents / Evals)

Job Description

Job Responsibility

Requirements

Nice to have

Looking for more opportunities?