Staff AI Systems Engineer

• Immerse yourself in the current system design and agent/tooling landscape. Understand the core customer use cases and data flows
• Support the team by shipping a few quick wins (e.g., refining tool APIs, prompt engineering, fixing bugs)
• Stand up the foundational eval and observability scaffolding (datasets, metrics, KPIs, reporting)
• Propose a technical architecture and implementation plan for an agent evaluation framework
• Deliver the MVP evaluation harness to produce initial metrics, enable debugging and perform regression testing
• Take on a system feature that offers demonstrated improvement against your MVP evaluation suite
• Productionize the evaluation and observability platform and make it the source of truth for quality and safety. (e.g. Online/offline tracing, alerting, dashboards, evaluations and PR-gated regression suite)
• Own the roadmap for evolving the agent evaluation platform
• Lead deeper R&D threads (e.g., lightweight fine-tuned projection layers, specialized embeddings, multimodal understanding) that can improve system performance on core metrics

• Familiarity with Agentic Systems: Hands-on experience with LLM agents including: LLM API use (e.g. LangChain/LangGraph, vLLM, OpenAI/Gemini/Anthropic APIs)
• Agent Design: tool use (e.g. via MCP), retrieval, memory, grounding/attribution for claims, and guardrails
• Architectural patterns: planning and hand-off for multi-agent systems, context management
• RAG: vector/hybrid search (e.g. pgvector, turbopuffer, rerankers, etc.)
• ML Platform expertise: 5+ years building and shipping ML systems to production
• Backend Python and JS familiarity required
• Typescript/Golang familiarity welcome
• Web services (e.g. Express/FastAPI, REST, SSE, JWTs)
• Cloud Infrastructure (e.g. AWS, Terraform, VPC, Networking)
• Backend databases/stores (e.g. Postgres, Redis)
• Observability (e.g. Prometheus, Grafana, OpenTelemetry, LangSmith/Langfuse)
• Experience with LLM Evaluations at scale: You’ve built offline/online eval harnesses and are familiar with the methodologies and metrics to measure: Search, retrieval, and recommendation performance
• Safety & robustness (security, compliance, red-teaming, regression testing)
• Cost, performance and latency trade-offs

• [Preferred] Durable execution (e.g. Temporal, Hatchet)
• [Preferred] OLAP (e.g. ClickHouse, Bigquery)
• [Preferred] ML Inference (e.g. PyTorch, TensorRT, NVIDIA Triton), ideally in multimodal domains (text/image/video)
• [Preferred] Compute orchestration (e.g. Kubernetes, Prefect, Ray)
• [Preferred] Agentic task success, trajectory quality, preference learning (e.g. SFT, DPO, RLHF, LLM-as-judge)

Flock Safety Stock Options