Staff AI Engineer Job at Heidi (Sydney)

Staff AI Engineer

Heidi

Location:
Australia , Sydney

Category:
IT - Software Development

Contract Type:
Not provided

Salary:

Not provided

Save Job

Apply Position

Job Description:

You will operate as a Staff+ AI scientist/engineer on the Orchestration team. You will own the design and delivery of a clinician‑grade retrieval and question‑answering stack across data ingestion, indexing, ranking, grounding, and safe deployment. You will set technical direction, establish quality bars, and lead cross‑functional execution with engineering, product, and clinical experts. You will move between research and production, turning prototypes into reliable services with clear SLAs, traceable outputs, and unit/acceptance metrics that matter in clinical contexts.

Job Responsibility:

Define the end‑to‑end architecture for literature and guideline ingestion, normalization, metadata extraction, de‑duplication, and versioning
Build hybrid search and retrieval: lexical + vector + re‑ranking, with tight latency budgets and cost controls
Design grounding and answer synthesis that cite sources, preserve provenance, and expose confidence and abstention
Lead model work across prompting, fine‑tuning, distillation, and tool use to improve faithfulness, coverage, and utility
Stand up gold‑standard evaluation: offline IR metrics (nDCG, MAP, recall), factuality/faithfulness audits, and human review with adjudication
Run online experiments at scale. Define guardrails, KPIs, and ship A/Bs to measure impact on clinician workflows
Productionize services with observability, tracing, canaries, rollbacks, and incident playbooks
Set data governance for medical content: access control, PHI handling, audit logs, and retention policies
Partner with clinicians to define intents, schemas, and acceptance criteria. Convert ambiguous questions into testable specs
Coach engineers and scientists. Raise the technical bar through design docs, reviews, and reusable components

Requirements:

Staff‑level track record shipping search, NLP, or LLM systems that serve real users at scale
Mastery of Python and SQL
Strong software engineering fundamentals, testing strategy, and API/service design
Depth in modern IR/NLP: embeddings, ANN indexes, re‑rankers, retrieval‑augmented generation, and prompt/program synthesis
Experience building data pipelines: parsing PDFs/HTML, OCR when needed, metadata extraction, and content hashing/versioning
Familiarity with PyTorch, plus distributed training/inference patterns
MLOps and reliability: containers, Kubernetes, feature/model registries, experiment tracking, monitoring, and alerting
Evidence of rigorous evaluation design: offline metrics, human‑in‑the‑loop judging, power analysis for online tests
Clear thinking on safety: hallucination controls, calibration, abstention, red‑teaming, and privacy/security by design
Ability to lead cross‑functional initiatives and make crisp decisions with incomplete information

Nice to have:

Search relevance expertise for long‑form, citation‑heavy domains
Knowledge of biomedical ontologies and standards (e.g., SNOMED CT, UMLS, ICD, RxNorm, FHIR)
Prior work with literature and guideline corpora, de‑duplication, and document lineage tracking
Experience with hybrid retrieval stacks (e.g., BM25 + ANN) and learned re‑rankers
Familiarity with clinical evaluation methods, EBM hierarchies, and annotation workflows
Strong cost‑performance tuning for LLM inference, caching, and batching in production

What we offer:

Flexible hybrid working environment, with 3 days in the office
Additional paid day off for your birthday and wellness days
Special corporate rates at Anytime Fitness in Melbourne, Sydney tbc
A generous personal development budget of $500 per annum
Learn from some of the best engineers and creatives, joining a diverse team
Become an owner, with shares (equity) in the company, if Heidi wins, we all win
The rare chance to create a global impact as you immerse yourself in one of Australia’s leading healthtech startups
If you have an impact quickly, the opportunity to fast track your startup career

Additional Information:

Job Posted:
February 18, 2026

Employment Type:

Fulltime

Work Type:

Hybrid work

Heidi - All Job Offers

Job Link Share: