Benchmarks you can run (and we are running)

Ambient scribes need more than NPS surveys. These are the best public frameworks for comparing transcription, note quality, and workflow fit in Canadian primary care.

SCRIBE evaluation framework

npj Digital Medicine (2025) · 2025

  • Simulation testing (robustness, bias, fairness)
  • Computational metrics (WER, ROUGE, BLEU, BERTScore, toxicity)
  • Human reviewer assessment
  • LLM-as-judge + clinical best practice

Transcription, diarization, and note generation governance

Primary publication

Primary-care competitive analysis framework

JMIR Human Factors (2025) · 2025

  • Usability (12-item framework, 3 domains)
  • Effectiveness & technical performance
  • Accuracy & quality vs standardized encounters

Head-to-head vendor comparison using 4 standardized patient audios + SOAP benchmarks

Primary publication

Physician Documentation Quality Instrument (PDQI-9)

Peer-reviewed validation literature · 2025

  • Accurate
  • Thorough
  • Useful
  • Organized
  • Succinct
  • Internally consistent
  • Synthesized
  • Comprehensible
  • Sufficient for continuity

Blinded comparison of AI notes vs physician gold notes

Primary publication

Canada AI Scribe Evidence Hub — open encounter battery

This site (in development) · 2026

  • 4 JMIR-style standardized primary-care scripts (EN + optional FR)
  • 4 synthetic multi-speaker stress cases (noise, tangents, third party in room)
  • Latency: record-to-draft and draft-to-EMR-push
  • Error taxonomy: omission, deletion, fabrication, SOAP mis-structure
  • EMR push fidelity: field-level match score

Repeatable public benchmark for Canadian EMR workflows

Primary publication

Recommended reading

What this hub will test next

  1. JMIR four standardized encounters (SOAP gold notes + 3-point Likert where applicable).
  2. PDQI-9 blinded review by two Canadian physicians per note.
  3. Latency: end-of-visit to draft; draft to Accuro field push fidelity.
  4. SCRIBE-style error injection (medication changes, third-party speakers) for robustness.
  5. Publish open audio scripts + scoring sheets for reproducibility.

← Back to evidence hub