Before reasoning
Identity/scope, consent, data quality, critical-event policy, freshness and eligibility.
Presenter view
Safety, cost & evaluation speaker notes
Lead with blast radius, defend the autonomy model, show the rough math and explain how a bad version is stopped before panel exposure.
Opening cue
“An async system can repeat the same mistake across 1,200 patients without anyone watching. Safety therefore has to be structural, measurable and enforced before the model reaches the panel.”
01 · Approval model
Separate preparation from consequential action
Summarize, retrieve, rank
Autonomous preparation
Visible provenance, audit, freshness and deterministic safety gates
Prepare patient message, task or order request
Propose-and-confirm
Named reviewer sees exact payload and consequence before release
Routine scheduling/admin action
Policy-configurable
Narrow allowlist, reversible action, idempotency and exception monitoring
Medication, diagnosis, clinical order, high-risk escalation
Human required; no autonomy
Named accountable clinician, current context and explicit approval
Unsupported or materially conflicting recommendation
Never execute
Abstain, expose uncertainty and request clinical review
“I allow autonomous preparation: retrieve, summarize, rank and draft. Patient communication is propose-and-confirm. Medication, diagnosis, clinical orders and high-risk escalation require a named clinician. Unsupported or conflicting work never executes.”
02 · Defense in depth
Safety at every transition
During reasoning
Scoped context, allowlisted tools, schemas, budgets, versioned guidelines and no direct execution.
After reasoning
Claim-evidence verification, temporal supersession, contradiction detection, calibration and deduplication.
Before action
Re-check state version, named owner, autonomy policy, idempotency key and exact approval payload.
After action
Confirm external result, record outcome, monitor exceptions and support reversal/escalation.
“Grounding is claim-level. I verify a patient evidence reference, current effective time and guideline version for every material clinical claim. Contradiction or missing evidence produces abstention, not prettier prose.”
Before execution, re-check
Patient-state version
Candidate expiry
Named owner
Autonomy policy
Exact approval payload
Idempotency key
03 · Cost math
State assumptions before stating savings
Tier 0 rules
1,200 patients · SQL/features
$0 model cost
Tier 1 triage
30 calls · 2k input + 150 output
~$0.04
Tier 2 specialists
16 calls · 8k input + 800 output
~$0.58
Tier 3 reconciliation
2 calls · 24k input + 1.5k output
~$0.57
Planning buffer
25% · Retries + token variance
~$0.30
Selective weekly run
One 1,200-patient panel · Inference only
~$1.50
Contrast
1,200 × 4 × $0.285
≈ $1,368
Naive frontier fan-out versus roughly $1.50 for the selective illustrative funnel.
“These are planning assumptions, not vendor quotes. The point is the topology: deterministic work for 1,200, cheap triage for about 30, mid-tier reasoning for roughly 16 specialist calls and frontier reconciliation only for two conflicts.”
• Illustrative planning rates—not vendor quotes: small $0.50/$2, mid $3/$15, frontier $10/$30 per 1M input/output tokens.
• Naive alternative: 1,200 × 4 frontier condition calls × ~$0.285 ≈ $1,368 per panel run.
• Caches, deterministic features and fresh-candidate reuse reduce repeat work; infrastructure and human-review time are measured separately.
04 · Ground truth + metrics
Each workflow has a different definition of good
Panel prioritization
Offline
Critical-case recall · precision@K · NDCG · duplicate/stale/unsupported rate
Online
Accept/modify/dismiss · time-to-review · action completion · alert burden
Guardrail
No critical-recall or grounding regression; sampled clinician adjudication
Inbox drafting
Offline
Correct route · unsafe-response recall · grounding · draft completeness · edit distance
Online
Approval/edit rate · time saved · escalation rate · response SLA
Guardrail
Zero autonomous clinical sends; unsafe messages must route to human
Practice improvement
Offline
Metric correctness · cohort validity · subgroup stability · temporal reproducibility
Online
Insight usefulness · drill-down use · experiment adoption · measured change
Guardrail
Suppress small/unstable cohorts and disclose missing/confounded data
“For panel ranking, two clinicians independently identify who needs attention, assign urgency and expected action, then adjudicate. I optimize critical-case recall and ranking quality. Acceptance rate alone is dangerous because it can reward easy, low-value recommendations.”
05 · Release, observability + PHI
Earn panel exposure progressively
Deterministic CI
Schemas, policies, ranking, permissions, freshness and Temporal replay tests.
Fixture + historical replay
Engineered edge cases plus prior panels with clinician-created labels.
Shadow
Generate outputs without clinician exposure; compare to current system and human review.
Canary
A small clinician cohort with strict rollback thresholds and daily sampling.
Progressive release
Expand only while quality, safety, cost, latency and burden remain inside budgets.
Trace what matters
• trace_id + workflow_id
• tenant/clinician/patient pseudonymous IDs
• state + guideline + agent + prompt + model versions
• routing decision + tool IDs
• token, latency and cost totals
• grounding + safety outcomes
• approval + execution status
• encrypted evidence references—not raw PHI in logs
Keep traces from becoming a chart
Use BAA-approved vendors, minimum-necessary context, pseudonymous identifiers, encrypted evidence references, short retention and tenant/patient scoped access. Raw PHI should not be a default log field.
Closing transition
“The design is safe enough to test, not assumed safe enough to scale. The final section shows how I would sequence proof, migration and deliberate debt.”