Part 07Safety, cost & evaluation

Async multiplies value—and the blast radius of error.

Safety is a control system: constrain autonomy, verify evidence and freshness, measure the right outcomes, and stop bad versions before they reach an entire panel.

Risk-tiered action policy

Preparation may be autonomous; clinical accountability is not

Action classAutonomyRequired controls

Summarize, retrieve, rank

Autonomous preparation

Visible provenance, audit, freshness and deterministic safety gates

Prepare patient message, task or order request

Propose-and-confirm

Named reviewer sees exact payload and consequence before release

Routine scheduling/admin action

Policy-configurable

Narrow allowlist, reversible action, idempotency and exception monitoring

Medication, diagnosis, clinical order, high-risk escalation

Human required; no autonomy

Named accountable clinician, current context and explicit approval

Unsupported or materially conflicting recommendation

Never execute

Abstain, expose uncertainty and request clinical review

Named accountable human

Every clinical action resolves to a specific clinician role and person before approval. “The practice” or “the system” is not an accountable owner.

Defense in depth

Five gates from trigger to verified outcome

Before reasoning

Identity/scope, consent, data quality, critical-event policy, freshness and eligibility.

During reasoning

Scoped context, allowlisted tools, schemas, budgets, versioned guidelines and no direct execution.

After reasoning

Claim-evidence verification, temporal supersession, contradiction detection, calibration and deduplication.

Before action

Re-check state version, named owner, autonomy policy, idempotency key and exact approval payload.

After action

Confirm external result, record outcome, monitor exceptions and support reversal/escalation.

Temporal grounding

Check later evidence that may supersede an earlier alarming value; never cite only the first matching fact.

Claim-level support

Every clinical claim maps to a patient evidence reference and, where applicable, a versioned guideline section.

Calibrated abstention

Missing, contradictory or low-quality evidence becomes an explicit human-review reason—not confident prose.

Scale + cost math

Spend inference only where it can change the queue

Illustrative assumptions

StageVolumeBudget/callEstimate

Tier 0 rules

1,200 patients

SQL/features

$0 model cost

Tier 1 triage

30 calls

2k input + 150 output

~$0.04

Tier 2 specialists

16 calls

8k input + 800 output

~$0.58

Tier 3 reconciliation

2 calls

24k input + 1.5k output

~$0.57

Planning buffer

25%

Retries + token variance

~$0.30

Selective weekly run

One 1,200-patient panel

Inference only

~$1.50

Naive frontier fan-out

1,200 × 4 × $0.285

≈ $1,368

per panel run, before retries and validation. Selective reasoning is roughly three orders of magnitude cheaper under the planning assumptions.

Illustrative planning rates—not vendor quotes: small $0.50/$2, mid $3/$15, frontier $10/$30 per 1M input/output tokens.

Naive alternative: 1,200 × 4 frontier condition calls × ~$0.285 ≈ $1,368 per panel run.

Caches, deterministic features and fresh-candidate reuse reduce repeat work; infrastructure and human-review time are measured separately.

Evaluation by workflow shape

Ground truth is a clinician-created ranking—not an LLM preference

Panel prioritization

Offline: Critical-case recall · precision@K · NDCG · duplicate/stale/unsupported rate
Online: Accept/modify/dismiss · time-to-review · action completion · alert burden
Release guardrail: No critical-recall or grounding regression; sampled clinician adjudication

Inbox drafting

Offline: Correct route · unsafe-response recall · grounding · draft completeness · edit distance
Online: Approval/edit rate · time saved · escalation rate · response SLA
Release guardrail: Zero autonomous clinical sends; unsafe messages must route to human

Practice improvement

Offline: Metric correctness · cohort validity · subgroup stability · temporal reproducibility
Online: Insight usefulness · drill-down use · experiment adoption · measured change
Release guardrail: Suppress small/unstable cohorts and disclose missing/confounded data

Panel ground-truth protocol

Two clinicians independently review engineered or historical panels, identify patients requiring attention, assign urgency and expected action, then adjudicate disagreements. Critical omissions receive more weight than minor ordering differences.

Deployment gates

A model version earns the right to see a panel

Deterministic CI

Schemas, policies, ranking, permissions, freshness and Temporal replay tests.

Fixture + historical replay

Engineered edge cases plus prior panels with clinician-created labels.

Shadow

Generate outputs without clinician exposure; compare to current system and human review.

Canary

A small clinician cohort with strict rollback thresholds and daily sampling.

Progressive release

Expand only while quality, safety, cost, latency and burden remain inside budgets.

Observability + HIPAA

Debug the system without turning traces into a second chart

Trace envelope

trace_id + workflow_id

tenant/clinician/patient pseudonymous IDs

state + guideline + agent + prompt + model versions

routing decision + tool IDs

token, latency and cost totals

grounding + safety outcomes

approval + execution status

encrypted evidence references—not raw PHI in logs

PHI controls

BAA-approved vendors only

Minimum-necessary context per agent

Encryption in transit and at rest

Role/tenant/patient scoped access

Redacted logs with short retention

Access and export audit trail

Assurance principle

Optimize for critical-case recall and clinician attention first; model elegance, raw acceptance rate and autonomous volume are secondary.

Data & state Next: Roadmap & non-goals