Idempotency in Patient Data Pipelines
Distributed systems inevitably produce duplicate messages, and in clinical pipelines an unhandled duplicate can mean a doubled medication order. This post covers idempotency keys and idempotent side effects, with one rule of thumb: design consumers so processing an event once or ten times leads to the same end state.
A Duplicate Row Is Not Always Harmless
In most software domains, a duplicate API call is a minor annoyance maybe a double entry you can clean up later. In healthcare, a duplicate isn’t always cosmetic. A duplicated medication order can mean a patient receives a drug twice. A duplicated lab result feed can trigger two conflicting alerts to two different clinicians. Retries, network blips, and message redelivery are normal facts of distributed systems but in clinical pipelines, the cost of getting deduplication wrong is measured in patient safety, not just data cleanliness.
Where Duplicates Actually Come From
- Network retries an HL7/FHIR feed times out and the sending system resends the same message.
- At-least-once message brokers Kafka, SQS, and similar systems guarantee delivery, not uniqueness, so consumers must handle redelivery themselves.
- Multi-system integrations a pharmacy system and an EHR both submit the “same” order through different channels because of a sync delay.
Designing for Idempotency
The core technique is an idempotency key: a unique identifier attached to each meaningful clinical event (an order, a result, a dispense action) that lets the receiving system recognize “I’ve already processed this” and safely no-op on a repeat.
A few practical patterns:
- Natural keys where they exist. An HL7 message has a message control ID; a FHIR resource has an identifier. Use these rather than inventing your own when the source system already provides one.
- Composite keys where they don’t. For homegrown systems, a combination like
(patient_id, order_type, timestamp_bucket, source_system)can approximate uniqueness when no single field is reliable. - A processed-events ledger. Before acting on an event, check a durable store (a database table, a Redis set with TTL) for whether that key has already been handled. This needs to be atomic check-and-set, not check-then-set with a race condition in between.
- Idempotent side effects, not just idempotent checks. If your idempotency check fails silently, the side effect itself (e.g., “dispense medication”) should also tolerate being invoked twice without doubling the real-world outcome. This is harder and often requires designing the downstream action to be a state transition (“mark as dispensed”) rather than an increment (“add one dispensed unit”).
The Cost of Getting This Wrong
Skipping idempotency doesn’t just risk data quality it risks trust. Once clinicians see one duplicate alert or one duplicated order, they start distrusting the system itself, which leads to workarounds and shadow processes that are far harder to unwind than the original bug.
A Reasonable Default
If you’re building or integrating clinical pipelines and unsure where to start: treat every inbound event as if it will arrive at least twice, and design your consumers so that processing the same event ten times produces exactly the same end state as processing it once. That single discipline eliminates most of the dangerous duplicate scenarios before they can happen.