Daniel Flippo
On this page

Integration and data pipelines
currently healthcare EDI

Reading this with an agent? llms.txt has the same facts in plain text.

Daniel Flippo

Integration and data pipelines
currently healthcare EDI

Reading this with an agent? llms.txt has the same facts in plain text.

Home

Case study · Integration failure lab

One scenario file, two runners: a browser simulation and a Java service that execute the same failures.

Break a small healthcare-style claims pipeline on purpose, watch the traces, retries and queue state, then turn on the safeguards every integration relies on and run the same failure again.

  • JavaScript
  • Java 21
  • Spring Boot
  • Synthetic data

Who it’s for

Anyone who wants to see why integration reliability is hard and what the standard fixes cost: a hiring manager skimming the site, or an engineer who has been paged for a duplicate payment. The visitor picks a failure, runs it and reads the outcome in plain language.

The problem

Most of my work is moving data between systems that don’t fail politely. Messages time out, arrive twice, show up out of order, or change shape without warning. None of those failures is healthcare-specific, but the claims domain makes the damage obvious: a claim paid twice, or one that ends up “Accepted” after it was “Paid”.

Reading about retries, dead-letter queues and idempotency does not show what each one costs. The lab makes you cause the failure first.

Constraints

  • Deterministic. A virtual clock with fixed latencies and no randomness, so each scenario replays identically.
  • Dependency-free. Plain JavaScript with no dependencies, so it embeds directly in a static site.
  • Synthetic data only. One claim, C-1001, expected to end Paid at $120.00 with three events applied exactly once in order. No real payers, patients or PHI.
  • Honest labelling. The lab says it is simulated, and the Java runs are labelled as recordings.

Decisions and tradeoffs What was chosen, what was passed on, and what it costs

One scenario format, shared by the simulator and the runner

Chose
Each scenario is a JSON file in scenarios/ holding the failure injections, the safeguard settings and the expected final record. The browser engine and the Java runner both read the same files, and the UI renders a trace without knowing which one produced it.
Passed on
Writing the scenarios twice, once per implementation.
Cost
The format has to stay neutral between a virtual-clock simulator and real HTTP, so anything only one side can express is left out.

Simulate in the browser, and execute the same scenarios in Java

Chose
The simulation is a pure, dependency-free function on a virtual clock with fixed latencies and no randomness, so every scenario replays identically and embeds in a static site. The Java 21 / Spring Boot runner then runs the same files over real HTTP: a delivery queue with read timeouts and retries, posting to an embedded claim-status service that injects faults on the server side.
Passed on
A simulation alone, or a live backend behind the website.
Cost
The two clocks differ. The runner uses a 300 ms timeout and a 150 ms backoff base instead of the simulation’s seconds, and every recorded trace says so. Nothing here measures timing or load.

Replay recorded traces instead of calling a live backend

Chose
The Java runs are exported to executed/ and shipped with the site, and guided mode replays them next to the simulation.
Passed on
Running the Spring service for visitors.
Cost
A replay is a recording, not a running system.

Show the cost of each safeguard, not only the fix

Chose
Every run ends with expected versus observed records and a one-sentence verdict that names the tradeoff: idempotency needs a dedupe store and a retention window, and ordering adds latency.
Passed on
A tour that turns the safeguards on and stops at “fixed”.
Cost
More to read per run, and a verdict simplifies a real incident.

Reject at the boundary instead of guessing

Chose
With schema validation on, a renamed field is a non-retryable rejection that goes to the dead-letter queue and leaves the claim visibly unpaid.
Passed on
Lenient parsing that reads the missing amount as zero.
Cost
A claim stays unpaid until someone fixes the producer, which is exactly the work the DLQ exists to surface.

The scenarios The fault, the safeguard that handles it, and what that costs

The vendored lab has twelve scenarios. Each is injected into the same producer, queue and downstream, and the expected record is always the same.

ScenarioFault injectedSafeguardOutcome and cost
1a Downstream times out on attempts 1–2 of ClaimPaid Retry limit 3, exponential backoff Delivered on attempt 3, record correct, about 7 s of virtual delay. Retries absorb transient failures.
1b Downstream stays down for 5 attempts Retry limit 3, dead-letter queue ClaimPaid lands in the DLQ and the claim stays Accepted and unpaid. A retry limit trades completeness for bounded load, and somebody has to watch the DLQ.
1c Same outage as 1b Retry limit 0 (fire and forget) The event is silently lost with no DLQ entry. With no retries, a hiccup becomes data loss.
2a Ack for ClaimPaid lost, so the queue redelivers it Idempotency off ClaimPaid is applied twice and the paid amount is $240.00. At-least-once delivery means duplicates will happen.
2b Same lost ack as 2a Idempotency on (dedupe by eventId) The second delivery is skipped and the record is correct. It costs a dedupe store and a retention window.
3a ClaimAccepted delayed past ClaimPaid Ordering off Final status is Accepted after payment, so the record is wrong. Last-write-wins fails when events can arrive late.
3b Same delay as 3a Ordering on (buffer until the seq gap fills) ClaimPaid is held until ClaimAccepted arrives and the record is correct. It adds latency and needs a timeout for gaps that never fill.
4 Timeout on ClaimAccepted, so its retry arrives after ClaimPaid, plus a lost ack on ClaimPaid All safeguards on Correct record, and the trace shows each safeguard firing. Real incidents are combinations of failures.
5a Producer renames paidAmount to amountPaid Schema validation off The consumer reads the missing amount as zero: status Paid, paid $0.00. Lenient parsing turns a contract break into bad data.
5b Same rename as 5a Schema validation on (reject, don’t guess) ClaimPaid is rejected as non-retryable and sent to the DLQ, and the status stays Accepted, loudly. A rejected message is fixable; a wrong record may not be noticed.
6a Poison message: the producer ships ClaimPaid with the amount as text, so the consumer fails on every delivery Retry limit 3, DLQ, no replay Four failed attempts, then ClaimPaid sits in the DLQ; status stays Accepted and unpaid, visibly. Retrying a message that can never work only delays the failure; the DLQ holds it but nothing repairs the record.
6b Same as 6a Replay after the producer fix The fix ships and ClaimPaid is replayed 5 s after it was parked, with a fresh attempt count; the DLQ drains and the record is correct. Separate “retry later” from “never going to work”: park it, fix the cause, replay.

How it was built

One producer emits three synthetic events for a claim: ClaimSubmitted, ClaimAccepted and ClaimPaid. Delivery is at-least-once: on a timeout or a 5xx the queue retries with backoff up to a limit, and what it gives up on goes to a dead-letter queue. The downstream applies events to the claim record, which is where duplicates and misordering do their damage. Every hop is written to a trace log.

How the lab runs a scenario A scenario file feeds two runners, the browser engine on a virtual clock and the Java runner over real HTTP. Each produces a trace in the same format, which the UI renders. scenarios/*.jsonfailures · safeguards · expected record Browser engine Java 21 runner Virtual clockno randomness Real HTTPserver-side faults Tracesimulated Tracerecorded, replayed UI: one trace format, either source
The UI cannot tell whether a trace came from the simulator or from a recorded Java run. Redrawn from the lab’s ARCHITECTURE.md.

The Java runner

A Spring Boot 4.1.1 / Java 21 app. A delivery queue posts each event to POST /claims/{id}/events with the JDK HttpClient and a read timeout, retries with backoff, treats a 422 as non-retryable, and dead-letters or drops what it gives up on. The claim-status controller holds a request past the timeout without processing it, or processes it and holds the ack, and applies the same consumer safeguards as the engine.

Where AI agents did the work, and what I decided

AI coding agents built the lab under my direction. The lab’s architecture doc records the calls decided with me: the Java/Spring Boot second stage, the schema-change scenarios in the first version, and the Guided and Free-form picker. It lists the rest as agent decisions open to revision.

What’s proven so far

Results

  • npm test runs the engine tests, which check all twelve scenarios against their expected records and that each replays identically, plus UI accessibility-structure tests.
  • ./mvnw verify in java/ runs the same twelve scenarios over real HTTP against the same expectations.
  • Both run in GitHub Actions on pushes and pull requests, alongside a SonarCloud analysis.

Limits

  • Synthetic only: one claim and three events. There is no production broker, and nothing here ran in production.
  • No timing or load measurements, and the runner’s wall-clock timings are shorter than the simulation’s.
  • Keyboard use, the 390 px layout and reduced motion are not fully covered by automated tests.

Try it

The playable lab lives on the homepage, so there is only one copy to keep working.

Open the lab on the homepage

Source and docs

Links are pinned to lab commit 60771f9, the version vendored into this site.

Back to the homepage lab