Fault Injection and Worker Recovery

An end-to-end suite can pass every assertion and still miss the failure that duplicates a payment, notification, or external update: the worker successfully performs the side effect, then crashes before recording that it happened.

The Failure Between Side Effect and Acknowledgment

Consider a background job that calls a provider, receives a successful response, and is terminated before saving its outcome or acknowledging the message. After restart, the queue may deliver the same work again.

The interesting question is no longer whether the job completes when every dependency behaves, but whether a second execution can distinguish an unfinished operation from one whose effect already exists outside the process.

Injecting Failures at Each Boundary

To test that boundary, inject a deterministic crash immediately after the provider confirms the effect, restart the worker, and let the normal recovery path run against a durable database and a controlled provider stub.

Repeat the experiment after each meaningful boundary, state persisted, external call completed, result recorded, and message acknowledged, because changing the interruption point changes which evidence survives the process.

Then assert the invariants rather than the final HTTP status: the external effect was not repeated when prohibited, persisted state remains coherent, and the workflow either resumes safely or exposes a reconcilable outcome.

Testing the Recovery Contract

A mock that forgets previous calls on restart cannot reveal this class of bug, just as catching an exception inside the same process cannot reproduce the loss of in-memory state or an unacknowledged delivery.

Fault injection will not prove exactly-once effects across arbitrary providers, but it exposes the recovery contract you actually built: which steps can replay, which require reconciliation, and which failures remain ambiguous.