Designing for Partial Failure in Multi-System Workflows
A business process that spans multiple systems will eventually have one of them fail mid-transaction. Designing for that up front is cheaper than discovering it in an incident.
The Happy Path Is Not the Design
A workflow that touches three systems — say, an order that has to update inventory, trigger a payment, and notify a fulfillment partner — has a happy path that's trivial to design and a set of partial-failure paths that determine whether the system is actually reliable. What happens if inventory updates successfully but the payment call times out? That question, asked at design time, is cheap. Asked during an incident, it's expensive and usually answered under pressure with a manual database fix.
Idempotency Is Not Optional
Any operation that might be retried — and in a distributed workflow, almost everything might be retried — needs to be safe to run more than once with the same effect as running it once. Without idempotency, a retry after a timeout can double-charge a customer or double-ship an order. This has to be designed into the operation itself, usually with an idempotency key tied to the originating request, not patched on after the first duplicate-charge incident.
Explicit State Beats Implicit Assumption
A workflow that spans multiple systems needs its own explicit state — a record of exactly which steps have completed and which haven't — rather than inferring progress from the state of the downstream systems themselves. Without that, recovering from a partial failure means manually reconstructing what happened from logs across three different systems, which is slow and error-prone exactly when speed and accuracy matter most.
Compensating Actions, Not Just Retries
Some failures can't be fixed by retrying — the payment succeeded but the fulfillment step failed permanently, for instance. Designing the reverse action (refund, cancellation, reversal) as a first-class part of the workflow, not an afterthought handled manually by support, is what makes a multi-system process actually trustworthy under failure.
Observability Has to Match the Failure Modes
Monitoring a multi-system workflow means more than checking whether each system is up. It means being able to answer, quickly, 'how many workflows are currently stuck in a partial state, and where.' That requires designing the logging and state tracking around the actual failure modes identified at design time — not generic uptime monitoring that has nothing to say about a workflow stuck between two systems that are each individually healthy.