When integrations fail: design recovery before launch

Classify the failure before retrying it

A temporary connection problem may be worth retrying. A missing customer reference usually needs correction. An access failure may require an administrator. Treating all three as identical can generate repeated errors while delaying the action that would actually resolve them.

AWS Step Functions documents retry and catch behaviour for different error conditions, including timing controls. The business principle applies beyond that platform: decide which failures are temporary, which are invalid requests and which require human intervention. Set a finite retry policy and an explicit route when those attempts are exhausted.

Know what already happened

The most awkward failure occurs after an external action succeeded but before its confirmation reached the workflow. If the system retries blindly, it may create a second project, send another customer message or duplicate a financial request. Record a stable business reference and check whether the intended result already exists before repeating consequential work.

This property is often described as idempotency: repeating an operation should not create an unintended additional effect. The implementation depends on the systems involved. Some APIs support dedicated request keys; others require an application-level record of completed actions. Do not assume a platform’s general retry feature makes every external action safe to repeat.

Give exceptions a useful operational home

An exception record should show the business item, failed step, time, error category and responsible person. Include what has already completed and whether replay is safe. Keep sensitive payloads out of broad notification channels; the alert can link an authorised colleague to the necessary detail.

Microsoft’s guidance on organising flows into scopes describes grouping related actions and supporting error handling. That structure helps technical investigation, but business users still need a readable status. "Project creation waiting for valid customer reference" is more actionable for an operations coordinator than a raw connector error.

A practical recovery scenario

Imagine a signed order triggers project creation and a welcome email. The project platform creates the project, but its response times out. A naïve retry creates a duplicate and sends two welcome emails. A safer workflow records the order reference, checks for an existing project and resumes only the unfinished step.

If the lookup is inconclusive, the workflow pauses for review instead of guessing. The coordinator sees the order, the possible project match and the pending email. This hypothetical scenario demonstrates why recovery needs business context. A green retry indicator alone does not establish that the customer experienced one coherent process.

Rehearse the failure checklist

Run controlled tests in a safe environment before launch. Make a field invalid, simulate an unavailable destination and repeat the same event. Verify the resulting business state, not just whether the automation reports success. Avoid using live customer communication for these rehearsals.

• Classify errors into retry, correction and escalation routes.
• Limit retry attempts and avoid immediate repeated requests.
• Record a stable reference for consequential actions.
• Document which completed steps must never be replayed blindly.
• Give every exception an owner and a safe resolution action.
• Test partial success, duplicate events and recovery after a pause.

Make recovery part of the handover

Consider the order of recovery actions. Restoring an integration connection is different from releasing a backlog. A large backlog may overwhelm the destination or send outdated messages. Review which items remain relevant, process them in a controlled way and keep a record of anything intentionally skipped so that reconciliation remains possible.

The person supporting the workflow needs a short runbook: where to inspect a failure, how to check the destination system, when to retry and when to escalate. Record a recovery action so the next person can understand what changed. Manual fixes without a trace make later reconciliation harder.

Reliability also has a cost. A low-volume internal notification may justify a simple manual retry, while order creation needs stronger duplicate protection. Match the recovery design to the consequences of being wrong. The goal is dependable business behaviour, with complexity added where it protects something that matters.