An integration is easy to demonstrate when every request succeeds in order. The production design becomes visible when a provider is slow, an event arrives twice, or the connection fails after the other system has already completed the action. These are normal operating conditions to plan for.
A dependable integration makes those situations understandable and recoverable. The business should know whether an order was exported, which system owns its current state, and what an operator can safely retry. Start by designing that lifecycle, then implement the API calls inside it.
The key ideas
- Treat an uncertain response as a state to investigate.
- Make repeated delivery safe before enabling retries.
- Build reconciliation and operator recovery into the workflow.
Write down the business contract
Identify the source of truth for each field and the event that starts a change. If two systems can update the same value, define how conflicts are resolved. Record which identifiers cross the boundary, which API version you use, and what successful completion means to the business.
For a catalog export, accepted for processing might be different from visible in the destination. Represent those states separately. Keep a mapping between internal records and provider identifiers so a later update can find the same object. Without that map, recovery often turns into manual comparison of records that look similar but are not equivalent.
Verify and record incoming events
Follow the provider's signature-verification procedure before trusting a webhook. Stripe, for example, documents verification using the raw request body and warns that events may be delivered more than once or out of order. Do not assume all providers share the same signature scheme or retry policy.
After verification, durably record or enqueue the work before acknowledging acceptance. Keep the synchronous path short, but do not return success for work that exists only in process memory. Decide which event fields are safe to retain and how long they are needed. Keep secrets and unnecessary personal data out of diagnostic logs.
Reference: Stripe: webhook verification and delivery behavior
Make repeated work safe
Use a stable operation identifier to recognize a repeated business action. Enforce uniqueness where the result is committed, not only in an in-memory check. If two workers receive the same event together, a check-then-insert sequence without a suitable constraint can still produce duplicate work.
Record the outcome alongside the operation identifier where possible. For an outbound side effect, use the provider's documented idempotency mechanism when available. If the remote outcome is unknown, query or reconcile it before creating a second object. A worker restart should not change the meaning of the original request.
Bound retries and respect the downstream system
Set connection and request timeouts, then distinguish temporary failures from invalid requests. AWS's guidance explains that retries can add load to a struggling dependency and that a timeout does not prove a side effect failed. Use bounded backoff with jitter for retryable work, and avoid stacking independent retry loops at every layer.
Respect the provider's rate-limit contract, including retry instructions when supplied. Stop automatic retries after a defined attempt or elapsed-time limit and move the item into an inspectable failure state. An authorization error or a rejected field usually needs correction; repeating it indefinitely hides the real issue.
Reference: AWS Builders' Library: timeouts, retries, and backoff with jitter
Add a reconciliation path
Webhooks and successful API responses are useful signals, but an integration also needs a way to compare intended state with destination state. Schedule a bounded reconciliation process for important records. Make it safe to resume from a checkpoint and explicit about conflicts it cannot resolve automatically.
Provide an operator view with the business identifier, current status, last attempt, and a safe next action. A replay should preserve the original operation identity where appropriate. Require deliberate review for actions that could create a duplicate charge, shipment, or customer notification. Recovery is a product workflow, not just a queue administration task.
Test the failures that change business outcomes
Before launch, exercise the integration with duplicate deliveries, reordered updates, timeouts, rate limits, and a worker crash around the commit boundary. Confirm the business result, not merely that a handler returned a status code. Include a provider schema change in the maintenance plan.
- Deliver the same event twice and concurrently; verify one intended outcome.
- Interrupt a request after the destination may have accepted it; resolve the unknown result.
- Replay a failed item after correction and verify the audit trail.
- Check that backlog growth and repeated failures reach an accountable owner.
Sources & further reading
The guidance above combines our engineering recommendations with the official references below. Service features and platform requirements can change; check the current documentation when implementing.