Retry work without charging twice

A queue answers “which worker should receive this job?” A workflow answers “which step is this business process in, and what happens next?” Do not add orchestration until the process needs explicit state.

List the states, branches, parallel work, waits, callbacks, compensation, duration, audit history, and volume. Independent jobs with the same retry policy may need only SQS workers. A payment that waits hours for a callback and may compensate needs a durable process record.

Separate transient failure from business rejection and poison input. A timeout may deserve retry. A declined payment should follow a nonretry branch. Malformed input may need quarantine and review.

Every retry needs matched errors, bounded attempts, backoff, and a timeout. Unlimited retries amplify an outage. Sending every first failure immediately to a DLQ gives up before a transient dependency can recover.

Idempotency remains required. A task can complete a charge and then time out before the workflow observes success. The next attempt must find the completed business result rather than charge again.

Step Functions Standard workflows fit durable, long-running coordination, callbacks, and execution history. The four-hour payment callback belongs here because the process must wait and remain auditable.

Express workflows fit short, high-rate processing. Asynchronous Express uses at-least-once workflow execution; Synchronous Express uses an at-most-once boundary. Verify current duration and rate limits before choosing.

Those execution guarantees do not make an external side effect happen exactly once. A payment API, inventory update, or email can still complete before a timeout is observed. Each irreversible task needs its own stable operation key and stored result.

SQS stores work but not an end-to-end sequence of business states. EventBridge routes events but does not maintain callback and compensation state. A Lambda that sleeps or polls hides the process in runtime and makes recovery harder to inspect.

Choose the workflow type before polishing the state diagram. Duration, invocation semantics, history, and volume are deciding boundaries; a familiar visual editor is not.

Give each side effect an operation key tied to the business action, such as capture-order-482. Before executing, check whether that key already has a completed result. After success, persist the result where a retry can find it.

Use Retry only for errors that may recover. Keep attempts and total time bounded. Use Catch to send exhausted technical failure or a business rejection to the correct next state rather than treating every failure alike.

Compensation is a new business action, not a database rollback. Releasing reserved inventory after a failed payment can itself fail, so compensation also needs state, retry rules, and an owner.

For callbacks, bind a token to the expected order and caller, keep it out of logs, reject replayed or late callbacks, and time out the waiting state. A callback without authentication can advance the wrong execution.

Observe the branches separately: transient failure then success, business rejection, retry exhaustion, callback timeout, and compensation failure. Alarm on stuck or failed executions and retain enough context for review without exposing secrets.

For the payment scenario, Standard records the four-hour wait and every transition. A duplicate-injection test must show one charge for one operation key, while a failed downstream step enters compensation or review without losing the execution history.