Sending email, creating a PDF, scanning a file, updating a search index or notifying an external provider often should not happen inside an HTTP request. The user request can finish quickly; work is placed on a queue and a worker handles it later. Messaging infrastructure usually does not guarantee that a job runs exactly once.
A worker can finish work and crash before acknowledgement. The queue may deliver the message again. A timeout means the outcome is unknown. Reliable design starts here: a job can run again, arrive out of order and be delayed. The application has to produce a correct result under those conditions.
Accept at-least-once delivery
Many queue systems behave close to at-least-once delivery. To prevent a message from disappearing, they wait for explicit acknowledgement from the consumer. If acknowledgement does not arrive, the same message can be redelivered.
If an PaymentReceived message is processed twice, two customer emails can be annoying. If the same message increases a balance twice, the result is much more serious. In both cases, the queue is not the problem. The consumer is unsafe against repeated processing.
Establish idempotency in the data layer
An idempotent operation leaves the same target result when it is called more than once with the same input. A memory-only list of “processed” jobs is not safe: it disappears when a worker restarts. The durable data layer should be the source of that decision.
Every message or job request needs a unique identifier. A consumer can store that identifier in a processed-operations table or on the target record. When the same identifier arrives again, the system returns the successful result without producing the side effect again. A database unique index is an important defence when two workers handle the same message at once.
Key scope matters. Two clicks on “regenerate report” may be one job or separate requests. Design the key around business meaning; a random identifier per call cannot distinguish a retry.
Retry based on error type
Not every error deserves a retry. A temporary network break, overload, timeout or short external-service failure can be retried. Invalid data, denied authorisation, a corrupt file or a business-rule violation will not become valid through repetition. A system that does not distinguish them both fills its queue and hides actual problems.
Exponential backoff is common for transient failures: retry after a short wait, then at increasing intervals. Adding jitter prevents many jobs from retrying together and overloading a dependency again. Maximum attempt count and total duration also need limits. Endless retries are a quiet loop that consumes a permanent failure.
The worker log or job record should show attempt count, last error category and next retry time. Support can then answer “why is this work still pending?” with a concrete state instead of a generic error message.
Do not separate database change from message publication
A common mistake writes a database record first and publishes a message in a separate step. If the application crashes between the two, the record exists but the event is never published. In the reverse case, a message is published but the record rolls back; consumers act on an object that never existed.
The outbox pattern closes this gap. A business change and the event to publish are stored in the same database transaction. A separate publisher reads outbox records, delivers them to the queue and marks successful delivery. Because the publisher can also run again, the message keeps its unique identifier and consumers remain idempotent.
This does not create exactly-once magic. It makes lost events and duplicate delivery explicit and manageable.
Keep ordering assumptions narrow
Many job flows have no global message order. Two events for the same resource can reach different partitions, run on parallel workers or arrive in reverse order because of retry. Assuming that OrderCreated always arrives before OrderCancelled is unsafe unless the system preserves it explicitly.
Where ordering is required, preserve it per business key rather than across the entire system. Routing events for the same order or document to one partition is one option. In other cases, a consumer can check an event version and ignore an outdated update. These choices depend on domain meaning; unnecessary ordering limits capacity.
Treat the dead-letter queue as a work surface
Messages that exceed maximum attempts or encounter permanent failure can move to a dead-letter queue (DLQ). This queue is not a signal that the problem is gone. It is a failure work surface that needs human or automated review.
For every failed message, retain a unique identifier, error category, first-seen time, attempt count and enough business context to reproduce the issue without carrying sensitive content or credentials. Can operations safely replay the message, or must data be corrected first? That path needs definition before an incident.
Resending an entire DLQ with one click is risky. A persistent schema mistake or authorisation issue can cause thousands of messages to fail again. Filtered, controlled and observable replay is safer.
Measure user impact, not only queue depth
An empty queue does not prove a workflow is healthy. A required email may never have been sent, a search index may not be current or a file scan may have stopped halfway. Track completion time, failure rate, age of the oldest pending job, retry rate and DLQ depth.
For critical work, design a user-visible status too: “processing”, “completed”, “will retry” or “sent to support” reduce uncertainty. Avoid showing definite success when the job has not actually completed.
Reliable background jobs are not about eliminating failure. They are about recovering from duplicate delivery, partial outages and dependency errors without creating incorrect or irreversible side effects. Idempotency, selective retry, an outbox and an operable failure queue are the foundation of that resilience.