Background Jobs: Designing Work That Survives a Server Restart

Returning a response and starting a promise in the background can look like a queue. So can scheduling a timer in a web process. Both may work during development, then lose work when a server restarts or a hosting platform stops an idle instance. Use background processing when the operation should outlive the request that created it. Report generation, imports, scheduled publishing, and notifications often fit this category. The important property is durability, not whether the work happens on another thread. ## Save the job before promising success Persist the work in a database or durable queue before telling the caller it has been accepted. Store a job identifier, input reference, status, attempt count, and relevant scheduling information. If creating the job accompanies a business change in the same database, consider recording both in one transaction. Otherwise the application can save the business record and fail to enqueue its required follow-up. Keep large files outside the queue payload. Pass a durable storage reference and verify the worker has permission to read it. A temporary local path may disappear when the job runs on another instance. ## Design for interruption and duplicate delivery A worker needs a way to claim work without competing workers all executing it. A lease or visibility timeout allows another worker to recover a job after the first disappears. Long-running work may need heartbeats to keep that lease valid. The recovery path means execution can happen more than once. Make business effects idempotent, checkpoint large operations, and distinguish retryable failures from invalid input that will never succeed. Put limits on attempts and backoff. Failed jobs should become visible for investigation instead of retrying indefinitely. Cancellation also needs a contract: stopping future work does not necessarily undo effects already committed. ## Give schedules an explicit clock For recurring jobs, record when each occurrence is due and how duplicate execution is prevented. Define whether a missed run should be caught up, skipped, or combined with a later run. Store timestamps consistently and show the user's intended timezone. Daylight-saving changes can make a local time ambiguous or nonexistent. A documented UTC schedule is simpler, but it should be translated clearly in the interface. Monitor the age of waiting jobs, failure rate, duration, and worker availability. An API health check cannot tell you whether a report has been stuck for three hours. A durable queue does not eliminate failure. It changes failure from silently forgotten work into recorded work that can be retried, inspected, or deliberately cancelled. ## Further reading - [Amazon SQS: Visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) - [BullMQ: Idempotent jobs](https://docs.bullmq.io/patterns/idempotent-jobs)