Jobs that don’t fall over: designing idempotent queue workers

A customer pays an invoice. The worker charges the card, and the process dies before the job is deleted. Laravel retries it. The second run charges the card again. The useful question is not "will this run twice?" It will. The question is what the invoice looks like afterwards.

An idempotent job can run more than once and still leave the data in the same correct state. Charging a card is not idempotent. Recording that this attempt already paid the invoice, and refusing to charge when that record exists, is.

01 The attempt is a value you create at dispatch

The retry must be able to tell "this is the same payment again" from "the customer clicked pay a second time". Those are different events, and they need different keys. Generate the key when you dispatch. A UUID is enough. The queue stores constructor arguments, so a retry receives the same string. A second click generates a new one.

// app/Http/Controllers/InvoicePaymentController.php
use App\Jobs\RecordInvoicePayment;
use App\Models\Invoice;
use Illuminate\Http\RedirectResponse;
use Illuminate\Support\Str;

class InvoicePaymentController
{
    public function store(Invoice $invoice): RedirectResponse
    {
        RecordInvoicePayment::dispatch($invoice->id, (string) Str::uuid());

        return back();
    }
}

$attempt is not magic, and it is not $this->attempts(), which is how many times the worker has picked the job up. The UUID is the payment. The worker's attempt count is the retry budget. Keep them apart.

02 Remember that this attempt already ran

Store the UUID on the invoice when the payment is recorded. If handle() sees that UUID, or sees the invoice already paid, it returns. The second run does not charge again.

// app/Jobs/RecordInvoicePayment.php
namespace App\Jobs;

use App\Models\Invoice;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;

class RecordInvoicePayment implements ShouldQueue
{
    use Queueable;

    public function __construct(public int $invoiceId, public string $attempt)
    {
    }

    public function handle(): void
    {
        $invoice = Invoice::query()->findOrFail($this->invoiceId);

        if ($invoice->payment_attempt === $this->attempt || $invoice->status === 'paid') {
            return;
        }

        $this->charge($invoice);

        $invoice->forceFill([
            'status' => 'paid',
            'payment_attempt' => $this->attempt,
        ])->save();
    }
}

forceFill() writes payment_attempt even when it is not fillable. That column should not be mass-assignable from a form. The job is the only writer.

The gap is the crash between charge() and save(). The card was charged, the column is still null, and the retry charges again. The early return cannot see a charge that was never recorded. Pass $this->attempt to the gateway as its idempotency key, so the gateway treats the retry as the same charge. The column is your record. The gateway key is the gateway's. You need both.

03 The same idea, with a cache key

A column is the right store when the fact belongs on the invoice. A cache key is the right store when you only need "the first worker to arrive wins", and you do not want a migration. Cache::add() writes the key only if it is absent, and it returns false when another worker already wrote it. The check and the write are one operation.

// app/Jobs/RecordInvoicePayment.php
use App\Models\Invoice;
use Illuminate\Support\Facades\Cache;
use Throwable;

class RecordInvoicePayment
{
    public function handle(): void
    {
        $key = 'invoice:'.$this->invoiceId.':payment:'.$this->attempt;

        if (! Cache::add($key, true, now()->addDay())) {
            return;
        }

        try {
            $invoice = Invoice::query()->findOrFail($this->invoiceId);
            $this->charge($invoice);
            $invoice->forceFill([
                'status' => 'paid',
                'payment_attempt' => $this->attempt,
            ])->save();
        } catch (Throwable $exception) {
            Cache::forget($key);

            throw $exception;
        }
    }
}

Forget the key when the work throws. Otherwise a retry finds the key, returns, and the payment never happens. That is the failure mode of claiming the key before the side effect. The model column does not have this trap, because you only write it after the charge. Use the column when you can. Use Cache::add() when the claim has to be atomic across workers and you remember to release it on failure.

04 Stop two copies running together

The early return helps after a run has finished. It does not help when two workers enter handle() before either has saved. WithoutOverlapping takes a cache lock so the second job waits.

By default the middleware releases a blocked job straight back onto the queue. Each time a worker picks that job up, it counts as an attempt. A tight loop burns $tries, and the job lands in failed_jobs without ever getting the lock. releaseAfter() is the pause. expireAfter() is how long the lock may be held if the worker dies holding it. Set the expiry longer than the job can legitimately run.

// app/Jobs/RecordInvoicePayment.php
use Illuminate\Queue\Middleware\WithoutOverlapping;

class RecordInvoicePayment implements ShouldQueue
{
    use Queueable;

    public int $tries = 10;

    public function __construct(public int $invoiceId, public string $attempt)
    {
    }

    /**
     * @return array<int, WithoutOverlapping>
     */
    public function middleware(): array
    {
        return [
            (new WithoutOverlapping('invoice:'.$this->invoiceId))
                ->shared()
                ->releaseAfter(15)
                ->expireAfter(180),
        ];
    }
}

shared() stops Laravel prefixing the lock with the class name. A refund job that uses the same string cannot run beside this one. Without shared(), each class has its own lock, and they overlap on the same invoice.

dontRelease() is the other choice. It sets the release delay to null, and the middleware then drops the blocked job instead of putting it back. That copy does not consume further attempts, and it does not appear in failed_jobs. Use it when a second dispatch is pure waste and the first job will finish the work. Do not use it for a retry of a payment that has not been recorded yet: if the holder crashes, the dropped copy is the one that would have finished the job.

$tries has to cover the waits. Ten tries and a fifteen-second release is a couple of minutes of waiting, then a real failure. If the only reason the job is released is the lock, and the work itself is quick, that budget is enough. If the gateway is slow, raise the expiry before you raise the tries.

05 Edge cases

findOrFail() fails the job when the invoice was deleted. That is what you want for a payment. A notification job might prefer to return, which is $deleteWhenMissingModels on a job that type-hints the model. This job takes an id on purpose, so the choice is visible in handle().

The lock ends when handle() returns, including when it returns early. A retry after that should find payment_attempt already set. The lock is not a record of success. Stop the same job running twice is the lock on its own. This article is the record the lock cannot keep.

A second click is a new UUID. The early return on status === 'paid' is what stops that click charging a paid invoice. The UUID check on its own would allow it, because the new UUID does not match payment_attempt. You want both conditions.

Do not put the card number in the constructor. The payload is what Horizon and failed_jobs store. The id and the attempt string are enough for the worker to load the rest.

06 Testing it

Call handle() twice with the same attempt. One invoice is paid, and the stored attempt is that UUID. The test does not need a worker.

// tests/Feature/RecordInvoicePaymentTest.php
use App\Jobs\RecordInvoicePayment;
use App\Models\Invoice;

it('records a payment once when the job runs twice', function () {
    $invoice = Invoice::factory()->create(['status' => 'open']);
    $job = new RecordInvoicePayment($invoice->id, 'attempt-1');

    $job->handle();
    $job->handle();

    $fresh = $invoice->fresh();

    expect(Invoice::query()->where('status', 'paid')->count())->toBe(1)
        ->and($fresh->payment_attempt)->toBe('attempt-1');
});

That test misses the crash between charge and save. Fake the gateway, throw after a successful charge on the first call, forget nothing, and assert the second call does not charge again only because the gateway saw attempt-1 before. If you cannot fake the gateway, the column test above is still the one that stops a casual second save().

07 Checklist

• Create the attempt UUID at dispatch with Str::uuid(), and pass it into the job. A retry keeps it. A second click does not.
• Return early when payment_attempt matches, or the invoice is already paid.
• Give the gateway that same UUID. The column cannot protect a charge that was not recorded.
• If you claim the work with Cache::add(), forget the key when the work throws.
• releaseAfter() paces a blocked job. Each pickup still counts toward $tries. dontRelease() drops the blocked copy instead.
• shared() is how a second job class joins the same lock. The lock still is not a record of success.

08 Further reading

• Stop the same job running twice is the middleware on its own.
• Dispatch after commit is the other race: the job running before the invoice exists.
• Unique jobs and WithoutOverlapping compares the lock with ShouldBeUnique.
• Safe to run twice is the same idea in the series.
• Queues covers WithoutOverlapping, releaseAfter(), and dontRelease().