Scaling Background Queues Beyond 10,000 Jobs Per Second

At a hundred jobs per second, a queue is a convenience. At ten thousand, it is a distributed system with all the properties that implies — and most of the advice written for the first case actively misleads you in the second.

This article is about the architecture. If you are tuning a Laravel application that is not yet at this scale, the driver choice, payload size, and worker sizing covered in Laravel queue throughput will get you considerably further with less disruption.

The constraint moves#

At low throughput, the bottleneck is per-job overhead — framework boot, payload deserialisation, container resolution. You fix it by making each job cheaper.

Past a few thousand jobs per second, three different constraints dominate, and they are all shared resources:

The broker's single-key hotspot. A Redis list is one key on one node. Every push and pop is serialised through one CPU on one machine. Redis will do a few hundred thousand simple operations per second, but your job pipeline is not one operation — it is a push, a pop, a reserve, a heartbeat, and a delete.

Downstream capacity. Ten thousand jobs per second each making one database write is ten thousand writes per second. The queue is now a very efficient mechanism for overwhelming something else.

Coordination cost. More workers means more polling, more lock contention, more connections, and more heartbeats. Beyond a point, adding workers reduces throughput.

Partition, do not enlarge#

The instinct is to make the queue faster. The correct move is to make it plural.

$shard = crc32((string) $tenantId) % 16;

ProcessEvent::dispatch($event)->onQueue("events.{$shard}");

Sixteen independent queues, each with its own key, potentially on its own Redis node. Contention drops by roughly the shard count, and throughput scales close to linearly because the shards do not talk to each other.

The choice of partition key is the design decision that matters. Partitioning by tenant preserves per-tenant ordering, which is usually what you actually need, while allowing unrelated tenants to proceed in parallel. Partitioning randomly gives the best balance and no ordering at all.

The failure mode is a hot partition. If one tenant produces 40% of your events, one shard carries 40% of the load and you have gained less than you hoped. Monitor per-shard depth, and be prepared to give large tenants dedicated shards.

Choose the broker for the guarantee you need#

Brokers differ less in raw speed than in what they promise, and the promise determines the cost.

The two questions that narrow this quickly: do you need ordering, and can you tolerate duplicates? Almost every expensive design decision follows from those answers.

Kafka is genuinely different in kind rather than degree. It is a distributed log, not a queue — consumers track their own offset, messages are not deleted on consumption, and replay is a first-class operation. If you need to reprocess a week of events after fixing a bug, that capability is worth the operational weight. If you never will, it is weight you are carrying for nothing.

Backpressure is not optional#

An unbounded queue does not fail. It absorbs everything, grows, and converts a throughput problem into a latency problem that nobody notices until jobs are hours old.

Bound it, and make producers feel the limit:

public function dispatchEvent(Event $event): void
{
    $depth = Redis::llen("queues:events.{$this->shard($event)}");

    if ($depth > self::MAX_DEPTH) {
        Metrics::counter('events_shed_total')->increment();

        if ($event->priority === Priority::Low) {
            return;                        // shed load deliberately
        }

        throw new QueueSaturated;          // propagate to the caller
    }

    ProcessEvent::dispatch($event)->onQueue("events.{$this->shard($event)}");
}

Two behaviours, chosen per event class. Low-value work — analytics pings, cache warming, recommendation refreshes — is shed. High-value work propagates the failure so the caller can retry or degrade.

Deciding in advance which of your job types are sheddable is the single most valuable half-hour of planning in a high-throughput system. Under load you will shed something; the only question is whether you chose it.

Aggregate before you enqueue#

The cheapest job is the one never dispatched. At high volume, many jobs are increments and appends that can be combined.

// Instead of dispatching a job per pageview
Redis::hincrby("pageviews:pending:{$bucket}", "post:{$postId}", 1);

Then flush periodically:

class FlushPageviews implements ShouldQueue
{
    public function handle(): void
    {
        $bucket = now()->subMinute()->format('YmdHi');
        $key = "pageviews:pending:{$bucket}";

        $counts = Redis::hgetall($key);

        if (empty($counts)) {
            return;
        }

        DB::transaction(function () use ($counts) {
            foreach (array_chunk($counts, 500, true) as $chunk) {
                $this->bulkIncrement($chunk);
            }
        });

        Redis::del($key);
    }
}

Ten thousand pageviews per second become one job per minute writing a few hundred aggregated rows. The queue never sees the volume at all.

This works whenever the operation is commutative and you can tolerate a delay — counters, metrics, search index updates, denormalised totals. It does not work for anything a user is waiting on.

Delivery guarantees, honestly#

Systems advertise at-least-once, at-most-once, or exactly-once. Only the first two are real.

At-least-once is what almost every broker gives you. A job may run more than once — after a worker crash, after a visibility timeout expiry, after a network partition. This is fine, and it puts the burden on your job being idempotent.

At-most-once means acknowledging before processing. Faster, and you lose jobs on crash. Appropriate for telemetry, never for anything with a side effect.

Exactly-once does not exist across a network boundary in the general case. What Kafka calls exactly-once is at-least-once delivery combined with transactional offset commits within Kafka — genuinely useful, and it does not extend to the third-party API your consumer calls.

The practical conclusion is unavoidable: make your jobs idempotent. Not as a nicety, but because the alternative is a system that is wrong under conditions that will certainly occur.

public function handle(): void
{
    $done = Cache::add("job:done:{$this->idempotencyKey()}", 1, now()->addDay());

    if (! $done) {
        return;                    // a previous attempt completed
    }

    try {
        $this->doWork();
    } catch (Throwable $e) {
        Cache::forget("job:done:{$this->idempotencyKey()}");
        throw $e;
    }
}

Cache::add is atomic — it sets only if the key is absent — which is what makes this safe under concurrent duplicate delivery. The same reasoning appears in more detail in designing idempotent APIs.

Protecting the downstream#

Your queue is now capable of generating more load than anything it talks to can absorb. Rate-limit at the consumer, not by reducing worker count:

public function middleware(): array
{
    return [new RateLimited('search-index')];
}

RateLimiter::for('search-index', fn () => Limit::perSecond(500));

A throttled job is released back to the queue rather than failing, so it does not consume a retry attempt. This is meaningfully better than sleeping inside the job, which occupies a worker doing nothing.

Add a circuit breaker for dependencies that fail hard:

if (Cache::get('circuit:search-index:open')) {
    $this->release(30);
    return;
}

try {
    $this->index();
    Cache::forget('circuit:search-index:failures');
} catch (ConnectionException $e) {
    if (Cache::increment('circuit:search-index:failures') > 20) {
        Cache::put('circuit:search-index:open', true, now()->addMinute());
    }

    $this->release(10);
}

Without this, a downstream outage converts your entire worker fleet into a retry storm, which delays recovery for the thing that is already struggling.

Measuring at this scale#

Queue depth is close to useless on its own — a depth of 50,000 is healthy at 10,000/s and catastrophic at 50/s.

The metric that matters is oldest-job age per queue, because it maps directly onto how long someone has been waiting and stays meaningful as throughput changes.

foreach ($this->shards() as $shard) {
    $oldest = Redis::lindex("queues:events.{$shard}", -1);
    $payload = json_decode($oldest ?: '{}', true);

    Metrics::gauge('queue_oldest_age_seconds')
        ->withLabels(['queue' => "events.{$shard}"])
        ->set(time() - ($payload['pushedAt'] ?? time()));
}

Alongside it, track arrival rate against completion rate. When arrivals exceed completions for a sustained period, you have a capacity problem that no amount of worker scaling within the current architecture will fix — and knowing that early is what buys you the time to shed load deliberately rather than discovering the limit the hard way.

What actually gets you past 10,000#

In rough order of impact:

1. Partition the queue by a key that preserves the ordering you genuinely need, and no more.
2. Aggregate before enqueueing anything commutative. Volume you never dispatch costs nothing.
3. Bound the queue and shed load deliberately, deciding in advance which job classes are expendable.
4. Make every job idempotent, because at-least-once is the only guarantee you have.
5. Rate-limit and circuit-break at the consumer to protect whatever the jobs talk to.
6. Alert on oldest-job age, and on arrival rate exceeding completion rate.

Notice that adding workers is not on the list. At this scale it is the last lever, not the first, and pulling it early usually makes things worse by increasing contention on the very resource that is already the bottleneck.

Related reading#

• Graceful Shutdown: Why Your Workers Lose Jobs on Deploy