Concurrency Caps

Per-worker vs fleet-wide limits

Concurrency(n) limits how many jobs a single worker process runs at once. If you run 5 workers with Concurrency(10), the fleet can run up to 50 jobs.

ConcurrencyCap(name, limit, opts...) adds an optional database-backed cap that is shared by every worker using a storage backend that supports concurrency slots. It is additive: storage backends without the optional capability keep the existing per-worker behavior.

Fleet-wide cap

Without a key function, the cap uses its name as the slot name. This limits the job class across the whole fleet:

w := jobs.NewWorker(q,
	jobs.WorkerQueue("emails", jobs.Concurrency(10)),
	jobs.ConcurrencyCap("email-provider", 25),
)

Even if many workers are running, at most 25 jobs admitted under email-provider hold live slots at the same time.

Per-key cap

Use CapKey when each customer, tenant, or account needs an independent cap:

w := jobs.NewWorker(q,
	jobs.WorkerQueue("imports", jobs.Concurrency(20)),
	jobs.ConcurrencyCap("tenant-imports", 2, jobs.CapKey(func(job *jobs.Job) string {
		return job.UniqueKey
	})),
)

The effective slot name is tenant-imports:<key>, so tenant A can run 2 imports while tenant B also runs 2.

Admission caveat

Concurrency slots are acquired after a job is dequeued. If the cap is full, the worker releases the job back to pending so another poll can try later.

Slots are leases. If a worker crashes, its slot is reclaimed automatically after the lease expires, using the same database-clock anchoring pattern as job locks. That makes crash recovery automatic, but it also means the effective cap can briefly exceed N for up to the lease window if a crashed worker’s job is picked up again before the old slot expires.

The built-in GORM storage implements this optional capability. Custom storage backends can opt in by implementing the worker’s optional concurrency slot methods; otherwise ConcurrencyCap is ignored and per-worker Concurrency continues to apply.

Keep the cap key bounded

When you partition a cap with CapKey, the effective slot name is Name + ":" + key. Each distinct slot name leaves a permanent admission-serialization row in the concurrency_slots table — a sentinel that the lease-expiry sweep intentionally preserves, so concurrent contenders for the same slot always have a row to lock before they count and insert.

That permanent sentinel is what makes the cap correct, but it means the key must come from a bounded, enumerable set — a fixed list of tenants, regions, or resource classes. Partitioning on a high-cardinality value (a job ID, a UUID, or arbitrary user input) creates a new permanent row per distinct value and grows concurrency_slots without bound. For “only N of these at a time” use a bounded key; do not derive it from job.ID or free-form data.

Prefer a cap key that is stable per job

Bounded is not quite the whole story — the key should also be stable for a given job across re-dispatches. A job can be dequeued twice by the same worker: the stale-lock reaper can release a lock this worker then reclaims, and the two runs briefly overlap.

If the key changed between those two dequeues, the runs hold different slot rows, and the job counts against the cap under both keys at the same time — which is not what “at most N of these at a time” is meant to mean.

It does not leak a row: the departing run hands its slot names to the surviving one, so whoever finishes last releases the union. (An aggressive pause is not a route to this either; it deletes every slot row for the job id, so the next run starts clean.) The double-counting is the real cost.

Derive the key from something immutable for the life of the job (its tenant, its queue, a field of the payload) rather than from mutable metadata a handler rewrites.