Benchmarks

Methodology

These are single-machine measurements for relative guidance on this library’s own storage backends. They are not a performance-lab cross-vendor comparison, and they should not be read as River or Asynq comparison numbers because those systems were not benchmarked under identical conditions.

Run date: June 6, 2026. Branch: feat/adoption-wave at 90e2907. Command: BENCH_COUNT=3 TEST_DATABASE_URL='postgres://jobs:jobs@localhost:15433/jobs_test?sslmode=disable' TEST_MYSQL_URL='root:jobs@tcp(localhost:13307)/jobs_test?parseTime=true' scripts/bench.sh sqlite postgres mysql.

Host: AMD EPYC 9R14 96-Core Processor, nproc=192, Linux 6.8.0-124-generic x86_64 GNU/Linux, go version go1.25.11 linux/amd64.

Database versions: SQLite 3.50.2; PostgreSQL 16.11 on x86_64-pc-linux-musl with max_connections=100; MySQL 8.0.42 with max_connections=151.

Database setup: SQLite used a temp file through jobs.SafeSQLiteDSN, which enables WAL, busy_timeout=5000, and txlock=immediate. Postgres and MySQL used the repository docker-compose.yml containers on task-provided host ports 15433 and 13307. The compose file does not override server max_connections; the measured values above are from SQL queries. The benchmark client pool was capped at 16 open connections and 8 idle connections for every backend so a 192-thread host would not swamp the shared dev containers.

Benchmark shapes: BenchmarkEnqueue enqueues one job per operation through the public queue API. BenchmarkEnqueueBatch enqueues 100 jobs per operation and reports jobs/sec. BenchmarkEndToEnd drains a saturated queue of 1000 pre-enqueued no-op jobs per benchmark operation, starts one worker with concurrency 16 and WithPollInterval(50ms), and reports jobs/sec from first handler start to last completion. With the explicit single-row dequeue path (WithDequeueBatchSize(1)), each claim asks storage for one job; the work-conserving dispatcher may still repeat those single-row claims while capacity remains. BenchmarkEndToEndBatchDequeue uses the same saturated-queue drain shape plus WithDequeueBatchSize(10). BenchmarkWorkflowSteps runs b.N workflow jobs, each doing 5 checkpointed jobs.Call steps, and reports steps/sec. BenchmarkDispatchLatency starts an already-running idle worker first, then enqueues one job at a time and records enqueue-to-handler-start latency before proceeding to the next sample.

The worker default poll interval is 100ms. These benchmarks use 50ms because that is the worker poll-interval floor: WithPollInterval clamps any positive value below 50ms up to that floor (and ignores a non-positive value, keeping the existing one). Idle-worker dispatch latency is poll-bound: p50 is roughly half to one poll interval on this host, and the tail is roughly one poll interval plus scheduling noise. It is tunable with WithPollInterval, but lower polling means more database load. Dispatch-latency p95 and p99 are based on about 25 samples per run, so treat the tail values as a smoke signal rather than a statistically deep latency distribution. SQLite has a single-writer ceiling even with WAL, so use Postgres or MySQL for heavy multi-process write concurrency.

The CI benchmark matrix covers SQLite and Postgres. MySQL numbers below are measured locally only against the repository’s MySQL container on localhost:13307; there is no checked-in CI MySQL baseline.

Throughput Knobs

The explicit single-row dequeue path (WithDequeueBatchSize(1)) is the conservative claim mode: each storage request claims one job. Pair it with a slower WithPollInterval(...) when you want to approximate the older slow-poll behavior during rollback.

The worker default is WithDequeueBatchSize(50), capped at the worker’s free concurrency slots — so at the default concurrency of 10, the effective per-poll claim is 10. The tables below were measured when the default was 10 and have not been re-run since it was raised, so read them as the batch-size-10 shape. At batch size 10, a 50ms poll can claim roughly 10 / 0.050s = 200 jobs/sec before storage and handler work dominate. That arithmetic explains why the default EndToEnd row is similar across backends, while enqueue, batch enqueue, workflow, and SQLite’s batched-drain shape are not near-identical. Raising concurrency is what unlocks the larger default batch.

SQLite Results

PostgreSQL Results

MySQL Results

MySQL is included here as a local measurement only. CI currently publishes SQLite and Postgres artifacts and compares those two backends against checked-in baselines.

Reproduce It

Start the external backends, then run the benchmark suite:

POSTGRES_HOST_PORT=15433 MYSQL_HOST_PORT=13307 docker compose -p sdj-test up -d --wait postgres mysql
BENCH_COUNT=3 \
  TEST_DATABASE_URL='postgres://jobs:jobs@localhost:15433/jobs_test?sslmode=disable' \
  TEST_MYSQL_URL='root:jobs@tcp(localhost:13307)/jobs_test?parseTime=true' \
  scripts/bench.sh sqlite postgres mysql

The script isolates each backend leg by blanking unrelated DSN environment variables before invoking go test, so the command above genuinely attributes SQLite, Postgres, and MySQL results to the intended backend even when both external DSNs are exported in the parent shell. Raw local output is written to bench-results/<backend>.txt; CI uploads the same raw files as artifacts and reports WARN-only benchstat deltas against the checked-in SQLite and Postgres baselines. The runner default is BENCH_COUNT=3, matching CI and the checked-in baselines.