Advanced Topics
Advanced Topics
Production tuning for Simple Durable Jobs. These pages cover the knobs you reach for once jobs are flowing and you’re hardening for scale and failure.
The honest contract: at-least-once execution, the idempotency requirement, backend support tiers, crash-recovery tuning, and how the guarantees are verified.
Deployment topology, fleet-safe migrations, graceful drain, health probes, sdj CLI, throughput tuning, metrics, DLQ, and retention runbooks.
How the worker reclaims jobs whose owners crashed or disappeared, and the two options (WithStaleLockInterval, WithStaleLockAge) that tune it.
The four connection-pool presets, what each optimizes for, and how to hand-roll a custom PoolConfig with the option functions.
Why the worker has two retry knobs (WithStorageRetry, WithDequeueRetry), what each protects against, and tuning for slow storage.
Configure worker-default and per-handler retry delays with BackoffPolicy, ExponentialBackoff, and jitter.
List, count, inspect, and requeue jobs that exhausted retries, with explicit DLQ metadata and retention caveats.
Use Batch entries and Queue.EnqueueBatch to insert many producer jobs with one storage batch operation.
Use GetVersion markers to evolve Call, fan-out, and signal-wait sequences while in-flight runs keep their recorded branch.
Pause workflows with Sleep and SleepUntil without holding a worker slot, with checkpointed deadlines that survive restart.
Opt in to bounded worker-side batch dequeue to claim multiple due jobs per poll when storage supports it.
Commit a phase checkpoint in the same GORM transaction as the business effect it protects.
Persist jobs inside your own GORM transaction so business rows and jobs commit or roll back together.
Encrypt or transform job arguments, results, checkpoints, signal payloads, and error text (last_error / dead_letter_reason) at the GORM storage boundary.
Add optional per-action authorization for mutating dashboard RPCs while keeping identity extraction in your own middleware.
Wrap handler execution with interceptors for logging, auth context, panic classification, and error mapping.
Cap parallelism fleet-wide and per-key (e.g. at most N jobs per customer) with DB-coordinated concurrency slots.
Throttle execution to a requests-per-window ceiling with per-queue token buckets and an optional fleet-wide limiter — without burning retry attempts.
Automatically prune terminal jobs by per-status age window with a background retention worker, so the jobs table stays bounded.
Export Prometheus / OpenTelemetry metrics — queue depth, throughput, wait/run latency, attempts, failures, retries — via an opt-in instrument.
Cancel a running job fleet-wide via a first-class verb across the facade, Connect-RPC, and the dashboard (cooperative cancellation).
The jobstest package: isolated SQLite fixtures and rivertest-style enqueue assertions for testing job producers and consumers.