Metrics

pkg/metrics adds optional OpenTelemetry metrics for queues and workers. It is not enabled by default and is complementary to pkg/otel tracing: tracing shows individual job executions, while metrics provide aggregate time series for scraping and alerting.

Prometheus Handler

Use NewPrometheusHandler when you want a ready-to-mount /metrics endpoint. Pass the returned meter provider into Instrument.

package main

import (
	"context"
	"net/http"

	jobs "github.com/jdziat/simple-durable-jobs/v4"
	jobsmetrics "github.com/jdziat/simple-durable-jobs/v4/pkg/metrics"
	"gorm.io/driver/sqlite"
	"gorm.io/gorm"
)

func main() {
	ctx := context.Background()
	db, err := gorm.Open(sqlite.Open("jobs.db?_journal_mode=WAL&_busy_timeout=5000&_txlock=immediate"), &gorm.Config{})
	if err != nil {
		panic(err)
	}
	store := jobs.NewGormStorage(db)
	if err := store.Migrate(ctx); err != nil {
		panic(err)
	}

	q := jobs.New(store)
	handler, meterProvider, err := jobsmetrics.NewPrometheusHandler()
	if err != nil {
		panic(err)
	}
	defer meterProvider.Shutdown(ctx)

	jobsmetrics.Instrument(q, jobsmetrics.WithMeterProvider(meterProvider))
	http.Handle("/metrics", handler)
	http.ListenAndServe(":8080", nil)
}

Bring Your Own MeterProvider

If your service already owns OpenTelemetry SDK setup, pass that provider instead.

jobsmetrics.Instrument(queue, jobsmetrics.WithMeterProvider(meterProvider))

Without WithMeterProvider, instrumentation uses otel.GetMeterProvider().

Metric Catalog

The throughput, latency, depth, backlog-age, dead-letter-depth, and reclaimed metrics are wired automatically by Instrument; jobs.leases.reclaimed is registered through the same call (it hooks OnJobReclaimed) and needs no extra setup. Unlike the throughput, latency, and depth series, it carries no queue or job.type attribute — reason is its only label. reason=stale_lock is the actor side (this worker’s reaper recovered a job from a presumed-dead peer, the crash leading-indicator), while reason=ownership_audit is the victim side (this worker observed a peer reclaim a job it was still running). In a multi-process fleet the same logical reclaim can surface once per side on different workers, so alert on each reason separately and do not sum across reason values.

Queue depth, backlog age, and dead-letter depth are collected through optional storage capabilities returning plain Go maps, not UI protobufs. GormStorage supports these capabilities. Custom storage backends that do not implement them still get throughput, latency, failure, retry, and reclaimed metrics; only the unsupported storage-side gauges are skipped.

jobs.queue.saturation is worker-side because storage does not know a worker’s configured per-queue capacity. Register it per worker with InstrumentQueueSaturation(workerID, capacities, running, ...); the gauge carries worker.id. Alert with avg by (queue), not sum, so two workers at 50% saturation do not appear as a fake 100% fleet value:

avg by (queue) (jobs_queue_saturation) > 0.9

Dequeue churn (rate-limit saturation throttle)

jobs.dequeue.released and jobs.dequeue.suppressed_ticks are worker-side counters that surface dispatch churn. They are registered per worker with InstrumentWorkerDequeue(workerID, worker.DequeueReleasedByReason, worker.DequeueSuppressedTicks, ...) and both carry worker.id.

A worker dequeues (claims) a job before the final admission gates run; if a gate denies, the job is released back to pending — a “bounce” counted under jobs.dequeue.released by reason. When every configured fleet RateLimit is unkeyed, a saturated limit engages a claim-rate throttle: the worker stops claiming until the limit’s rate window rolls over, rather than bouncing the full concurrency budget every poll tick. Each suppressed tick increments jobs.dequeue.suppressed_ticks.

The healthy steady state under a saturated unkeyed fleet limit is suppressed_ticks rising while released{reason="fleet_rate"} stays low (one probe batch per rate window) — the throttle collapsing the claim/release write amplification, not a stuck worker.

For a keyed RateLimit the worker can’t pre-check a bucket before claiming (the key needs the held job), so it does not whole-fleet suppress; instead a per-key cooldown skips the expensive DB rate transaction for a bucket already denied this window. Those skipped bounces are attributed to released{reason="fleet_rate_cached"}, distinct from reason="fleet_rate" (the bounces that actually paid the DB transaction). The healthy keyed signal is fleet_rate_cached dominating fleet_rate — the cooldown eliding the contended locked transaction. Register the cooldown’s cache-size gauge with InstrumentWorkerRateSaturation(workerID, worker.DequeueRateSaturationCacheSize, ...); when jobs_dequeue_rate_saturation_cache_size sits at the configured WithRateSaturationCacheSize cap, new saturated buckets are falling back to the DB transaction (a high-cardinality-RateLimitKey signal).

Note the per-key cooldown removes the DB rate transaction but not the claim/release itself, so for keyed configs released{reason="fleet_rate_cached"} (and the underlying claim/release write volume) can stay high even when healthy — it is the DB-transaction count, not the release count, that the cooldown reduces. Trade-offs: a just-recovered limit can wait up to one rate window before this worker probes it again, and the keyed saving is per-worker and locality-dependent (a worker only caches buckets it personally denied).

# unkeyed throttle engaging
rate(jobs_dequeue_suppressed_ticks_total[5m])
# keyed cooldown eliding the DB rate tx (fleet_rate_cached should dominate fleet_rate)
rate(jobs_dequeue_released_total{reason=~"fleet_rate.*"}[5m])

See Production Operations for the full alerting guidance and CLI runbooks.