River job stuck at running

[didntpay]

We recently noticed that a lot of our river job(around 5000) is stuck at a state of running. We have a global timeout of 1 hour and thought that the rescue is suppose to resolve this? These are across different job queues.

River job stats

SELECT
  COUNT(*) AS stuck_count,
  COUNT(DISTINCT kind) AS distinct_kinds,
  COUNT(DISTINCT queue) AS distinct_queues,
  MIN(created_at) AS oldest_created_at,
  MAX(created_at) AS newest_created_at,
  NOW() - MIN(created_at) AS oldest_age,
  NOW() - MAX(created_at) AS newest_age,
  AVG(NOW() - created_at) AS avg_age,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM NOW() - created_at)) * INTERVAL '1 second' AS median_age,
  MIN(attempt) AS min_attempts,
  MAX(attempt) AS max_attempts,
  AVG(attempt)::numeric(10,2) AS avg_attempts,
  COUNT(*) FILTER (WHERE attempted_at < NOW() - INTERVAL '1 hour') AS not_touched_in_1h,
  COUNT(*) FILTER (WHERE attempted_at < NOW() - INTERVAL '6 hours') AS not_touched_in_6h
FROM river_jobs.river_job
WHERE state = 'running'
  AND created_at < NOW() - INTERVAL '1 day';

[
	{
		"stuck_count": 5133,
		"distinct_kinds": 42,
		"distinct_queues": 7,
		"oldest_created_at": "2026-02-24 22:08:09.618834+00",
		"newest_created_at": "2026-05-25 19:10:35.474235+00",
		"oldest_age": "91 days 00:29:38.338464",
		"newest_age": "1 day 03:27:12.483063",
		"avg_age": "40 days 29:02:53.066961",
		"median_age": "1155:35:03.048939",
		"min_attempts": 1,
		"max_attempts": 4,
		"avg_attempts": 1.08,
		"not_touched_in_1h": 5133,
		"not_touched_in_6h": 5133
	}
]

River version

github.com/riverqueue/river v0.31.0
github.com/riverqueue/river/rivertype v0.31.0

Let me know if any other information is helpful

[brandur]

These are likely what we refer to as "stuck" jobs, which are jobs that don't respond to context timeout. This can occur when, for example, you have a wait on something like a channel in a job, but without also waiting on ctx.Done() in a select.

If you look for num_jobs_stuck in your logs, you should be able to see this number (requires info-level logging).

[peter941221]

Opened a docs follow-up here in case it's useful: #1264

I kept it scoped to the cancellation semantics that came up in this issue:

1. Expand the Worker.Work contract to call out blocking on channels, timers, and network operations explicitly.
2. Add the same guidance to the package docs and README.
3. Point people at num_jobs_stuck as the first diagnostic when jobs appear stuck in running.
4. Add a short note to the graceful shutdown example so the ctx.Done() expectation shows up in an example that's already about stop behavior.

No runtime behavior changes in this one, just trying to make the existing semantics easier to discover.

[didntpay]

I see, thanks team. I still have one confusion, so when we cancel the worker's context in hard stop mode, does that also set the job to cancelled? Are we saying that because we have a timer, channel, etc that does not respect the ctx.Done() from worker, in hard stop mode we are unable to set it to cancelled?

[brandur]

@didntpay Yeah, currently the idea is that you should very much try to have your jobs behave well and respond to context cancellation. A feature to hard error any jobs that didn't respond to context cancellation might be possible, but it'd complicate the stop process in ways that might not be super desirable.

[didntpay]

Make sense, in the case that our job does not respond to the ctx.Done(), would the rescuer eventually come in and clean those up? Trying to see what the safest strategy is to clean up our 5k stuck job currently in the db.

[brandur]

Yeah, the rescuer will clean those up after RescueStuckJobsAfter has elapsed (default 1 hour). I also have a patch open in #1289 that would try to clean these up before River stops.

[didntpay]

Ah, I think one of our problem is also the rescuer not cleaning those job after 1 or 2 hour. We did not specify a special timeout for those.