Add priority queue - #93
Merged
Merged
Conversation
fmcc
approved these changes
Aug 20, 2026
django-q 1.3.9 has been unmaintained since 2021; django-q2 is the maintained drop-in fork (same django_q import path and INSTALLED_APPS entry). Ships additional django_q migrations, applied by the existing 'manage.py migrate' step. Prerequisite for routing tasks to multiple queues via ALT_CLUSTERS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /customers/{customer}/queue/priority accepts the same payload as
the standard endpoint but routes processing through a VIP lane at both
levels, mirroring the DLCS API's own priority queue:
- locally, tasks are enqueued to a dedicated queue/cluster (named by
PRIORITY_QUEUE_NAME) with its own worker pool, so they are not
blocked by a standard-queue backlog. Each engine container runs a
second qcluster process for it; no new instances are required.
- at ingest, batches for priority collections are POSTed to the DLCS
/queue/priority endpoint.
If PRIORITY_QUEUE_NAME is unset the feature is inert: priority
submissions use the standard local queue but still ingest via the DLCS
priority endpoint. See docs/priority-queue.md for the full design.
Also repairs the integration test stack, which referenced a
Dockerfile.CompositeHandler left over from the protagonist monorepo
import and was unrunnable: builds the root Dockerfile instead, supplies
the required env vars, gates services on a postgres healthcheck, and
bumps pytest-docker for Docker Compose v2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pinning was to resolve a specific issue related to django-q, now that we are on django-q2 it no longer applies
donaldgray
force-pushed
the
feature/priority
branch
from
August 24, 2026 14:49
5631c19 to
aa1e241
Compare
Member
Author
|
Note - force-push was to update after merging #94. Also removed pinning of setuptools, detailed on that PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new endpoint:
POST /customers/{customer}/queue/priority(mimicsPOST /customers/{customer}/queuebut skips queue and ends up in priority DLCS queue).A priority submission is expedited both in terms of processing and ingest:
PRIORITY_QUEUE_NAME) with its own small worker pool, so they are never blocked by a standard-queue backlog. Each engine container runs a secondqclusterprocess for it (entrypoint-worker.sh);. This means no new instances or services running as I expect use to be rare and want to avoid infra overhead./customers/{customer}/queue/priorityendpoint, so they also skip any DLCS-side backlog.If
PRIORITY_QUEUE_NAMEis unset the behaviour is as beforeCommits
django-q==1.3.9→django-q2==1.10.0, the maintained drop-in fork. Prerequisite for multi-queue routing (ALT_CLUSTERS), also seems like sensible update regardless.Collection.priorityfield + migration, cluster routing in the view, DLCS priority ingest, worker entrypoint. Also repaired the integration test stack, which referenced an oldDockerfile.CompositeHandlerleft over from when this was part of the protagonist.Configuration
New configuration:
PRIORITY_QUEUE_NAMEPRIORITY_WORKER_COUNT1Infrastructure
New priority SQS queue (same settings as the standard one) and set the above env vars on the API and engine services.
Migrations
New migration to add
priorityfield.