RabbitMQ library processing queues
RabbitMQ is provisioned in the dedicated Kubernetes namespace prochain-rabbitmq.
The independent Argo CD application/project of the same name reconciles the
rabbitmq/ directory in aidj-gitops.
It runs RabbitMQ 4.3.5 with a pinned image, one StatefulSet replica and a retained
20 GiB hostpath PVC. The current single-node cluster provides persistence across
pod restarts, but no node-level high availability.
Connection
Trusted dispatchers and workers in prochain-aidj connect to
rabbitmq.prochain-rabbitmq.svc.cluster.local:5672, using virtual host
prochain_aidj. The prochain-aidj/aidj-rabbitmq Kubernetes Secret contains the
worker amqp-url and separate connection fields. The worker account has no
administration rights or permission to create arbitrary queues.
There is no public ingress or NodePort. NetworkPolicy allows AMQP from the AI-DJ namespace and management access from the broker namespace. Traffic is internal TCP; TLS is not configured. Credentials must stay in backend Secret mounts and must never be returned to the browser or embedded in job messages.
Stages
| Purpose | Exchange / routing key | Queue |
|---|---|---|
| Native audio/MIR | aidj.library.jobs / native | aidj.library.native |
| Neural inference | aidj.library.jobs / neural | aidj.library.neural |
| Qdrant indexing | aidj.library.jobs / index | aidj.library.index |
| Delayed retry | aidj.library.retry / stage name | aidj.library.<stage>.retry |
| Failed jobs | aidj.library.failed / any key | aidj.library.failed |
All queues are durable quorum queues, with one member on this installation. Retry queues delay messages for 30 seconds, then route back to the same stage. Main queues have a broker redelivery limit of five and dead-letter failures with at-least-once transfer. Application retries still need a bounded attempt counter: publishing a fresh retry message resets the broker redelivery history.
Main queues reject additional publishes above 256 MiB. Messages are limited to 1 MiB and should contain references, never audio. Failed messages remain for operator review. Workers must use publisher confirms, mandatory routing, persistent messages, manual acknowledgements and idempotent result publication.
Implementation boundary
The Rust library workers are connected to RabbitMQ. The library section can submit tenant-scoped analysis jobs, display progress and retry failed stages. See Music library and background processing for the implemented job ledger, worker recovery and user-facing pages.
MongoDB remains authoritative for ownership, job state and analysis versions; Qdrant remains the derived similarity index. Full-folder ingestion remains paused until the Windows source is available in the cluster.
Every task should identify its user, track, audio hash, analysis stage and algorithm/model/configuration versions. Workers must resolve authorized storage from that identity instead of trusting arbitrary paths in messages. An ACK is sent after results are durably published. Duplicate delivery must be harmless.
Operations
kubectl -n prochain-rabbitmq exec rabbitmq-0 -- rabbitmq-diagnostics -q check_local_alarms
kubectl -n prochain-rabbitmq exec rabbitmq-0 -- rabbitmqctl list_queues -p prochain_aidj name messages consumers
kubectl -n prochain-rabbitmq port-forward service/rabbitmq 15672:15672
The management UI is then available at http://localhost:15672. Administrator
credentials are in prochain-rabbitmq/rabbitmq-admin; this operator interface is
separate from AI-DJ's Keycloak login.
The bootstrap helper tools/configure-rabbitmq.py creates Secrets outside Git and
preserves existing credentials on rerun. Boot definitions include password hashes
and are also stored in a Secret. Back up the admin, worker and bootstrap Secrets
alongside the broker data. Secret edits alone do not rotate existing accounts.
After changing reviewed topology, rerun the helper and restart the broker to
import it. Deleting users/queues or changing queue types requires an explicit
migration; definition imports are additive.
The pod uses a 2 GiB memory limit, 1 GiB broker memory watermark and a 2 GiB free-disk alarm. A two-hour acknowledgement timeout accommodates expensive tasks; application heartbeats and job deadlines remain necessary. Hostpath requested storage is not a filesystem quota or a backup. Do not delete the PVC during routine recovery. Adding brokers requires configuring clustering and quorum membership, not merely increasing the replica count.
See the GitOps broker runbook, RabbitMQ boot definitions and quorum queue documentation.