Music analysis and library search

The analyzer is an offline Rust worker that produces reusable DJ measurements before playback. MongoDB prochain_aidj owns metadata, queue state and immutable reports; originals and canonical PCM remain on disk. Analysis does not call an LLM. Track similarity is only one future ranking input: a good transition also depends on the outgoing and incoming regions, phrase alignment, tempo, harmony, vocals and intended energy progression.

Implemented layers

Layer	Current implementation	Qualification
Ingest	Recursive read-only source scan, file inventory, hash deduplication, resumable native queue, per-file errors	Native-ready does not mean ready for autonomous mixing
Native audio	Symphonia, Rubato, rustfft, ebur128; waveform, RMS, spectral/band descriptors, chroma/key, onset/BPM estimates, LUFS and true peak	BPM/key and regular phrase grid still require review
Musical timeline	Rust analyzerd, 2-second windows, local tempo evidence, multiscale spectral/rhythmic section hypotheses, local key estimates	Downbeats, variable-tempo beat grids and semantic labels still require validation
Neural evaluation	Rust ort loading standalone ONNX Runtime 1.24.4 and an explicitly supplied YAMNet model; sound-event timelines, mean embedding and optional music-trained voice/genre/mood/danceability heads	Music heads remain uncalibrated; generic instrument scores are not reliable stem/presence estimates
Search/UI	Paginated track search, review filter, source file/path search, queue status, complete detail metrics, timeline, preview and JSON export	Advanced metric filtering and professional grid editing remain future work
Candidate retrieval	Qdrant embedding-space/review/BPM/energy/key filters, Mongo freshness checks and at most ten compact candidates	Full region-to-region DJ compatibility and live planner integration remain open

Rust is the production boundary. A separate Python environment benchmarks librosa/TensorFlow and converts model assets to ONNX; it is not installed in the runtime container. Aubio and Bliss are not linked into this implementation. Adoption requires dependency/licensing review. Essentia model licenses must be checked individually. Stem separation and instrument transcription are deferred.

Scan and queue lifecycle

Mount D:\_Music\_DJ_Deck_Main read-only (WSL path /mnt/d/_Music/_DJ_Deck_Main). The initial Rust inventory recorded 6,005 audio files, approximately 65.4 GiB on 2026-09-12. This number is a snapshot, not a fixed library size. The source remains paused while a small pilot and operator-selected benchmark recordings are evaluated.

Scanning stores relative paths, size, modification time and a pipeline signature without decoding every file. States are pending, processing, native_ready, failed, blocked and missing. Rescanning unchanged signatures avoids repeating work; import hashes detect identical content. Same size and modification time can conceal a content change, so this is not continuous integrity verification. Missing source files retain imported assets and history.

New sources start paused. Explicitly enable a source, run a bounded worker, then pause it. Pausing takes effect between files. A lease and owner token recover interrupted claims; the worker renews leases during analysis. One local worker is permitted per asset folder. A low-disk check blocks new imports below 2 GiB free. Failed/blocked files can be retried independently.

analyzerd scan /mnt/music
analyzerd status SOURCE_ID
booth-cli library-resume SOURCE_ID
analyzerd work-native SOURCE_ID 3
booth-cli library-pause SOURCE_ID
analyzerd analyze TRACK_ID_1 TRACK_ID_2

Commands require the existing Mongo/asset configuration. work-native without a positive limit drains the enabled source. UI enable/pause controls change the gate; they do not launch a worker. Browser uploads enqueue the native → neural → index pipeline through RabbitMQ; selected tracks can also be queued from the analysis workspace. Folder ingestion remains separately gated. See the processing runbook. The source folder is never modified. Supported files and duration/size limits are described in the library documentation.

Report semantics

Each musical report records source PCM SHA-256, native revision, analyzer code and dependency hash, model/label-map hashes, configuration and processing duration. Identical inputs reuse an immutable cached result. Old reports remain available; the API flags a report that predates the current native revision. Model replacement requires an explicit rerun to produce a new report.

Measurements and estimates are distinct. Estimate<T> retains nullable value and confidence plus evidence/version. An absent confidence means uncalibrated; it is never silently converted into certainty. Integrated loudness is LUFS, waveform energy is RMS, and spectral bass power is not a separated bass stem.

The current classifier produces raw vocal/instrument-group scores. It does not produce a calibrated vocal ratio, vocal absence guarantee, lyrics or instrument stems. Its bass group includes bass drum. The generic YAMNet model's low vocal scores on initial real-music pilots prevent its use as a vocal-collision exclusion rule. Numerical parity with TensorFlow validates ONNX conversion, not musical accuracy. See the official YAMNet tutorial.

Transition candidates describe 8/16/32 bars. Native/classical grids retain an explicit four-beat assumption; v5 neural regions require a supported bar-length candidate and consistent downbeat intervals. Phrase/downbeat alignment is unverified. Their current score reflects level stability only, and every candidate remains unapproved. Energy rises/falls are change points, not confident build/drop/breakdown labels. Global tempo plus locally aligned peaks cannot safely represent every accelerating or live-tempo recording.

The user-selected benchmark set is Dua Lipa — Don't Start Now, Eric Prydz — Opus, deadmau5 — Strobe, Daft Punk — Aerodynamic, M.A.N.D.Y./Booka Shade — Body Language and Bicep — Glue. The available Dua Lipa recording is a VDJ JD edit, so it is tracked as that version. Operator-supplied example timestamps and expected low vocal scores are hypotheses, not labels copied into analyzer output. Segment-level listening annotations are still needed to measure recall, false positives and boundary error. See the measured benchmark results, including failed vocal and Opus structure checks.

Next quality gates

1. Annotate vocal and instrumental windows in the exact benchmark recordings, including Glue's ambiguous textures; evaluate a music-specific detector against them.
2. Evaluate beat/downbeat and variable-tempo tracking, then phrase and section boundaries, especially Opus and Strobe. Retain uncertainty and human corrections.
3. Build pairwise transition-region compatibility with tempo stretch limits, harmonic uncertainty, vocal overlap and energy intent. Keep technical scores separate from listening ratings.
4. Validate the new filtered Qdrant retrieval and connect its bounded shortlist to the DJ agent. Improve embedding quality and region-to-region compatibility before enabling autonomous selection.
5. Connect the selected transition recipe to live booth execution; then evaluate AI-vs-AI and AI-vs-human handoffs.

The platform analyzer guide gives exact model preparation, runtime and verification commands. Full-library processing stays paused until the analysis quality gate is accepted.

Analysis v2

The v2 implementation and benchmark adds a tempo prior, local tempo evidence, multiscale feature novelty, music-specific voice classification and optional genre/mood/danceability heads. A bounded summary is available through analyzerd summary TRACK_ID POSITION_SECONDS and in the UI. It includes the current section and upcoming event hypotheses without raw frames or embedding vectors. These outputs improve the evidence available to a DJ agent; they do not certify that a transition is safe or update the live planner automatically.

Analysis correctness update — 12 September 2026

rust-dj-analysis-3 addresses the review of 001 - Avicii - Levels - Radio Edit. Existing immutable reports remain historical evidence; rerunning neural analysis creates a new report and refreshes the Qdrant point. The processing profile is library-pipeline-20260912-v2. Updated code hashes separate new embeddings from old embedding spaces; reanalyse selected comparison tracks before expecting cross-track retrieval in the new space.

• Section RMS: sum duration-weighted linear powers, divide by total duration, then convert to dBFS. The final partial window is weighted by its actual duration. Section half-to-half energy changes use the same calculation. Recomputing the review's four rounded windows gives approximately −35.124 dBFS, rather than the arithmetic dB mean of −56.385. This is an aggregation correction, not a listening judgement.
• Grid identity: the report retains native and musical grid IDs, their first beats, modulo-beat phase difference, and a conflict flag above 50 ms. Sections and transition regions refer to analysis_grid_id; v5 phrase hypotheses explicitly reference their neural grid. A previously reviewed native grid takes precedence; otherwise the musical grid is an explicitly unreviewed analysis hypothesis and canonical_grid_id is null. The playback engine still uses the native grid. No unreviewed analysis grid grants execution permission. This does not add a professional downbeat editor or establish which conflicting hypothesis is right.
• Local tempo: retain robust spread and periodicity evidence, but independently flag every window more than 2.5% from the selected global tempo. These are review thresholds, not calibrated tempo-change probabilities. Passing a periodicity threshold is not equivalent to timing validity.
• Onsets: attack selection now smooths the positive RMS-difference function over 25 ms, uses a one-second adaptive threshold, gates below −60 dBFS and suppresses weaker peaks within 60 ms. Tempo estimation retains its separate detection function. Counts are provisional attack-density evidence, not a drum transcription; labelled onset accuracy remains unmeasured.
• Regions: level_stability_score replaces ambiguous score on newly generated single-track regions. Compact context also normalizes old field names. These are track_region_candidate objects, not scored overlaps between two songs. Phrase hypotheses and bass/drum entries remain unverified. Pairwise tempo/EQ/phrase/harmonic/vocal evaluation and its execution gate remain future work.
• Inference: float PCM is channel-averaged and resampled to 16 kHz. There is no repeated integer-PCM scaling. When resampled amplitude exceeds unity, only model input is divided by its peak to meet the input range; original PCM and loudness measurements are unchanged. YAMNet score and embedding dimensions are checked independently. The voice head verifies metadata order instrumental, voice, takes output index 1, and validates both softmax outputs. Context averaging remains explicitly part of the production voice pipeline.
• Peaks and display: sample peak and true peak have separate implementations. ebur128 uses 4× polyphase FIR interpolation at 48 kHz and returns at least the sample peak, so equality can be legitimate. A synthetic inter-sample-peak regression confirms different results when appropriate. Reports identify the measurement methods. The UI displays tiny nonzero evidence values with additional precision and full-scale sample fraction as a percentage.

The offline parity_dump Rust example plus tools/audio-analysis/reference_parity.py compares identical prepared waveform input, raw YAMNet scores, all 1,024 embedding dimensions, and the unsmoothed voice-head output with local TensorFlow references. A six-second Levels excerpt starting at 53 seconds passed with maximum absolute differences of 1.79×10⁻⁶, 4.30×10⁻⁶, and 8.95×10⁻⁸, respectively. This is numerical parity on an excerpt, not a vocal-presence, vocal-dominance or phrase-boundary benchmark. Do not infer negative vocal labels from it.

Reference contracts: official YAMNet tutorial and official voice-head metadata. Model assets retain their existing licensing restrictions.

recommendation_ready remains false. The next validation step is manually annotated, edition-specific vocal intervals and reviewed timing/phrase boundaries, followed by held-out region-pair evaluation. Adding a model or a high stability score cannot substitute for these checks.

Grid contract and capability coverage — analyzer v4

rust-dj-analysis-4 / queue profile library-pipeline-20260912-v3 adds explicit zero-based start_beat_index, end_beat_index, and beat_count to region candidates. Start and end seconds are direct lookups in the same referenced beat array. Regions with no end beat are skipped; the final boundary is never extrapolated from average BPM. Both native playback-grid and detected-beat arrays are retained in the full report. Compact agent context omits those arrays.

Native grid identity includes PCM identity, native revision, BPM, origin and beat timestamps. The report API marks regions invalid when the current native grid or revision differs, and removes stale mix-in/mix-out suggestions from its returned compact context. Immutable historical reports remain available. Reanalysis creates new regions and a new report; it does not rewrite old evidence.

The offline B2B renderer now checks every planned track against the current approved native grid before decoding or compiling controls, including on replay. The grid must be reviewed, finite and constant-tempo; locally jittered detected beats are not accepted as a playback grid. Plan snapshots must match the current grid identity and cues must match a grid beat within one output sample. A revision or withdrawal of approval rejects the plan. Successful preparation records playback-grids.json alongside the session artifacts. Integration tests verify that revoked and changed timing produce no mix audio, while unchanged replay remains reproducible. This is an offline timing gate, not a new live automatic transition executor or proof of vocal/phrase compatibility.

Each timeline row separates the two-second voice output interval from its audio context interval. A prediction uses up to nine centered YAMNet frames (maximum 4.815 seconds); a two-second bucket aggregates several such predictions, so the union of their model-window contexts can be wider. End padding is recorded. These fields describe model context, not human-labelled vocal starts and stops.

Section boundary_changes preserve measured level, bass-fraction and onset-density changes even when semantic event rules produce nothing. events: [] means no supported event predictions, not “no drops.” The old onset-dependent semantic thresholds have not been declared validated or tuned to force particular labels. Listening annotations remain necessary before claiming improved structure accuracy.

Coverage is different from validation

The track detail page now shows Analysis coverage and validation, using these states rather than one “Still unavailable” list:

Capability	Current qualification
Downbeats and meter	v5 neural beats/downbeats; bar-count evidence; denominator and approval unverified
Phrase boundaries	Grid and feature-boundary hypotheses; musical validation pending
Chord progression	Not implemented; key/chroma are not a chord sequence
Genre and mood	Research-model scores when configured; uncalibrated
Lyrics and language	No transcription/language pipeline
Separated stems	No stem assets generated
Instrument transcription	Category scores exist; note-level transcription does not
Human vocal labels	v5 listening annotation form and immutable reference records; predictions are not human validation

The compact report distinguishes provisional evidence for candidate selection, review-only transition planning, and blocked automatic execution. It does not silently activate LLM selection or relax the existing audition gates. The next priority is downbeat/meter estimation, phrase validation and timestamped vocal annotations on exact recordings. Chords, lyrics, stems and transcription are separate later implementations, not required for the first safe track shortlist.

Downbeats, phrases and vocal boundaries (analyzer v5)

rust-dj-analysis-5 adds an optional Beat This! ONNX layer using the pinned beat-this = 1.0.0 Rust adapter. Production uses the full model; preprocessing and model SHA-256 hashes are recorded per analysis. The published Beat This! model and reference implementation are MIT licensed; the Rust adapter provides ONNX preprocessing, chunking and event decoding. This is distinct from the noncommercial research music-classification heads already installed.

The neural output contains a separate grid ID, beat timestamps, predicted downbeats and their exact beat indices. Bar-length histograms retain irregular intervals. A three- or four-beat bar candidate requires at least four observed bar intervals and 80% agreement. This support fraction is not calibrated confidence; a denominator such as 4/4 is not inferred. A reviewed native grid still takes precedence. The existing classical region grid is retained while neural predictions are evaluated. Neural timestamps define analysis regions only as a fallback when classical beats are unavailable; they never automatically approve the playback grid.

Phrase candidates now require a measured structural change within two seconds of a predicted downbeat. Candidates retain the measured time, snapped time, grid index and underlying bass/onset/level changes. Distances between candidates can support common phrase lengths without assuming that the first bar is the first phrase. No verse/chorus accuracy is claimed. On a neural grid, transition regions start and end on predicted downbeats and require consistent bar counts throughout the region. Ambiguous meter produces no such region candidates.

The existing voice head also runs on individual YAMNet embeddings: 480 ms hop, 975 ms audio context, with explicit output and context intervals. This new vocal_intervals layer complements the existing smoother nine-frame voice summary. Hysteresis defaults are 0.65 to enter, 0.35 to exit, and two frames minimum. These identify model activation episodes, not calibrated presence or dominance. Outside an episode remains unknown, not certified instrumental.

The bounded DJ summary includes up to three upcoming phrase hypotheses and vocal activation candidates. Full beat arrays, frame scores and embeddings remain outside the LLM context. Automatic transition readiness remains false.

Listening annotations and boundary evaluation

Track details now include Listen and validate. Load the audio preview, choose downbeats, phrases or vocal presence, specify the range actually heard, and enter timestamps or mark the playhead. Vocal labels are explicit present, absent or uncertain intervals; gaps are unknown. Do not copy predictions as human labels without listening. Reviews can be exported from the same panel.

Authenticated GET/POST /api/v1/library/{id}/annotations reads/appends immutable MongoDB track_annotations records in the current workspace's collection prefix. Each submission pins PCM SHA-256 and native revision, includes the reviewed range and server-generated workspace scope/time, and rejects stale sources, overlapping vocal spans and invalid event ordering. It records a manual authenticated workspace submission, not a separately certified annotator identity. Saving labels does not change reviewed, approve a grid, or calibrate a model. A null layer means not reviewed; an empty event array means the reviewer found no such events inside the specified range.

Export the analysis and one listening review, then run:

python3 tools/audio-analysis/evaluate_timing.py track-analysis.json annotations.json

This reports beat-event F1 at 70 ms, phrase-event F1 at one second, vocal presence on explicitly labeled audio at 100 ms integration steps, and local vocal start/end recall within 500 ms where contiguous present/absent labels establish a boundary. Review edges, unknown gaps and uncertain labels do not create vocal boundary references. Results describe that recording and review only; a held-out, manually annotated set is still needed before claiming library-wide accuracy.

The full-model adapter was checked against the published Python golden fixture (beat-this-rs revision 089b509247e6fdcec666511c0dcf0d5f39c21e73): 575/575 beats and 144/144 downbeats matched within 70 ms, with mean matched absolute differences of 3.44 ms and 3.47 ms. This establishes reference parity on that fixture, not accuracy on the DJ library. The Levels radio-edit pilot produced mixed two/four-beat bar intervals (75% four-beat support), so the 80% meter rule correctly leaves its meter ambiguous rather than forcing 4/4.

Partial-window level stability

V5 weights each reporting-window dB level by its actual overlap with the region: max(0, min(region_end, window_end) - max(region_start, window_start)). rms_std_db is the weighted population standard deviation of these levels; level_stability_score remains clamp(1 - rms_std_db / 12, 0, 1). This is a variation descriptor, not combined signal RMS and not transition quality. The method, weighted mean and covered duration are exported. Partial windows still approximate their level from the complete two-second measurement.

The Levels regression starts at 133.185 seconds. Moving its end from 193.997984 to 194.006385 seconds now changes the deviation from approximately 4.0854 to 4.1022 dB, rather than counting 6 ms as a full two-second window. Both endpoints continue to follow their referenced beat grid. Structural-change records also include the exact before/after time spans, window counts and aggregation methods, allowing the level/bass/onset deltas to be reproduced.

The next musical acceptance step is a reviewed, auditionable overlap: compare scoring predictions with rendered audio using matching approved playback grids and the relevant manually labeled phrase/vocal intervals. timing_current is freshness only and cannot authorize execution. The offline render/replay integration test rejects both revoked approval and replaced grid snapshots before producing audio.

Local timing consistency (analyzer v6)

rust-dj-analysis-6 / processing profile library-pipeline-20260912-v5 addresses the next Levels export review. It retains the v5 indexed region endpoints, overlap-weighted window-level variance, duration-weighted section power aggregation and explicit boundary measurement intervals. Regression tests cover these contracts; this revision does not revise those measurements.

Phrase hypotheses now retain predicted downbeat intervals separately from detected beat intervals. Every intervening downbeat interval must contain the same three or four detected beats, and the span must contain no flagged timing anomaly, before predicted_bar_count and local_beats_per_bar_candidate are populated. common_phrase_length_support is nullable: it is unknown when those checks fail, otherwise it describes only whether the predicted bar count is 4, 8, 16 or 32. validated_bar_count remains null. This is internal consistency, not verified meter or phrase structure, and no denominator is inferred.

The old bars_since_previous_candidate field is removed from new reports; predicted_downbeat_intervals replaces it without implying real bars. Likewise, the meter summary names its observation count predicted_downbeat_intervals, not observed_complete_bars. Historical reports remain immutable.

For the stored Levels timestamp fixture, 38.140–51.460 seconds spans eight predicted downbeat intervals but only 28 detected beat intervals; 131.460–139.080 spans four and 17 respectively. Both now have unknown common-length support. The fixture contains model timestamps only, no recording, and is a numerical regression case rather than listening ground truth.

Neural tempo now uses a least-squares fit of timestamp against event index on longer sequences. An adjacent interval differing by more than 25% from the median within 16 intervals either side is flagged and splits the sequence. No event is silently deleted, inserted, or renumbered. Fits require at least 32 intervals; the longest fit with RMS residual at most 30 ms and maximum absolute residual at most 60 ms supplies the neural BPM summary. These are engineering review thresholds, not calibrated confidence. If no fit qualifies, BPM and selected_fit are null. Gradually changing tempo can fail the residual checks; this implementation does not claim a complete variable-tempo segmentation model.

tempo_evidence includes all fit spans, fitted origins and periods, residuals, anomalous intervals, and the old median eight-beat estimate as a separate quantization-sensitive diagnostic. Extrapolation residuals across the whole raw beat array are included for inspection only: raw indices across insertion or omission candidates are not validated musical beat counts. A selected fit is a local, unapproved constant-tempo hypothesis, never automatically adopted as a canonical or playback grid. Grid review and transition execution gates remain separate.

The Levels fixture reproduces approximately 126.00055 BPM when fitting beat indices 32–256, while the short-span diagnostic stays near 126.31569 BPM. The extra short intervals around 137 seconds remain visible for listening review. Synthetic tests cover 20 ms quantization, an inserted beat, insufficient data, variable tempo, and consistent three- and four-beat local bar candidates.

The music-detail UI exposes fit coverage, residuals, clickable anomaly times, and phrase count uncertainty. Review the exact recording around 0–16, 38–52 and 136–140 seconds, plus the proposed entry/exit vocal coverage. Predicted vocal intervals still do not establish absence or safe overlap. No listening annotations are fabricated by this patch; an auditioned, reviewed two-track transition remains the next musical validation step.

Region evidence in planning context

dj-region-evidence-1 enriches the selected mix_in and mix_out with planning_evidence, joined against the proposed interval rather than only against the current playhead. Candidate retrieval and the position-specific context endpoint use this same builder. Report responses rebuild the compact summary from stored measurements, then apply stale-grid invalidation. Existing tracks do not need decoding or neural inference again for this context update.

Each region includes intersecting neural tempo fits with their grid identity, source fit index, applicable interval, fitted origin/period, BPM, eligibility, review state and residual summaries. Full-region containment is explicit. The original region grid and timestamps remain unchanged. A good fit is not an approved playback grid, and residuals measure agreement with predictions, not timing accuracy against the recording.

Vocal activation intersections include source candidate indices, their full intervals and the actual overlap. Zero intersections does not establish vocal absence. Phrase hypotheses inside the region retain provisional bar support and anomaly flags. Context is bounded to eight fits, sixteen vocal intervals and eight phrase hypotheses per region, with full counts and truncation flags; raw beat arrays, embeddings and residual arrays are excluded.

The Levels numerical analysis remains the v6 regression baseline. Tests cover separate entry/exit fit selection, vocal evidence beyond the first upcoming candidates, half-open vocal intersections, partial fit coverage, context truncation and unchanged approval/coordinates. Anomaly review buttons show milliseconds so adjacent short intervals have distinct labels.

This supplies evidence for planning; it does not implement pairwise musical compatibility or approve autonomous execution. The next acceptance milestone is a reviewed two-track overlap: approve playback timing, map the regions to that grid, review phrase/vocal intervals, render and listen, and verify rejection of stale or mismatched execution commands. No human listening annotation or musical approval is inferred from numerical regression checks.