Training, test sets and manual feedback

Updated 2026-09-13. Scoring has three distinct meanings: measured technical behavior, human preference, and a model prediction. Store their sources separately. Do not display an arbitrary weighted score as a probability of a good mix.

Manual review implemented

The Transition memory page auditions a rendered overlap and stores overall, rhythm, harmony, energy and coherence ratings from 1–5 plus notes. Low overall ratings reduce the planner bonus for that recording pair; positive technical health cannot cancel a negative human contribution. Lowest-ratings-first helps review bad results.

Avoid excludes that recipe from subsequent B2B planning for the same source PCM hashes, analysis revisions and engine fingerprint. Cue positions and target BPM are no longer part of the match: they were exact floats, so an exclusion recorded at one playhead second never applied again. An avoid is a judgement about that recording pair and never propagates through bucketed transfer to other pairs. It does not ban the whole song or recipe globally. The latest review per reviewer label per attempt counts; earlier entries remain in history. Submit a new review with Avoid cleared to withdraw that reviewer's exclusion. Any other current exclusion still applies. All candidates excluded means no feasible choice; the planner must not silently restore an avoided option.

The API adds optional avoid: boolean to POST /api/v1/transitions/{attempt_id}/reviews. Old records default to false. Evidence.human_excluded is passed through the stored candidate evidence. Storage uses the existing workspace Mongo/SQLite transition store.

Reviewer labels are still self-entered, not verified Keycloak identity or expert credentials. These are listener ratings, not certified professional labels. Live supervisor handoffs currently record events but do not create these rendered attempts; this review flow applies to the B2B lab. Whole-set, sample and overlay review forms still need implementation.

Offline training foundation implemented

Rust prochain-evaluation::learning exports rated attempts, validates a versioned dataset, builds grouped train/validation/test partitions, fits a small supervised recipe-rating regression model, and evaluates held-out mean absolute error against a constant training-mean baseline. No external Python/ML service is required.

The baseline's twelve inputs are six recipe-shape features — target tempo, recipe duration, bass-swap location and duration, filter depth and fader-curve type — plus six describing the musical context the transition was played in: outgoing and incoming section ordinals, phrase-alignment error in beats, suspected vocal collision fraction, energy delta and harmonic fit. All are known before rendering. Post-render faults and human ratings are targets/evidence, never input features. The first six alone could only ever rank fader curves; the model still does not understand style, lyrics or arrangement beyond these summaries.

The pointwise regression above is kept, but the model that reaches ranking is the pairwise one: learning::train_pairwise fits the same weight vector by logistic regression on feature differences between the winner and loser of each blind comparison. A constant cancels in every comparison, so there is no bias to fit, and accuracy is reported against the only honest reference for a pairwise task, the 0.5 coin flip. Comparisons are held out by connected recording/session group so a held-out pair never shares a recording with training.

Dataset dj-learning-2 stores feature names, source and rights declarations plus examples with IDs, kind, normalized features, target, label source, provenance and leakage keys. Kinds distinguish transition/mix/sample/overlay, but the current export and CLI trainer only implement transition ratings. Synthetic and AI-critic labels cannot train the human-preference baseline. Existing unreviewed attempts are omitted, rather than invented as successful demonstrations. Practice renders are no longer excluded: excluding them had discarded every experiment the worker ever produced, and the corpus is where the renders are.

Rows sharing any session/mix, source PCM or rendered-audio identity stay in one connected component, including transitive links. Components are hash-assigned approximately 80/10/10; small datasets may yield empty partitions and cannot train. A common track can connect many mixes into one component. This is intentional. For external data, also include canonical recording/song-family IDs across edits; PCM hashes alone cannot detect a remaster or another encoding of the same song.

A canonical SHA-256 fingerprints the complete dataset. Freeze the dataset and split together: adding labels creates a new revision, not a silent change to the held-out benchmark. The trainer checks the exact split, requires at least 20 training/5 validation/5 test examples of one kind/source, and uses fixed hyperparameters. Validation/test labels never update weights. The minimum is an engineering gate, not evidence of statistical sufficiency. Correlated examples remain correlated even when split correctly; uncertainty estimates and wider independent test sets remain needed.

Models record dataset hash, feature schema, weights, training mean, validation errors and deployment_approved: false. Test reports compare held-out error with the baseline. No command auto-promotes a model. Repeated experimentation against the same test set can overfit it; freeze an additional final listening benchmark before deployment and use validation for model selection.

From aidj-platform, with the normal Rust/native build environment:

cargo run -p prochain-evaluation --example learning -- export /path/to/library > dataset.json
cargo run -p prochain-evaluation --example learning -- split dataset.json > split.json
cargo run -p prochain-evaluation --example learning -- train dataset.json split.json > model.json
cargo run -p prochain-evaluation --example learning -- test dataset.json split.json model.json > test.json

Export honors the existing Mongo workspace environment when configured; otherwise it reads the local transition store. Keep snapshots private. The CLI is a developer example, not currently packaged as a deployed training worker. The regression checks use labelled synthetic fixtures to test the software, not to claim a trained professional DJ model.

Dataset research and what it can teach

Source	Available evidence	Appropriate use and limitation
UnmixDB	Automatically generated mixes; source/cue/tempo/speed ground truth; underlying CC track material	Validate alignment and reconstruction of known mixing operations. It is synthetic, not expert taste labels. Preserve per-track terms and attribution.
DJ Mix Dataset	Human DJ mix/track metadata and acquisition tooling	Investigate authentic sequence/transition examples. Metadata availability does not grant blanket redistribution or training rights for linked recordings.
DJtransGAN	Research implementation, pretrained model and data-generation pipeline	Reference for learned fader/EQ control. Authors explicitly do not distribute their training dataset because of licensing issues.

These are researched sources, not imported training data. Their annotations do not automatically supply “good versus bad,” professional ratings, lyrics/overlay placement or labelled sample opportunities. External adapters, asset rights review, recording identity mapping and benchmark imports remain work. The user's product specification defines the learning loop but is not itself a labelled dataset.

Next connection to the DJ GPS

Capture a stable recommendation ID, candidate set, analysis/model versions, actual choice, rendered outcome and later review. A skipped recommendation is weak behavioral evidence: the DJ may have been browsing or preparing another deck. Only explicit feedback is an explicit preference label. Separate private per-DJ preferences from a shared technical policy and obtain consent before using private session material in a shared corpus.

Use learned scores to rank permitted candidates/templates. A score never bypasses approved timing, current region coordinates, authority, headroom or human Avoid. Test challengers against the deployed baseline, archive comparisons, promote with a rollback path, then measure outcomes again. The idle practice scheduler is deployed as aidj-practice-worker on the aidj.learning.practice.v1 queue, and since 2026-09-14 the deployment gate and model loading exist: analyzerd dj-learn trains and stores a model unapproved, dj-learn approve LABEL deploys it only if held-out accuracy beat the coin flip, and both rankers load the approved model and apply a bonus bounded to ±0.10. Rollback is removing approval. Live attempt capture and authenticated expert reviews remain unimplemented.

The state on deployment day is worth recording plainly: 1,028 attempts, 1,027 of them practice, zero human reviews, zero comparisons, no model. The loop had been producing data and learning nothing, and practice was switched off in settings. It was re-enabled that day; nothing learned has yet been demonstrated on real renders. The Transition memory page's loop-status panel says this in the product.