Human DJ corpus and timing baseline

The subsequent Learning studio adds the full DJ Mix Dataset, a 63,038-track external catalog and bounded private practice. The Raveform-only results below remain a versioned baseline.

Raveform metadata is now imported into dedicated MongoDB research collections. It is not yet used by the live candidate selector. It supplies observed cue and overlap evidence, not professional quality ratings or executable mixer controls. The current live Kimi adviser still uses the user's analyzed library and stored sequential-fade recipe.

What was actually imported

The Raveform dataset archive at revision f0fbae800603608c7d7a4f860524888313f92395 has SHA256 10c97fa9213fe4ca032195e73b6a9d068c0d5ca8a8f603615bb1bdbabffb34de. Its schema documentation distinguishes DTW-estimated mix-to-track alignments from human structure annotations.

Our import found 4,911 mix metadata records and accepted 57,416 adjacent-track observations. These are our archive/import counts, not the paper's headline transition count. Unknown or repeated track identities, missing alignment pairs and reversed alignment order are excluded rather than bridged or guessed. This first importer does not import the separate structure annotations.

Each observation retains source/target external IDs, mix URL, tracklist index, track and mix cue coordinates, signed overlap in beats and seconds, match rates, and leakage identities. quality_score, controls and meter remain null. Negative overlap describes a gap; it is not clamped into a fictitious overlap. Beats are not converted to bars without meter evidence. Linked audio is not fetched. No inferred EQ/fader curves or vocal safety labels are created.

The source dataset card reports CC-BY-4.0. Provenance preserves source attribution and revision; the importer does not infer rights to the linked recordings from the metadata license.

Train, validation and test

Connected components over mix URLs and YouTube track IDs are assigned together using deterministic SHA256 buckets. This prevents known shared identities from crossing partitions. Different uploads of the same recording can still have different IDs; cross-corpus recording resolution remains necessary.

Partition	All observations	Screened overlap examples
Training	56,822	20,733
Validation	237	77
Test	357	118

A very large component contains 53,895 observations. Thus nominal 80/10/10 hash buckets do not produce balanced row counts. The small disconnected holdouts are not representative evidence of broad generalization.

The first baseline uses only training observations with both match rates at least 0.5, at least 32 matched beats each, and overlap within 0–256 beats. These are screening rules, not calibrated confidence thresholds. The training median is 67 beats; the constant-median predictor has validation MAE 43.06 beats and test MAE 40.17 beats. This is a weak descriptive baseline, not a recommended 67-beat recipe or a successful DJ-quality model. No model is promoted.

Human performance is an observed choice, not necessarily a positive taste rating. Tracks a DJ did not choose are not automatically bad alternatives. Expert reviews, technical measurements and synthetic damage labels retain separate meanings in the existing learning pipeline.

Reproduce and store

python3 tools/import_raveform.py /path/to/raveform.zip /new/output/directory
python3 -m unittest discover -s tools -p test_import_raveform.py

Run those in aidj-platform. The importer verifies the pinned archive hash and writes observations.jsonl, manifest.json and timing-prior.json. It records archive, importer and observation hashes. It only reads named ZIP members; it does not extract the archive into the filesystem.

From aidj-gitops:

python3 tools/import-raveform-mongo.py /new/output/directory

This uses the existing mounted Mongo credential through the UI pod and writes prochain_aidj.research_dj_observations and research_dj_corpora. Immutable IDs include the observation-corpus hash. Importing the same corpus is idempotent. A completed corpus manifest is published only after count/hash verification. These collections are research metadata, separate from user libraries and live recipe permissions. No new broker, audio import or library reanalysis is needed.

Next integration

1. Import the DJ Mix Dataset into the same observed-decision contract, resolve common mix/recording identities, then regenerate joint leakage partitions. Its rows must not simply be added to Raveform as independent examples.
2. Join structure annotations and properly identified local audio analysis; retain estimates separately from reviewed boundaries and musical quality.
3. Evaluate conditional mix-point/overlap predictors against this constant baseline; expose useful bounded evidence to Kimi and the planner.
4. Add replayable two-track auditions and expert preference labels. Promote improved recipe/ranking versions only after held-out evaluation.

DJ Mix Dataset, UnmixDB and Jamendo were not imported in this Raveform-only iteration. The DJ Mix Dataset is now imported by the subsequent Learning studio update; UnmixDB and Jamendo remain unimported. No neural quality critic is trained. The full corpus is stored in MongoDB; Git contains the importer and compact validation record rather than tens of megabytes of third-party observations.