From e14cebd081f8678e0d9cbaa2ccece510ad0f0004 Mon Sep 17 00:00:00 2001 From: Ethanfel Date: Thu, 2 Jul 2026 13:33:19 +0200 Subject: [PATCH] docs: audio workspace tab design (2-phase: extract/edit/classify + dataset) Co-Authored-By: Claude Opus 4.8 --- docs/plans/2026-07-02-audio-tab-design.md | 155 ++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 docs/plans/2026-07-02-audio-tab-design.md diff --git a/docs/plans/2026-07-02-audio-tab-design.md b/docs/plans/2026-07-02-audio-tab-design.md new file mode 100644 index 0000000..72afbaa --- /dev/null +++ b/docs/plans/2026-07-02-audio-tab-design.md @@ -0,0 +1,155 @@ +# Audio workspace tab — Design + +**Goal:** Promote audio from a pair of cramped transport-row controls to a +first-class **Audio** deck tab that unifies everything audio: exact-length +extraction, light non-destructive editing of the extracted sound, an +audio-dedicated dataset workflow, and the existing scan/classify controls. + +**Runs in:** Python/Qt client (`main.py`) + `core/` helpers. No server changes. + +**Approach:** Replace the current **Scan** deck tab with an **Audio** tab. +Because the tab now covers four concerns, its body is a `QToolBox` accordion +(one pane open at a time) rather than flat stacked group-boxes. Existing +scan/extraction widgets are re-parented (same handlers/signals) — no behavior +change beyond relocation. + +**Deck order after:** `Export | Crop & Track | Audio`. The `ScanResultsPanel` +(3rd splitter pane) is untouched. + +Delivered in **two phases**. Phase 1 is self-contained and shippable; Phase 2 +adds the dataset workflow. + +--- + +## Deck wiring (both phases) + +`_build_control_deck` (main.py:4661): swap `_tab_scan` → `_tab_audio` — +`_deck_key="audio"`, `_label="Audio"`, `_pinned=False`; replace it in +`_deck_panels`; one `addTab`. The pin/split machinery is `_deck_panels`-driven, +so it picks the new panel up automatically. Old persisted `"scan"` pin-state is +ignored (harmless). Nothing auto-switches to the old scan tab — verified: +`_tab_scan` is referenced only in construction + its builder. + +`_build_scan_tab` is renamed `_build_audio_tab` and rebuilt as a `QToolBox` +with the panes below. + +--- + +## Phase 1 — Extract, Edit, Classify + +### Pane 1 — Extract & Edit + +**Relocated from the transport row** (removed there): `_spn_audio_len` (length) +and `_btn_extract_audio` (`♪ Extract`). The teal timeline region band is +unaffected — still driven by `_spn_audio_len`. + +**New controls:** +- `_cmb_audio_fmt` — output format (WAV / MP3 / FLAC / M4A / OGG / OPUS), + persisted to QSettings `audio_extract_fmt`; pre-selects the save-dialog filter + so the format isn't re-picked each time. +- **Waveform strip** — a compact widget rendering the current region (or + last-extracted clip). Generated from decoded PCM (reuse + `core/audio_scan._load_audio_ffmpeg`) or `ffmpeg showwavespic`. Read-only in + v1 (no drag handles). +- **Play / Stop** — audition the current region/clip. Playback backend decided + in the plan (likely a short ffmpeg-decoded buffer through a + `QMediaPlayer`/`QSoundEffect`, kept separate from the mpv video player). + +**Editing (non-destructive, ffmpeg filter chain at render time):** + +| Control | Filter | +|---|---| +| Trim in / out (refine past the length spinbox) | `atrim` / adjusted `-ss` / `-t` | +| Fade in / out (s) | `afade=t=in` / `afade=t=out` | +| Normalize (toggle) | `loudnorm` | +| Gain (dB) | `volume` | + +`core/ffmpeg.build_audio_clip_command` gains an optional `filters` param → +appended as `-af ""`. Foley/existing callers pass none → byte-identical +output. Extract applies the current edit settings; nothing is written until you +save. + +### Pane 2 — Scan / Classify + +Relocated verbatim from the old Scan tab: model combo + `_btn_model_history`; +`_btn_scan`, `_btn_auto_export`, `_btn_speech`, `_btn_scan_mode`; +`_spn_auto_fuse`, `_sld_threshold`. Same handlers, same `ScanResultsPanel` +output. + +*(Train / Hard-negatives / Dataset-stats buttons join Pane 3 in Phase 2; in +Phase 1 they stay wherever they are today — the menu bar.)* + +--- + +## Phase 2 — Audio dataset + +### Pane 3 — Dataset (audio-dedicated) + +A dedicated audio dataset **workflow + view**, combining browser-plus-label, +add-from-extraction, and in-tab access to the existing tools. + +**Key data-model finding (the one open decision — confirm before building):** +the existing classifier trains on **(source_video, time-ranges)** per profile — +`db.get_training_data(...)` returns `(source_video_path, positive_times, +soft_times, negative_times)` per video; standalone clip *files* are not the +training unit. An extracted audio region always knows its +`(source_video, start, end)`. So the cheapest, most powerful integration is +**merged**: "Add to dataset" records the region as a labeled positive/negative +time-range in the **same** per-profile DB the video-export workflow already +feeds — no new trainer, no schema change. The Audio dataset pane is then an +**audio-oriented view/filter** over that shared dataset. + +- **Recommended (merged):** audio extraction becomes another way to label + training data; reuses `get_training_data`, `get_training_stats`, + hard-negatives, and `train_classifier` unchanged. +- **Alternative (separate):** a standalone folder of `.wav` clips + own + `dataset.json` + a new file-based training path. More code, duplicates the + trainer. Only needed if the audio set must stay entirely independent of the + video profiles. + +*(Brainstorming default was "separate"; inspecting the DB flips the +recommendation to "merged." Flagged for confirmation.)* + +**Pane contents (assuming merged):** +- **Add to dataset** — from the current/last extraction: + `[class ▾]` + `(•positive / ○negative)` + `[+ Add]`. Writes a labeled + time-range into the DB for the active profile (and, for standalone-clip + parity, writes the clip's `dataset.json` label via `upsert_clip_annotation` + when a file was saved). +- **Browser** — a list of the audio dataset entries (class, pos/neg, source, + time-range), filterable by class; select to seek the source video to that + region; remove entries. +- **Tools** — in-tab buttons for the existing dialogs: *Dataset Statistics* + (`DatasetStatsDialog`), *Manage Hard Negatives* (`HardNegativesDialog`), + *Train classifier* (existing Train dialog). + +--- + +## Persistence & migration + +- QSettings gains `audio_extract_fmt` (+ edit defaults: fade/normalize/gain) + alongside existing `audio_extract_len`, `audio_extract_dir`. +- Deck pin-state key `"scan"` → `"audio"`; absent → default unpinned. +- Phase 2 (merged): no schema change — reuses existing training tables / + `dataset.json`. Phase 2 (separate) would add a folder convention only. + +## Testing + +- `tests/test_ui_structure.py`: update `test_extract_audio_controls_exist` to + locate controls in the Audio tab; add a test that the Audio deck tab exists as + a `QToolBox` with Extract/Classify panes and that scan controls remain + reachable. +- `tests/test_utils.py`: extend the audio-command tests for the new `-af` + filter chain (trim/fade/normalize/gain), keeping existing codec-by-extension + assertions. +- Phase 2: a DB test that "add to dataset" produces a training-data row + consumable by `get_training_data`. + +## What this does NOT do + +- No full waveform-drag editor in v1 — editing is parametric + (trim/fade/normalize/gain). +- No change to the Foley/LTX-2 export pipeline or `ScanResultsPanel`. +- No new ML model or embedding change. +- Phase 1 makes no DB/schema change; the dataset-model decision is confined to + Phase 2.