docs: audio workspace tab design (2-phase: extract/edit/classify + dataset)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -0,0 +1,155 @@
|
|||||||
|
# Audio workspace tab — Design
|
||||||
|
|
||||||
|
**Goal:** Promote audio from a pair of cramped transport-row controls to a
|
||||||
|
first-class **Audio** deck tab that unifies everything audio: exact-length
|
||||||
|
extraction, light non-destructive editing of the extracted sound, an
|
||||||
|
audio-dedicated dataset workflow, and the existing scan/classify controls.
|
||||||
|
|
||||||
|
**Runs in:** Python/Qt client (`main.py`) + `core/` helpers. No server changes.
|
||||||
|
|
||||||
|
**Approach:** Replace the current **Scan** deck tab with an **Audio** tab.
|
||||||
|
Because the tab now covers four concerns, its body is a `QToolBox` accordion
|
||||||
|
(one pane open at a time) rather than flat stacked group-boxes. Existing
|
||||||
|
scan/extraction widgets are re-parented (same handlers/signals) — no behavior
|
||||||
|
change beyond relocation.
|
||||||
|
|
||||||
|
**Deck order after:** `Export | Crop & Track | Audio`. The `ScanResultsPanel`
|
||||||
|
(3rd splitter pane) is untouched.
|
||||||
|
|
||||||
|
Delivered in **two phases**. Phase 1 is self-contained and shippable; Phase 2
|
||||||
|
adds the dataset workflow.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deck wiring (both phases)
|
||||||
|
|
||||||
|
`_build_control_deck` (main.py:4661): swap `_tab_scan` → `_tab_audio` —
|
||||||
|
`_deck_key="audio"`, `_label="Audio"`, `_pinned=False`; replace it in
|
||||||
|
`_deck_panels`; one `addTab`. The pin/split machinery is `_deck_panels`-driven,
|
||||||
|
so it picks the new panel up automatically. Old persisted `"scan"` pin-state is
|
||||||
|
ignored (harmless). Nothing auto-switches to the old scan tab — verified:
|
||||||
|
`_tab_scan` is referenced only in construction + its builder.
|
||||||
|
|
||||||
|
`_build_scan_tab` is renamed `_build_audio_tab` and rebuilt as a `QToolBox`
|
||||||
|
with the panes below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — Extract, Edit, Classify
|
||||||
|
|
||||||
|
### Pane 1 — Extract & Edit
|
||||||
|
|
||||||
|
**Relocated from the transport row** (removed there): `_spn_audio_len` (length)
|
||||||
|
and `_btn_extract_audio` (`♪ Extract`). The teal timeline region band is
|
||||||
|
unaffected — still driven by `_spn_audio_len`.
|
||||||
|
|
||||||
|
**New controls:**
|
||||||
|
- `_cmb_audio_fmt` — output format (WAV / MP3 / FLAC / M4A / OGG / OPUS),
|
||||||
|
persisted to QSettings `audio_extract_fmt`; pre-selects the save-dialog filter
|
||||||
|
so the format isn't re-picked each time.
|
||||||
|
- **Waveform strip** — a compact widget rendering the current region (or
|
||||||
|
last-extracted clip). Generated from decoded PCM (reuse
|
||||||
|
`core/audio_scan._load_audio_ffmpeg`) or `ffmpeg showwavespic`. Read-only in
|
||||||
|
v1 (no drag handles).
|
||||||
|
- **Play / Stop** — audition the current region/clip. Playback backend decided
|
||||||
|
in the plan (likely a short ffmpeg-decoded buffer through a
|
||||||
|
`QMediaPlayer`/`QSoundEffect`, kept separate from the mpv video player).
|
||||||
|
|
||||||
|
**Editing (non-destructive, ffmpeg filter chain at render time):**
|
||||||
|
|
||||||
|
| Control | Filter |
|
||||||
|
|---|---|
|
||||||
|
| Trim in / out (refine past the length spinbox) | `atrim` / adjusted `-ss` / `-t` |
|
||||||
|
| Fade in / out (s) | `afade=t=in` / `afade=t=out` |
|
||||||
|
| Normalize (toggle) | `loudnorm` |
|
||||||
|
| Gain (dB) | `volume` |
|
||||||
|
|
||||||
|
`core/ffmpeg.build_audio_clip_command` gains an optional `filters` param →
|
||||||
|
appended as `-af "<chain>"`. Foley/existing callers pass none → byte-identical
|
||||||
|
output. Extract applies the current edit settings; nothing is written until you
|
||||||
|
save.
|
||||||
|
|
||||||
|
### Pane 2 — Scan / Classify
|
||||||
|
|
||||||
|
Relocated verbatim from the old Scan tab: model combo + `_btn_model_history`;
|
||||||
|
`_btn_scan`, `_btn_auto_export`, `_btn_speech`, `_btn_scan_mode`;
|
||||||
|
`_spn_auto_fuse`, `_sld_threshold`. Same handlers, same `ScanResultsPanel`
|
||||||
|
output.
|
||||||
|
|
||||||
|
*(Train / Hard-negatives / Dataset-stats buttons join Pane 3 in Phase 2; in
|
||||||
|
Phase 1 they stay wherever they are today — the menu bar.)*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — Audio dataset
|
||||||
|
|
||||||
|
### Pane 3 — Dataset (audio-dedicated)
|
||||||
|
|
||||||
|
A dedicated audio dataset **workflow + view**, combining browser-plus-label,
|
||||||
|
add-from-extraction, and in-tab access to the existing tools.
|
||||||
|
|
||||||
|
**Key data-model finding (the one open decision — confirm before building):**
|
||||||
|
the existing classifier trains on **(source_video, time-ranges)** per profile —
|
||||||
|
`db.get_training_data(...)` returns `(source_video_path, positive_times,
|
||||||
|
soft_times, negative_times)` per video; standalone clip *files* are not the
|
||||||
|
training unit. An extracted audio region always knows its
|
||||||
|
`(source_video, start, end)`. So the cheapest, most powerful integration is
|
||||||
|
**merged**: "Add to dataset" records the region as a labeled positive/negative
|
||||||
|
time-range in the **same** per-profile DB the video-export workflow already
|
||||||
|
feeds — no new trainer, no schema change. The Audio dataset pane is then an
|
||||||
|
**audio-oriented view/filter** over that shared dataset.
|
||||||
|
|
||||||
|
- **Recommended (merged):** audio extraction becomes another way to label
|
||||||
|
training data; reuses `get_training_data`, `get_training_stats`,
|
||||||
|
hard-negatives, and `train_classifier` unchanged.
|
||||||
|
- **Alternative (separate):** a standalone folder of `.wav` clips + own
|
||||||
|
`dataset.json` + a new file-based training path. More code, duplicates the
|
||||||
|
trainer. Only needed if the audio set must stay entirely independent of the
|
||||||
|
video profiles.
|
||||||
|
|
||||||
|
*(Brainstorming default was "separate"; inspecting the DB flips the
|
||||||
|
recommendation to "merged." Flagged for confirmation.)*
|
||||||
|
|
||||||
|
**Pane contents (assuming merged):**
|
||||||
|
- **Add to dataset** — from the current/last extraction:
|
||||||
|
`[class ▾]` + `(•positive / ○negative)` + `[+ Add]`. Writes a labeled
|
||||||
|
time-range into the DB for the active profile (and, for standalone-clip
|
||||||
|
parity, writes the clip's `dataset.json` label via `upsert_clip_annotation`
|
||||||
|
when a file was saved).
|
||||||
|
- **Browser** — a list of the audio dataset entries (class, pos/neg, source,
|
||||||
|
time-range), filterable by class; select to seek the source video to that
|
||||||
|
region; remove entries.
|
||||||
|
- **Tools** — in-tab buttons for the existing dialogs: *Dataset Statistics*
|
||||||
|
(`DatasetStatsDialog`), *Manage Hard Negatives* (`HardNegativesDialog`),
|
||||||
|
*Train classifier* (existing Train dialog).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Persistence & migration
|
||||||
|
|
||||||
|
- QSettings gains `audio_extract_fmt` (+ edit defaults: fade/normalize/gain)
|
||||||
|
alongside existing `audio_extract_len`, `audio_extract_dir`.
|
||||||
|
- Deck pin-state key `"scan"` → `"audio"`; absent → default unpinned.
|
||||||
|
- Phase 2 (merged): no schema change — reuses existing training tables /
|
||||||
|
`dataset.json`. Phase 2 (separate) would add a folder convention only.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- `tests/test_ui_structure.py`: update `test_extract_audio_controls_exist` to
|
||||||
|
locate controls in the Audio tab; add a test that the Audio deck tab exists as
|
||||||
|
a `QToolBox` with Extract/Classify panes and that scan controls remain
|
||||||
|
reachable.
|
||||||
|
- `tests/test_utils.py`: extend the audio-command tests for the new `-af`
|
||||||
|
filter chain (trim/fade/normalize/gain), keeping existing codec-by-extension
|
||||||
|
assertions.
|
||||||
|
- Phase 2: a DB test that "add to dataset" produces a training-data row
|
||||||
|
consumable by `get_training_data`.
|
||||||
|
|
||||||
|
## What this does NOT do
|
||||||
|
|
||||||
|
- No full waveform-drag editor in v1 — editing is parametric
|
||||||
|
(trim/fade/normalize/gain).
|
||||||
|
- No change to the Foley/LTX-2 export pipeline or `ScanResultsPanel`.
|
||||||
|
- No new ML model or embedding change.
|
||||||
|
- Phase 1 makes no DB/schema change; the dataset-model decision is confined to
|
||||||
|
Phase 2.
|
||||||
Reference in New Issue
Block a user