8-cut/docs/plans/2026-04-17-audio-scan-design.md

Audio Similarity Scanning — Design

Goal: Scan a video's audio track and highlight segments that match the sound profile of existing reference clips, so the user can quickly find similar moments without scrubbing manually.

Runs in: Python/Qt client (main.py), not the server.

---

Core Module: core/audio_scan.py

New module alongside core/tracking.py. Two main functions:

• build_profile(clip_paths: list[str]) -> dict — extracts MFCCs (20 coefficients) from each clip using librosa, returns a profile containing both the averaged vector and individual clip vectors.
• scan_video(video_path: str, profile: dict, mode: str, threshold: float, hop: float) -> list[tuple[float, float, float]] — slides an 8s window across the video's audio, returns (start_time, end_time, score) tuples for segments above threshold.

Feature Extraction

• Audio loaded via librosa.load() (handles video files directly, mono, 22050Hz).
• MFCCs: librosa.feature.mfcc(n_mfcc=20), averaged over time axis to produce a single vector per window/clip.
• Similarity: cosine similarity (numpy dot product on L2-normalized vectors).

Matching Modes

• Average mode: Compare each window to the mean of all reference MFCC vectors. Fast, good when references are homogeneous.
• Nearest mode: Compare each window to every reference vector, take the max score. Better when references have variety within the style.

Parameters

• threshold (float, 0.0–1.0): minimum cosine similarity to include a segment. Default 0.7.
• hop (float, seconds): step size for the sliding window. Default 1.0s.
• Window size fixed at 8s to match reference clip length.

---

UI Integration in main.py

Controls

Added near the existing tracking checkbox area:

• "Scan" button — triggers audio scan on current video.
• Threshold slider (0.0–1.0, step 0.05) — controls match strictness.
• Mode combobox — "Average" / "Nearest".
• Reference source combobox — "Current Profile" / "Custom Folder" (shows folder picker when "Custom Folder" selected).

Scan Workflow

1. User clicks Scan.
2. Reference clips collected: either all export output_path values from the current profile (via DB) or all audio/video files in a custom folder.
3. Scan runs in a QThread so UI stays responsive.
4. On completion, results sent to Timeline widget via signal.

Timeline Display

• New set_scan_regions(regions: list[tuple[float, float, float]]) method on Timeline.
• Drawn as semi-transparent colored rectangles behind existing markers.
• Color intensity proportional to score (brighter = higher match).
• Cleared on file change or re-scan.

Keyboard Shortcut

• S — jump cursor to the next scan region (similar to M for next marker).

---

Data Flow

Reference clips (DB export paths or folder)
    |
librosa.load() each -> MFCC vectors (20-dim)
    |
Profile: { mean_vector, clip_vectors[] }
    |
Current video -> librosa.load() full audio (mono 22050Hz)
    |
Sliding 8s window (hop=1s) -> MFCC per window
    |
Cosine similarity vs profile -> score per position
    |
Threshold filter -> [(start, end, score), ...]
    |
Timeline: semi-transparent highlight regions

Performance

• 2-hour video at 22050Hz mono ~ 380MB memory.
• MFCC extraction + sliding window: ~10-30s.
• QThread keeps UI responsive.

What This Does NOT Do

• No DB schema changes — scan results are ephemeral (visual only).
• No auto-export — user decides what to cut.
• No server integration — runs entirely in the Python client.
• No GPU/ML model dependency — just librosa + numpy.