8-cut/docs/plans/2026-04-19-comfyui-node-pack-design.md

# ComfyUI-8cut Node Pack Design

Date: 2026-04-19

## Goal

Port 8-cut's video scanning, training, review, and export workflow to a ComfyUI node pack. The primary motivation is **remote access** — ComfyUI's web UI allows browser-based operation over the network, and HTML5 `<video>` handles streaming compression natively. No tensor-based image pipeline; videos stay as file paths throughout.

## Architecture

### Approach

Monolithic Review Node + simple pipeline nodes. One central **VideoReview** node embeds the full interactive player/timeline/region table as a large DOM widget. Other nodes (Scan, Train, Export) are headless pipeline nodes that pass lightweight metadata.

### Core reuse

The entire `8-cut/core/` package is Qt-free and reusable as-is:
- `core/audio_scan.py` — `scan_video()`, `train_classifier()`, `load_classifier()`
- `core/db.py` — `ProcessedDB` (SQLite, all scan/training/export persistence)
- `core/ffmpeg.py` — `build_ffmpeg_command()` (clip export)
- `core/tracking.py` — YOLO-based subject tracking
- `core/paths.py` — path helpers, `format_time()`

No porting required — these are imported directly.

---

## Node Pack Structure

```
ComfyUI-8cut/
  __init__.py                    # NODE_CLASS_MAPPINGS, WEB_DIRECTORY
  core/                          # symlink or copy of 8-cut/core/
  data/
    8cut.db                      # separate SQLite DB (can copy from ~/.8cut.db)
  models/                        # trained classifiers (.joblib)
  nodes/
    load_video.py
    audio_scan.py
    video_review.py
    train_model.py
    export_clips.py
  server_routes.py               # custom API routes
  web/
    js/
      video_review.js            # timeline + player + scan panel widget
```

---

## Custom Types

No tensors anywhere in the pipeline. All data flows as lightweight metadata:

| Type | Python value | Purpose |
|------|-------------|---------|
| `VIDEO_PATH` | `str` (absolute path) | Video file reference |
| `SCAN_REGIONS` | `list[dict]` with start/end/score/model/disabled | Scan output / review edits |
| `SCAN_MODEL` | `str` (path to .joblib) | Trained classifier |

---

## Nodes

### LoadVideo

| | |
|---|---|
| **Input** | `video_path` (STRING, file browser), `profile` (STRING combo from DB profiles) |
| **Output** | `VIDEO_PATH`, `filename` (STRING) |
| **Logic** | Validates path exists, returns it. Populates profile combo via API route. |

### AudioScan

| | |
|---|---|
| **Input** | `VIDEO_PATH`, `SCAN_MODEL`, `threshold` (FLOAT 0-1), `hop` (FLOAT) |
| **Output** | `SCAN_REGIONS` |
| **Logic** | Calls `core.audio_scan.scan_video()` directly. Progress via `PromptServer.send_sync("progress", ...)`. |

### VideoReview (interactive, blocking)

| | |
|---|---|
| **Input** | `VIDEO_PATH`, `SCAN_REGIONS` (optional) |
| **Output** | `SCAN_REGIONS` (edited) |
| **OUTPUT_NODE** | `True` |
| **Logic** | Execution pauses here. User interacts via the widget. Clicks "Continue" to pass edited regions downstream. |

The widget layout:

```
+-------------------------------------+
|  [video player (HTML5 <video>)]     |
|  +- timeline with scan regions ----+|
|  |  cursor + region drag/resize    ||
|  +---------------------------------+|
|  +- model tabs [EAT_LARGE][HuBERT]+|
|  | Time   | End    | Score         ||
|  | 1:23   | 1:31   | 0.92          ||
|  | 3:45   | 3:53   | 0.87          ||
|  | [Add Negative] [Export] [Continue]|
|  +---------------------------------+|
+-------------------------------------+
```

Widget size: ~640x500px minimum, resizable via LiteGraph.

**Blocking mechanism**: The node's `run()` method blocks on a server-side event/queue. The frontend signals completion via `POST /8cut/review_done/{node_id}`, which unblocks `run()` and returns the edited `SCAN_REGIONS`.

### TrainModel

| | |
|---|---|
| **Input** | `profile` (STRING combo), `positive_folder` (STRING combo), `negative_folder` (STRING combo, optional), `embed_model` (STRING combo from `_EMBED_MODELS`), `use_hard_negatives` (BOOL) |
| **Output** | `SCAN_MODEL` |
| **Logic** | Queries `db.get_training_data()` to assemble `video_infos`, calls `core.audio_scan.train_classifier()`. Saves to `models/{profile}_{embed_model}.joblib` with version rotation. Progress via ComfyUI progress bar. |

### ExportClips

| | |
|---|---|
| **Input** | `VIDEO_PATH`, `SCAN_REGIONS`, `output_folder` (STRING), `short_side` (INT), `format` (combo MP4/WEBM), `spread` (FLOAT), `clip_count` (INT), `fuse_gap` (FLOAT) |
| **Output** | exported file paths (list) |
| **Logic** | Region fusion via `_build_export_spans()`, then `core.ffmpeg.build_ffmpeg_command()` per clip. Records each clip in DB via `db.add()`. |

### Typical workflow

```
[LoadVideo] --> [AudioScan] --> [VideoReview] --> [ExportClips]
                    ^
              [TrainModel]
```

### Training loop (hard negatives round-trip)

1. Scan with existing model -> regions in VideoReview
2. Review -> mark false positives as negatives (DB)
3. Train -> new model uses hard negatives
4. Rescan -> better results
5. Repeat

---

## API Routes

### Video serving

| Route | Method | Purpose |
|-------|--------|---------|
| `/8cut/video` | GET | Serve raw video file via `web.FileResponse`. Query param: `path`. Browser decodes mp4/h264 natively — key for remote streaming. |
| `/8cut/video_transcode` | GET | Fallback: transcode to webm on-the-fly via ffmpeg `StreamResponse` for browser-incompatible formats (some MKV, odd codecs). |

### Region editing (from VideoReview widget)

| Route | Method | Purpose |
|-------|--------|---------|
| `/8cut/toggle_region` | POST | `toggle_scan_result_disabled()` |
| `/8cut/resize_region` | POST | `update_scan_result()` |
| `/8cut/delete_region` | POST | `delete_scan_result()` |
| `/8cut/add_negatives` | POST | `add_hard_negatives()` |
| `/8cut/scan_versions` | GET | `get_scan_versions()` |
| `/8cut/review_done/{node_id}` | POST | Unblock the VideoReview node's `run()`, pass final regions |

### Data queries (for combo widget population)

| Route | Method | Purpose |
|-------|--------|---------|
| `/8cut/profiles` | GET | `db.get_profiles()` |
| `/8cut/export_folders` | GET | `db.get_export_folders()` |
| `/8cut/models` | GET | List available `.joblib` models |

---

## Frontend JS Widget (`web/js/video_review.js`)

Registered via `app.registerExtension()`. Hooks into the VideoReview node's `onNodeCreated` and `onExecuted` callbacks.

### Components

1. **Video player** — HTML5 `<video>` element, src pointed at `/8cut/video?path=...`
2. **Timeline** — `<canvas>` overlay below the video. Renders:
   - Scan region rectangles (color-coded by score, red for negatives, gray for disabled)
   - Cursor line (click to seek)
   - Drag handles on region edges (resize)
   - Waveform (optional, fetched via separate route)
3. **Region table** — HTML table with model tabs. Click row to seek. Columns: Time, End, Score.
4. **Action buttons** — Add Negative, Export, Continue
5. **Version combo** — dropdown to switch scan history versions

### Interaction flow

- Widget activates when `onExecuted` fires with scan regions
- User clicks/drags timeline, edits regions, marks negatives
- Each edit hits an API route (immediate DB persistence)
- "Continue" sends `POST /8cut/review_done/{node_id}` with final region state
- Node's `run()` unblocks, passes `SCAN_REGIONS` downstream

---

## DB

Separate SQLite DB at `ComfyUI-8cut/data/8cut.db`. Uses the existing `ProcessedDB` class unchanged — same schema, same migration code. Users can copy their existing `~/.8cut.db` to carry over scan history, training data, and hard negatives.

---

## Dependencies

Same as 8-cut's `requirements.txt` minus PyQt6/python-mpv:
- `torch`, `torchaudio`, `torchvision` (from CUDA index)
- `transformers>=4.30,<5.0`, `timm>=0.9`
- `librosa`, `scikit-learn`, `joblib`, `soundfile`, `numpy`
- `ultralytics` (YOLO tracking)

ComfyUI already provides torch. The node pack's install script just needs the audio/ML extras.

---

## Implementation Priority

1. **Node pack skeleton** — structure, `__init__.py`, custom types, API routes for video serving
2. **LoadVideo + AudioScan** — headless nodes, no widget needed yet
3. **VideoReview widget (minimal)** — video player + static region display + Continue button
4. **VideoReview interactivity** — timeline click/drag, region editing, negative marking
5. **TrainModel + ExportClips** — complete the pipeline
6. **Polish** — version history, waveform overlay, transcode fallback