ComfyUI-SelVA/docs/plans/2026-04-04-selva-integration-design.md

# SelVA Integration Design

**Date:** 2026-04-04
**Branch:** feature/selva-integration (new from master)
**Status:** Approved, ready for implementation

---

## Problem

PrismAudio's sync conditioning is text-agnostic: Synchformer extracts features from
all visual motion equally. In multi-source videos (person walking near a car), the DiT
receives unfocused sync guidance and struggles to match audio events to the correct
visual source.

SelVA (CVPR 2026, arXiv:2512.02650) solves this with TextSynchformer — text conditioning
is injected inside the Synchformer encoder via cross-attention, so sync features only
encode motion relevant to the requested sound. This is the core architectural improvement
needed for reliable V2A sync.

---

## Architecture

### New directory layout

```
selva_core/          ← vendored SelVA source (model + ext + utils)
nodes/
  selva_model_loader.py
  selva_feature_extractor.py
  selva_sampler.py
```

### New custom types

- `SELVA_MODEL` — `{generator, video_enc, feature_utils, variant, strategy, dtype}`
- `SELVA_FEATURES` — `{clip_features, sync_features, duration}`

### No subprocess

SelVA is pure PyTorch. Feature extraction runs inline in ComfyUI — no managed venv,
no JAX/TF, no pip install on first run.

### Dependencies

Zero new pip packages. ComfyUI already ships:
- `open_clip_torch` (CLIP ViT-H-14-384, auto-downloads via `hf-hub:` on first use)
- `transformers` (flan-t5-base, auto-downloads from HuggingFace on first use)
- `torch`, `torchaudio`, `einops`

---

## Nodes

### `SelvaModelLoader` → `SELVA_MODEL`

| Input | Type | Default | Notes |
|---|---|---|---|
| variant | dropdown | medium_44k | small_16k / small_44k / medium_44k / large_44k |
| precision | dropdown | bf16 | bf16 / fp16 / fp32 |
| offload_strategy | dropdown | auto | auto / keep_in_vram / offload_to_cpu |

Resolves weights from `models/selva/`. Raises descriptive errors with download
instructions if files are missing.

### `SelvaFeatureExtractor` → `SELVA_FEATURES`, `FLOAT` (fps)

| Input | Type | Default | Notes |
|---|---|---|---|
| video | IMAGE | — | ComfyUI video tensor [T,H,W,C] |
| prompt | STRING | — | Used by TextSynchformer to select relevant motion |
| video_info | VHS_VIDEOINFO | opt | Auto-sets fps when connected |
| fps | FLOAT | 30.0 | Fallback fps if video_info not connected |
| cache_dir | STRING | "" | Empty = system temp dir |

Feature extraction steps (all inline, no subprocess):
1. Resize frames to 384×384 → CLIP video features `[B, T, 1024]`
2. Resize frames to 224×224 + encode prompt with flan-T5 → TextSynchformer → text-conditioned sync features `[B, T, 768]`
3. Save to `.npz` cache keyed by hash(frames[:1MB] + prompt + fps)

### `SelvaSampler` → `AUDIO`

| Input | Type | Default | Notes |
|---|---|---|---|
| model | SELVA_MODEL | — | |
| features | SELVA_FEATURES | — | |
| prompt | STRING | — | Should match extractor prompt; drives CLIP text guidance |
| negative_prompt | STRING | "" | Steers away from unwanted sounds |
| duration | FLOAT | 0.0 | 0 = auto from features duration |
| steps | INT | 25 | Euler steps (25 is SelVA default, fast) |
| cfg_strength | FLOAT | 4.5 | CFG scale (SelVA default) |
| seed | INT | 0 | |

Generation steps:
1. Encode prompt → CLIP text features (for MMAudio)
2. Encode negative prompt → empty conditions for CFG
3. `net_generator.preprocess_conditions(clip_f, sync_f, text_clip)`
4. Flow matching Euler ODE (`num_steps` iterations) with CFG
5. `feature_utils.decode(latent)` → mel spectrogram
6. `feature_utils.vocode(spec)` → waveform (BigVGAN for 16k, direct for 44k)

**Note on dual prompt:** The extractor prompt is baked into sync_features via
TextSynchformer at extraction time. The sampler prompt drives CLIP text conditioning
at generation time. They should match — a tooltip explains this.

---

## Data Flow

```
[VHS LoadVideo] ──► [SelvaFeatureExtractor]
                         │  prompt: "dog barking"
                         │  video_info: (fps auto)
                         ▼
                    SELVA_FEATURES
                    {clip_features [B,T,1024],
                     sync_features [B,T,768],  ← text-conditioned
                     duration: 8.2s}
                         │
[SelvaModelLoader] ──► [SelvaSampler]
  variant: medium_44k    │  prompt: "dog barking"
  precision: bf16        │  negative: "wind noise"
                         │  cfg_strength: 4.5, steps: 25
                         ▼
                       AUDIO (44.1kHz or 16kHz)
```

---

## Model Weights

Location: `models/selva/`

```
video_enc_sup_5.pth                  ← TextSynch, shared across all variants
generator_small_16k_sup_5.pth
generator_small_44k_sup_5.pth
generator_medium_44k_sup_5.pth
generator_large_44k_sup_5.pth
ext/
  v1-16.pth                          ← VAE for 16k variants
  v1-44.pth                          ← VAE for 44k variants
  best_netG.pt                       ← BigVGAN vocoder (16k only)
```

`synchformer_state_dict.pth` is reused from `models/prismaudio/` — no duplicate.

---

## selva_core vendoring

Copy from `jnwnlee/selva` (pinned to a specific commit for stability):
- `selva_core/model/` — MMAudio, TextSynch, transformer layers, embeddings, flow matching
- `selva_core/ext/` — autoencoder, BigVGAN, synchformer, rotary embeddings, mel converters
- `selva_core/utils/` — transforms, generate() helper

Rename all internal imports from `selva.*` → `selva_core.*`.

---

## What stays the same

- All PrismAudio nodes unchanged
- `models/prismaudio/` unchanged
- Synchformer checkpoint shared (not duplicated)
- Branch: new `feature/selva-integration` off master (LoRA work stays separate)