Add SxCP Audio Prompt Guide node (audio -> waveform image + timing summary)

New node for audio-guided video prompts (LTX): takes a ComfyUI AUDIO clip + free-text
motion notes and outputs (1) a rendered energy-envelope IMAGE with beat/segment markers
so the vision model can see the audio shape, and (2) an audio_summary STRING with
duration, tempo/beats (librosa optional), per-segment energy + 8n+1-snapped frame counts
+ stage hints + the notes. Wire waveform_image -> Judge.reference_image and audio_summary
-> Judge.user_prompt (chat mode, json_output) for beat-aligned LTX JSON. librosa optional
(energy envelope + segments work without it). Registered in __init__; README workflow added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-04 22:02:12 +02:00
co-authored by Claude Opus 4.8
parent 8b567cb531
commit 6e6232d4ab
4 changed files with 204 additions and 2 deletions
+24
View File
@@ -62,6 +62,30 @@ Set **`json_output=true`** for JSON-producing system prompts — it extracts the
from the reply (stripping any reasoning, prose, or ```fences) and returns it clean and
re-serialized (falls back to raw text if none parses). Works even with `enable_thinking` on.
## Audio-guided prompts (`SxCP Audio Prompt Guide`)
For audio-driven video (e.g. LTX prompt-relay timed to music), the `AudioPromptGuide` node
turns an audio clip + your free-text motion **notes** into inputs the vision model can use:
- **`waveform_image`** (IMAGE) — the energy envelope with beat markers + segment boundaries,
so the model can *see* the audio's shape.
- **`audio_summary`** (STRING) — duration, tempo/beats (if `librosa` is installed), a
per-segment energy breakdown with **8n+1-snapped frame counts** and stage hints
(establish→build→peak→settle), plus your notes.
Wire it into the judge node in **chat mode**:
```
LoadAudio ─► SxCP Audio Prompt Guide ─┬─ waveform_image ─► Judge.reference_image
(your notes: "fast on the drop") └─ audio_summary ─► Judge.user_prompt
(LTX system prompt text node) ─────────────────────────► Judge.system_prompt
Judge (mode=chat, json_output=true) ─► LTX beats JSON
```
The model then gets the audio's timing/energy + your motion notes and writes beat durations,
camera moves, and deltas that escalate with the music. `librosa` is optional (BPM/beats);
without it you still get the energy envelope + segments.
## Performance / speed
This node runs models through **transformers `.generate()`** — the simplest path, but the