Add SxCP Audio Prompt Guide node (audio -> waveform image + timing summary)
New node for audio-guided video prompts (LTX): takes a ComfyUI AUDIO clip + free-text motion notes and outputs (1) a rendered energy-envelope IMAGE with beat/segment markers so the vision model can see the audio shape, and (2) an audio_summary STRING with duration, tempo/beats (librosa optional), per-segment energy + 8n+1-snapped frame counts + stage hints + the notes. Wire waveform_image -> Judge.reference_image and audio_summary -> Judge.user_prompt (chat mode, json_output) for beat-aligned LTX JSON. librosa optional (energy envelope + segments work without it). Registered in __init__; README workflow added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -62,6 +62,30 @@ Set **`json_output=true`** for JSON-producing system prompts — it extracts the
|
||||
from the reply (stripping any reasoning, prose, or ```fences) and returns it clean and
|
||||
re-serialized (falls back to raw text if none parses). Works even with `enable_thinking` on.
|
||||
|
||||
## Audio-guided prompts (`SxCP Audio Prompt Guide`)
|
||||
|
||||
For audio-driven video (e.g. LTX prompt-relay timed to music), the `AudioPromptGuide` node
|
||||
turns an audio clip + your free-text motion **notes** into inputs the vision model can use:
|
||||
|
||||
- **`waveform_image`** (IMAGE) — the energy envelope with beat markers + segment boundaries,
|
||||
so the model can *see* the audio's shape.
|
||||
- **`audio_summary`** (STRING) — duration, tempo/beats (if `librosa` is installed), a
|
||||
per-segment energy breakdown with **8n+1-snapped frame counts** and stage hints
|
||||
(establish→build→peak→settle), plus your notes.
|
||||
|
||||
Wire it into the judge node in **chat mode**:
|
||||
|
||||
```
|
||||
LoadAudio ─► SxCP Audio Prompt Guide ─┬─ waveform_image ─► Judge.reference_image
|
||||
(your notes: "fast on the drop") └─ audio_summary ─► Judge.user_prompt
|
||||
(LTX system prompt text node) ─────────────────────────► Judge.system_prompt
|
||||
Judge (mode=chat, json_output=true) ─► LTX beats JSON
|
||||
```
|
||||
|
||||
The model then gets the audio's timing/energy + your motion notes and writes beat durations,
|
||||
camera moves, and deltas that escalate with the music. `librosa` is optional (BPM/beats);
|
||||
without it you still get the energy envelope + segments.
|
||||
|
||||
## Performance / speed
|
||||
|
||||
This node runs models through **transformers `.generate()`** — the simplest path, but the
|
||||
|
||||
Reference in New Issue
Block a user