Learns K CLIP token embeddings ([K, 1024]) with all model weights frozen,
keeping generated latents on the decoder's natural manifold — avoids the
quality degradation that affects LoRA on BJ's audio dataset.
- selva_textual_inversion_trainer.py: trains learned_tokens via AdamW,
injects into last K positions of 77-token CLIP embedding, checkpoints
with eval audio + spectral metrics
- selva_textual_inversion_loader.py: loads .pt bundle, returns
TEXTUAL_INVERSION dict for sampler
- selva_sampler.py: optional textual_inversion input; injects into both
text_clip and neg_text_clip before preprocess_conditions
- __init__.py: registers both new nodes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two ComfyUI nodes to reduce domain mismatch between custom training audio
and the MMAudio VAE's expected spectral distribution:
SelvaHfSmoother: blends a low-pass filtered copy (biquad) with the original
at a configurable cutoff and blend ratio. Attenuates extreme HF content that
BigVGANv2 handles poorly. RMS-preserving.
SelvaSpectralMatcher: computes the log-mel energy profile of the clip,
compares it per-band to the VAE's normalization means (DATA_MEAN_80D/128D),
and applies a smooth STFT-domain gain correction to match the codec's training
distribution. Configurable strength and max_gain_db clamp. RMS-preserving.
Recommended workflow: SpectralMatcher → HfSmoother → feature extraction.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AutoEncoderModule unconditionally asserts vocoder_ckpt_path is not None
even when need_vae_encoder=True. Pass best_netG.pt to satisfy the assert;
the vocoder weights are not actually used since decode+vocode go through
model["feature_utils"].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Load fresh FeaturesUtils only for encoding; use model["feature_utils"] for
decode+vocode to mirror the exact path the sampler takes
- Apply generator.normalize() → unnormalize() around the encoded latent so the
decoder receives latents in the same space it expects from inference
- Log both encoded and norm→unnorm latent stats to diagnose round-trip fidelity
- Normalize output to -27 dBFS (matching training clip RMS) and clamp to [-1, 1]
to prevent clipping artifacts in the output waveform
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without this the decoder produced 7s instead of 8s due to STFT rounding.
Same fix as _prepare_dataset uses for training data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Encodes audio through the VAE then decodes straight back, bypassing the
diffusion model entirely. Use this to isolate whether saturation artifacts
are introduced by the codec reconstruction (VAE/DAC) or by the LoRA.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _eval_sample gains clip_idx param (default 0, backward compatible)
- Evaluator loops over all dataset clips per adapter, saves one WAV per clip
- Reference metrics computed for all clips and averaged
- Comparison chart and summary use avg_metrics across all clips
- Eliminates bias from evaluating on an unrepresentative single clip
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
shutil.copy2 was writing FLAC binary to reference.wav — unplayable.
Now copies as reference{.flac/.wav/etc} matching the source extension.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Loads the first clip's original audio (same clip used for inference),
copies it to output_dir/reference.wav, runs spectral metrics and
saves a spectrogram. Appears first in the comparison chart so generated
samples can be judged against the target sound.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generates audio samples from a list of adapters against a fixed reference
clip, collects spectral metrics for each, and outputs a comparison bar
chart + eval_summary.json. Useful for comparing sweep candidates before
committing to a next round of training.
JSON format: name, data_dir, output_dir, steps, seed, adapters[{id, path}].
Empty path = baseline (no LoRA).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Training clips at -23 LUFS measure -25 to -31 dBFS RMS (avg ~-27).
Normalizing output to -23 dBFS was 4-8 dB too loud, causing saturation
on clips with high crest factor and peaks near 0 dBFS.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
normalize() uses in-place ops so it worked, but reading the return value
makes the intent clear and guards against future refactors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add lr_schedule param (constant|cosine) to SelvaLoraTrainer
- Cosine decays LR from initial value to ~0 after warmup, preventing
the oscillation observed at steps 6000-8000 with lr=2e-4 flat
- Wire lr_schedule through scheduler _PARAM_DEFAULTS and _train_inner call
- Add g5_r128_lr_2e4_cosine and g5_r128_lr_3e4_cosine to r128_sweet_spot sweep
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New node: SelVA Skip Experiment — writes skip_current.flag from UI,
queue in a second workflow tab while scheduler is running
- SkipExperiment now attaches partial loss/grad/spectral data to the
exception so the scheduler saves all collected scalars in the summary
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Create the flag file in the sweep output_root to skip the running
experiment at the next log interval (every 50 steps):
touch /path/to/experiment/skip_current.flag
Scheduler marks it as 'skipped' in the summary and continues.
Skipped experiments are NOT resumed on restart (unlike failed ones).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Computes hf_energy_ratio (>4kHz), spectral_centroid_hz, spectral_rolloff_hz
at each save_every checkpoint. Logged to console and stored in
experiment_summary.json under results.spectral_metrics[step].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Log-frequency dB spectrogram (inferno colormap, 100Hz–16kHz) saved as
step_XXXXX.png next to step_XXXXX.wav in samples/ subfolder.
Makes high-frequency rolloff (low bitrate signature) immediately visible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
RMS normalize to target then scale back if peaks exceed 1.0,
preserving dynamics instead of hard-clipping transients.
Eval sample target updated to -23 dBFS to match training data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Peak norm was slamming output to full scale regardless of content level,
making generated audio several times louder than training clips.
RMS norm to -20 dBFS matches typical processed audio level.
Sampler exposes target_lufs (-40 to -6, default -20) for user control.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Inference is fast on RTX PRO 6000 — 8 steps was washing out quality
differences between experiments.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scheduler: on re-run, reads existing experiment_summary.json and skips
already-completed experiments — safe to stop and restart mid-sweep.
tier1_thorough: adds g5 (lr 3e-5/3e-4), g6 (full target attn.qkv+linear1
at r16 and r64), and g4_full_r64_6k (6000-step extended run) — 17 total.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Always sample dataset[0] with fixed noise seed so checkpoints are
directly comparable (hear the model improve step by step)
- Save to output_dir/samples/step_XXXXX.wav instead of alongside checkpoints
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dataset browser: audio/features now resolve through features/ subdir
- tier1_sweep.json: update data_dir to BJ dataset path
- tier1_thorough.json: 12-experiment overnight sweep across 4 groups
(rank 16/32/64, alpha scaling, LoRA+/dropout/curriculum isolation,
full Tier 1 stack at r16 and r64) — output to BJ/experiment/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Companion node for inspecting dataset.json entries by integer index.
Outputs video (.mp4), audio (.wav/.flac), features (.npz), frames dir,
mask dir, label, and max_index for constraining the index widget range.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- trainer: raise ValueError early when remaining steps < log_interval (50)
instead of UnboundLocalError on smoothed_img/final_path at return
- trainer: use None in grad_norm_history instead of silent 0.0 when
grad_accum > log_interval and no optimizer step fired in the interval
- trainer: include start_step in _train_inner return dict
- scheduler: use start_step from result dict for min_loss_step and
loss_at_steps (fixes wrong step labels on resumed experiments)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
trainer:
- Track gradient norm before clipping at each optimizer step
- Log avg grad_norm per log_interval alongside loss in console output
- Include grad_norm_history in _train_inner return dict
scheduler:
- Add system block to summary (GPU name, VRAM, torch/CUDA version)
- Include full loss_history and grad_norm_history arrays in each
experiment result (50-step resolution, not just save_every checkpoints)
- Add loss_std_last_quarter stability metric (std dev of raw loss over
last 25% of steps — high value indicates unstable training)
- Add log_interval field so consumers know the x-axis resolution
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract _prepare_dataset() from SelvaLoraTrainer.train() as a module-level
function so the dataset can be encoded once and reused across experiments
- Change _train_inner() return value from tuple to dict (adds loss_history,
meta, completed; train() unpacks for ComfyUI — no change to node outputs)
- New SelvaLoraScheduler node: reads a JSON sweep file, runs N experiments
sequentially, writes experiment_summary.json (updated after each run) and
loss_comparison.png with all smoothed curves overlaid on the same axes
- Register SelvaLoraScheduler in nodes/__init__.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- LoRA dropout: applied to the LoRA path only (not frozen base weights),
0.05–0.1 helps regularize on small datasets (arXiv:2404.09610)
- LoRA+: separate optimizer param groups for lora_A and lora_B with
configurable LR ratio; ratio=16 enables LoRA+ (arXiv:2402.12354)
- Curriculum mode: logit_normal for first N% of steps then uniform,
directly addresses early convergence + fine-detail degradation at
boundaries (arXiv:2603.12517)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
logit_normal reaches lower loss but perceptual improvement over uniform
is dataset-dependent. Keeping uniform as default to match original MMAudio
training behavior; logit_normal remains available as an option.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On Windows, /folder is drive-relative (no drive letter) rather than a real
absolute path. Redirect these to ComfyUI's output directory so files don't
land at C:\folder. Also redirects plain relative paths (e.g. lora_output)
to output/ instead of the process working directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wraps training loop in try/finally so adapter_final.pt and loss PNGs are
always written. On cancellation the adapter is named
adapter_cancelled_stepXXXXX.pt so it can be used with --resume.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clips from shorter videos produce fewer CLIP frames (e.g. 2s → 16 frames,
8s → 64 frames). Mixed-length datasets would cause torch.stack() to fail
during batching. Normalize to seq_cfg.clip_seq_len / sync_seq_len at load,
same as latents are already normalized to latent_seq_len.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Uniform timestep sampling undertrained t>0.8 (the final denoising steps),
leaving residual noise that CFG amplifies at inference. Logit-normal sampling
concentrates training near t=0.5 while still covering the full range, improving
high-t coverage and reducing noise floor in generated audio.
Default changed from uniform to logit_normal (sigma=1.0). Previous behavior
available with timestep_mode=uniform.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces single-sample steps with batched sampling via random.choices().
Tensors are stacked to [B, T, C] before the forward pass; t is now [B].
Default grad_accum lowered to 1 since real batching gives stable gradients.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Raw curve shown in light blue, EMA-smoothed (beta=0.9) overlay in darker
blue. Both saved as PNG at end of training. The node IMAGE output now
returns the smoothed version. Live preview also uses the smoothed overlay.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
500 warmup steps is 25% of a 2000-step run — too long. 100 steps lets
the full lr kick in much earlier without sacrificing stability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The third element in ComfyUI's preview tuple is max_size in pixels, not
JPEG quality. Passing 85 was capping the live loss curve at 85×40px.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
torch.enable_grad() alone is insufficient: operations on inference tensors
(created inside ComfyUI's outer inference_mode context) produce inference
tensors even inside enable_grad, breaking autograd. inference_mode(False)
exits the inference context so the deepcopy, apply_lora, and training loop
run with a fully clean autograd context.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
torch.enable_grad() re-enables grad tracking but nn.Parameters created while
torch.inference_mode() is active are inference tensors that can't enter autograd
regardless. Splitting into _train_inner() and calling it inside enable_grad()
ensures the deepcopy, apply_lora, and the training loop all run with a clean
autograd context.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ComfyUI executes all nodes inside torch.no_grad(), which prevents gradient
tracking and makes loss.backward() fail. torch.enable_grad() overrides it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
STFT hop-size rounding produces ±1 latent frame vs the expected seq length.
Clamp to seq_cfg.latent_seq_len after transpose so generator.forward assertion passes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Recent torchaudio defaults to torchcodec as the audio backend, which requires
FFmpeg shared libraries. Falls back to soundfile for envs where torchcodec
can't load (e.g. containerised ComfyUI without system FFmpeg).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ethanfel