Diagnosis: learned tokens grew to norm ~3.2 while real CLIP content tokens
sit at ~1.0. Model never trained on embeddings that large — activates buzz
artifact instead of semantic style shift.
Fix: measure mean token norm from content positions (1–20) of dataset CLIP
embeddings at startup, clamp learned_tokens per-token after every optimizer
step to max 1.5× that reference (50% headroom). Token norm is now logged
as current/limit for easy monitoring.
ti_sweep_1.json: rebuild around norm_clamp group — n4_clamped (primary
diagnostic), prefix_clamped, n8_prefix_clamped, warm_clamped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
n4_baseline showed token_norm growing linearly without plateau — classic
sign of lr too high relative to parameter count. With only K×1024 params,
gradient signal per param is already high-magnitude; high lr causes
overshoot rather than convergence.
- Default lr: 1e-3 → 2e-4 (matches LoRA working regime)
- Default batch_size: 16 → 4 (more diverse gradients, helps norm saturate)
- ti_sweep_1.json: add lr_batch group (lr_low_b4, lr_mid_b8,
lr_low_b4_prefix, lr_2e3), restructure with clearer groups,
annotate n4_baseline as completed with findings
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Saves baseline.wav + baseline.png in the checkpoint dir using the same
seed as the TI eval samples — direct A/B comparison at every checkpoint
without re-generating the baseline each time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Observation: n4_baseline loss barely moved (1.025→0.965 over 3000 steps),
token_norm grew linearly without plateau — generator likely ignores last-K
CLIP positions (EOS/padding zone) where suffix injects.
Fix: add inject_mode parameter throughout the pipeline:
- "suffix": replace last K positions (original behavior, model may ignore)
- "prefix": replace positions 1:1+K right after BOS — highest attention
weight in CLIP, much stronger gradient signal expected
Changes:
- selva_textual_inversion_trainer.py: _inject_tokens() helper centralises
the torch.cat construction for both modes; used in training loop and eval;
inject_mode stored in checkpoint files
- selva_textual_inversion_loader.py: reads inject_mode from checkpoint,
includes in TEXTUAL_INVERSION bundle
- selva_sampler.py: uses _inject_tokens() via bundle's inject_mode field
- selva_ti_scheduler.py: inject_mode in _PARAM_DEFAULTS, config, and
_train_inner call
- ti_sweep_1.json: updated with prefix_inject group (n4, n8, n4+warm);
n4_baseline marked completed; suffix experiments retained for comparison
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reuses _draw_loss_curve + _smooth_losses + _pil_to_tensor from the LoRA
trainer — raw loss in light blue, smoothed overlay in blue, matches the
LoRA trainer's visual style.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SelvaTiScheduler: runs a JSON-defined sweep of TI training experiments,
loading the dataset once and reusing it across runs
- Collects per-experiment loss history, final/min loss, stability metric
(loss_std_last_quarter), and duration — written to experiment_summary.json
after each completed run so partial sweeps survive interruption
- Resume-aware: skips experiments already marked completed in an existing
summary file
- Outputs smoothed loss comparison chart (same axes, one curve per experiment)
- SelvaTextualInversionTrainer._train_inner now returns a dict
{embeddings_path, loss_history} so the scheduler can read results;
train() extracts just the path for ComfyUI
JSON format: name, description, data_dir, output_root, base config,
experiments list with id + param overrides
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Wrap _spectral_metrics + _save_spectrogram in try-except so a matplotlib
or STFT error doesn't abort the checkpoint save (matches LoRA trainer)
- Remove unused `import math` and `_pil_to_tensor` import
- Drop dead `img` variable (_save_spectrogram returns None)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace in-place text_clip assignment with torch.cat so the computation
graph correctly links text_input → learned_tokens; in-place assignment
into a requires_grad=False leaf severs the graph and learned_tokens
receives no gradients
- _spectral_metrics(wav, sr): was passing wav.unsqueeze(0) [1,1,L] instead
of wav [1,L]; stft mean(dim=1) would return wrong shape [1,T] not [n_freqs]
- _save_spectrogram(wav, sr, ...): was passing wav.squeeze(0) [L] (1D)
instead of wav [1,L] as the function expects
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Learns K CLIP token embeddings ([K, 1024]) with all model weights frozen,
keeping generated latents on the decoder's natural manifold — avoids the
quality degradation that affects LoRA on BJ's audio dataset.
- selva_textual_inversion_trainer.py: trains learned_tokens via AdamW,
injects into last K positions of 77-token CLIP embedding, checkpoints
with eval audio + spectral metrics
- selva_textual_inversion_loader.py: loads .pt bundle, returns
TEXTUAL_INVERSION dict for sampler
- selva_sampler.py: optional textual_inversion input; injects into both
text_clip and neg_text_clip before preprocess_conditions
- __init__.py: registers both new nodes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ethanfel