fix(ti): lower default lr/batch, add lr_batch sweep group
n4_baseline showed token_norm growing linearly without plateau — classic sign of lr too high relative to parameter count. With only K×1024 params, gradient signal per param is already high-magnitude; high lr causes overshoot rather than convergence. - Default lr: 1e-3 → 2e-4 (matches LoRA working regime) - Default batch_size: 16 → 4 (more diverse gradients, helps norm saturate) - ti_sweep_1.json: add lr_batch group (lr_low_b4, lr_mid_b8, lr_low_b4_prefix, lr_2e3), restructure with clearer groups, annotate n4_baseline as completed with findings Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -175,12 +175,12 @@ class SelvaTextualInversionTrainer:
|
||||
"tooltip": "Training steps. 3000 is a reasonable starting point.",
|
||||
}),
|
||||
"lr": ("FLOAT", {
|
||||
"default": 1e-3, "min": 1e-5, "max": 1e-1, "step": 1e-5,
|
||||
"tooltip": "Learning rate. 1e-3 is a good default for textual inversion (higher than LoRA since there are far fewer parameters).",
|
||||
"default": 2e-4, "min": 1e-5, "max": 1e-1, "step": 1e-5,
|
||||
"tooltip": "Learning rate. 2e-4 matches the LoRA working regime. Higher LR (1e-3) causes token norm to drift without plateauing on small datasets.",
|
||||
}),
|
||||
"batch_size": ("INT", {
|
||||
"default": 16, "min": 1, "max": 64,
|
||||
"tooltip": "Clips sampled per training step.",
|
||||
"default": 4, "min": 1, "max": 64,
|
||||
"tooltip": "Clips sampled per training step. Smaller batch (4–8) gives more diverse gradients and helps token norm saturate rather than drift.",
|
||||
}),
|
||||
"seed": ("INT", {"default": 42, "min": 0, "max": 0xFFFFFFFF}),
|
||||
"save_every": ("INT", {
|
||||
|
||||
Reference in New Issue
Block a user