feat: add inject_mode (suffix/prefix) to TI pipeline
Observation: n4_baseline loss barely moved (1.025→0.965 over 3000 steps), token_norm grew linearly without plateau — generator likely ignores last-K CLIP positions (EOS/padding zone) where suffix injects. Fix: add inject_mode parameter throughout the pipeline: - "suffix": replace last K positions (original behavior, model may ignore) - "prefix": replace positions 1:1+K right after BOS — highest attention weight in CLIP, much stronger gradient signal expected Changes: - selva_textual_inversion_trainer.py: _inject_tokens() helper centralises the torch.cat construction for both modes; used in training loop and eval; inject_mode stored in checkpoint files - selva_textual_inversion_loader.py: reads inject_mode from checkpoint, includes in TEXTUAL_INVERSION bundle - selva_sampler.py: uses _inject_tokens() via bundle's inject_mode field - selva_ti_scheduler.py: inject_mode in _PARAM_DEFAULTS, config, and _train_inner call - ti_sweep_1.json: updated with prefix_inject group (n4, n8, n4+warm); n4_baseline marked completed; suffix experiments retained for comparison Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -3,6 +3,7 @@ import comfy.utils
|
||||
import comfy.model_management
|
||||
|
||||
from .utils import SELVA_CATEGORY, get_device, get_offload_device, soft_empty_cache
|
||||
from .selva_textual_inversion_trainer import _inject_tokens
|
||||
|
||||
|
||||
class SelvaSampler:
|
||||
@@ -118,16 +119,15 @@ class SelvaSampler:
|
||||
neg_text_clip = feature_utils.encode_text_clip([negative_prompt]) \
|
||||
if negative_prompt.strip() else None
|
||||
|
||||
# Inject textual inversion tokens into last K positions of CLIP embedding
|
||||
# Inject textual inversion tokens into CLIP conditioning
|
||||
if textual_inversion is not None:
|
||||
emb = textual_inversion["embeddings"].to(device, dtype) # [K, 1024]
|
||||
K = emb.shape[0]
|
||||
text_clip = text_clip.clone()
|
||||
text_clip[:, -K:, :] = emb.unsqueeze(0)
|
||||
emb = textual_inversion["embeddings"].to(device, dtype) # [K, 1024]
|
||||
K = emb.shape[0]
|
||||
inject_mode = textual_inversion.get("inject_mode", "suffix")
|
||||
text_clip = _inject_tokens(text_clip, emb, K, inject_mode)
|
||||
if neg_text_clip is not None:
|
||||
neg_text_clip = neg_text_clip.clone()
|
||||
neg_text_clip[:, -K:, :] = emb.unsqueeze(0)
|
||||
print(f"[SelVA] Textual inversion: injected {K} tokens into CLIP conditioning",
|
||||
neg_text_clip = _inject_tokens(neg_text_clip, emb, K, inject_mode)
|
||||
print(f"[SelVA] Textual inversion: {K} tokens mode={inject_mode}",
|
||||
flush=True)
|
||||
|
||||
conditions = net_generator.preprocess_conditions(clip_f, sync_f, text_clip)
|
||||
|
||||
Reference in New Issue
Block a user