Ethanfel/ComfyUI-Prompt-Calibrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add describe (first-pass) mode to the judge node

Browse Source 
New mode on QwenVLImageJudge: 'describe' looks at the reference alone and returns
a prompt-ready caption + per-axis target spec to seed the very first prompt (the
generator has nothing to reproduce yet). 'compare' is the existing ref-vs-gen
scoring. generated_image is now optional (required only for compare); shared
generation refactored into _generate_from_messages; third output renamed
diff_analysis -> analysis (mode-agnostic). agent_bridge gains --mode (describe
needs no receptor/prompt); added workflow_describe_api.json. Docs updated with the
first-pass bootstrap step. Fixed error-return arity to 5-tuple.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Ethanfel
2026-06-26 23:04:09 +02:00
parent 959ec70065
commit c7ef756a71
6 changed files with 211 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+21 -13
View File
@@ -31,11 +31,12 @@ can act on it.
| name | type | default | notes |
|---|---|---|---|
| `reference_image` | IMAGE | — | the target |
| `generated_image` | IMAGE | — | the candidate to score |
| `mode` | compare / describe | compare | `describe` = first pass over the reference only → caption + target spec (seeds the prompt). `compare` = score ref vs generated |
| `generated_image` | IMAGE (optional) | — | the candidate to score (required for `compare`, ignored for `describe`) |
| `model_path` | STRING | `/media/p5/qwen3vl_4b_abliterated_comfy_convert/hf_bf16` | local dir, **HF repo id** (`org/name`), or alias (`30b-a3b` / `8b` / `4b`) |
| `precision` | bf16 / fp16 / fp8 / nf4 | bf16 | `nf4` = 4-bit (run the 30B judge on 32 GB); `fp8` with the `hf_fp8` copy |
| `axes` | STRING | ~20 axes (identity, body, wardrobe, action, affect, camera, render) | scored axes; granular for explicit content. Edit to taste |
| `max_new_tokens` | INT | 512 | |
| `axes` | STRING | ~20 axes (identity, body, wardrobe, action, affect, camera, render) | scored/described axes; granular for explicit content. Edit to taste |
| `max_new_tokens` | INT | 1024 | |
| `temperature` | FLOAT | 0.0 | 0 = greedy/repeatable |
| `swap_eval` | BOOL | true | run twice with images swapped, average → cuts position bias |
| `keep_loaded` | BOOL | true | cache weights across loop iterations |
@@ -50,10 +51,11 @@ default skip download entirely.
| name | type | use |
|---|---|---|
| `overall_score` | FLOAT 0..1 | loop stop-condition / objective |
| `axis_scores_json` | STRING (JSON) | per-axis `{score, ref, gen}` — target vs current, for the agent |
| `diff_analysis` | STRING | readable summary, worst axes first (`score ref:[…] gen:[…]`) |
| `overall_score` | FLOAT 0..1 | compare: loop stop-condition / objective. describe: `1.0` placeholder |
| `axis_scores_json` | STRING (JSON) | compare: per-axis `{score, ref, gen}`. describe: per-axis target values `{axis: value}` |
| `analysis` | STRING | compare: summary, worst axes first (`score ref:[…] gen:[…]`). describe: the prompt-ready `caption` |
| `raw` | STRING | raw model output (both passes if `swap_eval`) |
| `report_path` | STRING | path to the written `calib_<tag>.json` |
## Install
@@ -91,20 +93,26 @@ black-box optimizer → LLM-in-the-loop) are in the methodology doc.
## End-to-end loop
1. Run ComfyUI with `--listen`, install this node pack, put your reference at `ComfyUI/input/reference.png`.
2. Load `workflow/workflow_api.json` (SDXL `waiIllustriousSDXL_v160` example — swap the checkpoint for Flux/Krea as needed).
3. Drive it from your agent following `docs/CALIBRATION_POLICY.md`:
2. **First pass (describe):** the judge looks at the reference alone and returns a prompt-ready
`caption` + per-axis target spec to seed the initial prompt:
```bash
python agent_bridge.py --mode describe --workflow workflow/workflow_describe_api.json \
--run-tag seed --analysis-dir /media/p5/Comfyui/output/calibrator
```
3. **Compare loop:** load `workflow/workflow_api.json` (SDXL `waiIllustriousSDXL_v160` example —
swap the checkpoint for Flux/Krea as needed) and iterate, following `docs/CALIBRATION_POLICY.md`:
```bash
python agent_bridge.py --workflow workflow/workflow_api.json \
--prompt "1 woman, red lingerie, bedroom, full body, warm light" \
--prompt "<caption from step 2, then calibrated>" \
--run-tag iter001 --analysis-dir /media/p5/Comfyui/output/calibrator
```
stdout = the analysis JSON → agent calibrates → next iteration.
stdout = the analysis JSON (`{score, ref, gen}` per axis) → agent steers toward `ref` → next iteration.
## Status
- [x] Methodology + node selection (`docs/METHODOLOGY.md`)
- [x] Qwen3-VL Image Judge node (structured JSON scoring, swap-eval, model caching, file report)
- [x] Agent-driven architecture (`docs/AGENT_LOOP.md`) — Receptor node + `agent_bridge.py`
- [x] Example end-to-end workflow (`workflow/workflow_api.json`)
- [x] Qwen3-VL Image Judge node — `describe` (first pass) + `compare` (scoring), swap-eval, file report
- [x] Agent-driven architecture (`docs/AGENT_LOOP.md`) — Receptor node + `agent_bridge.py` (`--mode`)
- [x] Example workflows: `workflow_describe_api.json` (first pass) + `workflow_api.json` (compare loop)
- [x] Agent calibration policy (`docs/CALIBRATION_POLICY.md`)
- [ ] Optional: structured-config receptor (carry Prompt-Builder knobs instead of a flat string)