diff --git a/docs/plans/2026-03-15-civitai-stats-shim-design.md b/docs/plans/2026-03-15-civitai-stats-shim-design.md new file mode 100644 index 0000000..cbc85b4 --- /dev/null +++ b/docs/plans/2026-03-15-civitai-stats-shim-design.md @@ -0,0 +1,157 @@ +# CivitAI Extended Stats Shim — Design + +## Problem + +The LoRA Manager fetches metadata from CivitAI but discards stats (download count, rating, thumbs up). Users want to sort and filter loras by popularity/quality, and see these stats on model cards — like CivitAI's own UI. + +## Constraints + +- No modifications to the original LoRA Manager codebase (avoid merge conflicts on upstream sync) +- Lives inside ComfyUI-LM-Remote, which already has proxy middleware intercepting API responses +- Must handle CivitAI API rate limits gracefully when bulk-fetching stats +- Stats should persist across restarts (SQLite) + +## Architecture + +### Hybrid Shim Approach + +- **Separate SQLite DB** (`civitai_stats.db`) for extended stats — avoids schema conflicts with the manager's own cache +- **Proxy enrichment** — the existing middleware intercepts `/api/lm/loras/list` responses from the remote and merges stats into each item before forwarding to the browser +- **New API endpoints** on LM-Remote for the bulk fetch button and status checks +- **Frontend JS** injected via LM-Remote's `web/comfyui/` directory, patches sort dropdown and card rendering + +### Data Flow + +``` +List request: + Browser → GET /api/lm/loras/list + → proxy fetches response from remote + → enriches each item with stats from local civitai_stats.db + → returns merged response to browser + +Bulk fetch (initial): + Browser → POST /api/lm-extra/fetch-stats + → handler reads models from remote list API + → extracts civitai_model_id + sha256 for each + → batches CivitAI API calls (/api/v1/models?ids=id1,id2,...) + → stores stats in civitai_stats.db + → WebSocket progress updates + → frontend auto-refreshes grid when done + +Piggyback (ongoing): + When proxy intercepts metadata sync responses that contain stats, + opportunistically store them in civitai_stats.db +``` + +## SQLite Schema + +File: `civitai_stats.db` (stored alongside LM-Remote's config) + +```sql +CREATE TABLE model_stats ( + sha256 TEXT PRIMARY KEY, + civitai_model_id INTEGER, + civitai_version_id INTEGER, + download_count INTEGER, + rating REAL, + rating_count INTEGER, + thumbs_up_count INTEGER, + fetched_at REAL +); +``` + +Keyed by `sha256` — the common identifier between the manager's cache and CivitAI. Joins naturally regardless of file paths or renames. + +## API Endpoints (LM-Remote) + +| Method | Path | Purpose | +|--------|------|---------| +| POST | `/api/lm-extra/fetch-stats` | Bulk fetch stats for all models (button) | +| GET | `/api/lm-extra/stats-status` | Check if stats DB is populated, staleness info | + +## Proxy Response Enrichment + +The middleware intercepts the `/api/lm/loras/list` response and: + +1. Parses the JSON response body +2. Collects all `sha256` values from items +3. Batch-queries `civitai_stats.db` (single SELECT with IN clause) +4. Merges `download_count`, `rating`, `rating_count`, `thumbs_up_count` into each item +5. Returns the enriched response + +## Sorting + +### New Sort Options + +Added to the sort dropdown alongside existing options (name, date, size, usage): + +| Value | Label | +|-------|-------| +| `downloads:desc` | Most downloaded | +| `downloads:asc` | Least downloaded | +| `rating:desc` | Highest rated | +| `rating:asc` | Lowest rated | +| `thumbsup:desc` | Most liked | +| `thumbsup:asc` | Least liked | + +### Sort Implementation + +When the proxy sees these sort values in the list request, sorting is handled locally: + +1. Fetch the full (unpaginated) list from remote +2. Join with stats DB +3. Sort locally +4. Apply pagination +5. Return result + +Items without stats sort to the end. + +## Frontend Changes + +All in `ComfyUI-LM-Remote/web/comfyui/`. + +### Card Badges + +Small badges on each model card (styled like the existing base model badge): + +- Download icon + count (e.g., "12.3k") +- Star icon + rating (e.g., "4.8") +- Thumbs-up icon + count (e.g., "89") + +Numbers use compact formatting (1000 → "1k", 12345 → "12.3k"). + +### Toolbar Button + +A "Fetch Stats" button in the toolbar (next to existing Fetch/Download buttons): + +- Chart-bar icon +- Triggers POST to `/api/lm-extra/fetch-stats` +- Shows progress via WebSocket +- Auto-refreshes grid on completion + +### Sort Dropdown Patch + +JS extension adds the new sort options to the existing `#sortSelect` dropdown on page load. + +## Bulk Fetch Strategy + +CivitAI's `/api/v1/models?ids=` endpoint accepts multiple model IDs per request. + +1. Collect all unique `civitai_model_id` values from the lora list +2. Batch into groups (e.g., 50 IDs per request) +3. For each batch, fetch model data and extract `stats` from the matching version +4. Rate-limit: 1-2 requests/second with backoff on 429 responses +5. Broadcast progress via WebSocket (e.g., "Fetching stats: 150/300 models") + +## Piggyback Strategy + +When the existing metadata sync runs (triggered by the manager's refresh/fetch), the proxy can intercept the CivitAI API response if it flows through, and opportunistically extract stats. This covers new models added after the initial bulk fetch. + +## Files to Create/Modify + +| File | Action | +|------|--------| +| `stats_db.py` | NEW — SQLite wrapper for civitai_stats.db | +| `stats_service.py` | NEW — bulk fetch logic, CivitAI API calls, rate limiting | +| `proxy.py` | MODIFY — intercept list response, enrich with stats; add sort handling | +| `web/comfyui/lm_stats_ui.js` | NEW — card badges, toolbar button, sort dropdown patch |