Ethanfel
afb3840105
Design for a shim that stores CivitAI stats (downloads, rating, thumbs up) in a separate SQLite DB and enriches the list API response via the proxy middleware. Includes bulk fetch button, card badges, and new sort options. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
158 lines
5.3 KiB
Markdown
158 lines
5.3 KiB
Markdown
# CivitAI Extended Stats Shim — Design
|
|
|
|
## Problem
|
|
|
|
The LoRA Manager fetches metadata from CivitAI but discards stats (download count, rating, thumbs up). Users want to sort and filter loras by popularity/quality, and see these stats on model cards — like CivitAI's own UI.
|
|
|
|
## Constraints
|
|
|
|
- No modifications to the original LoRA Manager codebase (avoid merge conflicts on upstream sync)
|
|
- Lives inside ComfyUI-LM-Remote, which already has proxy middleware intercepting API responses
|
|
- Must handle CivitAI API rate limits gracefully when bulk-fetching stats
|
|
- Stats should persist across restarts (SQLite)
|
|
|
|
## Architecture
|
|
|
|
### Hybrid Shim Approach
|
|
|
|
- **Separate SQLite DB** (`civitai_stats.db`) for extended stats — avoids schema conflicts with the manager's own cache
|
|
- **Proxy enrichment** — the existing middleware intercepts `/api/lm/loras/list` responses from the remote and merges stats into each item before forwarding to the browser
|
|
- **New API endpoints** on LM-Remote for the bulk fetch button and status checks
|
|
- **Frontend JS** injected via LM-Remote's `web/comfyui/` directory, patches sort dropdown and card rendering
|
|
|
|
### Data Flow
|
|
|
|
```
|
|
List request:
|
|
Browser → GET /api/lm/loras/list
|
|
→ proxy fetches response from remote
|
|
→ enriches each item with stats from local civitai_stats.db
|
|
→ returns merged response to browser
|
|
|
|
Bulk fetch (initial):
|
|
Browser → POST /api/lm-extra/fetch-stats
|
|
→ handler reads models from remote list API
|
|
→ extracts civitai_model_id + sha256 for each
|
|
→ batches CivitAI API calls (/api/v1/models?ids=id1,id2,...)
|
|
→ stores stats in civitai_stats.db
|
|
→ WebSocket progress updates
|
|
→ frontend auto-refreshes grid when done
|
|
|
|
Piggyback (ongoing):
|
|
When proxy intercepts metadata sync responses that contain stats,
|
|
opportunistically store them in civitai_stats.db
|
|
```
|
|
|
|
## SQLite Schema
|
|
|
|
File: `civitai_stats.db` (stored alongside LM-Remote's config)
|
|
|
|
```sql
|
|
CREATE TABLE model_stats (
|
|
sha256 TEXT PRIMARY KEY,
|
|
civitai_model_id INTEGER,
|
|
civitai_version_id INTEGER,
|
|
download_count INTEGER,
|
|
rating REAL,
|
|
rating_count INTEGER,
|
|
thumbs_up_count INTEGER,
|
|
fetched_at REAL
|
|
);
|
|
```
|
|
|
|
Keyed by `sha256` — the common identifier between the manager's cache and CivitAI. Joins naturally regardless of file paths or renames.
|
|
|
|
## API Endpoints (LM-Remote)
|
|
|
|
| Method | Path | Purpose |
|
|
|--------|------|---------|
|
|
| POST | `/api/lm-extra/fetch-stats` | Bulk fetch stats for all models (button) |
|
|
| GET | `/api/lm-extra/stats-status` | Check if stats DB is populated, staleness info |
|
|
|
|
## Proxy Response Enrichment
|
|
|
|
The middleware intercepts the `/api/lm/loras/list` response and:
|
|
|
|
1. Parses the JSON response body
|
|
2. Collects all `sha256` values from items
|
|
3. Batch-queries `civitai_stats.db` (single SELECT with IN clause)
|
|
4. Merges `download_count`, `rating`, `rating_count`, `thumbs_up_count` into each item
|
|
5. Returns the enriched response
|
|
|
|
## Sorting
|
|
|
|
### New Sort Options
|
|
|
|
Added to the sort dropdown alongside existing options (name, date, size, usage):
|
|
|
|
| Value | Label |
|
|
|-------|-------|
|
|
| `downloads:desc` | Most downloaded |
|
|
| `downloads:asc` | Least downloaded |
|
|
| `rating:desc` | Highest rated |
|
|
| `rating:asc` | Lowest rated |
|
|
| `thumbsup:desc` | Most liked |
|
|
| `thumbsup:asc` | Least liked |
|
|
|
|
### Sort Implementation
|
|
|
|
When the proxy sees these sort values in the list request, sorting is handled locally:
|
|
|
|
1. Fetch the full (unpaginated) list from remote
|
|
2. Join with stats DB
|
|
3. Sort locally
|
|
4. Apply pagination
|
|
5. Return result
|
|
|
|
Items without stats sort to the end.
|
|
|
|
## Frontend Changes
|
|
|
|
All in `ComfyUI-LM-Remote/web/comfyui/`.
|
|
|
|
### Card Badges
|
|
|
|
Small badges on each model card (styled like the existing base model badge):
|
|
|
|
- Download icon + count (e.g., "12.3k")
|
|
- Star icon + rating (e.g., "4.8")
|
|
- Thumbs-up icon + count (e.g., "89")
|
|
|
|
Numbers use compact formatting (1000 → "1k", 12345 → "12.3k").
|
|
|
|
### Toolbar Button
|
|
|
|
A "Fetch Stats" button in the toolbar (next to existing Fetch/Download buttons):
|
|
|
|
- Chart-bar icon
|
|
- Triggers POST to `/api/lm-extra/fetch-stats`
|
|
- Shows progress via WebSocket
|
|
- Auto-refreshes grid on completion
|
|
|
|
### Sort Dropdown Patch
|
|
|
|
JS extension adds the new sort options to the existing `#sortSelect` dropdown on page load.
|
|
|
|
## Bulk Fetch Strategy
|
|
|
|
CivitAI's `/api/v1/models?ids=` endpoint accepts multiple model IDs per request.
|
|
|
|
1. Collect all unique `civitai_model_id` values from the lora list
|
|
2. Batch into groups (e.g., 50 IDs per request)
|
|
3. For each batch, fetch model data and extract `stats` from the matching version
|
|
4. Rate-limit: 1-2 requests/second with backoff on 429 responses
|
|
5. Broadcast progress via WebSocket (e.g., "Fetching stats: 150/300 models")
|
|
|
|
## Piggyback Strategy
|
|
|
|
When the existing metadata sync runs (triggered by the manager's refresh/fetch), the proxy can intercept the CivitAI API response if it flows through, and opportunistically extract stats. This covers new models added after the initial bulk fetch.
|
|
|
|
## Files to Create/Modify
|
|
|
|
| File | Action |
|
|
|------|--------|
|
|
| `stats_db.py` | NEW — SQLite wrapper for civitai_stats.db |
|
|
| `stats_service.py` | NEW — bulk fetch logic, CivitAI API calls, rate limiting |
|
|
| `proxy.py` | MODIFY — intercept list response, enrich with stats; add sort handling |
|
|
| `web/comfyui/lm_stats_ui.js` | NEW — card badges, toolbar button, sort dropdown patch |
|