Ethanfel/ComfyUI-Tween
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Redesign README: add SVG model comparison, reorganize by priority

Browse Source 
Move installation to top, add shields.io badges, create visual model
comparison chart, use collapsible sections for node reference, condense
acknowledgments into a table with citations in a collapsible block.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Ethanfel
2026-02-27 13:37:50 +01:00
parent c178f756da
commit a60fb2a25e
2 changed files with 271 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files

301
README.md
View File

@@ -1,40 +1,86 @@
# ComfyUI BIM-VFI + EMA-VFI + SGM-VFI + GIMM-VFI # Tween — Video Frame Interpolation for ComfyUI
ComfyUI custom nodes for video frame interpolation using [BiM-VFI](https://github.com/KAIST-VICLab/BiM-VFI) (CVPR 2025), [EMA-VFI](https://github.com/MCG-NJU/EMA-VFI) (CVPR 2023), [SGM-VFI](https://github.com/MCG-NJU/SGM-VFI) (CVPR 2024), and [GIMM-VFI](https://github.com/GSeanCDAT/GIMM-VFI) (NeurIPS 2024). Designed for long videos with thousands of frames — processes them without running out of VRAM. [![ComfyUI](https://img.shields.io/badge/ComfyUI-Custom_Node-0a7ef0)](https://registry.comfy.org/)
[![Python 3.10+](https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white)](https://www.python.org/)
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Models](https://img.shields.io/badge/VFI_Models-4-8B5CF6)](#which-model-should-i-use)
Four video frame interpolation models in one package — **BIM-VFI**, **EMA-VFI**, **SGM-VFI**, and **GIMM-VFI**. Designed for long videos with thousands of frames without running out of VRAM.
<p align="center">
<img src="assets/model-comparison.svg" alt="Model Comparison" width="720"/>
</p>
## Installation
Install from the [ComfyUI Registry](https://registry.comfy.org/) (recommended) or clone manually:
```bash
cd ComfyUI/custom_nodes
git clone https://github.com/Ethanfel/ComfyUI-Tween.git
pip install -r requirements.txt
```
All dependencies (`gdown`, `timm`, `omegaconf`, `easydict`, `yacs`, `einops`, `huggingface_hub`) are declared in `pyproject.toml` and `requirements.txt`, installed automatically by ComfyUI Manager or pip.
### cupy (required for BIM-VFI, SGM-VFI, GIMM-VFI)
[cupy](https://cupy.dev/) provides GPU-accelerated optical flow warping. **EMA-VFI works without it.**
1. Find your CUDA version:
```bash
python -c "import torch; print(torch.version.cuda)"
```
2. Install the matching package:
| CUDA | Command |
|------|---------|
| 12.x | `pip install cupy-cuda12x` |
| 11.x | `pip install cupy-cuda11x` |
> Make sure to run pip in the same Python environment as ComfyUI. If cupy is missing, the Load node shows an error with your CUDA version and the exact install command.
<details>
<summary>cupy troubleshooting</summary>
| Problem | Solution |
|---------|----------|
| `ModuleNotFoundError: No module named 'cupy'` | Install cupy using the steps above |
| `cupy` installed but `ImportError` at runtime | CUDA version mismatch — uninstall and reinstall the correct version |
| Install hangs or takes very long | cupy wheels are ~800 MB, be patient |
| Docker / no build tools | Use the prebuilt wheel: `pip install cupy-cuda12x` (not bare `cupy` which compiles from source) |
</details>
## Which model should I use? ## Which model should I use?
| | BIM-VFI | EMA-VFI | SGM-VFI | GIMM-VFI | | | BIM-VFI | EMA-VFI | SGM-VFI | GIMM-VFI |
|---|---------|---------|---------|----------| |---|---------|---------|---------|----------|
| **Best for** | General-purpose, non-uniform motion | Fast inference, light VRAM | Large motion, occlusion-heavy scenes | High multipliers (4x/8x) in a single pass | | **Best for** | General-purpose | Fast, low VRAM | Large motion | High multipliers (4x/8x) |
| **Quality** | Highest overall | Good | Best on large motion | Good | | **Quality** | Highest | Good | Best on large motion | Good |
| **Speed** | Moderate | Fastest | Slowest | Fast for 4x/8x (single pass) | | **Speed** | Moderate | Fastest | Slowest | Fast for 4x/8x |
| **VRAM** | ~2 GB/pair | ~1.5 GB/pair | ~3 GB/pair | ~2.5 GB/pair | | **VRAM** | ~2 GB/pair | ~1.5 GB/pair | ~3 GB/pair | ~2.5 GB/pair |
| **Params** | ~17 M | ~14–65 M | ~15 M + GMFlow | ~80 M (RAFT) / ~123 M (FlowFormer) | | **Params** | ~17 M | ~14–65 M | ~15 M + GMFlow | ~80 M (RAFT) / ~123 M (FlowFormer) |
| **Arbitrary timestep** | Yes | Yes (with `_t` checkpoint) | No (fixed 0.5) | Yes (native single-pass) | | **Arbitrary timestep** | Yes | Yes (`_t` checkpoint) | No (fixed 0.5) | Yes (native) |
| **4x/8x mode** | Recursive 2x passes | Recursive 2x passes | Recursive 2x passes | Single forward pass (or recursive) | | **4x/8x** | Recursive passes | Recursive passes | Recursive passes | Single forward pass |
| **Requires cupy** | Yes | No | Yes | Yes |
| **Paper** | CVPR 2025 | CVPR 2023 | CVPR 2024 | NeurIPS 2024 | | **Paper** | CVPR 2025 | CVPR 2023 | CVPR 2024 | NeurIPS 2024 |
| **License** | Research only | Apache 2.0 | Apache 2.0 | Apache 2.0 |
**TL;DR:** Start with **BIM-VFI** for best quality. Use **EMA-VFI** if you need speed or lower VRAM. Use **SGM-VFI** if your video has large camera motion or fast-moving objects that the others struggle with. Use **GIMM-VFI** when you want 4x or 8x interpolation without recursive passes — it generates all intermediate frames in a single forward pass per pair. **TL;DR:** Start with **BIM-VFI** for best quality. Use **EMA-VFI** for speed or if you can't install cupy. Use **SGM-VFI** for large camera motion. Use **GIMM-VFI** for 4x/8x without recursive passes.
## VRAM Guide
| VRAM | Recommended settings |
|------|----------------------|
| 8 GB | `batch_size=1, chunk_size=500` |
| 24 GB | `batch_size=2–4, chunk_size=1000` |
| 48 GB+ | `batch_size=4–16, all_on_gpu=true` |
| 96 GB+ | `batch_size=8–16, all_on_gpu=true, chunk_size=0` |
## Nodes ## Nodes
### BIM-VFI All Interpolate nodes share a common set of controls:
#### Load BIM-VFI Model
Loads the BiM-VFI checkpoint. Auto-downloads from Google Drive on first use to `ComfyUI/models/bim-vfi/`.
| Input | Description |
|-------|-------------|
| **model_path** | Checkpoint file from `models/bim-vfi/` |
| **auto_pyr_level** | Auto-select pyramid level by resolution (&lt;540p=3, 540p=5, 1080p=6, 4K=7) |
| **pyr_level** | Manual pyramid level (3-7), only used when auto is off |
#### BIM-VFI Interpolate
Interpolates frames from an image batch.
| Input | Description | | Input | Description |
|-------|-------------| |-------|-------------|
@@ -42,7 +88,7 @@ Interpolates frames from an image batch.
| **model** | Model from the loader node | | **model** | Model from the loader node |
| **multiplier** | 2x, 4x, or 8x frame rate (recursive 2x passes) | | **multiplier** | 2x, 4x, or 8x frame rate (recursive 2x passes) |
| **batch_size** | Frame pairs processed simultaneously (higher = faster, more VRAM) | | **batch_size** | Frame pairs processed simultaneously (higher = faster, more VRAM) |
| **chunk_size** | Process in segments of N input frames (0 = disabled). Bounds VRAM for very long videos. Result is identical to processing all at once | | **chunk_size** | Process in segments of N input frames (0 = disabled). Bounds VRAM for very long videos |
| **keep_device** | Keep model on GPU between pairs (faster, ~200 MB constant VRAM) | | **keep_device** | Keep model on GPU between pairs (faster, ~200 MB constant VRAM) |
| **all_on_gpu** | Keep all intermediate frames on GPU (fast, needs large VRAM) | | **all_on_gpu** | Keep all intermediate frames on GPU (fast, needs large VRAM) |
| **clear_cache_after_n_frames** | Clear CUDA cache every N pairs to prevent VRAM buildup | | **clear_cache_after_n_frames** | Clear CUDA cache every N pairs to prevent VRAM buildup |
@@ -52,28 +98,43 @@ Interpolates frames from an image batch.
| Output | Description | | Output | Description |
|--------|-------------| |--------|-------------|
| **images** | Interpolated frames at the target FPS (or at the multiplied rate when target_fps = 0) | | **images** | Interpolated frames at the target FPS (or at the multiplied rate when target_fps = 0) |
| **oversampled** | Full power-of-2 oversampled frames before target FPS selection. Same as `images` when target_fps = 0. Useful for inspecting the raw interpolation or feeding into another pipeline | | **oversampled** | Full power-of-2 oversampled frames before target FPS selection. Same as `images` when target_fps = 0 |
#### BIM-VFI Segment Interpolate <details>
<summary><strong>BIM-VFI</strong></summary>
Same as Interpolate but processes a single segment of the input. Chain multiple instances with Save nodes between them to bound peak RAM. The model pass-through output forces sequential execution. #### Load BIM-VFI Model
### Tween Concat Videos Loads the BiM-VFI checkpoint. Auto-downloads from Google Drive on first use to `ComfyUI/models/bim-vfi/`.
Concatenates segment video files into a single video using ffmpeg. Connect from any Segment Interpolate's model output to ensure it runs after all segments are saved. Works with all four models.
### EMA-VFI
#### Load EMA-VFI Model
Loads an EMA-VFI checkpoint. Auto-downloads from Google Drive on first use to `ComfyUI/models/ema-vfi/`. Variant (large/small) and timestep support are auto-detected from the filename.
| Input | Description | | Input | Description |
|-------|-------------| |-------|-------------|
| **model_path** | Checkpoint file from `models/ema-vfi/` | | **model_path** | Checkpoint from `models/bim-vfi/` |
| **tta** | Test-time augmentation: flip input and average with unflipped result (~2x slower, slightly better quality) | | **auto_pyr_level** | Auto pyramid level by resolution (&lt;540p=3, 540p=5, 1080p=6, 4K=7) |
| **pyr_level** | Manual pyramid level (3–7), used when auto is off |
#### BIM-VFI Interpolate
Common controls listed above.
#### BIM-VFI Segment Interpolate
Processes a single segment of the input. Chain multiple instances with Save nodes between them to bound peak RAM. The model pass-through output forces sequential execution.
</details>
<details>
<summary><strong>EMA-VFI</strong></summary>
#### Load EMA-VFI Model
Auto-downloads from Google Drive to `ComfyUI/models/ema-vfi/`. Variant and timestep support are auto-detected from the filename.
| Input | Description |
|-------|-------------|
| **model_path** | Checkpoint from `models/ema-vfi/` |
| **tta** | Test-time augmentation (~2x slower, slightly better quality) |
Available checkpoints:
| Checkpoint | Variant | Params | Arbitrary timestep | | Checkpoint | Variant | Params | Arbitrary timestep |
|-----------|---------|--------|-------------------| |-----------|---------|--------|-------------------|
| `ours_t.pkl` | Large | ~65 M | Yes | | `ours_t.pkl` | Large | ~65 M | Yes |
@@ -81,51 +142,47 @@ Available checkpoints:
| `ours_small_t.pkl` | Small | ~14 M | Yes | | `ours_small_t.pkl` | Small | ~14 M | Yes |
| `ours_small.pkl` | Small | ~14 M | No (fixed 0.5) | | `ours_small.pkl` | Small | ~14 M | No (fixed 0.5) |
#### EMA-VFI Interpolate #### EMA-VFI Interpolate / Segment Interpolate
Interpolates frames from an image batch. Same controls as BIM-VFI Interpolate (including target FPS mode). Same controls as above.
#### EMA-VFI Segment Interpolate </details>
Same as EMA-VFI Interpolate but processes a single segment. Same pattern as BIM-VFI Segment Interpolate. <details>
<summary><strong>SGM-VFI</strong></summary>
### SGM-VFI
#### Load SGM-VFI Model #### Load SGM-VFI Model
Loads an SGM-VFI checkpoint. Auto-downloads from Google Drive on first use to `ComfyUI/models/sgm-vfi/`. Variant (base/small) is auto-detected from the filename (default is small). Auto-downloads from Google Drive to `ComfyUI/models/sgm-vfi/`. Requires cupy.
| Input | Description | | Input | Description |
|-------|-------------| |-------|-------------|
| **model_path** | Checkpoint file from `models/sgm-vfi/` | | **model_path** | Checkpoint from `models/sgm-vfi/` |
| **tta** | Test-time augmentation: flip input and average with unflipped result (~2x slower, slightly better quality) | | **tta** | Test-time augmentation (~2x slower, slightly better quality) |
| **num_key_points** | Sparsity of global matching (0.0 = global everywhere, 0.5 = default balance, higher = faster) | | **num_key_points** | Global matching sparsity (0.0 = global everywhere, 0.5 = default, higher = faster) |
Available checkpoints:
| Checkpoint | Variant | Params | | Checkpoint | Variant | Params |
|-----------|---------|--------| |-----------|---------|--------|
| `ours-1-2-points.pkl` | Small | ~15 M + GMFlow | | `ours-1-2-points.pkl` | Small | ~15 M + GMFlow |
#### SGM-VFI Interpolate #### SGM-VFI Interpolate / Segment Interpolate
Interpolates frames from an image batch. Same controls as BIM-VFI Interpolate (including target FPS mode). Same controls as above.
#### SGM-VFI Segment Interpolate </details>
Same as SGM-VFI Interpolate but processes a single segment. Same pattern as BIM-VFI Segment Interpolate. <details>
<summary><strong>GIMM-VFI</strong></summary>
### GIMM-VFI
#### Load GIMM-VFI Model #### Load GIMM-VFI Model
Loads a GIMM-VFI checkpoint. Auto-downloads from [HuggingFace](https://huggingface.co/Kijai/GIMM-VFI_safetensors) on first use to `ComfyUI/models/gimm-vfi/`. The matching flow estimator (RAFT or FlowFormer) is auto-detected and downloaded alongside the main model. Auto-downloads from [HuggingFace](https://huggingface.co/Kijai/GIMM-VFI_safetensors) to `ComfyUI/models/gimm-vfi/`. The matching flow estimator (RAFT or FlowFormer) is auto-detected and downloaded alongside.
| Input | Description | | Input | Description |
|-------|-------------| |-------|-------------|
| **model_path** | Checkpoint file from `models/gimm-vfi/` | | **model_path** | Checkpoint from `models/gimm-vfi/` |
| **ds_factor** | Downscale factor for internal processing (1.0 = full res, 0.5 = half). Lower = less VRAM, faster, less quality. Try 0.5 for 4K inputs | | **ds_factor** | Downscale factor for internal processing (1.0 = full, 0.5 = half). Try 0.5 for 4K inputs |
Available checkpoints:
| Checkpoint | Variant | Params | Flow estimator (auto-downloaded) | | Checkpoint | Variant | Params | Flow estimator (auto-downloaded) |
|-----------|---------|--------|----------------------------------| |-----------|---------|--------|----------------------------------|
| `gimmvfi_r_arb_lpips_fp32.safetensors` | RAFT | ~80 M | `raft-things_fp32.safetensors` | | `gimmvfi_r_arb_lpips_fp32.safetensors` | RAFT | ~80 M | `raft-things_fp32.safetensors` |
@@ -133,90 +190,40 @@ Available checkpoints:
#### GIMM-VFI Interpolate #### GIMM-VFI Interpolate
Interpolates frames from an image batch. Same controls as BIM-VFI Interpolate (including target FPS mode), plus: Common controls plus:
| Input | Description | | Input | Description |
|-------|-------------| |-------|-------------|
| **single_pass** | When enabled (default), generates all intermediate frames per pair in one forward pass using GIMM-VFI's arbitrary-timestep capability. No recursive 2x passes needed for 4x or 8x. Disable to use the standard recursive approach (same as BIM/EMA/SGM) | | **single_pass** | Generate all intermediate frames per pair in one forward pass (default on). No recursive 2x passes needed for 4x/8x. Disable to use the standard recursive approach |
#### GIMM-VFI Segment Interpolate #### GIMM-VFI Segment Interpolate
Same as GIMM-VFI Interpolate but processes a single segment. Same pattern as BIM-VFI Segment Interpolate. Same pattern as other Segment nodes.
**Output frame count (all models):** </details>
- Multiplier mode: 2x = 2N-1, 4x = 4N-3, 8x = 8N-7
- Target FPS mode: `floor((N-1) / source_fps * target_fps) + 1` frames. Automatically oversamples to the nearest power-of-2 above the ratio, then selects frames at exact target timestamps. Downsampling (target < source) also works — frames are selected from the input with no model calls
## Installation ### Tween Concat Videos
Install from the [ComfyUI Registry](https://registry.comfy.org/) or clone into your ComfyUI `custom_nodes/` directory: Concatenates segment video files into a single video using ffmpeg. Connect from any Segment Interpolate's model output to ensure it runs after all segments are saved. Works with all four models.
```bash ### Output frame count
cd ComfyUI/custom_nodes
git clone https://github.com/Ethanfel/ComfyUI-Tween.git
pip install -r requirements.txt
```
### cupy (required for BIM-VFI, SGM-VFI, GIMM-VFI) - **Multiplier mode:** 2x = 2N-1, 4x = 4N-3, 8x = 8N-7
- **Target FPS mode:** `floor((N-1) / source_fps * target_fps) + 1` frames. Automatically oversamples to the nearest power-of-2 above the ratio, then selects frames at exact target timestamps. Downsampling (target < source) also works — frames are selected from the input with no model calls.
[cupy](https://cupy.dev/) is a GPU-accelerated array library used for optical flow warping. It is required by **BIM-VFI**, **SGM-VFI**, and **GIMM-VFI**. EMA-VFI does **not** need cupy and works without it.
cupy must match your PyTorch CUDA version. If it is missing or mismatched, the Load node will show an error in ComfyUI with your CUDA version and the exact install command.
#### How to install cupy
**Step 1 — Find your CUDA version:**
```bash
python -c "import torch; print(torch.version.cuda)"
```
This prints something like `12.4` or `11.8`.
**Step 2 — Install the matching cupy package:**
| CUDA version | Install command |
|---|---|
| 12.x | `pip install cupy-cuda12x` |
| 11.x | `pip install cupy-cuda11x` |
> **Note:** Make sure to run pip in the same Python environment as ComfyUI. If you use a venv or conda, activate it first.
#### Troubleshooting
| Problem | Solution |
|---------|----------|
| `ModuleNotFoundError: No module named 'cupy'` | Install cupy using the steps above |
| `cupy` installed but `ImportError` at runtime | CUDA version mismatch — uninstall with `pip uninstall cupy-cuda12x` and reinstall the correct version |
| Install hangs or takes very long | cupy wheels are large (~800MB). Use a fast connection and be patient |
| Docker / no build tools | Use the prebuilt wheel: `pip install cupy-cuda12x` (not `cupy` which compiles from source) |
#### Can I skip cupy?
Yes — just use **EMA-VFI**, which does not require cupy. It is the fastest model and uses the least VRAM. The other three models (BIM-VFI, SGM-VFI, GIMM-VFI) will not load without cupy.
### Other dependencies
All other dependencies (`gdown`, `timm`, `omegaconf`, `easydict`, `yacs`, `einops`, `huggingface_hub`) are listed in `pyproject.toml` and `requirements.txt`, and are installed automatically by ComfyUI Manager or pip.
## VRAM Guide
| VRAM | Recommended settings |
|------|---------------------|
| 8 GB | batch_size=1, chunk_size=500 |
| 24 GB | batch_size=2-4, chunk_size=1000 |
| 48 GB+ | batch_size=4-16, all_on_gpu=true |
| 96 GB+ | batch_size=8-16, all_on_gpu=true, chunk_size=0 |
## Acknowledgments ## Acknowledgments
This project wraps the official [BiM-VFI](https://github.com/KAIST-VICLab/BiM-VFI) implementation by the [KAIST VIC Lab](https://github.com/KAIST-VICLab), the official [EMA-VFI](https://github.com/MCG-NJU/EMA-VFI) implementation by MCG-NJU, the official [SGM-VFI](https://github.com/MCG-NJU/SGM-VFI) implementation by MCG-NJU, and the [GIMM-VFI](https://github.com/GSeanCDAT/GIMM-VFI) implementation by S-Lab (NTU). GIMM-VFI architecture files in `gimm_vfi_arch/` are adapted from [kijai/ComfyUI-GIMM-VFI](https://github.com/kijai/ComfyUI-GIMM-VFI) with safetensors checkpoints from [Kijai/GIMM-VFI_safetensors](https://huggingface.co/Kijai/GIMM-VFI_safetensors). Architecture files in `bim_vfi_arch/`, `ema_vfi_arch/`, `sgm_vfi_arch/`, and `gimm_vfi_arch/` are vendored from their respective repositories with minimal modifications (relative imports, device-awareness fixes, inference-only paths). | Model | Authors | Venue | Links |
|-------|---------|-------|-------|
| **BIM-VFI** | Seo, Oh, Kim (KAIST VIC Lab) | CVPR 2025 | [Paper](https://arxiv.org/abs/2412.11365) · [Code](https://github.com/KAIST-VICLab/BiM-VFI) · [Project](https://kaist-viclab.github.io/BiM-VFI_site/) |
| **EMA-VFI** | Zhang et al. (MCG-NJU) | CVPR 2023 | [Paper](https://arxiv.org/abs/2303.00440) · [Code](https://github.com/MCG-NJU/EMA-VFI) |
| **SGM-VFI** | Zhang et al. (MCG-NJU) | CVPR 2024 | [Paper](https://arxiv.org/abs/2404.06913) · [Code](https://github.com/MCG-NJU/SGM-VFI) |
| **GIMM-VFI** | Guo, Li, Loy (S-Lab NTU) | NeurIPS 2024 | [Paper](https://arxiv.org/abs/2407.08680) · [Code](https://github.com/GSeanCDAT/GIMM-VFI) |
**BiM-VFI:** GIMM-VFI adaptation from [kijai/ComfyUI-GIMM-VFI](https://github.com/kijai/ComfyUI-GIMM-VFI) with checkpoints from [Kijai/GIMM-VFI_safetensors](https://huggingface.co/Kijai/GIMM-VFI_safetensors). Architecture files in `bim_vfi_arch/`, `ema_vfi_arch/`, `sgm_vfi_arch/`, and `gimm_vfi_arch/` are vendored from their respective repositories with minimal modifications.
> Wonyong Seo, Jihyong Oh, and Munchurl Kim.
> "BiM-VFI: Bidirectional Motion Field-Guided Frame Interpolation for Video with Non-uniform Motions." <details>
> *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2025. <summary>BibTeX citations</summary>
> [[arXiv]](https://arxiv.org/abs/2412.11365) [[Project Page]](https://kaist-viclab.github.io/BiM-VFI_site/) [[GitHub]](https://github.com/KAIST-VICLab/BiM-VFI)
```bibtex ```bibtex
@inproceedings{seo2025bimvfi, @inproceedings{seo2025bimvfi,
@@ -225,45 +232,21 @@ This project wraps the official [BiM-VFI](https://github.com/KAIST-VICLab/BiM-VF
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025} year={2025}
} }
```
**EMA-VFI:**
> Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, and Limin Wang.
> "Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation."
> *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023.
> [[arXiv]](https://arxiv.org/abs/2303.00440) [[GitHub]](https://github.com/MCG-NJU/EMA-VFI)
```bibtex
@inproceedings{zhang2023emavfi, @inproceedings{zhang2023emavfi,
title={Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation}, title={Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation},
author={Zhang, Guozhen and Zhu, Yuhan and Wang, Haonan and Chen, Youxin and Wu, Gangshan and Wang, Limin}, author={Zhang, Guozhen and Zhu, Yuhan and Wang, Haonan and Chen, Youxin and Wu, Gangshan and Wang, Limin},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2023} year={2023}
} }
```
**SGM-VFI:**
> Guozhen Zhang, Yuhan Zhu, Evan Zheran Liu, Haonan Wang, Mingzhen Sun, Gangshan Wu, and Limin Wang.
> "Sparse Global Matching for Video Frame Interpolation with Large Motion."
> *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.
> [[arXiv]](https://arxiv.org/abs/2404.06913) [[GitHub]](https://github.com/MCG-NJU/SGM-VFI)
```bibtex
@inproceedings{zhang2024sgmvfi, @inproceedings{zhang2024sgmvfi,
title={Sparse Global Matching for Video Frame Interpolation with Large Motion}, title={Sparse Global Matching for Video Frame Interpolation with Large Motion},
author={Zhang, Guozhen and Zhu, Yuhan and Liu, Evan Zheran and Wang, Haonan and Sun, Mingzhen and Wu, Gangshan and Wang, Limin}, author={Zhang, Guozhen and Zhu, Yuhan and Liu, Evan Zheran and Wang, Haonan and Sun, Mingzhen and Wu, Gangshan and Wang, Limin},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024} year={2024}
} }
```
**GIMM-VFI:**
> Zujin Guo, Wei Li, and Chen Change Loy.
> "Generalizable Implicit Motion Modeling for Video Frame Interpolation."
> *Advances in Neural Information Processing Systems (NeurIPS)*, 2024.
> [[arXiv]](https://arxiv.org/abs/2407.08680) [[GitHub]](https://github.com/GSeanCDAT/GIMM-VFI)
```bibtex
@inproceedings{guo2024gimmvfi, @inproceedings{guo2024gimmvfi,
title={Generalizable Implicit Motion Modeling for Video Frame Interpolation}, title={Generalizable Implicit Motion Modeling for Video Frame Interpolation},
author={Guo, Zujin and Li, Wei and Loy, Chen Change}, author={Guo, Zujin and Li, Wei and Loy, Chen Change},
@@ -272,12 +255,12 @@ This project wraps the official [BiM-VFI](https://github.com/KAIST-VICLab/BiM-VF
} }
``` ```
</details>
## License ## License
The BiM-VFI model weights and architecture code are provided by KAIST VIC Lab for **research and education purposes only**. Commercial use requires permission from the principal investigator (Prof. Munchurl Kim, mkimee@kaist.ac.kr). See the [original repository](https://github.com/KAIST-VICLab/BiM-VFI) for details. **BIM-VFI:** Research and education only. Commercial use requires permission from Prof. Munchurl Kim (mkimee@kaist.ac.kr). See the [original repository](https://github.com/KAIST-VICLab/BiM-VFI).
The EMA-VFI model weights and architecture code are released under the [Apache 2.0 License](https://github.com/MCG-NJU/EMA-VFI/blob/main/LICENSE). See the [original repository](https://github.com/MCG-NJU/EMA-VFI) for details. **EMA-VFI, SGM-VFI, GIMM-VFI:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). GIMM-VFI ComfyUI adaptation based on [kijai/ComfyUI-GIMM-VFI](https://github.com/kijai/ComfyUI-GIMM-VFI).
The SGM-VFI model weights and architecture code are released under the [Apache 2.0 License](https://github.com/MCG-NJU/SGM-VFI/blob/main/LICENSE). See the [original repository](https://github.com/MCG-NJU/SGM-VFI) for details. **This wrapper code:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
The GIMM-VFI model weights and architecture code are released under the [Apache 2.0 License](https://github.com/GSeanCDAT/GIMM-VFI/blob/main/LICENSE). See the [original repository](https://github.com/GSeanCDAT/GIMM-VFI) for details. ComfyUI adaptation based on [kijai/ComfyUI-GIMM-VFI](https://github.com/kijai/ComfyUI-GIMM-VFI).

80
assets/model-comparison.svg Normal file
View File

@@ -0,0 +1,80 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 320" width="720" height="320">
<defs>
<linearGradient id="gQ" x1="0" y1="0" x2="1" y2="0">
<stop offset="0%" stop-color="#7aa2f7"/><stop offset="100%" stop-color="#7dcfff"/>
</linearGradient>
<linearGradient id="gS" x1="0" y1="0" x2="1" y2="0">
<stop offset="0%" stop-color="#9ece6a"/><stop offset="100%" stop-color="#73daca"/>
</linearGradient>
<linearGradient id="gV" x1="0" y1="0" x2="1" y2="0">
<stop offset="0%" stop-color="#bb9af7"/><stop offset="100%" stop-color="#d2a8ff"/>
</linearGradient>
</defs>
<!-- Background -->
<rect width="720" height="320" rx="16" fill="#0d1117"/>
<!-- ═══ BIM-VFI (top-left) ═══ -->
<rect x="10" y="10" width="340" height="145" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="11" y="22" width="3" height="121" fill="#3fb950"/>
<text x="30" y="38" fill="#e6edf3" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="15" font-weight="600">BIM-VFI</text>
<text x="30" y="56" fill="#3fb950" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">&#9733; Recommended &#183; Best quality &#183; CVPR 2025</text>
<line x1="30" y1="64" x2="330" y2="64" stroke="#30363d" stroke-width="0.5"/>
<text x="30" y="82" fill="#7aa2f7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Quality</text>
<rect x="88" y="72" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="88" y="72" width="244" height="11" rx="3" fill="url(#gQ)" opacity="0.85"/>
<text x="30" y="100" fill="#9ece6a" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Speed</text>
<rect x="88" y="90" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="88" y="90" width="146" height="11" rx="3" fill="url(#gS)" opacity="0.85"/>
<text x="30" y="118" fill="#bb9af7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">VRAM</text>
<rect x="88" y="108" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="88" y="108" width="195" height="11" rx="3" fill="url(#gV)" opacity="0.85"/>
<!-- ═══ EMA-VFI (top-right) ═══ -->
<rect x="370" y="10" width="340" height="145" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="371" y="22" width="3" height="121" fill="#58a6ff"/>
<text x="390" y="38" fill="#e6edf3" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="15" font-weight="600">EMA-VFI</text>
<text x="390" y="56" fill="#58a6ff" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Fastest &#183; No cupy needed &#183; CVPR 2023</text>
<line x1="390" y1="64" x2="690" y2="64" stroke="#30363d" stroke-width="0.5"/>
<text x="390" y="82" fill="#7aa2f7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Quality</text>
<rect x="448" y="72" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="448" y="72" width="146" height="11" rx="3" fill="url(#gQ)" opacity="0.85"/>
<text x="390" y="100" fill="#9ece6a" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Speed</text>
<rect x="448" y="90" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="448" y="90" width="244" height="11" rx="3" fill="url(#gS)" opacity="0.85"/>
<text x="390" y="118" fill="#bb9af7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">VRAM</text>
<rect x="448" y="108" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="448" y="108" width="244" height="11" rx="3" fill="url(#gV)" opacity="0.85"/>
<!-- ═══ SGM-VFI (bottom-left) ═══ -->
<rect x="10" y="165" width="340" height="145" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="11" y="177" width="3" height="121" fill="#f0883e"/>
<text x="30" y="193" fill="#e6edf3" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="15" font-weight="600">SGM-VFI</text>
<text x="30" y="211" fill="#f0883e" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Large motion specialist &#183; CVPR 2024</text>
<line x1="30" y1="219" x2="330" y2="219" stroke="#30363d" stroke-width="0.5"/>
<text x="30" y="237" fill="#7aa2f7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Quality</text>
<rect x="88" y="227" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="88" y="227" width="195" height="11" rx="3" fill="url(#gQ)" opacity="0.85"/>
<text x="30" y="255" fill="#9ece6a" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Speed</text>
<rect x="88" y="245" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="88" y="245" width="98" height="11" rx="3" fill="url(#gS)" opacity="0.85"/>
<text x="30" y="273" fill="#bb9af7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">VRAM</text>
<rect x="88" y="263" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="88" y="263" width="98" height="11" rx="3" fill="url(#gV)" opacity="0.85"/>
<!-- ═══ GIMM-VFI (bottom-right) ═══ -->
<rect x="370" y="165" width="340" height="145" rx="10" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="371" y="177" width="3" height="121" fill="#bc8cff"/>
<text x="390" y="193" fill="#e6edf3" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="15" font-weight="600">GIMM-VFI</text>
<text x="390" y="211" fill="#bc8cff" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Single-pass 4&#215;/8&#215; &#183; NeurIPS 2024</text>
<line x1="390" y1="219" x2="690" y2="219" stroke="#30363d" stroke-width="0.5"/>
<text x="390" y="237" fill="#7aa2f7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Quality</text>
<rect x="448" y="227" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="448" y="227" width="146" height="11" rx="3" fill="url(#gQ)" opacity="0.85"/>
<text x="390" y="255" fill="#9ece6a" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">Speed</text>
<rect x="448" y="245" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="448" y="245" width="195" height="11" rx="3" fill="url(#gS)" opacity="0.85"/>
<text x="390" y="273" fill="#bb9af7" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','Noto Sans',Helvetica,Arial,sans-serif" font-size="11">VRAM</text>
<rect x="448" y="263" width="244" height="11" rx="3" fill="#21262d"/>
<rect x="448" y="263" width="146" height="11" rx="3" fill="url(#gV)" opacity="0.85"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 7.2 KiB