Conversation
|
Thank you for this. This 2D VAE is fantastic! However, sd-scripts' Qwen VAE already incorporates a memory-saving mechanism, reducing memory usage to 1/5 of the original. I also believe that maintaining weight compatibility is important for existing users. Therefore, directly merging this PR may be difficult, but I will use this idea to improve the VAE implementation in sd-scripts. Thank you again for letting me know this amazing idea! |
|
Besides saving memory, the speedup is also real when we need to cache a lot of latents. I can also implement tiling with the 2D VAE if we want to further save memory. For weight compatibility, I did not change the code to load the 3D VAE, and we can still use It's ok to close this PR if you're not planning to merge it. |
|
Inspired by this, I created a 2D version of Qwen-Image VAE in #2382. I also added automatic weight conversion, allowing users to seamlessly utilize the 2D VAE. Thank you again for sharing this idea! |
2 participants
Qwen-Image VAE is used in Anima and it's notoriously slow. We can convert 3d convs into 2d convs while keeping the encoded latents unchanged within floating point errors. On my machine (RTX 3080) when encoding a 1024x1024 image, it's 2x faster and takes only 1/3 peak VRAM.
See the model https://huggingface.co/Anzhc/Qwen2D-VAE and the ComfyUI inference code https://github.com/Anzhc/anzhc-qwen2d-comfyui
In the training config we can set
vae = '/path/to/Qwen2D_VAE.safetensors', and it will automatically detect the 2D VAE architecture.This is also worth porting to musubi-tuner.