Original Project found here: https://huggingface.co/Djrango/Qwen2vl-Flux
Qwen2vl-Flux is a state-of-the-art multimodal image generation model that enhances FLUX with Qwen2VL's vision-language understanding capabilities. This model excels at generating high-quality images based on both text prompts and visual references, offering superior multimodal understanding and control.
ComfyUI currently doesn't support and there is no available nodes to load the CLIP+LLM portion into it
This is just for reviewing/testing the finetuned trained part of the Flux model
CFG set to 1 on KSampler
Rendered an image in 150s using 8GB GPU @ 512px / 10 steps using the bf16 model
This model comes will be available in 3 formats named after the folder it should be in
diffusion_models - This one is in diffusers format, it is just the merged safetensors file from HuggingFace page
checkpoints - This one has been converted to Flux Transformers format and prefix for stable_diffusion compatibility, does not include CLIP and VAE
unet - I will provide the q4_0 and q8 variants, make a comment if you'd like to see any other quants
Description
This version goes in the checkpoints folder
This version is used with the Load Checkpoint node
VAE and CLIP not included, use standard Flux setup
FAQ
Comments (5)
I hope it comes out Q4_K_M and q5-1 , Best performance and quality ratio for small video memory
How to use the model? Any demo code?
Q1: Is it possible to replace to replace qwen2-vl with qwen2.6-VL?
https://huggingface.co/chohtet/Qwen2.5-7B-Instruct-H3-VLLM/tree/main
Q2: Is there some way to load GGUF quant of qwen2,5 using llama.cpp?
Qwen 2.5 based models outperforms many others. But loading non GGUF versions can be impossible on moderate home hardware.
Did I get it correctly that version "diffusion model bf16" have qwen2-vl CLIP model embedded?
So, just to make things more clear for everybody...
At advertised HF page you will find a sample python code whi SHOULD allow you to use a FLUX UNet module with QWEN2 (QWEN2.5) based tokenizer. On the following page you can see the usage of "FluxModel" module.
https://huggingface.co/Djrango/Qwen2vl-Flux
This is where the module goes from
https://github.com/erwold/qwen2vl-flux
So, we have the following state:
1. We have a custom nodes for ComfyUI which allows you to generate the prompts from text instruction and/or images and video. These nodes allows you to play with VRAM consumption by replacing the QWEN2 7B with QWEN2 3B for example.
2. We have a custom python code which is not yet converted into ComfyUI node, but uses QWEN2 as a tokenizer. With this python code you are bound to Full Size FLUX checkpoint in diffusers format from this folder https://huggingface.co/Djrango/Qwen2vl-Flux/tree/main/flux
In theory you should be able to replace QWEN V2 7B with just any other VLM. By "any" I mean not only qwen2, but also QWEN2.5 [1 | 3 | 7 | 100500]B AND CogVLM_v2 for example. How other VLM's like CogVLM_v2 will work with finetuned checkpoint I have no idea, but technically it seems possible if I did not messed smth up.
Details
Available On (2 platforms)
Same model published on other platforms. May have additional downloads or version variants.