Original Project found here: <a target="_blank" rel="ugc" href="https://huggingface.co/Djrango/Qwen2vl-Flux">https://huggingface.co/Djrango/Qwen2vl-Flux</a>Qwen2vl-Flux is a state-of-the-art multimodal image generation model that enhances FLUX with Qwen2VL's vision-language understanding capabilities. This model excels at generating high-quality images based on both text prompts and visual references, offering superior multimodal understanding and control.<ul><li>ComfyUI currently doesn't support and there is no available nodes to load the CLIP+LLM portion into it</li><li>This is just for reviewing/testing the finetuned trained part of the Flux model</li><li>CFG set to 1 on KSampler</li><li>Rendered an image in 150s using 8GB GPU @ 512px / 10 steps using the bf16 model</li><li>This model comes will be available in 3 formats named after the folder it should be in<ul><li>diffusion_models - This one is in diffusers format, it is just the merged safetensors file from HuggingFace page</li><li>checkpoints - This one has been converted to Flux Transformers format and prefix for stable_diffusion compatibility, does not include CLIP and VAE</li><li>unet - I will provide the q4_0 and q8 variants, make a comment if you'd like to see any other quants</li></ul></li></ul>

Qwen 2vl Flux

<ul><li>This version goes in the checkpoints folder</li><li>This version is used with the Load Checkpoint node</li><li>VAE and CLIP not included, use standard Flux setup</li></ul>

Files with SHA256: b784c6c5261073a48ef6750b1b2931aacbb9c1f327e93a98c390182fd3e36e31