Full Checkpoint with improved TE do not load additional CLIP/TE
FLUX.1 (Base UNET) + Google FLAN
All uploaded models sourced from 65GB Full FP32
Per the Apache 2.0 license FLAN is attributed to Google
Description
FAQ
Comments (31)
This checkpoint doesn't work in diffusion loader? It doesn't work in the unet folder of models in comfyui, is there something else I need to set up?
You need the NF4 checkpoint loader if your using comfy, forge would be native. Forge also has Lora support for NF4 comfy does not right now
@Felldude thx!
@Felldude you said Do not load VAE, CLIP or TE - FP32 Quantized versions baked in Does this mean that I shouldn't load VAE, but only load one of CLIP or TE?
I am adding a FP8 version for easier Comfy UI use
What's the benefit of using this Google FLAN encoder over the normal t5xxl_fp8_e4m3fn?
Much higher accuracy by quantizing from FP32 on a model Google finetuned
Looks nice. Can you tell me how much vram we are talking about here? Perhaps you should upload more samples to show that flan is better than t5xxl.
Google themselves published the stats on the finetune of T5xxl, I am quantizing with the highest publicly available quality FP32
Comfy and Forge have CPU offloading so VRAM shouldn't be an issue. I wouldn't attempt to use all the models in FP32 as that would be close to 100GB...it would be funny to post that checkpoint though. (Unfortunately Blackforest has never published a FP32 version of FLUX)
Did you remove the decoder part of the FLAN encoder? If not, that would definite decrease the file size.
Can you share only the Google FLAN FP32/NF4/FP8 so we can test it also on other models?
So currently the model gets some block errors when extracting, I will have to look into if it is being pruned incorrectly or extracted incorrectly. I am leaning toward extracting incorrectly as I have no clip errors in the merged models
@Felldude Getting full black image when using this clip only model (dual clip loader)
@jayepe6189480 If you will note in bold you should not be loading any clips
@Felldude Hey there Felldude, I'd like to try this out but it seems to have been "archived" - I can't download it. Currently testing off the model loader
@ericreator I released a simple tool to extract the T5 Flan
quality i will see
but fits perfect my 16GB VRAM
and its 25% faster than normal models (sec/it)
hmmm seems a bit less details and for long prompts it dont get all objects like the original FLUX_DEVfp8 tested on 3 prompts a 10 image
The UNET is the same, regarding prompt adherence the logic should be mathematically better per google but that is meaningless if the results don't look aesthetically pleasing.
It did appear faster but it can be hard to tell with CPU offloading
well downloading now, going to test it on 8gb vram and 32gb of ram. Let you all know if it works or not (forgeUI)
I recommend the NF4 for forge
How are you planning to fit a 15gb file into 8gb vram so it runs at decent speed? I don't see how's that possible
@LiteSoulHD With CPU offloading, before that it was not possible at all
It does using a 2070 it takes a good 90 to 120 seconds per image though.
@LiteSoulHD with multigpu node gguf its possible!
both models work with Forge for me, I have 128Gb Ram and 3090 24Gb, but I prefer to use NF4 with Forge.
Is there a way to make photos look less comic/hdr? Like a natural photograph, soft lighting
The Dev/NF4 model will have the highest quality apart from Full FP16/BF16