Lol, this is how I feel, was just perfecting my SD3 model, then came Flux, been working on a Flux model and boom, there are GGUF files, this morning I never heard of them and now made a shitload of images with Q4 and Q8 😂

@JayNL Like i said, i'm pulling step back. Deleted all models except regular schnell and dev.

I haven't tried NF4, so what I'm about to say is mostly educated speculation. Q4 (with the ComfyUI-GGUF extension) should be slower than NF4, but with the output much more faithful to the full precision model's. Q5 is even slower, but also better.

It's slower because the extension doesn't call any GGML backend to do computation directly with quantized tensors (like llama.cpp). Instead, the extension dequantizes the tensors on the fly during inference, then does the computations with full precision using pytorch. With proper GGML inference support, the speed should be about the same.

@stduhpf893 Thanks for the explanation. I tested FP8 version almost a week. Then I came back to SDXL and it was really refreshing to see the strengths of SDXL fine tuned model vs. Flux-Schnell FP8 fresh in mind. Then day or two later came NF4 and I was back to Flux again. Maybe I should wait and see what versions sticks. Who knows yet what Flux variants come next week? I'm happy with NF4 for now, especially if it's implied that Q5 could be slower than NF4 for me.

for reminder to everyone - nf4 is the worst quality. Better use q4 instead.

@PirateGirl do not use nf4 , better use q4 which has better quality

@PirateGirl exactly. i'm pulling the plug, no more thinkerin, downloading..... very soon, i hope, situation will be much clearer, there to many options, too little time to try it all, and in my case, to little, if any, noticeable difference.

@stduhpf893 I actually fomoed last night to Q5_1 and like you said it's clearly slower than NF4. About 5s. slower generation times. However, the time cap is about the same in 1024x and Full HD. So that's good. My initial tests suggest that NF4 make better looking images. Out of 5 tests with total of 25 test images, only 30% of GGUF images looked better than NF4 images. With NF4 images having clearly better details by wide margin (personal taste is always subjective, but I tried to be neutral with my opinion). Which is of course strange, because GGUF is supposed to be much more faithful to FP16 quality. However, I haven't tested text accuracy yet, but I assume that GGUF is better. edit: After some testing I have to say no, GGUF version is not better at text than NF4. With both of them being about equal and nailing it in few tries.

@stduhpf893 I don't see much advantage of using FS GGUF Q5_1 than FP8 version. It's only few seconds faster per image than FS FP8 with fp8_e4m3fn weights and clips. But I guess all versions have a use case. If you have less than 12Gb vram and 32Gb system ram then some GGUF version might be the best for you. There's even Q2_K and Q2_K_S versions now too. Just released a few hours ago.

@mirek190 I disagree. I actually just tested FD GGUF 4.0 and FD NF4v2. NF4 version is clearly faster and generates nicer and better quality images, imho. I think I'm going to stick with NF4 versions and use them as my primary flux versions from now on.

@mirek190 Yep, FS GGUF Q5_K_S is clearly the best version I've tried now. About 7s. slower generation time per image for me, but it's absolutely worth for the quality bump. I recommend it anyone.

@RalFinger ah, do we just change the .zip to .gguf or do we run winzip or something?

@RalFinger sadly that also means tools like civitai-helper on automatic1111 and forge will not be able to connect this "repo" to get metadata and such.
The model used will also not be detected automatically by civitai when uploading images generated locally.
Nothing you can do about it, but it is a fact, unless someone has a solution for it that I don't know about...?

ultimate sdupscale

If you don't have a really really slow card try Dev Q4?

S version really needed. Flux.Dev - non-commercial. You can't make even cover for you music-album, if you want to sell it.
But this checkpoint also non-commercial. What is the need for a shnell model with a ban on commerce? I'm guessing too.
And... Do you remember that many users do not have 12 GB of VRAM?

I tried to run Q8 with 12gb and 32gb ram, 1 single image 1024x1024 and the sampler never starts to denoise, the vram meter go up to 99% and stuck there. I need to restart ComfyUI to solve it. How did you managed it?

@Diana111 I run Q8 with 12GB, you can use Q4, ppl there are running it with 3060 8GB.

@JayNL Thanks! But it's absolutely useless for me to run checkpoint, learn how to work in it, and even download it so that it takes up space on an SSD if I can't even make a cover for my selling book in it.
And in any case I have 2080.

@Diana111 I tested a few FD models today and I just found out about the non-commercial license. I will now delete all FD models I've tried. This is about principles more than if miracle happens and I'm able to sell some of my images for a few yen.

I find this site is far too slow with any FLUX model, and I cannot run any locally. So, I registered with fal.ai with my GIT Hub username and pay $10.00 - That allows me to Run the [schnell] version 333 times for $1.00 which is not bad. They have many other models available. They allow LoRA developing for just $5.00.

You pay for what you use, as in GPU and CPU - basically a win, win, if you ask me. You can top-up with just $5.00 if you want. It's entirely up to you.

Here you go: https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/1050