LTX2.3 FP4 Models

yes.. that is a issue with comfy atm, they broke a shit load of stuff with the recent update.
you can try this on comfy start and see if it helps --disable-dynamic-vram
i had to recode a bunch of crap in the multigpu loaders to fix this.
and its nothing to do with the models, its the loaders because comfy added a bunch of crap that loaders arn't expecting so.. funny errors.

i use dual clip loader, not gemma loader, i know the most rencent multigpu loaders do work if you --disable-dynamic-vram on comfy start, but will crash if you don't have --disable-dynamic-vram set.
we are in for a rough week while everything gets fixed.
at first i though it was issues with ltx2.3, then found even my ltx2 workflows were bricked.

also, my workflow is in all my videos if you wanted to check those, it is multigpu setup, but no reason you can't just set it all to use same gpu.

@Gericho222 yeah, i fixed my errors, but now ive manually rewrote a bunch of code in the multigpu nodes, so can't update it now, lol.
i did try there most recent version, it crashes if you have have dynamic vram on, so went back to my version which allows me to control it per node.
so i can turn off dynamic vram from vae and clip, but leave it on for unet if I want.

@Minase460 its the loading of the model issue, comfy, at least of my last update, doesn't know how to load ltx2.3 right, it assumes ltx2 and errors. i use the checkpoint loader instead of unet/model loaders to fix it. (there is meta data in the files that tells the checkpoint loader how to use them, as long as you comfy ui is updated enough)

more blocks are in fp4 making it smaller, and even some of the ones you shouldn't really put in fp4, i put in fp8 instead of leaving them at bf16, i just wanted to test pushing more of the model into fp4/fp8 to save on vram, 14.8gb vs 19gb, the extra 4gb means another 10 seconds to length or higher resolution (it will do 1920x960 10 seconds in 24gb vram)

even beyond that, if you are running on lower vram cards, the swap should be faster, i don't really activly swap while generating, but when i do, the more in fp4, the faster the swapping is.

i don't know, not sure if 3xxx series can run fp4.
fp4 isn't "officially" supported by 4xxx series either but it works, not any faster but saves vram.
won't know unless you try, i would say maybe try to find a smaller model to test on so you don't waist time downloading 14-20gb.
the worst that would happen i think is it just expands it back out to fp8.

let me know if it does work though, im curious to know.

@FantasticFever the issue i have with gguf's and the reason i stick with fp4, is as soon as you add a lora to a gguf models, it starts swapping data, even if the model can fit in vram, fp4 doesn't do that, it just injects the lora into the fp4 model at fp4 weights, so if the model and latent fits in vram, no active swap so its faster. but again, like i say allot, my workflow is multi gpu, so model, gemma/clip, and vae's all have there own cards, so i do everything I can to prevent data swaps, everything is fully on a card. 1920x960 10 second gens in less then 2 minutes right now. and I can only do that because I have the fp4 distilled "extreme" down to 14gb, leaving almost 10gb for the latent on the 4090.

its all about use case, I have reasons for preferring fp4 over gguf, actually i kind of despise gguf's because the lora swap issue. i just don't like any swapping that isn't under my control with distorch.

@FantasticFever yeah, i had to turn dynamic vram off, with it on, my generations go from 3 minutes to 10 minutes.

im still figuring that out, i have some small issue on my end with i2v, but not sure where its coming from yet, not sure if its model or something else in my workflow, i have alot of issues with decode in my setup, so my issue is probably related to that.
but even with the decode issue, my fp4 extreme distilled looked great on itv at 2048x1024 except for the small 5 frame glitch at the start.

its a clip, if your using dual clip loader, put it in your clip folder, if your using it with the gemm3 loader put it in diffusion models and treat it as the "model"

what loader?
i use distorchmultigpu loader, so it doesn't even ask, it just loads what ever it is.

@bobatoms it is a mixed precision model, one or 2 things are in bf16. but 95% is fp4/fp8 mix.
what happens in hardware that cant run in fp4, like mine, is it keeps it in fp4 in vram, and very quickly, during generation, expands to fp8, samples, discards fp8. and it does this on a per block layer, so its not like its doubling in size during generation, each block layer maybe consumes an extra 500mb-1gb during inference. it sits in vram at 13gb.
that's how all this work for non blackwell gpu's, as long as you can run fp8, you should be able to run these models, just not as fast, but still faster then swapping fp8 to and from ram/hd.

check with the people posting ggufs, i haven't done much of anything with int8 outside llm's and from my understand int8 is horrible for image/videos models. the fp4's as far as I know do still work on 3x series cards for memory savings, just no speed boost that you would get on the blackwell cards, i don't even have a blackwell card, im on 4x series, so only memory savings, well, do get some speed boost vs fp8 or higher because don't need to swap during generation because my extreme varient is 14gb.

@ZelashZelash all my videos have workflows attached. they are all multi gpu setups. first thing I would suggest, turn off any lora's, this is a big issue with my extreme fp4 model, working on that, i don't really run that one with lora's. and it really prefers low steps, with dpm++ 2m sde heun gpu it runs at 4 steps, quality degrades at 5, euler a likes about 6 steps. (these are either beta11 or sgm uniform) this is my custom sigma i've been runing with dpm++ 2m sde heun gpu, 1.0, 0.925, 0.725, 0.421875, 0.0. play with the 0.925, between 0.95 and 0.90, depending need. to get that small size its very compressed so very fickle.

@MrTitsworth id be curious to, i just know ive read a lot that dit models don't do well in int8, and was that a typo or are you really running a 5090 on 8gb ram?
i guess its a trade now, prices are as equally insane.

@ZelashZelash hey, just wanted to follow up on this. i run custom sigmas for stage 2 upscale, euler ancestor.
for quick run - 0.91218, 0.725, 0.0
for more detailed run - 0.91218, 0.909375, 0.725, 0.10546875, 0.052734375, 0.0
that second one helps when going from reaaly low res base, as it helps add detail and fix anatomical issues.
the reason for the reaaly crazy low number ones, is for some reason this, 0.421875, is mathematically significant, and those low numbers i based on that.

@MrTitsworth yeah, its all about need, i made these 14gb fp4's because I wanted the model fully on card with no swap while still getting decent resolution/length outputs.
and the main reason is my 4090's are on 1x pcie risers (this machine was built for 3d rendering), so active swapping is a huge no for me, so 30gb models, i don't even look at.