PONY CLIP 100k Finetune
JoyCLIP is a further advancement of this CLIP
Note this is still a great base to start a finetune of PONY CLIP as the gradient was 50 out. Starting with this model you will be in the 4-8 range.
100k is a full finetune of base pony CLIP-L and CLIP-G
100k can be used in any model base V6, Autism, Anime or Realistic (Even non-pony SDXL Models)
CLIP-G took 68GB and 30 hours to train.
Forge users, you will need to download Comfy UI as CLIP replacement is only supported for FLUX, this may apply to Auto 1111 also. Once a model is saved with replaced clip it can be used in Forge or Auto
Can be run in any UI (Forge, Auto1111, Comfy UI) the model will be downcast by default. These settings improve complexity but are not required. (Full FP32 is not recommended but FP32 CLIP is)
Comfy UI --fp32-text-enc OR --force-fp32
Forge/Auto1111 --clip-in-fp32 OR --all-in-fp32
Description
FAQ
Comments (10)
The Laion8b CLIP-G seems to necessitate a more natural-language prompt style, as opposed to booru tags. Will need to do more testing.
Either way, it seems like the 100k Pony CLIP-L is a nice alternative to base PONY CLIP without having to change up prompting. Nice work!
Thank you
Can you do one for illustrious?
Couple of notes... Using your own model from this card.
I don't see much difference between the new 100k_CLIP_L and the older PONY_CLIP_L-FP32, the results are nearly identical. There are change, but mini changes. From what I can tell, the older FP32 yields the better results. They are also exactly the same size on disk.
During the testing above, I had to use the Laion-8B CLIP because the other CLIP-G model you have for PDXL just produced lineart, which wasn't what I was prompting for.
The two older "FP32" CLIP's listed in this card is showing as based on "Other" model, perhaps changing them to "Pony" or whatever model they're trained for could clear some confusion.
The LAION-8B CLIP is also listed as based on "SD1.5", don't know if that's correct.
-
Other than that, I find the Laion-8B CLIP to produce better fingers in all models I've tested, it's good! Don't know why you teased me with that "do not use" label but now I'm using it as default :P
Well I updated the 100k model today as it had a chance to go from 0.01 loss to 0.003 loss, If the 8B Laion CLIP is working that means that SDXL was likely merged back into the model your running. The 100k CLIP is made to work with either PONY-G or Laion-G, so it can use WD14 Booru tags or natural language.
The biggest difference seed to seed is "name triggers" with Laion + 100k if you compare seed to seed Laion with 100k to Laion with FP32 PONY or base PONY you can see the difference
I am running into an "issue" where a lot of the models I use appear to have a trained CLIP.
Is there a way to "integrate" the 100k CLIP-L with a checkpoint's existing CLIP? I tried using CLIPMergeAdd in ComfyUI but that didn't work properly at all.
dual clip loader using the --fp32-text-enc if you want them in FP32 - then save as checkpoint
Details
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.