Fine-tune of AnimateDiff mm_sd_v15_v2.ckpt to improve coherency with moving objects
Should generally have better coherence, but can be worse for some cases
Tested with ComfyUI AnimateDiff
Stable diffusion model used in examples: DreamShaper8
Recommended to use the negatives embeddings as well: BadDream, UnrealisticDream
Examples have been interpolated from 8 fps to 16 fps
Description
FAQ
Comments (19)
super awesome work and nice examples to test this model. cool and nice and thanks for the share.
why would you make this Clip Skip 2?
Some Stable diffusion models need clip skip -2 so I just put it there. Don't worry about it unless the stable diffusion model you're using needs it.
To make it work right away, add this to mm_zoo.json inside your animatediff extensions folder:
"d683556a3b72686b9909d97755054b344869431cddb26853da2178108da195f5": {
"name": "improved3DMotion_improved3DV1.ckpt",
"arch": 2
I don't have an extensions folder or any json file under that name, are you using comfy?
@Indelible_Soul If you have animtediff installed (which this model is for) you do have the folder inside the extensions folder. You mean to say you don't even have an extensions folder? Did you install A1111 correctly? :P
@olivetty I'm using comfyui
@olivetty Also already using it and its awesome!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
There are a few ways to use Animatediff, CMD only, ComfyUI, and Auto1111. You should specify.
@RyogaVee If you look above I did
@Indelible_Soul So did you have to add this file or it was not needed for comfyui? How did you install it? Just want to confirm and let others know as well.
@KINGLIFER With latest updates this shouldn't be needed anymore!
Excellent work. Is there a guide that talks about how to do this additional training?
There's a repo but not a ton of explanation on how to train: https://github.com/B34STW4RS/AD-Evo-Tuner
Also, I froze the module and specifically trained the mid-layer, which needs custom code. Hopefully, we can get options for it in the trainer itself and better explanations for how to use it.
Great work, your input and effort is much appreciated !!
Please excuse my ignorance, but in what folder in Comfy does this need to go? TYVM!
I assume Custom_Nodes/Comfy-UI-AnimateDiff-Evolved/Models
gave this a spin and got some awesome results so far and really would like to know how we can improvise and improve on this as this model gives a lot of smooth movement and action ! How was this model trained and what can be done to add to this that will make a great model !
They will never tell us sadly.