This model was created by CiaraRowles, posted to Civitai.com with permission. Basic Getting Started Tutorial available at the Civitai Education Hub!
TemporalDiff is a finetune of the original AnimateDiff weights on a higher resolution dataset (512x512).
Testing so far indicates a higher level of video coherency than the original weights, i also adjusted the stride from 4 to 2 frames to improve how smooth the motion was.
Current limitations are that the labelling for my dataset was a bit off, so it has slightly reduced ability to interpret the prompt, i'll be releasing a new version that fixes that soon.
This should work the same as any the base model in terms of use, just drag and drop it into comfy ui or the animatediff repository and use as normal.
This does not require any additional memory to run as the generations were 512x512 before, the training was just done at 256x256.
Description
FAQ
Comments (35)
Hello! Mind if I ask for a link of a tutorial or post about the usage of this extension? It looks very interesting and I would love to try it!
I will definitely put something together! Keep an eye on education.civitai.com!
@theally thanks so much!
@thesilvermoth Beginner’s Guide to AnimateDiff – Civitai Education
@theally yeeeeees! Thanks a lot!!
A tutorial for your model upload is missing.
What do you mean with
"This should work the same as any the base model in terms of use, just drag and drop it into comfy ui or the animatediff repository and use as normal."
What is "normal" here? ;)
- How to prompt?
- What can be animated?
- How to animate a sequence?
- What do people have to care about to receive good results.
The text is lifted directly from Ciara's HuggingFace - i'm not the author here. You'll have to do a little digging, or wait for me to get a tutorial up on education.civitai.com :)
heya, dev here, look into the various methods of using animate diff between their repo, the automatic extension or comfy, in these they give you the option of using v14 or v15 or v15v2 models, you can just drop temporaldiff into the models folder next to these other models and you'll be able to select it same as the others.
the prompting should be pretty much the same although i haven't tested it extensively to give full recommendations, the main thing is it was trained on a youtube dataset over a shutterstock dataset so expect motion to skew more towards youtube videos than stock footage.
@Strawberries Hi thx for the hints :)
@theally Thx a million for lifting the usage secrets creating a beauty guide :)
@theally The extension doesn't load for me :(
I followed the Beginner's Guide for installing this locally on Automatic1111. But Im stuck at the step where you're supposed to open AnimateDiff in the accordion on the left. The problem is that it doesn't even show up on the left...
@dogma4000 When you load Automatic1111 do you see any errors in the console window relating to AnimateDiff? When you go to the Extensions tab, do you see that AnimateDiff is installed and up to date?
Hola,
¿algún tutorial?
Hola! ¡Aún no! Pero pronto publicaré en nuestro centro de Educación: education.civitai.com 😃
I think these are the comfyUI and A1111 integrations this model is used with:
https://github.com/ArtVentureX/comfyui-animatediff
https://github.com/continue-revolution/sd-webui-animatediff
(Haven’t yet used it myself.)
Does it work with SDXL checkpoints?
No, these ones don't, unfortunately. The injection of the motion modules only works with SD 1.5 based models currently!
I always get two scenes even when under 75 tokens. Running on 8GB VRAM with xformers 512x512
pos/neg prompt needs to be same length
https://rentry.org/AnimAnon-AnimDiff
Enable Pad prompt/negative prompt to be same length and Batch cond/uncond and click Apply settings in Settings. You must do this to prevent generating two separate unrelated GIFs.
try under 32
doesn't work for me I followed every step in the tutorial. It starts generation but the end result is always just random colors using toonyou_bet3 as model and the temporaldiffmotion as animediff model
I have the same problem.
Did you tried on ComfyUI or Automatic1111 ?
its Automatic1111 but i managed to fix it and it works now
@nonozone222653 how
@nonozone222653 What was your fix? :)
basically, you need to restart the UI and try to use only 512x512 at first without any high res fixes and fewer prompts that is how I fixed it
if all else fails, try using RealisticVision or something made for animation, idk what toonyou_bet3 is, but if it's not meant for motion it won't work
Even now that the V3 motion model is out, I still find that this works best for my vid2vid generations. I don't really do txt2vid so I can't comment on that.
So, any chance you can be convinced to do a similar finetune for V3? Maybe it's something you could teach others how to do?
Where was he installed
Guys, can you please tell which Checkpoints and VAEs and maybe negative prompts work better with this model? ^_^
Does not work with AnimateDiff extension in A1111
*** Error completing request 3.03s/it] *** Arguments: ('task(1bymk1qh5mlbdd0)', <gradio.routes.Request object at 0x000001E87959F490>, 'female figure skater performing', '', [], 1, 1, 7, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', 'Use same scheduler', '', '', [], 0, 20, 'DPM++ 2M', 'Automatic', False, '', 0.8, -1, False, -1, 0, 0, 0, 0, False, 1, False, 1, 1, 30, 1, 0, -4, 1, 0.4, 0.5, 2, False, '[How to set parameters? Check our github!](https://github.com/scraed/CharacteristicGuidanceWebUI/tree/main)', 'More ControlNet', 0, 1, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', <scripts.animatediff_ui.AnimateDiffProcess object at 0x000001E8799D9DE0>, False, 0.6, 0.9, 0.25, 1, True, False, ControlNetUnit(is_ui=True, input_mode=<InputMode.SIMPLE: 'simple'>, batch_images='', output_dir='', loopback=False, enabled=False, module='none', model='mm_sd15_v3_sparsectrl_rgb [9f17d62f]', weight=1.0, image={'image': array([[[ 76, 33, 113], *** [ 75, 32, 118], *** [ 75, 31, 114], *** ..., *** [ 42, 18, 94], *** [ 43, 18, 92], *** [ 41, 19, 92]], *** *** [[ 81, 33, 114], *** [ 80, 32, 119], *** [ 76, 29, 115], *** ..., *** [ 41, 17, 94], *** [ 42, 18, 93], *** [ 45, 18, 95]], *** *** [[ 79, 27, 115], *** [ 78, 30, 118], *** [ 74, 29, 115], *** ..., *** [ 41, 17, 93], *** [ 41, 18, 93], *** [ 45, 18, 94]], *** *** ..., *** *** [[212, 216, 224], *** [215, 219, 225], *** [221, 225, 232], *** ..., *** [172, 172, 182], *** [171, 169, 180], *** [178, 175, 187]], *** *** [[223, 228, 235], *** [219, 223, 230], *** [216, 219, 227], *** ..., *** [171, 170, 180], *** [176, 176, 185], *** [178, 176, 187]], *** *** [[217, 223, 227], *** [219, 221, 230], *** [222, 223, 230], *** ..., *** [172, 171, 180], *** [175, 173, 181], *** [181, 180, 188]]], dtype=uint8), 'mask': array([[[0, 0, 0], *** [0, 0, 0], *** [0, 0, 0], *** ..., *** [0, 0, 0], *** [0, 0, 0], *** [0, 0, 0]], *** *** [[0, 0, 0], *** [0, 0, 0], *** [0, 0, 0], *** ..., *** [0, 0, 0], *** [0, 0, 0], *** [0, 0, 0]], *** *** [[0, 0, 0], *** [0, 0, 0], *** [0, 0, 0], *** ..., *** [0, 0, 0], *** [0, 0, 0], *** [0, 0, 0]], *** *** ..., *** *** [[0, 0, 0], *** [0, 0, 0], *** [0, 0, 0], *** ..., *** [0, 0, 0], *** [0, 0, 0], *** [0, 0, 0]], *** *** [[0, 0, 0], *** [0, 0, 0], *** [0, 0, 0], *** ..., *** [0, 0, 0], *** [0, 0, 0], *** [0, 0, 0]], *** *** [[0, 0, 0], *** [0, 0, 0], *** [0, 0, 0], *** ..., *** [0, 0, 0], *** [0, 0, 0], *** [0, 0, 0]]], dtype=uint8)}, resize_mode=<ResizeMode.INNER_FIT: 'Crop and Resize'>, low_vram=False, processor_res=512, threshold_a=0.5, threshold_b=0.5, guidance_start=0.0, guidance_end=1.0, pixel_perfect=True, control_mode=<ControlMode.BALANCED: 'Balanced'>, inpaint_crop_input_image=False, hr_option=<HiResFixOption.BOTH: 'Both'>, save_detected_map=True, advanced_weighting=None, effective_region_mask=None, pulid_mode=<PuLIDMode.FIDELITY: 'Fidelity'>, ipadapter_input=None, mask=None, batch_mask_dir=None, animatediff_batch=False, batch_modifiers=[], batch_image_files=[], batch_keyframe_idx=None), ControlNetUnit(is_ui=True, input_mode=<InputMode.SIMPLE: 'simple'>, batch_images='', output_dir='', loopback=False, enabled=False, module='none', model='None', weight=1.0, image=None, resize_mode=<ResizeMode.INNER_FIT: 'Crop and Resize'>, low_vram=False, processor_res=-1, threshold_a=-1.0, threshold_b=-1.0, guidance_start=0.0, guidance_end=1.0, pixel_perfect=False, control_mode=<ControlMode.BALANCED: 'Balanced'>, inpaint_crop_input_image=False, hr_option=<HiResFixOption.BOTH: 'Both'>, save_detected_map=True, advanced_weighting=None, effective_region_mask=None, pulid_mode=<PuLIDMode.FIDELITY: 'Fidelity'>, ipadapter_input=None, mask=None, batch_mask_dir=None, animatediff_batch=False, batch_modifiers=[], batch_image_files=[], batch_keyframe_idx=None), ControlNetUnit(is_ui=True, input_mode=<InputMode.SIMPLE: 'simple'>, batch_images='', output_dir='', loopback=False, enabled=False, module='none', model='None', weight=1.0, image=None, resize_mode=<ResizeMode.INNER_FIT: 'Crop and Resize'>, low_vram=False, processor_res=-1, threshold_a=-1.0, threshold_b=-1.0, guidance_start=0.0, guidance_end=1.0, pixel_perfect=False, control_mode=<ControlMode.BALANCED: 'Balanced'>, inpaint_crop_input_image=False, hr_option=<HiResFixOption.BOTH: 'Both'>, save_detected_map=True, advanced_weighting=None, effective_region_mask=None, pulid_mode=<PuLIDMode.FIDELITY: 'Fidelity'>, ipadapter_input=None, mask=None, batch_mask_dir=None, animatediff_batch=False, batch_modifiers=[], batch_image_files=[], batch_keyframe_idx=None), False, 0, 1, 0, 'Version 2', 1.5, 0.9, 0, 0.5, 0, 1, 1.6, 0.2, 0, 0.5, 0, 1, 1, 1, 0, 0.5, 0, 1, False, 1, 0.8, 3, 0, 0, 150, 4, False, 0, 0, 150, False, 'Constant', 0, 100, False, False, False, 2, 0.1, 0.5, 0, '', 0, 25, 1, False, False, False, 'BREAK', '-', 0.2, 10, False, False, 0, None, [], 0, False, [], [], False, 0, 1, False, False, 0, None, [], -2, False, [], False, 0, None, None, False, '0', False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False, 5, 'all', 'all', 'all', '', '', '', '1', 'none', False, '', '', 'comma', '', True, '', '20', 'all', 'all', 'all', 'all', 0, '', True, '0', False, 'SDXL', 'Standard', 'dynamic', 'none', '', '', None, None, False, None, None, False, None, None, False, 50, 'Long prompt to short prompt', '', 0, 8, 1000, 0, False, '0.1', '0.1', '1', '1', None, None, None, None, None, None, None, None, None) {} Traceback (most recent call last): File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 410, in reduce return _apply_recipe(recipe, tensor, reduction_type=reduction) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 233, in _apply_recipe _reconstruct_from_shape(recipe, backend.shape(tensor)) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 198, in _reconstruct_from_shape_uncached raise EinopsError("Shape mismatch, can't divide axis of length {} in chunks of {}".format( einops.EinopsError: Shape mismatch, can't divide axis of length 1 in chunks of 2 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\AI\stable-diffusion-webui\modules\call_queue.py", line 57, in f res = list(func(*args, **kwargs)) File "D:\AI\stable-diffusion-webui\modules\call_queue.py", line 36, in f res = func(*args, **kwargs) File "D:\AI\stable-diffusion-webui\modules\txt2img.py", line 109, in txt2img processed = processing.process_images(p) File "D:\AI\stable-diffusion-webui\modules\processing.py", line 839, in process_images res = process_images_inner(p) File "D:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 48, in processing_process_images_hijack return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs) File "D:\AI\stable-diffusion-webui\modules\processing.py", line 987, in process_images_inner x_samples_ddim = decode_latent_batch(p.sd_model, samples_ddim, target_device=devices.cpu, check_for_nans=True) File "D:\AI\stable-diffusion-webui\modules\processing.py", line 627, in decode_latent_batch sample = decode_first_stage(model, batch[i:i + 1])[0] File "D:\AI\stable-diffusion-webui\modules\sd_samplers_common.py", line 76, in decode_first_stage return samples_to_images_tensor(x, approx_index, model) File "D:\AI\stable-diffusion-webui\modules\sd_samplers_common.py", line 58, in samples_to_images_tensor x_sample = model.decode_first_stage(sample.to(model.first_stage_model.dtype)) File "D:\AI\stable-diffusion-webui\modules\sd_hijack_utils.py", line 22, in <lambda> setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs)) File "D:\AI\stable-diffusion-webui\modules\sd_hijack_utils.py", line 36, in __call__ return self.__orig_func(*args, **kwargs) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "D:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 826, in decode_first_stage return self.first_stage_model.decode(z) File "D:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 90, in decode dec = self.decoder(z) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "D:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 630, in forward h = self.mid.block_1(h, temb) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "D:\AI\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py", line 131, in forward h = self.norm1(h) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "D:\AI\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_mm.py", line 86, in groupnorm32_mm_forward x = rearrange(x, "(b f) c h w -> b c f h w", b=2) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 487, in rearrange return reduce(tensor, pattern, reduction='rearrange', **axes_lengths) File "D:\AI\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py", line 418, in reduce raise EinopsError(message + '\n {}'.format(e)) einops.EinopsError: Error while processing rearrange-reduction pattern "(b f) c h w -> b c f h w". Input tensor shape: torch.Size([1, 512, 64, 64]). Additional info: {'b': 2}. Shape mismatch, can't divide axis of length 1 in chunks of 2