This workflow generates an image with SD1.5, then uses Grounding Dino to mask portions of the image to animate with AnimateLCM. It animates 16 frames and uses the looping context options to make a video that loops. The denoise on the video generation KSampler is at 0.8 so that some of the structure of the original image generated is retained.
Based on: https://comfyworkflows.com/workflows/14d8d52b-1297-4d58-8469-2bdab4f5ebf1
Models needed:
1) Your favourite SD1.5 checkpoint
2) AnimateLCM and LORA (https://huggingface.co/wangfuyun/AnimateLCM): Put the Lora in models/loras and the model in models/animatediff_models
3) SAM Model (Manager > Install Models, search for SAM): sam_vit_b_01ec64.pth
4) Grounding Dino Model (Should be automatically downloaded)
5) Your favourite upscale model (Manager > Install Models, search for upscale): 4x_NMKD-Siax_200k.pth
Instructions
1) The cyan nodes are the models needed for the workflow
2) The blue nodes are the user inputs that are required. There are 3 prompts that are required for this workflow. a) Image generation prompt: Describe what you want in the overall image here, e.g. "Mountains, River, Clouds" b) GroundingDino prompt: Describe the object within the image that you want animated, e.g. "River" and c) the video prompt: Describe what you want the animated object to do, e.g. "Flowing River"
3) For those who are having insufficient VRAM, you can disable the "Upscale Image (Using Model)" node. This will reduce the input resolution to the KSampler to 256 x 256 x 16 frames. If this works, you can bypass the "Upscale Image By" node as well, to increase the resolution to 512 x 512 x 16 frames. The original workflow, without any nodes bypassed will result in a 1024 x 1024 x 16 frames video. Thanks to mojoflojo who helped with testing on GPU with lower VRAM.
Description
FAQ
Comments (8)
Wow his workflow is really great! I'm trying to move the clouds, by changing the mask to the sky. and changing the prompt to :: clouds moving , or clouds in the wind: but for a reason I don't understand no cloud animation here. where am i wrong?
1) Check that your grounding dino masks are masking the correct part of the image to be animated. If it is not, then you need to amend the prompt in the "GroundingDinoSAMSegment (segment anything)" node.
2) Look for the Apply AnimateDiff Model node, drag out a new node from scale_multival and add a ADE_MultivalDynamic node. Change the value to something like 1.5 and try again. Sometimes, the scale_multival value is too low for things like clouds to be animated.
@PixelMuseAI thank you it worked, but the ksampler is now very very slow... which node do you think is better to recompose the image? I tried but the mask seems to pixelate the composition
@TontonFrancky if the ksamper is slow, check your VRAM usage. reduce the image resolution.
@PixelMuseAI thanks, found the problem. I Enhanced the workflow, I found where to add more frames to render and how to mix the different mask. worked great
@TontonFrancky the reason why I keep the frames to 16 is because the context window for AnimateDiff is 16 frames. If you increase the frames, you might get some morphing every 16 frames.
But I'm really glad you managed to improve on the workflow and made it your own.
@PixelMuseAI no in the repeat latent batch right click and convert widget to input then convert amount to input. then add primitive integer node then choose the number of frame for your render aka 128
@TontonFrancky Hi tontonfrancky, if you have an updated workflow going, would you mind uploading yours?