To initiate the generation process, simply drag and drop an image into the orange "Load Image" node.
Feel free to adjust the main prompt and image qualifiers to refine the context as desired. The workflow base settings generate some awesome animations.
1. Loader
While any SD1.5 model is compatible, it's important to calibrate the LCM Lora weight accordingly:
•For the SD1.5 model, set the weight between 0.7 and 1.
•For the SD1.5 LCM model, adjust the weight to fall between 1 and 2.
Main Model:
•https://civarchive.com/models/306814/photon-lcm
LCM Lora:
•https://huggingface.co/wangfuyun/AnimateLCM/blob/main/AnimateLCM_sd15_t2v_lora.safetensors
2. IPAdapter Section:
In the IPAdapter Section, the first image loader is designated for the primary image, which will be cropped to match the composition size. The secondary loader is intended for a composition target image.
ClipVision:
•https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/blob/main/model.safetensors
IPAdapter PLUS:
•https://github.com/cubiq/ComfyUI_IPAdapter_plus
IPAdapter Composition:
•https://huggingface.co/ostris/ip-composition-adapter/tree/main
3. AnimateDiff Section:
Modify the motion strength using the "Multival dynamic" node.
Motion Model:
•https://huggingface.co/wangfuyun/AnimateLCM/blob/main/AnimateLCM_sd15_t2v.ckpt
4. ControlNet Section
Choose from three available ControlNets:
Tile:
•https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11f1e_sd15_tile.pth
Reference ControlNet:
•Comes integrated with the Advanced ControlNet custom nodes.
SparseCtrl:
•https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_sparsectrl_rgb.ckpt
5. KSampler / HighRes Fix
The "PatchModelAdd" node is designed to prevent deformations and artifacts when exceeding the sd1.5 aspect ratio limits. This step can be bypassed.
No adjustments needed for Ksampler options.
Fine-tune the Highres fix script based on your initial dimensions and VRAM capacity.
Choose between an empty latent or one with your image injected; the latter typically retains more of the original image and often results in less motion.
6. Output
"GMFSS Fortuna" will interpolate your video, providing a smoother and elongated output. The multiplier is usually set to 2 or 3.
Specify a custom directory and filename below the video output to maintain an organized output folder.
An optional HDR output flow has been included for your convenience.
Without altering any resolution settings, expect this workflow to:
•Complete a render in approximately 2 minutes (32 frames + interpolation)
•Utilize 16GB of VRAM
•Produce an output of 752x1344 pixels