This is a vid2vid workflow to transfer the style from photos onto the video.
This workflow is based on the ideas from: https://civarchive.com/models/372584/ipivs-morph-img2vid-animatediff-lcm
Models Needed
Your favourite SD1.5 Checkpoint. Put in the models/checkpoints folder
Controlnet. control_v11p_sd15_lineart_fp16.safetensors. Install using the manager, search for "controlnet"
AnimateLCM. https://huggingface.co/wangfuyun/AnimateLCM Download AnimateLCM_sd15_t2v_lora.safetensors and place in the models/loras folder. Download AnimateLCM_sd15_t2v.ckpt and place in the models/animatediff_models folder
IPAdapter Models. I am asking you to install more than what is needed so that if you change some of the options, it will still work. If you are getting errors, you will have to install the exact model that you are missing.
CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors and CLIP-ViT-bigG-14-laion2B-39B-b160k.safetensors. Install using the manager, search for "clip".
ip-adapter_sd15.safetensors, ip-adapter_sd15_light.safetensors, ip-adapter_sd15_vit-G.safetensors, ip-adapter-plus_sd15.safetensors. Install using the manager, search for "ipadapter"
How to use the Workflow
Load the workflow and install the missing custom nodes using the manager. Restart ComfyUI
Teal nodes are the models which you need to load. Blue nodes are the user inputs you need to provide.
Load your video into the load video node. If your video is in landscape, set force_size to ?x512. Set frame_load_cap to 32 for testing out the workflow with the first 32 frames of your video. Set frame_load_cap to zero to use the whole video.
Load the 4 images from which you want to transfer the style.
Edit the frames in the CreateFadeMaskAdvance nodes. You must decide which frame you want to start applying the style and which frame you want to end applying the style. Set the mask to zero at the frame before you want to start and after you want to end. Set the start and end frame to 1.0. For example, if I want to apply the style from frame 90 - 135, you would have to do this: 89:(0.0), 90:(1.0), 135:(1.0), 136:(0.0)
Enter your prompts as needed.
Amend the controlnet as depending on your video. I use lineart because I wanted to transfer the style from artistic images.
Description
FAQ
Comments (6)
Hello. I need you help. Some nodes i can't identify. Could i ask you add snapshot or updated workflow?
Could I bother you to send me a DM with an image of which nodes you are having problems with? I'll identify them for you.
These are the custom nodes that you need for this workflow: Video Helper Suite, IP Adapter Plus, KJ Nodes, AnimateDiff Evolved, Controlnet Auxilliary Preprocessors, Advanced Controlnet, Impact Pack, Frame Interpolation
I just updated my comfyui and the workflow still loads in correctly. I do not think it needs any updating.
@PixelMuseAI thanx
How did you get the video to look smooth? When I try this the generative look is so strong and every frame morphs so much
Hmm, I might need to take a look at this workflow again. There might have been some code change in the nodes/ ComfyUI that changes the behaviour of the workflow.
When I released the workflow, it worked to transfer the style to get the video I posted.