This workflow can do a couple of things: it can make a character from a reference image copy the pose from a reference video (while keeping the original background), and it can also completely swap that character into the reference video, making them perform the same actions. Based on this idea, I also tested swapping just the character's head into the reference video, but I'm still debugging some issues with the results. If I find a good solution, I'll release that workflow too. For important notes on how to use this, please see the "Workflow Testing and Usage Instructions" below.
💻I've already set up an online ➡️ workflow for you so you can quickly try out the effect.
🎁Bonus: If you're signing up for the first time, you can get 1,000 free RH Coins by using my link and the invite code ➡️rh-v1182. Plus, you'll get another 100 RH Coins for logging in daily.
🚀Workflow Testing and Usage Instructions:
1. First, a big thank you to eddy for open-sourcing several Loras. When combined with KJ's Wan2.2 animate workflow, these Loras have great performance in both character consistency and video motion. It's especially important to note that "lightx2v_elite_it2v_animate_face" already has the lightx2v acceleration built-in, so you don't need any other speed-up Loras. This Lora also helps maintain the reference character's consistency, so I recommend a strength value between 1.0 and 1.2. If you need high consistency with the reference image character, I suggest a strength of 0.35-0.5 for "WAN22_MoCap_fullbodyCOPY ED". If you want to lean more towards the character in the reference video, a strength of 0.7-1 is better.
2. Because this involves loading the Wan2.2 model and SAMSegment, it requires a lot of VRAM. That's why I've enabled WanVideo Block Swap by default. In my testing, the entire workflow can run on a 24G GPU. However, I recommend using the 48G GPUs on RunningHub for a much smoother experience.
3. For different kinds of reference videos, I've preset two different masking methods in the workflow. You only need to choose one of them. If your reference video has only one character, use the "Single-character usage" group. If your video has multiple characters and you only want to mask certain areas, use the "Multi-role usage" group.
4. This workflow can generate longer-than-usual videos, but I don't recommend using it for anything over 30 seconds. In my tests, I found that for a 20-second video, the character's consistency starts to decay around the 10-second mark, and the color tone also shifts slightly. I believe this is caused by the influence of the reference video during the context looping process. That's why I'd recommend keeping the videos you generate with this workflow to around 20 seconds and no longer than 30. Going past 30 seconds could lead to unpredictable degradation or a serious loss of consistency.
5. For more detailed instructions, please see the notes inside the workflow.
Description
Version 1.1 mainly builds on V1.0 by adding two upscaling groups to help improve the blurriness that happens during character transfer. They've been tested and work correctly on both 24G and 48G GPUs, and the effect is more noticeable when using SeedVR2 for upscaling. For the V1.0 workflow, you can also try using different schedulers and sampling steps to get different video effects. For specific usage instructions, please see the description in the V1.0 version.
FAQ
Comments (23)
in version 11, the sampling settings were lost
Could you describe what you mean by "lost"? What is the specific issue you're seeing? The only difference in V1.1 compared to V1.0 is the addition of two upscaling groups; everything else is identical to V1.0.
For some reason, the "sampling" section didn't have any links to the other sections..... but anyway, after trying to launch your thread, I realized that I needed a list of loras, vae, and models) maybe a separate annotation somewhere on the side?
The ouput is great but the face i get is somewhat off, what could it be? Im using a fairly high res face pic
You can try adjusting the strength values for the "lightx2v_elite_it2v_animate_face" and "WAN22_MoCap_fullbodyCOPY ED" Loras. For specific instructions on how to adjust them, see the description in the V1.0 version.
SeedVR2ExtraArgs，closed source node？
SeedVR2 is an open-source node. If you have it installed locally, make sure to update it to the latest version.
@emilkwok 3Q
no matter what reference image i use, the face is always an asian woman. how do i get it to use the original reference face?
I'm not sure I completely understand your question. This workflow either makes the character from the reference image perform the actions from the reference video, or it swaps the reference character into the reference video. The "lightx2v_elite_it2v_animate_face" Lora is there to improve facial consistency, so you can try increasing its strength to get a better result. However, as I mentioned in the instructions, character consistency will start to fade as the video gets longer. Taking all these factors into account, you can try adjusting the relevant parameters to see what works best.
Disabling the mask fixed it. I didn't need to the video background anyway.
Can we just change the clothes from the reference video and keep the rest the same?
Yeah, you can totally do that. I'd suggest trying it with a reference image of the clothing on a white background. For the mask, you'll just need to mask the clothing on the character in the reference video.Looking forward to seeing your awesome work!
@emilkwok thank u very much
where to download this: lightx2v_elite_it2v_animate_face?
can't find anywhere
I just checked the post from Eddy, the creator of this Lora, on HuggingFace, and it looks like he deleted it. I don't have a copy of it saved locally. If you want to use this Lora, you can use it online on RunningHub through my link. If you want to use it on your local machine, you'll probably have to find it somewhere else.
Please add the workflow on Tensor.art
Hi, where can I download the files vitpose-l-wholebody.onnx and yolov10m.onnx? thankyou. :)
You can find the download links for both of those files at this URL:https://github.com/kijai/ComfyUI-WanAnimatePreprocess
@emilkwok Thank you very much.
Wish I'd read comments first before even wasting my time on this. I hate workflows that use models and nodes that are near impossible to find, or are impossible to find because they don't exist anymore.
Marvelous~But How did you find out about these LoRa files and how to use them? I can find download links, but I can't find documentation for these LoRa files.
So far this one make very stable result near the reference)
But (on multi role usage) working not too stable as single.And in 2 top Loras have "lora key not loaded" many times , in console. So, don't know they are working, or not🤷