Input a Video and inpaint a new face onto your subject character, then swap that face into the rest of the video! Lip movement is tough with just this workflow. You may want to use LatentSync to do a lipsync pass if accurate lip motion is required.
V1: Use if you don't care about facial expressions
V2: Use for portraits/close-up (degrades further away because it uses MediaPipe)
V3: Use for anything else where you want facial expressions but it isn't a portrait! (Quality is the worst with this one, may have to do a little cleaning up after)
For a demo & tutorial, you can watch here! Unfortunately it doesn't have the facial expressions versions in the video, but everything else is the same!
Description
Added A Depth/OpenPose Blend for Mouth Movements and non-portraits
FAQ
Comments (41)
whoa..
Did not expect it to work this well when I started working on the workflow
Mine shows a bunch of nodes missing, and the manager doesn't seem to be able to fetch them.
WanVideoExperimentalArgs
WanVideoVACEModelSelect
DownloadAndLoadSAM2Model
WanVideoVACEEncode
Sam2Segmentation
MediaPipe-FaceMeshPreprocessor
Any pointers?
Mediapipe: https://github.com/Fannovel16/comfyui_controlnet_aux
WanVideo: Broken/Not Updated WanWrapper version. Delete the folder and reinstall
SAM2 Seg: https://github.com/kijai/ComfyUI-segment-anything-2
Getting this error: https://gist.github.com/4DeepThought2/bd536cf0f70755570f9dca92846c0c19
I got the same error. I think you need to downgrade protobuf, at least that's what google and chagpt told me. Couldn't get it to work though.
Try V3! It doesn’t use mediapipe
huh, i cant get any of them to run. They actually completely crash out the comfyui process itself. crazy.
ok, i got it finally working, sort of. It doesnt actually crash everything anymore, but the video itself comes out the wrong resolution/size. Working on fixing it.
Hey, Im about 3 hours into trying to get this to work. Ive fixed a lot of the issues. However, it seems like this workflow is very incomplete? Even some of the nodes are missing input links. The input video and output videos are two totally different sizes, the masking output is rotated 90 degrees from the input video. Many, many issues. Ill go and watch your video again and try to compare it to the workflows and make more fixes. But right now this workflow is almost impossible to work with. Excited to get it working though.
@singularity42 Yeah, the video shows where to update the sizes. I probably should update it so that there is one size input.
Same. Anyone got the Mediapipe working?
Woah this looks incredible! Would be cool to have further automation in the face swap process (maybe without PuLID) as it has strict depedencies conflict.
Thanks for the video - I might recommend to add some notes at certain areas in the workflow to precise what you mention and also notes on the order of bypass.
Thanks for the share!
Yeah, there still isn't really a better option than PuLID that I've found. The only other thing is Gemini API, which is decent, but I hate having an API call in the workflow for some reason haha.
@theartofficialtrainer how about reactor face swap + face detector or flux inpaint with face detect for custom new face
@Catz reactor swap kind of defeats the purpose though since the objective is to get a better faceswap than reactor. Might as well just use reactor for the whole thing if we use it for the initial image. Never tried face detect though! That could be cool. Have a link?
@theartofficialtrainer It's basically the Face Detailer node with Ultralytics Adetailers. It'll automatically detect the face and let you generate on top. From Impact Pack nodes.
https://github.com/ltdrdata/ComfyUI-Impact-Pack
Posted an example in the gallery.
@Catz Oh yeah I know that one! That requires a Lora already generated though, right? There’s no way to use a reference unless maybe using redux or PuLID
@theartofficialtrainer Hmm you can generate it without any lora, just a checkpoint. You can use a Lora for enhancement sure, but you can use SDXL or Flux to quickly reconstruct the face. I think the Adetailer option would just be for creating a brand new face.
For using a reference image face, then yes I think something like PuLID would be great (I tried PuLID a year ago on a virgin comfyui, but never got it working with my main comfy). I forget if PuLID automatically face detect and swap it or if you need to precise the area.
@theartofficialtrainer @Catz If either of you can implement the option of using a face Lora into the workflow I would greatly appreciate it. I have trained a face lora in flux that I would need to have this for.
@pandasarethebest4910 just put it in the flux image generation part of the workflow! Just need to add “loadloramodelonly” node after load diffusion model
This workflow, along with VACE model, works GREAT! thanks!
When I move around within the workflow, it freezes. Is this only happening in my environment?
I think it has something to do with SAM2. Either try updating it or uninstalling if you want to see if that is what’s causing the issue.
@theartofficialtrainer I uninstalled it and still freezing
same for me
@theartofficialtrainer Also happening here. This doesn't happen with any other workflow I am using so there is something in your workflow that is causing freezing for a lot of users. Can you update the workflow omitting whatever is causing the freezing as it is unusable as it is right now.
@tcla75542 I don’t know what is causing the freezing, it doesnt freeze for me
same for me... also i'm trying another workflow Wan2.1-Fun Control Workflows Native+Wrapper + First Frame Control.. the same problem. it freezes when i'm trying to move around or scale it. Btw Remove ANYTHING with VACE+Wan2.1 workflow had no problem, it worked like a charm.
A fresh install was the solution, I don't know why. But now it's works
@killtheegg What did you fresh install?
For an updated version maybe add a bypassed lora loader since you said that works better, and note next to the loaders where it says step by step what to do. Other wise quite clean workflow
It makes really good results for texture compared to faceswap, but, could you say how much generation time /second of video, with a reference setup ?
I am on a 5090, but this took about 1.5mins per clip.
Thanks for the reply, so its 20x video length + limited to some seconds. Really good but not sutable (for now) as a reactor new challenger
Hmm.. any idea what could cause this.. when I try to generate the video.
ImagePadKJ
The expanded size of the tensor (160) must match the existing size (832) at non-singleton dimension 0. Target sizes: [160, 480, 3]. Tensor sizes: [832, 480, 3
One of your videos is 160x480 or it’s being resized to that. Everything should be 480x832 when passed through the wan sampler
I was able to fix this error. by changing the target width and target height to match the image. this allowed me get further, but then it died on the depth section.
odel_path is D:\apps\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts\LiheYoung\Depth-Anything\checkpoints\depth_anything_vitl14.pth
using MLP layer as FFN
input_masks shape torch.Size([210, 832, 480])
refs shape torch.Size([1, 3, 1, 832, 480])
!!! Exception during processing !!! Invalid WanVideo model selected
Ouch!
also, all the frames of the video are being dumped in my output folder.. why is it doing that?
Its not doing anything for me in v2