v4.2: Made my own Remb tool thats more like a quasi video editor. Added that swap out backgrounds. https://github.com/GeekyGhost/ComfyUI-GeekyRemB.git
v 2.0: My normal video maker daisy chain. Using Cascade, SVD, and AnimateDiff with Wav2Lip and IF Whisper to make Talking Avatars. You can use your preferred video method. This just gives me a bit cleaner video for my specific setup. Advanced and you'll need to hunt down models. Using the ComfyUI specific cascade models. Using a version of AnimateLCM for SVD. Using an LCM animateDiff model and an LCM SD 1.5 model. Wav2lip has a few models out there. Reactor can be an issue to install for some. IF Whisper seems to be giving an error for others (May be related to recent updates to both ComfyUI and other nodes so you may need to update both to newest versions).
V 1.0 Made some Updates to ComfyUI wav2lip extension. Have been testing it out. Just a fork, I'm not affiliated with the maintainer. You can sub in the official comfyUI extension. Mine just adds an intensity slider for the lipsync. 2.0 gives a more prominent lipsync, 1.0 is the default, 0 gives no lipsync. Combined it with some other tools. Adding in video next to create the animated avatar video.
For model info you can check my fork of the repo or the main repo. The only difference is the intensity scale I added. 1.5 is pretty choice. And the Smoothing slider. Everything in my repo is the same instruction wise besides these two additions. wav2lip.py and wav2lip_node.py
original repo for ComfyUI wav2lip - https://github.com/ShmuelRonen/ComfyUI_wav2lip.git
Description
FAQ
Comments (8)
any option to restore lips/face without faceswap?
yeah, there's a lot of face restore only nodes. Reactor itself even has one. They usually use codeformer and Gfpgan models.
@TheGeekyGhost i tried face restore with model, nut quality still not good. also face swap affected lip movements, so i stopped using it on lipsync videos.
@myprivacy27091991221 yeah, that's kind of why I added the intensity slider lol. Some videos would come out okay, others not so much. Now I can adjust the intensity to make it work and with the mask padding and gaussian blur addition to the mask it made things a lot cleaner in the end. I had to make a some modifications to wav2lip to get better output.
Added a video using just face restore without faceswap.
@TheGeekyGhost thank you
below error when using voice generaion
OSError: [WinError 1314] A required privilege is not held by the client: 'C:\\Users\\myprivacy\\.cache\\huggingface\\hub\\models--speechbrain--spkrec-ecapa-voxceleb\\snapshots\\eac27266f68caa806381260bd44ace38b136c76a\\hyperparams.yaml' -> '~\\.cache\\speechbrain\\hyperparams.yaml'
Someone else had this issue. This is related to IF Whisper I think. I haven't encountered but I did almost help someone with this issue but they fixed it first lol. Let me see what I can find.