Just some experimental lip syncing tool i made a while ago. Requires a 16 or 32 fps video of a character and a voice audio input. The output is the same video but with the added lip motion of the character synced to the audio. Works best for simple anime style, but can also work great for detailed realistic videos (check the examples in different WF versions above).The workflow allows to insert the inpainted lip motion within any part of the video with smooth in-and-out transitions. It uses SAM3 segmentation to precisely target the face and mouth of the character to resample them at low denoise using Wan2.2 S2V.Detailed description of how to use the workflow can be found within the notes inside!<h3 id="v2-changes:">V2 CHANGES:</h3><ul><li>Split the final stage into two parts for easier mouth movement rerolling.</li><li>Fixed the ability to choose speaking character (for videos with multiple faces detected) and added new controls.</li><li>Removed slow color correct node in the beginning and swapped video input from VHS node to native loader because it doesn't introduce color drift when decoding video frames into Comfy's operational format.</li><li>Improved compatibility: reduced number of custom node packs used to bare essentials (swapped the problematic sound info node to a different pack; workflow should now be compatible with transformers 5.x)</li><li>Added an option to combine input speech audio with background sounds/music before the final render.</li><li>Slightly improved memory management by swapping interpolation to native nodes (requires new models to be downloaded into ComfyUI/models/frame_interpolation/). Didn't do the same for SAM3 segmentation tho because comfyui-easy-sam3 custom node pack is way superior to native implementation. Overall, workflow still remains very memory-heavy, but you don't need insane RAM capacity because comfy can also offload to paging file on your disc. So if you are inpainting long videos make sure you have like &gt;100 GB of paging file disc space.</li><li>Other minor fixes and tweaks to some nodes.</li></ul><h3 id="important:">IMPORTANT:</h3>All the custom nodes used in this workflow can be installed through the Manager. I tried to reduce the amount of custom packs to only those that are already widely used by everyone. I recommend installing them one by one in case some of them change the versions of your existing python dependencies. As always, don't be lazy and check the requirements.txt for each custom node pack before you install. Otherwise you risk getting conflicts with your other installed node packs or even BRICKING your ComfyUI installation entirely! So fat WARNING right here.<h3 id="required-resources:">Required resources:</h3><ul><li>For the Wan2.2 S2V itself, i personally use DaSiWa's finetuned checkpoint because it works well with anime styles and has lightning lora embedded. You can check it out here: <a target="_blank" rel="ugc" href="https://civarchive.com/models/2151205/dasiwa-wan-22-14b-s2v">https://civarchive.com/models/2151205/dasiwa-wan-22-14b-s2v</a> The normal Wan2.2 S2V would probably work just as fine with some appropriate adjustments to KSamplers.</li><li>As for the VAE, text and audio encoders, i used the models provided by the official ComfyUI documentation. They can be found here: <a target="_blank" rel="ugc" href="https://docs.comfy.org/tutorials/video/wan/wan2-2-s2v">https://docs.comfy.org/tutorials/video/wan/wan2-2-s2v</a></li><li>The key part of this workflow is automatic segmentation which is done using SAM3 model. I used the custom node pack "ComfyUI-Easy-Sam3". From my testing, this node pack provides WAY better controllability and segmentation then the native implementation of SAM3.1. Find the download links and were to put the model (official fp32 release and pruned fp16, which is just as good and more memory-efficient) in the node pack documentation here: <a target="_blank" rel="ugc" href="https://github.com/yolain/ComfyUI-Easy-Sam3#model-downloads">https://github.com/yolain/ComfyUI-Easy-Sam3#model-downloads</a></li><li>To use "32 fps input" mode you would need models for native frame interpolation node, which can be found in the official repo:<a target="_blank" rel="ugc" href="https://huggingface.co/Comfy-Org/frame_interpolation">https://huggingface.co/Comfy-Org/frame_interpolation</a>Placed in ComfyUI/models/frame_interpolation/.</li><li>If you can't find the Audio Info node pack in the manager, here is the manual link: <a target="_blank" rel="ugc" href="https://github.com/afloy011-spec/afloy_audio_tools">https://github.com/afloy011-spec/afloy_audio_tools</a> Great pack for precise audio trimming that has no additional requirements.</li></ul><h3 id="system-requirments:">System requirments:</h3>I tried to add some memory-clearing nodes after each step to make the workflow more memory friendly but those can be kinda buggy. Due to Comfy's smart native offloading 12GB VRAM should probably be enough to run this, however the workflow still remains very RAM-heavy (excess memory is offloaded into the paging file on your disc if RAM is full). For my specs (RTX4080 with 16GB VRAM and 64 GB of RAM) the total run time is around 3 min and RAM/VRAM peak usage is at maximum. My honest bet would be that if you can fit normal Wan2.2 S2V, than you can handle this workflow also. If not, then try swapping the model loaders for GGUFs idk.<h3 id="known-issues:">Known issues:</h3><ul><li>If the input video has intense movements, the output might have the mouth of the character slightly swaying relative to their face.</li><li>Crossfading stage for first and last frames can produce buggy mouth morphs. To avoid this, try using the input with closed mouth or change the number of frames to crossfade.</li><li>Segmentation can produce jittering in the 1st stage face track video. Small shaking is fine, the model and the smart crop-and-stich will handle it. But sometimes SAM3 tries to segment ears/neck differently for each frame which leads to major jittering. This can be somewhat negated by tweaking segmentation, masking and cropping parameters (check notes inside the workflow for more detailed guide).</li><li>Mouth movement sometimes doesn't match the speech. Try different controls or generation seed (again, check notes inside the workflow for guidance).</li></ul><h3 id="example-videos-above:">Example videos above:</h3><ul><li>All of the example video were generated using taek75799's Enhanced WAN 2.2, the FASTMOVE V2 FP8 version specifically. Check out this awesome model here: <a target="_blank" rel="ugc" href="https://civarchive.com/models/2053259/wan-22-enhanced-nsfw-or-svi-or-camera-prompt-adherence-lightning-edition-i2v-and-t2v-fp8-gguf?modelVersionId=2477539">https://civarchive.com/models/2053259/wan-22-enhanced-nsfw-or-svi-or-camera-prompt-adherence-lightning-edition-i2v-and-t2v-fp8-gguf?modelVersionId=2477539</a></li><li>The voice lines for characters were generated using Qwen3-TTS 1.7B (V1 showcase; requires transformers 4.x) and OmniVoice (V2 showcase; requires transformers 5.x). They can be easily run in ComfyUI via the following node packs (unfortunately, incompatible with each other): <a target="_blank" rel="ugc" href="https://github.com/flybirdxx/ComfyUI-Qwen-TTS">https://github.com/flybirdxx/ComfyUI-Qwen-TTS</a> <a target="_blank" rel="ugc" href="https://github.com/Saganaki22/ComfyUI-OmniVoice-TTS">https://github.com/Saganaki22/ComfyUI-OmniVoice-TTS</a></li><li>For background sound design i use MMAudio or Woosh-V2A (a better finetune).</li></ul>

LipSync any video to voice with SAM3+WanS2V inpainting workflow!

• Split the final stage into two parts for easier mouth movement rerolling.• Fixed the ability to choose speaking character (for videos with multiple faces detected) and added new controls.• Removed slow color correct node in the beginning and swapped video input VHS node to native loader because it doesn't introduce color drift when decoding video frames into Comfy's operational format.• Improved compatibility: reduced number of custom node packs used to bare essentials (swapped the problematic sound info node to a different pack; workflow should now be compatible with transformers 5.x)• Added an option to combine input speech audio with background sounds/music before the final render.• Slightly improved memory management by swapping interpolation to native nodes (requires new models to be downloaded into ComfyUI/models/frame_interpolation/). Didn't do the same for SAM3 segmentation tho because comfyui-easy-sam3 custom node pack is way superior to native implementation. Overall, workflow still remains very memory-heavy, but you don't need insane RAM capacity because comfy can also offload to paging file on your disc. So if you are inpainting long videos make sure you have like &gt;100 GB of paging file disc space.• Other minor fixes and tweaks to some nodes.

Files with SHA256: af3ff69c6241e93d3655e48bf793f85eb4f6cc15bbf1526736b768a56e5345a1