I'm replacing the sampling setup, currently getting much nicer output with er_sde/beta combo. Even 4 steps, with the 4step turbo of course, looks much nicer than restart sampling. I'd suggest trying that, and only enabling a turbo, for low steps, otherwise no LoRA at all for high. The ref LoRA is really only good with the full bf16 model. VBVR LoRA is awesome but it can totally ruin video face swaps. Just leave everything off to start.
Add the attention node of your choice after the loaders, should be able to cut sampling time in half. At maximum resolution, 10 seconds should take around 2 minutes. Unless you're not me, in which case I have no idea how long it should take.
There are a few ways to run this:
1) Your faces + voice, with a t2v style prompt will give you a t2v style output with your face and voice.
2) Your face + voice, with a reference image and an i2v style prompt will give you an i2v output. Ish. Not quite the same, though you can force it pretty hard. Generally I prompt it as a suggestion (eg. this is a general composition guide to the overall structure of the first frame...etc.)
3) Your face + voice + a source video will give you a face swap. I cannot emphasize enough what a great job this model is doing with swaps. The examples I gave do not even come close. Give it a minimum of four HQ pictures of someone. Notes on recording a good voice clone sample are below and in the WF.
There are three ways to supply the reference video: you can either
1) pipe it straight in (I didn't make this switchable but you can figure that out)
2) have the SAM detector make a mask for face/hair/etc. that will be used to make a composite that inverts the detected area. This tends to help H3 find what you want. This is default, I think.
2) have the SAM detector make a mask for face/hair/etc. that will completely black out (or white out if you like) the detected area. This can sometime be helpful.
I think when I've settled on a model and sampler and have attention/optimization added I'll post a nicer, more polished version.
Here's whatever I wrote before, though all requisite notes and links are in the WF itself.:
I think I forgot to link the SAM model in the WF. Here it is: https://huggingface.co/Comfy-Org/sam3.1/tree/main/checkpoints. At some point when I wasn't looking, comfy changed things up a bit with regards to SAM, so this is a checkpoint, and needs to go into checkpoints folder (duh). In case you're used to putting SAMs into their own folder.
My frame count logic isn't quite lining up the way I thought it would, so be careful with that if it matters to you. You can always just unplug and enter a manual cap in the loader and elsewhere as needed.
This workflow does things to your face.
Show the model your face. Give it a nice long look at your face.
Then tell it where you want your face to be. Have you heard of 'text to video'? It's just like this, with one small difference. It uses your face.
Oh, your voice too. You must give it your voice. Or a voice. Something.
If you choose to do something like the 'image to video' stuff, supply a reference image. There is some crap in there that will mask out the face of the reference so the stupid idiot model doesn't get confused, the moron. Point is, it's the same as above, except you're helping it out a bit with the composition of the video, how it should look at the start, and all that jazz.
It's basically like instant character training. Just feed it good data.
I'm switching to 20+ step full model generation now that the swap is working, have to see if H3 can actually put out a good image... what a shame that would be, an amazing likeness but garbage video, that would be appropriately Sisyphean, that's how all these models seem to work.
V2 Info - It will post whenever CivitAI's terrible content filter decides to rate my example video. Of course it will be rated X. You know, bathing suits and all that.
-Ha, wtf did I tell you? Rated X - how predictable was that...
v2 has a straight video face swap module. It's messy, I think face swap should be its own thing.
Be that as it may, it's working. Video is still pretty crappy, but I've just been focused on getting the swap to work.
My example faces aren't super high resolution, working with HQ real photos gives MUCH better results, cannot post that here though.
Should be able to dial in some quality now that I can get a consistent result.
Also there is no upscaling or refining or anything yet. You have to be super careful with that anyway when you do swaps. It's VERY easy to ruin a likeness.
Also needs attention nodes and all that jazz.
Every section has copious notes, you should be able to figure it out. I believe in you.
For obvious reasons I cannot show you how insanely good a job this model has been doing with real faces. So for the examples I had Gemini supply me with a few angles of a unique face. Well, unique as defined by an idiot model. So if it's hard to tell if the likeness matches, it's because A) Civitina is not real and B) even if she was you don't really know her so how the hell would you know if the likeness matches? So you will just have to trust me. I've been swapping heads for years, before diffusion models, doing it with my bare hands damnit, then all the way back to A1111 with original ReActor, I've tried absolutely every swap technique, and all of them together, plus loads of manual compositing with tedious roto masking. I've never seen anything close to the likeness matching I'm getting with this model.
So just give it at least four HQ closeups of your face, you can add more and shift the comp ref down, adjust prompt, mutatis mutandis, all the rest, you know what to do. This isn't for babies. It's not a complicated setup at all. If you don't have custom nodes, swap them out for what you do have.
Notes on each section are in the WF.
<SUBJECT> speaks with the exact vocal timbre, pitch, and voice identity of <Audio 1>.
IF AND ONLY IF your sample is crappy/noisy, tack on cleanser promptage like this:
<SUBJECT> speaks with the exact vocal timbre, pitch, and voice identity of <Audio 1>, but with clean studio isolation and zero background noise.
This is the text I use for voice cloning. It is designed to capture all the phonemes you need. Have people read it with a natural cadence, no special effort, just read it as they would if they were alone. With this model you can try multiple readings, however, so you can try reading angry or sad, etc. Might work if you need it, but my single source audio works GREAT.
---------
When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors. These take the shape of a long round arch, with its path high above, and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look, but no one ever finds it. When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow.
---------
It really should be CLEAN. A good recording.
It should be stereo, 48khz. A normal reading should end up at around 30 seconds.
Description
Source video face swap now integrated.
Nowhere near presentable really, but whatever. Hopefully when optimizations and cleanup are added this will be faster, and put out a decent image. For all the amazing things this model does, it's a huge step backwards in several respects. But at least they all look like they can be overcome. I certainly hope so, because ref and FLF are just insane.
FAQ
Comments (8)
where to get Minimax_bst_v1 & VBVR_h3_attn_only loras?
https://civitai.red/models/1343431/bouncing-boobs-ltx-wan-h3 isn't necessary for the workflow. https://civitai.red/models/2497207/h3-ltx-23-i2v-t2v-video-reasoning-lora-vbvr does help.
FLF with H3 is also insane, for anyone who cares about FLF, all three of you. I'll post some slopflows of that as well.
I have been sleeping on it. I didn't know it was really that different. Would love to see your other workflows
@SomeRandomUser23 Everyone seems to love it. As a general all-purpose model it has some serious problems, imo. But it's got some magic in the 'I need this and exactly this' department, which is priceless when making compositions. LTX 2.3 is awesome, quality is great and you can spit out 20 -30 seconds of full HD in one stage, with perfect sync. But it's stiff as hell, you need lots of help with camera and subject motion. And it has garbage frames that need removal when doing FLF, which results in a mismatch, which requires dissolves/morphs, and also has contrast and color jumps. All of that is not an issue with continuation, but still you get degradation eventually, which requires FLF to get back to a HQ keyframe, so back to square one.
From experiments so far H3 seems to be the first model I've used that actually does what it says on the tin with regards to FLF. That last frame is THE frame you give it, not something close. And though it still requires precise prompting to avoid unwanted cuts, success rate is pretty damn high when you have well-planned keyframes with matching backgrounds.
So though it's still DiT it is quite different from LTX and extremely useful as a tool. But beware, these models are HUGE. And slow. Slow on a 5090. Well, slow is relative of course, still it's a beast. And I am finding now that the quants and prunes and all other fruity versions are indeed detrimental to precise output. Which is pretty obvious, as that's kind of the point. Bring lots of RAM. All flavors.
@SomeRandomUser23 FLF WF is scheduled, will post whenever the example video clears manual review. I forgot to include an unclad tot with a lollypop, which guarantees instant PG rating and featured status.
is the a way to do full body swap? also is the ref video masked? if yes what type of mask solid color or inverted? i have hard time replacing hair, model renders the same hair as ref video
Take a look at the WF I linked to in the description, I based the detection on that setup. While this is set up for faces specifically, that one is for the entire body (but can do just the face as well). To be precise about what 'mask' means here: to help the model understand what needs to be swapped, we make a copy of the ref video and invert the colors. At the same time we use the SAM module to detect the face/body (or whatever you prompt for). It makes a mask of this, but it is immediately used with a composite node (ref video + inverted ref video wherever mask is white). So you wind up with your ref video altered such that the part(s) you want to swap are inverted. There is no actual mask data after this point. However, I did set up a switch you can use to black out/white out the detected parts instead of inverting, which can be helpful in some instances. I believe the linked WF has something similar, disconnected.
The inversion addresses this exact problem that you are having - the ref is too strong. It's very clever and dead simple. And it works. But like I said, you can try solid b/w too. It's a model, every click is a roll of the dice.