Warning
If this workflow works for you and you update to latest comfy ui it will remove the IC lora node... I hate when they do this crap.
I will try to put another version out without it soon... ffs.
Every single time i do a LTX workflow they do this... they put nodes out, they take them away... they pissed me off last time doing the same shit.
You can replace it with a normal addguide node now with the model with the id lora connected to it via the ic lora parameters node
LTX 2.3
REUPLOADED to include the silence MP3 needed for padding.
Simple LTX 2.3 video to video Frame ref workflow
This is a cut down version of my first flow
Removed
Audio voice ref
Multi frame inserts
Main uses
- Copy video movements of a source video onto an image with lipsync
Img2vid
This is a cut down version of my first flow
Removed
Audio voice ref
Video to video
Those two i think were most peoples hangups with the first version.
Main uses
- Lip sync music
- Create scenes with one or multiple speakers.
- Interpolate first/last frames
3 presets to show you how to use it.
Comment any questions.
V1.1
No changes to the main flow
Added 3 example flows with example files and instructions
2 frame interpolation example
Img2vid with motion transfer example
remake video adding sync or sound example
All have their own json flow with everything set up for the task and instructions.
All have the needed files, video ref, image refs, audio files, etx.
This should show you how to use it for the most part.
1-3 stages
1. Basic image to video
2. First/last frame interpolation
3. Multi frame interpolation
4. Video motion transfer with guide frames
5. Video insertion as a mid video ref.
6. Video reproduction adding sound and sync
Audio
1. Custom voice ref to guide the generated voice
2. Custom audio can be synced
All options are optional, if no video or image ref is used it does regular text to video.
All options can be used together when needed.
Included the images i used for interpolation for you to test with. She is an ai girl.
Notes everywhere, read. Enjoy :)
Description
No changes to the main flow
Added 3 example flows with example files and instructions
2 frame interpolation example
Img2vid with motion transfer example
remake video adding sync or sound example
All have their own json flow with everything set up for the task and instructions.
All have the needed files, video ref, image refs, audio files, etx.
This should show you how to use it for the most part.
FAQ
Comments (19)
thanks alot
That looks very promising. Thank you! Cannot find hqtalk lora anywhere. Is it original file name in the workflow?
Sorry :)
hq i speak of is
https://huggingface.co/AviadDahan/LTX-2.3-ID-LoRA-CelebVHQ-3K
The other choice is this
https://huggingface.co/AviadDahan/LTX-2.3-ID-LoRA-TalkVid-3K
@sy0ww4bb1984 thank you so much. Whatever I try, I cannot make the sound work, though. It's just high pitch noise, if i remove the previewer from SamplerCustomAdvanced, Video Combine fails with error, that audio frames are not provided.
@comthumb The audio is probably the most confusing thing about this flow tbh. I gave a few different ways to use audio and did not properly explain what each did and how to combine or not combine them. I still have much to do with this flow to make it easier, i just threw it out here because the video copy was actually cool and i did not see others doing it.
I will try to explain the what happens to the audio
First with no options enabled (audio ref bypassed and custom audio bypassed) it uses an empty audio latent by default allowing normal LTX audio gen.
When you enable the audio ref group and give it a ref voice, using the HQtalk lora it will attempt to change the voice of the generation to the one provided.
When you enable custom audio this removes all need for LTX to do audio. It takes the audio you give it and uses it to form the video using its tone, speed, etc to drive motion and sync.
You cant use custom audio and audio ref together. They do different things.
Now when using custom audio you have a mask, this is set at 0 to not change the audio. Custom audio group has this mask, as well as 2 small groups between the samplers to stop/start editing the audio.
EDIT - also when using the audio ref and HQ lora you need a specific prompt style i think i included and example
What are you trying to accomplish? I could probably explain better if i knew what you were trying to do. Easier than just telling you every possibility,
@sy0ww4bb1984 Thank you very much for the explanation about audio. I think there was compatibility issues with comfyui, as LTX's own sample workflows did not produce correct audio for me, but latest update fixed the problem. So far your workflow is the best I've tried. Great job.
@comthumb Thx for the comment tbh. None of my workflows are currently counting or showing any stats so i didnt really know if anyone is using any of them. I cant see likes, downloads or anything for over a week now.
Ok... trying to run it, got it in Compfy but have some errors...
Missing Models - Latent_upscale_models
ltx-2.3-spacial-upscaler... I downloaded that from huggingface, but what folder does it go in as the workflow can't find it.
Hi there,
The latent upscale models are here with the main LTX models
https://huggingface.co/Lightricks/LTX-2.3/tree/main
I use the v1.1 of the 2x upscale. There is a 1.0 its not as good.
https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-spatial-upscaler-x2-1.1.safetensors
These go in the folder inside models called latent_upscale_models
ComfyUI\models\latent_upscale_models
Ok... I am crawling forward... Got that taken care of but now my error is
ModuleNotFoundError: No module named 'sageattention'
When I go to comfy manager, I can find a comfyui-sageattention3 is that what I should use instead?
Also, is it ok to use gemma312BAbliterated_v10aExperime in lieu of the normal gemma?
Its running now... it was the patch block leading into the sage attention.... I took that out and it seems to be working. 5060 with 16gb ram is chugging along
@pjwhoopie Great :) Sage attention is 100% optional, it can speed up generation by about 10-15%, but also can make the output slightly worse. Some models use this better than others. Wan vace hates sage but ltx seems to work ok with it.
I also run a 5060 16gb. Its plenty for LTX and many others. even my old 3060 12gb worked fine. Its slower than high end cards but everything should work.
It works. Yes, better to try once—to understand.
Seems its not basic wf, not for novice.
I make workflows that work, but this is not as complex as many others i have done. It looks big and intimidating but its really just simple image to video, or frame inserts for first/mid/last frames. The complex part can be when using the video motion transfers. But even then i give an example and its not needed for most things.
I will put out another flow that is a bit less complex and smaller to allow people to learn to use it easier in a few days here. I find it hard to see the complexity in workflows when i make them, as i know how it all works and assume others would as well when they look through it. I know, i know... many cant follow my stuff. Its why i'm here, trying to show people.
It's actually very easy to follow.
It is very messy, dirty, and not simple at all. Above all, I have to reinstall comyfui due to a node crash error; it is a very messy workflow.
:P. I appreciate the comment and am sorry you needed to reinstall comfy... i have done this many times because comfy... is NOT comfy to use and updates and nodes break things all the time. This workflow works fine. No workflows i make are simple. That was an inside joke because those that know me... know it dont do simple. I release what i use, all the options, all the settings i use to make my videos. Its up to the person downloading it to understand what i am doing. Most people want a big green button that makes full length videos with no input. Making unique quality videos usually takes more than that when done locally. I spend weeks making my flows work for what i use them for and i still use them today with many more options i add as i need them.
This flow is large, but its not hard to follow if you are adept at comfy. I have done many many many flows that are... just silly in their scope. The problem with this flow as it is... is most likely that it can do multiple things with different options enabled and people who download this do not understand the options.
That being said, i have been working on making simple separated flows for different single uses. This would allow people to follow easier. This has all uses in one because thats how i work, i enable what i need when i need it but many struggle to follow my train of thought unless you are used to my flows as they all tend to follow the same lines.
Its odd when people look at my videos (stuff many cant even do locally to this day) and expect me to be able to lay it all out in a simple way. They assume its all the AI doing the work. Its not... its me... i do the work... I just share the workflow i use to do the work.
thanks for saving me the headache!