Description:
You can create a video of any length by specifying the "Length" and "Loop Count" according to the original video.
No adjustments are necessary except for the parameters in "Main Configuration" including the above.
For making this, Thanks to @BenjisAIPlayground(https://www.youtube.com/@BenjisAIPlayground) for this great opportunity.
Features (for those who want to modify it):
The first reference image (ref image) is best to use Flux Kontext or similar to match the starting frame and composition of the video. However, Wan2.1 VACE can be used without problems in my opinion because of its good tracking.
The chunk size (specified by Length) makes a difference in the number of Input and Output frames (and also in the first loop and the next and subsequent loops),I have corrected the number of frames in Output to always be "Length-1".
I found since the color of the reference image (ref image) saturates with each loop, so color matching is performed with the most recent reference image just before loop input.
Output video is assumed to be 30 fps; adjust "nth" if you are processing input video at 60 fps, for example.
Release Note:
(v3.0 -> v4.0)
Improvement (?):
When using Reference Video background/objects (using CN:DWPose), a new WanVaceToVideo for CN was set up so that CN control lines do not appear in the final output.
(Not sure if the quality is noticeably better than Ver 3.0. Waiting for feedback)
(v2.0 -> v3.0)
Improvement:
Improved output by injecting a CN (DWPose) element when using a video background (using SAM Mask).
when using a reference image background (using ControlNet), CN can be set more precisely; Depth is recommended.
In relation to above, the “Main Configuration” setting can be adjusted, including the case where the background is specified in the prompt (see Note in “How to Select#”).
Added prompt creation support function (using Ollama).
(v1.0 -> v2.0)
Improvement:
Background control is now possible.
Processing time could be reduced by selecting either image (ControlNet) or video (Masking).
Bug fix:
now includes sound in the output video.
Description
FAQ
Comments (7)
I tried loading this workflow and I have no missing dependencies, but I get a bunch of kjnode setnode routing errors when loading this workflow. Are you using an out of date version of kjnodes?
And the workflow fails validation:
873 is def invalid; BOTH origin node 293 doesn't have 873 and 293 target node doesn't have 873. Deleting link #873. splicing 65 from links Made no node link patches, and 1 stale link removals.
I noticed after loading there is no connections in a bunch of places: vae decode not connected to vae in anyway, get_audio not connected to the video combine, etc.
I removed get_audio because of an error. Also, the lack of connection, such as VAE, is due to antenna-based nodes such as Anything Everywhere, so I did not get an error because of this in my environment.
I will try to improve it here as well, but since you give me so much input, I think I can manage on my own. Thanks for the feedback, it's much appreciated.
@Usako_USA I also looked around because I remember having issues with this myself in the past, I actually had this node disabled, but use-everywhere and kjnodes get/set don't really work well with each other.
https://github.com/chrisgoringe/cg-use-everywhere/issues/213
You might want to switch to just using kjnodes get/set as that is always up to date and doesn't have the issues that use everywhere has been having lately with comfyUI updates to its node structure.
@psspsspsspssspss I have looked into it since then too, and the warning related to KJNode was because I was connected to the For (Start) node. I tried unplugging it and the warning disappeared. I will leave it as it is since it is the main process but it is working fine. Thanks for the advice.
This is awesome! However, after the first video is output, and the next round of frames are control-netted' I get this error:
vace_frames = torch.stack(vace_frames_out, dim=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: stack expects each tensor to be equal size, but got [1, 96, 27, 104, 64] at entry 0 and [1, 96, 26, 104, 64] at entry 1
And the for loop end node is highlighted.
I asked chatgpt and it fixed the issue with below, not sure if there's anything you can fix in the workflow:
Change model_base.py around line 1166:
vace_frames = torch.stack(vace_frames_out, dim=1)
Replace it with:
min_len = min(t.shape[2] for t in vace_frames_out) vace_frames = torch.stack([t[:, :, :min_len] for t in vace_frames_out], dim=1)
I'm not sure if it's exactly the same thing, but I've found these types of errors resolved with
1. making your number divisible by 4 for w/h and length or (if that doesn't work...)
2. making your w/h for your output image same aspect ratio
It can take a bit of playing with w/h/l numbers