What does this workflow do?
This will take your input image, crop/resize if needed to the ideal Cosmos render size, then automatically create an appropriate prompt for Cosmos to work its magic! The result will be a hopefully-amazing video that your family can cherish for generations.
This process is dependent on both Florence (for automatically describing the image) and an LLM (for creating a video prompt from the image description).
Further instructions and links are included in the workflow.
Extremely simple operation after initial setup (model load/LLM configuration):
1. Load an input image.
2. Queue prompt. Really, that's it. Every other setting should be good to go.
Have fun, and I look forward to seeing your creations!
Expectations setting: This model is HEAVY. Using the 7B model, with the included (optional) optimizations, I am running at about 15 minutes and 20GB VRAM usage on a 4090 to generate a 121 frame video at 1280x704.
Description
Exciting enhancement! I've developed a custom node https://github.com/EnragedAntelope/ComfyUI-ConstrainResolution and included it in this workflow with Cosmos-specific constraints.
This will automatically make the Cosmos output as close as mathematically possible to your input image's Aspect Ratio, resulting in less loss and greater fidelity to your source.
FAQ
Comments (4)
I adjusted it for my own setup using Ollama and it works really well. It's not getting enough love IMHO. Probably because the gooners can't really use it. ;)
19 march Workflow that worked few weeks ago is not working anymore. Used ollama LLM on local machine. Some parameters getting mixed up in Advanced Prompt Enhancer. I guess new version of it screw everything up. Help!
Thanks for the heads-up. Plush Nodes (which includes the Advanced Prompt Enhancer) has updated, which shifted all of the values that I had pre-selected in that node. To get running quickly, you should be able to right-click the node and choose "recreate", then select a good model (such as Llama 3 70B) from Groq.
I will also try to add a new version of the workflow but it may be a bit before I can do that.
Hi, I just published an updated version with the latest APE.