YuE2 Kit: generating openings and nesting workflows 🎼

The YuE2 Kit got two cool features that sound more complicated than they are. Generating openings and nesting workflows. This article tries to demystify them.

Update through ComfyUI-Manager to YuE2-Kit 0.8.0, ABC Preview 0.3.5+ and (optionally) Audio Visualizer 0.5.0+. Both templates are in the template browser under ComfyUI-YuE2-Kit.

Quick refresher: score first, then music

YuE2 Kit works in two steps. First it writes a score: sheet music in ABC notation. Then it renders that score into audio.

Because the score is just text, you can see it, play it back, and reuse parts of it. That's what makes both features possible.

Generating openings

Continuing a song is easy for YuE2: it writes music left to right, so you give it some bars and it carries on.

That means generation score before the bars you already have as reference is a new trick. The model can't "look ahead" to your chorus while writing an intro. So if you love a hook and want a proper build-up in front of it, you're stuck.

The main realization

More technically the model is writing text tokens, and it's autoregressive (AR), so everything that came before is context for what is generated after it. Nothing stops you from giving the model 0-bars of music and instead only write the ABC header.

X:1
T:
M:3/4
L:1/16
Q:1/4=86
V: Vocal clef=treble name="Vocal Melody" snm="Vocal"
V: Ins clef=treble name="Ins Melody" snm="Inst."
K:Bm
% intro

This is ABC for 3/4 meter, 86 BPM, B minor. Then % intro starts a new intro section. Your Lyrics + Style guidance and all the other parameters like temperature guide this. And the model is told, go generate the rest of the ABC from here...

That knowledge is enough to implement the "locked tempo/key/meter" functionality. The custom node just writes this header for you and continues from there. So what if we did this in several phases?

How YuE2 kit solves it as of v0.7.0+

Generate opening works in a multi-pass approach:

1. Write several opening scores. The planner writes the beginning of a song with your style and lyrics, several times with different seeds (Opening candidates, default 4).

2. Cut and check. Each candidate is cut to the number of bars you want (Opening bars, default 8). Candidates that don't fit your reference are thrown out: wrong meter, a different unit note length, or a broken last bar.

3. Rank and join. The rest are ranked by warnings (like a key that doesn't match) and by how well the last chord leads into your reference first chord. The best one gets selected.

4. Keep generating. If you use Generate continuation the model is given the combination (header, the chosen opening, your reference ABC) and continues generating score from there.

The Opening section label (default intro) tells the model what kind of section to write. You could open with a chorus and then start your intro 😜, have fun with this!

The Score report tells you which candidate won and why. If none fit, it lists each seed and what went wrong.

Just remember that the model still doesn't "look ahead" to your score that's coming up. It's the ranking of step 3 using heuristics that I came up with in Python, just to take the tedium out of this fun technique.

How to use it

1. In YuE2 Advanced Score, turn on Use reference.

2. Paste or drop ABC, or drop audio/video to transcribe it with SheetSage2. Saved songs work too via Song history → Use as reference.

3. On the reference timeline, select the bars you want to keep.

4. Turn on Generate opening. Optionally also Generate continuation to write what comes after.

5. Run.

The top timeline shows the overall plan: the generated opening, your kept bars, then the continuation.

Tips

• Want to change the tempo of a reference? locked score properties keep its meter, key and unit note length but let you adjust the tempo.

• generated score properties with a reference require an opening: the opening is what picks the new properties.

• No good candidates? Raise Opening candidates, or try a different seed.

• It's recommended to keep the BPM in your style prompt equal to the locked tempo.

Nesting workflows, introduced in v0.8.0

On Civitai you can't actually post a FLAC file as-is. I've included a visualizer feature and that will embed your workflow in the video.

But when you want to create actual music videos, like with H3 or LTX2, the details about your song tend to get lost. The video would just have a "load audio: SONG_1097.FLAC" in the workflow and it tells you nothing about how you generated the YuE2 song.

That's what I wanted to fix, letting others see how I generate the songs to a video.

Every song carries its own recipe

Each FLAC the kit saves embeds the workflow that made it, including the final score. Drop a saved FLAC into ComfyUI and you get the workflow back, score and all.

And retrieving your ABC no longer requires generating with the same seed or an .abc file next to it. Just extract it from the workflow that's inside.

Embed YuE2 Kit song

The new Embed YuE2 Kit song node takes things one step further. Pick a saved FLAC (or upload one) and the node copies the song's prompt, workflow and score into itself.

So any workflow you build on top of a song carries the song's recipe inside it. Render a music video from it, and the video's embedded workflow still knows the lyrics, style, seed and score the song was made with. Even if the FLAC is gone later.

Its outputs also give you the details, so you can plug it into your workflow:

• audio, abc, abc_seconds, bars

• tempo, meter, key

• lyrics, style, instrumental, seed

• title, artist (from the save node's optional FLAC player info)

Try it: Embedded song demo

Open YuE2 Kit - Embedded song demo. Or download the song from this screenshot.

Even if you don't have my FLAC file, the complete song workflow with exact seed/settings/loras/generated score should be all there. (If you don't have the custom node installed, you'll see the raw ABC and JSON of the nested workflow).

The demo workflow loads a saved song and adds a visualizer video, with the score in ABC Preview and the timeline. Use it as a starting point: swap the visualizer for your own video workflow, or add in an EQ like I did, and use the outputs you need to time scenes to the music or add lyrics in the subtitles.

Feedback

Questions, bugs, or a song with a great generated intro? Leave a comment! And post your tracks, they may end up in the Music Focus 🎸 collection.

New: Music Resources 🎼
⇒ Posts • Articles • Models and Tools