Audio YuE2 — Music Generation Guide v2

Caption version: Audio_YuE2_Captions

SmartLLM or manual lyrics and style · optional ABC planning · full-song FLAC saving · excerpt review and trimming · animated lyric captions · MP4 export

What Is This Workflow?

This workflow generates music with YuE2, saves the complete song, and lets you review and export a selected excerpt. Use SmartLLM to prepare style and lyrics from a song concept, or enter both yourself.

The caption version adds Render Lyric Captions, vocal separation for alignment, four caption styles, and viewers for subtitles, timing corrections, and alignment diagnostics. Captions retain your supplied wording. The original song remains the soundtrack; the separated vocals are used only for analysis.

The three groups are Generate Audio (YuE2), Load and/or Trim Audio, and Save Video. A selected cover image forms the background. Captions can float over it or appear in a fixed position. Turning captions off uses the same image and trimmed audio for a plain MP4. The cover itself remains still.

The caption version also replaces the repeated image batch with a direct image fallback, shares one trim-start control between audio and captions, and connects the excerpt's actual duration to the renderer.

For a first test, use a short song concept and a 60-second generation cap. Review the audio and caption report before increasing the length.

Before You Run It

Required Custom Nodes

Install or update these packages and their requirements in your ComfyUI environment, then restart ComfyUI:

ComfyUI Eclipse — use a version containing Render Lyric Captions, Any Multi-Switch Mixed, and the Audio source and Stop (Result Review) controls on Load Audio.
ComfyUI Smart Model Loader — the YuE2 checkpoint loader and shared seed/sampling controls.
ComfyUI SmartLLM — Smart LM Loader and Split YuE2 for optional lyrics/style assistance.
Audio Separation Nodes — AudioSeparation, which supplies the isolated vocal track.

ComfyUI must include the native YuE2 Generate ABC, YuE2 Generate Music, and Empty YuE2 Latent Audio nodes. If they are missing, update ComfyUI. The complete saved graph still references SmartLLM and Audio Separation Nodes when their branches are unused.

Installing Python Dependencies

If ComfyUI-Manager is available, install or update the required node packages there. Manager handles their Python dependencies during installation. Restart ComfyUI after installation finishes and refresh the browser. See the official custom-node installation guide for the available installation methods.

For a manual installation, or missing dependencies after an update, use the commands below. Use the Python environment that actually runs ComfyUI. Installing into a separate system Python will not make the packages available to ComfyUI. The examples assume the node folders are named comfyui_eclipse and audio-separation-nodes-comfyui; adjust those names to match the folders inside your own custom_nodes directory.

Windows Portable

Open PowerShell or Command Prompt in the portable root folder containing both python_embeded and ComfyUI, then run:

.\python_embeded\python.exe -m pip install -r ".\ComfyUI\custom_nodes\comfyui_eclipse\requirements.txt"
.\python_embeded\python.exe -m pip install -r ".\ComfyUI\custom_nodes\audio-separation-nodes-comfyui\requirements.txt"

Virtual Environment / Manual ComfyUI Installation

Activate the existing virtual environment you use to start ComfyUI. Open a terminal in the ComfyUI folder containing custom_nodes, then run:

python -m pip install -r "custom_nodes/comfyui_eclipse/requirements.txt"
python -m pip install -r "custom_nodes/audio-separation-nodes-comfyui/requirements.txt"

For ComfyUI Desktop, use Manager or its built-in terminal so dependency installation targets the app's environment. The ComfyUI environment instructions cover Portable, Desktop, and custom environments.

Also complete the installation instructions for Smart Model Loader and SmartLLM if installing those packages manually. The two commands above cover Eclipse and the vocal-separation package. Eclipse's requirements include stable-ts (imported as stable_whisper), faster-whisper, fonttools, and Pillow. Use the requirements file so its supported version constraints are applied. Model weights and fonts are configured separately below.

Leave --no-deps off for normal installation. It skips dependencies of the listed packages, potentially leaving caption requirements such as CTranslate2 or PyAV missing. Use it only when those dependencies are already installed and managed separately. See pip's option reference.

To preview changes, add --dry-run to an installation command. --no-deps does not freeze packages listed directly in requirements.txt, including Torch. If your setup requires specific Torch/CUDA builds, use matching version constraints instead of relying on that flag.

Quick Dependency Check

Run these in the same environment:

python -c "import sys; print(sys.executable)"
python -c "import stable_whisper, faster_whisper, fontTools, torchaudio; from PIL import features; print('Caption imports OK; RAQM:', features.check('raqm'))"

For Windows Portable, replace python at the start of each check with .\python_embeded\python.exe and run from the portable root. The first command shows the Python being checked. The second should report Caption imports OK; RAQM: True. It checks the Python imports and text-shaping support; model files still need the setup below.

If RAQM is False, follow Pillow's installation instructions for a build with RAQM support and its required FriBiDi runtime, then repeat the check. Restart the ComfyUI process after changing dependencies; refreshing the browser alone does not reload Python packages.

Music Models

Download the saved checkpoint from the Comfy-Org YuE2 bundle:

yue2_3b_int8_convrot.safetensors → ComfyUI/models/checkpoints/.

In Smart Model Loader, select Standard Checkpoint, this YuE2 file, and Baked for both CLIP and VAE.

Auto Prompt also needs a configured text-capable model in Smart LM Loader. The saved selection is huihui_ai-Qwen3.8-27B-abliterated-Ollama, using the YuE2 Music task. Select a model available in your own SmartLLM setup, or turn Auto Prompt off to use manual inputs.

Caption Models and Fonts

Install Eclipse's requirements for alignment and font support. Download these five files from the supported Systran faster-whisper-large-v3 revision into ComfyUI/models/whisper/large-v3/:

model.bin
config.json
tokenizer.json
vocabulary.json
preprocessor_config.json

The caption node verifies this exact model revision and does not download it automatically. The saved device is auto; explicit cpu and cuda choices are also available.

AudioSeparation uses torchaudio's Hybrid Demucs model. Its first use may download the checkpoint into the torch cache. The saved separation settings are linear fades, 10-second chunks, and 0.1 overlap.

The selected font is Quicksand-Bold.ttf, loaded from ComfyUI/models/fonts/. Eclipse supplies bundled defaults when that folder is empty. Choose a font containing the characters used by your lyrics; other scripts may need another font. Pillow must support RAQM text shaping.

Quick Start

Select the YuE2 checkpoint in Smart Model Loader and a locally available text model in Smart LM Loader — YuE2.
With Auto Prompt (YuE2) on, edit the LM loader's user_prompt: describe genre, mood, instruments, vocal character, language, and song idea. For manual text, turn Auto Prompt off and fill Style and Lyrics.
Leave Enable ABC planning on and both native generators in full mode. Set Duration (Seconds) to your generation cap; the saved value is 360 seconds (6 minutes). Current ComfyUI supports a cap up to 900 seconds (15 minutes), subject to the limits explained below.
Select your own cover in Load Image (Metadata Pipe). Set its dimensions in Image Resolution Simple [Pipe]; the saved target is 720 × 1280.
Set Shared trim start (seconds) and the duration on Load Master Audio. The saved excerpt starts at 0 and lasts up to 240 seconds. Duration 0 keeps everything from the selected start to the end.
Leave Stop (Result Review) on and queue. The full generated song is saved before Load Master Audio stops execution for excerpt review.
Audition and adjust the excerpt. Queue unchanged again to continue from the reviewed selection, or turn the review stop off and queue. Keep prompts and seeds unchanged to reuse available cached results.
Leave Render Lyrics on, select the sung language and font, and choose a caption mode. The saved mode is Floating words with English (en).
Review the MP4 and the Report, srt, timing_json, and cleaned-lyrics viewers. Unresolved lyrics need review or timing corrections; they are not guaranteed to appear automatically.

For a plain cover-and-audio video, follow Turning Captions Off below. For audio-only output, mute the Save Video group, including the caption renderer.

Prompts and ABC Planning

With Auto Prompt enabled, Smart LM Loader → Split YuE2 supplies style and lyrics to both native YuE2 generators. The selected lyrics also feed the caption renderer. The manual text boxes are ignored in this mode.

For manual mode, turn off the Fast Mode Toggle connected to Auto Prompt (YuE2). Enter musical direction in Style, for example:

English alternative rock, expressive male baritone, 110 BPM, crunchy guitars,
warm bass and live drums. Restrained verses, a hopeful anthemic chorus,
clear lead vocals and a short instrumental ending.

Enter the sung words in Lyrics, including repeated choruses in full. Section labels such as [Verse] and [Chorus] are supported; recognized section labels are removed for captions. Keep production directions in the style text. Review the cleaned lyrics before correcting timings. For instrumental output, turn captions off.

With Enable ABC planning on, the native YuE2 Generate ABC node prepares a score from the selected style and lyrics. It passes through ABC score preview into YuE2 Generate Music. SmartLLM prepares the text; the native planner generates the ABC notation. Turning planning off supplies an empty score for direct generation. Both saved generators use full mode; if you experiment with melody, change both mode controls together.

Length, Sampling, and Saved Settings

These values describe the saved caption version:

SettingSaved valueAuto Prompt / ABC planning / Render LyricsOnGeneration cap360 seconds (6 minutes), as saved in this workflowSampling32 steps, CFG 1.0, dpm_2 / sgm_uniform, denoise 1.0Audio VAERegular decode; switch false selects regular, true selects tiledShared trim start / excerpt duration0 / 240 secondsLoad Master Audio source / review stopAuto / onCover and caption size720 × 1280ResizeCustom, crop, center, lanczos, divisible by 2, CPUCaption mode / language / deviceFloating words / en / autoFont / sizeQuicksand-Bold.ttf / 80 pxCaption transparency / glow3% / enabledVideoMP4/H.264, 24 FPS, CRF 19, preset veryfastSave Video trim modevideo_to_audio for the IMAGE path

Change the shared seed and sampling settings in Smart Model Loader. Connected values override the local KSampler and native generator widgets, including their displayed 222 seeds. SmartLLM has its own generation controls. The floating-caption seed controls text placement, separately from music seeds.

Generation Cap: 360 Seconds Saved, Up to 900 Seconds Available

The current native YuE2 Generate Music node accepts max_duration up to 900 seconds (15 minutes) and defaults to 360 seconds (6 minutes). The workflow retains that default. See the native ComfyUI node definition.

To allow a longer song, increase Duration (Seconds). Its connection supplies max_duration to YuE2 Generate Music and overrides that node's local widget. Older ComfyUI versions limited this setting to 360 seconds; update ComfyUI if needed. The increase is documented in ComfyUI change #16292.

900 seconds is the selectable generation ceiling, not a guaranteed song length. Style, lyrics, and ABC notation share the model's context with music tokens. ComfyUI reduces the available generation budget when the prompt needs more space, and generation can finish naturally before reaching the cap.

The actual generated duration drives the audio latent. The full-song save occurs before trimming. To export the whole result, set Shared trim start to 0 and Load Master Audio → duration to 0; leaving duration at 240 still exports only that excerpt, even when the generated song is longer.

The generation cap and excerpt duration are separate. Trimming to 30 seconds does not reduce the work needed to generate a longer source song; lower Duration (Seconds) for a shorter generation.

Review, Trim, and Save

The Set/Get connections carry these paths across the canvas:

YuE2 → Save full song → master audio selector
                         ├─ Load Master Audio → Save trimmed FLAC → Save Video: audio
                         ├─ Render Lyric Captions: audio
                         └─ AudioSeparation: Vocals → Render Lyric Captions: vocals

Shared trim start → Load Master Audio: start_time + Render Lyric Captions: trim_start
Load Master Audio: duration → Render Lyric Captions: duration
Selected lyrics → Render Lyric Captions: lyrics

Load Image → Image Resize ┬─ Render Lyric Captions: background
                         └─ Any Multi-Switch Mixed: any_2
Render Lyric Captions: video → Any Multi-Switch Mixed: any_1
Any Multi-Switch Mixed → Save Video: images / video

Edit Shared trim start (seconds) to move the excerpt. Its cable overrides the local start_time widget on Load Master Audio and supplies the same offset to the renderer. Set excerpt length on Load Master Audio. Duration 0 means to the end; an oversized excerpt stops at the available audio's end. Its duration output tells the renderer the actual selected length.

Alignment always uses the full song. Rendering trims the original soundtrack and captions together. SRT timestamps start from the exported clip, while timing JSON retains full-song timestamps, including lyrics outside the clip.

With Stop (Result Review) enabled, Load Master Audio publishes its preview and stops downstream execution. Editing the selection updates the player without queueing; downstream audio changes on the next run. An unchanged repeat queue continues from the reviewed excerpt. Changed input needs review again.

Cover Size and Video Saving

Set width and height in Image Resolution Simple [Pipe]. Context Image sends those values to both Image Resize and the caption renderer, overriding their local width/height widgets. The saved divisible_by = 2 preserves the exact 720 × 1280 target. Center crop fills the frame; choose pad to retain the whole cover with borders. The resize node's size = 1024 is unused in custom mode.

The captioned VIDEO carries its original, trimmed soundtrack, frame rate, and duration into Save Video. Save Video's separate AUDIO and FPS controls apply to the IMAGE fallback. Keep its FPS and the renderer's FPS equal if you want both versions to use the same frame rate.

The fallback holds one image for the complete trimmed audio duration. No Loop Calculator or Repeat Image Batch is needed. Captioned video is passed as a file and saved without expanding the complete clip into image tensors. Keep the VIDEO connection through the mixed switch to Save Video for long songs.

Turning Captions Off

Any Multi-Switch Mixed selects the first available input. Captioned VIDEO is connected first and the resized IMAGE second. Save Video accepts either type.

Turn Render Lyrics off to save the cover with the trimmed soundtrack. Turn it on again for captions.

Setting caption transparency to 100% only hides the text; it still runs the caption branch. Use the toggle when you want to skip that work.

Using an Existing Audio File

For a plain cover-and-audio video, mute Generate Audio (YuE2), turn captions off and select or upload a file in Load Master Audio. Choose Audio source: Selected file to select it explicitly. Auto prefers usable incoming audio and otherwise falls back to the file.

For captions on an existing file, feed that file into the full-song path as well. Selecting a file only in Load Master Audio changes the excerpt branch; the renderer and AudioSeparation still receive audio from the upstream selector. Add these connections before running:

Add a separate Load Audio for the complete file, with start_time = 0, duration = 0, and Audio source: Selected file.
Connect its AUDIO to the spare second input of the audio RFFN / Any Multi-Switch immediately before Load Master Audio.
Mute the generated-audio branch, including its full-song Save Audio node. Keep the manual Lyrics node and its routing nodes active.
Turn Auto Prompt off and supply lyrics matching the file. Keep Load Master Audio on Auto or Incoming audio, then use the shared trim controls.

This supplies the same complete song to alignment, separation, and excerpt selection. Leave any separate review stop on the new loader off if you only want the existing Load Master Audio review step.

Using the Lyrics Renderer

Render Lyric Captions matches the supplied lyrics to the full song, then renders your selected excerpt. Use the first run to review the automatic timing. You can then adjust the appearance or supply edited timing JSON for subsequent renders of the same song.

Caption Styles and Appearance

ModeWhat appearsWhole line (whole-line)The complete line in a fixed position.Active word (active-word)The complete line with timed words highlighted.Floating words (floating-words)Individual words or short phrases with gentle movement and fades.Floating lines (floating-lines)Complete lines with floating movement and fades.

The saved appearance uses white text, yellow active-word highlights, a 3 px #616161 outline, and 40 px margins. Fixed modes use bottom-center positioning. The selected image supplies the background; background_color is used when no image or video is connected.

Caption transparency applies to text, outlines, highlights, and glow: 0% is fully visible and 100% is hidden. The saved 3% makes the effect slightly transparent. Glow is enabled with intensity 5, range 25 px, blur 15, inner color #2ec0ff, and outer color #006eff. Reduce font size, glow, or margins if the complete effect cannot fit.

The saved floating controls are a 30% circle radius, 1.5% drift, 0.5-second fade-in/out, 0.9-second preferred display time, up to 4 words per automatic phrase, 5 simultaneous items, and seed 42. Fast vocals shorten display/fade times. Untimed words stay attached to nearby timed text; a phrase-only match can appear as one complete item even when longer than the word limit.

Floating placement is scheduled over the full song before trimming, so an excerpt keeps the same positions and motion as that interval of the full video. Positive timing_adjustment delays captions without shifting the soundtrack. Changing appearance, background, or trim can reuse the bounded alignment cache in the current ComfyUI session.

Practical Tips

Check timing in Whole line first. It is easier to hear whether a line starts and ends correctly before adding floating motion. Use Active word to inspect individual word timings, then choose your final display mode.
Preview a short excerpt while adjusting the look. Change the shared trim start and Load Master Audio duration to focus on a verse or chorus. This reduces rendering work; automatic alignment still analyzes the full song.
Use timing_adjustment for a consistent offset. For example, 0.20 delays all captions by 0.2 seconds; -0.20 brings them forward. Correct individual lines in timing JSON when only part of the song is mistimed.
Keep floating text readable. Try a smaller font or fewer simultaneous items for dense lyrics. Shorter fades reduce lingering overlap. The position control applies to fixed modes; floating placement uses the circle controls.
Balance the effect against the cover. Outline and glow help separate text from a busy image. Reduce their size if long phrases become cramped, and check the result at the size viewers will actually watch.
Save a timing copy for each song. Once reviewed, reuse that JSON while experimenting with fonts, colors, glow, caption modes, and trim settings. Keep the source audio and lyrics unchanged so the timings still match.

Review the Alignment

The saved language is English (en). Choose another supported language code for other songs, or use Auto and inspect the language confidence and uncertainty in the report. Auto does not translate the lyrics.

Isolated vocals can help detection, but separation does not guarantee better matching. To compare against the original mix, disconnect the renderer's vocals input. Keep its main audio input connected to the complete song.

Inspect Report for unresolved lines, partial word timings, match scores, and analysis source. Text-match coverage describes how much supplied text was matched; it is not a timing-accuracy score. Changed arrangements, repeated choruses, sustained notes, and recognition errors can leave gaps.

Accepted lines keep the supplied wording even when some word timings are missing. Only supported words receive active-word highlighting. Completely unresolved lines are omitted from rendering and SRT until corrected; the node does not invent timestamps for them.

Correcting Timing Without Running Alignment Again

The timing_json output is also an editable input for the next render:

First run: Render Lyric Captions → timing_json viewer
                                  ↓ manually copy, paste, and edit
Next run:  String Multiline → Render Lyric Captions: corrected_timing

Generate the initial timing. Leave corrected_timing disconnected and run the workflow with your song and lyrics. Review the video and Report.
Copy the complete timing_json result. Copy the JSON from its viewer, from the opening { through the closing }. Use the timing JSON, rather than the SRT or Report output.
Paste it into String Multiline. Use the node titled Connect to corrected_timing after adjusting. Replace its entire saved example with your current result; the saved example belongs to a previous song. Paste plain JSON without Markdown code fences.
Adjust the timings while listening. Edit the affected line's start and end, and its word timings when needed. All times are seconds from the beginning of the full song. For example, a lyric 2.5 seconds into a clip trimmed from 60 seconds has a JSON start of 62.5.
Connect the multiline text output to the renderer's corrected_timing input. Keep the full-song audio, lyrics, and trim connections in place.
Adjust the renderer and queue again. Choose your caption mode, font, colors, glow, or excerpt settings. The renderer uses the edited JSON and renders a new video without repeating automatic alignment. If Load Master Audio pauses for review, review the excerpt and queue unchanged again.
Review and refine. Edit the same multiline text and rerun as needed. Keep a saved copy of your corrections. Disconnect corrected_timing when you want a fresh automatic alignment.

What to Edit in the JSON

Keep version: 1, the full-song duration, and the exported lines array. Preserve line order, exact lyric text, blank lines, and repeated lines so they match cleaned_lyrics. Its saved viewer is titled cleared_lyrics.

For each corrected line, start must be earlier than end. Timed lines must remain ordered, non-overlapping, and within the song. If you retain the words array, keep word timings ordered inside that line's interval. Preserve each word's text, char_start, and char_end; those offsets identify characters in the original line, not timestamps. Moving a line's boundaries does not automatically move its word timings.

For a simple line-level repair, set the line's start/end and replace its words with words: []. This displays the complete phrase without individual word highlighting. For example, one entry inside lines could look like this (illustrative text and times; use your own exported lyric text):

{
  "text": "Hello world!",
  "start": 62.5,
  "end": 65.0,
  "words": []
}

Keep the rest of the exported JSON around that entry. A missing line can be repaired this way after listening to establish its actual timing. Leave unsupported lines or words with start: null and end: null; blank lines keep empty text and null timing.

Connected corrections bypass recognition and model verification. They also skip vocal separation when no other executing branch needs it. Rendering still runs with your selected visual settings. This does not prevent upstream music generation from rerunning: keep its prompts and seeds fixed, or use the saved-audio route above. A different song or changed lyrics need matching timing data. The global timing_adjustment still applies to corrected timings, so return it to 0 if you have already incorporated that offset into the JSON.

Output Files

The saved prefixes produce numbered files under your ComfyUI output folder:

Full generated song: audio/Moonlight_*.flac.
Selected excerpt: audio/Moonlight_Trimmed_*.flac.
Captioned or plain cover video: video/YuE2_Captions_*.mp4.

Change these prefixes for your own project. The full-song save precedes trimming, so excerpt changes do not shorten it. The added existing-file route does not run the muted generation branch's full-song save.

srt, report, timing_json, and cleaned_lyrics are displayed as text. The saved graph does not write separate SRT or JSON files; copy and save those outputs if you need them. The text viewers have their review stops off and are not a review barrier before Save Video.

Troubleshooting

Missing nodes or ModuleNotFoundError: check the ComfyUI startup log for the failing package, then run that package's requirements installation using ComfyUI's Python. For stable_whisper, use Eclipse's requirements: its package name is stable-ts. Restart ComfyUI afterward.
requirements.txt cannot be found: check the terminal's current folder and the actual node-folder names. Portable commands run from the folder containing python_embeded; virtual-environment examples run from the ComfyUI folder containing custom_nodes.
Pip says packages are installed but ComfyUI still cannot import them: run the Python-path and import checks under Installing Python Dependencies with the same interpreter used to launch ComfyUI.
Manual lyrics have no effect: turn Auto Prompt (YuE2) off.
Turning Render Lyrics off produces a type error: set its Fast Mode Toggle to Mode: Mute, then turn it off again.
A selected file plays, but captions use another song: route the file into the full-song selector as described under Using an Existing Audio File. Changing Load Master Audio alone only changes its excerpt branch.
The excerpt or MP4 is not saved: after reviewing, queue unchanged again or turn Stop (Result Review) off. Select your own cover image as well.
The caption model is missing or fails verification: check all five files from the supported revision in models/whisper/large-v3/.
Text fails to render or characters are missing: choose a font covering the lyrics' script and run the RAQM check above. A False result means Pillow's text-shaping support needs fixing; changing fonts alone will not enable it.
Some lyrics are missing or mistimed: inspect the report, language, and supplied lyrics. Review against the full song and use corrected timing where needed. Singing alignment remains approximate.
Video export uses too much memory: keep VIDEO connected through the mixed switch to Save Video. Avoid converting a long clip into an IMAGE batch.
The song is shorter than requested: the generation duration is a cap. The native node allows up to 900 seconds, but generation can end earlier and long style/lyrics/ABC inputs can reduce the available budget. Check the full-song FLAC separately from the excerpt; the saved excerpt is 240 seconds.
Generation runs out of memory: lower the generation cap. If decoding fails, try the tiled VAE branch.
Older canvas notes show different node numbers, H3, or 75.5 seconds: those notes are inherited. The current caption graph uses a shared trim start, a 240-second excerpt, and the mixed IMAGE/VIDEO save path above.

Start with a short generation, review the song and caption timings, then increase the duration once the style and lyrics work for your project.

Files with SHA256: eaa0b505621a48fa00c280fc7c8adfb611a39d75abfdd34d21fd54352d7fccb0