<p>This is a test LoRA model based on LTX-2, trained on 44 videos (most with audio) using LTX Trainer. The Lora still has some problems, but here I'd like to share some experiences from the training process.</p><p>LTX-2 has significant limitations with Japanese language support, which appears to be a base model capability issue—most Japanese kanji characters cannot be pronounced correctly. However, there are some impressive aspects: <strong>lip-sync and audio cloning show great promise</strong>. The model fits quickly and shows results rapidly, but I believe more diverse training data (video dynamics, audio variations, etc.) is needed for better outcomes.</p><h2 id="dataset-caption">Dataset Caption</h2><p><strong>I highly recommend using Gemini for dataset captioning</strong>. Gemini can directly process both video and audio, delivering excellent results. While ComfyUI workflows with other caption nodes + Whisper audio recognition + refine are possible, they're not particularly recommended.</p><h3 id="agent-prompt">Agent Prompt</h3><p>I used the following prompt for video analysis:</p><pre><code>You are a professional video analysis assistant responsible for generating descriptive text for video training datasets. Please analyze this video and output the description in the following format:

【Output Format】
Visual content: [Detailed description of visual content]
Speech transcription: [Transcribe dialogue content, omit this line if none]
Sounds: [Describe sounds and music, omit this line if none]
On-screen text: [List on-screen text, omit this line if none]

【Rules】
1. Use plain text format, no Markdown bold or other formatting
2. "Visual content:" is required; omit entire lines (not "None") if other sections are empty
3. Descriptions should be natural and fluid, suitable for video generation model training
4. Focus on: actions, scenes, camera movement, lighting, atmosphere, emotions
5. Use present continuous tense for actions (e.g., is walking, is standing)

【Visual Content Description Points】
- Subject's actions and behaviors
- Scene environment and background details
- Camera movements (push, pull, pan, tilt, close-up, wide-angle, etc.)
- Lighting and color tone (warm light, cool light, backlight, soft light, etc.)
- Overall atmosphere and mood (serene, tense, warm, etc.)
- Composition and visual elements
</code></pre><h3 id="refine-prompt">Refine Prompt</h3><p>After generation, use this prompt to convert structured descriptions into natural paragraphs:</p><pre><code>You are a video description expert. Now you need to rewrite structured video descriptions into completely natural paragraph form.

【Task】
Rewrite the segmented video description into a smooth, natural paragraph, as if someone is describing what they see in the video.

【Rewriting Requirements】
1. Completely remove all labels (don't write "Visual content:", "Speech:", "Sounds:", etc.)
2. Naturally merge all information (visuals + dialogue + sounds into one fluid paragraph)
3. Like storytelling (use third-person perspective, present tense, vivid and specific with visual appeal)
4. Maintain technical details (camera movement, lighting, atmosphere descriptions should be preserved but naturally integrated)
</code></pre><h2 id="preprocessing-configuration">Preprocessing Configuration</h2><p>To prevent VRAM overflow, I used multiple bucket settings:</p><ul><li><p><strong>Maximum resolution</strong>: 768×?, 25 frames</p></li><li><p><strong>Minimum resolution</strong>: 256×?, 121 frames</p></li><li><p><strong>Unified frame rate</strong>: 24fps</p></li><li><p><strong>Frame count requirement</strong>: Must satisfy <code>frame_count % 8 == 1</code> (e.g., 1, 9, 17, 25...)</p></li></ul><h2 id="training-configuration">Training Configuration</h2><p>Using RTX 4090 (48GB VRAM), no fp8/quanto quantization needed, training config was barely modified.</p><p><strong>Training Progress</strong>:</p><ul><li><p>Sampling every 400 steps</p></li><li><p>First sample (400 steps) already captured basic character features</p></li><li><p>At 800 steps, ComfyUI testing showed incomplete character feature retention</p></li><li><p>Total training: 3400 steps (approximately 8 hours)</p></li><li><p><strong>Final weight used: 2200 steps</strong> (1400 steps might also work well)</p></li></ul><h2 id="testing-findings">Testing Findings</h2><h3 id="prompt-impact">Prompt Impact</h3><p>Prompts significantly affect results:</p><ul><li><p>Without background description, likely generates simple solid color backgrounds</p></li><li><p>Need detailed descriptions of character movements (hair flowing, etc.)</p></li><li><p>Overall motion tends to be minimal</p></li></ul><h3 id="workflow-configuration">Workflow Configuration</h3><p>Using official T2V workflow, found second-pass sampling has significant impact:</p><ul><li><p><strong>First pass (720p)</strong>: More aligned with LoRA characteristics but slightly blurry</p></li><li><p><strong>Second pass (spatial upscale)</strong>: Clearer, but audio quality slightly degraded</p></li><li><p>Without LoRA in second pass, noticeable differences appear—more 2.5D style, most obvious in lip-sync details (presence/absence of teeth)</p></li></ul><h3 id="interesting-discovery">Interesting Discovery</h3><p>Using cinematic-style descriptions produces interesting effects.</p><h2 id="future-plans">Future Plans</h2><p>Planning to test training with larger-scale datasets.</p><hr /><p><strong>Reference Documentation</strong>: <a target="_blank" rel="ugc" href="https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-trainer/docs/dataset-preparation.md">LTX-2 Official Training Documentation</a></p>

Files with SHA256: 4810a51c5f1afc0a5563fef6fb49f97139080ef0a768d62a54b74a27eb27bf98