<h2 id="trained-adapter-to-use-llm-as-text-encoder-for-rouwei-0.8-(and-other-sdxl-models).">Trained adapter to use LLM as text encoder for Rouwei 0.8 (and other sdxl models).</h2><p></p><h2 id="update-v0.2:">Update v0.2:</h2><p>New version for t5gemma-2b text encoder model with improvement performance.</p><p>To run you need <a target="_blank" rel="ugc" href="https://huggingface.co/google/t5gemma-2b-2b-ul2">t5gemma-2b</a> encoder model (<a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/RouWei-Gemma/tree/main/t5gemma-2b-2b-ul2_encoder_only">ungated mirror</a>, downloading instructions below).</p><p></p><h3 id="you-need-an-updated-set-of-custom-nodes-to-make-it-work">You need an updated set of <a target="_blank" rel="ugc" href="https://github.com/NeuroSenko/ComfyUI_LLM_SDXL_Adapter">custom nodes</a> to make it work</h3><h3 id="detailed-launch-instructions-and-prompting-tips-below">Detailed launch instructions and prompting tips below</h3><p></p><h2 id="what-is-it:">What is it:</h2><p>A drop-in replacement for Clip text encoders in SDXL models that allows to achieve better prompt adherence and understanding.</p><p>Kind of same as ELLA, SDXL-T5 and likely others, but this one is focused on anime models and advanced knowledge without censorship.</p><p></p><h2 id="key-features:">Key features:</h2><ul><li><p>State of the art prompt adherence and NL prompt understanding among SDXL anime models</p></li><li><p>Support of both long and short prompts, no 75 tokens limit per chunk</p></li><li><p>Preserves original knowledge of styles and characters while allowing amazing flexibility in prompting</p></li><li><p>Support of structured prompts that allows to describe individual features for characters, parts, elements, etc.</p></li><li><p>Maintains perfect compatibility with booru tags (alone or combined with NL), allowing easy and convenient prompting</p></li></ul><p></p><h1 id="how-to-run-latest-version:">How to run latest version:</h1><p>1. Install/update <a target="_blank" rel="ugc" href="https://github.com/NeuroSenko/ComfyUI_LLM_SDXL_Adapter">custom nodes</a> for Comfy</p><ul><li><p>Option a: Go t<code> ComfyUI/custom_nodes</code> and type<code>git clone https://github.com/NeuroSenko/ComfyUI_LLM_SDXL_Adapter</code></p></li><li><p>Option b: Open <a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/Rouwei-T5Gemma-adapter_v0.2/resolve/main/examples/sfw/8.png">example workflow</a>, go to ComfyUI Manager and press <code>Install Missing Custom Nodes</code> button.</p></li></ul><p>2. Make sure you have latest Transformers: Activate ComfyUI venv, type <code>pip install transformers -U</code></p><p>3. Download adapter and put it into <code>/models/llm_adapters</code></p><p>4. Download T5Gemma</p><ul><li><p>Option a: After activating ConfyUI venv type <code>hf download Minthy/RouWei-Gemma --include "t5gemma-2b-2b-ul2_*" --local-dir "./models/LLM"</code> (correct path if needed).</p></li><li><p><s>Option b: Download safetensors file and put in into ComfyUI/models/text_encoders (implemented within next nodes update)</s></p></li></ul><p>5. Download <a target="_blank" rel="ugc" href="https://civarchive.com/models/950531">Rouwei</a> (vpred or epsilon or <a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/RouWei-0.8/blob/main/rouwei_080_base_fp16.safetensors">base</a>) checkpoint if you don't have one yet</p><p>6. Use any image from showcase as a reference workflow, feel free to experiment</p><p></p><h3 id="instructions-for-previous-versions-based-on-gemma-3-1b-llm-model-can-be-found-in-this-hf-repo.">Instructions for previous versions based on gemma-3-1b llm model can be found in this <a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/RouWei-Gemma">HF repo</a>.</h3><p></p><h2 id="current-performance:">Current performance:</h2><p>This version stands above any clip text encoders from various models in terms of prompt understanding. It allows to specify more details and individual parts for each character/object that will work more or less consistent instead of pure randomness, make a simple comic (stability varies), define positions and more complex composition.</p><p>However it is still in early stage, there can be difficulties with rare things (especially artist styles) and some biases. And it works with quite old and small UNET than needs a proper training (and possibly modification), don't expect it to perform as top tier open source image generation models like Flux and QwenImage.</p><p></p><h2 id="usage-and-prompting-with-examples:">Usage and Prompting with examples:</h2><p>The model is quite versatile and can accept various formats, including multilangual inputs or even base64.</p><p>But it is better to stick to one of several prompting styles:</p><p>(Examples in showcase or in <a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/Rouwei-T5Gemma-adapter_v0.2">HF repo readme</a>)</p><h3 id="natural-language">Natural language</h3><pre><code>kikyou (blue archive) a cat girl with black hair and two cat tails in side-tie bikini swimsuit is standing on all fours balancing on top of swim ring. She is scared with tail raised and afraid of water around.</code></pre><p></p><p>Just a plain text. It is better to avoid very short and very long prompts.</p><h3 id="booru-tags">Booru tags</h3><p>Regular booru tags.</p><p>Until emphasis support will be added to nodes, <u>avoid adding of </u><code>\</code><u> before brackets</u>. Also unlike with clip misspelling will likely lead to wrong results.</p><h3 id="combination-of-tags-and-nl:">Combination of tags and NL:</h3><pre><code>masterpiece, best quality, by muk (monsieur).
1girl, kokona (blue archive), grey hair, animal ears, brown eyes, smile, wariza,
holding a yellow ball that resembles crying emoji</code></pre><p>Most easy and convenient approach for most cases.</p><h3 id="structured-prompting:">Structured prompting:</h3><p></p><pre><code>bold line, masterpiece, classroom.
## Asuka:
ouryuu Asuka Langley in school uniform with tired expression sitting at a school desk, head tilt.
## Zero two:
Zero two (darling in the franxx) in red bodysuit is standing behind and making her a shoulder massage.</code></pre><p>It can understand Markdown <code>#</code> for seperating), json, xlm or simple seperation with new lines and <code>:</code>. Prompt structuring allows to improve results when prompting several characters with individual features. Depending on specific case, it can work very stable, work in most cases above random level, or can require some rolls bo allowing to achieve things impossible otherwise due to biases or complexness.</p><h3 id="all-together:">All together:</h3><p>Any combinations of above. Recommended for most complex cases.</p><h3 id="quality-tags:">Quality tags:</h3><p><code>masterpiece</code> or <code>best quality</code> for positive.</p><p><code>worst quality</code> or <code>low qualit</code> for negative.</p><p>It is better to avoid spamming because it can cause unwanted biases.</p><p></p><p><strong>Current state of custom nodes does not support prompt weights and standard spells. Also (brackets) should be left as is, no need to add </strong><code>\</code><strong>.</strong></p><p>Other settings and recommendations are same as for original RouWei</p><p></p><h2 id="knowledge-and-train-dataset:">Knowledge and Train Dataset:</h2><p>Training dataset utilises about 2.7M of pictures from <a target="_blank" rel="ugc" href="https://huggingface.co/datasets/Minthy/Anime-Art-Multicaptions-v5.0">this dataset</a> and few other sources. Still quite a small number.</p><p></p><h2 id="training-and-code">Training and code</h2><p></p><p><a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/Rouwei-T5Gemma-adapter_v0.2/blob/main/code/llm_to_sdxl_adapter.py">Forward code</a> example, obtaining <a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/Rouwei-T5Gemma-adapter_v0.2/blob/main/code/t5gemma_states.py">hidden states from t5gemma</a> example.</p><p></p><p>Sd-scripts fork for <a rel="ugc" href="https://github.com/lRemixl/sd-scripts-T5">LORA training</a>.</p><p></p><p>Sd-scripts (dev branch) <a rel="ugc" href="https://github.com/MinthyBasis/sd-scripts_t5gemma_adapter">fork for Full Training</a>, supports fine-tuning for each part (t5gemma, adapter, unet).</p><p></p><h2 id="compatibility:">Compatibility:</h2><p></p><p>Designed to work with Rouwei, works with most of Illustrious-based checkpoints including NoobAi and popular merges. Unet parts of loras works, TE parts need to be retrained.</p><p></p><h2 id="near-future-plans:">Near future plans:</h2><p></p><ul><li><p>Custom nodes improvement including emphasis</p></li></ul><p>There will be another version trained on larger dataset to estimate capacity and decision about joint training with encoder or leaving it untouched.</p><p>If no flaws are found, then it will be used as text encoder for large training of next version of Rouwei checkpoint.</p><p></p><h3 id="i'm-willing-to-helpcooperate:">I'm willing to help/cooperate:</h3><p>Join <a target="_blank" rel="ugc" href="https://discord.gg/ZXHENAhqE9">Discord server</a> where you can share you thoughts, give proposal, request, etc. Write me directly here, or dm in discord.</p><p></p><h2 id="thanks:">Thanks:</h2><p>Part of training was performed using google TPU and sponsored by <strong>OpenRoot-Compute</strong></p><p>Personal: NeuroSenko (code), Rimuru (idea, discussions), Lord (testing), DraconicDragon (fixes, testing), Remix (nodes code)</p><h3 id="also-many-thanks-to-those-who-supported-me-before:">Also many thanks to those who supported me before:</h3><p>A number of anonymous persons, Bakariso, dga, Fi., ello, K., LOL2024, NeuroSenko, OpenRoot-Compute, rred, Soviet Cat, Sv1., T., TekeshiX</p><p></p><h2 id="donations:">Donations:</h2><p>BTC: bc1qwv83ggq8rvv07uk6dv4njs0j3yygj3aax4wg6c</p><p>ETH/USDT(e): 0x04C8a749F49aE8a56CB84cF0C99CD9E92eDB17db</p><p>XMR: 47F7JAyKP8tMBtzwxpoZsUVB8wzg2VrbtDKBice9FAS1FikbHEXXPof4PAb42CQ5ch8p8Hs4RvJuzPHDtaVSdQzD6ZbA5TZ</p><p></p><p></p><h2 id="license">License</h2><p></p><p>MIT license for adapter models.</p><p></p><p>This tool uses original or finetuned models google/t5gemma-2b-2b-ul2 and google/gemma-3-1b-it.</p><p>Gemma is provided under and subject to the Gemma Terms of Use found at [ai.google.dev/gemma/terms](ai.google.dev/gemma/terms).</p><p></p><p></p>

Rouwei-Gemma

Files with SHA256: c67ba5fb932887b9f0dd5178474c97707f7a04f4e14c52b681517a5c0fcd263c