New version is out: <a target="_blank" rel="ugc" href="https://civarchive.com/models/628865/sotediffusion-v2">https://civarchive.com/models/628865/sotediffusion-v2</a>Anime finetune of Würstchen V3.This release is sponsored by <a target="_blank" rel="ugc" href="https://fal.ai/grants?rel=sote-diffusion">fal.ai/grants</a>Trained on 6M images for 3 epochs using 8x A100 80G GPUs.This model can be used via API with <a target="_blank" rel="ugc" href="http://Fal.AI">Fal.AI</a>For more details: <a target="_blank" rel="ugc" href="https://fal.ai/models/fal-ai/stable-cascade/sote-diffusion">https://fal.ai/models/fal-ai/stable-cascade/sote-diffusion</a> Please refer to Huggingface for SD.Next UI, Diffusers or UNet models: <a target="_blank" rel="ugc" href="https://huggingface.co/Disty0/sotediffusion-wuerstchen3">https://huggingface.co/Disty0/sotediffusion-wuerstchen3</a> CivitAI page has only the ComfyUI checkpoint models. <h3 id="inference-parameters:-h0i2a8fq2">Inference Parameters:</h3>Download the Main model (8.14 GB file):<a target="_blank" rel="ugc" href="https://civarchive.com/api/download/models/563950?type=Model&amp;format=SafeTensor&amp;size=pruned&amp;fp=fp16">https://civarchive.com/api/download/models/563950?type=Model&amp;format=SafeTensor&amp;size=pruned&amp;fp=fp16</a> Download the Decoder model (4.24 GB file):<a target="_blank" rel="ugc" href="https://civarchive.com/api/download/models/563892?type=Model&amp;format=SafeTensor&amp;size=pruned&amp;fp=fp16">https://civarchive.com/api/download/models/563892?type=Model&amp;format=SafeTensor&amp;size=pruned&amp;fp=fp16</a>Positives:<pre><code>newest, extremely aesthetic, best quality,</code></pre>Negatives:<pre><code>very displeasing, worst quality, monochrome, realistic, oldest, loli,</code></pre><h3 id="main:-e3o1xo1nd">Main:</h3>Sampler: DDPM or DPMPP 2M with SGM Uniform CFG: 7 Steps: 30 or 40<h3 id="decoder:-s5ikrpa4j">Decoder:</h3>Sampler: Euler a Karras CFG: 1 or 1.2 Steps: 10Compression: 42 (or 32 to 64)Resolution: 1024x1536, 2048x1152.Anything works as long as it's a multiply of 128.<h1 id="training:-pf661l1ds">Training:</h1>Software used: Kohya SD-Scripts with Stable Cascade branch. <a target="_blank" rel="ugc" href="https://github.com/kohya-ss/sd-scripts/tree/stable-cascade">https://github.com/kohya-ss/sd-scripts/tree/stable-cascade</a>GPU used: 8x Nvidia A100 80GB GPU hours: 220<h2 id="base-gvk49eumc">Base</h2>parameters | value<ul><li>amp | bf16</li><li>weights | fp32</li><li>save weights | fp16</li><li>resolution | 1024x1024</li><li>effective batch size | 128</li><li>unet learning rate | 1e-5</li><li>te learning rate | 4e-6</li><li>optimizer | Adafactor</li><li>images | 6M</li><li>epochs | 3</li></ul><h2 id="final-75m2n5zdv">Final</h2>parameters | value<ul><li>amp | bf16</li><li>weights | fp32</li><li>save weights | fp16</li><li>resolution | 1024x1024</li><li>effective batch size | 128</li><li>unet learning rate | 4e-6</li><li>te learning rate | none</li><li>optimizer | Adafactor</li><li>images | 120K</li><li>epochs | 16</li></ul><h1 id="dataset:-cxh4lqwko">Dataset:</h1>GPU used for captioning: 1x Intel ARC A770 16GB GPU hours: 350Model used for captioning: SmilingWolf/wd-swinv2-tagger-v3Model used for text: llava-hf/llava-1.5-7b-hfCommand:<pre><code>python /mnt/DataSSD/AI/Apps/kohya_ss/sd-scripts/finetune/tag_images_by_wd14_tagger.py --model_dir "/mnt/DataSSD/AI/models/wd14_tagger_model" --repo_id "SmilingWolf/wd-swinv2-tagger-v3" --recursive --remove_underscore --use_rating_tags --character_tags_first --character_tag_expand --append_tags --onnx --caption_separator ", " --general_threshold 0.35 --character_threshold 0.50 --batch_size 4 --caption_extension ".txt" ./</code></pre> dataset name | total images<ul><li>newest : 1.85M</li><li>recent : 1.38M</li><li>mid : 993K</li><li>early : 566K</li><li>oldest : 160K</li><li>pixiv : 344K</li><li>visual novel cg : 231K</li><li>anime wallpaper : 105K </li><li>Total: 5.628.499 images</li></ul>Note:<ul><li>Smallest size is 1280x600 / 768.000 pixels</li><li>Deduped based on image similarity using czkawka-cli</li><li>Around 120K very high quality images got intentionally duplicated 5 times, making the total image count 6.2M</li></ul><h1 id="tags:-jzbj7tx2d"> Tags:</h1><h3 id="tag-format:-b15c2cmis">Tag Format:</h3>Model is trained with random tag order but this is the order in the dataset if you are interested:<pre><code>aesthetic tags, quality tags, date tags, custom tags, rating tags, character, series, rest of the tags</code></pre><h3 id="date:-o7yxmif0j">Date:</h3><ul><li>newest : 2022 to 2024</li><li>recent : 2019 to 2021</li><li>mid : 2015 to 2018</li><li>early : 2011 to 2014</li><li>oldest : 2005 to 2010</li></ul><h3 id="aesthetic-tags:-2zs1t8sgj">Aesthetic Tags:</h3>Model used: shadowlilac/aesthetic-shadow-2<ul><li>score &gt; 0.90 : extremely aesthetic</li><li>score &gt; 0.80 : very aesthetic</li><li>score &gt; 0.70 : aesthetic</li><li>score &gt; 0.50 : slightly aesthetic</li><li>score &gt; 0.40 : not displeasing</li><li>score &gt; 0.30 : not aesthetic</li><li>score &gt; 0.25 : slightly displeasing</li><li>score &gt; 0.10 : displeasing</li><li>rest of them : very displeasing</li></ul><h3 id="quality-tags:-6p4f4nab5">Quality Tags:</h3>Model used: <a target="_blank" rel="ugc" href="https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/models/aes-B32-v0.pth">https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/models/aes-B32-v0.pth</a><ul><li>score &gt; 0.980 : best quality</li><li>score &gt; 0.900 : high quality</li><li>score &gt; 0.750 : great quality</li><li>score &gt; 0.500 : medium quality</li><li>score &gt; 0.250 : normal quality</li><li>score &gt; 0.125 : bad quality</li><li>score &gt; 0.025 : low quality</li><li>rest of them : worst quality</li></ul><h3 id="rating-tags:-i4ucnneao">Rating Tags:</h3><ul><li>general</li><li>sensitive</li><li>nsfw</li><li>explicit nsfw</li></ul><h3 id="custom-tags:-xdvkprq26">Custom Tags:</h3><ul><li>image boards: date,</li><li>text: The text says "text",</li><li>characters: character, series</li><li>pixiv: art by Display_Name,</li><li>visual novel cg: Full_VN_Name (short_3_letter_name), visual novel cg,</li><li>anime wallpaper: date, anime wallpaper,</li></ul><h2 id="license-iqyvf9v5p">License</h2>SoteDiffusion models falls under <a target="_blank" rel="ugc" href="https://freedevproject.org/faipl-1.0-sd/">Fair AI Public License 1.0-SD</a> license, which is compatible with Stable Diffusion models’ license. Key points:<ul><li>1. Modification Sharing: If you modify SoteDiffusion models, you must share both your changes and the original license.</li><li>2. Source Code Accessibility: If your modified version is network-accessible, provide a way (like a download link) for others to get the source code. This applies to derived models too.</li><li>3. Distribution Terms: Any distribution must be under this license or another with similar rules.</li><li>4. Compliance: Non-compliance must be fixed within 30 days to avoid license termination, emphasizing transparency and adherence to open-source values.</li></ul>Notes: Anything not covered by Fair AI license is inherited from Stability AI Non-Commercial license.

SoteDiffusion Wuerstchen3

Did more experimenting than training this week and this release did reset the training. Disabled gradient accumulation steps, this saved me enough VRAM to train the TE. Enabled Text Encoder training, this forced me to reset the training. Since i did reset the training, i also decided to re-tag the entire dataset using these models instead (Currently in progress.): shadowlilac/aesthetic-shadow-v2 SmilingWolf/wd-swinv2-tagger-v3

Files with SHA256: f46c789a6635088caca27238359546081d1aaf39e66175c17bb931b5a57137f5