This post contains three types of fine-tuned results with the same dataset: LoHa, LoCon, and full checkpoint (all with clip skip 1).Associated hugging face repository: <a target="_blank" rel="ugc" href="https://huggingface.co/alea31415/roukin8-characters-partial">https://huggingface.co/alea31415/roukin8-characters-partial</a> (more versions can be found there).Note that the models are fine-tuned on top of <a target="_blank" rel="ugc" href="https://huggingface.co/Crosstyan/BPModel">BP</a>, so you get the style I show below by using BP. Using these models on orange could kill the orange style unless using with a properly adjusted weight.What is LoHa/LoCon?LoHa: Lora with hadamard decomposition (ref <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2108.06098">https://arxiv.org/abs/2108.06098</a>)LoCon: Lora extended to residual blocksSee <a target="_blank" rel="ugc" href="https://github.com/KohakuBlueleaf/LyCORIS/blob/lycoris/Algo.md">https://github.com/KohakuBlueleaf/LyCORIS/blob/lycoris/Algo.md</a> for more explanationHow to use?For webui install <a target="_blank" rel="ugc" href="https://github.com/KohakuBlueleaf/a1111-sd-webui-locon">https://github.com/KohakuBlueleaf/a1111-sd-webui-locon</a> and use it as a loraWhy LoHa?The following comparison should be quite convincing.(Left 4 LoHa with different number of training steps; Right 4 same thing but for LoCon; same seed same training parameters for the two training)<img src="https://imagecache.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/8bf8cc77-2c5a-46e0-eb4b-5dea65408c00/width=525/8bf8cc77-2c5a-46e0-eb4b-5dea65408c00" />In addition to the five characters I also train a bunch of style into the model (what I have always been doing actually).However, LoRa and LoCon do not combine styles with characters that well (characters are only trained with anime images) and this capacity gets largely improved in LoHa.Note that the two files have almost the same size (around 30mb). For this I set (linear_dim, conv_dim) to (16,8) for LoCon and (8,4) for LoHa. However with Hadamard product the resulting matrix could now be of rank 8x8=64 for linear layers and 4x4=16 for convolutional layers.P.S. Note that the 20000 step files are provided here so if you want other version please check hugging faceHow about full checkpoint?It is still the most performing as illustrated below. Maybe we can get such results with LoHa with larger dimension, better decomposition of the convolutional layers, and by adding back the few layers that are not yet currently trained in the LoCon/LoHa implementation. Let's see.<img src="https://imagecache.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/54fb993e-2701-4373-001b-378eef082c00/width=525/54fb993e-2701-4373-001b-378eef082c00" />Training details<ul><li>LoCon/LoHa: alpha 1, dimensions as specified above, batch size 8, learning rate 2e-4 throughout with constant scheduler, AdamW</li><li>Full checkpoint: batch size 8, learning rate 2.5e-6 with cosine scheduler, Adam8bit, conditional dropout 0.08</li></ul>Example ImagesImages 1 and 4 come from LoCon, 2 and 3 come from LoHa, and 5 and 6 come from full checkpoint.DatasetI have uploaded the anime part dataset of 3000 anime screenshots. The current format works with <a target="_blank" rel="ugc" href="https://github.com/victorchall/EveryDream2trainer">Everydream</a> with the multiply.txt and for <a target="_blank" rel="ugc" href="https://github.com/kohya-ss/sd-scripts">kohya trainer</a> you should use this script <a target="_blank" rel="ugc" href="https://github.com/cyber-meow/anime_screenshot_pipeline/blob/develop/utilities/flatten_folder.py">https://github.com/cyber-meow/anime_screenshot_pipeline/blob/develop/utilities/flatten_folder.py</a><pre><code>python flatten_folder.py --src_dir /path/to/dir</code></pre>

Files with SHA256: 7c2798c4c26176393f29d3bd06ab55443ed770ed8a4341318bd2d191b3fc2b5e