Imported into Civitai from <a target="_blank" rel="ugc" href="https://rentry.org/LazyDAdaptationGuide">https://rentry.org/LazyDAdaptationGuide</a>This guide is a repository for testing and tweaking DAdaptation V3 LoRAs, introduced by Kohya on 05/25/2023 .For reference to my guide on collating a dataset, and the old method of utilizing theAdamW8Bit optimizer, see - <a target="_blank" rel="ugc" href="https://rentry.org/lazytrainingguide">https://rentry.org/lazytrainingguide</a>Useful links:Kohya - <a target="_blank" rel="ugc" href="https://github.com/kohya-ss/sd-scripts">https://github.com/kohya-ss/sd-scripts</a><a target="_blank" rel="ugc" href="https://github.com/kohya-ss/sd-scripts￼bmaltais"> bmaltais</a> training GUI (Dreambooth <a target="_blank" rel="ugc" href="http://localhost">localhost</a>, DADAPT V3 HAS NOW BEEN IMPLEMENTED) - <a target="_blank" rel="ugc" href="https://github.com/bmaltais/kohya_ss">https://github.com/bmaltais/kohya_ss</a>With the inclusion of DADAPT V3 into bmailtais, I have provided a .json via <a target="_blank" rel="ugc" href="https://files.catbox.moe/8yh7ko.json">https://files.catbox.moe/8yh7ko.json</a> - edit your img folders, max epochs, dim/alpha, weight decay etc.DAdaptation is an adaptive optimizer that automatically selects and tweaks the TE/UNET/LR values as it trains.I have spent the past few months tweaking the older DAdaptation implementation (Assumingly 1.5) via the derriandestro repo. With the release of V3 and other versions of the DAdaptation optimizer, I firmly believe that this is the best optimizer to use and oust AdamW8Bit as the default optimizer.Here is an example of the shift in learning rates between DAdapt V2 (green) and DAdapt V3 (blue) - in general, it's less heavy in blasting out a higher learning rate and cooking the image faster, meaning we can keep the weight decay lower than previously practiced:<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/a1bf0f0c-bffe-45c7-8499-61712210de47/width=525/a1bf0f0c-bffe-45c7-8499-61712210de47.jpeg" />DAdaptation removes the need for setting restarts, setting learning rates, text encoders, unet learning rates etc. Simply put, you can flick a switch and it will generate your LoRA with minimal effort.DAdapt can be used for all LoRAs. Characters, concepts, artist, style etc.DAdapt needs a higher 'weight decay' value, as expressed in the research data and the LoRA results. This is the primary method for tweaking your LoRA. Is your LoRA not learning enough before it overfits? Increase the weight decay in .1 increments. With the release of V3, it seems to blast the learning rate a lot less heavily than V1.5 or 2, and so we can usually get away with reducing the weight decay more than we used to for the previous interations of the optimizer.DAdapt should have the learning rate, text encoder and unet learning rate set anywhere from 0.5 to 1. This is personal preference, but I've found keeping it at '1' and tweaking the weight decay is the simplest method of fine-tuning training.DAdapt needs the argument <code>--optimizer_args "decouple=True"</code> setting along with the weight decay settings (for example): <code>"weight_decay=0.4" "betas=0.9,0.99"</code> This can be done in the Bmaltais Kohya GUI through the parameters boxes.Here are a few of the settings I have confirmed work well:Learning Rate (LR) 1Text Encoder (TE) 1UNET 1Weight Decay (WD) - From 0.2-1 - will require longer training to learn more, but FAR less prone to overfittingRepeats 1 for each data subfolder - Allows for tweaking of steps via epochs and removes the need to do mathResolution 512 - DAdapt is a heavier train and needs better hardware. 768 is not worth itClip Skip 2 - For NAI based modelsTraining Model NAIScheduler Cosine - Constant is not good, and I'm not sure why it was originally recommended for DAdaptationBatch Size - As high as you can go for your hardware. For example, 6 for a 3060 - From my experience it's better to set this and forget it, and focus on more impactful settings for trainingEpochs - Set it as high as you want. As an example, I have tested a 36 image, 3 character LoRA. This started to burn out at around 230 epochs at weight decay 0.4 totalling around 2400 steps at batch 6 - Once again, DAdapt needs more steps to achieve results as we're actively hampering the learning through decay DIM32 - (or set it lower to '8' for example, increase training time/reduce weight decay)Alpha16 - (match the DIM if set to 8 or lower)Warmups - Relatively untested, but I just set it to 0.1 to give it a bit of time to calibrate (?) - set warmups to 0 if you're using the above .json and the "growth_rate=1.02\" argumentHow many training steps for my LoRA? - Final steps are divisible by your batch - for example, at batch 6, for a single character LoRA on DAdaptation I would probably expect anywhere from 600-1200 steps. Generally, training til overfitting and going back down the epochs will save needing to re-train the LoRA all over again with new settingsExamples:XYZ plot starting at 100 epochs, up to 234 - 1/1/1 TE/UNET/LR, 0.4 weight decay, 0.1 warmup steps - training at batch 6 totalling around 2,400 stepsNote that these are trained on a tiny dataset of 12 images per character with one outfit, and should not be a representation of the plability of character clothing, positions etc.<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/6d70d4ae-0375-493f-a253-58311241f2bd/width=525/6d70d4ae-0375-493f-a253-58311241f2bd.jpeg" />LoRA trained on 3 characters, example tags:Aru - Aru, long hair, breasts, bangs, shirt, gloves, ribbon, yellow eyes, white shirt, pink hair, horns, collard shirt, red ribbon, coat, blunt bangs, fur-trimmed coat, haloShiroko - Shiroko, long hair, breasts, bangs, blue eyes, animal ears, animal ear fluff, mismatched pupils, cross hair ornament, blue scarf, grey hair, haloKarin (Bunny) - Karin, large breasts, black hair, halo, animal ears, cleavage, hair between eyes, fake animal hears, playboy bunny, rabbit tail, yellow eyes, very long hair, dark skin(The final epoch version has been uploaded to this Civitai post)Example .safetensors -Final Epoch - <a target="_blank" rel="ugc" href="https://files.catbox.moe/muah1o.safetensors￼200">https://files.catbox.moe/muah1o.safetensors 200</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/luvylf.safetensors￼180">https://files.catbox.moe/luvylf.safetensors 180</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/xkemzm.safetensors￼160">https://files.catbox.moe/xkemzm.safetensors 160</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/ane6p6.safetensors￼140">https://files.catbox.moe/ane6p6.safetensors 140</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/frshlg.safetensors￼120">https://files.catbox.moe/frshlg.safetensors 120</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/47hsmp.safetensors￼100">https://files.catbox.moe/47hsmp.safetensors 100</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/qgbxs9.safetensors">https://files.catbox.moe/qgbxs9.safetensors</a>

Files with SHA256: 1cb7a73e142fee20fa1c3c550709f296078ef411228f1fcf3676b443da05ac5a