Troubleshooting Style LoRAs

Sometimes I get messages that my styles aren't working for some people, but they're clearly working for others. I've done a lot of experimenting with LoRAs and know from experience a lot of the things that can affect them. So I'm posting this article in case others might find it helpful.

I'm writing this specific to my style LoRAs, but the general idea behind these tips should apply to most LoRAs. I'm focusing on issues with styles being too weak, but these tips can also be used for ideas if you find a style too overpowering.

None of these are rules you always need to follow. I encourage experimentation! They're just things to check and be aware of if you aren't getting the results you expect.

You aren't using a trigger.

All of my styles come through stronger when using a trigger. For Krea 2, it's almost always "n0vuschr0ma style". For SDXL and Pony, it's usually "novuschroma style". There are some exceptions, but this is always specified in the trigger section of the model page.

If you're using the Civitai generator, you'll see the link to add the trigger below the prompt. Some other UIs have an option for a trigger placeholder so you don't have to remember or manually type it. For example, in SwarmUI this is <trigger>.

The trigger is buried too deep in the prompt.

Placement matters when models prioritize what to emphasize in prompts. If you're placing the trigger near the end of your prompt and it seems like it isn't strong enough, try placing it at the beginning.

Your weight is too weak.

I always start with weight 1 and nudge it up or down as needed.

You should try a short prompt.

My styles should still work with long prompts (at least for most recent models), but it can get lost in the noise if you're including a lot in the prompt. Try a short prompt. If that works, you can gradually lengthen the prompt to see when it starts to break down.

Moving the trigger closer to the beginning is often enough to push it through without changing the prompt length. Things like the trigger placement being at the beginning vs end matter more when the prompt is longer. You can also experiment with increasing the prompt weight.

Words in your prompt conflict with the LoRA.

Sometimes style modifications you request in your prompt change my style so much you might not get results you expect. For example, this can happen if you're using a LoRA trained for illustration and you're trying to generate a photo realistic style.

Sometimes it even happens when you ask for the style the LoRA was trained for. For example, the clay style I trained and what your checkpoint model was trained to recognize as clay might look very different. So by using "clay", you might be mixing the checkpoint's training data with mine and get a blend of the two. If this isn't what you want, try omitting "clay" from your prompt and only use the trigger.

You need to add some "helper" words.

The opposite of the above tip can also be an issue. If the trigger alone isn't giving you the look you want, try guiding it toward what you want with the words in your prompt. You might benefit from adding "made of clay" for a clay style or "watercolor" for a watercolor style.

Other LoRAs are overpowering mine.

If you're using multiple LoRAs, sometimes one's strength or style overrides the other. Try reducing the weight of the other LoRAs that might be overpowering mine.

You're using too many LoRAs.

If you're using a lot of a style LoRAs in a single prompt, you'll get diminishing returns. Try removing some of them. You can gradually add them back one by one to see when they start to break down.

Your checkpoint interprets my LoRA differently.

I always train my LoRAs on the official checkpoints. The same prompt can look very different with a different model checkpoint. Some checkpoints specialize in certain styles, like realism or illustration. For example, a realism model can have trouble rendering an illustration style LoRA. You can try using the official checkpoints if you really want the style I posted in my example images, or you can experiment and accept the interpretation other checkpoints give you.

Your subject has a strong bias toward a certain style.

Some subjects have a strong link to a specific style, and it can be difficult to override that with a style LoRA. One example might be generating a cartoon character in a photography style. Another would be niche subjects that might have only been trained on photography or highly detailed illustrations (I see this a lot when trying to render unusual animals in simplistic illustration styles).

Sometimes rephrasing your prompt to make it very clear you want the subject in a certain style helps. If you have access to img2img, controlnet, or image prompts, sometimes you can apply your style to the image afterwards without mentioning the subject keywords that are triggering the bias.

Sometimes applying the other tips in this article (weight, moving the trigger, etc) can help, even if you don't need to do it for other subjects with the same style.