WARNING : after rigorous tests, I can affirm that this Lora can induce some kind of, I guess, VRAM mix-ups wich completely messes up your generations if you switch models in Automatic1111.

I stumbled upon this while experimenting a bit. At first I was generating different pictures without changing the prompt or anything else, only the seed, and it was working fine. Then I tried switching models (all being XL models of course) and it screwed up, seemingly totally ignoring my prompt. The style of the pictures were still charcoal drawings, but the results were kinda random and very blurry, as if no prompt was entered.

At first I suspected it might be a problem with the new model I had switched to. I tried other ones, same problem. I restarted A1111, chose one the models which had been problematic just before, and this time the prompt was respected. If I tried generating multiple times without switching models, it went fine each time. But if I switched to another one, same problem as before, instantly.

Ok, so I thought there might be some kind of conflict between extensions or something, so I disabled them all, same tests, same problem. I then tried to rollback to an earlier version of A1111, same tests, same problem.

Finally, I tried using all the same parameters, just without using this Lora. And voila, this time I could switch models and the prompt was respected each time.

I insist on the fact that as long as I didn't switch models, it worked fine. I could change the prompt or seemingly any other paramater (seed, sampler, CFG, steps...) without issues. As soon as I switched models though, the problem appeared.

If, then, I removed the Lora from the prompt, no problem. Putting it back again ? Same problem. Restarting A1111 and using the problematic image generated just a bit earlier as source for the parameters ? No problem, the prompt is respected. Then not changing any parameter, model switch : sam probleme.

So, I don't know anything about how Lora really work technically, to be honest, but this one does mess up really badly in some ways when switching models in A1111, at least for me. Of course it could be because of some specific things in my setup, but still, I think it needed to be announced.