You might want to take a look at your lora. On site, bare-bones, PonyDifV6 only with or without prompting, I get an abstract oil painting of a mess. Even removing emphasis from "sexy nerd" does nothing. This is a promising lora. I'm waiting for a re-release :)

The lora results are too baked in. Such as background and clothes. I have to offset it with relatively high weights to force any meaningful change. Perhaps consider a new dataset where the lora character is bare without clothes and then train with that dataset from different angles, poses etc. Then do the same with clothes and with expressions. This should provide you a better result. But hey, don't take it from me. I've only read on it, not done it myself. Either way, I hope you keep at it :) Thanks for the lovely lora.