Can you post some of your samples? What is in your prompt?, you can lower the strength of the lora to get more flexibility, i recommend you to use the pony version to avoid doing schizophrenic prompts.