From all the tests I actually don't know what made it work but I'll list what I remember.

I did the training from the start each time the results came up bad.

This is from epoch no.04 from 10 which gave the best results and still flexible enough to except prompts.

Trained on 94 images, some are NSFW.

I also add a lot of captions to each image ( I think the minimum was 22 captions).

Trained on Chilloutmix Fp32Fix model.

For the best results use 0.5-0.7 for the Lora weight and a clip skip 2.