IP-Adapter but fine-tuned on anime images. The result is not perfect but definitely better than the non fine-tuned versions (for anime).
The model used to train is anything-v5 but it seems to work on other anime models too.
Note that this is not really a Controlnet but the 'other' category does not allow '.safetensors' files.
The original IP-Adapter repository: https://github.com/tencent-ailab/IP-Adapter
-----
Fun fact:
If you use the original IP-Adapter on an anime model and you supply a real image, it will give you an anime image that kind of looks like the original image.
If you use this fine-tuned IP-Adapter on a realistic model and you supply an anime image, it will every now and then give you a 'cosplay' image similar to the original image, but it will usually give you nightmares.
-----
How to use:
Tutorial how to use in Comfyui
(replace the ipadapter model with this model)
Description
fine tuned on the danbooru17 dataset
FAQ
Comments (19)
How do you use it ?
Added a section in the description. Hope it helps.
I noticed that the original IP Adapter Face ID also has a supporting Lora model that needs to be used. Does your version have a supporting Lora model?[1.1.428] IP-Adapter FaceID · Mikubill/sd-webui-controlnet · Discussion #2442 (github.com)
No, this is a fine tune of the regular IP Adapter, not Face ID so it does not have a supporting Lora model.
How can I use this with the diffusers library?
You could for example install the ip adapter from source and use this example https://github.com/tencent-ailab/IP-Adapter/blob/main/ip_adapter_demo.ipynb
Thanks for the great work! What type and amount of datasets did you use to train the Ip-adapter?
@arnon Thank you for your kind response! May I ask what hardware and time you used for the training?
@toyxyz , it took about 24h on an RTX 4090 laptop gpu.
@arnon Thank you! Did you use the example training parameters from the ip-adpater repo? Or did you modify them?
I kept the learning rate and weight decay. The other parameters are less clear as I edited the code for performance and memory usage (though the original code could also be used). E.g. an easy way to increase the performance of the training script is to precalculate the VAE step. I could share my code but it is kind of gross.
@arnon Thanks for the detailed explanation! Did you train from scratch or did you use a pre-trained ip-adpater model?
I used a pre-trained model.
@arnon Thanks! I'm trying to create an anime ip-adapater for SDXL. I am currently in the process of collecting the dataset.
@toyxyz I could probably indeed help you with that. I would have done it myself if I had time... The training didn't fit my local GPU and I didn't have time to look for alternatives online. So I decided to do it later when I had more time but eventually this never happened. So you will probably need a GPU with vram larger than 16 Gb (can't tell with certainty how much more is needed) but you can of course hire them online.
As for the dataset, I think the Danbooru2017 dataset should probably be enough? What datasets are you looking for? There is also the more recent danbooru2023 dataset as an alternative.
@arnon Thanks! I'm going to test it with a small dataset first. I'm currently looking at Danbooru 2017 (300,000) and AI generated images (20,000), and considering importing new images from the 2024 version if needed. There's not a lot of material out there on Ip-adapter training, so I'm going to approach this with caution.
@toyxyz Ok, if you have any questions, and you think I may be able to help, feel free to ask.
@arnon I found news that the developers of noob-xl are preparing an anime ip-adapter for SDXL!
Details
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.