This is an experimental LLLite Regional Controlnet model for Regional conditioning for Anima. https://huggingface.co/Sen-sou/Anima-LLLite-Regional-Controlnet
Usage
Use a color mask image as the conditioning input.
Any color can be used to define a region. There is no fixed palette or strict RGB requirement.
The mask background should be white.
Using simple solid colors (red, green, blue, yellow, etc.) for different regions is recommended for clarity.
The model was trained using manually masked conditioning images and therefore expects clearly separated regions.
Install https://github.com/kohya-ss/ComfyUI-Anima-LLLite in comfyui to use the model.
Controlnet end step value is recommended to keep at 0.45.
A higher controlnet strength can be used if the subject is leaking through the mask bounds
Prompting
Normal prompting works fine.
However, the model currently cannot determine which prompt corresponds to which colored region automatically. It only receives the region mask as additional conditioning and does not perform explicit prompt-to-region matching.
To guide concepts into specific regions, use spatial prompts such as:
"girl on the left, cat on the right"
"character in the foreground, city skyline in the background"
Combining this model with attention masking methods such as Forge Couple or the 'Attention couple' node by https://github.com/pamparamm/ComfyUI-ppm to get better results
Training
Trained on 580 images, 2 repeats, batch size 4, for 4400 steps
Conditioning images were manually masked for each image
Captions were generated with the help of wd-eva02-large-tagger v3 and https://github.com/pythongosssss/ComfyUI-WD14-Tagger custom node
Trained using https://github.com/kohya-ss/sd-scripts
Limitations
Because most training images consisted of close-up character compositions, generations involving distant subjects may not strictly adhere to the provided mask boundaries.
Support
If you want to support future training, you can support me via Ko-fi https://ko-fi.com/sensou
Description
Experimental Version v1
FAQ
Comments (13)
Insane.
Great work
If possible, please add support for Forge NEO.
Forge neo already supports LLLite models, you just have to upload your own control image in the controlnet control image section, load this model, set the controlnet recommended parameters and it works. You can also pair it with the forge couple to get even better regional adherence
@Sensou_ thank for reply~
@BIG_A this controlnet is officially implemented in forge neo, you can clone the latest commit
Help me out a little bit. I am confused on the masking part, do I need to add a masking image color the regions or use prompts manually?
just make colored regions using the comfyui mask editor on a white background and pass that image to the lllite apply node. I'll add the example workflow here for those who need it for comfyui
@Sensou_ Thank you that would be of great help
@Sensou_ Hi have you added example workflow? thanks
@Smexlo Under the optional files on the right. You can also download the workflow from the huggingface link
@Sensou_ Thank you will check it out.
I have a question. You’re using Turbo Lora for the Naruto and Hinata illustrations, but when I try to use it myself, the image gets really blurry. Is it just that Lora is still in my workflow even though I’m not actually using it? Or is there a way to prevent it from getting blurry?
i used turbo lora for that. Maybe its the cfg that you havent set to 1, that can cause blurred images if youre using the turbo lora.