Thank you very much for putting all of this together for us. I know it must be cumbersome at times! But for those of us just starting out with ControlNet and feeling our ways around, this helps to find models and documentation much more easier.

I'm so happy the atoms and genetics that make up your body and mind coalesced as they did because it has been a great help to all of us.

i cant get these to work on comfyui, i keep getting errors.

Error occurred when executing ControlNetApplyAdvanced: 'NoneType' object has no attribute 'copy'

i downloaded the controlnet from comfyui-manager and i am using apply controlnet.

@eurotaku I've just noticed that in here (kataragi inpaint model) they say it's a testing model that doesn't perform well. Is it worth adding it here yet? On the other hand, as a backup it may be not a bad option. In any case thank you for updating the list with new stuff!

The xinsir canny v2 is so good, I don't see much reason to use openpose anymore

Does anyone know how to use the BRIA AI Background Generator in A1111? I'm trying to find any documentation about what preprocessing to use, if any, in what form to enter images. Whenever I do I get the error message:

"RuntimeError: Given groups=1, weight of size [16, 5, 3, 3], expected input[1, 3, 1024, 1024] to have 5 channels, but got 3 channels instead"

I have no idea what input has five channels. Even a PNG with an alpha channel has only four. If there is a user manual then I can't find it.

these are checkpoints and loras? how touse them and where to put them? load them like normal checkpoints and loras? need more info

Thank you very much for compiling this list, it has been incredibly helpful to me.

For anyone using Forge/A1111 and Lineart not working...
Model won't work if you're not setting your resolution to 1024.

Very weird.

RECOMMENDED MODELS FOR PONY-BASED CHECKPOINTS (PLUS GENERAL TIPS ON HOW TO USE THEM)

Pony is a very overtrained finetune. Basically, this means that it was trained so much to a point where it has relatively little in common with the original XL model. As a consequence, ControlNet models who have been trained for XL may not work as well with Pony.

Here's an incomplete list of the best ControlNet models for Pony-based checkpoints. Keep in mind that this all comes from my limited experience, so the list is very wip now. Please feel free to add your own results and/or correct me. I'm also using Forge, so things might be different if you're using Comfy or something else.

I've been updating this list and ended up including some tips that may be useful even if you're using a non-Pony XL checkpoint, or even for 1.5 checkpoints as well.

DEPTH: Xinsir's depth model is the best. Diffusers' depth-full works well, but with some quality decrease or style corruption at higher weights. Haven't bothered to check others. I've only used the Midas preprocessor so I don't know if it does well with other preprocessors. Consider lowering the control weight (0.2 to 0.8) to give the AI more "artistic" freedom.

Update: I've done further testing and discovered the following: Unless you intend to generate a "difficult image" (like poses that the checkpoint does not understand well, image compositions that the checkpoint does not like to create, etc), you may be able to go very low with the control weight (like 0.05 or even 0.02) and still see a clear influence on pose/composition, while also giving the AI lots of freedom to change details and minor variables.

The depth preprocessor is VERY powerful and can really help you achieve a specific pose/image composition even if your reference image is very different from what you actually want. To fully tap this power, I really recommend going to an image editor like Gimp and make a manual mask to be used alongside the map.

OPEN-POSE: Thibaud's openpose (not Thibaud's openposelora) is the only one that works for me. Even then, it's not perfect and will often give me poses that are similar but not 100% like the one in the map. If this is an issue to you, you can either:

a) try increasing the control weight to 1.2 or even higher , but increasing the weight past 1.0 decreases image quality. The effect is usually tolerable up to 1.2 but past that it starts to become more noticeable.

b) combine OpenPose with Canny or, even better, Depth on a low control weight. The bad part is that it slows down your generation and may decrease the AI's "artistic" freedom, unless you manually make masks for the canny/depth maps.

Sometimes OpenPose has a hard time functioning properly when I'm using Loras with very specific poses that contradict the reference I'm using. When this happens, using a densepose preprocessor may help (besides the tips above)

INPAINTING: EcomXL's inpainting seems to be the best of them, but you can try out Kataragi's if EcomXL's fails to give you the results you want. Destitech's inpainting for some reason always creates a black hole in the image. Besides that, there are three important points to get good results with inpainting:

1) If your image turns out deepfried, you may need to decrease the weight control. It seems there's no rule of thumb here: sometimes I get a good image with weight 1, sometimes I have to turn it down to 0.8 or 0.6 to avoid deepfrying, sometimes I need to go all the way down to 0.05. Even low values (like 0.05) may have a positive effect on image coherence, but higher values are even better (or even necessary for high-complexity images), so you want to keep the weight as high as you can without deepfrying things.

2) The inpaint_global_harmonious preprocessor is the best one when it comes to not deepfrying things, which is fortunate because it's also the preprocessor you'll be using the most, as it's meant to be the best at keeping image coherence. If you want to remove an object from the picture (like removing a cup from a table, a detail from a shirt or something like that), you may have an easier time with inpaint_only+lama, depending on how complex the area around the to-be-removed object is.

3) If you feel the results blend poorly with the rest of the image, increase your steps by 10 or 20, even 30 if you're desperate.

One of the best parts about using ControlNet inpainting instead of the standard inpainting is that it allows you to set the denoising strength very high but still blending reasonably well with the rest of the image. I usually go for 0.95 and will sometimes go a bit lower.

OUTPAINTING: Once again, EcomXL's is the best one in my experience, followed by Kataragi's, but they are still terrible for outpainting. Consider using a non-ControlNet solution, like Mosaic Outpainting (if your image's composition allows for it) or Envy's Zoom lora (to make your generation more zoomed-out from the beginning so you don't need to outpaint).

IP ADAPTER: ip-adapter_xl works reasonably well so I haven't tested any of the options here. You will need to turn the control weight down (0.2 to 0.5) else it will horribly distort your generation. May also need to try out with the different preprocessors available to see which ones actually capture the characteristics you want. ip-adapter_xl is available here: https://huggingface.co/lllyasviel/sd_control_collection/tree/main

SCRIBBLE: Haven't managed to get good results with neither of the two models avaiable. Don't take my word for granted, though, as I should've run more extensive tests. All I can say is that none of the two models worked properly "out of the box" for me, though they might have with the proper settings.

CANNY: Diffuser's Canny-full seems to work just fine but Xinsir's Canny v2 seems to be even better. Try out different weights to see which one gives you best results.

LINEART: I tried Kataragi's lineart model and it worked so well I didn't bother trying out others. Also, I admitely only tried the lineart_anime_denoise preprocessor, but if the model worked with one preprocessor, then it should be able to work with all of them.

But there's an issue: I don't know why, but some images turn out to be excessively bright when using this model. If you have this problem or any other similar to this, I believe you could try diminishing the ending control step down to 0.8, 0.6 or even 0.4, so that Controlnet's image composition will be preserved but the lighting/style will be free to be determined by the checkpoint you're using.

Also, here's a comment by @mrjoshuatee: "I've also found that Mistoline seems to be the best result compared to any controlnet I've tried with Pony as a lineart. https://civitai.com/models/441432/mistoline

It must be set to balanced for some reason though as when I try it on controlnet is more important, it shows even less influence to the controlnet image."

TILE: I haven't tried any model tile on my own, but @Zapper commented: "I use this one https://civitai.com/models/330313?modelVersionId=444122

It says that model is realistic, however I found it working good enough for Pony too. Mostly I use it for upscale with denoise about 0.7 and tile controlnet in "my prompt is more important" mode and weight from 0.5 to 1.0 (for example use 1.0 if upscale from 1024x1024 to 2048x2048). Also it is good for minor inpainting if it is important to keep image consistent even with pretty high denoising."

Personally, I tried it out but the images it generated were blurry as fuck. This is just my experience, though, as I haven't tested it very thoroughly and it might be just skill issue on my part. Anyway, still looking for suggestions on Tile models if you guys have any.

I only tried inpainting but it doesn't work for me, when I do the test with and without controlnet I have the same result. Thanks anyway

Does anyone know how to solve this error:TypeError: 'NoneType' object is not iterable

And what's that? And why is that? From the description, the answer is not clear.

bdsqlsz depth is now one of my faves for doing letters and logos with no pre-proc. Just a mask made in Photoshop. But I have no idea how to pronounce it! LOL I end up saying something like "BeeDee Squizels".

Hetaneko, Diffusers and Kohya's controllite are good too. More fiddly than 1.5, but they definitely work once you're got everything dialed in.

If you're running Auto1111 v1.9.4, nonetype rsplit error is back unfortunately. It usually takes quite a few gens for it to happen, but it will and it's not the CN model's fault as it's totally random. If you switch CN models and get that error, just restart A1111.

Here is some model called "bg-gen"
But how to use it in A1111 ControlNet? Can anyone explain please?