This is a Proof of concept of having a simplified version of an image and 'rerender' it with different styles or shading. It sometimes causes Nans in the Unet, reset the UI to fix it, I hope this could be picked up and have better support.

1) Render Control Net
It uses a Kmeans posterized image or flat color to render an image, or change the style
Preprocessor:
- Kmeans posterize, use G'Mic posterize filter with 9 colors
- Swap channels RGB to BGR (Thanks OpenCV, 😒), in gimp you can change it easily with Colors > Components > Channels mixer

- Balanced or Control net more important, works with weights of 0.6-1.0, txt2img

2) HSX Control Net
It uses a posterized image or flat color to render and image, or change the style
Preprocessor:
- posterize with quantized HSV, use G'Mic HSX posterize-index filter with 12 hue bands, 4 (or 3) saturation bands, 4 (or 3) luminosity bands, and HSV color space

- Now a apply median blur of radius of 3
- Use "my prompt is more important" in Auto1111 with weight of 1.0, txt2img

3) Hue Control Net
It uses a Kmeans posterized image or flat color to control the hue and saturation of an image, made for 3D render passes or flat colors in drawings because it lacks the concept of shape, another controlnet has to give it. In the example the lineart was inverted with MSLD at 1.0, Render controlnet at 0.6 and Hue controlnet at 0.4, using txt2img.
Preprocessor:
- Kmeans posterize, use G'Mic posterize filter with 9 colors
- set value to 127, create a layer over the image and fill it with a 50% gray, then set the blending mode HSV value

- Balanced or Control net more important, works with weights of 0.4-0.6

Training:
1) Finetuned a model with a custom dataset of 150k images removing any color word, like red, blue, gold, magenta, etc from the prompt
2) Used diffusers repo to train a blank ControlNet with the preprocessors used in each network
https://huggingface.co/blog/controlnet