This ComfyUI workflow was mainly created to do one thing in SDXL: "Multiple characters Loras" that embeds every process of how the image is generated.
It's like semi-inpaint that starts from the beginning of the generative process, and is compatible with "Tile Upscale (Ultimate SD Upscale)".
It also works well without character lora.
It works well with AMD GPU, should work with an Nvidia GPU too, if the custom nodes work as intended.
📌Featuring
Illustrious 2.0 Focused Workflow (Supports SDXL models)
Regional Prompts With Regional Lora (Up to 3 characters)
All-in-one Workflow (KSampler/Ultimate SD Upscaler/Detailer/Color Correction/Upscale/Inpaint)
Optional ControlNets (OpenPose, DepthAnything, Canny, SoftEdge, Tile)
Separated Base/Detailer Prompts (less upscaler/detailer hallucinate)
Flexible Workflow Starts (Starts your generation from any part of the process)
Semi-Auto Mask (starts workflow right away without manual masking involvement)
Suitable for users who can navigate the workflow spaghetti.
However, I've arranged the node groups in order from left to right and up and down. Categorized separately by Header and Textboxes. It should make it easier to follow through and input text/number accordingly. With Hotkeys to quickly navigate the workflow
The Guide, how to use, and that sort of stuff are inside the workflow. Ask me if there are any confusing words in the guide that may need improvement in the comments.
Please avoid flooding me with personal chat as best as you can. I encourage posting your question in the comments.
⚠️Forewarning
Backup your Local ComfyUI Environment.
When installing/updating new custom nodes, things can break from forcing to install certain version of site-packages, which may be incompatible with your settings.
Not suitable for impatient users. The workflow is very delayed.
Depending on how many group toggles are enabled.
And if using "Regional Prompt/Lora", it will generate very slowly.
2-4 times slower (depending on how many characters/character loras enabled).
There's a basic auto masking, but complicated poses still require manual masking.
Similar to "Inpainting", it requires tracing the character from the image to give proper facial expression/pose and prevent character features from bleeding.
💾Download Section
🎛️ControlNet (models > controlnet)
Depth Anything - Rename file to "t2i-adapter-depth-midas-sdxl-1.0".
Canny - Rename file to "t2i-adapter-canny-sdxl-1.0".
🔍Detailer (models > ultralytics)
📦BBOX (ultralytics > bbox)
Eyeful v2 Paired or Eyeful v2 Individual (Can use either Eye Detector models)
📐SEGMENT (ultralytics > segm)
🧠Sams Detector (models > sams)
💎Upscale Model (models > upscale_models)
⚠️Known Issue
[If using Load Image node] You need to set your image "Aspect Ratio" to match your "Load Image". Still trying to fix the dependency cycles error when implementing "Auto Get Image Size".
VERY LAGGY/DELAYED when generating due to the sheer amount of nodes and checks on the frontend. Can be remedied by keeping the browser size small.
At the last part of every prompt. You have to manually add a comma "," at the end.
Overburn/Overbaked Image could still happen if using 3 characters Loras, the countermeasure is to lower CFG by 1-3 for KSampler or use Sampler: "Euler_a_cfg_pp" with CFG: 1.3-1.5.
Somehow Sampler: "Euler_a_cfg_pp" in Detailer causes artifacting in the corner, hence I use the standard Sampler/Scheduler combo for this part
[Detailer "Tile SEGS" and Detailer Fix's "Character Bound" Mode] It is somehow forcing the generation outside of the detected area. It still works, but takes unnecessary long generative time.
Credits
Nerdy Rodent (Video Link)
His video sparks my interest in creating this madness AIO of workflow using his method.
Not so simple (or is it?) Illustrious Workflow by @Gladas
took some good sampler/scheduler combo, nodes, and settings from there
Regional Conditioning with AttentionCouple(LoRA supported) by @Nyraen07
Took some inspiration from there too. Also works similarly to mine but with different custom nodes (uses "A8R8 Attention Coupling" + built-in "Set Cond Props" nodes, my workflow only uses ComfyUI built-in "Set Cond Props + Set Default Cond Props")
END NOTE
I took an interest in generating stuff that I'm... interested in, along with gaining some useful insight, and sharing images/videos. This workflow is part of my learning and experimenting node and workflows.
If you find this workflow on another site besides Civitai, the uploader is responsible for providing the support.
I have no way of knowing what gets changed by the uploader.
May you find my personal workflow useful and suits your needs.
Description
Test Release
Adds an example prompt. Can generate everything from start to finish.
Flexible generative process. Continue where you left off with Load Image.
Adds experimental Inpaint using Detailer and BBOX person detector. Manually fixing the bad anatomy part of the image is possible.
There are still a few known issues that aren't generative breaking bugs; just need to be a bit mindful when using them.
FAQ
Comments (10)
I've been enjoying this workflow quite a bit so far. The quality feels leagues above when I was getting by chaining together nodes in a clean tab.
The only issue I've run into is the SD upscale pass will sometimes go rogue and redraw each tile. But the controlnet seems to help with that for the most part.
The fact that it works with AMD and my crappy 8gig card helps too.
Very glad my workflow is serving you well.
As for SD Upscale part, do you mean it's hallucinating extra body parts or adding random characters in the background?
If that's the case, you could try lowering "Denoise" to 0.25-0.3 since my default setting pushes the limit with quality before hallucinating kicks in too often.
Worth noting that some "Checkpoints" are less compatible with Tile Upscale, so they require less denoise to mitigate annoying hallucination.
"Regional Prompt" method is also a good option to limit your character prompt to a specific area, though it does add some manual masking hassles and generates much slower.
@Sillyfools The biggest issue I was having is that on other models, the upscale would literally redraw the tile from scratch. It would be close to the base image, but the pose would be different, so when it combined all four tiles, she'd have limbs in places they shouldn't be. Or a forearm in the wrong spot. Lots of ghosting.
It mostly did it with other models. The one you have as default for the workflow almost never did and setting the tile controlnet caught the little that did slip through. And I prefer the aesthetic of that model vs the others I tried anyway. It seems to be pretty well rounded in both character and background quality.
The only other big issue I was noticing is the upscale passes was really cooking in the loras. One of the loras i used had a strong tendency for clothes or armor so she'd end up with one boot but the rest nude. So i just added a new checkpoint loader right before the upscale and connected that one. That fixed that issue. I know I could have gone from the original checkpoint loader but I wasn't sure if there's a relay or if I'd need to drag it halfway across the workflow. (Or if there's a toggle for that already.)
Now if only I could get her to look like she's supposed to look. So far I can get one or two key features, but the rest go out the window. She might just be too far outside the box for an AI to do right.
@lokiie1984 Try fixing the unwanted redraw by lowering USDU's denoise to ~0.2-0.3 (In the "Setup" section). Enable "Tile ControlNet" later too, if it still doesn't work.
If denoise is too high (and possibly vague prompts like "dynamic pose" or even empty pose prompts without anchoring like "standing" or "sitting"), it will more likely cause unwanted redraw because the sampler gets way too creative on individual tiles. Even more random if using a different "Seed" (if you use Load Image for USDU and forgot to lock "Seed").
Do avoid adding extra "Embeddings" that can conflict with the default quality prompt. Pick either "Embeddings" or default prompts. It possibly lowers other prompts' priority, amplifies the quality prompts, thus overcooking the image.
Prompts with emphasis above "(:1.3)" can also lower the other prompts' priority, sometimes overcook the image on some models.
As for the Lora and checkpoint toggle on USDU, there isn't a toggle to disable Lora on a specific sampler, as I don't have an issue using Lora with them. But it would be a nice feature to have. I do the "Image Fixing" using "Load Image" from the last good image.
@Sillyfools That's probably part of my issue, I really need to look into how to actually prompt these things. The embeddings + quality tags was probably part of my issue too. (And everything else you said lol.) I'm sure my prompts are a how-to guide on everything I shouldn't be doing. But I'm learning, I guess.
One of the things I both really like and hate is the randomness of it. Trying to get it to draw her right and it randomly adds this big demon dog standing over her. Is it gonna eat her? Protecting her? No idea, but it looks cool, so now I need to let it go all the way through the refining steps.
Sadly I couldn't get this to work I install all that was listed on this page and the node's that ComfyUI did list and some I found on my own but I still getting errors!.
invalid prompt: {'type': 'missing_node_type', 'message': "Node 'ID #1220' has no class_type. The workflow may be corrupted or a custom node is missing.", 'details': "Node ID '#1220'", 'extra_info': {'node_id': '1220', 'class_type': None, 'node_title': None}}
Might be because I'm using a Nvidia
Node 1220 is related to the compatibility toggle node for Nvidia/AMD. Search "cudnn" in ComfyUI Manager and find "ovum-cudnn-wrapper" (ID:3769). You could also try to download directly with this link: https://github.com/sfinktah/ovum-cudnn-wrapper
Last resort would be redirecting CLIP's link in that missing node using hold Shift + hold left click the missing node's right side of CLIP link, and drag into "CLIP Set Last Layer" node and deleting the missing node. That missing node is just a link pass to toggle AMD compatibility.
Note: That missing node is in the "Setup" section, top right of the group, between "Vae Check" node and "CLIP Set Last Layer" node.
@Sillyfools Well that part seems to be working now but when I do a test run it now.
[Errno 1094995529] Invalid data found when processing input: 'D:\\ComfyUI\\input\\example.png'
File "av/container/core.pyx", line 482, in av.container.core.open
File "av/container/core.pyx", line 298, in av.container.core.Container.__cinit__
File "av/container/core.pyx", line 318, in av.container.core.Container.err_check
File "av/error.pyx", line 424, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input: 'D:\\ComfyUI\\input\\example.png'
Prompt executed in 2.15 seconds
And when I bypass them as I don't want to load an image it then bring up
RGBYPMaskToRegularMasks
RGBYPMaskToRegularMasks.convert() missing 1 required positional argument: 'rgbyp_mask'
Which I have got installed
@Lucifer_kitsune Does your "Input" folder have the "example.png" image (the cartoonish kid drawing image)?
ComfyUI always has the "example.png" image file in the "Input" folder. So I put it as a default dummy image.
The "RGBYPMaskToRegularMasks.convert() missing 1 required positional argument: 'rgbyp_mask'" one is likely due to "RGBYP Load Image" inside "Regional Prompt" group is being bypassed/muted.
For a solution of "Load Image" nodes, you can basically add any dummy images to all of them (or add any image inside ComfyUI's "Input" folder and rename them to "example.png"). I made "Load Image" to use those images specifically when the prior samplers are turned off (For example: If KSampler is turned off, USDU will use its "Load Image" node instead of getting an image from KSampler).
@Sillyfools sorry for late reply, At first I could find that image because it was in my C:drive so I move to to the D:drive and it work funny how a small image can stop it from working.
It seem to be running all okay now will mess around with it still.
Thanks