The idea was inspired by 2 videos I watched yesterday.
The first was @Grockster's video "Flux EXTREME - Cutting-Edge Techniques for Ultimate Control" - https://youtu.be/aZt5CL3r1lQ?si=5jAB6SDfqOUfR5Vd - Where they showed using Mask to SEGS with SEGS Detailer for Flux Inpaint. I'd already been using SEGS Detailer but with BBOX Ultralytics and I didn't even know that Mask to SEGS node existed.
The second was @driftjohnson's video "Start using Promptable Masks, Segment Anything & Unlock FLUX Inpainting | Foda V10" - https://www.youtube.com/watch?v=dO2o0YbBvmI - Where he used Florence2 and SAM2 for setting the mask for Inpaint.
After watching both I thought can I connect Florence and SAM2 to the Mask to SEGS to do automatic masking with that and created this work flow.
Instructions are on the workflow.
Note: It sometimes throws up an error on the Florence2 Coordinates node "There's nothing in index: 0 File" It means the Florence Model has not been able to detect the prompted area and empty data has been passed to the node. Either change the prompt you're using to set the mask or try a different model. Base seems to be the most reliable and when I've had this switching to base has worked. I've tried a different image with the same model as well when this has got the error and that has worked fine.
Description
FAQ
Comments (17)
very easy to use, good notes. Thanks!
Thank you for the feedback and glad you like it :)
Hi, I am very interested in your workflow, but I am a little confused, what exactly is this?
switch mask
on_false
What kind of image does this option actually load?
————————
Is this Boolean a manual switch?
It's to give an alternative way to manually set the mask. When set to true it will use the mask automatically generated by the Florence2/SAM2 groups, when set to false you use the mask editor on the load image node near the switch to draw the mask and use that instead. Useful for when the automatic masking doesn't detect what you want masked.
@runebinder764 Thank you，very useful
Love this, I have seen a lot lot lot of inpaint workflows and this is the best IMO! simple in look but genius in design, I had to change the activate switches to different ones, they just gave a red outline w/ no reason why it wouldn't work....no big deal tho. I rarely switch to manual masking this thing is so good. I am going to be making some groups so then I can switch models from SDXL and SD1.5. I might also use this workflow to inpaint eyes and face and clothes a adetailer replacement. I like using sd1.5 as a detailer just to add find details to a flux upscale like skin texture, better eye detail and and fixing small faces (still haven't fully figured this out on how to make sure it only gets small faces)
I'm glad you like it and thanks for the feedback :)
Not working, I see it mask what I want then the workflow just stops how do I use that mask and actually produce an image?
I thought I'd replaced it with my most recent local version, just checked and I hadn't and have reuploaded to update. Try downloading it again and let me know how you get on. The manual mask was enabled on the previous version which might be why you were only getting the mask and nothing else, I've set the latest version to having the auto masing enabled by default.
If you don't want to download the new version, you'll see a pink group at the bottom of the workflow, change the toggle to True and that should fix the version you have.
@runebinder764 downloaded new version, doesn't really work I get an output, I tried to change the color of a dress it simply changes the material to like slightly loose, and not the color at all even though the mask is correct its not following my prompt at all, what box am I supposed to write in?
it's a fantastic tool :) However I've set the control bridge to Mute/Bypass, but I couldn't figure out how to run the second pass. I assume it'll automatically load the edited image, allowing the user to perform another inpaint, right??
For a second run I copy (clipspace) the result and paste it back into the Load Image node. I don't know how to do that automatically.
Works brilliant! Let's say it detects multiple elements. What setting do I need to change to inpaint all selected elements and not just the first one (hope that makes sense)?
Same question.
This is a really fascinating workflow, ive tried getting it to work.
Im struggling with understanding how to manually painting in the areas i want adjusted, not sure how im supposed to do that.
Also with "obedience" as in, if i type into the prompt "Closed mouth" on a photo with a person with an open mouth, and have the selector as "face" or "mouth", well, each time it fully ignores my request and just regenerates a different open mouthed photo with a slightly different style and i have no idea how to fix this
For the manual masking you need to go down to the bright pink group at the bottom (below the green one) called Manual Masking, set the Switch Mask and Switch Image toggles to False and load you image into the Load Image node in that group, not the one at the top. Right click on that image and choose Open in MaskEditior.
Inpaint in my experience can be a bit flaky with facial expressions, check out Expression Editor which is part of the AdvancedLivePortrait Custom Nodes and is perfect for chaning mouth, eyes etc. Video on how it works https://youtu.be/xzFjy6Cdz_c?si=wXT02fXotME_Svjk
Denoise just seems off. At .76-.79 it doesnt listen to CLIP at all. At 80 it does, but then the object being masked looks off.