IMPORTANT INFORMATION I FORGOT TO ADD
AMD Users have to have the arg --no-half
Here’s how you set the COMMANDLINE_ARGS environment variable across different operating systems.
Windows (Command Prompt or Batch script):
set COMMANDLINE_ARGS=--no-half
Windows (PowerShell):
$env:COMMANDLINE_ARGS = "--no-half"
Linux/macOS (Bash or Zsh):
export COMMANDLINE_ARGS=--no-half
Other Unix-like OS (BSD, etc.):
export COMMANDLINE_ARGS=--no-half
(Less Common)
Example for Fish shell:
set -x COMMANDLINE_ARGS --no-half
FOR THE BEST RESULTS USE THE SAMPLER EULER A
The samples are from Forge/Auto1111 img2img though i did most of my testing using openOutpaint because i prefer it over the default plus outpainting is so much easier but i chose to upload the img2img samples because openOutpaint doesn't hold the metadata in the images you generate and just want to be as transparent as possible, whether you use img2img or openOutpaint or even comfyui (although not tested on,i don't see any reason it wouldn't work)..
One thing i forgot to add for those that don't know, you should have your Inpainting conditioning mask strength set to 0.99 which you can do by going to settings and i believe it's in the img2img section, mines been moved with the Quicksettings list... So if you use openOutpaint like i do anytime it brings up that screen to change it to 1 just click no.
This took me over 2 months of trial and error with so many merges and combinations trying to get a ponydiffusion inpainting model because as I'm sure for the ones that have tried to create a ponydiffusion inpainting model the normal way, it will generate images but its basically not usable. It was trained on sdxl so idk why it's the only model that does this and It's not just ponydiffusion but any fine-tuned ponydiffusion model has the same problem but i came across two fine-tuned pd models called
and
that was very very close to the default ponydiffusion model but the problem wasn't nearly as bad but still unusable as it was so these were the steps that i took if you want to for whatever reason recreate it yourself. you'll also need
which is another fine-tuned ponydiffusion model and lastly
AnieaseXL_V2.0
which is the only model that's not trained on ponydiffusion but i used it because its a pretty good model overall with characters it already knows but mostly for the way the inpainting model's color blends into your image its very clean and seamless
This step idk if its mandatory but I'll list it, one test i did was i used the Model Toolkit Extension loaded the normal ponydiffusion_v6 model clicked on the advanced tab and then in the import i selected the t-ponynai3_v3.5 model and selected UNET-XL and clicked save and used this model in the merge i mention below (when i reference using the t-ponynai3 model, it was technically the ponydiffusion_v6XL model with a swapped UNET-XL)
so using the checkpoint merger in Forge/Auto1111 i started by creating the first inpainting model setting Add difference with a strength of 1 checking Save as float16 this will prune the model and create an fp16 model (i tried baking in a vae and not baking one in and since i use a normal sdxl vae by default i didn't really notice a difference)
for
model A = sdxl.inpainting
model B = luminaverse
model C = sdxl base model (specifically sdxl.09vae)
1. luminaverse.inpainting
Now that you have that you'll need to create an aniease inpainting model using the same method above.
2. Aniease.inpainting
then you'll need the 4thTailHentail_03 model but this one doesn't need an inpainting model (i tried creating the inpaint model and the results were just as bad)
3. 4thTailHentail_03
so first i took the luminaverse.inpainting model and the aniease.inpainting model and merged them together using the checkpoint model merger
Model A = luminaverse.inpainting
Model B = aniease.inpainting
Weighted Sum of 0.3
which resulted in luminaverse-x-aniease.inpainting model
then to add some of the pony back and also further help the color problem because the colors for the normal models are fine it's only after you create an inpainting model with them that it ends up the way that it does and i have no idea why...
but i took luminaverse-x-aniease.inpainting and used checkpoint model merger again
Model A = luminaverse-x-aniease.inpainting
Model B = 4thTailHentai_03
Weighted Sum of 0.2
luminaverse-x-aniease-x-4thTailhentai_03.inpainting
The results after this was alright, color was close to normal but it seemed to effect the inpainting model a little too much so i then used checkpoint model merger again
Model A = luminaverse-x-aniease-x-4thTailhentai_03.inpainting
Model B = T-Ponynai3_V3.5.inpainting
Weight Sum = 0.5
luminaverse-x-aniease-x-4thTailhentai_03-x-T-Ponynai3_V3.5.inpainting
so just a 50/50 split between the two which fixed the inpainting side of things but brought back the color issues so this is when i dropped checkpoint model merger and used the Model Mixer Extension
and im only listing the successful merge receipts if i listed every single merge i tried i would have to write a book on the process and no one wants that. and i would just like to stress im not an expert or even knowledgeable on Block Weights or the different things like Normal,Cosine,Inv.Cosine ect. all the tests that lead to this next merge were all trial and error and over 2 months of using different merging techniques and i almost gave up but thought i might try something i still don't fully understand, not even a little bit actually which is the DARE option that the extension model mixer recently added. and i also played around with the Adjust settings that it gives you which look like 
i also used the elemental merge which was also just trial and error im sure someone that actually knows about what these functions do can make a way better inpainting model, one were the artists styles are close to the same because they do still work but are just weaker as i'll show in my xyz plot.. that's just the default settings I'll just post the settings that were in the Metadata but if you just wanted to copy the settings over this part specifically is the "ModelMixer adjust" part which i bolded the important parts but what worked was
Model A = luminaverse-x-aniease-x-4thTailhentai_03-x-T-Ponynai3_V3.5.inpainting (i had a different name for it but it's the same as the Final-TestPonydiffusion_V6XLMix.inpainting model)
Model B = 4thTailHentaiModel_03.safetensors
ModelMixer merge mode b: DARE
ModelMixer alpha b: 0.5
ModelMixer mbw mode b: True
ModelMixer use elemental b: True
ModelMixer calcmode b: Simple Cosine
ModelMixer mbw b: ALL
ModelMixer mbw weights b: "0,0.125,0.15,0.125,0.15,0.2,0.25,0.2,0.15,0.2,0.25,0.125,0.15,0.125,0.15,0.2,0.25,0.2,0.15,0.2"
ModelMixer elemental b: "IN00-IN03 OUT06-OUT08:skip_connection out_layers op in_layers emb_layers:0.25"
ModelMixer model a: Final-TestPonydiffusion_V6XLMix.inpainting.safetensors [9fefd616ba], ModelMixer max models: 3, ModelMixer adjust: "0.05,0.05,0.15,0.1,0.025,-0.01,0.025,-0.15", ModelMixer use model b: True, ModelMixer model b: 4thTailHentaiModel_03.safetensors [a1f96244d3], ModelMixer merge mode b: DARE, ModelMixer alpha b: 0.5, ModelMixer mbw mode b: True, ModelMixer use elemental b: True, ModelMixer calcmode b: Simple Cosine, ModelMixer mbw b: ALL, ModelMixer mbw weights b: "0,0.125,0.15,0.125,0.15,0.2,0.25,0.2,0.15,0.2,0.25,0.125,0.15,0.125,0.15,0.2,0.25,0.2,0.15,0.2", ModelMixer elemental b: "IN00-IN03 OUT06-OUT08:skip_connection out_layers op in_layers emb_layers:0.25", ModelMixer use model c: False, ModelMixer use model d: False
After this merge i ended up with Mix100+Test4FixPonydiffusionXL.inpainting.safetensors
But still not done yet, i had to be sure it was basically still the pony model which has all the artist styles (over 750+ I'll post below) and we all know pony is a perfect nsfw model so that was the first test i did using different artists tags and I'm actually still shocked that for the most part work. I can't say they all work as i couldn't test all of them it would have taken to long, all the ones i did test worked so i figured It's good enough.
Last step - used Supermerger extension to merge 4thTailHentai's Lora Extract with the inpainting model which seemed to really help bring out some of the artist styles that were really hard to tell if they had any effect or not
THIS MODEL IS NOT USED FOR GENERATING IMAGES, FOR THAT USE THE NORMAL PONYDIFFUSION_V6XL MODEL
Sorry I forgot the artist list and i tried to upload the txt file but don't even know if you can on civitai so sorry in advance for having to scroll so much
Steps to Format this list is after it at the bottom. It's only 2 steps using Notepad++ Which I'll Screenshot.
matsunaga_kouyou m-da_s-tarou cutesexyrobutts gweda fumihiko_(fu_mihi_ko) kantoku cle_masahiro kame_(kamepan44231) homare_(fool's_art) 92m personal_ami sciamano240 kuro_kosyou bigrbear samip wisespeak asutora hayabusa fujima_takuya varniskarnis 40hara seraziel kairunoburogu nottytiffy asanagi vahn_yourdoom kase_daiki cluseller iwbitu themaestronoob syhan niliu_chahui ibuki_notsu tama_(tamago) ke-ta scottie_(phantom2) ricegnat ishikei as109 abmayo lvl_(sentrythe2310) mignon chita_(ketchup) nekoda_(maoda) aki99 tab_head mashuu_(neko_no_oyashiro) ndgd doitsuken6 chahan_(fried_rice0614) kukie-nyan beijuu yoako harris_hero missile228 nagishiro_mito mao_lian_(nekokao) puuzaki_puuna yoshizawa_tsubaki yukito_(dreamrider) hu_dako omone_hokoma_agm kfr alp sabamen deaver ouma_tokiichi ruu_(tksymkw) bluefield sukemyon tachimi_(basue) kisaragi_nana sakuraba_hikaru_(loveindog) casino_(casinoep) unfairr null_(nyanpyoun) mochirong koyuki_(kotatsu358) sasaame pottsness miyase_mahiro ergot cowfee re:shimashima shpo tony_guisado roropull everyntrge yuano hater_(hatater) monobe_yuri takunomi z.taiga jeneral camonome nekoya_(liu) sayori_(neko_works) aoi_ogata kaedeko_(kaedelic) himitsu_(hi_mi_tsu_2) misaka_12003-gou neko_neko_koneko ratatatat74 afrobull rokugou_daisuke masami_chie huke ushiyama_ame mathias_leth sakura_chiyo_(konachi000) cirilla_lin xiujia_yihuizi umakuchi_shouyu deel_(rkeg) masabodo chanta_(ayatakaoisii) sendo_(sendrawz) saburou_(hgmg) tateha_(marvelous_grace) nanoless sub-res wlop sogaya ge-b ateoyh idaten93 dearmybrothers pizza_(pizzania_company) ringeko-chan poco_(asahi_age) maccha_(mochancc) sigm@ neropaso tarakanovich putama beeeeen koretsuki_azuma nat_the_lich dearonnus naga_u usagihime watao subachi omoomomo horn/wood minaha_(playjoe2005) huyumitsu hoshizuki_(seigetsu) catstudioinc_(punepuni) efe cube85 starraisins muk_(monsieur) munashi_mujou hara_kenshi 3_(sanyako1) allenes danimaru greatodoggo nibiiro_shizuka kuria_(clear_trip_second) yaman nakamura_rohane mushi024 monaka_curl kanachirou nashidrop nokoru_sora suerte turewindwalker miko_(royal_milk) oopartz_yang ehrrr hinata_(user_rjkt4745) kanpa_(campagne_9) merrytail poligon_(046) nikorashi-ka shishi_juuroku genkung yuzuriha_(active_homing) chups mathew_(srmmk_mce) yukineko1018 shiratsuki_shiori zankuro lds_(dstwins97) akai_sashimi fal_maro skywalker0610 arpeggio_kaga jovejun nekoyashiki_pushio tidsean ogre_(illustogre) lunch_(shin_new) nopetroto songchuan_li yuuji_(yukimimi) mamuru hammer_(sunset_beach) kink_(tortoiseshell) inre_kemomimi shirokitsune merunyaa cenangam asou_(asabu202) rizento ddolbang kana616 hplay meion ohako_(ohako1818) midori_(misuriru8) observerz sheya nanashin_naomi mikozin raptor7 pinkcap kazami_karasu masanaga_(tsukasa) ebonyxh youmu-kun kanon_(kurogane_knights) onono_imoko aramaru namagome_negi welt_(kinsei_koutenkyoku) gustav_(telomere_na) ryota_tentei nito_(nshtntr) ooji_cha darumoon kanta_(kanta_077) chilakkk hotel01 tandohark kinhasu gijang nanana_(nanana_iz) raki_kr arinu mikisai ru_zhai liu_liaoliao oxxo_(dlengur) ao_oni_(onioni-aoi) satou_(3366_s) amashiro_natsuki iroha_(iroha_matsurika) ajifurai e.o. yanagi_yuu pixel_(yuxian) suminagashi ejami kanopan kaigen_1025 ibaraki_natou horokusa_(korai) ojay_tkym radioneet sonri megateru tksand joshua_(shisanli934) shone kokihanada rukako sp123 mafuyu_(chibi21) sy4 shiro_youduki akeyama_kitsune roke_(taikodon) rustle kird_a tanabe_(fueisei) da_mao_banlangen aos etsunami_kumita kamiyoshi_rika nokoppa kibisake zooanime fiz-rot suraimu_(suraimuraimu) kawakami_rokkaku liya itomugi-kun elsi kyou_039 born-to-die massakasama ayaoshiro rushian piyodesu momoko_(momopoco) kiriko_(araragikoyomi) hospital_king suzuno_(bookshelf) distr beudelb yua_(checkmate) mildt bartolomeobari neko_lu_(wofud453) senmura syuri22 syope krirk ayakura_juu pandarou mirin_chikuwa uechin_ewokaku mata_(matasoup) liclac mauve iesupa yan_pai amagi_(amagi626) fusu_(a95101221) kurousagi_tetora gusha_s sasaki_(glass1138) hiyashi_mirano rotte_(1109) sakuramochi_(sakura_frappe) garuku mamemochi kou_mashiro koume_keito sakurato_remi ayahi_4 robot_cat_(robotcatart) dan-98 nama_udon tyakomes enkyo_yuuichirou fh_moya akino_ell ke-su zunusama kamukamu_(ars) peso_(cheese_company) dio_nand terasu_mc damda gyouza_(mhea5724) mabing hanamoto410 miyo_(ranthath) ddddecade kutan toi1et_paper mizuga spam_(spamham4506) silver_(chenwen) shundou_heishirou arim0k0 nuu_(nu-nyu) doribae binggong_asylum kawayabug kasuya_baian tenmu_shinryuusai shika_miso hira_(nanika_no_heya) ransusan satsuki_mayuri kagiyama_shachou masuyama_ryou nakkar juejue syroh ma_draws shirosato laserflip deku_suke yume_shokunin muku_(muku-coffee) moda_(mo_da_3) sao_satoru aohashi_ame paraffin hatoyama_itsuru shikito mintoaisu tsune_(tune) paintrfiend maboroshi_mochi letdie1414 hagure_kedama nero_augustus daidai_ookami momozu_komamochi ini_(inunabe00) ribiadan trimbil kajimakoto xe_(execut3r) auguste primsla hongbaise_raw yana_mori kani_bonara oppaiserothicc gokuu_(acoloredpencil) ganesagi deetamu totoharu_(kujirai_minato) keita_naruzawa tohoyuukarin gaothun komaku_juushoku sincos inyuppo daizu_(melon-lemon) makuran kokuto_(kurousagi1210) hisona_(suaritesumi) gawako kuro_suto_sukii yokaze_(yokajie) kentllaall faustsketcher mirufui ramudia_(lamyun) haseru_(ginku_mh) movik dikko kaptivate hidebo akitsuki_karasu fufumi morino_hon minakami_(flyingman555) furukawa_wanosuke dombear melaton shiraue_yuu muuran mizuki_(kogetsu-tei) less ringouulu ayamy dokomon kochiya_(gothope) slugbox zerocat gins bee_(deadflow) arind_yudha aogisa mayogii tony_taka e_sdss misaki_yuu chirang matsuoka_(mtok_0) otokuyou suzuiro_(chocolatte) fieryonion xiaosamiao himajin_noizu paryi zap-nik luren_max colon_br miike_(992058) kaamin_(mariarose753) eddybird55555 yuccoshi kuya_(hey36253625) harmonist11 melon22 funa_(sakana) na_tarapisu153 ren_(gh) npcpepper shitacemayo tokiani jikuno shino_duka roki_(hirokix) yabby togekk0 fuuzasa tomatolover16 second_heaven sumishi_(sumisi_3) coreytaiyo pumpkinspicelatte k0ng club3 hecha_(01964237) wolflong kazana_(sakuto) nagomian 7saki kutsuki_kai kamehito togemaru34 shounen_(hogehoge) taisa_(kari) min_(120716) timitarcat sakura_ani akuma_(st.takuma) asanogawa_(tutufcc) sarasadou_dan ameto_yuki dokonjou_(odossan) itou_yuuji mcgunngu haryuu_(poetto) shuuko_(s_h_uuko) mitake_eil jun_(aousa0328) kakaricho_dairi yom_(ymayma00ss) kanijiru falken_(yutozin) ougiikun tofuubear yagoro_kusuriya whoru nyong_nyong ebifurya sasaoka_gungu tostantan chamaruk spacezin kazuki_seihou floofsmear momendoufu chiroshiron beleven nori_tamago rottenweiler calm_mashiro nejime poronegi yoshida_hideyuki optionaltypo tanaka_kusao shimada_fumikane koizumo konata_gazel chii_(tsumami_tsumamare) gurajio hukahire0120 ikeda_ruriko chikado nuko_miruku kitazinger supershrimpcakes mozukuzu_(manukedori) hano_haruka suzune_hapinesu techi_(techi35499) moyoron daichi_(tokoya) mochizuki_riku barbariank saeki_touma bori_(3910234) sakuma_hiragi tonchamon_san arnest peroponesosu. tochinoko 258n susukitten chima_q mirukurim ghost_(tama) tamahana ken_(1057168252) mazume hira-san shio_(shiofeifei) keiki8296 akisome_hatsuka shibori_kasu iganashi1 b_nosk101 fir3born azuki_osamitsu ichirugi maru_(maru1625) meng_ziya yamauchi_(conan-comy) shinmon_akika ni_(221) kkato lemtun yatanukikey kuri_dora harunoha shironeko_yuuki miura-n315 oeyama wada_arco kinkymation someyaya toutenkou mokoiscat kurotofu kolshica red_medicine mikan_(manmarumikan) mirufuaa haruwaka_064 niliu_chahuiobaoba_(monkeyix) risui_(suzu_rks) hamico torakichi_888 rariatto_(ganguri) roshin molyb hihiyama_yokikana katsushika_pachi guo582 dodok_(gj77230) sh_(shinh) rimukoro ryuuichi_(f_dragon) soranamae hjz_(artemi) fukahire_(ruinon) sinkai sifeizui sabaku_chitai done_(donezumi) gunnjou_yosio u_u_zan mame_komari ramie_(ramie541) mo_ne rank_alee wing228 meta_meta le_(huanglongen) maturiuta_sorato top-exerou gnai tabayashi hase_neet s16xue steed_(steed_enterprise) hijikawa_arashi shiratama_(hockey) wowoguni kemo_chiharu monaka_(gatinemiku) coldcat. akaoni_(zumt3548) nyantcha kyabekko sawaya_(mizukazu) neko_no_youchuu aoshima naguru_(cyoroama) dog-san yonago_miko daisy_cutter himanemuitoma y2 sznkrs dfra rtil 0_(znanimo) yuuppi shinonome_asu rylaera it_po_(sss_cmy) sunhyun foxykuro shikido_(khf) kujira_pasta micon n0r0 kakult2017 usaka_ray azure-maya kame_(kamepan44231) setakman drlee_lili kei_kei suzume_(simple0091) lily0428 fumei_(mugendai) 360_(taiyo360) ten_no_hoshi kurokan_(kokkyou_oudan) 54hao garasuno k_mugura mikoma_sanagi otomeza_ryuseigun xi_ying fenrir_(fenriluuu) kyogoku-uru yuuki_mix zhazhapao carbohydrate_(asta4282) op_na_yarou kyuu_(plastic_night_q) yunmi_0527 kajuu ichikushi_mojibake ebi_193 amazuyu_tatsuki yukinoshiro onkn_sxkn umyonge_(lkakeu) ya_kexi agahari alexi_oekaki lingshalan yukishiro_haku fujiko_(emnprmn) binware reijing_etrn nanome_to mitama_mudimudi ohno cactus41747280 monety sekiyu_(spartan) chis_(js60216) sakurasaka gomeifuku felutiahime qinglai_haiji tomyoo jytniliu_chahui namiki_(remiter00) pudding_(skymint_028) chanifge sthk iop5509 seika_okawari zounose bbc-chan nanashi_(nlo) foxyreine kagami_toufu matsuda_(matsukichi) sarukana rikuguma you_(noanoamoemoe) cine_(wazd0183) hitobashira_(1746164306) tetsurou_(fe+) itsumizu doraeshi bodhi_wushushenghua 463_jun amisu orchid_(orukido) dior-zi kaleka kaguura_(kagu) sumiyao_(amam) xing kuraki chintora0201 sora_mame_(princess_0sora) wlper dishwasher1910 umberblack izumi_sai mizore_arius vamjulian mikan_(chipstar182) k@bu cerevevisiae hyouta_(yoneya) nephlite ootsuki_wataru houkai_(collapse_illust) g_(genesis1556) tama_(soon32281) zuo_daoxing natsuba002 hews fumitsuki_(minaduki_6) fujisaki_hikari bannouyakunta if_f minazuki_mizu renxzd amino_dopple shironekoban kanzakietc parijennu222
copy the entire list, open up notepad++ (if you don't have it, i recommend getting it. It's free and can be very useful).
After opening Notepad++ paste the entire text and then go to Search tab and click on replace.
For Your Settings
Find What:
Replace With:\n
(it might look empty but you have to put one space in the find what for it to work)
(You Don't Have to worry about the Wrap Around, it's not needed but doesn't hurt anything)
select Extended
and then click replace all and that's it.
If anyone has any problems with this or the model itself please leave a comment
Description
All the score trigger words go in the positive prompt at the beginning.
It just makes the quality better.
All the source trigger words go in the negative prompt at the beginning.
I added them in a whole if you want to use them all or if you wanted to just use 1 or 2 you have the option of just coping whichever you want to use. it's not mandatory but I highly recommend it.
Since its an XL model Clip Skip doesn't matter idk why i see a lot of people mention Clip Skip 2 needed, on XL models Clip Skip doesn't change anything. I highly recommend only using Euler A, the other samplers work but I've mostly tested using Euler A it seems to be the best by far to use with 20-35 steps, I mostly stay around 30 with CFG of 8-9, if i go above 9 i use the DynamicThresholding (CFG-Fix) Extension or if your using forge already integrated
FAQ
Comments (63)
You did it you crazy motherfucker, first working pony inpainting model as far as I am aware.
Could you please explain (or direct to a source that explains it) the purpose of inpainting models? I've been wondering about it since SD1.5 and all people say is that it just works better without much explanation. And in my testing (probably not enough or done in a wrong way) there was not much difference from using a base model for inpainting. I use comfy (in other apps should be the same), I can generate, add a mask, feather it a bit and use the same model for inpainting. With diff-diff it became even easier (I'm sure there are other inpainting tools that can improve the result quality). Maybe I'm missing something important and an inpainting model can work miracles.
@green_anger i personally don't use comfyui i use forge after using auto1111 for the past few years but inpainting models are specifically made to replace things in an image so objects you don't want or just want something else (one example is getting rid of watermarks or bad hands, extra fingers ect.) and if you use something like openOutpaint you can basically expand on your image and if you tried to do that with a normal model it just wouldn't work, inpainting models depending on your settings seamlessly blend into the image so when it's done and you fixed or replaced whatever you can't tell that it was replaced if that makes sense
@green_anger this seems like a good guide, might want to check it out https://civitai.com/articles/161/basic-inpainting-guide
@HITTRAKKZ Thank you, I'll definitely check the guide and try your model.
@HITTRAKKZ Just quick update. I've had a look at the guide, it's mostly focused on using different inpainting technics in a1111. But there's one thing I noticed in section "Inpainting Models":
An inpainting model is a special type of model that specialized for inpainting. Personally, I haven't seen too much of a benefit when using inpainting model. I have occasionally noticed that inpainting models can connect limbs and clothing noticeably better than a non-inpainting model but I haven't seen too much of a difference in image quality.
This is close to my own impression of inpainting models, hence was my original question. Again, maybe I'm doing something wrong with them, will try to experiment a bit more since you gave a good reason for that by publishing the model.
@green_anger a better test might be to use openOutpaint and first use a normal model to try and expand to your image so that it looks natural and then use the same prompt, seed, steps ect. for the impainting model and compare both and there's mainly two different types i use which is Latent Noise and Original, if we're talking about using original and using a denoising strength of 0.3-0.5 then yea a normal model would have no problem at all inpainting the part you selected but if your using latent noise with a denoising strength of 1 then i highly doubt whatever you generate will be seamless (one example is masking over the entire characters body but keeping the head unmasked, what happens with a normal model is it'll generate a characters body but will almost always be scaled either bigger or smaller and the rest gets filled in with whatever colors or objects that don't go with the space that its given but if you then used an inpainting model you'll more than likely get a body that's perfectly scaled and connected to the characters head with just the clothing swapped/changed. I'm far from an expert on ML but this has been my experience with using inpainting models vs normal models to inpaint/outpaint
I got an error in IMG2IMG inpaint Automatic1111 1.9.3 "RuntimeError: Input type (float) and bias type (struct c10::Half) should be the same"
@DarkMaster13 i use forge but i dont see why they would be any different.. ill try it with auto1111
what command args are you using?, i would restart the webui and run auto1111 with these
set TF_ENABLE_ONEDNN_OPTS=0
set COMMANDLINE_ARGS=--allow-code --api --enable-insecure-extension-access --administrator --theme dark --disable-nan-check --cors-allow-origins=https://zero01101.github.io --cors-allow-origins-regex=https://.*\.github\.io$ --disable-safe-unpickle --timeout-keep-alive 240 --opt-sdp-attention
and test this model first dont run any other model before running this one to see if it works or if you still get that issue..
ChatGPT's answer
The error message "RuntimeError: Input type (float) and bias type (struct c10::Half) should be the same" typically occurs in PyTorch when there is a mismatch between the data types of tensors being used in a computation.
In this case, it seems that you're trying to perform an operation that involves tensors of different data types: one tensor is of type float and another tensor is of type half (half-precision floating-point format).
To resolve this issue, you need to ensure that the data types of the tensors involved in the computation are compatible. You can do this by either converting the tensors to the same data type before the operation or by ensuring that tensors of the same data type are used throughout your computation.
For example, if you have tensors tensor_float and tensor_half, you can convert tensor_half to float before performing the operation:
i tested it with automatic1111 now and it works as expected to, though if you haven't tried forge yet, i would switch over, forge kept everything that was good about auto1111 and then added some extras like integrated extensions for controlnet,freeu,CFG scheduler, more samplers and the biggest factor.. generation speed.. i can generate 8 images faster than i can a batch of 4 in auto1111 because my vram in forge uses a lot less
I quickly tried it in Forge, there is no error, but it doesn't work either, just a gray spot.
dreamshaper_8Inpainting
juggerxlInpaint_juggerInpaintV8
work fine.
If I figure out what the problem is, I'll write here.
I set
set TF_ENABLE_ONEDNN_OPTS=0
set COMMANDLINE_ARGS=--allow-code --api --enable-insecure-extension-access --administrator --theme dark --disable-nan-check --cors-allow-origins=https://zero01101.github.io --cors-allow-origins-regex=https://.*\.github\.io$ --disable-safe-unpickle --timeout-keep-alive 240 --opt-sdp-attention
Error gone, but I get a black spot in the inpaint area
@DarkMaster13 ok if your commandline args are the same then we can rule that out, it has to be something with your settings.. what I'll usually do is create a new virtual environment and clone the repo again and just symlink the model and outputs directory, if nothing else works you could do that just to see that it works because i know it's not the model but with how many different settings there are in the web ui i wouldn't know where to even start. You mentioned black images which you can get if you have the vae settings wrong like encoding to full and decoding Taesd, (At least that became a problem for me at one point).. Mine are both set to Taesd with all 3 options unchecked like this
Selected VAE overrides per-model preferences
(you can set per-model VAE either by editing user metadata for checkpoints, or by making the VAE have same name as checkpoint)
Automatically convert VAE to bfloat16
(triggers when a tensor with NaNs is produced in VAE; disabling the option in this case will result in a black square image; if enabled, overrides the option below)
Automatically revert VAE to 32-bit floats
(triggers when a tensor with NaNs is produced in VAE; disabling the option in this case will result in a black square image)
VAE type for encode
(method to encode image to latent (use in img2img, hires-fix or inpaint mask))
TAESD
VAE type for decode
(method to decode latent to image)
TAESD
here's a link to download my config.json file which would let you see exactly what's different
https://jumpshare.com/v/SI5r9sA2HUIoDNpFTkr8
Goodluck, Hope you get whatever it is resolved!
Thank you man. Will play with it on weekend
I fixed this error with adding "--no-half-vae" to set COMMANDLINE_ARGS in webui-user.bat
@DarkMaster13 Glad to hear it got resolved
After trying this out, I noticed the generated images tend to be very messy/blurry in the inpainted area, which reminds me of some of the issues of base ponyXL had. Would it be possible for you to make similar inpainitng models based on some of the other more used mixes such as autismix or more realistic variants like crystalclear/ponyrealism?
@Pupper which models do you suggest? ive already tried autismix though, and it has the same issue as pony but i haven't tried any of the realistic models
@HITTRAKKZ That's weird, among the more 2d-oriented models, autismix seems to be the go-to model for generating without the weird fried/grainy look, perhaps it's a result of trying to create an inpainting model from mixing with other models. Either way, my 2 cents regarding more realistic looking models would be Everclear PNY and Pony Realism, these two seem to be the go-to ones people use.
@Pupper i don't know what it is i just know it's a ponydiffusion problem because i went through the whole list of pony fine-tuned models on civitai and made inpainting models of each one (excluding the realistic models because i only generate anime type images) but every single one was bad, some better than others but still unusable but realistic models might be different, i'll try them both and get back to you, if they have no issue though i'll just upload them. also i didn't mention this but with this inpainting model some images blend in perfectly and some have a hard time and still look like the color is a bit off its really strange and don't know what would cause this so I'm still looking to better this model, hopefully pd7 doesn't have this issue but i bet it will and i really wish the creator of pd would create an inpainting model, since he trained it i would think he would know what is causing this problem
I don't know why but i get Gray Outpaintings,even tho i tried everything,do i have to add something to the commandline args or what?
@Santaonholidays are you using a vae? by grey do you mean the color looks faded and almost has no color? if that is the case then it's most likely because there's no vae being used, there's not a vae baked in with this model so you'll have to use one, i recommend the normal sdxl vae which i also posted so you can download it straight from this page
@HITTRAKKZ Hello author, I can use this paint model perfectly before updating the webui and comfyui. In the past few days, when I update to the latest version, the resulting images are all noisy
@fengyun321 i don't use comfyui but im on the latest auto1111 1.9.3 and running the latest forge update as well and it works as it's always worked, i know nothing about comfyui but i don't see why it wouldn't work the same, i guess the only solution i can give you is to try going back to the git commit when the model was working and test to see if the update is indeed the problem. (though i can't think of any reason why an update would mess with the model, if you can use other xl inpainting models, then this would also work. so first if you haven't already you might wanna test with another xl inpainting model and see if you get any problems, also i don't know what sampler you use but i would recommend only using euler a but if your still getting issues here's a detailed answer from chatgpt so you can use git checkout to revert back to the commit when it was working
Ah, I see! If you want to view the updates (i.e., all actions including checkouts, rebases, resets, etc.) that you've performed in your local repository, you should use git reflog. The reflog shows you the history of all changes made to the tip of branches and other references in your repository, including actions like git pull, git merge, git rebase, and even individual commits.
Here’s how to use git reflog to view all the updates you've performed since cloning the repository:
1. **Open your terminal or command prompt**:
- On Windows, you can use Git Bash or Command Prompt.
- On macOS or Linux, use the built-in terminal.
2. **Navigate to your Git repository**:
- Use the cd command to change the directory to the repository you cloned. For example:
```bash
cd path/to/your-repo
```
- Replace path/to/your-repo with the actual path to your repository.
3. **Run git reflog**:
- To view all actions you've performed, simply run:
```bash
git reflog
```
### Understanding git reflog
The git reflog command provides a list of all reference updates, which can include:
- Commits
- Checkouts
- Rebases
- Merges
- Pulls
- Resets
Each entry in the reflog includes:
- A reference number (e.g., HEAD@{0}, HEAD@{1}, etc.).
- The commit hash.
- The action performed.
- The timestamp.
- The user who performed the action.
### Example of Using git reflog
Let’s say you want to see the list of all updates since you cloned the repository. Here’s an example of what the output might look like:
```plaintext
a1b2c3d HEAD@{0}: commit: Fixed bug in feature X
b2c3d4e HEAD@{1}: checkout: moving from main to feature-branch
c3d4e5f HEAD@{2}: rebase finished: returning to main
d4e5f6g HEAD@{3}: pull: Fast-forward
e5f6g7h HEAD@{4}: checkout: moving from feature-branch to main
f6g7h8i HEAD@{5}: commit: Added unit tests
```
### Filtering git reflog
If you want to filter the reflog to find specific actions, you can use the grep command (available on Unix-based systems and Git Bash on Windows):
1. **Filtering by action (e.g., checkout)**:
```bash
git reflog | grep checkout
```
- This will show only the entries related to checkouts.
2. **Filtering by commit**:
```bash
git reflog | grep commit
```
- This will show only the entries related to commits you made.
3. **Combining with other options**:
- You can combine git reflog with other commands for more specific queries. For example, to find commits on a specific branch:
```bash
git reflog show main
```
### Conclusion
Using git reflog is an effective way to see a history of all actions performed in your local repository, including updates and changes. This is especially useful for tracking what you've done since cloning the repository, as it includes all reference changes, not just the commits you've authored.
or you can just check git log if you know the date of the previous commit you were on you can just go through that list, im not to familiar with git myself but im sure theres an option to revert back to your previous commit but using git reflog or git log should work just fine. sometimes i'll have to use git stash before git checkout will work
Gray Outpaintings - i fixed with selecting VAE for SDXL "sdxl_vae.safetensors"
Thank you very much and congratulations, I have tested the model and it works wonderfully, at least on Automatic1111.
Finally we can touch up those details in our images that until now seemed immutable in Pony.
@Spider_Ham Thanks for your feedback, I'm glad it works for you, I'm still working to make it better though but for the most part I'm happy with the results
Does it work in Fooocus for anyone? When I try to run it, ater it prepared everything for generation (load models, etc), generation stopping without even starting, with error
AttributeError: 'NoneType' object has no attribute 'shape'
@Lera123 I personally haven't tested it in fooocus because I heard you can just use the normal ponyxl model and use the fooocus patch which allows you to use any normal xl model as an inpainting model but I can check it out and see if I get any issues
@HITTRAKKZ what is the patch you are reffering to? available for download somewhere?
@www48 I recommend you check this out, it has the model to download but more importantly information on what it does and how it works https://github.com/Acly/comfyui-inpaint-nodes?tab=readme-ov-file
RuntimeError: Error(s) in loading state_dict for DiffusionEngine: size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).
@453453453253253 can you give me a little bit more information like which webui your using, if you've used other xl inpainting models ect. that message doesn't really tell me anything that i can help with
---
*** Error completing request
*** Arguments: ('task(vmvg3u7udjngdl5)', <gradio.routes.Request object at 0x000001DF33B5D720>, 3, '1 girl', '', [], None, None, {'image': <PIL.Image.Image image mode=RGBA size=1080x1620 at 0x1DF33B5C220>, 'mask': <PIL.Image.Image image mode=RGB size=1080x1620 at 0x1DF33B5CC40>}, <PIL.Image.Image image mode=RGB size=1080x1620 at 0x1DF33B5C250>, <PIL.Image.Image image mode=RGB size=1080x1620 at 0x1DF33B53DF0>, None, None, 4, 0, 1, 1, 1, 7, 1.5, 0.75, 0.0, 512, 512, 1, 0, 0, 32, 0, '', '', '', [], False, [], '', 0, False, 1, 0.5, 4, 0, 0.5, 2, 20, 'DPM++ 2M', 'Automatic', False, '', 0.8, -1, False, -1, 0, 0, 0, False, False, {'ad_model': 'face_yolov8n.pt', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M', 'ad_scheduler': 'Use same scheduler', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M', 'ad_scheduler': 'Use same scheduler', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 'DemoFusion', False, 128, 64, 4, 2, False, 10, 1, 1, 64, False, True, 3, 1, 1, True, 0.85, 0.6, 4, False, False, 2048, 128, True, True, True, False, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, True, 'MEAN', 'AD', 1, UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), 'NONE:0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0\nALL:1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1\nINS:1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0\nIND:1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0\nINALL:1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0\nMIDD:1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0\nOUTD:1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0\nOUTS:1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1\nOUTALL:1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1\nALL0.5:0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', True, 0, 'values', '0,0.25,0.5,0.75,1', 'Block ID', 'IN05-OUT05', 'none', '', '0.5,1', 'BASE,IN00,IN01,IN02,IN03,IN04,IN05,IN06,IN07,IN08,IN09,IN10,IN11,M00,OUT00,OUT01,OUT02,OUT03,OUT04,OUT05,OUT06,OUT07,OUT08,OUT09,OUT10,OUT11', 1.0, 'black', '20', False, 'ATTNDEEPON:IN05-OUT05:attn:1\n\nATTNDEEPOFF:IN05-OUT05:attn:0\n\nPROJDEEPOFF:IN05-OUT05:proj:0\n\nXYZ:::1', False, False, False, False, 0, None, [], 0, False, [], [], False, 0, 1, False, False, 0, None, [], -2, False, [], False, 0, None, None, '* CFG Scale should be 2 or lower.', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, 'start', '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, 50, '<p style="margin-bottom:0.75em">Will upscale the image depending on the selected target size type</p>', 512, 0, 8, 32, 64, 0.35, 32, 0, True, 0, False, 8, 0, 0, 2048, 2048, 2, 'NONE:0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0\nALL:1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1\nINS:1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0\nIND:1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0\nINALL:1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0\nMIDD:1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0\nOUTD:1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0\nOUTS:1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1\nOUTALL:1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1\nALL0.5:0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', True, 0, 'values', '0,0.25,0.5,0.75,1', 'Block ID', 'IN05-OUT05', 'none', '', '0.5,1', 'BASE,IN00,IN01,IN02,IN03,IN04,IN05,IN06,IN07,IN08,IN09,IN10,IN11,M00,OUT00,OUT01,OUT02,OUT03,OUT04,OUT05,OUT06,OUT07,OUT08,OUT09,OUT10,OUT11', 1.0, 'black', '20', False, 'ATTNDEEPON:IN05-OUT05:attn:1\n\nATTNDEEPOFF:IN05-OUT05:attn:0\n\nPROJDEEPOFF:IN05-OUT05:proj:0\n\nXYZ:::1', False, False) {}
Traceback (most recent call last):
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\call_queue.py", line 57, in f
res = list(func(*args, **kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\call_queue.py", line 36, in f
res = func(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\img2img.py", line 232, in img2img
processed = process_images(p)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\processing.py", line 845, in process_images
res = process_images_inner(p)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 59, in processing_process_images_hijack
return getattr(processing, '__controlnet_original_process_images_inner')(p, args, *kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\processing.py", line 981, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\processing.py", line 1741, in sample
samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_kdiffusion.py", line 172, in sample_img2img
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_common.py", line 272, in launch_sampling
return func()
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_kdiffusion.py", line 172, in <lambda>
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
denoised = model(x, sigmas[i] s_in, *extra_args)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_cfg_denoiser.py", line 237, in forward
x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input c_in, self.sigma_to_t(sigma), *kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_models_xl.py", line 44, in apply_model
return self.model(x, t, cond)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_hijack_utils.py", line 18, in <lambda>
setattr(resolved_obj, func_path[-1], lambda args, *kwargs: self(*args, **kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_hijack_utils.py", line 32, in call
return self.__orig_func(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\generative-models\sgm\modules\diffusionmodules\wrappers.py", line 28, in forward
return self.diffusion_model(
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_unet.py", line 91, in UNetModel_forward
return original_forward(self, x, timesteps, context, args, *kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\generative-models\sgm\modules\diffusionmodules\openaimodel.py", line 993, in forward
h = module(h, emb, context)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\generative-models\sgm\modules\diffusionmodules\openaimodel.py", line 102, in forward
x = layer(x)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\extensions-builtin\Lora\networks.py", line 518, in network_Conv2d_forward
return originals.Conv2d_forward(self, input)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\conv.py", line 456, in convforward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [320, 9, 3, 3], expected input[2, 4, 64, 64] to have 9 channels, but got 4 channels instead
@marlonmelodixxh67311
1. Check Recent Changes
Settings: Review any recent changes you may have made to the inpainting or img2img settings. Even a small tweak can sometimes lead to errors.
Extensions: If you've added or updated any extensions recently, such as ControlNet, they could be causing a conflict. Try disabling the most recent extensions and running the model again.
2. Inspect the ControlNet Extension
The traceback mentions the sd-webui-controlnet extension and a specific file (batch_hijack.py). Ensure this extension is up-to-date, and check if any recent updates might have introduced bugs or compatibility issues.
Disable ControlNet: As a test, disable the ControlNet extension and try running the model again to see if the error persists.
3. Review Image and Mask Inputs
Ensure that the images and masks you're using are properly formatted and compatible with the model. The image inputs should match the expected dimensions and color modes (e.g., RGBA, RGB).
Check Image Size: Sometimes, issues can arise if the image size is non-standard or if the mask size does not match the input image. Rescale the images to a more standard resolution (e.g., 512x512) and try again.
4. Verify the Model and Configuration
Model Checkpoint: Double-check that the model checkpoint and configuration files are correctly loaded. If you have multiple checkpoints, ensure the correct one is selected.
Reset Configuration: Consider resetting the configuration settings for the img2img model to their defaults and running the model with basic parameters to see if it works.
5. Reinstall or Update Dependencies
Update Environment: Ensure all dependencies are up-to-date. Sometimes, issues can arise if packages like torch, PIL, or others are outdated or incompatible with the current version of the model.
Reinstall Extensions: If you suspect a corrupted file, try reinstalling or updating the extension, especially the ones mentioned in the traceback.
6. Look at the Specific Error Line
The error points to specific lines in img2img.py and processing.py. If you're comfortable, you can inspect these lines in the source code to get a better understanding of what's going wrong. Alternatively, you can check online forums or the GitHub repository for similar issues reported by others.
7. Run a Simplified Test
Try running a very basic img2img task with minimal settings, no additional extensions, and standard images to isolate the problem.
8. Check for Debug Logs
Look for more detailed logs or enable debugging to get more information about what is causing the error. This can often provide clues that aren't immediately visible in the error message.
@marlonmelodixxh67311 also try using Euler A and Schedule type Automatic or Align Your Steps, theres been a bunch of times were ive got errors like this just for using a different sampler, scheduler type ect.
I had this working a month ago and even made custom inpainting models with it but my drive died, trying it again the base model gives me images that look like noise and when i try to merge custom models they give me the same result.
https://civitai.com/posts/3321799
Not sure what the issue is, I did everything the same way I did before.
@volkin555 what are you using? what webui and what are your settings? if you don't mind sending me your config information I can take a look but the model hasn't changed so what I recommend is if you use automatic1111/forge is to git clone the repo and make a new directory for stable diffusion but still keep the original one and just create symbolic links for your models and outputs directory so that way you won't have to move anything and it won't take up anymore space but you'll be able to use both if or when you find what the issue was. If your using windows I recommend downloading this https://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html .. theres a 32 bit and 64 bit version and it allows you to easily create symbolic links from the context menu by right clicking on whatever directory you want to copy and clicking on "Pick Link Source" and then you go to the new stable diffusion directory you just cloned and then right click inside the directory and you'll see an option "Drop as" and when you hover over it you'll see all the options it has but you just choose Symbolic Link which is the first option and its that easy. You'll want to delete the models directory that came with the cloned repo before you do this since you can't have 2 directories with the same name but this process will at least fix your problem because I've had a lot of problems in the past where I had to do this after spending days trying to find the problem and wasted a lot of time (Not related to this but a fresh install will still fix it) that is if thats the only issue your having and you can still use xl models just fine but if your having problems with both id recommend doing the same as above but also creating a new virtual environment. I use conda since i don't use the typical .bat file to start stable diffusion but instead i create my own virtual environment using this command
conda create -n forge python=3.10
and then once its done ill activate the venv with conda activate forge and once activated ill use this command
pip install torch==2.2.2 torchaudio==2.2.2 torchvision==0.17.2 --extra-index-url https://download.pytorch.org/whl/cu121
and once thats done ill use this to set my commandline args
set TF_ENABLE_ONEDNN_OPTS=0
set COMMANDLINE_ARGS=--allow-code --api --enable-insecure-extension-access --administrator --theme dark --disable-nan-check --cors-allow-origins=https://zero01101.github.io --cors-allow-origins-regex=https://.*\.github\.io$ --disable-safe-unpickle --timeout-keep-alive 240 --opt-sdp-attention
and then i always start the webui using
python launch.py
and after I've ran it once and everything's installed ill change my commandline args to this
set TF_ENABLE_ONEDNN_OPTS=0
set COMMANDLINE_ARGS=--allow-code --api --enable-insecure-extension-access --administrator --theme dark --disable-nan-check --cors-allow-origins=https://zero01101.github.io --cors-allow-origins-regex=https://.*\.github\.io$ --disable-safe-unpickle --timeout-keep-alive 240 --opt-sdp-attention --skip-prepare-environment
adding the skip-prepare-environment speeds up the launch a lot
I hope this helps
@HITTRAKKZ I figured it out, for some reason downloading it through the CivitAI Browser+ extension on Automatic1111 corrupts the model, I tried multiple times. Downloading it through civitai directly works fine.
@volkin555 thats odd but glad to hear its resolved
Does anyone know why my inpainted area is not very respectful of the prompt and gives low quality output?
@dasdasqqqq for the quality part i can only assume it's because its an inpainting model which i only use to fix small mistakes like hands, feet etc. but the prompt part i have no idea, id have to see what your prompt was and what was generated
this is almost there, someone should train it a little bit to inpaint so it stabilises a little since there are still artifacts. I might do it myself, but I need training data
@SomeName69 how big of a dataset?
honestly not that big, 500-2000 images should be enough since it's almost there. All it needs it's some stabilization with it's new weights. You can even freeze all weights except the first and last block for less vram consumption
@HITTRAKKZ I gave it a thought and I will train it next weekend
@SomeName69 to be honest this is something i thought about before but ive only ever trained lora's and being that it's an inpainting model i assume you have to fine tune it using a structure like this
dataset/ ├── images/ │ ├── image1.png │ ├── image2.png │ └── ... ├── masks/ │ ├── mask1.png │ ├── mask2.png │ └── ... ├── annotations/ │ ├── image1.json │ ├── image2.json │ └── ... └── metadata/ ├── image1.txt ├── image2.txt └── ...
do you know what software can create bulk masks from a directory? i can help you with the dataset part but i don't know much about training
@HITTRAKKZ I personally don't. I was planning on creating my own mask generator and then edit a little kohya's script so it takes masks into consideration. It shouldn't be that hard. Please do help me with datasets. If you want you can dm me your discord
@SomeName69 my discord 彡HiTZ#6997
@SomeName69 @HITTRAKKZ i'd love to see the update when you gents get toghether
sorry，but I fail to use this model
---
*** Error completing request
*** Arguments: ('task(vmvg3u7udjngdl5)', <gradio.routes.Request object at 0x000001DF33B5D720>, 3, '1 girl,', '', [], None, None, {'image': <PIL.Image.Image image mode=RGBA size=1080x1620 at 0x1DF33B5C220>, 'mask': <PIL.Image.Image image mode=RGB size=1080x1620 at 0x1DF33B5CC40>}, <PIL.Image.Image image mode=RGB size=1080x1620 at 0x1DF33B5C250>, <PIL.Image.Image image mode=RGB size=1080x1620 at 0x1DF33B53DF0>, None, None, 4, 0, 1, 1, 1, 7, 1.5, 0.75, 0.0, 512, 512, 1, 0, 0, 32, 0, '', '', '', [], False, [], '', 0, False, 1, 0.5, 4, 0, 0.5, 2, 20, 'DPM++ 2M', 'Automatic', False, '', 0.8, -1, False, -1, 0, 0, 0, False, False, {'ad_model': 'face_yolov8n.pt', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M', 'ad_scheduler': 'Use same scheduler', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M', 'ad_scheduler': 'Use same scheduler', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 'DemoFusion', False, 128, 64, 4, 2, False, 10, 1, 1, 64, False, True, 3, 1, 1, True, 0.85, 0.6, 4, False, False, 2048, 128, True, True, True, False, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, True, 'MEAN', 'AD', 1, UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=True, hr_option='Both', save_detected_map=True, advanced_weighting=None, ipadapter_input=None), 'NONE:0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0\nALL:1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1\nINS:1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0\nIND:1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0\nINALL:1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0\nMIDD:1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0\nOUTD:1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0\nOUTS:1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1\nOUTALL:1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1\nALL0.5:0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', True, 0, 'values', '0,0.25,0.5,0.75,1', 'Block ID', 'IN05-OUT05', 'none', '', '0.5,1', 'BASE,IN00,IN01,IN02,IN03,IN04,IN05,IN06,IN07,IN08,IN09,IN10,IN11,M00,OUT00,OUT01,OUT02,OUT03,OUT04,OUT05,OUT06,OUT07,OUT08,OUT09,OUT10,OUT11', 1.0, 'black', '20', False, 'ATTNDEEPON:IN05-OUT05:attn:1\n\nATTNDEEPOFF:IN05-OUT05:attn:0\n\nPROJDEEPOFF:IN05-OUT05:proj:0\n\nXYZ:::1', False, False, False, False, 0, None, [], 0, False, [], [], False, 0, 1, False, False, 0, None, [], -2, False, [], False, 0, None, None, '* CFG Scale should be 2 or lower.', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, 'start', '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, 50, '<p style="margin-bottom:0.75em">Will upscale the image depending on the selected target size type</p>', 512, 0, 8, 32, 64, 0.35, 32, 0, True, 0, False, 8, 0, 0, 2048, 2048, 2, 'NONE:0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0\nALL:1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1\nINS:1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0\nIND:1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0\nINALL:1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0\nMIDD:1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0\nOUTD:1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0\nOUTS:1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1\nOUTALL:1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1\nALL0.5:0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', True, 0, 'values', '0,0.25,0.5,0.75,1', 'Block ID', 'IN05-OUT05', 'none', '', '0.5,1', 'BASE,IN00,IN01,IN02,IN03,IN04,IN05,IN06,IN07,IN08,IN09,IN10,IN11,M00,OUT00,OUT01,OUT02,OUT03,OUT04,OUT05,OUT06,OUT07,OUT08,OUT09,OUT10,OUT11', 1.0, 'black', '20', False, 'ATTNDEEPON:IN05-OUT05:attn:1\n\nATTNDEEPOFF:IN05-OUT05:attn:0\n\nPROJDEEPOFF:IN05-OUT05:proj:0\n\nXYZ:::1', False, False) {}
Traceback (most recent call last):
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\call_queue.py", line 57, in f
res = list(func(*args, **kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\call_queue.py", line 36, in f
res = func(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\img2img.py", line 232, in img2img
processed = process_images(p)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\processing.py", line 845, in process_images
res = process_images_inner(p)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 59, in processing_process_images_hijack
return getattr(processing, '__controlnet_original_process_images_inner')(p, args, *kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\processing.py", line 981, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\processing.py", line 1741, in sample
samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_kdiffusion.py", line 172, in sample_img2img
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_common.py", line 272, in launch_sampling
return func()
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_kdiffusion.py", line 172, in <lambda>
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
denoised = model(x, sigmas[i] s_in, *extra_args)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_samplers_cfg_denoiser.py", line 237, in forward
x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input c_in, self.sigma_to_t(sigma), *kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_models_xl.py", line 44, in apply_model
return self.model(x, t, cond)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_hijack_utils.py", line 18, in <lambda>
setattr(resolved_obj, func_path[-1], lambda args, *kwargs: self(*args, **kwargs))
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_hijack_utils.py", line 32, in call
return self.__orig_func(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\generative-models\sgm\modules\diffusionmodules\wrappers.py", line 28, in forward
return self.diffusion_model(
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\modules\sd_unet.py", line 91, in UNetModel_forward
return original_forward(self, x, timesteps, context, args, *kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\generative-models\sgm\modules\diffusionmodules\openaimodel.py", line 993, in forward
h = module(h, emb, context)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\repositories\generative-models\sgm\modules\diffusionmodules\openaimodel.py", line 102, in forward
x = layer(x)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1511, in wrappedcall_impl
return self._call_impl(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\module.py", line 1520, in callimpl
return forward_call(*args, **kwargs)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\extensions-builtin\Lora\networks.py", line 518, in network_Conv2d_forward
return originals.Conv2d_forward(self, input)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
File "E:\sd-webui-aki\sd-webui-aki-v4.8\python\lib\site-packages\torch\nn\modules\conv.py", line 456, in convforward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [320, 9, 3, 3], expected input[2, 4, 64, 64] to have 9 channels, but got 4 channels instead
@marlonmelodixxh67311 what gpu are you using i forgot to ask, amd or nvidia.. i have an nvidia card but mostly everyone with issues has been amd users which needed to add --no-half to they're command line args because amd has a problem with fp16 half precision but if you have an nvidia card then i recommend creating a new venv and starting fresh to see it works and i don't know how many extensions you have or your settings but there's to many things that can cause issues to any inpainting model not just this one. its an sdxl inpainting model fp16 so it's not a 10gb file and i should have baked in a vae seeing many people don't use one apparently so next upload i'll make sure to do that but hope your problem is resolved.. i know creating a new venv and cloning the repo is a lot but its a 100% way to fix whatever issue it is that's conflicting with inpainting
@marlonmelodixxh67311 i dropped your error in to ChatGPT and this is what it recommends, id start at 2. because it is an xl inpainting model and all the people it works for verifies that
The error you're encountering while running the Stable Diffusion XL inpainting model suggests an issue with the arguments or settings you're using, particularly with image processing or model configuration. To help you resolve it, let's go through a few troubleshooting steps:
1. Check Model Compatibility
Ensure that the inpainting model you're using is compatible with Stable Diffusion XL. Some models are tailored for certain tasks, and an incompatible model can cause errors like this.
If you have other models, try switching to a different inpainting model and see if the error persists.
2. Verify Image Format and Mask
The input image should be in the correct format (RGBA, RGB), and the dimensions of the image and mask should match. The error shows that your image is in RGBA mode while the mask is in RGB mode. Ensure both are in compatible formats.
You can try converting the mask to RGBA or the image to RGB before processing.
3. Check Resolution and Inpainting Area
Confirm that the resolution of the image (1080x1620) is supported by the inpainting model. Some models may struggle with larger dimensions.
You can also try resizing the image to a more standard dimension like 512x512 or 1024x1024.
4. Update Stable Diffusion and Extensions
Make sure you're using the latest versions of Stable Diffusion and any extensions or plugins you have installed. Bugs and compatibility issues are often fixed in newer updates.
5. Simplify Your Inputs
It looks like you're passing a large number of settings and parameters. To identify the source of the error, try simplifying the configuration:
Prompt: Try using a simple prompt like '1 girl' and remove advanced settings like controlnet, noise multiplier, or guidance controls.Mask and Denoising Strength: Ensure that the mask and inpainting settings are appropriate. You could set the denoising_strength to a lower value, like 0.2, to start.
6. Inspect Logs
Review the console or application logs for more detailed error messages. Sometimes, errors not shown directly in the output will appear in logs, giving you a clearer idea of what went wrong.
7. Test with Default Settings
Reset the inpainting settings to their defaults and try running the process with the basic configuration. If this works, you can gradually reintroduce your custom parameters to pinpoint the issue.
@marlonmelodixxh67311 sorry for 3 replies but i just noticed the 1080x1620 dimensions and 1620 isn't a multiple of 8, 1624 would be so idk if that could be the issue
Great work! it's magnificent. Any plans on improvment on the few artifacts? sometimes I get a slight brlurry-ness.
You are basically amazing. I was just looking all over for something like this! thanks so much for sharing, inpainting is so much better now.
Since im using an AMD card(can afaik only start stable diffusion with --no half), i get RuntimeError: Input type (float) and bias type (c10::Half) should be the same
Is there any way around that?
thanks anyway for the work you put in this, great job
@Rhonokho i have an rtx 3060, never used an amd gpu but it should be the same, the command args that i use are these
set COMMANDLINE_ARGS=--allow-code --api --enable-insecure-extension-access --administrator --theme dark --disable-nan-check --cors-allow-origins=https://zero01101.github.io --cors-allow-origins-regex=https://.*\.github\.io$ --opt-sdp-attention
so this is chatgpt's answer to the problem
It seems that the error your user is encountering relates to the fact that AMD GPUs do not natively support half-precision (also known as FP16) computations as well as NVIDIA GPUs do. Stable Diffusion models are often optimized to run using half-precision because this helps save memory and improve performance, but this optimization requires hardware that supports FP16 computation efficiently, such as NVIDIA's Tensor Cores (which are available in RTX series GPUs).
The Error Explained:
The specific error they encountered:
RuntimeError: Input type (float) and bias type (c10::Half) should be the same
This error occurs because the model is trying to run in FP16 mode (half-precision floating-point), but the AMD card does not handle this well. Their setup is likely trying to mix FP16 and FP32 (full-precision floating-point), which causes a mismatch and results in the error.
Why You Don't Encounter This Issue:
Your NVIDIA GPU likely supports FP16 operations via its Tensor Cores, so you do not need to disable half-precision (--no-half). On NVIDIA cards, this optimization improves speed and lowers VRAM usage without causing errors. That's why the command-line arguments you're using work perfectly for you and others with NVIDIA GPUs.
Solution for AMD Users:
AMD users often need to use the --no-half argument to force the model to run in FP32 precision (full-precision), which avoids the issue but at the cost of using more VRAM and potentially running slower.
For your AMD user, the solution would be to start Stable Diffusion with the --no-half argument:
Command-Line Arguments for AMD GPUs:
Instruct the AMD user to modify their command-line arguments to include --no-half. For example, they could run the Web UI using:
set COMMANDLINE_ARGS=--no-half --allow-code --api --enable-insecure-extension-access --administrator --theme dark --disable-nan-check --cors-allow-origins=https://zero01101.github.io --cors-allow-origins-regex=https://.*\.github\.io$ --opt-sdp-attention
I honestly don't understand how to use this inpaint model.
Like, your instructions is a bit weird and odd.., i'm not sure from where to follow?
Do i really need to do merging model with this model in order to experience the good result?
Can't I just use it like normal inpaint model?
yea i only put the merges for those that wanted to recreate the model but if you just want to use the inpainting model, you can ignore all of that and just download it and load it also make sure to use a vae because i didn't bake one in so if you don't use a vae you'll get images that look faded
@HITTRAKKZ Ah, I see! It makes more sense now.
I'm trying to use your inpainting model for anime screenshots or Megami Magazine images, but I'm not satisfied with the result. Even though I'm using a mask, it's still not 'obedient' enough to do the job properly. I'll try using a VAE and come back to share the result later!