First successfully trained custom variant of PlayGround V2
Use the training version above in Kohya and it should work for you. But I suggest merging the trained data with the original PlaygroundV2 model, as this model is not pure PG.
https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic/blob/main/LICENSE.md
Recommended to use DPM SDE Karras/Exponential, 50 steps and a CFG of 3. But you can try lower step counts and experiment with CFG and other things. I have tested 4-8 steps and had some success even.
I like to use with a denoise of 0.88 on the Sampler.
A few caveats. It functions in place of an SDXL model, however it does not share compatibility directly with LoRA etc. As far as I have tested. They seem to effect the model differently than intended due to the nature of this model having been trained completely from scratch.
I personally think that this model is going to become the new base model for community work, going forwards, as it is a million times more flexible to train and already higher detail than many custom SDXL models.
Also this is technically an alpha release on my part and doesn't necessarily reflect the full degree of outputs the model is capable of.
Description
FAQ
Comments (33)
clip skip 2 and denoise of .88 .... definitely different. I noted that playground had their own VAE available for download. What do you recommend?
So far it seems like there is no real difference, except that, like the original SDXL VAE, The baked vae in the original model doesn't do FP16. So using a VAE from SDXL which does is what I recommend for most people.
They used the same CLIP and VAE as SDXL base. Not even the one that is fixed for fp16. They only trained a new Unet.
@dunkeroni hmm, ok. So they trained their own base, how much different is that from just taking the SDXL base and training it further?
@EricRollei21 It means they can release it under a different license that allow them to demand a commercial licensing fee from Civitai per the "additional commercial terms" in the Playground v2 Community License. They own it, and it is not open source. They allow you to use, modify, and redistribute it.... as long as you aren't a business with more than 1mil users. If you are, then even if you only provide Playground models for free, they can demand licensing payments for the privilege to distribute them for free because "image generation or image editing is a core business or product of Licensee".
As far as I can tell, they released the weights just so that people would make finetunes for them for free, and then Playground has the rights to sell the results on their service while preventing competitors like Civitai.com, Tensor.Art, and all the other generator sites from benefiting in the same way.
@dunkeroni Yeah, I wonder how many people on civitai are even thinking about licensing issues? I wonder how many people are making real money with these?
@EricRollei21 Mostly it's the websites that are making real money with these. I think creators on Tensor can get paid in tips, but I'm not sure on the economy there. I suspect it is minimal.
The real risk is that one day Playground contacts Civitai and says that because of their user count, Civitai is no longer allowed to distribute their model derivatives. No licencing fee offered. Then all of the community-trained Playground models shared here will be suddenly and irrevocably deleted, and the only place to access the best ones would be back on Playground's website by paying for their service. Unrestricted free redistribution is literally the first part of the Open Source Definition. https://opensource.org/osd/
Playground is attempting (and succeeding by the existence of this finetune) to carve out their own closed-source section of an otherwise open-source community by hiding behind the ignorance of users and volunteer model trainers. They want a piece of our pie, and that's a dangerous trend to start.
@dunkeroni I'm not sure if you read the same thing I did, but it clearly stated that the license of the model would only be revoked in such cases as the individual who owns the license, E.G me, runs a commercial enterprise using the model which gains the majority of its profits through onsite generations and those users on site, or perhaps just runs, exceed 1 million.
Forgive me for thinking that it is actually somewhat reasonable in an open source landscape for them to withhold the right to stop individual licensees from running competing businesses without at least being on friendly terms with them.
I may indeed be proven wrong in future, but as far as I am concerned any company that is secure enough to release their technology into the public domain - fully well knowing that the majority of users will have no idea what the difference is between open source offerings, going forwards - is one which is either incredibly smart, or incredibly stupid.
It is in their best interest to be as good as possible to us as a community and as creators etc. because we hold grudges forever. And enough of us know how to code and have equipment that if they do want to screw with us, it's like screwing with Anonymous or 4Chan. But even worse.
Our community as a whole is one I feel will win out over any attempts at subversion, and in my opinion - I'm in it for the progression of the technology and the capabilities of the models. Not for licensing or money etc. At least not at this stage in the game.
It's also a 2-way street. I'm not sure if they realize the implications of revoking a license. While I no longer would benefit from their active/passive consent at the moment the license is revoked, I am no longer beholden to it.
And I, and many others in the open source community came from an online environment which encouraged the free distribution of Intellectual properties without permission from the copyright holder. Now, times have changed in regards to how aware the powers that be are of such practices and we've all matured to create our own Intellectual properties which we now distribute freely for all.
So, while now we are perfectly happy to work within the boundaries of the system, that is likely in large part only up until such times as the system becomes inconvenient.
And such an act from a company would be INCREDIBLY inconvenient. I'm certainly not encouraging any kind of untoward behavior, but as someone who exists on the internet and has his eyes open, I know that it would be an inevitability.
Essentially, if PG aren't careful they will have people actively competing against them, at the very least, and if they do this right - they could be a positive driving force for the community and we all win.
@Triple_Headed_Monkey I can't imagine how the creators of Playground could manage to put the genie back in the bottle. My take is licensing for a lot of open source is to protect the creators from being sued later by someone using their work to do something malicious or bad.
@EricRollei21 Absolutely. At least for now, there is no way of dealing with individuals who choose not to comply.
In the first place, if I only allowed onsite generations, no downloads, how could anyone prove where the model came from? Especially if I hosted it myself.
So they are relying on good faith here. And I think that is a better move than I've seen from a lot of AI companies in the space.
If their company is willing to put faith in our community, I'm willing to put faith in them. Hopefully we all gain something out of this exchange.
Thanks for your thoughtful and cogent replies. I hate to see all discourse on this reduced "kewl, tiddies!" and insulting people who type more than 2 sentences on what is an incredibly complex topic (Reddit). Dumbing stable diffusion down to "press button, get gurl" is a sad waste of the first new creative tool to come along since 3D. I'm enjoying all your models and make sure to heart, star, whatever.
@EricRollei21 Depends on how you define "real". I don't do gens for hire, but I have a client who's AI-curious. I use Auto1111 but just about everything professional goes through Photoshop, then maybe animated in Cinema 4D or After Effects and is only a part of what I do for him. Nothing has ended up as a resource in a commercial app yet. Sometimes it's all just for creative inspiration. A fresh way of looking at things... even just a color scheme. It's like art directing a drunken kid sometimes but that freedom is wonderful. What I've done that's seen the light of day was for free anyway (and now seems ancient) - just some concert visuals for Peter Gabriel's last tour for one song. It all got me a new box with a 4090, and as an aging female former tech dev/mgr no one wants to hire, that's "real" enough for me.
@parallelepipedon New box with 4090 is pretty cool and perhaps higher-profile stuff will get you referrals? Agism in tech seems real, but I see it the other way. I was just at the AI user conference in SF and got to hear stuff like "Photography was invented in 1977" by young CEOs who were sure they understood what product brands really needed. They will be soon looking around for people with real knowledge about how to put together a proper image and layout. Gen AI would be different if it had started with some consult from experienced people first. Like the model would have been trained with vetted image sets, and built with stuff like color managment, depth and perspective control from the beginning.
@EricRollei21 Seriously? Someone though that photography was invented in the 1970s?
... I mean, maybe it is just because I studied animation in depth on the side for my electronic music production degree, but even then I was fully aware that photography began more than a hundred years ago.
https://www.youtube.com/watch?v=HoUKE_sKR7Q
I even did a small, really bad podcasty documentary thing about the subject. lol.
@parallelepipedon <3 I appreciate the support man! Glad you are getting use out of my work!
@Triple_Headed_Monkey I'm sure this CEO had intended to say 'digital' photography, but so much of the craft of photography needed specifically for product photography was also missing most importantly from their product. The lighting alone is an art, and perspective control requires a technical or view camera which no one knows how to use anymore. Impossible to do convincingly in post at the moment. Can do the whole scene in CGI .... maybe with AI in the future, but the lighting control requires the AI to know the depth of the individual objects. Perhaps with a combination of AI and something like Blender? Watched your video - wow - you covered quite a lot of history in that one. Very thorough!
I think is too soon to say this model will replace anything, for now not a lot of quality, etc. HOWEVER, I know it's baby steps but when I saw SDXL I thought other things. In Turbo had the same feeling as gamechanger.
I've been staring at a lot of pixels, and I can say that the density of pixel detail in this branch of the model appears far greater than SDXL. It doesn't have as many issues with architecture and things being warped in the background etc. It seems to me like a solid base.
At the very least it shows that their is a need to create a new base model.
It's not a "gamechanger" per say. But I think that with a bit of community work, this has less built-in problems to be ironed out that the base model does.
@Triple_Headed_Monkey Agree. In my opinion community will move with Demofusion pipeline cause its the same base model but with different pipeline and like an enhanced aug. res. SDXL turbo problem is licensing IMO and with SDXL or this if trainable plus that pipeline things will get forward fast. New model and from scratch is always good, for now looks quite similar to SDXL and we will need to test first like Foooocus did, for example to get our heads about the extent of the power of SDXL.
@LDWorksDavid I very much doubt anyone is going to move over to DemoFusion if it takes 15 minutes for a 4000x4000 image.
That is approximately the length of time it would take to generate an 8k image using existing methods.
@Triple_Headed_Monkey I mean using their pipeline with of course (and time will tell) reduced times. Probably using SDXL Turbo will reduce generation times, LCM and so on. Demofusion now is 3 minutes 2048x2048 on 3090 more or less (as they say, need to test further) but as you already saw with other long times it got reduced a lot. We will see but their enhancement on detail is really cool feature.
I strongly recommend against this becoming a new community standard for training. It is not open source. Read the license. Playground trained their own Unet from scratch so that they could release it under their own license that allows them to demand payment from any commercial enterprise that offers/uses the model (or any fine-tunes or merges that include it). The fees for using it commercially are not explicitly defined and "at their sole discretion".
As a side note, there are ways I have successfully merged it with SDXL base and possibly make it compatible with LoRA, but I don't want to get into that here.
Sure. I actually agree with you here. I've not used any of the weights that they released to fine-tune I just ran this thing through Kohya myself.
The model is great, unique and fresh. And Honestly I think that competition is a good thing on all fronts when it comes to their being more quality options available.
In fact I had to replace the text encoder on the original PlaygroundV2 model with the Text encoders from the base SDXL model in order to get Kohya to train it in the first place.
And then afterwards I merged the trained data back into the original model.
But anyway. In terms of licensing - I'm not sure if you are aware that Stability AI are doing something similar with their models also. They intend to make it so anyone who hits a million dollars revenue has to join their members club for a subscription fee in order to retain commercial licensing.
So we're kind of up shit creek without a paddle there.
I'm actually right now looking at Playground as a proof of concept more than anything. This proves that the base model is just downright in need of replacement - but that the technology is sound. And it feels like seeing the lack of certain glaring issues that have been present from the SDXL base model in all of the custom models is a breath of fresh air for me as a model creator.
What we need is a truly open source movement. One which actually is open source. Perhaps WurstchenV3 will be a good candidate.
I will make my own custom version of that when it releases too. As that will release with training support and even a set of nodes for ComfyUI integration.
@Triple_Headed_Monkey glad we're on the same page. Stability's new restrictions are also worrisome, but at least it's only on their Turbo models so far, and they did go as far as to explain how they finetuned to get there. SDXL and any modifications people make to mimic turbo or other features are not commercially restricted, afaik.
As for the model itself, I found Playground v2 to be oddly repetitive in strange ways (e.g. there is only one "old man" face, "haunted mansion" rooms always have the same layout and angled parquet floor, women with tattoos usually show up with neon signs). However, it does use color significantly better than SDXL base does. I do like the fact that someone besides Stability with a vested interest in furthering the technology is spending the resources to make something new.
I'm sure Playground claims to have the best of intentions and doesn't plan to go after creators of models, but when you have a model on Civitai you as the creator are not distributing it. You are giving a copy to Civitai with grants them the same license to distribute it. Whether or not Playground currently plans on exploiting that situation in the future doesn't change the fact that it gives them a dangerous amount of power going forward.
SDXL normal models will be the standard. About LCM unsure the target but XL normal 100% sure will be the way to go... and ei. 1.5 IS NOT DEAD. I mean, you can do anything (or almost anything) in 1.5 too. We should be looking too at license of Kandinsky, PixArt (superior to SDXL theorically), etc.
THM, you are the master! The model is very good, not perfect, but very very good, very artful. Thank you so much for the new discovery!
How did you get it to train? I keep getting an error in koyha at the unet startup step.
I'm uploading the version of the model that I trained on now, Link will be above in the versions section.
Ahhh. Thanks I will try it out. I guess they used something special to train.
@benjimon678 Probably. Most likely just did it through the scripts directly. Instead of using a compiled UI.
This version of the model is one I've created specifically to get around the Kohya bugs I encountered.
If you have any issues with this version, I can somewhat assist.
@Triple_Headed_Monkey It started without errors on the first try 😋
Details
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.