ATTENTION Currently, the only way to use TinyBreaker is through ComfyUI by installing:
these nodes can be easily installed using ComfyUI Manager.
Workflows
The example images I've posted here on CivitAI include embedded workflows for use in ComfyUI. First, click on the image to enlarge it, then drag and drop it directly from your browser window into ComfyUI to automatically load the associated workflow.
However, if that doesn't work, base workflows are available here:
TinyBreaker Model
TinyBreaker is specifically designed to consume minimal resources while still achieving stunning visuals with little effort. It's a hybrid model that utilizes PixArt as its base and Photon as a refiner. At present, the project is entirely experimental.
Overview
Resource-Efficient: Designed for minimal resource usage, likely less than 8GB
Generation Speed: Images at 1536×1024 in ~12 seconds on an RTX 3080.
High Prompt Adherence: Generated images closely follow the given prompt.
Hybrid Model: Combines PixArt and Photon models.
Experimental: Currently in the Prototype0 phase, fully in testing.
Description
FAQ
Comments (39)
Gemini says this should be compatible with SD Forge. Forge says ValueError: Failed to recognize model type! Is Gemini wrong?
It looks like Gemini got that one wrong. This model is currently only set up to work with ComfyUI and relies on some custom nodes. It's not in a standard format that Forge recognizes. I'm hoping to create a Forge extension for it when I have some time, but for now, unfortunately, it's only compatible with ComfyUI. 😔
@Photographer Thanks for that! I'll stop chasing my tail. ;)
@HardyBigg Don't worry, I'm considering Forge compatibility for future updates. So, keep your fingers crossed 🤞
@Photographer Right on! Thanks I'll send you some buzz for the effort.
@HardyBigg Your buzz and encouragement are fuel for TinyBreaker’s progress, thank you so much!
It's very easy to use and fun! Plus, the background and skin details are amazing.
Thanks so much! I'm so happy to hear you're enjoying the model! That's exactly what I aimed for: easy, fun, and amazing images. Keep on creating and thanks again for your kind feedback!
And thanks for the buzz! Your support really motivates me to keep improving TinyBreaker.
I’m truly grateful to you for inspiring me with new ways to use Pixart!
I see so much potential in this workflow—it feels like any style can be flexibly generated depending on the settings and model combinations, and the possibilities seem endless.
@hjhf Thank you and I couldn't agree more! Pixart Sigma is a true hidden gem, often overlooked because of its smaller parameter count. However, with the right approach, it proves to be incredibly powerful.
What I love most about Pixart is its inherent creativity. It reminds me of the early days of SD1.5, where the model would sometimes surprise you by generating unexpected elements that seamlessly integrated into the image, resulting in truly unique and captivating creations, but unlike SD1.5, which was largely based on chance, Pixart seems to understand the concept of the prompt almost completely.
The results of its creativity and understanding of prompt concepts are truly surprising and have become a new source of inspiration for me!
Honestly, I think I underestimated Pixart Sigma. I didn’t expect it to grasp such a wide range of concepts, especially given its training on a limited dataset.
4step ?
does your pngs have the prompt inside ?
does this model can do lower resolutions also or regular ones like sdxl or flux medium resolutions ?
any chance to develop the pixart one with parameters more than this ?
and thanks a lot , i hope it's fast cuz i can do only 4 steps i should try at 15 at least as shown , hope there chance to develop this model more , it looks very good from results , at least it can understand the prompt but still need to be trained more on subjects shapes poses and others.
Hi there! Thanks for your interest, let me answer your questions:
1. Yes, all the PNGs I've uploaded to CivitAI have both the ComfyUI workflow and the original prompt embedded in the metadata in a format compatible with A1111. As far as I've tested, Civitai seems to preserve image metadata, so simply downloading or dragging the image into either of those applications should allow you to access that information. Additionally, CivitAI itself displays the prompt on the right-hand panel next to each image.
2. The model performs relatively well within a resolution range of approximately 800x800 to 1248x1248 pixels (for square images). For other aspect ratios, the total pixel count should fall within this range. This translates to roughly 640,000 to 1,500,000 total pixels.
3. There are a couple of PixArt models with 0.9B parameters already implemented that could potentially be combined with a refiner to enhance details. However, my goal with TinyBreaker is to focus on lower parameter counts – fewer parameters in the refiner, and especially fewer in the T5 encoder which currently uses 5.0B parameters.
4. You're right, more fine-tuning is needed! The model is based on the raw originals from PixArt and Photon, so further training is essential. I need to experiment with training them together or exploring whether it would be beneficial to train PixArt to incorporate Photon's refinement capabilities, potentially eliminating the need for a internal refiner.
Thanks again for your support and feedback! Your insights are valuable as I continue to develop TinyBreaker.
The examples really show that prompting makes fun and opens our fantasies. Thumbs up.
Thanks! Prompting lets us explore amazing ideas. Glad you’re enjoying it!
Hi, is there any way to use your floo to refine already finished images or photographs? I can't figure out a VAE encode that works correctly. Thanks in advance for the reply!
Almost figured it out right away. They are compatible with SDXL's VAEs.
Hi!
TinyBreaker indeed has two models embedded: PixArt (SDXL based VAE) and Photon (SD1.5). Internally, the code first runs a certain number of steps with PixArt and then another set of steps with Photon, the number depending on the chosen style.
So, if you want to make a subtle refinement to an image (running just 3 or 4 steps of TinyBreaker on an existing image), it essentially behaves the same as running only Photon. However, for more pronounced refinement, utilizing the full TinyBreaker workflow might make sense.
The quickest solution I can suggest, which you might have already seen, is to use the "fully customizable" workflow and tweak the sigmas using the SplitSigmas node in comfyui. You'd place the SplitSigmas node right between the "Unpack Sampler Params" and "SamplerCustom" nodes.
I hope this helps! Please do let me know if you make any progress or have further questions.
(the "fully customizable example" can be found in the ¨/workflows" folder, https://github.com/martin-rizzo/ComfyUI-TinyBreaker/tree/master/workflows)
but sd15 knowledge is way differ from pixart , how it refine , is using sdxl will be better? and how to don't refine at all , you node is = closed , like telling normal user you can't do anything other than what you coded !
This is an interesting model. Dedistilled flux is my go-to, but its also 23 gigs. This is a pretty impressive model for a 3 gig footprint.
Thanks so much for checking it out and for the kind words! You're right, the really interesting thing is that it could probably be even smaller. For now, it's just a proof of concept where I 'pegged' a model behind the other without even training them together. I guess that, well trained, the refiner could have only 0.6B parameters, so the base + refiner pack would be around ~2.5Gb, and with only 0.6B parameters active at any given time (less than SD1.5).
Oh this is fun! How do you get a random seed in the prompt field "--seed " chatGPT suggested "--seed random" but that didn't work
I’m not sure of the solution either, but in the sample workflow "v03_tinybreaker_complete_control_workflow.json" the seed node can be used, so I’m utilizing that.
@hjhf Thanks! Was using different workflow. It's amazing model, and especially when I realized it's sd1.5 hybrid!
@pipgamemyrran187
You're very welcome! I also think this is an interesting approach where both of our models can complement each other's weaknesses while making the most of their strengths.
And in practice, it's impressive how the two models together still fit within just 3GB—so compact, yet capable of such flexible expression.
@hjhf It's genius idea to insert T5 there, I just wish there was an easy way to get T5 into sdxl. But from some results I would not have believed there's pretty much sd1.5 inside this model. The creator is a genius.
@pipgamemyrran187
Yes, thanks to him, I gained a new insight, and it's surprising that SD1.5 can achieve good results even at high resolution.pixart+sd1.5 I still see potential for growth in these models.Above all, it is lightweight and has a lower burden for fine-tuning, making it easier to improve.
It would have been great if Ella had been released for SDXL.
Kolors was very close to meeting those conditions, but it's really unfortunate that it didn't capture people's interest.It achieves both PixArt and SDXL in a single model, offering even greater expressive power.
Since novelai_v04 also uses U-Net + T5, I also thought that direction was the right path, but...
@hjhf Yeah I am too pretty impressed by this model and this model finally "released" me from the clip-style prompting I'm so used to. I've been stuck with only sd1.5 for a long time, but few weeks ago updated my hardware and have been dabbling with different models and am quite impressed with SDXL actually while Flux seems pretty underwhelming in comparison to quality and speed. I have been downloading models like insane :D and I think for some reason I have some files for Ella too, no idea if I have gotten to use it even. Kolors is actually what I browsed today but it didn't catch my attention, maybe I should give it a try. But TinyBreaker has been my go to model for few days now. Made an odd "workflow" 1. go outside and take pictures 2. give chatgpt the picture and ask for how to prompt for that 3. recreate picture with TinyBreaker
That's an interesting workflow! I used to stick only to tags, but thanks to TinyBreaker, I’ve come to appreciate the fun of natural language prompts.Upgrading your hardware definitely opens up more options—it's exciting! Enjoy exploring everything!
Thank you all for the kind words, it's incredibly motivating to continue improving TinyBreaker!. Regarding random seeds, @hjhf ’s suggestion is spot on. Using the seed node in the workflow is the best way to go for now. I included that alternative workflow specifically for testing and to handle situations where you need a bit more control beyond the prompt parameters.
The idea of using "--seed random" was really interesting! It did initially seem like it would create non-reproducible images, but it gave me a thought that I think could actually solve that. I'm currently working on releasing version 0.3 of the nodes, and if I have the time, I'm hoping to include this new approach to the "--seed random" parameter.
Hola, tengo este error:
DoubleStageSampler //TinyBreaker
Input type (float) and bias type (struct c10::BFloat16) should be the same
Hola! ¿Estás usando el workflow default?:
https://github.com/martin-rizzo/ComfyUI-TinyBreaker/tree/v0.2.0/workflows
Siguiendo el link tenés la imagen "tinybreaker_example.png" que contiene el workflow default.
Pero si te sigue dando error, podés probar con "tinybreaker_fully_customizable_example.png", es un workflow más complejo (y lo uso solamente para testear) pero también puede darte un mensaje de error más detallado.
De todas formas, estoy a punto de lanzar la versión 0.3.0 de los nodos para ComfyUI. En esta nueva versión estuve implementando y modificando varias cosas importantes. Tal vez te convenga probar dentro de 2 o 3 días cuando la nueva versión ya esté disponible, porque el código cambió bastante.
Igualmente, le voy a echar un vistazo a este problema. Si por casualidad tenés acceso al log de la consola y podés pasarme el mensaje completo del error, te lo agradecería muchísimo.
Saludos
Aby way to use wildcards with it? And make random seed?
You can right-click on the Prompt node and use the following menu to allow input from other nodes' text output—this might be useful. Connect the output from a wildcard node to it.
■ convert_widget_to_input > convert_text_to_input
As for random seeds, you can use them by trying the "Fully Customizable Workflow" included in the repository.
@hjhf It was helpful, thank you <3