@SECourses Most creators would extract a lora from the dreambooth model and upload this instead.

@SECourses thanks, where to put this file in swarm ui?

@SECourses thanks champion, one more thing, in the article, could you please write the setting which needs to be selected during image generation in swarmui, for example: sampler, clips, flux guidance etc... I am trying to replicate your images and huge fan of you :)

@SECourses I've not seen a decent one around off the top of my head either.

T5 is basically a mini LLM. I'm fairly certain that CLIP is still handling the majority of the heavy lifting all round including the tokenization process. I've been trying to work it out for a little while but I think the process is something like:

Text input > T5 breaks it down contextually based on the sentence structure and attempts to feed it to the CLIP tokenizer without giving it room to mistake intent > CLIP converts tokens into a vector with spaital/visual information > Transformer uses these vectors to generate an image.

There are a couple of other possible configurations, but the simple take away from all of the different variations was that T5 is not capable of interpreting or producing the output necessary to caption or generate imagery. It is inherently fully text and token based. Therefore it's contributions to the process must also be restricted to the domain of text and/or improving the tokenization process with semantic context.

By itself T5 is basically an autofill model. Which would generate text based on simple inputs/parameters. Things like finishing a sentence you've started writing or responding to simple questions relating to additional information being provided to it. For example when interrogating an image using a Vision adapter model or something like BLIP.

In other words I'm not sure it really matters much in this case. CLIP having been created originally for the purpose of captioning and sorting images into categories was accidentally found to have the ability, when used in reverse, to generate the image information it was trained to caption.

So instead of converting images to vectors and plotting them to category tags, it became possible to input category tags and it would output vectors.

The transformer is then trained on top of the vector inputs with the same, or similar enough, datasets that were used to train the CLIP models, which bridges the gap between the Transformers ability to generate/manipulate pixels and the CLIP's output of vector information.

In the case of FLUX in specifics, the architecture seems to include a secondary clip model inside the transformer model layers itself. Which handles the translation between the training done on the transformer and the untrained clip and T5 models and allows for results to be closer to if you had trained them in conjunction.

This is my take on it anyway.