A sleeping giant rises from the depths...

https://civarchive.com/user/speach1sdef178

Welcome all. This is a joint collaborative project between myself and https://civarchive.com/user/speach1sdef178. I've been developing the TruBass model series, while she has been developing the Project 0 model series.

Periodically, as the development progressed on our individual models we would share the results with each other and each of us would take turns creating our own merge.

Working towards similar goals with differing perspectives, our respective versions of the model began to become both similar and unique.

All of the training that we have done, has been done using paramaters achievable on 24GB of VRAM, and much of it was trained online using Tensor Art's online training, which allows users to train on their own custom models. What wasn't trained on Tensor art was trained locally, using AI Toolkit.

Furthermore, until recently, the majority of our work has been focused on training LORA and using them, in both positive and negative weights to substantiate impactful changes which steer the model towards our intended goals.

Generally speaking we have succeeded in a number of areas, but we have also identified several key areas in which the currently available Flux models are significantly lacking. And frankly both of us have been struggling with how to proceed with implementing our vision without losing the general functionality of the model and reducing overall prompt adherence.

As this is a FLUX DEV model, it is necessary to state all of the changes made to it as part of the model license agreement for using and customizing it.

In this case, it can be summarized as follows:

• We trained ~1000 styles into the model using our own captioning style for the mile high styler.

• We then tested over 1000 styles and identified which styles were lacking.

• We created synthetic datasets using the models own output, which reflected the worst mistakes of a particular style on a case by case basis.

• We trained individual bad LORA using those bad datasets, one by one.

• Then we follow this up by collecting and curating a small dataset of real-world data for the individual style. Which is then used to train a good lora.

• The "bad" lora is then used in negative weight to remove unwanted elements, while the "good" lora is used both to return some of the lost weight to the model and to shape it more accurately towards your intended style output.

• The resulting combination of positive and negative weighted lora is then merged with the model and saved as a checkpoint.

• We repeated this process multiple times a day, for approximately 3 months. Periodically merging our works together and occasionally incorporating additional community models like ShuttleDiffusion, Crystal Clear Super, Jibmix and ArtsyDream for added context.

• When we merged with a community model, it was necessary to re-apply the negative weighted lora to avoid it getting rid of my existing progress during the merge process.

• In some cases it was also necessary to re-apply the positive weighted lora also.

The intention of this model is to create a replacement for the FLUX DEV model which is so much improved that for most cases additional LORA aren't needed. We would have liked to have added characters to the model, including celebrities. But fundementally it seems to be above our paygrade as simple model merging lora trainers.

This has been an extremely difficult endeavour despite overall being a simple process. Much of the difficulty has stemmed from the research elements and developing the system by which the model can be consistently trained on different datasets with the same paramaters. We've had to make several compromises because the model architecture is either unfixable, or requiring a total overhaul from scratch.

The latest version(s) of the model can be tested online using Tensor Art. And I will, as I develop and test, release them here on Civitai for public download.

https://tensor.art/models/816904519431515667

The reason for this being that the cost to train on Tensor is really much lower than Civitai. Where a model release can take several days, to a month or more to generate enough Buzz to train a follow-up model onsite, Tensors prices make it possible to train a new lora every single day, even if you aren't succesful at all. And the more success you have on the platform the more LORA you can train each day without having to spend a thing.

So in order to make sure that all of us get the benefit of free access, I'll keep the test releases exclusively online only on Tensor to fund the continued ongoing training process.

We fall ever deeper into the abyss as we seek the light.

Collaborative Guidance: Building a Unified Model Framework

Welcome to the AI Model Collaboration Project! This guide will help you dive into refining and merging models while prioritizing prompt adherence above everything else. By focusing on modes for artistic mediums and styles for fashion, and using positive and negative LORA, we’re creating precise, adaptable, and groundbreaking models. Let’s collaborate and redefine what AI models can achieve.

1. Join the Collaboration

We’re a collaborative community focused on improving and expanding AI models. Connect with us here:

• Discord Server: AI Revolution Discord
  Join to share progress, get feedback, and collaborate in real-time. The community is managed by Olivio Sarikas, and populated by other trainers, mergers and developers, as well as AI enthusiasts.

• Tensor Art: Explore Models and Test Online
  Test the latest versions of the model, provide feedback, and support ongoing training.

2. The Foundation

This project prioritizes prompt adherence—making sure the model produces exactly what’s described in a prompt. Every step we take builds on this foundation.

Key elements:

• Modes: Represent artistic mediums (e.g., oil painting mode, pixel art mode).

• Styles: Reserved for fashion (e.g., cyberpunk fashion, baroque fashion).

• Positive/Negative LORA: Tools to fine-tune outputs by amplifying the good and suppressing the bad.

3. Prompt Structure

Prompt Template

Mode, artistic attributes, era, fashion style, subject count, unique identifier, Rating, detailed scene/action description, ¬ additional details, filter.

Examples

Oil Painting Mode

Oil painting mode, rich textures, detailed brushwork, era 1600s, baroque fashion, solo, intricate composition, Rating SFW, a nobleman standing in an opulent room holding a gilded scepter, ¬ light streaming through ornate windows, soft light filter.

Pixel Art Mode

Pixel art mode, 8-bit graphics, bright colors, era 1980s, casual fashion, duo, retro video game aesthetic, Rating SFW, two characters running through a pixelated jungle with glowing mushrooms, ¬ vibrant sprite animations, pixel glow filter.

4. Workflow

Step 1: Build Your Dataset

1. Mode Datasets:

   • Collect 10–30 high-quality images for each mode.

   • Example for oil painting mode: Include thick impasto textures, smooth tonal blending, and expressive compositions.

2. Style Datasets:

   • Gather images reflecting specific fashion (e.g., baroque fashion, cyberpunk fashion).

3. Detailed Prompts:

   • Select 5 standout images and write detailed prompts for them. These will anchor your training process.

Step 2: Train Positive and Negative LORA

Positive LORA

• Purpose: Reinforces desired characteristics and improves prompt adherence.

• How to Train: Use curated datasets representing the mode or style.

• Weighting: Use weights up to +0.4 during inference. Avoid exceeding this range to prevent outputs from becoming exaggerated.

Negative LORA

• Purpose: Suppresses artifacts and incorrect representations.

• How to Train:

  1. Generate outputs using problematic prompts.

  2. Create a "bad dataset" of images that fail to meet expectations.

  3. Train a LORA to target these issues.

• Weighting: Use weights up to -0.3 to remove unwanted elements without overcorrecting.

Step 3: Combine Positive and Negative LORA

Balance is key. Use both positive and negative LORA together for fine-tuned results:

oil painting mode, intricate brushwork, vibrant colors, era 1600s, baroque fashion, solo, thick impasto oil painting, Rating SFW, a nobleman in an ornate study holding a gilded staff, ¬ light reflecting on textured details, soft glow filter.
-0.3:(negative oil painting) +0.4:(positive oil painting)

Step 4: Test and Refine

1. Prompt Adherence:

   • Validate how well the model follows prompts across modes and styles.

2. Adjust Weights:

   • Fine-tune LORA weights based on test results.

3. Iterate:

   • Refine datasets and prompts to close any gaps.

5. Merging Models

Leverage the Google Drive LORA folder to create your unique model merges.

Process

1. Incrementally merge LORA, testing results at each step.

2. Apply positive and negative weights during merges to maintain balance.

6. Negative LORA Training Workflow

Step 1: Build the "Bad Dataset"

• Generate outputs using a problematic tag/prompt.

• Collect images that exhibit artifacts, distortions, or poor representation.

Step 2: Train the LORA

• Train the LORA using this dataset to suppress unwanted features.

Step 3: Apply During Inference

• Use a negative weight (up to -0.3) for the trained LORA during inference.

7. Tools and Platforms

• Discord Server: AI Revolution Discord

  • Share progress, collaborate with the community, and get real-time feedback.

• Tensor Art:

  • Test models and fund LORA training through platform engagement.

  • Explore Tensor Art.

8. Key Tips

• Weights Matter: Positive LORA weights should stay between +0.1 to +0.4, while negative weights should remain within -0.1 to -0.3.

• Work Incrementally: Avoid making too many changes at once. Iterate carefully to retain progress.

• Collaborate: Share results and learn from the community. Feedback is invaluable.

9. Goals Moving Forward

1. Perfect Modes:

   • Ensure that each mode performs consistently and accurately.

2. Strengthen Prompt Adherence:

   • Validate and improve outputs based on edge cases and detailed prompts.

3. Secondary Refinements:

   • Once adherence is perfected, focus on skin textures, anatomy, and lighting.

With these tools and guidance, you’ll have everything you need to create precise, adaptable models that excel in creative execution. Let’s build something incredible—together! 🚀