Whats new
beta:
Updated the workflow and custom node (both need to be updated).
Thanks to user Hysocs for the tip regarding the Anima timestep_sampling parameter (changed to uniform).
The model is now slightly better at understanding natural language prompting; the default style for natural prompting has improved in some areas but not others, and there are some issues with anatomy—the model is still undergoing training (currently trained on 150,000 images).
Description
an attempt to transition Anime base to an MoE architecture—two 2B models, each trained on a specific noise-reduction range.
The model handles natural language prompting quite well, but the resulting style often leaves something to be desired—I'm thinking about how to fix that. (Maybe it's because I recently increased the maximum prompt length from 512 to 2048.)
This model was used to make image captions: Minthy/ToriiGate-0.5 · Hugging Face
Workflow is included in posted images metadata and optional files
You need to put this into custom_nodes for the workflow to work.
real time training checkpoints are there:
low noise
https://huggingface.co/femboysLover/anima-looped-test-diffusion-blocks-idx-0
high noise
https://huggingface.co/femboysLover/anima-looped-test-diffusion-blocks-idx-1
the article from which I took the methodology:
[2506.14202] DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
Description
FAQ
Comments (12)
“You need to put this into for the workflow to work.”
What does this mean? Is there a reference workflow available?
The workflow data is embedded in the metadata of the post's model images—you can simply download them and drag and drop them into the ComfyUI interface. However, I’m going to add the workflow as a config file now; I tried adding it properly before, but CivitAI won't let me change the type from "config" to "workflow."
upd: done, check Optional files
Okay, it’s sorted out. Admittedly, none of the generated images are appealing. Also, as I mentioned earlier, prompt adherence stands out. I feel like it’s improved a little, but there isn’t an obvious boost you can clearly notice—only a slight enhancement.
@tuosun01970 Yeah, training needs to continue—it will likely take a month to achieve noticeable results.
How exactly are you training the models? Are you fine-tuning them while excluding certain timestep ranges, or are you training across the full timestep range and changing the noise distribution sampled at each timestep?
Add timestep block range sampling for Anima training · drimeF0/sd-scripts@34e3fed
I'm simply finetuning the model only on specific noise-reduction steps.
@spogocev172 You may want to double-check how Anima’s timestep shift interacts with this setup. Because Anima uses a custom shift that heavily favors high-noise timesteps, splitting the raw range 50/50 may not produce two models with equal effective coverage as shift changes the timestep after the fact. Ideally with this you would likely want to train uniform with no shift
That could leave the low-noise specialist with a much narrower or weaker training range, which undermines one of the main benefits of using a separate model for the refinement half of the denoising process.
You may already account for this but its just what comes to mind
@Hysocs If you're referring to discrete_flow_shift=3, I've actually removed that from the training config for now.
@Hysocs Thanks for the tip—I decided to review the default arguments and discovered that I’d been training using the sigmoid step sampling method all along; I’ve just switched to uniform.
@spogocev172 Yes, I just worked through the math and simulated the sampling. Base Anima uses this distribution across the full timestep range, but in your MoE setup you split that distribution in half.
That means each specialist only receives one side of the original curve rather than a complete timestep distribution. I doubt the effect is huge, but it does mean each sub-model has an entire section of timesteps excluded and receives uneven sampling within its assigned half.
So i do think trying timestep_sampling=uniform would be worthwhile here, because it would let each specialist cover its full assigned range evenly instead of inheriting only one side of the original hill-shaped distribution.
I train Anima almost daily and have built my own trainer, so I genuinely enjoy discussing the training logic behind it. Feel free to DM me if you want to compare ideas or speak on stuff like this
@spogocev172 And keep up the good work. I always appreciate seeing people experiment with new stuff like this, it always worth testing
@Hysocs That’s good to hear. I also tried retraining the Anima model using the Relaxed Transformer architecture (https://deepmind.google/research/publications/122290/), but I eventually abandoned the training—you can test it here: https://huggingface.co/femboysLover/anima-looped-test-lowlr4 (though there is a lot of hard-coded logic in there).