Photo Background - 2d Compositing｜写真背景・二次元合成
Trained on 2d illustrations composited on a photo background.
This is a small LoRA I thought would be interesting to see how models trained on illustrations or real world images/video can produce the composite, mixed reality effect.
ℹ️ LoRA work best when applied to the base models on which they are trained. Please read the About This Version on the appropriate base models and workflow/training information.
Metadata is included in all uploaded files, you can drag the generated videos into ComfyUI to use the embedded workflows.
Description
Trained on Anima Base 1
Updated dataset with a mix of natural language and tag captions
Partitioned dataset and trained at multi-res 512, 768, 1024, 1280, 1536
Training config:
# trained using diffusion-pipe commit b0aa4f1e03169f3280c8518d37570a448420f8be
# NCCL_P2P_DISABLE="1" NCCL_IB_DISABLE="1" NCCL_CUMEM_ENABLE="0" deepspeed --num_gpus=1 train.py --deepspeed --config anima-lora.toml --i_know_what_i_am_doing
output_dir = '/mnt/d/anima/training_output/anima-base-1-niji-v4'
dataset = 'dataset-anima-niji.toml'
# training settings
epochs = 2
# Per-resolution batch sizes
micro_batch_size_per_gpu = [[512, 64], [768, 48], [1024, 32], [1280, 24], [1536, 16]]
pipeline_stages = 1
gradient_accumulation_steps = 1
gradient_clipping = 1
warmup_steps = 30
lr_scheduler = 'cosine'
# misc settings
save_every_n_epochs = 1
activation_checkpointing = true
partition_method = 'parameters'
save_dtype = 'bfloat16'
caching_batch_size = 1
map_num_proc = 8
steps_per_print = 1
compile = true
[model]
type = 'anima'
transformer_path = '/mnt/c/workspace/models/diffusion_models/anima-base-v1.0.safetensors'
vae_path = '/mnt/c/workspace/models/vae/qwen_image_vae.safetensors'
llm_path = '/mnt/c/workspace/models/text_encoders/qwen_3_06b_base.safetensors'
dtype = 'bfloat16'
#cache_text_embeddings = false
llm_adapter_lr = 0
#timestep_sample_method = 'uniform'
flux_shift = true
multiscale_loss_weight = 0.5
sigmoid_scale = 1.3
[adapter]
type = 'lora'
rank = 32
dtype = 'bfloat16'
[optimizer]
type = 'adamw_optimi'
lr = 4e-5
betas = [0.9, 0.99]
weight_decay = 0.01
eps = 1e-8resolutions = [512, 768, 1024, 1280, 1536]
enable_ar_bucket = true
min_ar = 0.5
max_ar = 2.0
num_ar_buckets = 9
# micro_batch_size_per_gpu = [[512, 64], [768, 768, 48], [1024, 32], [1280, 24], [1536, 16]]
# images_niji\1536x1536\captions.json with 124 entries.
[[directory]]
path = '/mnt/d/training_data/images_niji/1536x1536'
resolutions = [512, 1024, 1280, 1536]
# images_niji\1280x1280\captions.json with 28 entries.
[[directory]]
path = '/mnt/d/training_data/images_niji/1280x1280'
resolutions = [512, 1024, 1280]
# images_niji\1024x1024\captions.json with 137 entries.
[[directory]]
path = '/mnt/d/training_data/images_niji/1024x1024'
resolutions = [512, 768, 1024]
# images_niji\768x768\captions.json with 43 entries.
[[directory]]
path = '/mnt/d/training_data/images_niji/768x768'
resolutions = [512, 768]
# images_niji\512x512\captions.json with 5 entries.
[[directory]]
path = '/mnt/d/training_data/images_niji/512x512'
resolutions = [512]
FAQ
Comments (16)
thanks, I like this one. one of the better lora for anima
Hi! I really like your LoRAs and I would like to train my own on Anima. I used to train on kohya-ss/sd-scripts for Illustrious, but I feel like the diffusion-pipe training method would work better for Anima. I read into diffusion-pipe and read your answers to some comments too, but a few things are unclear to me.
1. I have my dataset ready like this: image.png, image.txt (booru tags) and image_nl.txt (natural language captions). I tried to find information about how to make use of both the tags and nl captions when training but I could not find anything. I read here (https://civitai.red/models/2425904/arknights-endfield-or-operators-collection?dialog=commentThread&commentId=1133533) that you use DEFAULT_MIXED_WEIGHTS = {'tags': 50, 'nl': 10, 'tags_nl': 20, 'nl_tags': 20}. I assume this means that there is a 50% change that the image will use only booru tags, 10% chance that it will only use nl caption and a 40% chance that it will use a combination of both. If so how is tag shuffling and tag dropout works in this case? Wouldn't tag dropout ruin the captions? How is DEFAULT_MIXED_WEIGHTS implemented, I have not found it anywhere in the source code of diffusion-pipe.
2. Where do the tag dropout, keep tokens and options like these go? I assume they go under the dataset.toml under the [[directory]] part, but I don't see it anywhere in your training infos.
Thank you!
Hi @Yuno
I'd recommend you start with the training guide and example LoRA by the creator of diffusion-pipe and trainer of the Anima model itself:
https://civitai.red/models/2536147/greg-rutkowski-style-anima
The conversation you linked there predates the official support added to diffusion-pipe for Anima, and the DEFAULT_MIXED_WEIGHTS, dropout etc options are not available in the official repository, but were used in an early fork by bluvoll created before the official support.
If you want to combine your natural language captions with tags, I'd recommend just concatenating your files as part of the dataset preparation into a single image_filename.txt file.
You can also create variants of tags and natural language with any format with a captions.json file for use with diffusion-pipe that can contain multiple captions per image (length N of these corresponding to effective N repeats for an image).
This is what I do now to pre-compute my desired mix of natural language and tags, dropout, shuffling etc.
@motimalu Thank you very much for answering my questions!
I really like the captions.json idea. Do you use it while training, or do you only use it to create the dataset? If you use it while training, could you tell me what its structure looks like? What are its json keys?
What percentage of booru tags, natural language captions, and mix do you recommend? Do you still use 50-10-20-20%? What percentage of tag dropout is recommended? I usually go with 10%. Do you shuffle and use dropout on the natural language captions? I don't know its effects yet, I haven't used natural language for training before.
I also saw that in your dataset.toml you divide the images into many folders with different resolutions? Based on what do you group the images? I assume you use images under 1024x1024 so they don't fit into the bigger resolutions, and you don't want to upscale them either, so you create different directories for them.
What are the differences I need to be aware of when training a style vs. training a character?
Thank you!
@Yuno Yes I separate by resolution to avoid upscaling too much and control what gets placed in the higher res buckets.
The captions.json content is something like this, the key is the image filename and the values are an array of the captions to use for it during training:
{
"image_filename.png": [
"caption1 nl+tags",
"caption2 tags+nl"
],
}
Sharing a gist for the script I use to generate them from the _nlx.txt and tags .txt files that includes my current settings:
https://gist.github.com/motimalu/08ecfb90e1b4088e7d863b706da7a3ff
I don't do shuffle or dropout on the natural language, too complicated I think to not break sentences and make them nonsensical. Might use multiple NL captions from different VLMs though.
Dropout seems mostly helpful for learning characters by their activation tags, maybe not so important for styles.
@motimalu Thank you very much!!
I'm almost done with my setup but I still have a few small questions.
In my captions.json I see '\n' between the tags and captions and '\\' before every '(' and ')' in the tags part. So for example: shrug \\(clothing\\). Will this work correctly when diffusion-pipe reads it?
I used to downscale my images so that the max width or height would be 1920. Do I need to resize my images if they are abnormally large, or its fine if I put all the images in the 1536 directory?
Thank you!
Hey @Yuno yes shouldn't be an issue with diffusion-pipe reading the '\\' before every '(' and ')' or '\n'
I'm not actually sure on the handling of parenthesis '(' and ')' characters for the captions, they do work for prompt weighting at inference i.e. (tag:1.5) but are also common in booru tags, which is why they have needed to be escaped for training other tag-based models.
The official training sample dataset doesn't include any parenthesis in the captions so may need to ask circlestone labs about that.
You shouldn't need to downscale your images, diffusion-pipe will automatically do this for you to the specified resolution.
You can put everything in the 1536 directory and use the settings of the example default training sample to train those at [512, 1024, 1536] resolutions:
https://civitai.red/models/2536147/greg-rutkowski-style-anima
(Should try those settings first if not already)
The directories I use are separated by maximum image resolution, because I think diffusion-pipe will also upscale images to the specified resolutions for the dataset directory.
(Maybe not desired behavior if you have some very small images)
非常棒
I love you
This LoRA is funnily enough great as a general quality boost for background even when doing full anime image. Great work
I’ve been testing Anima1.0 and its base model, but it doesn't seem to be working. Am I using it incorrectly? I’m using it in a workflow where other LoRAs are producing the intended results.
Hi @SkyLaptor hard to say if you are using it incorrectly or not without seeing your workflow - it is loaded like any other LoRA.
The effect might not be as strong as other LoRAs, and it may not work well when stacked with other LoRA.
@motimalu
Thanks for the reply!
I’m using the "Advanced" version of the workflow from https://civitai.red/models/2426853/anima-preview-workflow almost exactly as-is, applying it via the LoRA Manager node. I have other LoRAs disabled.
This might be a bit off-topic, but what do people generally use to share workflows on CivitAI?
I’d like you to review my workflow.
@SkyLaptor Okay, that is a pretty loaded workflow.
If you want to test a LoRA with a few less steps I'd recommend using only the official Anima workflow from:
https://huggingface.co/circlestone-labs/Anima#installing-and-running
With the ComfyUI "Load LoRA (Model and Clip)" node that I've included in my preview images to load the LoRA.
To answer your other question, ComfyUI workflows are embedded into the image outputs.
So usually people are sharing these by default if they post the output images without any additional alterations.
All of the images shared on my Anima model cards should have the ComfyUI workflows embedded for example.
I don't have many additional custom nodes that I use - most notable one might be the custom comfyui-prompt-control node for schedule prompting syntax to mix concepts i.e. [word1|word2] which is needed to create the same outputs as I've shared.
@motimalu
Thank you.
After trying various things, I found that including words like "photorealistic" or "realistic" for the subject I want to render realistically yields good results.
The information about the workflow was also very helpful.
I only just realized this, but even with just Anima-B1, you can get reasonably photorealistic results if you include a "photorealistic" instruction in the prompt. That said, I feel like the details turn out better when using this LoRA.
I'll be making frequent use of this.