I added a lot of furry images for training, but the results are not ideal.
Due to limited GPU memory, the training resolution had to be kept low, and I used labels as prompts since I didn’t have time to convert them into natural language.
Please don’t judge the quality of the base model by the performance of this LoRA.
This is an experimental model that may generate NSFW content. The training data comes from an artist, so the model may be taken down later. It’s intended only for research and learning purposes.
Honestly, I was just advised to give this a try, but the results might turn out pretty bad 😛
I’m still pausing updates and treating this as a few simple experiments.
For better results, please look forward to the edited version of Z-image and Z-Image-Base. The Z-image team is also considering doing an anime-style finetune—perhaps that will be the promised land?
Description
---
job: "extension"
config:
name: "my_first_lora_v2"
process:
- type: "diffusion_trainer"
training_folder: "D:\\ai_training\\ai-toolkit\\output"
sqlite_db_path: "./aitk_db.db"
device: "cuda"
trigger_word: "syj style"
performance_log_every: 10
network:
type: "lora"
linear: 32
linear_alpha: 32
conv: 16
conv_alpha: 16
lokr_full_rank: true
lokr_factor: -1
network_kwargs:
ignore_if_contains: []
save:
dtype: "bf16"
save_every: 250
max_step_saves_to_keep: 4
save_format: "diffusers"
push_to_hub: false
datasets:
- folder_path: "D:\\ai_training\\ai-toolkit\\datasets/2464453_training_data"
mask_path: null
mask_min_value: 0.1
default_caption: ""
caption_ext: "txt"
caption_dropout_rate: 0.05
cache_latents_to_disk: true
is_reg: false
network_weight: 1
resolution:
- 512
- 768
controls: []
shrink_video_to_frames: true
num_frames: 1
do_i2v: true
flip_x: true
flip_y: false
train:
batch_size: 1
bypass_guidance_embedding: false
steps: 2000
gradient_accumulation: 1
train_unet: true
train_text_encoder: false
gradient_checkpointing: true
noise_scheduler: "flowmatch"
optimizer: "adamw8bit"
timestep_type: "weighted"
content_or_style: "balanced"
optimizer_params:
weight_decay: 0.0001
unload_text_encoder: false
cache_text_embeddings: true
lr: 0.0001
ema_config:
use_ema: false
ema_decay: 0.99
skip_first_sample: false
force_first_sample: false
disable_sampling: false
dtype: "bf16"
diff_output_preservation: false
diff_output_preservation_multiplier: 1
diff_output_preservation_class: "person"
switch_boundary_every: 1
loss_type: "mse"
model:
name_or_path: "Tongyi-MAI/Z-Image-Turbo"
quantize: true
qtype: "uint4"
quantize_te: true
qtype_te: "qfloat8"
arch: "zimage:turbo"
low_vram: true
model_kwargs: {}
layer_offloading: true
layer_offloading_text_encoder_percent: 1
layer_offloading_transformer_percent: 0
assistant_lora_path: "ostris/zimage_turbo_training_adapter/zimage_turbo_training_adapter_v2.safetensors"
sample:
sampler: "flowmatch"
sample_every: 250
width: 1024
height: 1024
samples:
- prompt: ""
- prompt: ""
neg: ""
seed: 42
walk_seed: true
guidance_scale: 1
sample_steps: 8
num_frames: 1
fps: 1
meta:
name: "[name]"
version: "1.0"
FAQ
Comments (2)
From what I've seen the model is heavily influenced by realism so its hard to decouple into pure 2D. And its probably even harder if you arent training 1girl characters.
The model really needs a 2D finetune to compete with SDXL variants.
Thank you for making this effort! :D