Network Rank (Dim) and Network Alpha

Network Dim

This is LoRA's "how many dimensions should information be held in?"

e.g.

Dim 4 → Small capacity

Dim 16 → Normal

Dim 32 → quite a margin

Dim 64 → Large

Dim 128 → even larger capacity

That's what it feels like.

If want to memorize a specific style, character, or detailed characteristics from a large number of images like this time, if Dim is too small,

"can't remember everything, so compress similar features together."

It turns out.

Conversely, increasing Dim increases the representation capacity, but it also makes it easier to overfit the data and increases the file size.

However, when Dim is small, the model does not have enough capacity to retain all of these features individually. Instead, it tends to move toward:

“These are the essential features shared across this large collection of images.”

In other words, it abstracts and compresses the features.

This can work particularly well with cartoon, where stylization and simplification are part of the appeal. In such cases, this kind of compression can actually be desirable.

Network Alpha

This one is a little confusing.

In LoRA, conceptually

Actual LoRA contribution ≒ Alpha / Dim

The scaling is used.

e.g.

Dim : Alpha : Alpha/Dim

16 : 16 : 1.0

32 : 32 : 1.0

64 : 64 : 1.0

64 : 32 : 0.5

128 : 32 : 0.25

Therefore,

Dim 64 / Alpha 64 and Dim 64 / Alpha 32

So, even with the same Dim, the effective scale of LoRA is different.