[LoRA] fix vanilla fine-tuned lora loading.

[sayakpaul]

What does this PR do?

Fixes vanilla (legacy) fine-tuning LoRA loading. The bug was introduced in #8316.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for fixing this. Could you please give a short explanation why this fails with legacy LoRA adapters? Also, would it be possible to catch this in a test?

[sayakpaul]

Could you please give a short explanation why this fails with legacy LoRA adapters?

It failed because of this check:

diffusers/src/diffusers/loaders/lora.py

Line 400 in 4ad7a1f

if any(key.startswith(cls.unet_name) for key in keys) and not only_text_encoder:

More specifically, any(key.startswith(cls.unet_name) for key in keys). The old format didn't have such prefix identifiers.

Unfortunately, it won't be possible to catch this in a fast test because the format is about 1.5 years old, and no one really uses it.

[BenjaminBossan]

Thanks for explaining. LGTM.

It would probably be a good idea to add the explanation to the commit message if there is ever the need to understand the history of this part of the code.