How to Fine-tune SDXL 0.9 using Dreambooth LoRA

Personalized generated images with custom datasets

Image by the author

By reading this article, you will learn to do Dreambooth fine-tuning of Stable Diffusion XL 0.9 via LoRA. For you information, DreamBooth is a method to personalize text-to-image models with just a few images of a subject (around 3–5).

This tutorial is based on Unet fine-tuning via LoRA instead of doing a full-fledged training. LoRA is a parameter-efficient fine-tuning technique introduced in LoRA: Low-Rank Adaptation of Large Language Models.

LoRA is extremely beneficial for SDXL training as it:

• is less likely to have catastrophic forgetting as the previous pre-trained weights are kept frozen
• has fewer parameters than the original model and can be easily portable
• allows control to which extent the model is adapted toward new training images (supports interpolation)

This tutorial is based on the diffusers package and requires at least version 0.18.2 or above.

Based on a local experiment with GeForce RTX 4090 GPU (24GB), the VRAM consumption is as follows:

• 512 resolution — 11GB for training, 19GB when saving checkpoint
• 1024 resolution — 17GB for training, 19GB when saving checkpoint