Introduction to Stable Diffusion Latent Upscaler

Upscale generated images by 2x natively

Image by the author

The topic for today is about a latent diffusion-based upscaler model to upscale Stable Diffusion generated images by 2x natively. The model was trained by Katherine Crowson in collaboration with Stability AI.

As of version 0.13.0, the diffusers package officially supports the stabilityai/sd-x2-latent-upscaler model under the StableDiffusionLatentUpscalePipeline class.

One main advantage of this pipeline is that you can use the latent output from any StableDiffusionPipeline and pass it as input to the upscaler before decoding it with the desired VAE.

Alternatively, you can encode an existing image to latent space before passing it to the upscaler and decode the output with any VAE.

Let’s proceed to the next section to install all the necessary modules.

Setup

Before that, it is highly recommended to create a new virtual environment.

diffusers

Activate the virtual environment and install the standard diffusers package via the following command:

pip install diffusers

Alternatively, install the latest diffusers package as follows:

pip install git+https://github.com/huggingface/diffusers

xformers (optional)

The xformers package helps to improve the inference speed. As of version 0.0.16, there are pip wheels support for PyTorch 1.13.1.

Pip install (win/linux)

For those with torch==1.13.1, simply run the following command to install xformers:

pip install -U xformers

Conda (linux)

For conda users, the installation only supports either torch==1.12.1 or torch==1.13.1

conda install xformers

Building from source

For the other use cases, consider building xformers directly from source:

# (Optional) Makes the build much faster
pip install ninja
# Set TORCH_CUDA_ARCH_LIST if running and building on different GPU types
pip install -v -U…