Rust Stable Diffusion with diffuser-rs

🦀 I curious that can we do Stable Diffusion things with Rust? Turn out there’s a way to do just that via FFI which I never think I will get a chance to use it so this worth noted in my book and re-post here 👉 https://book.gist.rs/rust/r2/hello-diffusers-stable-diffusion.html

Prequisites

1. Source code: diffusers-rs
2. HUGGINGFACE_TOKEN : Get it from here.
3. weights : 1.5 or 2.1

Manual Setup

For huggingface model v1.5, This will manual convert the weights so you will need installed Python, numpy, PyTorch.

git clone https://github.com/LaurentMazare/diffusers-rs
cd diffusers-rs
. ./scripts/download_weights.sh REPLACE_HERE_WITH_YOUR_HUGGINGFACE_TOKEN

Lazy Setup

For huggingface model v1.5

mkdir -p data
wget https://huggingface.co/lmz/rust-stable-diffusion-v1-5/resolve/main/weights/bpe_simple_vocab_16e6.txt -P data/
wget https://huggingface.co/lmz/rust-stable-diffusion-v1-5/resolve/main/weights/pytorch_model.ot -P data/
wget https://huggingface.co/lmz/rust-stable-diffusion-v1-5/resolve/main/weights/unet-inpaint.ot -P data/
wget https://huggingface.co/lmz/rust-stable-diffusion-v1-5/resolve/main/weights/unet.ot -P data/
wget https://huggingface.co/lmz/rust-stable-diffusion-v1-5/resolve/main/weights/vae.ot -P data/

For huggingface model v2.1

mkdir -p data
wget https://huggingface.co/lmz/rust-stable-diffusion-v2-1/resolve/main/weights/bpe_simple_vocab_16e6.txt -P data/
wget https://huggingface.co/lmz/rust-stable-diffusion-v2-1/resolve/main/weights/clip_v2.1.ot -P data/
wget https://huggingface.co/lmz/rust-stable-diffusion-v2-1/resolve/main/weights/unet_v2.1.ot -P data/
wget https://huggingface.co/lmz/rust-stable-diffusion-v2-1/resolve/main/weights/vae_v2.1.ot -P data/

Run

For huggingface model v1.5 (has flag --sd-version v1-5)

cargo run --example stable-diffusion --features clap -- --prompt "A rusty cat robot holding a fire torch." --sd-version v1-5

For huggingface model v2.1

cargo run --example stable-diffusion --features clap -- --prompt "A rusty cat robot holding a fire torch."

Take Away

It took almost 14 minutes for 1.5 and 49 minutes for 2.1 to generate 1 pics for my old mac.

💁‍♂️ If you like my work please consider support me so I can have better machine (mps) to play with = more content to blog.