How to use trained pixart lora for inference?

[segalinc]

I trained a lora using the script train_pixart_alpha_lora.py
How can I use it now to generate images?
The doc is a bit confusing as it's not clear if I need to covert the model to diffusers or just load_lora_weights from saved safetensor file

[surebert]

This example will output an image called demo.png. Replace the checkpoint definition with the path to your checkpoint, replace the prompt with the promp

from pathlib import Path

import torch
from diffusers import PixArtAlphaPipeline, AutoencoderKL
from peft import PeftModel

checkpoint = Path('/path/to/your/build/checkpoint-500')
prompt = 'A small cactus with a happy face in the Sahara desert.'

vae = AutoencoderKL.from_pretrained(
    'stabilityai/sd-vae-ft-ema',
)
pipe = PixArtAlphaPipeline.from_pretrained(
    "PixArt-alpha/PixArt-XL-2-1024-MS",
    vae=vae,
    torch_dtype=torch.float32,
).to("cuda")
pipe.transformer = PeftModel.from_pretrained(pipe.transformer, checkpoint)

# Enable memory optimizations.
pipe.enable_model_cpu_offload()
img = pipe(
    prompt,
    width=1024,
    height=1024,
    num_inference_steps=50,
).images[0]
img.save("./demo.png")

[segalinc]

Thank you for the help!

…

On Wed, Jan 10, 2024 at 6:20 PM Paul Visco ***@***.***> wrote: This example will output an image called demo.png. Replace the checkpoint definition with the path to your checkpoint, replace the prompt with the promp from pathlib import Path import torchfrom diffusers import PixArtAlphaPipeline, AutoencoderKLfrom peft import PeftModel checkpoint = Path('/path/to/your/build/checkpoint-500')prompt = 'A small cactus with a happy face in the Sahara desert.' vae = AutoencoderKL.from_pretrained( 'stabilityai/sd-vae-ft-ema', )pipe = PixArtAlphaPipeline.from_pretrained( "PixArt-alpha/PixArt-XL-2-1024-MS", vae=vae, torch_dtype=torch.float32, ).to("cuda")pipe.transformer = PeftModel.from_pretrained(pipe.transformer, checkpoint) # Enable memory optimizations.pipe.enable_model_cpu_offload()img = pipe( prompt, width=1024, height=1024, num_inference_steps=50, ).images[0]img.save("./demo.png") — Reply to this email directly, view it on GitHub <#68 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHLPES65FVFFDAIWMVABKDYN5EAFAVCNFSM6AAAAABBVLAAU2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DAOBZGIYTG> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>