train dreambooth result

[wen020]

This is the result of training dreambooth. I found that Pixart responds better to prompts, but its consistency with characters is not very good. Has anyone produced better results?

[quocanh34]

@wen020 I trained with 20 photos, 1600 steps, lr 5e-6, and it produces quite good results. Btw, have you tried to train the text encoder also?
I also want to ask @lawrence-cj for the option to train the text encoder in the script train_dreambooth.py

[mrT333]

[..] Once you train the text encoder, the text features will change, and the previous image-text alignment features will be destroyed. In this case, I don't think training the text encoder is a good choice.

That is what the regularization images in Dreambooth are there for: To avoid catastrophic forgetting / destroying previous image-text alignment.

Pixart is amazing. I'd love it if it would get a similar community support as Stable diffusion does with thousands of finetunes. I cannot even imagine how the quality of Pixart with its superior training efficiency would look like compared to the SD1.5 community finetunes once people start finetuning it on mass.

But for mass adoption of finetuning (a fully featured, e.g trainable text encoder w reg. images) Dreambooth as a low-barrier option might be crucial as it is way more "idiot proof" than a full finetune which in turn makes it more accessible to the countless hobby users who might not be as proficient in avoiding finetuning mistakes as power users are.

[lawrence-cj]

Agree with your suggestion. Is there any repo or paper u think it’s good enough for us to implement the SOTA Dreambooth LoRA.

[mrT333]

Agree with your suggestion. Is there any repo or paper u think it’s good enough for us to implement the SOTA Dreambooth LoRA.

If you are looking for the SOTA Dreambooth LoRA than that is definitely Hyperdreambooth ( https://arxiv.org/abs/2307.06949 ) by Google Research. It is an improved version over the original Dreambooth ( https://arxiv.org/abs/2208.12242) in every way, shape and form that was also published by Google Research & Boston University.

HyperDreambooth would have an interesting synergy with PixArt-alpha considering it would have 25x higher training efficiency and 10000x (!) smaller model size (few KB instead of hundeds of MB LORA) than the original Dreambooth that is currently used by everyone in the Stable Diffusion Community.

But the main benefit of HyperDreambooth goes beyond that: Despite its higher training efficiency and tiny size it is just way more capable than the original Dreambooth resulting in achieving better fidelity and retaining more stylistic flexibilty while using only one (!) reference image compared to the 10-25 (or in some cases even 40-80 dep. on subject & image quality) reference images necessary for original Dreambooth.

[mrT333]

If Hyperdreambooth would be implemented within Pixart it is just a matter of time until it would take over the whole SD community. Let my explain my reasoning:

1. Shareability is an important aspect in the frequency on how often LoRAs are used and downloaded, and therefore how high of a popularity they can establish in the community. That is one reason why there are thousands of popular LoRAs compared to only a handful of established full finetunes. Lots of people start to complain about the huge sizes of the SDXL-LoRAs compared to SDv1.5 LoRAs which is i.m.o one of the reason that is hampering SDXLs adoption rate despite its superior quality over base SD1.5. (Although textual inversions are even smaller, they are just not comparable as the fidelity falls too low)

2. The low number of required reference images makes Hyperdreambooth more accessible to a huge user base consisting of hobby users boosting usage numbers immensely. One reason why the original Dreambooth is so popular over full finetunes is that it can produce almost the same quality while being less error prone (e.g. preventing catastrophic forgetting through reg. images), needing much less training and a way smaller Dataset. Hyperdreambooth improves over every of those aspects while having higher fidelity in LoRA format than even a full Dreambooth would have (even more so compare to DB LoRA) .

3. Higher fidelity and more flexibility than orig. Dreambooth. LoRAs are often used to introduce a new concept into the model where fidelity is critical, like a TV-show character, a celebrity, anime character etc. (If you don't care about fidelity then it would make more sense to use a textual inversion. And if its not a specific concept but to capture a general style a full finetune would make more sense.)
   If it turns out that characters that are trained through HyperDreambooth LoRAs within Pixart-Alpha have higher resemblance than LoRAs or even full Dreambooths of Stable Diffusion models it is just no competition. People will always prefer the higher fidelity. They don't want to generate images where the character looks "almost as" or "similar to" the intended subject, they want the character to stay true to its identity.

What fascinates me of PixArt-Alpha over SD is on one hand the way higher training efficiency as I expect that to enable the community to create much more powerful finetunes and on the other hand the closer adherence to the prompt as that empowers every single user to create an image closer to their intent.
I think Hyperdreambooth would supplement both those Pixart-Alpha "superpowers" perfectly with its own higher training efficiency and staying closer to the intent of the artist by retaining higher subject fidelity while still allowing stylistic flexibility.

[mgyong]

@quocanh34 Can you share the results of your training dataset and the results of Pixart-dreambooth? so we can compare with what @wen020 did. If you have the code to share that will be great as well. I don't believe @wen020 trained the text encoder during the dreambooth process. Did you @quocanh34 ?

[lawrence-cj]

Glad to hear the result. I don't think the consistency issue is caused by PixArt. We also hope for a better solution.

[attashe]

Also tried a dreambooth and added a LoRA with PEFT, got some results with default setting, think that need regularization images and more precision prompting during training, also sks token hasn't a lot of influence

[mgyong]

@attashe Could you share your results for the Pixart dreambooth example and code? We like to see how it compares with what @wen020 did.

[mgyong]

@FurkanGozukara how did create a large regularization dataset for what you did for man but for woman using unsplash? Was it a very tedious process?

[FurkanGozukara]

it is beyond tedious. total i spent like 10 days now :D

[attashe]

Yes, I trained LoRA with r=8, 3 images, and detailed separated prompts. May be learning rate too high and I need lower and set more steps. In general concept learned, not so much variation, but dataset also very small. 60% generations trash like last image, but others from mediocre to ok.

[attashe]

@mgyong

[attashe]

@FurkanGozukara I started rewriting this part for support individual captions. As free alternative used a subset of CelebaHQ as regularization dataset.

[FurkanGozukara]

actually my dataset is captured from my phone : poco x3 pro in several minutes :d i didnt even bother to replace clothing

because i wanted it to be an easy dataset so people can replicate

currently what settings and script you are using to train hyper dreambooth? i can compare with kohya DreamBooth

[mrT333]

thats the problem... currently there are only a few poor WIP implementations of Hyperdreambooth available that are not even properly pretrained like the paper suggested.

That is why all we have are the comparisons in the Hyperdreambooth paper which all are done in a extreme scenario of just one reference image and therefor comparing Hyperdreambooth with the original Dreambooth when they both weren't used to their max potential. (Just look at the Dreambooth examples in that paper they are complete garbage as well due to the low reference image count. And your pictures above are proof that the original Dreambooth can do way better than what is shown in that scenario within that paper, same as i would assume the Hyperdreambooth would increase in quality with more reference images)

[mrT333]

actually my dataset is captured from my phone : poco x3 pro in several minutes :d i didnt even bother to replace clothing

Since you are doing a dreambooth of yourself you have control over how the subject is captured and it was quick and easy to change distance and angles. Most people have a limited training data set where they cannot control how the images look like. They have to work with whatever dataset they scrumbled together even if those images always have the same face angle or the same distance.

To have a good dataset you don't need a professional camera, most important is variety of face angles, distances and high visibility of the face, which I think is exactly what you wanted to demonstrate with your video and succeeded. Good datasets can be made easily & quickly with simple cameras if you have control over how your subject gets captured (which is not always the case).

[mgyong]

@FurkanGozukara The example dreambooth images you show here in the thread is done in SDXL right? not using Pixart? Have you been able to get Dreambooth working with Pixart?

[FurkanGozukara]

@mgyong they are SDXL . I am yet to train PixArt