How much learning capacity does Pixart have?

[Deathawaits4]

Hei, since pixart uses very low parameter count, how much information is it able to store? Stable diffusion is able to store lots of different concepts in a single checkpoint without bleeding into other concepts if trained right. Is pixart able to do the same? Im not too sure how pixart handles it, so any insight would be great.

[lawrence-cj]

IMO, the restriction is more likely from the dataset. As long as your dataset has enough concepts, you will get what you want. The model size may have less impact on concept learning. That's why Stable Diffusion XL extends millions of internal datasets for both concept learning and image quality improvement. PixArt has a similar parameter as SD2.1, so I don't think this will be an issue. BTW, a larger model definitely will improve the capacity, a smaller model, like PixArt can accelerate the training process. Therefore, after the concept learning, the Transformer-based model PixArt can be easily scaled up to a larger model size. We consider this training process should be more reasonable.

[Deathawaits4]

alright, sounds good. So basically there isnt a realy limitation on the amount of concepts it can learn. Thanks! Is there already anything about concept bleeding? Is the model able to handle concepts without bleeding too strong into other concepts if captioning and dataset is good? SD 1.5 and 2.1 had this issue, SDXL not so much anymore.

[lawrence-cj]

I consider the overfitting of the model on a smaller dataset as the concept 'bleeding' you said. If so, there are some tricks stop the model from from overfitting, like smaller learning_rate, ema_update or early stop strategy(discussed in MetaAI EMU paper). I believe the EMU model should have larger model size, like 3B+, but they still have to stop the finetuning of the model before 5K step. If you don't want any overfiting, I would suggest 200K~1M data at least.