Memory problems with custom dataset

[Mikey-E]

It appears that grads and grads_abs may not be getting cleared correctly between calls of densify_and_prune(). They seem to double in size each time, eventually leading to out of memory errors. As such, simply changing memory parameters or environment variables does little to help. What is the best way to manage this size during training?

[niujinshuchong]

Hi, the number of Gaussians depends on your scene. If you have very sparse initial point, more Gaussians will be added at the beginning. We increase the same number of Gaussians for grads and grads_abs. see here: https://github.com/autonomousvision/gaussian-opacity-fields/blob/main/scene/gaussian_model.py#L637-L640 Therefore, it will have more Gaussians at the beginning. You can decrease it by using a smaller ratio.

[wjy-666]

Same problem, very small dataset can result in out of memory errors. Have you solve this problem？

[Mikey-E]

@wjy-666 I have not.

[niujinshuchong]

Hi, maybe you can increase the threshold for gaussian allocation. Could you share a plot to show how the number of Gaussians change during training?