OOM issue when i try extract_mesh_tsdf

[ApdowJN]

When I try python extract_mesh_tsdf.py -m exp_playroom/release --iteration 30000, i found issues below:

Looking for config file in exp_playroom/release/cfg_args
Config file found: exp_playroom/release/cfg_args
Rendering exp_playroom/release
Loading trained model at iteration 30000
Reading camera 225/225
Loading Training Cameras
Loading Test Cameras
Rendering progress:   2%|█▍                                                               | 5/225 [00:00<00:29,  7.35it/s]
Traceback (most recent call last):
  File "extract_mesh_tsdf.py", line 107, in <module>
    extract_mesh(model.extract(args), args.iteration, pipeline.extract(args))
  File "extract_mesh_tsdf.py", line 90, in extract_mesh
    tsdf_fusion(dataset.model_path, "test", iteration, cams, gaussians, pipeline, background, kernel_size)
  File "extract_mesh_tsdf.py", line 67, in tsdf_fusion
    vbg.integrate(frustum_block_coords, o3d_depth, o3d_color, intrinsic,
RuntimeError: [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /root/Open3D/cpp/open3d/core/CUDAUtils.cpp:310: /root/Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp:25 CUDA runtime error: out of memory

How to solve this problem? Has anyone meet this error?

[niujinshuchong]

Hi, we use open3D's GPU version of TSDF fusion and when it is easy to run out of memory when applied to larger scenes. I think you could change the code to use CPU for TSDF fusion. But in general, TSDF results are worse then marching tetrahedral.

[ApdowJN]

Thank you for your reply! Is there a flag in the configuration to switch to CPU for TSDF fusion in the code?

[niujinshuchong]

Hi, unfortunately there is no single flag to support cpu TSDF fusion. But you can refer the code here https://github.com/hbb1/2d-gaussian-splatting/blob/main/utils/mesh_utils.py#L156-L180

[s3219521aa]

When I try python extract_mesh_tsdf.py -m exp_TNT/release/Caterpillar --iteration 30000 , I also meet this problem . Then I fix o3d_device = o3d.core.Device('CPU:0'), but it doesn't work. I'm running this code on a high-performance server with A100s and 750G RAM