Self-supervised monocular depth estimation in nighttime and dynamic scenes

Datasets

💾 KITTI Dataset

🔹 Please refer to the raw KITTI dataset for downloading the kitti Dataset.

💾 nuScenes Dataset

🔹 Please refer to the nuScenes official website for downloading the nuScenes Dataset.

💾 DDAD Dataset

🔹 Please refer to the ddad for downloading the DDAD Dataset.

Evaluation

Scripts for evaluation are found in eval/, including depth,

The following are a set of shared arguments to use with any of the evaluation scripts above.

• -l </PATH/TO/MODEL/CKPT> indicates which model checkpoint to be evaluated.
• --depth_model <MODEL_NAME> specifies which depth model ("litemono" or "monodepthv2") to use, with default "litemono".
• -d <DATASET_NAME> specifies which dataset ("DDAD", "nuscenes", or "kitti") to evaluate on, and the default is "DDAD".
• --eval_dir defines the output directory where the results would be saved, with default "./outputs".

Note: To access the trained models for Waymo Open, please fill out the Google Form, and raise an issue if we don't get back to you in two days. Please note that Waymo open dataset is under strict non-commercial license so we are not allowed to share the model with you if it will used for any profit-oriented activities.

📊 Depth

eval/depth.py evaluates monocular depth estimation, with results saved in ./outputs/<CKPT>_<DATASET>/depth/.

🔹 To replicate the results reported in the paper (Table 1 and 2), run the following lines.

## === Missing checkpoints will be downloaded automatically === ##

python3 eval/depth.py -l ckpt/W_Dynamo-Depth                                  ## please fill out the form for ckpt!!
python3 eval/depth.py -l ckpt/W_Dynamo-Depth_MD2 --depth_model monodepthv2    ## please fill out the form for ckpt!!
python3 eval/depth.py -l ckpt/N_Dynamo-Depth -d nuscenes
python3 eval/depth.py -l ckpt/N_Dynamo-Depth_MD2 --depth_model monodepthv2 -d nuscenes
python3 eval/depth.py -l ckpt/K_Dynamo-Depth -d kitti
python3 eval/depth.py -l ckpt/K_Dynamo-Depth_MD2 --depth_model monodepthv2 -d kitti

Model	Dataset	Abs Rel	Sq Rel	RMSE	RMSE log	delta < 1.25	delta < 1.252	delta < 1.253
KITTI_MD2	KITTI	0.117	0.842	4.848	0.193	0.869	0.958	0.982
[KITTI_LM]	KITTI	0.107	0.824	4.648	0.184	0.886	0.962	0.983
nuScenes_MD2	nuScenes	0.145	1.416	7.092	0.245	0.802	0.921	0.967
[nuScenes_LM]	nuScenes	0.147	1.423	6.871	0.243	0.800	0.922	0.968
DDAD_MD2	DDAD	0.166	3.643	17.291	0.286	0.764	0.902	0.949
[DDAD_LM]	DDAD	0.152	3.519	14.684	0.244	0.805	0.928	0.968

(*) Very minor differences compared to the results in the paper. Rest of the checkpoints are consistent with the paper.
(†) Please refer to the note above for obtaining access to the models trained on Waymo Open Dataset.

🔹 To replicate the results reported in the Appendix (Table 6 and 7), run the following lines.

## === Missing checkpoints will be downloaded automatically === ##

python3 eval/depth.py -l ckpt/N_Dynamo-Depth -d nuscenes --split nuscenes_dayclear
python3 eval/depth.py -l ckpt/N_Dynamo-Depth_MD2 --depth_model monodepthv2 -d nuscenes --split nuscenes_dayclear

Note that by adding --split nuscenes_dayclear, we evaluate on the nuScenes day-clear subset as defined in splits/nuscenes_dayclear/test_files.txt instead of the original splits/nuscenes/test_files.txt

Notice

Our complete code will be revised after the paper is published. The test program is complete and some weights have been uploaded.