BEVFormer: a Cutting-edge Baseline for Camera-based Detection

注：本代码库在BEVFormer算法所提供的代码的基础上新增了BEV分割代码(Note: This code library adds BEV segmentation code based on the code provided by the BEVFormer algorithm)

4.18.1.mp4

Abstract

In this work, the authors present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, the authors design a spatial cross-attention that each BEV query extracts the spatial features from the regions of interest across camera views. For temporal information, the authors propose a temporal self-attention to recurrently fuse the history BEV information. The proposed approach achieves the new state-of-the-art 56.9% in terms of NDS metric on the nuScenes test set, which is 9.0 points higher than previous best arts and on par with the performance of LiDAR-based baselines.

Methods

Getting Started

• Installation
• Prepare Dataset
• Run and Eval

Model Zoo

Backbone	Method	Lr Schd	NDS	mAP	memroy	Config	Download
R50	BEVFormer-tiny_fp16	24ep	35.9	25.7	-	config	model/log
R50	BEVFormer-tiny	24ep	35.4	25.2	6500M	config	model/log
R101-DCN	BEVFormer-small	24ep	47.9	37.0	10500M	config	model/log
R101-DCN	BEVFormer-base	24ep	51.7	41.6	28500M	config	model/log

origin

• 3D Detection checkpoints
• 3D Detection code
• Initialization

New

• BEV Segmentation checkpoints
• BEV Segmentation code
• multi-task visualization
• weight

改进(Improve)

map分支：仅训练分割分支，目前只支持单卡训练(map branch: only training split branch, currently only supports single card training)

model	Epoch	Divider	pred Crossing	Boundary	mIoU	weight
BEVFomer-small	5	0.4684	0.3328	0.4775	0.4262	model

双分支：实现了检测与分割的联合训练，并支持并行(Double branch: realize the joint training of detection and segmentation, and support parallelism)

det：

model	Epoch	mAP	NDS	weight
BEVFomer-small-150x150	19	0.3820	0.4869	model
BEVFomer-small-300x300	18	0.391	0.5031	model
BEVFomer-base-150x150	23	0.4186	0.5132	model

seg：

model	Epoch	Divider	pred Crossing	Boundary	mIoU	weight
BEVFomer-small-150x150	19	0.4531	0.3105	0.469	0.4109	model
BEVFomer-small-300x300	18	0.4329	0.322	0.4845	0.4132	model
BEVFomer-base-150x150	23	0.4881	0.3392	0.4955	0.4409	model

Acknowledgement

本项目参考了BEVerse, open-mmlab, BEVDet, HDMapNet等代码库，在这里表示感谢！(This project refers to code libraries such as BEVerse, open-mmlab, BEVDet, HDMapNet, etc. Thank you here!)