CooTest

This is the official implementation of paper. "CooTest: An Automated Testing Approach for V2X Communication Systems".

Installation

All experiments are conducted on a server with an Intel i7-10700K CPU(3.80 GHz), 32 GB RAM, and an NVIDIA GeForce RTX 4070 GPU (12GB VRAM).

Init Dataset

You need to check the V2V4real website and download the test datasets test1, test2, test3.

Basic Dependency

To set up the codebase environment, do the following steps:

Create conda environment (python >= 3.7)

conda create -n v2v4real python=3.7
conda activate v2v4real

Pytorch Installation (>= 1.12.0 Required)

conda install pytorch==1.12.0 torchvision==0.13.0 cudatoolkit=11.3 -c pytorch -c conda-forge

spconv 2.x Installation

pip install spconv-cu113

Install other dependencies

pip install -r requirements.txt
python setup.py develop

Install bbx nms calculation cuda version

python opencood/utils/setup.py build_ext --inplace

Model Donwload

You need to download the cooperative 3D detection models here, and unzip them in the model folder.

model
├── attfuse
├── early_fusion
├── late_fusion
├── PointPillar_Fcooper
├── PointPillar_V2VNet
└── PointPillar_V2XViT

Experiments

RQ1

You need to get the detecting boxes and scores of nofusion first.

python rq_test/rq1_inference.py --dataset_dir ${dataset}/rq1 --model_dir model/late_fusion--fusion_method nofusion

Generate transformation data and test the erroneous behaviors of cooperative perception tasks using MRs.

python rq_test/rq1_inference.py --dataset_dir ${dataset}/test --model_dir model/${MODEL}--fusion_method ${FUSION_STRATEGY} --data_augment True 

Arguments Explanation:

• model_dir: the path to your saved model, e.g. model/early, meaning you want to use "early_fusion "model for test. See Tutorial 1: Config System to learn more about the rules of the yaml files.
• fusion_method: indicate the fusion strategy, currently support 'nofusion', 'early', 'late', and 'intermediate'.
• dataset_dir: the "test" dataset path.

RQ2

1. Split dataset

Half of the sequences are randomly selected and saved as a training set for retrain and a test set for testing.

$ python rq_test/rq2_dataset_split.py --dataset_dir ${dataset_path}

2. Generate transformation data

You need to get the detecting boxes and scores of nofusion first.

python rq_test/rq2_inference.py --dataset_dir ${dataset}/rq2 --model_dir model/late_fusion--fusion_method nofusion

Generate augment data and test the erroneous behaviors of cooperative perception tasks using MRs.

python rq_test/rq2_inference.py --dataset_dir ${dataset}/test --model_dir model/${MODEL}--fusion_method ${FUSION_STRATEGY}

3. Using V2X-oriented guided transformation select data for retrain

You need to change the configuration file：

python rq_test/rq2_inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} --data_select formula --data_augment True 

RQ3

Retrain with transformed tests of rq2

python rq_test/rq3_train.py --dataset_dir ${dataset}/rq2/rq2_select --model_dir model/${MODEL}

Test the effect of retrain models

python rq_test/rq3_inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} --data_select formula --data_augment True 

Dataset structure

After generate the transformation datasets, the recommended dataset format like this:

├── v2v4real
│   ├── test
│   ├── rq1
│      ├── rq1_det_box
│   ├── rq2
│      ├── rq2_select
│      ├── rq2_det_box
│   ├── rq3
│      ├── rq3_test
└──    ├── rq3_det_box

Citation

@inproceedings{guo2024cootest,
  title={CooTest: An Automated Testing Approach for V2X Communication Systems},
  author={Guo, An and Gao, Xinyu and Chen, Zhenyu and Xiao, Yuan and Liu, Jiakai and Ge, Xiuting and Sun, Weisong and Fang, Chunrong},
  booktitle={Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis},
  pages={1453--1465},
  year={2024}
}