DistilBert.md

PyTorch Transformer model DistilBert-base-uncased for Natural Language Classification and Question Answering

This document describes evaluation of optimized checkpoints for transformer models DistilBert-base-uncased for NL Classification and Question Answering tasks.

AIMET installation and setup

Please install and setup AIMET (Torch GPU variant) before proceeding further.

NOTE

• All AIMET releases are available here: https://github.com/quic/aimet/releases
• This model has been tested using AIMET version 1.23.0 (i.e. set release_tag="1.23.0" in the above instructions).
• This model is compatible with the PyTorch GPU variant of AIMET (i.e. set AIMET_VARIANT="torch_gpu" in the above instructions).

Additional Setup Dependencies

pip install datasets==2.4.0
pip install transformers==4.11.3 

Model checkpoint

• Original full precision checkpoints without downstream training were downloaded through hugging face
• [Full precision model with downstream training weight files] are automatically downloaded using evaluation script
• [Quantization optimized model weight files] are automatically downloaded using evaluation script

Dataset

• For NLP tasks, we use the General Language Understanding Evaluation (GLUE) benchmark dataset for evaluation.
• For Question Answering tasks, we use the Stanford Question Answering Dataset (SQuAD) benchmark dataset for evaluation.
• Dataset downloading is handled by evaluation script

Usage

To run evaluation with QuantSim in AIMET, use the following

python distilbert_quanteval.py \
        --model_config <MODEL_CONFIGURATION> \
        --per_device_eval_batch_size 4 \
        --output_dir <OUT_DIR> \

• example

  python distilbert_quanteval.py --model_config distilbert_w8a8_rte --per_device_eval_batch_size 4 --output_dir ./evaluation_result 
  

• supported values of model_config are "distilbert_w8a8_rte","distilbert_w8a8_stsb","distilbert_w8a8_mrpc","distilbert_w8a8_cola","distilbert_w8a8_sst2","distilbert_w8a8_qnli","distilbert_w8a8_qqp","distilbert_w8a8_mnli", "distilbert_w8a8_squad"

Quantization Configuration

The following configuration has been used for the above models for INT8 quantization:

• Weight quantization: 8 bits, symmetric quantization
• Bias parameters are not quantized
• Activation quantization: 8 bits, asymmetric quantization
• Model inputs are quantized
• TF range learning was used as quantization scheme
• Mask values of -6 was applied in attention layers
• Quantization aware training (QAT) was used to obtain optimized quantized weights, detailed hyperparameters listed in Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort, "Understanding and Overcoming the Challenges of Efficient Transformer Quantization", EMNLP 2021.

Results

Below are the results of the Pytorch transformer model DistilBert for GLUE dataset:

Configuration	CoLA (corr)	SST-2 (acc)	MRPC (f1)	STS-B (corr)	QQP (acc)	MNLI (acc)	QNLI (acc)	RTE (acc)	GLUE
FP32	53.85	91.17	88.40	87.12	90.39	87.29	82.15	65.34	80.71
W8A8	52.99	90.48	89.34	86.76	89.77	86.88	83.35	65.54	80.26