Skip to content

Commit

Permalink
Update readme. Add patch for hexagon_nn_3.5.4 shared libs. Add patch …
Browse files Browse the repository at this point in the history 
…for ImagenetCv.
  • Loading branch information
@JsGjKJzi
JsGjKJzi committed Apr 26, 2024
1 parent 68b77ac commit db89901
Show file tree
Hide file tree
Showing 13 changed files with 476 additions and 0 deletions.
28 changes: 28 additions & 0 deletions RB5/linux_kernel_5_x/AI-ML-apps/HexgonSDK-Image-classification/LICENSE
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
Copyright (c) 2020 Qualcomm Innovation Center, Inc. All Rights Reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted
provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this list of conditions
and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of
conditions and the following disclaimer in the documentation and/or other materials
provided with the distribution.
* Neither the name of the copyright holder nor the names of its contributors may be used to
endorse or promote products derived from this software without specific prior written
permission.

NO EXPRESS OR IMPLIED LICENSES TO ANY PARTY'S PATENT RIGHTS ARE GRANTED BY THIS
LICENSE. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Where there is uncertainty as to how, or
where, to apply marks, open an OSR to escalate to OSG for review.

SPDX-License-Identifier: BSD-3-Clause-Clear
329 changes: 329 additions & 0 deletions RB5/linux_kernel_5_x/AI-ML-apps/HexgonSDK-Image-classification/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,329 @@
# Inception V3 on Qualcomm Robotics RB5 DSP

This project is designed to show how you can use the Qualcomm® Robotics RB5 development kit, specifically using the Qualcomm® Hexagon™ DSP with Hexagon Vector extensions (HVX), coupled with Hexagon DSP SDK to achieve high speed and high performance on device Machine Learning.

## Getting Started guide

### Step 1: Setting up the Host Linux Machine

Build Hexagon NN DSP library for the Qualcomm Robotics RB5 CDSP

- Download the Hexagon SDK version 3.5.4 for Linux from here.

- https://developer.qualcomm.com/software/hexagon-dsp-sdk

- Unzip the file and install the SDK using the following commands

```
cd qualcomm_hexagon_sdk_3_5_4_linux/
chmod +x qualcomm_hexagon_sdk_3_5_4_eval.bin
./qualcomm_hexagon_sdk_3_5_4_eval.bin
```

- Set the environment variable $HEXAGON_SDK_ROOT by the command

```
source ~/Qualcomm/Hexagon_SDK/3.4.2/setup_sdk_env.source
```

![image text](image/01_Set_ENV.png)

- Compile Hexagon NN using the following commands and generate a hexagon library

```
cd $HEXAGON_SDK_ROOT/libs/hexagon_nn/2.10.1/
make tree VERBOSE=1 CDSP_FLAG=1 V=hexagon_Release_dynamic_toolv83_v66 V66=1
```

![image text](image/02_make.png)

The libhexagon_nn_skel.so generated here is the shared library for the CDSP

### Step 2: Generate Testsig and sign the previously generated shared library-libhexagon_nn_skel.so with it.

- To generate a Testsig, the serial number of Target robotics development kit is needed.

- Get serial number from the robotics development kit ’s console using command.

```
cd $HEXAGON_SDK_ROOT/tools/elfsigner/getserial/UbuntuARM_Release_aarch64
adb push getserial /
adb shell chmod 777 getserial
adb shell
./getserial
```

The above command returns the serial number of the attached device:

```
####################Serial number (see below)###########################
Serial Num : 0x7f7595b7
####################Serial number (see above)###########################
```

- Pass the serial_number to elfsigner.py using the following commands:

```
python $HEXAGON_SDK_ROOT/tools/elfsigner/elfsigner.py -t 0x<serial number>
```

![image text](image/04_elfsigner.png)

- Sign the shared library libhexagon_nn_skel.so using elfsigner.py

```
cd $HEXAGON_SDK_ROOT
mkdir signed_libs
python tools/elfsigner/elfsigner.py -i libs/hexagon_nn/2.10.1/hexagon_Release_dynamic_toolv83_v66/ship/libhexagon_nn_skel.so -o signed_libs/
```

![image text](image/05_libhexagon_nn_skel.png)

### Step 3: Generate Inception v3 model graph for Hexagon DSP.

In this project, you will use an Inception v3 model’s frozen graph for the image classification. In the following steps, download TensorFlow, Bazel and generate a Hexagon DSP compatible version of the Inception V3 frozen graph.

- Download TensorFlow Inception V3 model Frozen graph using

```
curl http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz -o /tmp/inceptionv3.tgz
```

![image text](image/06_Download-TensorFlow.png)

and unzip it to /tmp directory.

```
tar -xzf /tmp/inceptionv3.tgz -C /tmp/
```

Download and install Bazel (version 0.20.0) using the following commands:

```
wget https://github.com/bazelbuild/bazel/releases/download/0.20.0/bazel-0.20.0-installer-linux-x86_64.sh
chmod +x bazel-0.20.0-installer-linux-x86_64.sh
./bazel-0.20.0-installer-linux-x86_64.sh --user
```

![image text](image/07_install_Bazel.png)
![image text](image/08_make_Bazel.png)

Download and build TensorFlow(version 1.13.2) using the following commands

```
wget https://github.com/tensorflow/tensorflow/archive/v1.13.2.tar.gz
tar -zxvf v1.13.2.tar.gz
cd tensorflow-1.13.2
TF_ENABLE_XLA=y \
TF_NEED_OPENCL_SYCL=n \
TF_NEED_ROCM=n \
TF_NEED_CUDA=n \
TF_DOWNLOAD_CLANG=n \
TF_NEED_MPI=n \
CC_OPT_FLAGS='-march=native -Wno-sign-compare' \
TF_SET_ANDROID_WORKSPACE=n \
./configure
```

Build transform_graph tool in TensorFlow using the following commands. The tensorflow binaries have changed, so we need up update the expected hash in the bazel file.

```
touch WORKSPACE
sed -i 's/e15ffd84606323cbad5515bf9ecdf8061cc3bf80fb883b9e6aa162e485aa9761/86b85fbf1b251d7a658de86ce5a0c8f34151027cc60b01e1b76f167379acf181/g' third_party/icu/workspace.bzl
bazel build tensorflow/tools/graph_transforms:transform_graph
```

Quantize protobuf using Graph Transform Tool

```
bazel-bin/tensorflow/tools/graph_transforms/transform_graph --in_graph=/tmp/classify_image_graph_def.pb --out_graph=/tmp/inception_v3_quantized.pb --inputs="Mul" --outputs='softmax' --transforms='add_default_attributes strip_unused_nodes(type=float, shape="1,299,299,3") remove_nodes(op=Identity, op=CheckNumerics) fold_constants(ignore_errors=true) fold_batch_norms fold_old_batch_norms quantize_weights quantize_nodes fold_constants strip_unused_nodes sort_by_execution_order'
```

Convert the quantized protobuf into a C file (iv3.c) using the following commands

```
cd $HEXAGON_SDK_ROOT/examples/hexagon_nn/
apt-get install virtualenv
virtualenv -p python2 env2
```

If this gives an error “virtualenv not found”/ “pip missing”, install pip using

```
apt-get install python-pip
apt-get install python-numpy
```

and install virtualenv using

```
sudo apt-get install python-virtualenv
source env2/bin/activate
pip install --upgrade 'setuptools<45.0.0'
pip install -r $HEXAGON_SDK_ROOT/Hexagon_SDK/3.5.4/tools/python_venv/environments/req2.txt
```

Enter the command below to generate Hexagon compatible iv3.c file，

```
python $HEXAGON_SDK_ROOT/examples/hexagon_nn/scripts/tensorflow_to_hexagon_nn.py /tmp/inception_v3_quantized.pb $HEXAGON_SDK_ROOT/examples/hexagon_nn/tutorials/007-tensorflow-to-hexagon-nn/inceptionv3_v1.yaml > iv3.c
```

And copy iv3.c to the folder as shown in the command

```
cp iv3.c $HEXAGON_SDK_ROOT/libs/hexagon_nn/2.10.1/
```

### Step 4: Download linaro64

Please download gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu.tar.xz from http://releases.linaro.org/archive/14.11/components/toolchain/binaries/aarch64-linux-gnu

Extract the tar file and copy folder gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu to <Hexagon_SDK_ROOT>/tools/ folder.

Rename gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu folder to linaro64

```
cd $HEXAGON_SDK_ROOT/tools
wget https://releases.linaro.org/archive/14.11/components/toolchain/binaries/aarch64-linux-gnu/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu.tar.xz
tar -xf gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu.tar.xz
mv gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu linaro64
rm gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu.tar.xz
```

### Step 5: Compile hexagon-nn shared user space library

```
cd $HEXAGON_SDK_ROOT/libs/hexagon_nn/2.10.1/
```

- Patch Hexagon NN with the downloaded patch to compile as shared library for user space

```
patch --binary -p1 < hexagon_nn_3.5.4_shared_lib.patch
make tree V=UbuntuARM_Release_aarch64 CDSP_FLAG=1 V66=1 GRAPHINIT="iv3.c"
```

![image text](image/12_make.png)

### Step 6: Push generated files to target and run it

```
adb push $HEXAGON_SDK_ROOT/libs/hexagon_nn/2.10.1/UbuntuARM_Release_aarch64/graph_app /data/
adb push $HEXAGON_SDK_ROOT/libs/hexagon_nn/2.10.1/UbuntuARM_Release_aarch64/libhexagon_nn.so /usr/lib
adb push $HEXAGON_SDK_ROOT/signed_libs/libhexagon_nn_skel.so /usr/lib/rfsa/adsp/
adb push $HEXAGON_SDK_ROOT/libs/hexagon_nn/2.10.1/test/panda_299x299.dat /data/
adb shell chmod a+x /data/graph_app
adb shell /data/graph_app --input_to_float 1 /data/panda_299x299.dat
```

You should expect output like the following:

```
Run!
output size=4032
Rank,Softmax,index,string
0,0.940308,169,giant panda
1,0.002943,7,lesser panda
2,0.000613,61,brown bear
3,0.000534,878,earthstar
4,0.000424,374,lawn mower
AppReported: 41682408
```

### Part 2 - On the Qualcomm Robotics RB5 Development Kit (Target)

### Step 1: Make hardware connections and power up Robotics Development Kit

- Connect Robotics Development Kit with USB Camera and wireless USB for Keyboard and mouse.
- Connect Robotics Development Kit with HDMI cable and the other end of the cable to the HDMI Screen/monitor that is powered on.
- Connect the power cable to Robotics Development Kit and wait for the desktop to pop up.

### Step 2: Start cdsprpcd daemon

```
adb shell
cdsprpcd&
```

### Step 3: Build OpenCV for Wayland

The Qualcomm Robotics RB5 development board supports Wayland/Weston as the display server, OpenCV now supports Wayland but needs to be compiled with support.

> :warning: OpenCV 4.0 had some kind of memory leak when using GStreamer that resulted in memory consumption growing continuously as images were captured. OpenCV 4.9 was tested and did not display the memory growth behavior.
```
root@qrb5165-rb5:/# apt-get install build-essential curl unzip
root@qrb5165-rb5:/# apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
root@qrb5165-rb5:/# apt-get install libjpeg-dev libpng-dev
root@qrb5165-rb5:/# apt-get install python-numpy libxkbcommon-dev libwayland-client0 libwayland-dev
root@qrb5165-rb5:/# cd /data
root@qrb5165-rb5:/# git clone https://github.com/opencv/opencv.git
root@qrb5165-rb5:/# cd opencv/
root@qrb5165-rb5:/# mkdir build
root@qrb5165-rb5:/# cd build
root@qrb5165-rb5:/# cmake -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/usr/local -DWITH_IPP=OFF -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_EXAMPLES=OFF -DWITH_WAYLAND=ON -DWITH_GTK=OFF ..
root@qrb5165-rb5:/# make -j8
```

### Step 4: Run Weston desktop on Qualcomm Robotics RB5 Development kit

If the HDMI cable connects properly, after the device boot up, you will see the device boot up logs and then see a shell is ready for command input. Run the following command to launch Weston desktop.

```
root@qrb5165-rb5:/# weston --connector=29
```

If it fails to run, you can try the following commands with adb on an Ubuntu PC. Please make sure the Qualcomm Robotics RB5 development kit is connected to the Ubuntu PC with a USB cable. Please ignore this step if the above command runs successfully.

```
root@qrb5165-rb5:/# mkdir -p /usr/bin/weston_socket
root@qrb5165-rb5:/# chmod 700 /usr/bin/weston_socket
root@qrb5165-rb5:/# export XDG_RUNTIME_DIR=/usr/bin/weston_socket
root@qrb5165-rb5:/# export LD_LIBRARY_PATH=/usr/lib:/usr/lib/aarch64-linux-gnu/
root@qrb5165-rb5:/# weston --tty=1 --connector=29 --backend=drm-backend.so
```

### Step 5: Run OpenCV ImageNet application for camera

Download the application from the camera branch and build using the following commands. The ImagenetCv code is designed to use a USB camera using Video4Linux 2. The RB5 main MIPI cameras do not work with Video4Linux and are only compatible with GStreamer. So apply the patch to capture from a gstreamer pipeline.

```
root@qrb5165-rb5:/# git clone https://git.linaro.org/people/srinivas.kandagatla/ImagenetCv.git
root@qrb5165-rb5:/# cd ImagenetCv
root@qrb5165-rb5:/# git branch -a
root@qrb5165-rb5:/# git checkout camera
patch --binary -p1 < imagenetcv.patch
```

Build application using

```
root@qrb5165-rb5:/# cmake .
root@qrb5165-rb5:/# make
```

Run the application using the command

```
root@qrb5165-rb5:/# ./ImagenetCv
```

If this shows an error like “install libgtk2.0-dev”

Install libgtk2.0-dev using the commands given below and run the application

```
root@qrb5165-rb5:/# apt-get update
root@qrb5165-rb5:/# apt-get upgrade
root@qrb5165-rb5:/# apt-get install libgtk2.0-dev
```

Running the application pops up a window that shows video captured by the camera and the image classification being done
![image text](image/wayland-screenshot.png)

## License

This is licensed under the BSD 3-Clause-Clear “New” or “Revised” License. Check out the [LICENSE](LICENSE) for more details.
Loading

0 comments on commit db89901

Please sign in to comment.