Skip to content

Commit c500ac6

Browse files
hiroyuki-sakamotohiroyuki-sakamoto
committed
Update to Version 1.0.3. Add AI Sample Applications (Hand Landmark Localization, Face Expression Recognition, Classification) in how-to.
1 parent c8c6a04 commit c500ac6

File tree

104 files changed

+3519
-4889
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

104 files changed

+3519
-4889
lines changed

‎.gitattributes

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +0,0 @@
1-
how-to/sample_app/exe/hrnet_onnx/deploy.so filter=lfs diff=lfs merge=lfs -text
2-
how-to/sample_app/exe/yolov2_onnx/deploy.so filter=lfs diff=lfs merge=lfs -text
3-
how-to/sample_app/exe/yolov3_onnx/deploy.so filter=lfs diff=lfs merge=lfs -text

‎apps/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ Provided fixed sequence is as follows.
190190

191191
| No. | Function | Details |
192192
|:---|:---|:---|
193-
| 1 |conv_yuv2rgb |Convert YUY2 to RGB. <br>Default input size is 4196x2160.|
193+
| 1 |conv_yuv2rgb |Convert YUY2 to RGB. <br>Default input size is 4096x2160.|
194194
| 2 |resize |Resize to specified size. <br>Default is 640x640. |
195195
| 3 |cast_to_fp16 | Cast data to FP16 for DRP-AI.|
196196
| 4 |normalize | Normalize pixel values with mean and standard deviation. <br>Default value are mean=[0, 0, 0] and std=[1/255, 1/255, 1/255].|

‎docs/Model_List.md

Lines changed: 166 additions & 109 deletions
Large diffs are not rendered by default.

‎how-to/README.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,21 +21,21 @@ This directory contains the solution to specific problems related to DRP-AI TVM[
2121
</tr>
2222
<tr>
2323
<td align="center"><a href="sample_app/docs/face_detection/ultraface"><img src="img/face_detection.png"></a></td>
24-
<td align="center"><a href="sample_app/docs"><img src="img/2d_hand_estimation_dev.png"></a></td>
25-
<td align="center"><a href="sample_app/docs"><img src="img/expression_dev.png"></a></td>
24+
<td align="center"><a href="sample_app/docs/hand_landmark_localization/hrnetv2"><img src="img/2d_hand_estimation.png"></a></td>
25+
<td align="center"><a href="sample_app/docs/emotion_recognition/emotion_ferplus"><img src="img/emotion.png"></a></td>
2626
</tr>
2727
<tr>
2828
<td align="center"><a href="sample_app/docs/face_detection/ultraface">Face Detection</a></td>
29-
<td align="center">Hand Landmark Localization </td>
30-
<td align="center">Face Expression Recognition </td>
29+
<td align="center"><a href="sample_app/docs/hand_landmark_localization/hrnetv2">Hand Landmark Localization</a></td>
30+
<td align="center"><a href="sample_app/docs/emotion_recognition/emotion_ferplus">Emotion Recognition</a></td>
3131
</tr>
3232
<tr>
33-
<td align="center"><a href=""><img src="img/classification_dev.png"></a></td>
33+
<td align="center"><a href="sample_app/docs/classification/googlenet"><img src="img/classification.png"></a></td>
3434
<td align="center"><a href=""><img src="img/semantic_segmentation_dev.png"></a></td>
3535
<td align="center"><a href=""><img src="img/age_classification_dev.png"></a></td>
3636
</tr>
3737
<tr>
38-
<td align="center">Classification </td>
38+
<td align="center"><a href="sample_app/docs/classification/googlenet">Classification</a></td>
3939
<td align="center">Semantic Segmentation </td>
4040
<td align="center">Age Classification </td>
4141
</tr>

‎how-to/img/2d_hand_estimation.png

17.7 KB
Loading

‎how-to/img/2d_hand_estimation_dev.png

-11.1 KB
Binary file not shown.

‎how-to/img/3d_pose_estimation_dev.png

-9.25 KB
Binary file not shown.

‎how-to/img/classification.png

5.01 KB
Loading

‎how-to/img/classification_dev.png

-6.56 KB
Binary file not shown.

‎how-to/img/emotion.png

17.2 KB
Loading

‎how-to/img/face_recognition_dev.png

-25.8 KB
Binary file not shown.

‎how-to/sample_app/README.md

Lines changed: 60 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
## Overview
44
This page explains about the sample application for DRP-AI TVM[^1] that uses USB Camera as an input and transfer the result via HTTP to display on HTML.
55
Sample application code and its execution environment are provided in this directory.
6-
This application is for RZ/V2MA Evaluation Board Kit.
6+
This application is for **RZ/V2MA Evaluation Board Kit**.
77

88
<img src=./img/web.JPG width=500>
99

@@ -44,19 +44,14 @@ In `src` directory, followings are provided.
4444

4545
### Execution Environment
4646
In `exe` directory, following files are provided as execution environment to be placed on target board.
47+
**Note that Model Object files (DRP-AI TVM[^1] compile result) are not provided.**
4748

4849
| File/Directory | Details |
4950
|:---|:---|
50-
|face_deeppose_pt/ | DeepPose model for DRP-AI mode. |
51-
|face_deeppose_cpu/ | DeepPose model for CPU mode. |
52-
|yolov3_onnx/ | YOLOv3 model for DRP-AI mode. |
53-
|yolov2_onnx/ | YOLOv2 model for DRP-AI mode. |
54-
|tinyyolov3_onnx/ | Tiny YOLOv3 model for DRP-AI mode. |
55-
|tinyyolov2_onnx/ | Tiny YOLOv2 model for DRP-AI mode. |
56-
|hrnet_onnx/ | HRNet model for DRP-AI mode. |
5751
|preprocess_tvm_v2ma/ | Pre-processing Runtime Object files. |
5852
|sample_app_drpai_tvm_usbcam_http | Application itself. |
59-
53+
|coco-labels-2014_2017.txt | Label list for Object Detection. |
54+
|synset_words_imagenet.txt | Label list for Classification. |
6055

6156
In `etc` directory, following files are provided as execution environment to be placed on client PC that displays HTTP result.
6257

@@ -79,8 +74,64 @@ Please refer to [Application Example](../../apps/README.md#how-to-compile-the-ap
7974
Please make sure to change the SDK path to the one generated in 1.
8075

8176
## Run the application
77+
Before running the application, please compile the AI models to generate following directories according to the instruction provided in each "How to create Model Object" column.
78+
Copy the generated directories to `exe` directory above so that Model Objet directories are placed in the same directory as `sample_app_drpai_tvm_usbcam_http` application.
79+
80+
| File/Directory | Details | How to create Model Object |
81+
|:---|:---|:---|
82+
|face_deeppose_pt/ | DeepPose model for DRP-AI mode. |[Face Landmark Localization](docs/face_landmark_localization/deeppose) |
83+
|face_deeppose_cpu/ | DeepPose model for CPU mode. |[Face Landmark Localization](docs/face_landmark_localization/deeppose) |
84+
|yolov3_onnx/ | YOLOv3 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
85+
|yolov2_onnx/ | YOLOv2 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
86+
|tinyyolov3_onnx/ | Tiny YOLOv3 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
87+
|tinyyolov2_onnx/ | Tiny YOLOv2 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
88+
|ultraface_onnx/ | UltraFace model for DRP-AI mode.|[Face Detection](docs/face_detection/ultraface) |
89+
|hrnet_onnx/ | HRNet model for DRP-AI mode.|[Human Pose Estimation](docs/human_pose_estimation/hrnet) |
90+
|hrnetv2_pt/ | HRNetv2 model for DRP-AI mode. |[Hand Landmark Localization](docs/hand_landmark_localization/hrnetv2) |
91+
|emotion_fp_onnx/ | Emotion FERPlus model for DRP-AI mode. |[Emotion Recognition](docs/emotion_recognition/emotion_ferplus) |
92+
|googlenet_onnx/ | GoogleNet model for DRP-AI mode. |[Classification](docs/classification/googlenet) |
93+
94+
Filesystem on the board should look like below.
95+
```sh
96+
/
97+
├── usr/
98+
│ └── lib64/
99+
│ └── libtvm_runtime.so
100+
└── home/
101+
└── root/
102+
└── exe/
103+
├── face_deeppose_pt/
104+
│ ├── deploy.json
105+
│ ├── deploy.params
106+
│ └── deploy.so
107+
├── face_deeppose_cpu/
108+
│ ... Other Model Object directories ...
109+
│
110+
├── preprocess_tvm_v2ma/
111+
├── coco-labels-2014_2017.txt
112+
├── synset_words_imagenet.txt
113+
└── sample_app_drpai_tvm_usbcam_http
114+
```
115+
82116
To run the application, please refer to the instruction for USB Camera HTTP version application in RZ/V2MA DRP-AI Sample Application Note provided in RZ/V2MA DRP-AI Support Package.
83117

118+
## Note
119+
When the web browser is closed to terminate the application following error may appear on the console.
120+
```sh
121+
[ERROR] Failed to enqueue _capture buffer.
122+
Send application message to web client.[Failed to enqueue _capture buffer.
123+
Restart the application.]
124+
125+
...
126+
127+
********************** END *********************
128+
***** Stop Recognize. *****
129+
<<<<<<<<<<<<<<<<<<<<< Message Thread Terminated >>>>>>>>>>>>>>>>>>
130+
All Finish
131+
```
132+
Since the application has already been terminated, the error does not affect the application.
133+
Application can be restarted without any special procedures, i.e. reboot the board.
134+
84135
## Application Specification
85136
### Model Information
86137
Please refer to [AI Sample Application](../README.md#ai-sample-application) for model information.

‎how-to/sample_app/docs/classification/googlenet/README.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Classification
2+
3+
### Model: [GoogleNet](#model-information)
4+
Sample application code and its execution environment are provided in **[here](../../../../sample_app)**.
5+
6+
## Overview
7+
This page explains about Classification in the [sample application](../../../../sample_app) for DRP-AI TVM[^1].
8+
9+
<img src=./img/googlenet.jpg width=500>
10+
11+
## Model Information
12+
- GoogleNet: [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/classification/inception_and_googlenet/googlenet) googlenet-9.onnx
13+
Dataset: [ILSVRC2014](https://image-net.org/challenges/LSVRC/2014/)
14+
Input size: 1x3x224x224
15+
Output size: 1x1000
16+
17+
### How to compile the model
18+
To run the Classification, `googlenet_onnx` Model Object is required.
19+
Follow the instuction below to prepare the Model Object.
20+
21+
1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
22+
2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/classification/inception_and_googlenet/googlenet).
23+
3. Place the onnx file in `$TVM_HOME/../tutorials`.
24+
4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.
25+
5. Run the with the command below.
26+
```sh
27+
$ python3 compile_onnx_model.py \
28+
-i data_0 \
29+
-s 1,3,224,224 \
30+
-o googlenet_onnx \
31+
googlenet-9.onnx
32+
```
33+
6. Confirm that `googlenet_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
34+
7. Before running the application, make sure to copy the `googlenet_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
35+
36+
37+
## Processing Details
38+
### DRP-AI mode
39+
- Source Code: [tvm_drpai_googlenet.cpp](../../../src/recognize/googlenet/tvm_drpai_googlenet.cpp)
40+
41+
Followings are processing details if user selected "GoogleNet (DRP-AI)".
42+
43+
#### Pre-processing
44+
Pre-processing is done by DRP-AI Pre-processing Runtime and CPU.
45+
46+
| Function | Details |
47+
|:---|:---|
48+
|conv_yuv2rgb |Convert YUY2 to RGB processed by DRP-AI Pre-processing Runtime.|
49+
|resize |Resize to 224x224 processed by DRP-AI Pre-processing Runtime.|
50+
|cast_to_fp16 | Cast data to FP16 for DRP-AI processed by DRP-AI Pre-processing Runtime.|
51+
|normalize | Normalize pixel values with mean values of {123.68, 116.779, 103.939}</br>processed by DRP-AI Pre-processing Runtime.|
52+
|transpose | Transpose HWC to CHW order processed by DRP-AI Pre-processing Runtime. |
53+
|cast_fp16_fp32 | Cast FP16 data to FP32 for DRP-AI TVM[^1] input</br> processed by DRP-AI Pre-processing Runtime.|
54+
|rgb2bgr | Convert RGB to BGR processed by CPU.|
55+
56+
#### Inference
57+
The Object files `googlenet_onnx` is generated from ONNX Model Zoo GoogleNet pre-trained model as described in [Model Information](#model-information).
58+
59+
#### Post-processing
60+
Post-processing is processed by CPU.
61+
62+
---
63+
[^1]: DRP-AI TVM is powered by EdgeCortix MERA™ Compiler Framework.

‎how-to/sample_app/docs/classification/googlenet/img/googlenet.jpg

84.2 KB
Loading

‎how-to/sample_app/docs/emotion_recognition/emotion_ferplus/README.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Emotion Recognition
2+
3+
### Model: [Emotion FERPlus](#model-information)
4+
Sample application code and its execution environment are provided in **[here](../../../../sample_app)**.
5+
6+
## Overview
7+
This page explains about Emotion Recognition in the [sample application](../../../../sample_app) for DRP-AI TVM[^1].
8+
9+
<img src=./img/emotionfp.jpg width=500>
10+
11+
## Model Information
12+
- Emotion FERPlus: [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus) emotion-ferplus-8.onnx
13+
Dataset: See [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus#dataset).
14+
Input size: 1x1x64x64
15+
Output size: 1x8
16+
17+
Emotion FERPlus can only classify the face expression of single person.
18+
To enable multiple face emotion recognition, this application used [UltraFace](../../../docs/face_detection/ultraface/) as pre-processing.
19+
To see more details on UltraFace, please see [Face Detection](../../../docs/face_detection/ultraface/).
20+
21+
22+
### How to compile the model
23+
To run the Emotion Recognition, `emotion_fp_onnx` Model Object and `ultraface_onnx` Model Object are required.
24+
Follow the instuction below to prepare the `emotion_fp_onnx` Model Object.
25+
For `ultraface_onnx` Model Object, please refer to [Face Detection](../../../docs/face_detection/ultraface/).
26+
27+
28+
1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
29+
2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus).
30+
3. Place the onnx file in `$TVM_HOME/../tutorials`.
31+
4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x442d0000`.
32+
Note that the value **must NOT** be default value `0x438E0000` in order to avoid conflict with UltraFace Model Object.
33+
5. Run the with the command below.
34+
```sh
35+
$ python3 compile_onnx_model.py \
36+
-i Input3 \
37+
-s 1,1,64,64 \
38+
-o emotion_fp_onnx \
39+
emotion-ferplus-8.onnx
40+
```
41+
6. Confirm that `emotion_fp_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
42+
7. Before running the application, make sure to copy the `emotion_fp_onnx` directory and `ultraface_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
43+
44+
45+
## Processing Details
46+
### DRP-AI mode
47+
- Source Code: [tvm_drpai_emotionfp.cpp](../../../src/recognize/emotionfp/tvm_drpai_emotionfp.cpp)
48+
49+
Followings are processing details if user selected "Emotion FERPlus (DRP-AI)".
50+
51+
#### Pre-processing
52+
As a pre-processing, Face Detection model, UltraFace, is used.
53+
To see details, please refer to [Face Detection Processing Details](../../../docs/face_detection/ultraface/README.md#processing-details).
54+
55+
For each face detected, following pre-processing is done by CPU..
56+
Note that some of them are processed by C++ OpenCV.
57+
58+
| Function | Details |
59+
|:---|:---|
60+
|Crop | Crop YUYV image. Processed by CPU. |
61+
|cvtColor | C++ OpenCV. Convert YUY2 to Grayscale.|
62+
|resize |C++ OpenCV. Resize to 64x64.|
63+
|transpose |Transpose HWC to CHW order. Processed by CPU.|
64+
65+
#### Inference
66+
The Object files `emotion_fp_onnx` is generated from ONNX Model Zoo Emotion FERPlus pre-trained model as described in [Model Information](#model-information).
67+
68+
#### Post-processing
69+
Post-processing is processed by CPU.
70+
71+
72+
#### About processing time
73+
Details of processing time, which is displayed on web browser, are as follow.
74+
75+
| Processing | Details |
76+
|:---|:---|
77+
|Pre-processing | Sum of time taken for following operations. </br>- Face Detection pre-processing, inference and postprocessing</br>- Emotion recognition pre-processing for all detected faces. |
78+
|Inferene | Time taken to run inference for all detected faces.|
79+
|Post-processing |Time taken to run post-processing for all detected faces.|
80+
81+
For example, if there are two bounding box detected in face detection, emotion recognition will be carried out for two times.
82+
Therefore, inference time will be approximately two times by single inference processing time and same applies for other processing time.
83+
84+
---
85+
[^1]: DRP-AI TVM is powered by EdgeCortix MERA™ Compiler Framework.

‎how-to/sample_app/docs/emotion_recognition/emotion_ferplus/img/emotionfp.jpg

112 KB
Loading

‎how-to/sample_app/docs/face_detection/ultraface/README.md

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,27 @@ Dataset: See [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/bo
1414
Input size: 1x3x240x320
1515
Output size: 1x4420x2, 1x4420x4
1616

17+
18+
### How to compile the model
19+
To run the Face Detection, `ultraface_onnx` Model Object is required.
20+
Follow the instuction below to prepare the `ultraface_onnx` Model Object.
21+
22+
1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
23+
2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/ultraface).
24+
3. Place the onnx file in `$TVM_HOME/../tutorials`.
25+
4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.
26+
5. Run the with the command below.
27+
```sh
28+
$ python3 compile_onnx_model.py \
29+
-i input \
30+
-s 1,3,240,320 \
31+
-o ultraface_onnx \
32+
version-RFB-320.onnx
33+
```
34+
6. Confirm that `ultraface_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
35+
7. Before running the application, make sure to copy the `ultraface_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
36+
37+
1738
## Processing Details
1839
### DRP-AI mode
1940
- Source Code: [tvm_drpai_ultraface.cpp](../../../src/recognize/ultraface/tvm_drpai_ultraface.cpp)
@@ -34,7 +55,6 @@ Pre-processing is done by DRP-AI Pre-processing Runtime, which allows following
3455

3556
#### Inference
3657
The Object files `ultraface_onnx` is generated from ONNX Model Zoo pre-trained model as described in [Model Information](#model-information).
37-
Please refer to [Compile Tutorial](../../../../../tutorials) for more details on compiling model.
3858

3959
#### Post-processing
4060
Post-processing is processed by CPU.

‎how-to/sample_app/docs/face_detection/ultraface/img/ultraface.jpg

-92 KB
Loading

0 commit comments

Comments
 (0)