Update to Version 1.0.3. Add AI Sample Applications (Hand Landmark Lo…

…calization, Face Expression Recognition, Classification) in how-to.

.gitattributes

@@ -1,3 +0,0 @@
-how-to/sample_app/exe/hrnet_onnx/deploy.so filter=lfs diff=lfs merge=lfs -text
-how-to/sample_app/exe/yolov2_onnx/deploy.so filter=lfs diff=lfs merge=lfs -text
-how-to/sample_app/exe/yolov3_onnx/deploy.so filter=lfs diff=lfs merge=lfs -text

apps/README.md

@@ -190,7 +190,7 @@ Provided fixed sequence is as follows.
 
 | No. | Function | Details |   
 |:---|:---|:---|  
-| 1 |conv_yuv2rgb |Convert YUY2 to RGB. <br>Default input size is 4196x2160.|  
+| 1 |conv_yuv2rgb |Convert YUY2 to RGB. <br>Default input size is 4096x2160.|  
 | 2 |resize |Resize to specified size. <br>Default is 640x640. |  
 | 3 |cast_to_fp16 | Cast data to FP16 for DRP-AI.|  
 | 4 |normalize | Normalize pixel values with mean and standard deviation.  <br>Default value are mean=[0, 0, 0] and std=[1/255, 1/255, 1/255].|  

how-to/README.md

@@ -21,21 +21,21 @@ This directory contains the solution to specific problems related to DRP-AI TVM[
 </tr>
 <tr>
     <td align="center"><a href="sample_app/docs/face_detection/ultraface"><img src="img/face_detection.png"></a></td>
-    <td align="center"><a href="sample_app/docs"><img src="img/2d_hand_estimation_dev.png"></a></td>
-    <td align="center"><a href="sample_app/docs"><img src="img/expression_dev.png"></a></td>
+    <td align="center"><a href="sample_app/docs/hand_landmark_localization/hrnetv2"><img src="img/2d_hand_estimation.png"></a></td>
+    <td align="center"><a href="sample_app/docs/emotion_recognition/emotion_ferplus"><img src="img/emotion.png"></a></td>
 </tr>
 <tr>
     <td align="center"><a href="sample_app/docs/face_detection/ultraface">Face Detection</a></td>
-    <td align="center">Hand Landmark Localization </td>
-    <td align="center">Face Expression Recognition </td>
+    <td align="center"><a href="sample_app/docs/hand_landmark_localization/hrnetv2">Hand Landmark Localization</a></td>
+    <td align="center"><a href="sample_app/docs/emotion_recognition/emotion_ferplus">Emotion Recognition</a></td>
 </tr>
 <tr>
-    <td align="center"><a href=""><img src="img/classification_dev.png"></a></td>
+    <td align="center"><a href="sample_app/docs/classification/googlenet"><img src="img/classification.png"></a></td>
     <td align="center"><a href=""><img src="img/semantic_segmentation_dev.png"></a></td>
     <td align="center"><a href=""><img src="img/age_classification_dev.png"></a></td>
 </tr>
 <tr>
-    <td align="center">Classification </td>
+    <td align="center"><a href="sample_app/docs/classification/googlenet">Classification</a></td>
     <td align="center">Semantic Segmentation </td>
     <td align="center">Age Classification </td>
 </tr>

how-to/sample_app/README.md

@@ -3,7 +3,7 @@
 ## Overview
 This page explains about the sample application for DRP-AI TVM[^1] that uses USB Camera as an input and transfer the result via HTTP to display on HTML.  
 Sample application code and its execution environment are provided in this directory.  
-This application is for RZ/V2MA Evaluation Board Kit.  
+This application is for **RZ/V2MA Evaluation Board Kit**.  
 
 <img src=./img/web.JPG width=500>  
 
@@ -44,19 +44,14 @@ In `src` directory, followings are provided.
 
 ### Execution Environment
 In `exe` directory, following files are provided as execution environment to be placed on target board.  
+**Note that Model Object files (DRP-AI TVM[^1] compile result) are not provided.**
 
 | File/Directory | Details |  
 |:---|:---|  
-|face_deeppose_pt/ | DeepPose model for DRP-AI mode. |  
-|face_deeppose_cpu/ | DeepPose model for CPU mode. |  
-|yolov3_onnx/ | YOLOv3 model for DRP-AI mode. |  
-|yolov2_onnx/ | YOLOv2 model for DRP-AI mode. |  
-|tinyyolov3_onnx/ | Tiny YOLOv3 model for DRP-AI mode. |  
-|tinyyolov2_onnx/ | Tiny YOLOv2 model for DRP-AI mode. |  
-|hrnet_onnx/ | HRNet model for DRP-AI mode. |  
 |preprocess_tvm_v2ma/ | Pre-processing Runtime Object files. |  
 |sample_app_drpai_tvm_usbcam_http | Application itself. |  
-
+|coco-labels-2014_2017.txt | Label list for Object Detection. |  
+|synset_words_imagenet.txt | Label list for Classification. |  
 
 In `etc` directory, following files are provided as execution environment to be placed on client PC that displays HTTP result.  
 
@@ -79,8 +74,64 @@ Please refer to [Application Example](../../apps/README.md#how-to-compile-the-ap
 Please make sure to change the SDK path to the one generated in 1.  
 
 ## Run the application
+Before running the application, please compile the AI models to generate following directories according to the instruction provided in each "How to create Model Object" column.  
+Copy the generated directories to `exe` directory above so that Model Objet directories are placed in the same directory as `sample_app_drpai_tvm_usbcam_http` application.  
+
+| File/Directory | Details | How to create Model Object |  
+|:---|:---|:---|  
+|face_deeppose_pt/ | DeepPose model for DRP-AI mode. |[Face Landmark Localization](docs/face_landmark_localization/deeppose) |  
+|face_deeppose_cpu/ | DeepPose model for CPU mode. |[Face Landmark Localization](docs/face_landmark_localization/deeppose) |  
+|yolov3_onnx/ | YOLOv3 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |  
+|yolov2_onnx/ | YOLOv2 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |  
+|tinyyolov3_onnx/ | Tiny YOLOv3 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |  
+|tinyyolov2_onnx/ | Tiny YOLOv2 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |  
+|ultraface_onnx/ | UltraFace model for DRP-AI mode.|[Face Detection](docs/face_detection/ultraface) |  
+|hrnet_onnx/ | HRNet model for DRP-AI mode.|[Human Pose Estimation](docs/human_pose_estimation/hrnet) |  
+|hrnetv2_pt/ | HRNetv2 model for DRP-AI mode. |[Hand Landmark Localization](docs/hand_landmark_localization/hrnetv2) |  
+|emotion_fp_onnx/ | Emotion FERPlus model for DRP-AI mode. |[Emotion Recognition](docs/emotion_recognition/emotion_ferplus) |  
+|googlenet_onnx/ | GoogleNet model for DRP-AI mode. |[Classification](docs/classification/googlenet) |  
+
+Filesystem on the board should look like below.  
+```sh
+/
+├── usr/
+│   └── lib64/
+│       └── libtvm_runtime.so
+└── home/
+    └── root/
+        └── exe/
+            ├── face_deeppose_pt/
+            │   ├── deploy.json
+            │   ├── deploy.params
+            │   └── deploy.so
+            ├── face_deeppose_cpu/
+            │   ... Other Model Object directories ...
+            │  
+            ├── preprocess_tvm_v2ma/
+            ├── coco-labels-2014_2017.txt
+            ├── synset_words_imagenet.txt
+            └── sample_app_drpai_tvm_usbcam_http
+```
+
 To run the application, please refer to the instruction for USB Camera HTTP version application in RZ/V2MA DRP-AI Sample Application Note provided in RZ/V2MA DRP-AI Support Package.  
 
+## Note
+When the web browser is closed to terminate the application following error may appear on the console.  
+```sh
+[ERROR] Failed to enqueue _capture buffer.
+Send application message to web client.[Failed to enqueue _capture buffer.
+Restart the application.]
+
+...
+
+********************** END *********************
+***** Stop Recognize. *****
+<<<<<<<<<<<<<<<<<<<<< Message Thread Terminated >>>>>>>>>>>>>>>>>>
+All Finish
+```
+Since the application has already been terminated, the error does not affect the application.  
+Application can be restarted without any special procedures, i.e. reboot the board.  
+
 ## Application Specification
 ### Model Information
 Please refer to [AI Sample Application](../README.md#ai-sample-application) for model information.

how-to/sample_app/docs/classification/googlenet/README.md

@@ -0,0 +1,63 @@
+# Classification
+
+### Model: [GoogleNet](#model-information)
+Sample application code and its execution environment are provided in **[here](../../../../sample_app)**.  
+
+## Overview
+This page explains about Classification in the [sample application](../../../../sample_app) for DRP-AI TVM[^1].  
+
+<img src=./img/googlenet.jpg width=500>  
+
+## Model Information
+- GoogleNet: [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/classification/inception_and_googlenet/googlenet) googlenet-9.onnx  
+Dataset:  [ILSVRC2014](https://image-net.org/challenges/LSVRC/2014/)  
+Input size: 1x3x224x224  
+Output size: 1x1000  
+
+### How to compile the model
+To run the Classification, `googlenet_onnx` Model Object is required.  
+Follow the instuction below to prepare the Model Object.  
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).  
+2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/classification/inception_and_googlenet/googlenet).  
+3. Place the onnx file in `$TVM_HOME/../tutorials`.
+4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.  
+5. Run the  with the command below.  
+```sh
+$ python3 compile_onnx_model.py \
+-i data_0 \
+-s 1,3,224,224 \
+-o googlenet_onnx \
+googlenet-9.onnx
+```
+6. Confirm that `googlenet_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.  
+7. Before running the application, make sure to copy the `googlenet_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.  
+
+
+## Processing Details
+### DRP-AI mode
+- Source Code: [tvm_drpai_googlenet.cpp](../../../src/recognize/googlenet/tvm_drpai_googlenet.cpp)  
+
+Followings are processing details if user selected "GoogleNet (DRP-AI)".  
+
+#### Pre-processing
+Pre-processing is done by DRP-AI Pre-processing Runtime and CPU.  
+
+| Function | Details |  
+|:---|:---|  
+|conv_yuv2rgb |Convert YUY2 to RGB processed by DRP-AI Pre-processing Runtime.|  
+|resize |Resize to 224x224 processed by DRP-AI Pre-processing Runtime.|  
+|cast_to_fp16 | Cast data to FP16 for DRP-AI processed by DRP-AI Pre-processing Runtime.|  
+|normalize | Normalize pixel values with mean values of {123.68, 116.779, 103.939}</br>processed by DRP-AI Pre-processing Runtime.|  
+|transpose | Transpose HWC to CHW order processed by DRP-AI Pre-processing Runtime. |  
+|cast_fp16_fp32 | Cast FP16 data to FP32 for DRP-AI TVM[^1] input</br> processed by DRP-AI Pre-processing Runtime.|  
+|rgb2bgr | Convert RGB to BGR processed by CPU.|  
+
+#### Inference
+The Object files `googlenet_onnx` is generated from ONNX Model Zoo GoogleNet pre-trained model as described in [Model Information](#model-information).  
+
+#### Post-processing
+Post-processing is processed by CPU.
+
+---
+[^1]: DRP-AI TVM is powered by EdgeCortix MERA™ Compiler Framework.

how-to/sample_app/docs/emotion_recognition/emotion_ferplus/README.md

@@ -0,0 +1,85 @@
+# Emotion Recognition
+
+### Model: [Emotion FERPlus](#model-information)
+Sample application code and its execution environment are provided in **[here](../../../../sample_app)**.  
+
+## Overview
+This page explains about Emotion Recognition in the [sample application](../../../../sample_app) for DRP-AI TVM[^1].  
+
+<img src=./img/emotionfp.jpg width=500>  
+
+## Model Information
+- Emotion FERPlus: [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus) emotion-ferplus-8.onnx  
+Dataset:  See [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus#dataset).  
+Input size: 1x1x64x64  
+Output size: 1x8  
+
+Emotion FERPlus can only classify the face expression of single person.  
+To enable multiple face emotion recognition, this application used [UltraFace](../../../docs/face_detection/ultraface/) as pre-processing.  
+To see more details on UltraFace, please see [Face Detection](../../../docs/face_detection/ultraface/).
+
+
+### How to compile the model
+To run the Emotion Recognition, `emotion_fp_onnx` Model Object and `ultraface_onnx` Model Object are required.  
+Follow the instuction below to prepare the `emotion_fp_onnx` Model Object.  
+For `ultraface_onnx` Model Object, please refer to [Face Detection](../../../docs/face_detection/ultraface/).
+
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).  
+2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus).  
+3. Place the onnx file in `$TVM_HOME/../tutorials`.
+4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x442d0000`.  
+Note that the value **must NOT** be default value `0x438E0000` in order to avoid conflict with UltraFace Model Object.
+5. Run the  with the command below.  
+```sh
+$ python3 compile_onnx_model.py \
+-i Input3 \
+-s 1,1,64,64 \
+-o emotion_fp_onnx \
+emotion-ferplus-8.onnx
+```
+6. Confirm that `emotion_fp_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.  
+7. Before running the application, make sure to copy the `emotion_fp_onnx` directory and `ultraface_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.  
+
+
+## Processing Details
+### DRP-AI mode
+- Source Code: [tvm_drpai_emotionfp.cpp](../../../src/recognize/emotionfp/tvm_drpai_emotionfp.cpp)  
+
+Followings are processing details if user selected "Emotion FERPlus (DRP-AI)".  
+
+#### Pre-processing
+As a pre-processing, Face Detection model, UltraFace, is used.  
+To see details, please refer to [Face Detection Processing Details](../../../docs/face_detection/ultraface/README.md#processing-details).  
+
+For each face detected, following pre-processing is done by CPU..  
+Note that some of them are processed by C++ OpenCV.
+
+| Function | Details |  
+|:---|:---|  
+|Crop | Crop YUYV image. Processed by CPU. |  
+|cvtColor | C++ OpenCV. Convert YUY2 to Grayscale.|  
+|resize |C++ OpenCV. Resize to 64x64.|  
+|transpose |Transpose HWC to CHW order. Processed by CPU.|  
+
+#### Inference
+The Object files `emotion_fp_onnx` is generated from ONNX Model Zoo Emotion FERPlus pre-trained model as described in [Model Information](#model-information).  
+
+#### Post-processing
+Post-processing is processed by CPU.
+
+
+#### About processing time
+Details of processing time, which is displayed on web browser, are as follow.  
+
+| Processing | Details |  
+|:---|:---|  
+|Pre-processing | Sum of time taken for following operations. </br>- Face Detection pre-processing, inference and postprocessing</br>- Emotion recognition pre-processing for all detected faces. |  
+|Inferene | Time taken to run inference for all detected faces.|  
+|Post-processing |Time taken to run post-processing for all detected faces.|  
+
+For example, if there are two bounding box detected in face detection, emotion recognition will be carried out for two times.  
+Therefore, inference time will be approximately two times by single inference processing time and same applies for other processing time.  
+
+---
+[^1]: DRP-AI TVM is powered by EdgeCortix MERA™ Compiler Framework.

how-to/sample_app/docs/face_detection/ultraface/README.md

@@ -14,6 +14,27 @@ Dataset: See [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/bo
 Input size: 1x3x240x320  
 Output size: 1x4420x2, 1x4420x4
 
+
+### How to compile the model
+To run the Face Detection, `ultraface_onnx` Model Object is required.  
+Follow the instuction below to prepare the `ultraface_onnx` Model Object.  
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).  
+2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/ultraface).  
+3. Place the onnx file in `$TVM_HOME/../tutorials`.
+4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.  
+5. Run the  with the command below.  
+```sh
+$ python3 compile_onnx_model.py \
+-i input \
+-s 1,3,240,320 \
+-o ultraface_onnx \
+version-RFB-320.onnx
+```
+6. Confirm that `ultraface_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.  
+7. Before running the application, make sure to copy the `ultraface_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.  
+
+
 ## Processing Details
 ### DRP-AI mode
 - Source Code: [tvm_drpai_ultraface.cpp](../../../src/recognize/ultraface/tvm_drpai_ultraface.cpp)  
@@ -34,7 +55,6 @@ Pre-processing is done by DRP-AI Pre-processing Runtime, which allows following
 
 #### Inference
 The Object files `ultraface_onnx` is generated from ONNX Model Zoo pre-trained model as described in [Model Information](#model-information).  
-Please refer to [Compile Tutorial](../../../../../tutorials) for more details on compiling model.
 
 #### Post-processing
 Post-processing is processed by CPU.