@@ -44,19 +44,14 @@ In `src` directory, followings are provided.
### Execution Environment
In `exe` directory, following files are provided as execution environment to be placed on target board.
+**Note that Model Object files (DRP-AI TVM[^1] compile result) are not provided.**
| File/Directory | Details |
|:---|:---|
-|face_deeppose_pt/ | DeepPose model for DRP-AI mode. |
-|face_deeppose_cpu/ | DeepPose model for CPU mode. |
-|yolov3_onnx/ | YOLOv3 model for DRP-AI mode. |
-|yolov2_onnx/ | YOLOv2 model for DRP-AI mode. |
-|tinyyolov3_onnx/ | Tiny YOLOv3 model for DRP-AI mode. |
-|tinyyolov2_onnx/ | Tiny YOLOv2 model for DRP-AI mode. |
-|hrnet_onnx/ | HRNet model for DRP-AI mode. |
|preprocess_tvm_v2ma/ | Pre-processing Runtime Object files. |
|sample_app_drpai_tvm_usbcam_http | Application itself. |
-
+|coco-labels-2014_2017.txt | Label list for Object Detection. |
+|synset_words_imagenet.txt | Label list for Classification. |
In `etc` directory, following files are provided as execution environment to be placed on client PC that displays HTTP result.
@@ -79,8 +74,64 @@ Please refer to [Application Example](../../apps/README.md#how-to-compile-the-ap
Please make sure to change the SDK path to the one generated in 1.
## Run the application
+Before running the application, please compile the AI models to generate following directories according to the instruction provided in each "How to create Model Object" column.
+Copy the generated directories to `exe` directory above so that Model Objet directories are placed in the same directory as `sample_app_drpai_tvm_usbcam_http` application.
+
+| File/Directory | Details | How to create Model Object |
+|:---|:---|:---|
+|face_deeppose_pt/ | DeepPose model for DRP-AI mode. |[Face Landmark Localization](docs/face_landmark_localization/deeppose) |
+|face_deeppose_cpu/ | DeepPose model for CPU mode. |[Face Landmark Localization](docs/face_landmark_localization/deeppose) |
+|yolov3_onnx/ | YOLOv3 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
+|yolov2_onnx/ | YOLOv2 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
+|tinyyolov3_onnx/ | Tiny YOLOv3 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
+|tinyyolov2_onnx/ | Tiny YOLOv2 model for DRP-AI mode. |[Object Detection](docs/object_detection/yolo) |
+|ultraface_onnx/ | UltraFace model for DRP-AI mode.|[Face Detection](docs/face_detection/ultraface) |
+|hrnet_onnx/ | HRNet model for DRP-AI mode.|[Human Pose Estimation](docs/human_pose_estimation/hrnet) |
+|hrnetv2_pt/ | HRNetv2 model for DRP-AI mode. |[Hand Landmark Localization](docs/hand_landmark_localization/hrnetv2) |
+|emotion_fp_onnx/ | Emotion FERPlus model for DRP-AI mode. |[Emotion Recognition](docs/emotion_recognition/emotion_ferplus) |
+|googlenet_onnx/ | GoogleNet model for DRP-AI mode. |[Classification](docs/classification/googlenet) |
+
+Filesystem on the board should look like below.
+```sh
+/
+├── usr/
+│ └── lib64/
+│ └── libtvm_runtime.so
+└── home/
+ └── root/
+ └── exe/
+ ├── face_deeppose_pt/
+ │ ├── deploy.json
+ │ ├── deploy.params
+ │ └── deploy.so
+ ├── face_deeppose_cpu/
+ │ ... Other Model Object directories ...
+ │
+ ├── preprocess_tvm_v2ma/
+ ├── coco-labels-2014_2017.txt
+ ├── synset_words_imagenet.txt
+ └── sample_app_drpai_tvm_usbcam_http
+```
+
To run the application, please refer to the instruction for USB Camera HTTP version application in RZ/V2MA DRP-AI Sample Application Note provided in RZ/V2MA DRP-AI Support Package.
+## Note
+When the web browser is closed to terminate the application following error may appear on the console.
+```sh
+[ERROR] Failed to enqueue _capture buffer.
+Send application message to web client.[Failed to enqueue _capture buffer.
+Restart the application.]
+
+...
+
+********************** END *********************
+***** Stop Recognize. *****
+<<<<<<<<<<<<<<<<<<<<< Message Thread Terminated >>>>>>>>>>>>>>>>>>
+All Finish
+```
+Since the application has already been terminated, the error does not affect the application.
+Application can be restarted without any special procedures, i.e. reboot the board.
+
## Application Specification
### Model Information
Please refer to [AI Sample Application](../README.md#ai-sample-application) for model information.
diff --git a/how-to/sample_app/docs/classification/googlenet/README.md b/how-to/sample_app/docs/classification/googlenet/README.md
new file mode 100755
index 0000000..2b4f031
--- /dev/null
+++ b/how-to/sample_app/docs/classification/googlenet/README.md
@@ -0,0 +1,63 @@
+# Classification
+
+### Model: [GoogleNet](#model-information)
+Sample application code and its execution environment are provided in **[here](../../../../sample_app)**.
+
+## Overview
+This page explains about Classification in the [sample application](../../../../sample_app) for DRP-AI TVM[^1].
+
+
+
+## Model Information
+- GoogleNet: [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/classification/inception_and_googlenet/googlenet) googlenet-9.onnx
+Dataset: [ILSVRC2014](https://image-net.org/challenges/LSVRC/2014/)
+Input size: 1x3x224x224
+Output size: 1x1000
+
+### How to compile the model
+To run the Classification, `googlenet_onnx` Model Object is required.
+Follow the instuction below to prepare the Model Object.
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/classification/inception_and_googlenet/googlenet).
+3. Place the onnx file in `$TVM_HOME/../tutorials`.
+4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.
+5. Run the with the command below.
+```sh
+$ python3 compile_onnx_model.py \
+-i data_0 \
+-s 1,3,224,224 \
+-o googlenet_onnx \
+googlenet-9.onnx
+```
+6. Confirm that `googlenet_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
+7. Before running the application, make sure to copy the `googlenet_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
+
+
+## Processing Details
+### DRP-AI mode
+- Source Code: [tvm_drpai_googlenet.cpp](../../../src/recognize/googlenet/tvm_drpai_googlenet.cpp)
+
+Followings are processing details if user selected "GoogleNet (DRP-AI)".
+
+#### Pre-processing
+Pre-processing is done by DRP-AI Pre-processing Runtime and CPU.
+
+| Function | Details |
+|:---|:---|
+|conv_yuv2rgb |Convert YUY2 to RGB processed by DRP-AI Pre-processing Runtime.|
+|resize |Resize to 224x224 processed by DRP-AI Pre-processing Runtime.|
+|cast_to_fp16 | Cast data to FP16 for DRP-AI processed by DRP-AI Pre-processing Runtime.|
+|normalize | Normalize pixel values with mean values of {123.68, 116.779, 103.939}processed by DRP-AI Pre-processing Runtime.|
+|transpose | Transpose HWC to CHW order processed by DRP-AI Pre-processing Runtime. |
+|cast_fp16_fp32 | Cast FP16 data to FP32 for DRP-AI TVM[^1] input processed by DRP-AI Pre-processing Runtime.|
+|rgb2bgr | Convert RGB to BGR processed by CPU.|
+
+#### Inference
+The Object files `googlenet_onnx` is generated from ONNX Model Zoo GoogleNet pre-trained model as described in [Model Information](#model-information).
+
+#### Post-processing
+Post-processing is processed by CPU.
+
+---
+[^1]: DRP-AI TVM is powered by EdgeCortix MERA™ Compiler Framework.
diff --git a/how-to/sample_app/docs/classification/googlenet/img/googlenet.jpg b/how-to/sample_app/docs/classification/googlenet/img/googlenet.jpg
new file mode 100755
index 0000000..b4f5cb6
Binary files /dev/null and b/how-to/sample_app/docs/classification/googlenet/img/googlenet.jpg differ
diff --git a/how-to/sample_app/docs/emotion_recognition/emotion_ferplus/README.md b/how-to/sample_app/docs/emotion_recognition/emotion_ferplus/README.md
new file mode 100755
index 0000000..370e8c8
--- /dev/null
+++ b/how-to/sample_app/docs/emotion_recognition/emotion_ferplus/README.md
@@ -0,0 +1,85 @@
+# Emotion Recognition
+
+### Model: [Emotion FERPlus](#model-information)
+Sample application code and its execution environment are provided in **[here](../../../../sample_app)**.
+
+## Overview
+This page explains about Emotion Recognition in the [sample application](../../../../sample_app) for DRP-AI TVM[^1].
+
+
+
+## Model Information
+- Emotion FERPlus: [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus) emotion-ferplus-8.onnx
+Dataset: See [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus#dataset).
+Input size: 1x1x64x64
+Output size: 1x8
+
+Emotion FERPlus can only classify the face expression of single person.
+To enable multiple face emotion recognition, this application used [UltraFace](../../../docs/face_detection/ultraface/) as pre-processing.
+To see more details on UltraFace, please see [Face Detection](../../../docs/face_detection/ultraface/).
+
+
+### How to compile the model
+To run the Emotion Recognition, `emotion_fp_onnx` Model Object and `ultraface_onnx` Model Object are required.
+Follow the instuction below to prepare the `emotion_fp_onnx` Model Object.
+For `ultraface_onnx` Model Object, please refer to [Face Detection](../../../docs/face_detection/ultraface/).
+
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/emotion_ferplus).
+3. Place the onnx file in `$TVM_HOME/../tutorials`.
+4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x442d0000`.
+Note that the value **must NOT** be default value `0x438E0000` in order to avoid conflict with UltraFace Model Object.
+5. Run the with the command below.
+```sh
+$ python3 compile_onnx_model.py \
+-i Input3 \
+-s 1,1,64,64 \
+-o emotion_fp_onnx \
+emotion-ferplus-8.onnx
+```
+6. Confirm that `emotion_fp_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
+7. Before running the application, make sure to copy the `emotion_fp_onnx` directory and `ultraface_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
+
+
+## Processing Details
+### DRP-AI mode
+- Source Code: [tvm_drpai_emotionfp.cpp](../../../src/recognize/emotionfp/tvm_drpai_emotionfp.cpp)
+
+Followings are processing details if user selected "Emotion FERPlus (DRP-AI)".
+
+#### Pre-processing
+As a pre-processing, Face Detection model, UltraFace, is used.
+To see details, please refer to [Face Detection Processing Details](../../../docs/face_detection/ultraface/README.md#processing-details).
+
+For each face detected, following pre-processing is done by CPU..
+Note that some of them are processed by C++ OpenCV.
+
+| Function | Details |
+|:---|:---|
+|Crop | Crop YUYV image. Processed by CPU. |
+|cvtColor | C++ OpenCV. Convert YUY2 to Grayscale.|
+|resize |C++ OpenCV. Resize to 64x64.|
+|transpose |Transpose HWC to CHW order. Processed by CPU.|
+
+#### Inference
+The Object files `emotion_fp_onnx` is generated from ONNX Model Zoo Emotion FERPlus pre-trained model as described in [Model Information](#model-information).
+
+#### Post-processing
+Post-processing is processed by CPU.
+
+
+#### About processing time
+Details of processing time, which is displayed on web browser, are as follow.
+
+| Processing | Details |
+|:---|:---|
+|Pre-processing | Sum of time taken for following operations. - Face Detection pre-processing, inference and postprocessing- Emotion recognition pre-processing for all detected faces. |
+|Inferene | Time taken to run inference for all detected faces.|
+|Post-processing |Time taken to run post-processing for all detected faces.|
+
+For example, if there are two bounding box detected in face detection, emotion recognition will be carried out for two times.
+Therefore, inference time will be approximately two times by single inference processing time and same applies for other processing time.
+
+---
+[^1]: DRP-AI TVM is powered by EdgeCortix MERA™ Compiler Framework.
diff --git a/how-to/sample_app/docs/emotion_recognition/emotion_ferplus/img/emotionfp.jpg b/how-to/sample_app/docs/emotion_recognition/emotion_ferplus/img/emotionfp.jpg
new file mode 100755
index 0000000..5afcd47
Binary files /dev/null and b/how-to/sample_app/docs/emotion_recognition/emotion_ferplus/img/emotionfp.jpg differ
diff --git a/how-to/sample_app/docs/face_detection/ultraface/README.md b/how-to/sample_app/docs/face_detection/ultraface/README.md
index 3497dfc..022be8f 100644
--- a/how-to/sample_app/docs/face_detection/ultraface/README.md
+++ b/how-to/sample_app/docs/face_detection/ultraface/README.md
@@ -14,6 +14,27 @@ Dataset: See [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/bo
Input size: 1x3x240x320
Output size: 1x4420x2, 1x4420x4
+
+### How to compile the model
+To run the Face Detection, `ultraface_onnx` Model Object is required.
+Follow the instuction below to prepare the `ultraface_onnx` Model Object.
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Download the onnx file from [ONNX Model Zoo](https://github.com/onnx/models/tree/main/vision/body_analysis/ultraface).
+3. Place the onnx file in `$TVM_HOME/../tutorials`.
+4. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.
+5. Run the with the command below.
+```sh
+$ python3 compile_onnx_model.py \
+-i input \
+-s 1,3,240,320 \
+-o ultraface_onnx \
+version-RFB-320.onnx
+```
+6. Confirm that `ultraface_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
+7. Before running the application, make sure to copy the `ultraface_onnx` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
+
+
## Processing Details
### DRP-AI mode
- Source Code: [tvm_drpai_ultraface.cpp](../../../src/recognize/ultraface/tvm_drpai_ultraface.cpp)
@@ -34,7 +55,6 @@ Pre-processing is done by DRP-AI Pre-processing Runtime, which allows following
#### Inference
The Object files `ultraface_onnx` is generated from ONNX Model Zoo pre-trained model as described in [Model Information](#model-information).
-Please refer to [Compile Tutorial](../../../../../tutorials) for more details on compiling model.
#### Post-processing
Post-processing is processed by CPU.
diff --git a/how-to/sample_app/docs/face_detection/ultraface/img/ultraface.jpg b/how-to/sample_app/docs/face_detection/ultraface/img/ultraface.jpg
index 170ba95..4a41d53 100644
Binary files a/how-to/sample_app/docs/face_detection/ultraface/img/ultraface.jpg and b/how-to/sample_app/docs/face_detection/ultraface/img/ultraface.jpg differ
diff --git a/how-to/sample_app/docs/face_landmark_localization/deeppose/README.md b/how-to/sample_app/docs/face_landmark_localization/deeppose/README.md
index fedc1dc..a369ae8 100644
--- a/how-to/sample_app/docs/face_landmark_localization/deeppose/README.md
+++ b/how-to/sample_app/docs/face_landmark_localization/deeppose/README.md
@@ -14,6 +14,140 @@ Dataset: [WFLW](https://wywu.github.io/projects/LAB/WFLW.html)
Input size: 1x3x256x256
Output size: 1x98x2
+
+### How to compile the model
+To run the Face Landmark Localization, `face_deeppose_pt` Model Object is required for DRP-AI mode and `face_deeppose_cpu` is required for CPU mode.
+#### Operating Environment
+- mmcv-full v1.6.1
+- MMPose v0.28.1
+
+#### 1. Save the AI model from MMPose
+Follow the instuction below to prepare the DeepPose model.
+
+1. Prepare the save script as below in the MMPose clone directory.
+```py
+import numpy as np
+
+# PyTorch imports
+import torch
+import torchvision
+
+from mmpose.apis import init_pose_model
+
+def _convert_batchnorm(module):
+ """Convert the syncBNs into normal BN3ds."""
+ module_output = module
+ if isinstance(module, torch.nn.SyncBatchNorm):
+ module_output = torch.nn.BatchNorm3d(module.num_features, module.eps,
+ module.momentum, module.affine,
+ module.track_running_stats)
+ if module.affine:
+ module_output.weight.data = module.weight.data.clone().detach()
+ module_output.bias.data = module.bias.data.clone().detach()
+ # keep requires_grad unchanged
+ module_output.weight.requires_grad = module.weight.requires_grad
+ module_output.bias.requires_grad = module.bias.requires_grad
+ module_output.running_mean = module.running_mean
+ module_output.running_var = module.running_var
+ module_output.num_batches_tracked = module.num_batches_tracked
+ for name, child in module.named_children():
+ module_output.add_module(name, _convert_batchnorm(child))
+ del module
+ return module_output
+
+config_file = 'configs/face/2d_kpt_sview_rgb_img/deeppose/wflw/res50_wflw_256x256.py'
+checkpoint_file = 'deeppose_res50_wflw_256x256-92d0ba7f_20210303.pth'
+model = init_pose_model(config_file, checkpoint_file, device='cpu')
+model = _convert_batchnorm(model)
+model = model.eval()
+
+# implement the forward method
+if hasattr(model, 'forward_dummy'):
+ model.forward = model.forward_dummy
+
+# We grab the TorchScripted model via tracing
+input_shape = [1, 3, 256, 256]
+input_data = torch.randn(input_shape)
+scripted_model = torch.jit.trace(model, input_data).eval()
+
+scripted_model.save('deeppose.pt')# Save
+print("Torch model saved to ./deeppose.pt")
+```
+2. Download the checkpoint file(`.pth`) from [the mmpose website](https://mmpose.readthedocs.io/en/latest/topics/face.html#deeppose-resnet-on-wflw) and place them in the same directory as the save script above.
+3. Run the save script and confirm that `deeppose.pt` is generated.
+
+#### 2. Compile pytorch model for DRP-AI mode
+Follow the instuction below to prepare the `face_deeppose_pt` Model Object.
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Place the `deeppose.pt` file in `$TVM_HOME/../tutorials`.
+3. Change the `addr_map_start` setting in `compile_pytorch_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.
+4. Run the script with the command below.
+```sh
+$ python3 compile_pytorch_model.py \
+-s 1,3,256,256 \
+-o face_deeppose_pt \
+deeppose.pt
+```
+5. Confirm that `face_deeppose_pt` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
+6. Before running the application, make sure to copy the `face_deeppose_pt` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
+
+
+#### 3. Compile pytorch model for CPU mode
+Follow the instuction below to prepare the `face_deeppose_cpu` Model Object.
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Place the `deeppose.pt` file in `$TVM_HOME/../tutorials`.
+3. Copy and rename the script `compile_cpu_only_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) as shown below.
+```sh
+$ cp compile_cpu_only_onnx_model.py compile_cpu_only_pytorch_model.py
+```
+4. Change the `compile_cpu_only_pytorch_model.py` script as shown below.
+**Note that this script is only for DeepPose CPU mode and not guaranteed for other models.**
+
+Before
+```py
+#L23
+import onnx
+
+#L59~66
+ # 2. Load onnx model and set input shape.
+ shape_dict = {input_name: input_shape}
+ # 2.1 Load onnx model
+ onnx_model = onnx.load_model(model_file)
+ # 2.2 Set input data information
+
+ # 3.1 Run TVM Frontend
+ mod, params = tvm.relay.frontend.from_onnx(onnx_model, shape_dict)
+```
+After
+```py
+#L23
+import torch
+
+#L59~68
+ # 2. Load model and set input shape.
+ # 2.1 Load model
+ model = torch.jit.load(model_file)
+ model.eval()
+ # 2.2 Set input data information
+ input_name = "input0"
+ shape_list = [(input_name, opts["input_shape"])]
+
+ # 3.1 Run TVM Frontend
+ mod, params = tvm.relay.frontend.from_pytorch(model, shape_list)
+```
+5. Run the script with the command below.
+```sh
+$ python3 compile_cpu_only_pytorch_model.py \
+-s 1,3,256,256 \
+-o face_deeppose_cpu \
+deeppose.pt
+```
+6. Confirm that `face_deeppose_cpu` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
+7. Before running the application, make sure to copy the `face_deeppose_cpu` directory into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
+
+
## Processing Details
### DRP-AI mode
- Source Code: [tvm_drpai_deeppose.cpp](../../../src/recognize/deeppose/tvm_drpai_deeppose.cpp)
@@ -34,7 +168,6 @@ Pre-processing is done by DRP-AI Pre-processing Runtime, which allows following
#### Inference
The Object files `face_deeppose_pt` is generated from MMPose DeepPose pre-trained model as described in [Model Information](#model-information).
-Please refer to [Compile Tutorial](../../../../../tutorials) for more details on compiling model.
#### Post-processing
Post-processing is processed by CPU.
@@ -57,7 +190,6 @@ Note that some of them are processed by C++ OpenCV.
#### Inference
The Object files `face_deeppose_cpu` provided in this directory is generated from MMPose DeepPose pre-trained model as described in [Model Information](#model-information) using CPU-only deploy mode.
-Please refer to [Compile Tutorial CPU-only deploy mode](../../../../../tutorials/README.md#3-compile-using-cpu-only-deploy-mode) for more details on compiling model.
#### Post-processing
Post-processing is processed by CPU.
diff --git a/how-to/sample_app/docs/hand_landmark_localization/hrnetv2/README.md b/how-to/sample_app/docs/hand_landmark_localization/hrnetv2/README.md
new file mode 100755
index 0000000..a9ada74
--- /dev/null
+++ b/how-to/sample_app/docs/hand_landmark_localization/hrnetv2/README.md
@@ -0,0 +1,125 @@
+# Hand Landmark Localization
+
+### Model: [HRNet(High-Resolution Network) v2](#model-information)
+Sample application code are provided in **[here](../../../../sample_app)**.
+
+## Overview
+This page explains about Hand Landmark Localization in the [sample application](../../../../sample_app) for DRP-AI TVM[^1].
+
+
+
+## Model Information
+- HRNetv2: [MMPose Topdown Heatmap + Hrnetv2 + Coco + Wholebody on Coco_wholebody_hand](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#topdown-heatmap-hrnetv2-coco-wholebody-on-coco-wholebody-hand)
+Dataset: [COCO](https://cocodataset.org/#home)
+Input size: 1x3x256x256
+Output size: 1x21x64x64
+
+### How to compile the model
+#### Operating Environment
+- mmcv-full v1.6.1
+- MMPose v0.28.1
+
+#### Save the AI model from MMPose
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Clone mmpose repository.
+
+```sh
+git clone -b v0.28.1 https://github.com/open-mmlab/mmpose.git
+```
+
+3. Download the checkpoint file from [the mmpose website](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#topdown-heatmap-hrnetv2-coco-wholebody-on-coco-wholebody-hand) and place it in the MMPose clone directory.
+
+4. Save the script below in this directory and run it.
+
+```py
+import numpy as np
+
+# PyTorch imports
+import torch
+import torchvision
+
+from mmpose.apis import init_pose_model
+
+def _convert_batchnorm(module):
+ """Convert the syncBNs into normal BN3ds."""
+ module_output = module
+ if isinstance(module, torch.nn.SyncBatchNorm):
+ module_output = torch.nn.BatchNorm3d(module.num_features, module.eps,
+ module.momentum, module.affine,
+ module.track_running_stats)
+ if module.affine:
+ module_output.weight.data = module.weight.data.clone().detach()
+ module_output.bias.data = module.bias.data.clone().detach()
+ # keep requires_grad unchanged
+ module_output.weight.requires_grad = module.weight.requires_grad
+ module_output.bias.requires_grad = module.bias.requires_grad
+ module_output.running_mean = module.running_mean
+ module_output.running_var = module.running_var
+ module_output.num_batches_tracked = module.num_batches_tracked
+ for name, child in module.named_children():
+ module_output.add_module(name, _convert_batchnorm(child))
+ del module
+ return module_output
+
+config_file = 'configs/hand/2d_kpt_sview_rgb_img/topdown_heatmap/coco_wholebody_hand/hrnetv2_w18_coco_wholebody_hand_256x256.py'
+checkpoint_file = 'hrnetv2_w18_coco_wholebody_hand_256x256-1c028db7_20210908.pth'
+model = init_pose_model(config_file, checkpoint_file, device='cpu')
+model = _convert_batchnorm(model)
+model = model.eval()
+
+# implement the forward method
+if hasattr(model, 'forward_dummy'):
+ model.forward = model.forward_dummy
+
+# We grab the TorchScripted model via tracing
+input_shape = [1, 3, 256, 256]
+input_data = torch.randn(input_shape)
+scripted_model = torch.jit.trace(model, input_data).eval()
+
+scripted_model.save('hrnetv2.pt') # Save
+print("Torch model saved to ./hrnetv2.pt")
+```
+5. Copy the hrnetv2.pt to the `drp-ai_tvm/tutorials` directory and run the sample script [compile_pytorch_model.py](../../../../../tutorials/compile_pytorch_model.py) as shown below.
+Make sure to change the `addr_map_start` setting in `compile_pytorch_model.py` to `0x438E0000`.
+The output directory name `hrnetv2_pt` is used in the sample application and should not be changed.
+Confirm that three files, deploy.so, deploy.params, and deploy.json, have been created in the `hrnetv2_pt` directory.
+
+```sh
+# Run DRP-AI TVM[*1] Compiler script
+python3 compile_pytorch_model.py \
+ ./hrnetv2.pt \
+ -o hrnetv2_pt \
+ -s 1,3,256,256
+```
+
+6. Copy the `hrnetv2_pt` directory into the execution environment directory where the compiled sample application sample_app_drpai_tvm_usbcam_http is located.
+
+## Processing Details
+
+### DRP-AI mode
+- Source Code: [tvm_drpai_hrnet.cpp](../../../src/recognize/hrnet/tvm_drpai_hrnet.cpp).
+
+The source code is common to that of the HRNet, except for the macro definitions. See [the header file](../../../src/recognize/hrnet/tvm_drpai_hrnet.h).
+Followings are processing details if user selected "HRNetv2 (DRP-AI)".
+
+#### Pre-processing
+First, crop process is executed by CPU. Then pre-processing is done by DRP-AI Pre-processing Runtime, which allows following pre-processing on DRP-AI.
+
+| Function | Details |
+|:---|:---|
+|crop |Crop image processed by CPU.|
+|conv_yuv2rgb |Convert YUY2 to RGB processed by DRP-AI Pre-processing Runtime.|
+|resize |Resize to 256x256 processed by DRP-AI Pre-processing Runtime.|
+|cast_to_fp16 | Cast data to FP16 for DRP-AI processed by DRP-AI Pre-processing Runtime.|
+|normalize | Normalize pixel values with mean and standard deviation processed by DRP-AI Pre-processing Runtime.|
+|transpose | Transpose HWC to CHW order processed by DRP-AI Pre-processing Runtime. |
+|cast_fp16_fp32 | Cast FP16 data to FP32 for DRP-AI TVM[^1] input processed by DRP-AI Pre-processing Runtime.|
+
+#### Inference
+Inference performs the processing of `hrnetv2_pt` generated in [Model Information](#model-information).
+
+#### Post-processing
+Post-processing is processed by CPU.
+
+---
+[^1]: DRP-AI TVM is powered by EdgeCortix MERA™ Compiler Framework.
diff --git a/how-to/sample_app/docs/hand_landmark_localization/hrnetv2/img/hrnetv2.jpg b/how-to/sample_app/docs/hand_landmark_localization/hrnetv2/img/hrnetv2.jpg
new file mode 100755
index 0000000..c40e6c1
Binary files /dev/null and b/how-to/sample_app/docs/hand_landmark_localization/hrnetv2/img/hrnetv2.jpg differ
diff --git a/how-to/sample_app/docs/human_pose_estimation/hrnet/README.md b/how-to/sample_app/docs/human_pose_estimation/hrnet/README.md
index f8322f7..6da414a 100644
--- a/how-to/sample_app/docs/human_pose_estimation/hrnet/README.md
+++ b/how-to/sample_app/docs/human_pose_estimation/hrnet/README.md
@@ -13,9 +13,33 @@ This page explains about Human Pose Estimation in the [sample application](../..
Dataset: [COCO](https://cocodataset.org/#home)
Input size: 1x3x256x192
Output size: 1x17x64x48
-
-### ONNX format model
- Here, we follow this [tutorial](https://mmpose.readthedocs.io/en/latest/tutorials/5_export_model.html#prerequisite) to convert HRNet model provided by MMPose into ONNX format. The MMpose version we checked is v0.26.0.
+
+### How to compile the model
+To run the Human Pose Estimation, `hrnet_onnx` Model Object is required.
+
+#### Operating Environment
+- mmcv-full v1.5.1
+- MMPose v0.26.0
+
+#### Compile onnx model
+Follow the instuction below to prepare the HRNet Model Object.
+
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Follow the [MMPose tutorial](https://mmpose.readthedocs.io/en/latest/tutorials/5_export_model.html#prerequisite) to convert HRNet model provided by MMPose into ONNX format.
+3. Next, run the sample script [compile_onnx_model.py](../../../../../tutorials/compile_onnx_model.py) as shown below.
+Make sure to change the `addr_map_start` setting in `compile_onnx_model.py` to `0x438E0000`.
+The output directory name `hrnet_onnx` is used in the sample application and should not be changed.
+
+```sh
+# Run DRP-AI TVM[*1] Compiler script
+$ python3 compile_onnx_model.py \
+./hrnet.onnx \
+-o hrnet_onnx \
+-s 1,3,256,192 \
+-i input.1
+```
+4. Confirm that three files, deploy.so, deploy.params, and deploy.json, have been created in the `hrnet_onnx` directory.
+5. Copy the `hrnet_onnx` directory into the execution environment directory where the compiled sample application sample_app_drpai_tvm_usbcam_http is located.
## Processing Details
### DRP-AI mode
@@ -38,7 +62,6 @@ First, crop process is executed by CPU. Then pre-processing is done by DRP-AI Pr
#### Inference
The Object files `hrnet_onnx` are generated from HRNet pre-trained model provided by MMPose framework as described in [Model Information](#model-information).
-Please refer to [Compile Tutorial](../../../../../tutorials) for more details on compiling model.
#### Post-processing
Post-processing is processed by CPU.
diff --git a/how-to/sample_app/docs/object_detection/yolo/README.md b/how-to/sample_app/docs/object_detection/yolo/README.md
index 79136fd..1e5c471 100644
--- a/how-to/sample_app/docs/object_detection/yolo/README.md
+++ b/how-to/sample_app/docs/object_detection/yolo/README.md
@@ -21,13 +21,18 @@ Dataset: [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/index.html)
Input size: 1x3x416x416
Output size: 1x13x13x125
-### ONNX format model
+
+### How to compile the model
+To run the Object Detection, `yolov3_onnx`, `yolov2_onnx`, `tinyyolov3_onnx` and `tinyyolov2_onnx` Model Object are required.
+Follow the instuction below to prepare the Model Object.
+
+#### 1. Save ONNX format model
Here, we use the ONNX format model converted from Darknet weight file by using the scripts provided in this directory.
-#### Requirement
+##### 1.1 Requirement
To run the script, PyTorch 1.8.0 must be installed.
-#### File Configuration
+##### 1.2 File Configuration
| File | Details |
|:---|:---|
|darknet_cfg.py |Darknet cfg file parser.|
@@ -36,16 +41,16 @@ To run the script, PyTorch 1.8.0 must be installed.
|yolo.py | Conversion configuration file.|
|yolo.ini | Conversion configuration file parser. |
-#### ONNX conversion
-Download the YOLO *.cfg file and *.weights file from [Darknet](https://pjreddie.com/darknet/yolo/) and place them in the same directory of the conversion scripts.
-Run the following commands to convert YOLOv3 model.
+##### 1.3 ONNX conversion
+1. Download the YOLO *.cfg file and *.weights file from [Darknet](https://pjreddie.com/darknet/yolo/) and place them in the same directory of the conversion scripts.
+2. Run the following commands to convert YOLOv3 model.
```sh
$ python3 convert_to_pytorch.py yolov3
# --> yolov3.pth will be generated.
$ python3 convert_to_onnx.py yolov3
# --> d-yolov3.onnx will be generated.
```
-If you would like to convert other models, specify the parameter instead of `yolov3` according to the following table.
+3. If you would like to convert other models, specify the parameter instead of `yolov3` according to the following table.
| Parameter | Model |
|:---|:---|
@@ -54,6 +59,29 @@ If you would like to convert other models, specify the parameter instead of `yol
|tinyyolov3| Tiny YOLOv3|
|tinyyolov2 | Tiny YOLOv2 |
+#### 2. Compile onnx models
+1. Set the environment variables, i.e. `$TVM_HOME` etc., according to [Installation](../../../../../setup/).
+2. Place the onnx file in `$TVM_HOME/../tutorials`.
+3. Change the `addr_map_start` setting in `compile_onnx_model.py` provided in [Compile Tutorial](../../../../../tutorials) to `0x438E0000`.
+4. Run the script with the command below to compile YOLOv3 model.
+```sh
+$ python3 compile_onnx_model.py \
+-i input1 \
+-s 1,3,416,416 \
+-o yolov3_onnx \
+d-yolov3.onnx
+```
+5. Confirm that `yolov3_onnx` directory is generated and it contains `deploy.json`, `deploy.so` and `deploy.params` files.
+6. Repeat the step for other models to generate following Model Object.
+
+| Model | Model Object Name |
+|:---|:---|
+|d-yolov2.onnx |yolov2_onnx|
+|d-tinyyolov3.onnx |tinyyolov3_onnx|
+|d-tinyyolov2.onnx| tinyyolov2_onnx|
+
+7. Before running the application, make sure to copy the `yolov3_onnx`, `yolov2_onnx`, `tinyyolov3_onnx`, and `tinyyolov2_onnx` directories into the execution environment directory `exe` where the compiled sample application `sample_app_drpai_tvm_usbcam_http` is located.
+
## Processing Details
### DRP-AI mode
@@ -75,7 +103,6 @@ Pre-processing is done by DRP-AI Pre-processing Runtime, which allows following
#### Inference
The Object files `yolov3_onnx`, `yolov2_onnx`, `tinyyolov3_onnx` and `tinyyolov2_onnx` are generated from Darknet YOLO pre-trained ONNX model as described in [Model Information](#model-information).
-Please refer to [Compile Tutorial](../../../../../tutorials) for more details on compiling model.
#### Post-processing
Post-processing is processed by CPU.
diff --git a/how-to/sample_app/etc/Websocket_Client/index.html b/how-to/sample_app/etc/Websocket_Client/index.html
index 5ff6325..61d5ef6 100644
--- a/how-to/sample_app/etc/Websocket_Client/index.html
+++ b/how-to/sample_app/etc/Websocket_Client/index.html
@@ -71,6 +71,9 @@