Advanced.md

MobileNet V1 BFloat16 inference - Advanced Instructions

MobileNet v1 BFloat16 inference depends on Auto-Mixed-Precision to convert graph from FP32 to BFloat16 online. Before evaluating MobileNet v1 BFloat16 inference, please set the following environment variables:

export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_REMOVE=BiasAdd \
export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE=Softmax \
export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_ALLOWLIST_ADD=BiasAdd,Softmax

1. The ImageNet dataset is required for testing accuracy and can also be used when running online or batch inference. If no dataset is provided when running online or batch inference, synthetic data will be used.

   Download and preprocess the ImageNet dataset using the instructions here. After running the conversion script you should have a directory with the ImageNet dataset in the TF records format.

2. Download the pre-trained model.

   wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
   

3. Clone the tensorflow/models repository.

   git clone https://github.com/tensorflow/models
   

   The tensorflow/models files are used for dependencies when running the model.

4. Clone the intelai/models repo and then navigate to the benchmarks directory:

   git clone https://github.com/IntelAI/models.git
   cd models/benchmarks
   

   MobileNet V1 can be run for either online or batch inference using the commands below. The --data-location should be the path to the ImageNet validation data from step 1, the --checkpoint arg should be the path to the checkpoint files from step 2, and the --model-source-dir should point to the tensorflow/models repo that was cloned in step 3.

   • Run for online inference (with --batch-size 1, --in-graph with a path to the frozen graph .pb file, and the --data-location is optional):

     python launch_benchmark.py \
         --precision bfloat16 \
         --model-name mobilenet_v1 \
         --mode inference \
         --framework tensorflow \
         --docker-image intel/intel-optimized-tensorflow:latest \
         --model-source-dir /home/<user>/tensorflow/models \
         --batch-size 1 \
         --socket-id 0 \
         --data-location /dataset/Imagenet_Validation \
         --in-graph /home/<user>/mobilenet_v1_1.0_224_frozen.pb \
         -- input_height=224 input_width=224 warmup_steps=10 steps=50 \
         input_layer="input" output_layer="MobilenetV1/Predictions/Reshape_1"
     

   • Run for batch inference (with --batch-size 100, --in-graph with a path to the frozen graph .pb file, and the --data-location is optional):

     python launch_benchmark.py \
        --precision bfloat16 \
        --model-name mobilenet_v1 \
        --mode inference \
        --framework tensorflow \
        --docker-image intel/intel-optimized-tensorflow:latest \
        --model-source-dir /home/<user>/tensorflow/models \
        --batch-size 100 \
        --socket-id 0 \
        --data-location /dataset/Imagenet_Validation \
        --in-graph /home/<user>/mobilenet_v1_1.0_224_frozen.pb \
        -- input_height=224 input_width=224 warmup_steps=10 steps=50 \
        input_layer="input" output_layer="MobilenetV1/Predictions/Reshape_1"
     

   • Run for accuracy (with --batch-size 100, --accuracy-only and --in-graph with a path to the frozen graph .pb file):

     python launch_benchmark.py \
        --precision bfloat16 \
        --model-name mobilenet_v1 \
        --mode inference \
        --framework tensorflow \
        --docker-image intel/intel-optimized-tensorflow:latest \
        --model-source-dir /home/<user>/tensorflow/models \
        --batch-size 100 \
        --accuracy-only \
        --data-location /dataset/Imagenet_Validation \
        --in-graph /home/<user>/mobilenet_v1_1.0_224_frozen.pb
     

     Note that the --verbose or --output-dir flag can be added to any of the above commands to get additional debug output or change the default output location.

5. The log files for each run are saved at the value of --output-dir.

   • Below is a sample log file snippet when testing online inference:

     [Running warmup steps...]
     steps = 10, ...6 images/sec
     [Running benchmark steps...]
     steps = 10, ... images/sec
     steps = 20, ... images/sec
     steps = 30, ... images/sec
     steps = 40, ... images/sec
     steps = 50, ... images/sec
     Average Throughput: ... images/s on 50 iterations
     Ran inference with batch size 1
     Log file location: {--output-dir value}/benchmark_mobilenet_v1_inference_bfloat16_20210610_143741.log
     

   • Below is a sample log file snippet when testing batch inference:

     [Running warmup steps...]
     steps = 10, ... images/sec
     [Running benchmark steps...]
     steps = 10, ... images/sec
     steps = 20, ... images/sec
     steps = 30, ... images/sec
     steps = 40, ... images/sec
     steps = 50, ... images/sec
     Average Throughput: ... images/s on 50 iterations
     Ran inference with batch size 100
     Log file location: {--output-dir value}/benchmark_mobilenet_v1_inference_bfloat16_20210610_144252.log
     

   • Below is a sample log file snippet when testing accuracy:

     Iteration time: ... ms
     Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7117, 0.8992)
     Iteration time: ... ms
     Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7118, 0.8992)
     Iteration time: ... ms
     Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7119, 0.8992)
     Ran inference with batch size 100
     Log file location: {--output-dir value}/benchmark_mobilenet_v1_inference_bfloat16_20210610_144534.log