HS Model Conversion
HSDK now supports conversion of Caffe and Tensorflow models. Models base on other CNN frameworks, e.g. MXNet, Darknet(YOLO), PyTorch, etc. would need to convert to Caffe/Tensorflow first. This tool is similar to the conversion tool provided by NCSDK.
Get model conversion toolkit
This conversion tool is modified based on NCSDK (Python), implemented in the newest SungemSDK-Python.
Conversion tool is located in SungemSDK-Python/tool.
The directory tool has following structure:
tool
├── Controllers # Model analyser (impl. from NCSDK)
├── Models # Model structures, setups, device configurations, util functions, etc. (impl. from NCSDK)
├── Views # Visualisation, log (impl. from NCSDK)
├── caffe # Simplified local CPU version Caffe-SSD for model validation tool.
└── example # Caffe\TF conversion examples
Setup
Tested on Ubuntu 16.04, should work on other platforms.
Python dependencies
Note: graphviz may need to be installed via apt
pip3 install graphviz, pyyaml
Install Caffe-SSD CPU
Note: Makefile.config has been rewritten, but may need modification in case that error occurs.
Error relates to libboost:
cd /usr/lib/x86_64-linux-gnu
sudo ln -s libboost_python-py35.so libboost_python3.so
Please seek help from us or community for other issues.
cd caffe
make -j4
make pycaffe
Install Tensorflow
Please refer to Tensorflow official guide .
Model conversion, validation and profile
Compile
This script converts Caffe or Tensorflow model to HS Graph.
Usage:
python3 mvNCCompile.py network # network - *.prototxt(Caffe)* or *.meta(Tensorflow)*
-h # Show help
-w WEIGHTS # Model weight (Caffe only), 如 *.caffemodel*
-in INPUTNODE # Name of input node, default: Caffe: *data*, TF: *input*
-on OUTPUTNODE # Name of output node, default: Caffe: Last layer, TF: *output*
-o OUTFILE # Output path of HS Graph
-s NSHAVES # Number of Movidius Shave, recommend 8, default 1
-is INPUTSIZE INPUTSIZE # Change network input size, default: original network size.
Example: Run under tool:
python3 mvNCCompile.py example/deploy.prototxt -w example/deploy.caffemodel -o example/graph_mnist -s 8
-----------
mvNCCompile v02.00, Copyright @ Movidius Ltd 2016
...
Generates HS Graph: graph_mnist
Validation
This script is used for validating the output of Caffe/TF and HS model.
The evaluation method used in this script is comparing the result with Euclidean distance and sorting. Note: therefore if both somehow results in zero value, the test is incorrectly passed. Because of the limitaion of hardware and NCSDK conversion tool, the test may fail if some layer are not supported or behaving differently. If test fails, please check the network structure to ensure that all layers are supported. In some case you may have to retrain the model to workaround the limitation. As all calculations are based on half-precision float point (FP16), there would be errors due to precision loss, especially on deep and large model. In some case the test fails but the model is still usable. This can be analysed through the log.
Usage:
python3 mvNCCheck.py network # network - *.prototxt(Caffe)* or *.meta(Tensorflow)*
-h # Show help
-w WEIGHTS # Model weight (Caffe only), *.caffemodel*
-in INPUTNODE # Name of input node, default: Caffe: *data*, TF: *input*
-on OUTPUTNODE # Name of output node, default: Caffe: Last layer, TF: *output*
-o OUTFILE # Output path of HS Graph
-s NSHAVES # Number of Movidius Shave, recommend 8, default 1
-is INPUTSIZE INPUTSIZE # Change network input size, default: original network size.
-i IMAGE # Image for validation, default: random float number
-S SCALE # Scale for preprocessing, default: 1
-M MEAN # Mean for preprocessing, after scale multiplication, default: 0
-cs [CHANNEL_SWAP] # Can modify to RGB [0,1,2], default: BGR [2,1,0]
-metric METRIC # Metric for validation, options are: top1, top5, accuracy_metrics, ssd_pred_metric. Default: top1
Example: Run under tool:
python3 mvNCCheck.py example/deploy.prototxt -w example/deploy.caffemodel
-----------
mvNCCompile v02.00, Copyright @ Movidius Ltd 2016
...
USB: Transferring Data...
USB: Myriad Execution Finished
USB: Myriad Connection Closing.
USB: Myriad Connection Closed.
Result: (10,)
1) 3 8.3
2) 8 4.64
3) 2 4.617
4) 5 3.217
5) 7 1.299
Expected: (10,)
1) 3 8.3
2) 8 4.637
3) 2 4.61
4) 5 3.21
5) 7 1.303
------------------------------------------------------------
Obtained values
------------------------------------------------------------
Obtained Min Pixel Accuracy: 0.14124293811619282% (max allowed=2%), Pass
Obtained Average Pixel Accuracy: 0.06709039444103837% (max allowed=1%), Pass
Obtained Percentage of wrong values: 0.0% (max allowed=0%), Pass
Obtained Pixel-wise L2 error: 0.07816364273015063% (max allowed=1%), Pass
Obtained Global Sum Difference: 0.0556640625
------------------------------------------------------------
The results show that Graph has similar output with Caffe, with neglectable error. This model is usable by HS device.
Profile
This script is used for profiling model structure and overall/per layer time cost.
Usage:
python3 mvNCProfile.py network # network - *.prototxt(Caffe)* or *.meta(Tensorflow)*
-h # Show help
-w WEIGHTS # Model weight (Caffe only), *.caffemodel*
-in INPUTNODE # Name of input node, default: Caffe: *data*, TF: *input*
-on OUTPUTNODE # Name of output node, default: Caffe: Last layer, TF: *output*
-o OUTFILE # Output path of HS Graph
-s NSHAVES # Number of Movidius Shave, recommend 8, default 1
-is INPUTSIZE INPUTSIZE # Change network input size, default: original network size.
Example: Run under tool:
python3 mvNCProfile.py example/deploy.prototxt -w example/deploy.caffemodel
-----------
mvNCCompile v02.00, Copyright @ Movidius Ltd 2016
...
USB: Transferring Data...
Time to Execute : 5.75 ms
USB: Myriad Execution Finished
Time to Execute : 4.32 ms
USB: Myriad Execution Finished
USB: Myriad Connection Closing.
USB: Myriad Connection Closed.
Network Summary
Detailed Per Layer Profile
Bandwidth time
# Name MFLOPs (MB/s) (ms)
===============================================================================
0 data 0.0 935.8 0.002
1 conv1 0.6 59.7 0.658
2 pool1 0.0 220.2 0.100
3 conv2 3.2 223.5 0.829
4 pool2 0.0 149.4 0.041
5 ip1 0.8 2300.8 0.332
6 ip2 0.0 263.2 0.040
-------------------------------------------------------------------------------
Total inference time 2.00
-------------------------------------------------------------------------------
Result:
Python overall time cost: 5.75ms, USB cost 4.32ms, internal model inference time 2.00ms, and time cost for every layer.
MFLOPS(Unit): million FP16 calculation.
Detailed report can be inspected from output.gv.svg or output_report.html. For small model that below 10ms, e.g. the LeNet model used in this example, the time cost for interaction/USB transfer might be higher than inferencing.
About model conversion on Tensorflow
Converting Tensorflow model is more complicated than Caffe.
Generate Tensorflow deploy model
This tutorial also adapts NCSDK.
Mostly, Tensorflow model cannot be directly converted to HS Graph, the reasons are below:
- Model format is not compatible. The Tensorflow model must be converted first to include these 3 files: .meta + .index + .data-00000-of-00001.
- Cannot directly convert using training model. Many operations for training cannot be recognised by the conversion tool, e.g. the operations for optimisation. As shown in the figures below (model for training / deploy). We recommend users to convert their TF models for deploy first before converting to HS model.
3. Defining output node name. Defining output node name is necessary for Tensorflow model conversion. The name can be acquired with Tensorboard, as shown below. The name is located at top-right corner. In this example, the name is UNet/Conv_end/conv1/bias_add.
Example
Generate HS graph file under tool:
python3 mvNCCompile.py -s 8 ./example/tf_wave/example/wave.ckpt.meta -in input -on UNet/Conv_end/conv1/bias_add -o graph_wave
python3 mvNCProfile.py -s 8 ./example/tf_wave/example/wave.ckpt.meta -in input -on UNet/Conv_end/conv1/bias_add
python3 mvNCCheck.py -s 8 ./example/tf_wave/example/wave.ckpt.meta -in input -on UNet/Conv_end/conv1/bias_add
This model is our recently released small demo for style transfer
Usage guide:
First load model via HSAPI, resize image to 256x256 (float32), then directly load it to the HS device (no preprocessing). Reshape and convert the result (256x256x3) from BGR to RGB, resize back to the original size.
Output:
Graph usage
Please refer to API, examples and open-sourced codes to learn how to deploy the generated graph to applications.