SungemSDK-Python

Development guide

hsapi’s package structure is as follows：

hsapi
├── __init__.py
├── core
│   ├── __init__.py
│   ├── base.py
│   ├── device.py
│   └── graph.py
├── high
│   ├── __init__.py
│   ├── net.py
│   └── task.py
└── easy
    ├── __init__.py
    └── prefab.py
  • core Low-Level Interface.
  • high High-Level Interface. Abstract a neural network base class to simplify the operation process.
  • easy Provides some simple classes and methods that use built-in neural networks.

Low-Level Interface

Device Management (class Device)

# Import libs
import numpy
import hsapi as hs

# Get Device
device_list = hs.EnumerateDevices() # Get all connected devices
device = hs.Device(device_list[0])

# Open Device
device.OpenDevice()

"""
# [Optional, according to business needs]
# Read the image of the camera built into the Horned Sungem
image = device.GetImage(zoomMode=True) # zoomMode: True(640x360) False(1920x1080)
"""

# Manage neural network resources, processing data, business logic, etc.
...

# Close the device and the Horned Sungem will automatically reset
device.CloseDevice()

Manage neural network resources (class Graph)

  • Single-device Multi-model: Same Device instance Load multiple Graph files and get multiple Graph instances
  • Multi-device Multi-model: Different Device instances Load different Graph files to get the corresponding Graph instance
# Load the Graph file, get the Graph instance
with open('Graph filePath', mode='rb') as f:
    data = f.read()
graph = device.AllocateGraph(data, scale, mean) # scale/mean Image preprocessing parameters

# Neural network image input, there are two ways [2 choose 1]
# 1. Load external image, here to read the USB camera as an example
"""
import cv2
capture = cv2.VideoCapture(0)
_, image = capture.read()

image = image_preprocess(image) # Preprocess the image and process it into the format and size required by the neural network
graph.LoadTensor(image.astype(numpy.float16), None) # Load image as input to neural network
"""
# 2. Use the built-in camera of the Horned Sungem as input
# Use the built-in camera as input to return images
image = graph.GetImage(True)  # zoomMode: True(640x360) False(1920x1080)

# Read neural network output
output, _ = graph.GetResult()

# Analyze neural network output to implement business logic
...

# Release neural network resources
graph.DeallocateGraph()

Simple interface call flow

# Single-device Single-model
import cv2, numpy
import hsapi as hs # import libs

scale = 0.007843 # Image preprocessing parameters
mean = -1.0 # Image preprocessing parameters

device_list = hs.EnumerateDevices() # Get all connected devices
device = hs.Device(device_list[0]) # Get a Device instance

device.OpenDevice() # Open device

with open('Graph filePath', mode='rb') as f:
    data = f.read()
graph = device.AllocateGraph(data, scale, mean) # Get a Graph instance

try:
    while True:
        # Use the built-in camera of the Horned Sungem as input
        image = graph.GetImage(True) # Use the built-in camera as input to return images
        output, _ = graph.GetResult() # Read neural network output
        print(output)
        cv2.imshow("horned-sungem", image)
        cv2.waitKey(1)
finally:
    graph.DeallocateGraph() # Release neural network resources
    device.CloseDevice() # Close device

High-Level Interface

Net is an abstract class that encapsulates some simple device operations.
Based on this class, you can implement your own neural network class.
Refer to the model class preset in the easy.prefab module.

Built-in Model - Face Detection

# Single-device Single-model
import cv2, numpy 
from hsapi import FaceDetector # import libs

# The model path can be specified by the 'graphPath' attribute. The default path is under 'examples/graphs'. 

net = FaceDetector(graphPath="./graph_face_SSD", zoom = True, thresh=0.55) # Create a face detection network

try:
    while True:
        # Use the built-in camera of the Horned Sungem as input
        result = net.run(image=None) # Use the built-in camera as input to return images
        image = net.plot(result) # Draw an image based on the output
        cv2.imshow("FaceDetector", image)
        cv2.waitKey(1)
finally:
    net.quit() # Exit the neural network, release resources, reset the device

Built-in Model - Face Detection + Object Detection

# Single-device Multi-model
import cv2
import hsapi as hs

device_list = hs.EnumerateDevices() # Get all connected devices
device = hs.Device(device_list[0]) # Get a Device instance

face_net = hs.FaceDetector(device=device, zoom = True, thresh=0.55) # Create a face detection network
obj_net = hs.ObjectDetector(device=device, zoom = True, thresh=0.55) # Create a object detection network

"""
Multi-Device Multi-Model: Use different Device instances when initializing Net
"""

capture = cv2.VideoCapture(0) # Get a USB camera

try:
    while True:
        # Use a USB camera as input
        _, image = capture.read() # Read USB camera image data
        face_result = face_net.run(image) # Input images into the face detection network
        obj_result = obj_net.run(image) # Input images into the object detection network

        # Draw on the image according to the recognition result of the face detection network
        image = face_net.overlay(image, face_result[1])
        
        # Draw on the image according to the recognition result of the object detection network
        image = obj_net.overlay(image, obj_result[1]) 

        cv2.imshow("Face/Obj Detector", image)
        cv2.waitKey(1)
finally:
    face_net.quit() # Reset the device, the same device only needs to exit one network

About Preprocessing Parameters (scale & mean)

The input of the neural network requires pre-processing of the image according to the needs of the network.

When using the built-in camera of the horned hummingbird as the neural network input, only the unified preprocessing operation of the RGB channel of the image is currently supported.
Here Horned Sungem internals and Python API both use multiply plus operations to preprocess images.

image *= scale
image += mean

If the image 3 channel unified processing is required, or the preprocessing requirements are not strict, the mean mean can be taken as a parameter input.

mean = -1.0 # [-1.0, -1.0, -1.0]
scale = 0.007843
graph = device.AllocateGraph(data, scale, mean) # scale & mean

# The built-in device image has been pre-processed and loaded into the neural network. 
# The returned image is the unprocessed original image.
image = graph.GetImage(True) 

output, _ = graph.GetResult() # Read neural network output

If grayscale input is required, or different pre-processing operations need to be performed on different channels, the image can be processed by itself and loaded into the Horned Sungem.

image = preprocess(image) # preprocess by yourself
graph.LoadTensor(image.astype(numpy.float16), None)

API Documentation

  • hsapi.core
    Low-Level API
    • class Status(enum.Enum)
      Return status
      • OK
      • BUSY
      • ERROR
      • OUT_OF_MEMORY
      • DEVICE_NOT_FOUND
      • INVALID_PARAMETERS
      • TIMEOUT
      • NO_DATA
      • GONE No Graph found or closed
      • UNSUPPORTED_GRAPH_FILE
      • MYRIAD_ERROR
    • class GlobalOption(enum.Enum)
      Global option
      • LOG_LEVEL
    • GetGlobalOption(opt)
      Get the value of the global option
      • opt refer to GlobalOption
    • SetGlobalOption(opt, data)
      Set the value of global option
      • opt refer to GlobalOption
      • data value
    • BootUpdateApp(fileName)
      Upgrade firmware through ‘Boot mode’, mainly used for recovery and upgrade after firmware update failure
      • fileName Firmware file
    • EnumerateDevices()
      Enumerate all connected devices
    • class Device
      • init(self, name)
        Instantiating class objects
      • OpenDevice(self)
        Open Device
      • CloseDevice(self)
        Close Device
      • UpdateApp(self)
        Upgrade firmware
      • AllocateGraph(self, graphfile, std_value=1.0, mean_value=0.0)
        Assign neural network resources and return to the ‘Graph’ instance
        • graphfile graph file
        • std_value Image preprocessing parameters
        • mean_value Image preprocessing parameters
      • GetImage(self, zoomMode=True)
        Get device image
        • zoomMode Image zoom mode (True: 640x360, False: 1920x1080)
    • class Graph
      Neural network container class, which can be called by the Device instance AllocateGraph()
      • init(self, handle, std_value, mean_value)
        Instantiating class objects
        • handle graph handle
        • std_value Image preprocessing parameters
        • mean_value Image preprocessing parameters
      • DeallocateGraph(self)
        Release allocated neural network resources
      • LoadTensor(self, tensor, userobj)
        Load image data input by neural network
        • tensor The preprocessed image data must be in the form of a semi-precision floating point (float16) type numpy ndarray
        • userobj User-defined parameter
      • GetResult(self)
        Get the output of the neural network forward inference
      • GetImage(self, zoomMode=True)
        Use the built-in camera as a neural network input to return the input image
        • zoomMode Image zoom mode (True: 640x360, False: 1920x1080)
  • hsapi.high
    High-Level API
    • GetDevice(index=0)
      Get connected device
    • class Net(object)
      Abstract class, which performs some simple management on the Device class and the Graph class.
      • scale(self)
        Image preprocessing parameters (subclasses must implement)
      • mean(self)
        Image preprocessing parameters (subclasses must implement)
      • netSize(self)
        input image size (subclasses must implement)
      • graphPath(self)
        graph file path (subclasses must implement)
      • cleanup(self)
        Release neural network resources
      • quit(self)
        Release resources and close the device
      • run(self, image=None, **kwargs)
        Perform one neural network
        • image Input image, ‘None’ means using built-in camera
    • class SingleTask
      This class manages two child threads to perform a single neural network task.
      • init(self, net, **kwargs)
        Instantiation method
        • net Net instance
      • start(self)
        Start task
      • stop(self)
        Stop task
  • hsapi.easy
    Some convenient APIs using built-in neural networks
    • GetDefaultGraphRelPath(fileName)
      Get the default path of graphs
    • GetDefaultMiscRelPath(fileName)
      Get the default path of misc
    • class SSD(object)
      Built-in ‘MobileNet-SSD’ abstract class
      • labels(self)
        The labels of categories (subclasses must implement)
      • thresh(self) threshold (subclasses must implement)
      • getBoundingBoxes(self, output, image_size)
        Parsing location information from the results of the output of the neural network
        • output Neural network output
        • image_size Image size
      • overlay(self, image, boxes)
        Overlay position information on the image
        • image image
        • boxes position information
      • plot(self, result)
        Overlay position information on the image based on the output
        • result Output result
      • crop(self, result, square=True)
        Crop the recognized image based on the output
        • result Output result
        • square Square
    • class ObjectDetector(Net, SSD)
      Built-in Model - SSD-based object detection
    • class FaceDetector(Net, SSD)
      Built-in Model - SSD-based face detection
    • class FaceDetectorPlus(Net, SSD)
      Built-in Model - SSD-based face detection +
    • class Mnist(Net)
      Built-in Model - Mnist
    • class GoogleNet(Net)
      Built-in Model - GoogleNet
    • class SceneRecorder(GoogleNet)
      Built-in Model - GoogleNet-based scene recorder
      • record(self, result, key, **kwargs)
        Perform an ANN search based on GoogleNet output, returning similarity results
        • result GoogleNet output
        • key command
    • class SketchGuess(Net)
      Built-in Model - Sketch Recognition