Nnet manager

NNetManager is a class that is made to help you with setting up neural networks (NN). It is also responsible for all NN related functionalities. It’s capable of creating appropriate nodes and connections, decoding neural network output automatically or by using external handler file.

Getting started

To get started we first have to know some things that the manager offers. Firstly the manager is responsible for running our NN, which means that our manager will need a blob to work with. We can pass the blob to our NNetManager either with our BlobManager or we can pass the blob directly from the blobconverter module. We now have our blob, now we need to declare our pipeline through which our NN will be receiving data. This step is best done with the help of PipelineManager, as the manager already contains methods for NN nodes (addNN and setNnManager methods). After initializing all of that we are almost done. For convenience we can use PreviewManager to parse our frames, but this step is not needed as te NNetManager is able to parse raw frames. Bellow you can see 2 projects that use the NNetManager. Every major step is also commented for better understanding.

Face detection

from depthai_sdk import Previews
from depthai_sdk.managers import PipelineManager, PreviewManager, NNetManager, BlobManager
import depthai as dai
import cv2

# create pipeline
pm = PipelineManager()

# define camera source (in this case color and change winow size to 600 x 500)
pm.createColorCam(xout=True)

# define project that you wish to run
bm = BlobManager(zooName="face-detection-retail-0004")

# define Neural network configs
nm = NNetManager(inputSize=(300, 300), nnFamily="mobilenet")
nn = nm.createNN(pipeline=pm.pipeline, nodes=pm.nodes, source=Previews.color.name,
                 blobPath=bm.getBlob(shaves=6, openvinoVersion=pm.pipeline.getOpenVINOVersion()))
pm.addNn(nn)

# connect to device
with dai.Device(pm.pipeline) as device:
    # define configs for above sources
    pv = PreviewManager(display=[Previews.color.name])

    # create stream and neural network queues
    pv.createQueues(device)
    nm.createQueues(device)
    nnData = []

    while True:
        # read frames
        pv.prepareFrames()
        inNn = nm.outputQueue.tryGet()

        if inNn is not None:
            nnData = nm.decode(inNn)

        # draw information on frame and show frame
        nm.draw(pv, nnData)
        pv.showFrames()

        # end program with 'q'
        if cv2.waitKey(1) == ord('q'):
            break

In this above example we will use all classes that we learned before and run the face detection project. First we define the pipeline and initialize the streams. After that we load in our blob (face-detection-retail-0004) and send it in to our NNetManager. Every project has its own inputSize (desired NN input size, which should match input size defined in the network itself (width, height)) the and familyName (supported NN types / family names are YOLO and nobilenet). After all that is initialized, we add our neural network to our pipeline and connect to our device. In our device we set our Previews, to see our stream and create our stream queues. Like every other class that we covered, we need a loop, that will keep our project running, and in our loop, we get our frames, use our neural network to draw over our frames, and then we show them on the stream.

Outputs of our above program should look like this:

If our face is shown, our neural network detects it, but if we cover it, our neural network will not detect it.

Mobile net

from depthai_sdk import Previews
from depthai_sdk.managers import PipelineManager, PreviewManager, NNetManager, BlobManager
import depthai as dai
import cv2

# create pipeline
pm = PipelineManager()

# define camera source (in this case color and change winow size to 600 x 500)
pm.createColorCam(xout=True)

# define project that you wish to run
bm = BlobManager(zooName="mobilenet-ssd")

# define Neural network configs
nm = NNetManager(inputSize=(300, 300), nnFamily="mobilenet")
nn = nm.createNN(pipeline=pm.pipeline, nodes=pm.nodes, source=Previews.color.name,
                 blobPath=bm.getBlob(shaves=6, openvinoVersion=pm.pipeline.getOpenVINOVersion()))
pm.addNn(nn)

# connect to device
with dai.Device(pm.pipeline) as device:
    # define configs for above sources
    pv = PreviewManager(display=[Previews.color.name])

    # create stream and neural network queues
    pv.createQueues(device)
    nm.createQueues(device)
    nnData = []

    while True:
        # read frames
        pv.prepareFrames()
        inNn = nm.outputQueue.tryGet()

        if inNn is not None:
            nnData = nm.decode(inNn)

        # draw information on frame and show frame
        nm.draw(pv, nnData)
        pv.showFrames()

        # end program with 'q'
        if cv2.waitKey(1) == ord('q'):
            break

This example shows how to use the MobileNetSSD project. The code should be almost the same as the one that we used in the above example, with the only difference being the blob. In this example we load the mobilenet-ssd blob and pass it to our neural network.

class depthai_sdk.managers.NNetManager

Manager class handling all NN-related functionalities. It’s capable of creating appropriate nodes and connections, decoding neural network output automatically or by using external handler file.

__init__(inputSize, nnFamily=None, labels=[], confidence=0.5, sync=False)

Parameters

• inputSize (tuple) – Desired NN input size, should match the input size defined in the network itself (width, height)

• nnFamily (str, Optional) – type of NeuralNetwork to be processed. Supported: "YOLO" and mobilenet

• labels (list, Optional) – Allows to display class label instead of ID when drawing nn detections.

• confidence (float, Optional) – Specify detection nn’s confidence threshold

• sync (bool, Optional) – Store NN results for preview syncing (to be used with SyncedPreviewManager

sourceChoices = ('color', 'left', 'right', 'rectifiedLeft', 'rectifiedRight', 'host')

List of available neural network inputs

Type

list

source = None

Selected neural network input

Type

str

inputSize = None

NN input size (width, height)

Type

tuple

openvinoVersion = None

OpenVINO version, available only if parsed from config file (see readConfig())

Type

depthai.OpenVINO.Version

inputQueue = None

DepthAI input queue object that allows to send images from host to device (used only with host source)

Type

depthai.DataInputQueue

outputQueue = None

DepthAI output queue object that allows to receive NN results from the device.

Type

depthai.DataOutputQueue

buffer = {}

nn data buffer, disabled by default. Stores parsed nn data with packet sequence number as dict key

Type

dict

readConfig(path)

Parses the model config file and adjusts NNetManager values accordingly. It’s advised to create a config file for every new network, as it allows to use dedicated NN nodes (for MobilenetSSD and YOLO) or use custom handler to process and display custom network results

Parameters

path (pathlib.Path) – Path to model config file (.json)

Raises

• ValueError – If path to config file does not exist

• RuntimeError – If custom handler does not contain draw or show methods

createNN(pipeline, nodes, blobPath, source='color', useDepth=False, minDepth=100, maxDepth=10000, sbbScaleFactor=0.3, fullFov=True, useImageManip=True)

Creates nodes and connections in provided pipeline that will allow to run NN model and consume it’s results.

Parameters

• pipeline (depthai.Pipeline) – Pipeline instance

• nodes (types.SimpleNamespace) – Object cointaining all of the nodes added to the pipeline. Available in depthai_sdk.managers.PipelineManager.nodes

• blobPath (pathlib.Path) – Path to MyriadX blob. Might be useful to use together with depthai_sdk.managers.BlobManager.getBlob() for dynamic blob compilation

• source (str, Optional) – Neural network input source, one of sourceChoices

• useDepth (bool, Optional) – If set to True, produced detections will have spatial coordinates included

• minDepth (int, Optional) – Minimum depth distance in centimeters

• maxDepth (int, Optional) – Maximum depth distance in centimeters

• sbbScaleFactor (float, Optional) – Scale of the bounding box that will be used to calculate spatial coordinates for detection. If set to 0.3, it will scale down center-wise the bounding box to 0.3 of it’s original size and use it to calculate spatial location of the object

• fullFov (bool, Optional) – If set to False, manager will include crop offset when scaling the detections. Usually should be set to True (if you don’t perform aspect ratio crop or when keepAspectRatio flag on camera/manip node is set to False

• useImageManip (bool, Optional) – If set to False, manager will not create an image manip node for input image scaling - which may result in an input image being not adjusted for the NeuralNetwork node. Can be useful when we want to limit the amount of nodes running simultaneously on device

Returns

Configured NN node that was added to the pipeline

Return type

depthai.node.NeuralNetwork

Raises

RuntimeError – If source is not a valid choice or when input size has not been set.

getLabelText(label)

Retrieves text assigned to specific label

Parameters

label (int) – Integer representing detection label, usually returned from NN node

Returns

Label text assigned to specific label id or label id

Return type

str

Raises

RuntimeError – If source is not a valid choice or when input size has not been set.

parse(blocking=False)

decode(inNn)

Decodes NN output. Performs generic handling for supported detection networks or calls custom handler methods

Parameters

inNn (depthai.NNData) – Integer representing detection label, usually returned from NN node

Returns

Decoded NN data

Raises

RuntimeError – if outputFormat specified in model config file is not recognized

draw(source, decodedData)

Draws NN results onto the frames. It’s responsible to correctly map the results onto each frame requested, including applying crop offset or preparing a correct normalization frame, then draws them with all information provided (confidence, label, spatial location, label count).

Also, it’s able to call custom nn handler method draw to hand over drawing the results

Parameters

• source (depthai_sdk.managers.PreviewManager | numpy.ndarray) –

  Draw target. If supplied with a regular frame, it will draw the count on that frame

  If supplied with depthai_sdk.managers.PreviewManager instance, it will print the count label on all of the frames that it stores

• decodedData – Detections from neural network node, usually returned from decode() method

createQueues(device)

Creates output queue for NeuralNetwork node and, if using host as a source, it will also create input queue.

Parameters

device (depthai.Device) – Running device instance

closeQueues()

Closes output queues created by createQueues()

sendInputFrame(frame, seqNum=None)

Sends a frame into inputQueue object. Handles scaling down the frame, creating a proper depthai.ImgFrame and sending it to the queue. Be sure to use host as a source and call createQueues() prior input queue.

Parameters

• frame (numpy.ndarray) – Frame to be sent to the device

• seqNum (int, Optional) – Sequence number set on ImgFrame. Useful in synchronization scenarios

Returns

scaled frame that was sent to the NN (same width/height as NN input)

Return type

numpy.ndarray

Raises

RuntimeError – if inputQueue is None (unable to send the image)

countLabel(label)

Enables object count for specific label. Label count will be printed once draw() method is called

Parameters

label (str | int) – Label to be counted. If model is using mappings in model config file, supply here a str label to be tracked. If no mapping is present, specify the label as int (NN-default)