C++ inferencing APIs

TensorFlow Lite utilities

The following APIs simplify your code when working with a tflite::Interpreter.

namespace coral¶

Functions

bool MatchShape(absl::Span<const int> shape, const std::vector<int> &pattern)¶

Checks whether a vector/tensor shape matches a dimensional pattern.

Negative numbers in the pattern indicate the corresponding shape dimension can be anything. Use -1 in the pattern for consistency.

Parameters

shape: The shape you want to evaluate.
pattern: The pattern to compare against.

Return

True if the shape matches, False if not.

absl::Span<const int> TensorShape(const TfLiteTensor &tensor)¶: Gets the tensor shape.

int TensorSize(const TfLiteTensor &tensor)¶: Gets the tensor size.

template<typename T> absl::Span<const T> TensorData(const TfLiteTensor &tensor)¶: Gets the immutable data from the given tensor.

template<typename T> absl::Span<T> MutableTensorData(const TfLiteTensor &tensor)¶: Gets the mutable data from the given tensor.

template<typename InputIt, typename OutputIt> OutputIt Dequantize(InputIt first, InputIt last, OutputIt d_first, float scale, int32_t zero_point)¶: Dequantizes the specified vector space.

template<typename T, typename OutputIt> OutputIt Dequantize(absl::Span<const T> span, OutputIt d_first, float scale, int32_t zero_point)¶: Returns a dequantized version of the given vector span.

template<typename T> std::vector<T> DequantizeTensor(const TfLiteTensor &tensor)¶: Returns a dequantized version of the given tensor.

template<typename InputIt, typename OutputIt> OutputIt Quantize(InputIt first, InputIt last, OutputIt d_first, float scale, int32_t zero_point)¶: Quantizes the specified vector space.

absl::Status MakeEdgeTpuInterpreter(const tflite::FlatBufferModel &model, edgetpu::EdgeTpuContext *tpu_context, tflite::ops::builtin::BuiltinOpResolver *resolver, tflite::StatefulErrorReporter *error_reporter, std::unique_ptr<tflite::Interpreter> *interpreter)¶

Creates a new interpreter instance for an Edge TPU model.

Also consider using MakeEdgeTpuInterpreterOrDie().

Parameters

model: The tflite model.
tpu_context: The Edge TPU context, from coral::GetEdgeTpuContext(). If left null, the given interpreter will not resolve an Edge TPU delegate. PoseNet custom op is always supported.
resolver: Optional. May be null to use a default resolver.
error_reporter: Optional. May be null to use default error reporter, but beware that if null, tflite runtime error messages will not return.
interpreter: The pointer to receive the new interpreter.

std::unique_ptr<tflite::Interpreter> MakeEdgeTpuInterpreterOrDie(const tflite::FlatBufferModel &model, edgetpu::EdgeTpuContext *tpu_context = nullptr, tflite::ops::builtin::BuiltinOpResolver *resolver = nullptr, tflite::StatefulErrorReporter *error_reporter = nullptr)¶

Returns a new interpreter instance for an Edge TPU model, crashing if it cannot be created.

For example:

const auto model = coral::LoadModelOrDie(model_path);
auto edgetpu_context = coral::ContainsEdgeTpuCustomOp(*model)
                           ? coral::GetEdgeTpuContextOrDie()
                           : nullptr;
auto interpreter =
    coral::MakeEdgeTpuInterpreterOrDie(*model, edgetpu_context.get());

Parameters

model: The tflite model.
tpu_context: The Edge TPU context, from coral::GetEdgeTpuContext(). If left null, the given interpreter will not resolve an Edge TPU delegate. PoseNet custom op is always supported.
resolver: Optional. May be null to use a default resolver.
error_reporter: Optional. May be null to use default error reporter, but beware that if null, tflite runtime error messages will not return.

Return

The new interpreter instance.

absl::Status SetTensorBuffer(tflite::Interpreter *interpreter, int tensor_index, const void *buffer, size_t buffer_size)¶

Replaces existing tensor buffer with the provided one.

Caller owns provided buffer. Tensor quantization parameters are preserved. This function is a required ‘hack’ for performance reasons until this functionality would become a part of TensorFlow Lite API.

std::shared_ptr<edgetpu::EdgeTpuContext> GetEdgeTpuContext(const std::string &device, const edgetpu::EdgeTpuManager::DeviceOptions &options = {})¶

Returns TPU context or nullptr if requested TPU context is not available.

Parameter device:

”” any TPU device
”usb” any TPU device on USB bus
”pci” any TPU device on PCIe bus
”:N” N-th TPU device, e.g. “:0”
”usb:N” N-th TPU device on USB bus, e.g. “usb:0”
”pci:N” N-th TPU device on PCIe bus, e.g. “pci:0”

Parameter options: See edgetpu.h for details.

All TPUs are always enumerated in the same order assuming hardware configuration doesn’t change (no added/removed devices between enumerations). Under the assumption above, the same index N will always point to the same device.

Consider 2 USB devices and 4 PCIe devices connected to the host. The way to reference specifically USB devices: “usb:0”, “usb:1”. The way to reference specifically PCIe devices: “pci:0”, “pci:1”, “pci:2”, “pci:3”. The generic way to reference all devices (no assumption about device type): “:0”, “:1”, “:2”, “:3”, “:4”, “:5”.

std::shared_ptr<edgetpu::EdgeTpuContext> GetEdgeTpuContextOrDie(const std::string &device, const edgetpu::EdgeTpuManager::DeviceOptions &options = {})¶: The same as above but crashes if requested TPU context is not available.

std::shared_ptr<edgetpu::EdgeTpuContext> GetEdgeTpuContext(absl::optional<edgetpu::DeviceType> device_type = absl::nullopt, absl::optional<int> device_index = absl::nullopt, const edgetpu::EdgeTpuManager::DeviceOptions &options = {})¶

The same as previously defined GetEdgeTpuContext except device parameter is replaced with two separate ones: device_type and device_index.

Custom options would only be passed when device_type and device_index are non-empty.

std::shared_ptr<edgetpu::EdgeTpuContext> GetEdgeTpuContextOrDie(absl::optional<edgetpu::DeviceType> device_type = absl::nullopt, absl::optional<int> device_index = absl::nullopt, const edgetpu::EdgeTpuManager::DeviceOptions &options = {})¶: The same as above but crashes if requested TPU context is not available.

std::unique_ptr<tflite::FlatBufferModel> LoadModelOrDie(const std::string &path)¶: Load a tflite model at the given file path or die trying.

std::unique_ptr<tflite::FlatBufferModel> LoadModelOrDie(const flatbuffers::FlatBufferBuilder &fbb)¶: Load a tflite model via flatbuffer or die trying.

absl::Status InvokeWithMemBuffer(tflite::Interpreter *interpreter, const void *buffer, size_t in_size, tflite::StatefulErrorReporter *reporter = nullptr)¶

Invokes tflite::Interpreter using a given buffer as an input tensor.

Parameters

interpreter: An initialized interpreter.
buffer: The interpreter input. We assume there is only one tensor.
in_size: The number of elements in the input buffer, which can have padding elements at the end. in_size can be larger than the input tensor size, denoted by n, and only the first n elements of the input buffer will be used. in_size can not be smaller than n.
reporter: Optional. If left null, tflite runtime error messages will not be returned. To get tflite runtime error messages, reporter must be set to the one that is used to create interpreter.

absl::Status InvokeWithDmaBuffer(tflite::Interpreter *interpreter, int dma_fd, size_t in_size, tflite::StatefulErrorReporter *reporter = nullptr)¶

Invokes tflite::Interpreter using a given DMA file descriptor as an input tensor.

Works only for Edge TPU models running on PCIe Edge TPU devices.

Parameters

interpreter: An initialized interpreter.
dma_fd: The DMA file descriptor to use as input.
in_size: The number of elements in the input buffer, which can have padding elements at the end. in_size can be larger than the input tensor size, denoted by n, and only the first n elements of the input buffer will be used. in_size can not be smaller than n.
reporter: Optional. If left null, tflite runtime error messages will not be returned. To get tflite runtime error messages, reporter must be set to the one that is used to create interpreter.

bool ContainsEdgeTpuCustomOp(const tflite::FlatBufferModel &model)¶: Checks whether a tflite model contains any Edge TPU custom operator.

std::unordered_set<std::string> GetInputTensorNames(const tflite::Interpreter &interpreter)¶: Returns all input tensor names for the given tflite::Interpreter.

const TfLiteTensor *GetInputTensor(const tflite::Interpreter &interpreter, const char *name)¶

Returns the input tensor matching name in the given tflite::Interpreter.

Returns nullptr if such tensor does not exist.

Image classification

Use the following APIs with image classification models.

[adapter.h source]

namespace coral

Functions

bool operator==(const Class &x, const Class &y)¶

bool operator!=(const Class &x, const Class &y)¶

std::string ToString(const Class &c)¶

std::ostream &operator<<(std::ostream &os, const Class &c)¶

std::vector<Class> GetClassificationResults(absl::Span<const float> scores, float threshold = -std::numeric_limits<float>::infinity(), size_t top_k = std::numeric_limits<size_t>::max())¶

Converts classification output tensors into a list of ordered classes.

Parameters

scores: The classification output tensor (dequantized).
threshold: The score threshold for results. All returned results have a score greater-than-or-equal-to this value.
top_k: The maximum number of predictions to return.

Return

The top_k Class predictions, <score, label_id>, ordered by score (first element has the highest score).

std::vector<Class> GetClassificationResults(const tflite::Interpreter &interpreter, float threshold = -std::numeric_limits<float>::infinity(), size_t top_k = std::numeric_limits<size_t>::max())¶

Gets results from a classification model as a list of ordered classes.

Parameters

interpreter: The already-invoked interpreter for your classification model.
threshold: The score threshold for results. All returned results have a score greater-than-or-equal-to this value.
top_k: The maximum number of predictions to return.

Return

The top_k Class predictions, <score, label_id>, ordered by score (first element has the highest score).

Class GetTopClassificationResult(const tflite::Interpreter &interpreter)¶

Gets only the top result from a classification model.

Parameters

interpreter: The already-invoked interpreter for your classification model.

Return

The top Class prediction, <score, label_id>.

struct Class¶

#include <adapter.h>

Represents a single classification result.

Public Members

int id¶: The class label id.

float score¶: The prediction score.

Object detection

Use the following APIs with object detection models.

[adapter.h source]

namespace coral

Functions

bool operator==(const Object &x, const Object &y)¶

bool operator!=(const Object &x, const Object &y)¶

std::string ToString(const Object &obj)¶

std::ostream &operator<<(std::ostream &os, const Object &obj)¶

std::vector<Object> GetDetectionResults(absl::Span<const float> bboxes, absl::Span<const float> ids, absl::Span<const float> scores, size_t count, float threshold = -std::numeric_limits<float>::infinity(), size_t top_k = std::numeric_limits<size_t>::max())¶

Converts detection output tensors into a list of SSD results.

Parameters

bboxes: Bounding boxes of detected objects. Four floats per object (box-corner encoding [ymin1,xmin1,ymax1,xmax1,ymin2,xmin2,…]).
ids: Label identifiers of detected objects. One float per object.
scores: Confidence scores of detected objects. One float per object.
count: The number of detected objects (all tensors defined above have valid data for only this number of objects).
threshold: The score threshold for results. All returned results have a score greater-than-or-equal-to this value.
top_k: The maximum number of predictions to return.

Return

The top_k Object predictions, <id, score, bbox>, ordered by score (first element has the highest score).

std::vector<Object> GetDetectionResults(const tflite::Interpreter &interpreter, float threshold = -std::numeric_limits<float>::infinity(), size_t top_k = std::numeric_limits<size_t>::max())¶

Gets results from a detection model as a list of ordered objects.

Parameters

interpreter: The already-invoked interpreter for your detection model.
threshold: The score threshold for results. All returned results have a score greater-than-or-equal-to this value.
top_k: The maximum number of predictions to return.

Return

The top_k Object predictions, <id, score, bbox>, ordered by score (first element has the highest score).

struct Object¶

#include <adapter.h>

Represents a detected object.

Public Members

int id¶: The class label id.

float score¶: The prediction score.

BBox<float> bbox¶: A BBox defining the bounding-box (ymin,xmin,ymax,xmax).

[bbox.h source]

namespace coral

Functions

template<typename T> bool operator==(const BBox<T> &a, const BBox<T> &b)¶

template<typename T> bool operator!=(const BBox<T> &a, const BBox<T> &b)¶

template<typename T> std::string ToString(const BBox<T> &bbox)¶

template<typename T> std::ostream &operator<<(std::ostream &stream, const BBox<T> &bbox)¶

template<typename T> BBox<T> Intersection(const BBox<T> &a, const BBox<T> &b)¶: Gets a BBox representing the intersection between two given boxes.

template<typename T> BBox<T> Union(const BBox<T> &a, const BBox<T> &b)¶: Gets a BBox representing the union of two given boxes.

template<typename T> float IntersectionOverUnion(const BBox<T> &a, const BBox<T> &b)¶: Gets the intersection-over-union value for two boxes.

template<typename T> struct BBox¶

#include <bbox.h>

Represents the bounding box of a detected object.

Public Functions

T width() const¶: Gets the box width.

T height() const¶: Gets the box height.

T area() const¶: Gets the box area.

bool valid() const¶: Checks whether the box is a valid rectangle (width >= 0 and height >= 0).

Public Members

T ymin¶: The box y-minimum (top-most) point.

T xmin¶: The box x-minimum (left-most) point.

T ymax¶: The box y-maximum (bottom-most) point.

T xmax¶: The box x-maximum (right-most) point.

Public Static Functions

BBox<T> FromCenterSize(T center_y, T center_x, T height, T width)¶

Creates a BBox (box-corner encoding) from centroid box encodings.

Parameters

center_y: Box center y-coordinate.
center_x: Box center x-coordinate.
height: Box height.
width: Box width.

Return

A BBox instance.

Is this content helpful?