BRFv5 - API Reference

The BRFv5 API is the same for all platforms. Everything described on this page can be applied to JavaScript, ActionScript 3, C++ etc..

Before you read this API, please take a look at what you can do with BRFv5, especially what results to expect from Face Detection and Face Tracking (link: What can I do with it?).

There are three main classes: BRv5FManager, BRFv5Config and BRFv5Face.

BRFv5Manager controls the SDK.

BRFv5's configuration defaults are reasonably set for a 640x480 landscape input and to track a single face.

If that's the case for your app then you don't need to configure the SDK. But in most cases, you need to call configure() before starting the tracking by calling update().

The JavaScript SDK package is a good example for that. It covers configuration for desktop, mobile, landscape and portrait apps.

Please see BRFv5Config for more info about each parameter.

To finish the configuration, once all BRFv5Config properties are set, call:

configure( config : BRFv5Config )

When everything is configured you need to prepare the input imageData before calling:

update( image : ImageData )


BRFv5 consists of two steps: Face Detection and Face Tracking.

Face Detection operates globally, it is not bound to a BRFv5Face.

The Face Detection results can be retrieved by calling:

getDetectedRects() : Vector<BRFv5Rectangle>
getMergedRects() : Vector<BRFv5Rectangle>

The merged rectangles are the starting areas for the Face Tracking step.

The detailed Face Tracking data can be retrieved by calling:

getFaces() : Vector<BRFv5Face>

which gives you a list of BRFv5Faces with a length based on your setting of:

BRFv5Config.faceTrackingConfig.numFacesToTrack : int

And that's all about it. Configure, then continuously update and use the results.

For more information about the implementation details for each specific platform, see the platform specific example packages.

configure( config : BRFv5Config ) : void

This function configures the SDK. Please see BRFv5Config for more information about each parameter.

update( image : ImageData ) : void

This function does the tracking job.

In C++ implementations, eg. OpenCV (camera handling) or iOS, you need to update the imageData on your own before calling update.

All other platforms have their known image containers and BRFv5 is already set up to deal with those.

The results can be retrieved after calling update() as arrays of BRFv5Rectangle or BRFv5Face:

getDetectedRects() : Vector<BRFv5Rectangle>
getMergedRects() : Vector<BRFv5Rectangle>
getFaces() : Vector<BRFv5Face>

imageData (a platform dependent image data type)

This is platform dependent. In C++ implementations, eg. OpenCV (camera handling) or iOS, you need to update the imageData on your own before calling update. All other platforms have their known image containers and BRFv5 is already set up to deal with those, eg. Android(Java): Bitmap, AS3: BitmapData etc.

reset() : void

All face trackers reset and start over by performing Face Detection. And by the way: To pause the tracking, simply don't call update anymore.

getDetectedRects( ) : Vector<BRFv5Rectangle>

Returns all rectangles in the image that the algorithm thinks is a face (blue). If enough of those rects are found in one area a merged rectangle is created that is the starting area for the Face Tracking step. Will be empty if all face trackers are active and no new face needs to be detected.

getMergedRects( ) : Vector<BRFv5Rectangle>

Returns all actual merged/detected faces (white) of the last update call. Will be empty if all face trackers are active and no new face needs to be detected.

getFaces( ) : Vector<BRFv5Face>

Returns the list of currently tracked faces. The length of this list depends on your

BRFv5Config.faceTrackingConfig.numFacesToTrack : int

value.

Please see BRFv5Face for details.

Setting the correct BRFv5Config parameters before you start tracking is crucial.

BRFv5's configuration defaults are reasonably set for a 640x480 landscape input and to track a single face.

But in most cases, you need to call configure() before starting the tracking by calling update().

It's highly recommended to take a look at the JavaScript SDK package on GitHub to see how to configure BRFv5 for each use case.


Internally BRFv5 works on XYZx480 (landscape) or 480xXYZ (portrait) image for it's analysis. So 480px is the base size that every other input size scales to (eg. 1280x720 -> 854x480 or 1920x1080 -> 854x480).

The minimum detectable face sizes for the following resolutions are:

  640 x   480: 24px (  480 / 480 = 1.00 * 24 = 24) (base)
1280 x   720: 36px (  720 / 480 = 1.50 * 24 = 36)
1920 x 1080: 54px (1080 / 480 = 2.25 * 24 = 54)

enableFaceTracking : Boolean

If you don't need the Face Tracking part of BRFv5, set this value to false to deactivate it and only detect faces. Might also come in handy, if you want to configure your Face Detection part properly.

imageConfig.inputWidth : int

This value tells the SDK what width the input imageData will have. It's recommended to use a 640x480 or 480x640 imageData, since any other size will trigger an additional scaling step internally.

imageConfig.inputHeight : int

This value tells the SDK what height the input imageData will have. It's recommended to use a 640x480 or 480x640 imageData, since any other size will trigger an additional scaling step internally.

regionOfInterest : BRFv5Rectangle

Sets the region of interest, the area within the imageData where Face Detection is performed.

minFaceSize : int

The minimum detectable face sizes (in pixel) for the following resolutions are:

  640 x   480: 24px (  480 / 480 = 1.00 * 24 = 24) (base)
1280 x   720: 36px (  720 / 480 = 1.50 * 24 = 36)
1920 x 1080: 54px (1080 / 480 = 2.25 * 24 = 54)

Increase that value as much as possible for your use case. The smaller faces you want to detect, the less performance you will get for Face Detection.

maxFaceSize : int

The maximum detectable face sizes (in pixel) for the following resolutions are:

  640 x   480: 480px (base)
1280 x   720: 720px
1920 x 1080: 1080px

Not so much a performance hit, if you use the maximum possible value. But the smaller the better.

faceSizeIncrease : int

A value multiple of 12. Default is 24. Faces (detectedRects: blue) are detected in 'layers'. The detection window size is increased by faceSizeIncrease. Setting this to 12 will double the amount of detection attempts, because the detection windows will only grow by 12, instead of 24.

For 24:

  640 x   480: 24,   48,   72, ...,   456,   480
1280 x   720: 36,   72, 108, ...,   684,   720
1920 x 1080: 54, 108, 162, ..., 1026, 1080

For 12:

  640 x   480: 24, 36,   48,   60,   72, ...,   456,   468,   480
1280 x   720: 36, 54,   72,   90, 108, ...,   684,   702,   720
1920 x 1080: 54, 81, 108, 135, 162, ..., 1026, 1053, 1080

stepSize : int

Values: 0 (default, means it's dynamically adjusted by the SDK), (don't set it to 1, this will kill your app), >= 2

At a window size layer the detection window will move by stepSize in x direction and once it's at the end of the row, will move stepSize in y direction and do the next row.

0 is the (recommended) default. The SDK will have a smaller stepSize for smaller windows and a larger stepSize for larger windows.
2 is the most detailed detection, but performs the slowest.
> 2 might be sufficient for your use case, but 0 is recommended.

minNumNeighbors : int

Detected faces (blue) get merged (white) if they are

+ roughly placed in the same location,
+ roughly the same size and
+ have at least minNumNeighbors of other detected faces in the same spot.

rectMergeFactor : float

Detected faces (blue) get merged (white) if they are

+ roughly placed in the same location,
+ roughly the same size and
+ have at least minNumNeighbors of other detected faces in the same spot.

rectMergeFactor is used to check if the center points of detected rects are close together and not more than rectMergeFactor * width px away from each other.
0.0 won't create merged rectangles.
1.0 would merge rectangles, even if they are width px apart.
Default: 0.2, don't bother, I guess.

rectSkipFactor : float

rectSkipFactor is used to skip detection windows that are less rectSkipFactor * width px away from an already merged rectangle.
0.0 won't skip any rectangles.
1.0 would skip rectangles, even if they are width px apart.
Default: 0.5, don't bother, I guess.

filterNoise : Boolean

If true, filters all detected rectangles, that don't belong to a merged rectangle. If false, detectedRects will include all stray detected rects.

regionOfInterest : BRFv5Rectangle

If a tracked faces moves outside of this area, it's being reset.

numFacesToTrack : int

The number of faces to track. Default is 1 (single Face Tracking). The more detailed face trackings you want to do in parallel, the slower the SDK will perform. Two faces might be possible in real time, anything more than that won't be real time anymore.

numTrackingPasses : int

Values: 1, 2, 3, 4, 5, 6. Default: 3.

The number of Face Tracking passes on a face. If this value is >=3 a simple confidence value is calculated, that helps resetting the face, if it was lost. The more tracking passes you perform, the slower the tracking will be. It's a trade-off between fast execution and accuracy. The JavaScript SDK examples set numTrackingPasses dynamically depending on the CPU usage.

minFaceScaleStart : float

By setting these *Start parameters you can control the conditions for picking up a face. Restrict the distance/range to the camera by setting minFaceScaleStart and maxFaceScaleStart. Smaller values mean more distance to the camera. Restricting the starting angles may help to create a consistent user experience.

If BRFv5Face.scale is smaller than minFaceScaleStart, the face is reset and not tracked.

maxFaceScaleStart : float

If BRFv5Face.scale is larger than maxFaceScaleStart, the face is reset and not tracked.

maxRotationXStart : float

If BRFv5Face.rotationX is larger than maxRotationXStart, the face is reset and not tracked.

maxRotationYStart : float

If BRFv5Face.rotationY is larger than maxRotationYStart, the face is reset and not tracked.

maxRotationZStart : float

If BRFv5Face.rotationZ is larger than maxRotationZStart, the face is reset and not tracked.

confidenceThresholdStart : int

If BRFv5Face.confidence is smaller than confidenceThresholdStart, the face is reset and not tracked (only applies, if BRFv5Config.faceTrackingConfig.numTrackingPasses >= 3).

minFaceScaleReset : float

By setting these parameters you can control the conditions for resetting the tracking of a face.

If BRFv5Face.scale is smaller than minFaceScaleReset, the face is reset and not tracked.

maxFaceScaleReset : float

If BRFv5Face.scale is larger than maxFaceScaleReset, the face is reset and not tracked.

maxRotationXReset : float

If BRFv5Face.rotationX is larger than maxRotationXReset, the face is reset and not tracked.

maxRotationYReset : float

If BRFv5Face.rotationY is larger than maxRotationYReset, the face is reset and not tracked.

maxRotationZReset : float

If BRFv5Face.rotationZ is larger than maxRotationZReset, the face is reset and not tracked.

confidenceThresholdReset : int

If BRFv5Face.confidence is smaller than confidenceThresholdReset, the face is reset and not tracked (only applies, if BRFv5Config.faceTrackingConfig.numTrackingPasses >= 3 and if confidenceThresholdReset > 0.0).

enableStabilizer : Boolean

BRFv5 stabilizes fast and slow movements. You can turn that off, eg. if your users will have their face always in the same spot, to save some CPU cycles.

enableFreeRotation : Boolean

BRFv5 makes it possible to tilt the head more than 90 degrees (rotationZ). There is some extra calculations involved to achieve this. Turn it off to restrict the head tilt to 34 degrees and save some CPU cycles.

BRFv5Face holds all information about a tracked face. BRFv5 is able to track more than one face, so every BRFv5Face has it's individual state and tracked properties. BRFv5 sets the state automatically depending on the detection or tracking results.

BRFv5 includes two types of models. Models with 68 landmarks (standard) and models with 42 landmarks (for 3D object placement). In the 42 landmark models the indices 37, 38, 40, 41, 43, 44, 46, 47 and all mouth landmarks, except 60 and 64 are missing.



The origin for the 3D transformation (scale, translationX, translationY, rotationX, rotationY, rotationZ) is right behind landmark index 27 (top of the nose). So if you want to put a 3D object or PNG file on top of a tracked face you know exactly where to position it.

state : String

The state of the face. It depends on the current tracking results and is either one of the following BRFv5States:

BRFv5State.RESET

This face was reset because of a call to BRFv5Manager.reset() or because it exceeded the reset parameters.

BRFv5State.FACE_DETECTION

BRFv5 is looking for a face, but still found not enough evidence.

BRFv5State.FACE_TRACKING

BRFv5 aligned the face and is now tracking it.

confidence : float

When BRFv5Config.faceTrackingConfig.numTrackingPasses >= 3, a simple confidence value is calculated. This value lies between 0.0 and 1.0. Usually you want >= 0.8 value for the start and if it drops to 0.0, the face was lost (maybe through fast movements or bad lighting conditions). This value is just a difference calculation of the different tracking passes, nothing too fancy.

vertices : float[]

An array of landmark positions in the form of [x, y, x, y, ..., x, y], length is therefore: 68 * 2 (or 42 * 2)



Getting the position of a certain landmark would be like:

var vx = vertices[index * 2]; // where index is eg. 27 for the top of the nose.
var vy = vertices[index * 2 + 1];

landmarks : Vector<BRFv5Landmark>

Same as vertices, but a list of points in the form of [{x, y}, {x, y}, ..., {x, y}], length is therefore: 68 (or 42).

Getting the position of a certain point would be like:

var px = landmarks[index].x; // where index is eg. 27 for the top of the nose.
var py = landmarks[index].y;

bounds : BRFv5Rectangle

This is the outline rectangle of all landmarks.

scale : float

The scale of the whole face.

translationX : float

The x position of the origin of face on the image. The origin is right behind landmark index 27 (top of the nose).

translationY : float

The y position of the origin of face on the image. The origin is right behind landmark index 27 (top of the nose).

rotationX : float

A degree angle. Rotation around the X axis (pitch, turning head up/down)

rotationY : float

A degree angle. Rotation around the Y axis (yaw, turning head to the left/right)

rotationZ : float

A degree angle. Rotation around the Z axis (roll, tilt head to the left/right)

A BRFv5Rectangle is a rectangle, which marks an area. Usually used to define regions of interest.

x : float

The x value of that rectangle.

y : float

The y value of that rectangle.

width : float

The width value of that rectangle.

height : float

The height value of that rectangle.

setTo( x : float, y : float, width : float, height : float) : void

With this function x, y, width and height can be set in one go.

A BRFv5Landmark is a point with x and y property. A convenience class.

x : float

The x value of that landmark. (Yeah!)

y : float

The y value of that landmark. (Seriously!)

setTo( x : float, y : float) : void

With this function x and y can be set in one go.

A BRFv5Face can be in one of the following states.

FACE_DETECTION : String

The BRFv5Face tried to initially detect a face in the image data. If BRFv5Config.enableFaceTracking = false, the BRFv5Face will always be in this state, except when you called BRFv5Manager.reset() manually.

FACE_TRACKING : String

For this BRFv5Face Face Tracking was performed and all properties (landmarks, vertices, translationX etc.) were updated.

RESET : String

The BRFv5Face triggered a reset or you called BRFv5Manager.reset() manually. Resetting occurs automatically, if a *Start or *Reset parameter was exceeded, or, if BRFv5Config.faceTrackingConfig.numFacesToTrack >= 3, the confidence value is lower than expected.