Face recognition with Flutter. The Fastest Way

If you're tasked with integrating face recognition into your mobile application and have chosen Flutter for development, you might encounter challenges in finding a suitable SDK or API. Fortunately, this article will guide you through the initial steps of face recognition (detection and optimal shot extraction) using Flutter and 3DiVi SDK 3.18.

In the latest 3DiVi Face SDK release, we've enhanced our Flutter API. With this API, you can:

Detect faces in photos and video frames.
Assess face liveness in both passive and cooperative modes.
Identify and verify individuals based on facial features.

The process of detection and optimal shot extraction looks like this: the client application captures frames from the smartphone camera, detects the user's face, selects the best shot, and forwards it to the server. The server application then evaluates the shot and provides feedback.

To accomplish face recognition and optimal shot selection, we utilize the `VideoWorker` class, which is tailored for tracking faces in video files and streams. In our application, the `VideoWorker` object runs on two distinct threads. On one thread, it consistently receives frames from the front camera.

format = Format.FORMAT_YUV_NV21;
convertRAW(img.planes, _data); // concatenates Android camera image planes
                            	  // into one array
_ri = RawImageF(img.width, img.height, format, _data.pointer!.cast()); // Create RawImage
                                                                       // object _videoWorker.addVideoFrame(_ri!, time); // Pass frame and current timestamp to VideoWorker

Concurrently, on the second thread, results from the `VideoWorker` are retrieved at regular intervals, for instance, every 50 milliseconds.

final callbackData = _videoWorker.poolTrackResults();
final rawSamples = callbackData.tracking_callback_data.samples;
final samplesQuality = callbackData.tracking_callback_data.samples_quality;

The `poolTrackResults` method returns tracking outcomes, encompassing objects like `TrackingCallbackData`, `TemplateCreatedCallbackData`, and `TrackingLostCallbackData`. For our needs, we focus on `TrackingCallbackData`, which offers data on face detections and their quality. Using methods from `RawSample`, we can get the track identifier for an individual and also record the quality rating of the detection in the frame.

if (rawSamples.length > 0) {
    for(var i = 0; i < rawSamples.length; i+=1) {
   	 personId = rawSamples[i].getID();
   	 sampleQuality = samplesQuality[i]

After storing this information for a few real-time seconds, we can identify the highest quality face image based on the person's identifier. For the `RawSample` object corresponding to the best image quality, we can then obtain the best shot.

Uint8List bestShot = bestRawSample.cutFaceImage(ImageFormatCode.IMAGE_FORMAT_JPG,
                                                FaceCutType.FACE_CUT_FULL_FRONTAL);

The above example demonstrates the ease with which you can integrate the Face SDK Flutter API into your project and address your challenges. To learn more about the capabilities of the Flutter API, check out the following resources:

Flutter Plugin Tutorial >

Flutter API Documentation >

Note: To use our Flutter API in your applications, please purchase the ESL Mobile license.

With the power of Flutter and the capabilities of the 3DiVi Face SDK, you can create a robust face recognition system that not only detects faces but also ensures that the best shots are captured and verified. Whether you're a seasoned developer or just starting out, this combination of tools and technology can help you achieve your project goals efficiently and effectively.