How to build AI facial expression detection into your videos with React

In today's tech-driven world, enhancing user interaction and creating immersive experiences is crucial. One exciting way to achieve this is by integrating AI facial expression detection into your video content. In this blog, we'll guide you through the steps to integrate AI facial expression detection to control your videos using React.

Let's start.

To learn how to do the integration with Javascript, head to this blog.

1. Create a minimal React app

As the first step, we need to create a new React app. Run the following command for the same.

Now add the MediaPipe and the api.video React player packages

2. Add a video player

You’re going to use the official api.video’s React player in this application, but feel free to use your favorite one and adapt the callbacks to it.

Replace the code inside App.tsx with the following. The playerRef allows us to control the api.video player.

Run your application with the npm run start command and go to http://localhost:3000/, you should see a video that you can play, pause or mute.

3. Record your facial expressions

To control the player with your facial expressions, you need to stream your face through your webcam and pass this streamed data to the trained machine-learning model.

Add a <video /> HTML tag to App.tsx. Its webcamRef allows the video element to be targeted. Apply a display: "none" inline style to hide the element, and an autoPlay attribute so the webcam stream time is updated.

Note: You can remove the display: "none" style and customize your CSS, it doesn’t matter.

4. Load the machine-learning model

Before we can use the machine-learning model, we must wait for it to finish loading. These models can be large and take a moment to get everything needed to run.

Put this logic inside a useEffect and store the faceLandmarker class as a React stateful value.

5. Start detecting the webcam stream

Once the machine-learning model has been loaded and the faceLandmarker class has been instantiated, you can start detecting the stream.

Create a new useEffect

Run your application with the npm run start command, go to http://localhost:3000/, and open your console. You should see a lot of console logs appearing each millisecond.

Each one of them logs an array of objects of type:

6. Use the facial expression detection results

You now have access to a list of facial expression detection results. Use them to do whatever you want and in this case, to control your video player.

Let’s assume that you want to play the video when you smile, and pause it when you do a kiss.

Check the facial expression detection results on each frame and play or pause your video depending on the score of the wanted result by replacing your console.log with the following.

Once again, run your application with the npm run start command, go to http://localhost:3000/, and try to smile at your webcam. It should trigger the player’s play callback and play the video. Then, try to do a kiss ( 😗 ) in front of your webcam, and the video should pause.

7. Display the results

It can be helpful to have a visual representation of the detection results. Display these results as a list and highlight the detected facial expressions by adding a <ul /> HTML tag to your JSX and creating a new ref to target it.

Add the code to update the HTML according to the results.

Now, you should see a list of results displayed in your application, with a green highlight on the ones with more than 60% confidence.

8. Code samples

Check the full App.tsx code below.

We hope this blog helped you understand the coding behind integrating AI facial expression detection to your videos.

For any questions or doubts you may have, chat with us on our website. If you would like to try out api.video for your videos, sign-up for a free sandbox account here.