Face Landmark Detection using Python

使用Python进行面部关键点检测

Face Landmark Detection using Python

使用Python进行面部关键点检测

The comparison between dlib and mediapipe library.

dlib与mediapipe库的比较

Introduction

引言

Face landmark detection is a computer vision task where we want to detect and track keypoints from a human face. This task applies to many problems.

面部关键点检测是一项计算机视觉任务，我们想要从人脸中检测并跟踪关键点。该任务适用于许多问题。

For example, we can use the keypoints for detecting a human’s head pose position and rotation. With that, we can track whether a driver is paying attention or not. Also, we can use the keypoints for applying an augmented reality easier. And there are so many solutions that we can generate based on this task.

例如，我们可以利用关键点来检测人头的姿态位置和旋转角度。据此，我们可以追踪驾驶员是否在专心驾驶。此外，我们还可以利用关键点更轻松地应用增强现实技术。基于这项任务，我们可以衍生出许多解决方案。

Thankfully, we don’t have to understand the concepts of face landmark detection in detail. We can use the prebuilt library like dlib, OpenCV, and mediapipe. In this article, I will show you how to implement face landmark detection with dlib and mediapipe.

幸运的是，我们不需要详细了解面部关键点检测的概念。我们可以使用像dlib、OpenCV和mediapipe这样的预建库。在本文中，我将向您展示如何使用dlib和mediapipe实现面部关键点检测。

Without further, let’s get started!

事不宜迟，让我们开始吧！

Face Landmark Detection with Dlib

使用Dlib进行面部关键点检测

Dlib is a library for applying machine learning and computer vision solutions. This library is based on the C++ language, but we can use a language like Python for using the library. One of the solutions that we can apply by using this library is face landmark detection. Now let’s get into the implementation.

Dlib是一个用于应用机器学习和计算机视觉解决方案的库。该库基于C++语言，但我们也可以使用Python等语言来使用它。我们可以通过该库实现的解决方案之一就是面部关键点检测。现在让我们进入具体实现。

Install the library

安装库

Installing a library can become a problem. If we don’t have a good guide, installing the library can take several days. Dlib is one of them. Because it uses C++ as the primary language, we have to install C++ tools for installing the library.

安装库有时会成为一个难题。如果没有好的指南，安装库可能需要好几天的时间。Dlib就是其中之一。由于它使用C++作为主要语言，我们必须安装C++工具才能安装该库。

There are several steps that we should do to install it. Here are the steps:

安装它需要几个步骤，具体如下：

First, install the CMake. You can download the software here. If you are using Windows, please locate the CMake file path first. Then, set the path to the executable path on the environment variable.

首先，安装CMake。你可以从这里下载该软件。如果你使用的是Windows，请先找到CMake文件路径，然后将路径添加到环境变量的可执行路径中。

Then, install Visual Studio with the C++ dependencies to it. You can download the software here. For the dependencies, you can look at this screenshot below:

然后，安装带有C++依赖项的Visual Studio。你可以从这里下载该软件。关于依赖项，你可以参考下面的截图：

After you install the Visual Studio, the next step is to install the Python. To make your installation simpler, I recommend you for installing Anaconda. You can download it here. For the Python version, I recommend you for using the 3.6.6 version to avoid any errors.

安装完Visual Studio之后，下一步是安装Python。为了让安装更简单，我建议你安装Anaconda。你可以从这里下载。关于Python版本，我建议使用3.6.6版本以避免任何错误。

Lastly, install the CMake, dlib, and OpenCV library by using pip. Here is the command for doing that:

最后，使用pip安装CMake、dlib和OpenCV库。以下是执行该操作的命令：

Import the libraries

导入库

After we’ve installed the libraries, the next step is to import them into our code. We will import OpenCV for retrieving inputs from the webcam, NumPy for numerical computation, and Dlib for detecting keypoints from a face.

安装完库之后，下一步是将它们导入到我们的代码中。我们将导入OpenCV用于从网络摄像头获取输入，NumPy用于数值计算，以及Dlib用于从人脸检测关键点。

Here is the code for doing that:

以下是执行该操作的代码：

Initialize the objects

初始化对象

Now let’s initialize several variables. There are three must need variables that we will initialize:

现在让我们初始化几个变量。我们需要初始化三个必需的变量：

A detector for detecting one or more faces. We set the dlib.get_frontal_face_detector function inside the variable.

一个用于检测一张或多张人脸的检测器。我们将dlib.get_frontal_face_detector函数赋值给该变量。

A predictor for detecting keypoints from faces. We set the dlib.shape_predictor function inside the variable. This function needs a pretrained model location as the parameter, which you can download here.

一个用于从人脸检测关键点的预测器。我们将dlib.shape_predictor函数赋值给该变量。该函数需要一个预训练模型的位置作为参数，你可以从这里下载。

The cv2.VideoCapture object for capturing images from the webcam. Also, we set a parameter with value 0 for capturing images from a webcam.

用于从网络摄像头捕获图像的cv2.VideoCapture对象。同时，我们设置参数值为0，用于从网络摄像头捕获图像。

Let’s write this code for initializing variables:

让我们编写以下代码来初始化变量：

Face landmark detection mechanism

面部关键点检测机制

As you can see from above, we initialize the face landmark detector by using the pretrained model. The model is based on ensemble regression trees because the model will predict continuous numbers. You can read the details about the model here.

从上面可以看出，我们使用预训练模型初始化了面部关键点检测器。该模型基于集成回归树，因为模型要预测连续数值。你可以在这里阅读关于该模型的详细信息。

That model is trained on the iBUG-300 W dataset, where it contains images and their corresponding 68 face landmark points. In general, those landmark points belong to the nose, the eyes, the mouth, and the edge of a face. You can download the dataset here.

该模型在iBUG-300 W数据集上训练，该数据集包含图像及其对应的68个面部关键点。一般来说，这些关键点属于鼻子、眼睛、嘴巴和面部轮廓。你可以在这里下载该数据集。

Here is the visualization of the face landmark locations below:

以下是面部关键点位置的可视化图示：

Implement the face landmark detection

实现面部关键点检测

Now you know how the face landmark detection algorithm works. Now let’s implement the algorithm. For implementing that, you can see the code below along with explanations on each line of code:

现在你了解了面部关键点检测算法的工作原理。现在让我们实现该算法。你可以在下面看到代码以及每行代码的解释：

By combining all the code as one, now let’s try the code! If the code doesn’t have any errors, the webcam will display the result along with the keypoints. In my case, here is the result:

将所有代码整合在一起，现在让我们运行代码！如果代码没有错误，网络摄像头将显示带有关键点的结果。以我为例，结果如下：

Face Landmark Detection with Mediapipe

使用Mediapipe进行面部关键点检测

Mediapipe is a tool for implementing ML-based computer vision solutions. The tool is created by Google.

Mediapipe是一个用于实现基于机器学习的计算机视觉解决方案的工具。该工具由Google创建。

This tool contains varieties computer vision solutions, such as face detection, pose estimation, object detection, and many more.

该工具包含各种计算机视觉解决方案，如人脸检测、姿态估计、目标检测等。

The advantage of this library is that you can apply the solutions on many platforms, such as web, mobile, PC, and many more.

该库的优点是你可以将解决方案应用于多种平台，如网页、移动端、PC等。

I’ve already explained in the previous section to you how to implement face landmark detection using dlib. Now let’s implement the face landmark detection using Mediapipe.

我已经在上一节向你展示了如何使用dlib实现面部关键点检测。现在让我们使用Mediapipe来实现面部关键点检测。

The mechanism

工作原理

The library uses the BlazeFace model for detecting face landmarks. BlazeFace is a deep learning model that is already optimized for low spec devices like smartphones. Therefore, we can use the model in real-time.

该库使用BlazeFace模型来检测面部关键点。BlazeFace是一个深度学习模型，已经针对智能手机等低规格设备进行了优化。因此，我们可以实时使用该模型。

BlazeFace contains two main steps. First, the model detects one or more faces on an image. Second, the image detects around 468 face keypoints by using regression.

BlazeFace包含两个主要步骤。首先，模型检测图像中的一张或多张人脸。其次，通过回归在图像上检测约468个面部关键点。

Different from the dlib library, this model detects 3D coordinates. Those x and y coordinates are normalized from the image scale. The z coordinate is retrieved by taking the relative calculation between the screen and the model x coordinates. You can read more details here.

与dlib库不同，该模型检测的是3D坐标。其中x和y坐标根据图像尺寸进行了归一化。z坐标则通过屏幕与模型x坐标之间的相对计算得出。你可以在这里阅读更多细节。

Here is the flattened mesh from a face with their corresponding indexes:

以下是人脸展开网格及其对应的索引：

Implementing the face landmark detection

实现面部关键点检测

In general, the pipeline for implementing face landmark detection is the same as the dlib library. It starts from importing libraries, initializing objects, detect face and its landmarks, and done.

总的来说，实现面部关键点检测的流程与dlib库相同。从导入库、初始化对象、检测人脸及其关键点，到完成。

Here is the code for doing that:

以下是执行该操作的代码：

If you implement the code correctly, the image will display on your computer. Here is the preview of my result:

如果你正确实现了代码，图像将显示在你的电脑上。以下是我的结果预览：

The comparison

比较

We have already take a walkthrough of face landmark detection libraries using dlib and mediapipe. We can say that both libraries are easy to use. Therefore, we can build our solution rapidly.

我们已经分别介绍了使用dlib和mediapipe进行面部关键点检测的库。可以说这两个库都易于使用。因此，我们可以快速构建我们的解决方案。

However, there are differences between them. The dlib library needs C++ dependencies it. That’s why we need CMake and Visual Studio for installing the library.

然而，它们之间也存在差异。dlib库需要C++依赖项，这就是为什么我们需要CMake和Visual Studio来安装该库。

Also, this library needs a specific python library. Therefore, you have to create a virtual environment if you don’t have the supported Python version to run the library.

此外，该库需要特定版本的Python库。因此，如果你没有支持的Python版本来运行该库，就必须创建一个虚拟环境。

On the other side, installing mediapipe is easier. All you need to do is to install from pip only. Therefore, you don’t have to worry about installation more while using the mediapipe.

另一方面，安装mediapipe更加容易。你只需要通过pip安装即可。因此，使用mediapipe时你不需要担心安装问题。

In the case of the solution, the dlib can detect only the 2D coordinates of the keypoints. On the other hand, the mediapipe can detect the 3D coordinates of the keypoints. Therefore, you can use those keypoints from the mediapipe library for estimating the head pose.

在解决方案方面，dlib只能检测关键点的2D坐标。而mediapipe可以检测关键点的3D坐标。因此，你可以使用mediapipe库中的这些关键点来估计头部姿态。

Final Remarks

结语

Well done! Now you know how to implement face landmark detection using Python. I’ve shown you the libraries like dlib and mediapipe for implementing the solution.

做得很好！现在你知道了如何使用Python实现面部关键点检测。我已经向你展示了dlib和mediapipe这两个库来实现该解决方案。

I hope it helps you in implementing a computer vision solution. Also, I hope it can become your foundation to build more complex applications.

希望这能帮助你在实现计算机视觉解决方案方面有所帮助。同时，也希望这能成为你构建更复杂应用的基础。

If you are interested in my articles, you can follow me on Medium for more articles like this. Also, if you have any questions, you can contact me on LinkedIn.

如果你对我的文章感兴趣，可以在Medium上关注我，获取更多类似文章。此外，如果你有任何问题，可以在LinkedIn上联系我。

Thank you for reading my article!

感谢阅读我的文章！

References

参考文献

[1] https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

[1] https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

[2] https://www.analyticsvidhya.com/blog/2021/07/facial-landmark-detection-simplified-with-opencv/

[2] https://www.analyticsvidhya.com/blog/2021/07/facial-landmark-detection-simplified-with-opencv/