Explaining how I reached the top ranks of the new Data-Centric competition
解释我如何在新一届数据为中心（Data-Centric）竞赛中跻身前列
Explaining how I reached the top ranks of the new Data-Centric competition
解释我如何在新一届数据为中心（Data-Centric）竞赛中跻身前列
Following the endorsement of Andrew NG himself(!) regarding my last article, it felt natural to share all tips (with code!) of how I handled DeepLearning.ai's new challenge.
在Andrew Ng本人对我上一篇文章表示认可之后（！），我觉得有必要分享我是如何应对DeepLearning.ai新挑战的所有技巧（附代码！）。
The competition explained… again!
再次解释一下这个竞赛……！
If you are not familiar yet with the new Data-Centric challenge launched by DeepLearning.ai a few weeks ago, you might have a look at the article I wrote a few weeks ago to describe this challenge.
如果你还不熟悉DeepLearning.ai几周前发起的新一届数据为中心挑战赛，你可以看看我几周前写的那篇描述该挑战的文章。
Not sure this is worth it? Just follow Andrew Ng advice ? :
不确定值不值得看？那就听从Andrew Ng的建议吧？：
And if you are in a hurry, here is the long story short: the objective of the competition is to produce the best possible set of pictures to train a predefined model (ResNet50) to recognize roman numerals. The competition offers a “starting base” of approx. 3000 pictures, including noisy and mislabeled numbers, as you can observe below:
如果你赶时间，长话短说：本次竞赛的目标是生成尽可能优质的图片集，用于训练一个预定义模型（ResNet50）来识别罗马数字。竞赛提供了约3000张图片的"起始基础集"，其中包括带有噪声和错误标注的数字，如下所示：
Let’s the game begin!
游戏开始！
I am going to present the different steps to reach a good performance in a smooth way but this is obviously the result of many tests and trials I undertook to find the optimum combination!
我将以循序渐进的方式介绍达到良好性能的不同步骤，但这显然是我经过多次测试和尝试后找到的最优组合！
I have also created a dedicated repository on GitHub (link at the end of the article) if you want to explore my solution further.
如果你还想进一步探索我的解决方案，我还在GitHub上创建了一个专用代码库（文章末尾附链接）。
1. Pictures review
1.图片审查
The first task is probably the most demanding: reviewing each picture to check a few criteria. Here are the ones I had used:
第一个任务可能是最耗时的：审查每张图片并检查几个标准。以下是我使用的标准：
Does this look like a roman number? (If not, we should remove it!)
这看起来像罗马数字吗？（如果不是，我们应该移除它！）
Is the picture correctly labeled? (ex. “II” in “III” folder or vice versa)
图片的标签正确吗？（例如，"III"文件夹中的"II"，反之亦然）
What is the number quality? (rating from 1: good to 4:poor)
数字质量如何？（评分从1：良好到4：差）
What is the background quality? (same rating as above)
背景质量如何？（同上评分标准）
What is the font style? (“Arial” or “Roman”)
字体样式是什么？（"Arial"还是"Roman"）
What is the exact format of the number? (“viii” or “VIII”)
数字的确切格式是什么？（"viii"还是"VIII"）
Could we apply symmetries? (horizontal or vertical symmetries are usually suiting “I, II, III, or X” numbers but not for “i”, “ii”, or “VII”)
我们可以应用对称变换吗？（水平或垂直对称通常适用于"I、II、III或X"这类数字，但不适用于"i"、"ii"或"VII"）
Here are three examples of my evaluations (stored in a tabular way):
以下是我评估的三个示例（以表格形式存储）：
While reviewing the 3000 pictures (which took me approximately 2 or 3 hours ?), I sometimes had the feeling of “déjà vu” and I started to wonder whether some duplicates were hidden in the dataset? That would have not been very surprising so I had to take this also into consideration.
在审查3000张图片的过程中（花了我大约2到3个小时？），我有时会有"既视感"，开始怀疑数据集中是否隐藏着重复图片？这并不奇怪，所以我也必须考虑这一点。
I also designed a simple function to automatically check the content of the folders to evaluate the results of the different operations I would perform. Like all other functions I would use afterward in the notebook, I stored it in a dedicated “dcc_functions.py” (available on the GitHub repository).
我还设计了一个简单的函数来自动检查文件夹内容，以评估我将执行的不同操作的结果。就像我之后在notebook中使用的所有其他函数一样，我将其存储在一个专用的"dcc_functions.py"中（可在GitHub代码库中找到）。
Here is the output on the initial dataset:
以下是在初始数据集上的输出结果：
2. Dataset cleaning
2.数据集清洗
2.1 Noise removal
2.1噪声移除
I started by removing all the pictures that I had identified as pure noise or, at least, too noisy to train properly the model. This is obviously a personal choice and each participant has probably ended up with a different selection. I identified approx. 260 pictures to be removed (the corresponding list is stored in an Excel file on the GitHub repo).
我首先移除了所有被我识别为纯噪声或至少噪声太大而无法正常训练模型的图片。这显然是个人的选择，每位参与者最终可能都有不同的筛选结果。我识别出约260张需要移除的图片（对应的列表存储在GitHub代码库的Excel文件中）。
2.2 Duplicates removal
2.2 重复图片移除
As explained before, I had the feeling that some pictures were exactly the same but manually identifying them was impossible. There were a few technics I knew that could allow solving this issue:
如前所述，我有种感觉有些图片完全相同，但手动识别它们是不可能的。我知道有几种技术可以解决这个问题：
Pairing files with identical sizes… but a lot of false positives would arise
配对具有相同文件大小的图片……但会产生大量误报
Pairing files with identical sizes & configurations (like two “II” or “viii”)
配对具有相同大小和相同类别配置的图片（如两个"II"或两个"viii"）
Pairing files according to their statistics (using, for ex., PIL’s ImageStat)
根据图片的统计信息进行配对（例如，使用PIL的ImageStat）
Pairing files using Structure Similarity Index (some explanations here)
使用结构相似性指数进行配对（此处有一些解释）
Pairing files according to their “hash” number
根据图片的"哈希"值进行配对
As it was not a “life or death” matter, I decided to use the second solution which was both easy and quick to implement. The script (available here) identified approx 200 pairs of twin pictures, out of which 53 were actually genuine duplicates (some examples below):
由于这不是"生死攸关"的问题，我决定使用第二种方案，既简单又易于实现。该脚本（可在此处获取）识别出约200对孪生图片，其中53对是真正的重复图片（部分示例如下）：
2.3 Moving some pictures in the right folders
2.3 将图片移动到正确的文件夹
No need to spend a lot of time on that one: when a picture was mislabeled, I simply moved it back to the folder it belongs to.
这一点不需要花太多时间：当图片被错误标注时，我只需将其移回它所属的文件夹。
2.4 Edgy or not edgy?
2.4 边缘案例还是非边缘案例？
Before we go further, I’d like to share an interesting finding: I reviewed the pictures twice: when I entered the competition and, a second time, when I had a better idea of what to look for in the pictures.
在进一步讨论之前，我想分享一个有趣的发现：我审查了两次图片：第一次是在我参加竞赛时，第二次是在我更清楚应该在图片中寻找什么之后。
When reviewing the original pictures for the first time, I had excluded a lot of “edgy cases” that seemed too ambiguous to train the model.
当我第一次审查原始图片时，我排除了许多看起来过于模糊、不适合训练模型的"边缘案例"。
But a few weeks after the competition started, I started to get used to these edgy cases and consider them differently, like: “Well, it could be good to include this one to teach to the model that this case might happen.” I ended up adding approx. 80 pictures to the dataset.
但在竞赛开始几周后，我开始习惯这些边缘案例，并以不同的方式看待它们，比如："嗯，把这张加进来可能也不错，可以告诉模型这种情况可能会发生。"我最终向数据集中添加了约80张图片。
Counter-intuitively, the performance was decreasing with this new selection, including more edgy pictures. How come?
与直觉相反，包含更多边缘图片的新选择反而导致性能下降。这是怎么回事？
One of the participants, Mohamed Mohey, highlighted on the dedicated Discourse thread that the 32x32 transformation (applied to the dataset before the training) would sometimes completely denature the essence of the picture, as shown in the example below:
其中一位参与者Mohamed Mohey在专门的Discourse讨论帖中指出，32x32变换（在训练前应用于数据集）有时会完全改变图片的本质，如下例所示：
We can observe that, due to this 32x32 transformation, an obvious “III” is becoming a plausible “II”, explaining why some edgy cases would not necessarily bring valuable information to the model.
我们可以观察到，由于这种32x32变换，一个明显的"III"变成了看似"II"，这解释了为什么一些边缘案例不一定会为模型带来有价值的信息。
It would probably have been a good thing to review the pictures after a 32x32 transformation but I did not!
在32x32变换后重新审查图片可能是个好主意，但我没有这样做！
2.5 Using the “label book” pictures to train the model
2.5 使用"标签书"图片训练模型
The organizers from DeepLearning.ai had provided a set of 52 pictures, not existing in the “train” or “validation” folders, to evaluate our model’s performance when the ResNet50 training was over.
DeepLearning.ai的组织者提供了一组52张图片，这些图片不存在于"训练"或"验证"文件夹中，用于在ResNet50训练结束后评估我们模型的性能。
It was a good way to have a sense of how the model would be performing on the final and hidden dataset but I had also “guessed”, thanks to the scores displayed on the leaderboard, that the final evaluation on the hidden dataset was including 2420 pictures (see the corresponding notebook here). So 52 pictures were not very representative anyway!
这是了解模型在最终隐藏数据集上表现如何的好方法，但我也通过排行榜上显示的分数"猜测"出，最终隐藏数据集的评估包含2420张图片（参见相应的notebook）。所以52张图片无论如何都不太有代表性！
So I simply included these pictures in my training folder! The merrier, the funnier ?
所以我干脆把这些图片包含到我的训练文件夹中！越多越热闹嘛？
2.6 Evaluating the impact of the augmentation technics
2.6 评估数据增强技术的影响
As you might know, it is quite common to use augmentation technics on a dataset composed of pictures to help deep learning models identify the features that allow to properly infer the classes.
你可能知道，在由图片组成的数据集上使用数据增强技术是很常见的，这有助于深度学习模型识别能够正确推断类别的特征。
I decided to consider a few of them:
我决定考虑其中几种技术：
Horizontal and Vertical Symmetries
水平和垂直对称
Clockwise and Anti-clockwise rotations (10° and 20°)
顺时针和逆时针旋转（10°和20°）
Horizontal and Vertical Translations
水平和垂直平移
Cropping the white areas in the pictures
裁剪图片中的白色区域
Adding synthetic “salt and pepper” noise
添加合成"椒盐"噪声
Transfering noise of some pictures to some others
将某些图片的噪声迁移到其他图片
2.7 Implementing the customs functions
2.7 实现自定义函数
The first functions are quite simple and easily implemented with PIL, OpenCV, or even “packaged solutions” such as ImgAug. I thought it would be more interesting to share some tips regarding some of the custom functions I had designed ?
前几个函数相当简单，可以很容易地用PIL、OpenCV甚至"打包解决方案"（如ImgAug）实现。我觉得分享一些关于我设计的自定义函数的技巧会更有趣？
2.7.1 Squared Cropping Function
2.7.1 方形裁剪函数
The cropping operation is an interesting one! As the picture will, ultimately, be converted to a 32x32 picture, it might be better to zoom in on the area where the number is located.
裁剪操作是一个有趣的操作！由于图片最终会被转换为32x32图片，最好放大到数字所在的区域。
However, if the number does not have a “squared” shape, the result could be distorted when converted to 32x32 (as shown below). I redesigned the function so that the cropped output will always have a square shape and avoid this distortion effect:
然而，如果数字不具有"方形"形状，转换为32x32时结果可能会失真（如下所示）。我重新设计了该函数，使得裁剪输出始终具有方形形状，避免这种失真效果：
2.7.2 “Salt and Pepper” Function
2.7.2 "椒盐"噪声函数
As the background is probably not always plain white on the final evaluation dataset, I tried to augment pictures by adding a synthetic background.
由于最终评估数据集中的背景可能不总是纯白色的，我尝试通过添加合成背景来增强图片。
I used the “salt & pepper” function which is, basically, adding “0” and “1” randomly into the NumPy arrays describing the pictures:
我使用了"椒盐"噪声函数，它基本上是在描述图片的NumPy数组中随机添加"0"和"1"：
2.7.3 Background Noise Transfer Function
2.7.3 背景噪声迁移函数
I was not fully happy with the results of the “Salt and Pepper” function as the noise was always homogeneous so I imagined another way to add noise to the pictures.
我对"椒盐"噪声函数的结果并不完全满意，因为噪声总是均匀的，所以我想到了另一种为图片添加噪声的方法。
I recycled some of the pictures that I had originally considered as unreadable and made them become some “noisy background” basis. There were also some pictures with a “heavy background” for which I removed the number (as shown below) to get more samples.
我回收了一些最初被我视为不可读的图片，将它们变成"噪声背景"库。还有一些具有"厚重背景"的图片，我移除了其中的数字（如下所示）以获得更多样本。
It provided me a “noisy backgrounds bank” of 10 pictures that I added randomly to some pictures after applying horizontal or vertical symmetries:
这为我提供了一个包含10张图片的"噪声背景库"，我可以在对某些图片应用水平或垂直对称后随机添加这些背景：
2.8 Choosing the best augmentations
2.8 选择最佳增强策略
As the number of pictures allowed could not exceed 10.000 elements, I had to know which transformations were providing the highest impact.
由于图片数量不能超过10，000张，我需要知道哪些变换产生的影响最大。
I decided to benchmark them by comparing a baseline (a cleaned dataset with no transformation) and the individual performance of each of the augmentation technics (summary below):
我决定通过比较基线（未经任何变换的清洗后数据集）与每种数据增强技术的单独表现来进行基准测试（总结如下）：
We can observe that the rotations, translations, and cropping were bringing a significant impact compared to others so I decided to focus on that ones.
我们可以观察到旋转、平移和裁剪相比其他技术带来了显著影响，所以我决定重点关注这些。
And “voilà”!
"好了"！
As the process is stochastic (transformations are applied with a 50% probability and some random parameters), each iteration of the script will produce a unique combination of pictures.
由于该过程是随机的（变换以50%的概率和一些随机参数应用），脚本的每次迭代都会产生独特的图片组合。
Many of my tests produced a performance of around 84% while the highest competitor had reached 86% (with a 64% baseline). Honorable I guess ?
我的许多测试产生了约84%的性能，而最高分的竞争对手达到了86%（基线为64%）。还算体面吧？
There would have been some additional tweaks to consider (like creating my own pictures and adding them to the dataset but I choose to only rely on the initial pictures provided). Some others probably gave it a try!
本来还有一些额外的调整可以考虑（比如创建自己的图片并添加到数据集中，但我选择只依赖提供的初始图片）。其他人可能尝试过了！
A global overview of the competitors’ performance
参赛者表现的整体概览
It is probably also worth mentioning that I have analyzed the performance of competitors during the first weeks of the challenge (until 26/08) and we can see how quickly most of the participants reached an acceptable performance, converging quickly towards 75% and above:
可能也值得一提的是，我分析了竞赛前几周（截至8月26日）参赛者的表现，我们可以看到大多数参与者多快达到了可接受的性能，迅速收敛到75%及以上：
Final words
结语
As mentioned earlier, after a proper review/cleaning of the data and a script executed in less than 30 seconds, you can easily outperform what state-of-the-art models could produce with noisy data!
如前所述，经过适当的数据审查/清洗以及一个运行时间不到30秒的脚本，你可以轻松超越最先进的模型在噪声数据上的表现！
I really enjoyed participating in this challenge which, according to me, was more demanding of “fresh ideas” than “GPU power” and I am really looking forward to the next one!
我非常享受参加这次竞赛，在我看来，它更需要的是"新鲜想法"而非"GPU算力"，我非常期待下一届！
We had a lot of fun and rich interactions with other competitors and Lynn (from DeepLearning.ai) to share our views on this “first-of-its-kind” contest. Many participants were more seeking to share their views and findings along the way rather than being at the top of the leaderboard.
我们与其他参赛者以及Lynn（来自DeepLearning.ai）之间进行了很多有趣且富有成果的互动，分享我们对这场"首创性"竞赛的看法。许多参赛者更倾向于一路分享他们的观点和发现，而不是仅仅追求排行榜上的排名。
I also know how difficult generating “average and noisy data” can be… so congratulations to the organizers for delivering such good material to work on!
我也知道生成"平均且带噪声的数据"是多么困难……所以祝贺组织者提供了如此优质的素材供我们使用！
And, of course, I hope you liked the second part of this Deep-Dive on the Data-Centric Challenge from DeepLearning.ai!
当然，希望你喜欢这篇对DeepLearning.ai数据为中心挑战赛深入剖析的第二部分！
As promised, here is the link to the GitHub repository and feel free to share your experience and/or findings in the comments:
如约，这是GitHub代码库的链接，欢迎在评论中分享你的经验和/或发现：
