6 Common Metrics For Your Next Regression Project

下一次回归项目的6个常用指标

6 Common Metrics For Your Next Regression Project

下一次回归项目的6个常用指标

Advantages, Disadvantages & Major Pitfalls

优势、劣势与主要陷阱

So you trained a model, now what? Or, you’ve trained multiple models; how do you decide which one is best? Ok, let’s ask Google. Hmm, Google suggests many metrics that can be used to evaluate your model(s). But now, this becomes a meta-problem of what metric should I use to determine what model to use? So, here’s a list of some common metrics along with their advantages, disadvantages & nuances to start with:

你训练好了一个模型，然后呢？或者，你训练了多个模型，怎么决定哪个最好？好吧，问问谷歌。嗯，谷歌推荐了许多可以用来评估模型的指标。但现在这变成了一个元问题：我应该用什么指标来决定使用哪个模型？所以，这里列出了一些常用指标，以及它们的优势、劣势和细微差别，供你参考：

Mean absolute error (MAE)

平均绝对误差（MAE）

Mean squared error (MSE)

均方误差（MSE）

Root mean squared error (RMSE)

均方根误差（RMSE）

Normalized root mean squared error (NRMSE)

归一化均方根误差（NRMSE）

Root mean squared log error (RMSLE)

均方根对数误差（RMSLE）

Mean absolute percentage error (MAPE)

平均绝对百分比误差（MAPE）

For the following sections, y is the true value, y-hat is the predicted value, n is the number of test instances, and i goes from 1 to n. Also, all metrics are evaluated on an unseen test set.

在以下章节中，y 表示真实值，y-hat 表示预测值，n 表示测试样本数量，i 从 1 到 n。另外，所有指标均在未见过的测试集上评估。

Mean Absolute Error

平均绝对误差

Mean absolute error is a very intuitive and, therefore, popular metric. It is simply the average distance between predicted and true values. To avoid errors canceling one another out, we take the absolute of every error that we compute. The best model is usually the one with the lowest MAE. However, there are a few features to consider when picking MAE as your metric.

平均绝对误差是一个非常直观的指标，因此也很流行。它简单来说是预测值与真实值之间的平均距离。为了避免正负误差相互抵消，我们计算每个误差的绝对值。通常最好的模型是 MAE 最低的那个。不过，在选择 MAE 作为评估指标时，有几个特点需要注意。

Although it is straightforward to interpret, MAE has a few disadvantages. For example, it doesn’t tell you whether your model tends to over-estimate or under-estimate since any direction information is destroyed by taking the absolute value. Also, the metric can be insensitive to large outliers. Take a look at the example below.

尽管 MAE 易于解释，但它也有一些缺点。例如，它不能告诉你模型是倾向于高估还是低估，因为绝对值破坏了所有方向信息。另外，该指标对大的离群值可能不敏感。请看下面的例子。

On the left, the model is off by a bit here and there. However, on the right, the model misses the marks at the tail end by a much wider margin while being perfect at the beginning and in the middle. Yet, the MAEs for both are the same. If you decide to use this metric, it’s a good idea to plot the errors to see any outliers like case 2. All in all, if you want a metric that penalizes large errors, you’ll have better luck somewhere else.

左边这个模型在个别点上有些偏差。而右边这个模型在开头和中间都完美拟合，但在尾部却偏离很大。然而，两者的 MAE 却相同。如果你决定使用这个指标，最好画出误差图来检查是否存在类似案例2那样的离群点。总而言之，如果你需要一个能惩罚大误差的指标，那 MAE 可能不是最佳选择。

Mean Squared Error

均方误差

This brings me to mean squared errors. Like MAE, we’re destroying directional information when we square every computed error. The MSE is also always larger than or equal to 0. However, we are now able to distinguish between the two models above.

这就引出了均方误差。与 MAE 类似，当我们对每个计算出的误差进行平方时，也破坏了方向信息。MSE 也总是大于等于 0。但现在我们能够区分上面两个模型了。

Interestingly, MSE is related to the infamous bias-variance trade-off. It can be shown that the expected test MSE for a given test point can be written as [1]:

有趣的是，MSE 与著名的偏差-方差权衡有关。可以证明，对于给定的测试点，期望测试 MSE 可以写成 [1]：

where the 0 subscript is the index of a test data point, and ϵ is the irreducible or noise in the data.

其中下标 0 是测试数据点的索引，ϵ 是数据中不可约的误差或噪声。

Variance refers to the amount by which y-hat changes when we change the training set. Usually, more flexible methods have a higher degree of variance. Variance is also dependent on how much data we have. The larger the training dataset, the lower the variance. Therefore, we can interpret the MSE for a given test data point in terms of the model’s bias and random noise in the limit of huge data.

方差指的是当我们改变训练集时，y-hat 变化的幅度。通常，更灵活的方法具有更高的方差。方差也取决于我们拥有的数据量。训练集越大，方差越低。因此，在大数据极限下，我们可以将给定测试数据点的 MSE 解释为模型偏差和随机噪声的组合。

Bias occurs when we try to estimate a complicated relationship between predictors and targets with something simpler. For example, we often assume that x and y have linear or polynomial relationships because we know the form of these equations, which reduces the problem to a few parameters we can estimate. In reality, x and y might not have such a relationship.

当我们试图用更简单的模型来估计预测变量与目标之间复杂的关系时，就会产生偏差。例如，我们经常假设 x 和 y 具有线性或多项式关系，因为我们知道这些方程的形式，从而将问题简化为几个可估计的参数。但在现实中，x 和 y 可能并不存在这样的关系。

Finally, one major disadvantage to MSE is that the units of y are squared, which means it’s easy to misinterpret the final results.

最后，MSE 的一个主要缺点是 y 的单位被平方了，这意味着最终结果容易被误解。

Roor Mean Squared Error

均方根误差

How about we take the square root? RMSE and MSE are very similar, except that RMSE is more convenient because it has the same unit as whatever y has.

那取平方根呢？RMSE 和 MSE 非常相似，只是 RMSE 更方便，因为它具有与 y 相同的单位。

However, have you ever tried to transform your targets to see if you get a better fit? Like taking the log of y, for example. Then you compute the RMSE of both approaches and see that one is higher than the other. This is not a fair comparison because the two values have different units. A way to get around this is to divide by some property of y to get a unit-less metric called normalized root mean squared error.

但是，你是否尝试过对目标变量做变换以获得更好的拟合效果？比如对 y 取对数。然后你计算两种方法的 RMSE，发现一个高于另一个。这种比较并不公平，因为两个值具有不同的单位。解决这个问题的方法是用 y 的某个属性来除，得到一个无单位的指标，称为归一化均方根误差。

Normalized Root Mean Squared Error

归一化均方根误差

I don’t have any equations here because there are many things you can divide RMS with. Some common choices are the mean of y, the difference between the max and min of y, standard deviation, and interquartile range. When to choose what is a subtle business, and [2] can provide a much more in-depth explanation if you’re interested.

这里我没有列出公式，因为可以用来除以 RMSE 的东西有很多。常见的选择有 y 的均值、最大值与最小值之差、标准差和四分位距。何时选择哪个是一个微妙的问题，如果你感兴趣，[2] 提供了更深入的解释。

Root Mean Squared Log Error

均方根对数误差

I think this metric was introduced by Kaggle a few years back [3]. This metric should be used when you want to add direction to how errors are penalized. In this case, we’re telling the metric to penalize underestimation more than overestimation. For example, suppose y=1. If our model gives y-hat = 0, then the error is [log(1/2)]² = 0.09. However, if the model gives y-hat=2, then the error is [log(3/2)]² = 0.03. If we used MSE, then our error would be 1 either way.

我认为这个指标是几年前由 Kaggle 引入的 [3]。当你希望对误差的惩罚增加方向性时，应该使用这个指标。在这种情况下，我们告诉指标对低估的惩罚要大于高估。例如，假设 y=1。如果我们的模型预测 y-hat=0，则误差为 [log(1/2)]² = 0.09。但如果模型预测 y-hat=2，则误差为 [log(3/2)]² = 0.03。如果我们使用 MSE，那么无论哪种情况误差都是 1。

Also, the metric takes into the relative scale of the true and predicted values. For example, if y=9 and y-hat=99, then the error is 1. If, on the other hand, y=99 and y-hat=999, then the error is still 1. So, this metric is handy when the range of your target values is high, and you don’t want to penalize large errors when both the predicted and true values are large.

此外，该指标考虑到了真实值和预测值的相对尺度。例如，如果 y=9 且 y-hat=99，则误差为 1。另一方面，如果 y=99 且 y-hat=999，误差仍然是 1。因此，当目标值范围很大，并且你不想在预测值和真实值都很大时惩罚大误差，这个指标就很方便。

However, I haven’t been able to find a discussion on a few pitfalls to keep in mind here. If you look at the log(x) graph, you’ll see that x cannot be equal to or smaller than 0. This means that the argument inside our logarithm (y-hat+1 / y+1) can’t be equal to or smaller than 0. Then, there’s the added constraint that y can’t be -1. Otherwise, the whole thing blows up.

然而，我还没能找到关于此指标需要注意的陷阱的讨论。如果你看 log(x) 的图形，你会发现 x 不能等于或小于 0。这意味着我们对数内部的参数 (y-hat+1 / y+1) 不能等于或小于 0。另外还有一个约束，y 不能等于 -1，否则整个式子会发散。

Mean Absolute Percentage Error

平均绝对百分比误差

MAPE looks like a cousin of MAE, with the added benefit that it’s unitless. However, introducing division also comes with some major drawbacks. For example, with very small y_i or y_i = 0, MAPE can blow up or not be computable at all. It also inherits a problem from MAE, whether there isn’t an upper bound for how large each error can be (even though the name says ‘percentage’).

MAPE 看起来像是 MAE 的表亲，额外的好处是无单位。然而，引入除法也带来了一些主要缺点。例如，当 y_i 非常小或 y_i=0 时，MAPE 会变得很大甚至无法计算。它也继承了 MAE 的一个问题，即每个误差没有上限（尽管名称里有“百分比”）。

Wrap-Up

总结

I hope you liked the list and learned something new. What’s next is just my opinion. I often see the mention of ‘interpretability’ when I read up on these metrics. I’m not exactly sure what ‘interpretability’ means or whether there’s even a consensus on it. My explanations of and how I understand these metrics are often based on how they behave in certain situations. All in all, how much should we emphasize the ‘interpretability’ of a metric when there are other metrics that behave more like the way we want but is harder to interpret?

希望你喜欢这个列表并学到了一些新知识。接下来只是我个人的看法。我在阅读这些指标时经常看到“可解释性”这个词。我不太确定“可解释性”到底是什么意思，或者是否有一个共识。我对于这些指标的解释和理解，往往是基于它们在特定情况下的表现。总而言之，当有其他指标更符合我们的需求但却更难解释时，我们究竟应该在多大程度上强调指标的“可解释性”呢？

Stay Connected

保持联系

I like to write about data science & science. If you like this post, follow me on Medium and/or join my email list. See you next post! ?

我喜欢写数据科学和科学相关的内容。如果你喜欢这篇文章，请在 Medium 上关注我或加入我的邮件列表。下一篇见！？

Sources

资料来源

[1] Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani. Introduction to Statistical Learning: With Applications in R (2013)

[1] Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani. Introduction to Statistical Learning: With Applications in R (2013)

[2] Otto, S.A. (2019, Jan.,7). How to normalize the RMSE [Blog post]. https://www.marinedatascience.co/blog/2019/01/07/normalizing-the-rmse/

[2] Otto, S.A. (2019, Jan.,7). How to normalize the RMSE [Blog post]. https://www.marinedatascience.co/blog/2019/01/07/normalizing-the-rmse/

[3] https://www.kaggle.com/carlolepelaars/understanding-the-metric-rmsle

[3] https://www.kaggle.com/carlolepelaars/understanding-the-metric-rmsle

[4] https://en.wikipedia.org/wiki/Mean_absolute_percentage_error

[4] https://en.wikipedia.org/wiki/Mean_absolute_percentage_error