西安电子科技大学教学平台-试译宝

本校课程

英语基础训练

CATTI笔译备考课程

试译宝点评功能体验

好课推荐 前往试译宝，备考CATTI MTI

智能翻译技术2026全年班

校企合作

西安电子科技大学—学生翻译实践

以下内容为西安电子科技大学学生最近一个月内的翻译实践成果

Efficient Data Valuation with Exact Shapley Values

Efficient Data Valuation with Exact Shapley Values Produce Better Models with Less Data. This post is an overview of the work done in the paper — ‘Efficient task-specific data valuation for nearest neighbour algorithms’. Does more data always produce better results? Given the trends in machine learning, you would expect as much. More data seems king given the recent rise in larger and larger machine learning models with exceptional performance. This pattern is exemplified in advances in Machine learning like OpenAIs GPT-3 model, trained on over 45 terabytes of data from books and the internet to support its 175 billion parameters. But this model is effectively dwarfed by Google’s new switch Transformer architecture with 1.6 trillion parameters. So it appears that bigger and bigger models with more and more data are inevitable. If data is so valuable, should we aim to gather more? Why shouldn’t we have more of what is so valuable? There is a problem with this approach. What if the additional data you gather is irrelevant to your problem. What if it is purely noise? Then any additional data will actively hurt the performance of your model. So what can you do to mitigate this problem? Well, for one, you can painstakingly look through your data. This approach is probably one of the most effective tools but is highly time-consuming when you have a lot of data. For computer vision tasks, there are a lot of different techniques that you can use. This post shows some exciting techniques for precisely that. For many cases, the applicable data is relatively easy to identify. However, for others, it is an unclear task at best. Often part of the problem is that the relationships between the data and the outcome are what we’re trying to model. Then identifying what data is most applicable is an arduous task. Fortunately, there is a new method that is perfect for this scenario based on Shapley values. SHAP Values SHapley Additive exPlanations (SHAP) is a game-theoretic approach to explain the output of any machine learning model. This method is fairly well known, but the attribution is based solely on the features. The method provides attribution values from features to the output of the model. SHAP values are based on the Shapley values, which determine how to distribute payment among multiple players in a game fairly. For SHAP values, the coalitions of players are based on the features. SHAP values are incredibly flexible. For example, in computer vision tasks, SHAP values represent the attribution of different pixels to the model’s output. There are many different methods to calculate SHAP values, including a KernelSHAP method which is model-independent. For each variation of SHAP values, the attributions are always related to the features of the models. However, there is an alternative. Use the instances as the players and calculate attribution for each instance. Data Valuation ‘Efficient task-specific data valuation for nearest neighbour algorithms’ is a recent paper providing novel algorithms to calculate exact Shapley values. For the rest of this post, I refer to the Shapley values produced for each instance as Data Shapley Values. Data Shapley Values are a recent innovation that utilizes Shapley values to determine the attribution of different data instances. The motivation of the research was inspired by optimizing records selected from a data market for privacy-preserving machine learning. The data market consists of many different medical records. Therefore, the data buyer chooses a subset of records to purchase from the data market. Because of the data cost, the buyer aim’s to select an optimal subset of patients for their model. The Shapley values in this problem configuration measure the marginal improvements of the utility attributed to each data point average overall possible data subsets. The most significant issue for computing Shapley values is the high degree of complexity. Generally, this is on the level of O(2^N) for exact calculations. However, the researchers have developed a novel algorithm for exact Shapley values designed for K-NN classifiers using a KNN utility. This algorithm relies on the fact that the KNN utility satisfies a piecewise utility difference property. I’ll leave the exact mathematical formulation for the paper and the curious readers. But here is the result. The algorithm performs exact computation with O(N log N) complexity. Experiments The structure of the experiments follows a simple format. First, the data is separated into training, validation, and test sets. Then, the exact Shapley values are calculated based on the attribution of the training instances to the validation instances. Then the training instances are ordered according to their Shapley values. This setup provides the user with the attributions of each instance to the validation set. Next, the user can select only those instances with the highest Shapley value, providing the user with a smaller subset of data. As this process selects data that contributes the most to the performance on the validation set, removing instances will low Shapley values often removes the noisiest instances within the data. And, at the same time, maintaining the most representative examples. The experiments use the diamonds dataset. This dataset contains almost 54,000 diamonds. The features include diamond attributes such as the carat of the diamond, the cut, the colour, and several other features. Some of the features are categorical, and for these experiments, I transform these features into boolean features for each category. The target is the price of the diamonds. I’ve taken 5000 instances for validation and testing. The remainder of the data is used for training. The aim is to produce a model to predict the price of diamonds using less data. I’ve evaluated the performance with a single decision tree regressor. The data is ordered based on the Shapley value, and the performance is measured with an R² score. Next, the Shapley values are calculated for the validation. The chunk of code also sequentially fits a model on smaller amounts of data. The data is considering according to decreasing Shapley Values. Once the model is trained on the subset of training data, the model is evaluated on the test dataset. The experiment results show that with less data, the model performs better on the test data. The peak of the test performance occurs when almost 50% of the training data is removed. What is also intriguing is that the model performance on the validation set peaks when over 60% of the training data is removed. Since the instances are removed based on the Shapley values on the validation set, this pattern makes some sense. Conclusion Despite the increasing availability of a massive amount of data, the quality of the data is crucial. Building your models on low-quality data produces low-quality models. Utilizing Shapley values on instances offer an alternative approach. You can create your models with fewer instances and get improved performance. The experiments show that less data can improve the model performance even with a dramatically smaller subset of data. Data is the new oil, but the quality of that oil matters. Consider using data Shapley values in your next machine learning model.

2026-08-12 李宇轩人工智能英-中

Explaining how I reached the top ranks of the new Data-Centric competition

Explaining how I reached the top ranks of the new Data-Centric competition Following the endorsement of Andrew NG himself(!) regarding my last article, it felt natural to share all tips (with code!) of how I handled DeepLearning.ai's new challenge. The competition explained… again! If you are not familiar yet with the new Data-Centric challenge launched by DeepLearning.ai a few weeks ago, you might have a look at the article I wrote a few weeks ago to describe this challenge. Not sure this is worth it? Just follow Andrew Ng advice ? : And if you are in a hurry, here is the long story short: the objective of the competition is to produce the best possible set of pictures to train a predefined model (ResNet50) to recognize roman numerals. The competition offers a “starting base” of approx. 3000 pictures, including noisy and mislabeled numbers, as you can observe below: Let’s the game begin! I am going to present the different steps to reach a good performance in a smooth way but this is obviously the result of many tests and trials I undertook to find the optimum combination! I have also created a dedicated repository on GitHub (link at the end of the article) if you want to explore my solution further. 1. Pictures review The first task is probably the most demanding: reviewing each picture to check a few criteria. Here are the ones I had used: Does this look like a roman number? (If not, we should remove it!) Is the picture correctly labeled? (ex. “II” in “III” folder or vice versa) What is the number quality? (rating from 1: good to 4:poor) What is the background quality? (same rating as above) What is the font style? (“Arial” or “Roman”) What is the exact format of the number? (“viii” or “VIII”) Could we apply symmetries? (horizontal or vertical symmetries are usually suiting “I, II, III, or X” numbers but not for “i”, “ii”, or “VII”) Here are three examples of my evaluations (stored in a tabular way): While reviewing the 3000 pictures (which took me approximately 2 or 3 hours ?), I sometimes had the feeling of “déjà vu” and I started to wonder whether some duplicates were hidden in the dataset? That would have not been very surprising so I had to take this also into consideration. I also designed a simple function to automatically check the content of the folders to evaluate the results of the different operations I would perform. Like all other functions I would use afterward in the notebook, I stored it in a dedicated “dcc_functions.py” (available on the GitHub repository). Here is the output on the initial dataset: 2. Dataset cleaning 2.1 Noise removal I started by removing all the pictures that I had identified as pure noise or, at least, too noisy to train properly the model. This is obviously a personal choice and each participant has probably ended up with a different selection. I identified approx. 260 pictures to be removed (the corresponding list is stored in an Excel file on the GitHub repo). 2.2 Duplicates removal As explained before, I had the feeling that some pictures were exactly the same but manually identifying them was impossible. There were a few technics I knew that could allow solving this issue: Pairing files with identical sizes… but a lot of false positives would arise Pairing files with identical sizes & configurations (like two “II” or “viii”) Pairing files according to their statistics (using, for ex., PIL’s ImageStat) Pairing files using Structure Similarity Index (some explanations here) Pairing files according to their “hash” number As it was not a “life or death” matter, I decided to use the second solution which was both easy and quick to implement. The script (available here) identified approx 200 pairs of twin pictures, out of which 53 were actually genuine duplicates (some examples below): 2.3 Moving some pictures in the right folders No need to spend a lot of time on that one: when a picture was mislabeled, I simply moved it back to the folder it belongs to. 2.4 Edgy or not edgy? Before we go further, I’d like to share an interesting finding: I reviewed the pictures twice: when I entered the competition and, a second time, when I had a better idea of what to look for in the pictures. When reviewing the original pictures for the first time, I had excluded a lot of “edgy cases” that seemed too ambiguous to train the model. But a few weeks after the competition started, I started to get used to these edgy cases and consider them differently, like: “Well, it could be good to include this one to teach to the model that this case might happen.” I ended up adding approx. 80 pictures to the dataset. Counter-intuitively, the performance was decreasing with this new selection, including more edgy pictures. How come? One of the participants, Mohamed Mohey, highlighted on the dedicated Discourse thread that the 32x32 transformation (applied to the dataset before the training) would sometimes completely denature the essence of the picture, as shown in the example below: We can observe that, due to this 32x32 transformation, an obvious “III” is becoming a plausible “II”, explaining why some edgy cases would not necessarily bring valuable information to the model. It would probably have been a good thing to review the pictures after a 32x32 transformation but I did not! 2.5 Using the “label book” pictures to train the model The organizers from DeepLearning.ai had provided a set of 52 pictures, not existing in the “train” or “validation” folders, to evaluate our model’s performance when the ResNet50 training was over. It was a good way to have a sense of how the model would be performing on the final and hidden dataset but I had also “guessed”, thanks to the scores displayed on the leaderboard, that the final evaluation on the hidden dataset was including 2420 pictures (see the corresponding notebook here). So 52 pictures were not very representative anyway! So I simply included these pictures in my training folder! The merrier, the funnier ? 2.6 Evaluating the impact of the augmentation technics As you might know, it is quite common to use augmentation technics on a dataset composed of pictures to help deep learning models identify the features that allow to properly infer the classes. I decided to consider a few of them: Horizontal and Vertical Symmetries Clockwise and Anti-clockwise rotations (10° and 20°) Horizontal and Vertical Translations Cropping the white areas in the pictures Adding synthetic “salt and pepper” noise Transfering noise of some pictures to some others 2.7 Implementing the customs functions The first functions are quite simple and easily implemented with PIL, OpenCV, or even “packaged solutions” such as ImgAug. I thought it would be more interesting to share some tips regarding some of the custom functions I had designed ? 2.7.1 Squared Cropping Function The cropping operation is an interesting one! As the picture will, ultimately, be converted to a 32x32 picture, it might be better to zoom in on the area where the number is located. However, if the number does not have a “squared” shape, the result could be distorted when converted to 32x32 (as shown below). I redesigned the function so that the cropped output will always have a square shape and avoid this distortion effect: 2.7.2 “Salt and Pepper” Function As the background is probably not always plain white on the final evaluation dataset, I tried to augment pictures by adding a synthetic background. I used the “salt & pepper” function which is, basically, adding “0” and “1” randomly into the NumPy arrays describing the pictures: 2.7.3 Background Noise Transfer Function I was not fully happy with the results of the “Salt and Pepper” function as the noise was always homogeneous so I imagined another way to add noise to the pictures. I recycled some of the pictures that I had originally considered as unreadable and made them become some “noisy background” basis. There were also some pictures with a “heavy background” for which I removed the number (as shown below) to get more samples. It provided me a “noisy backgrounds bank” of 10 pictures that I added randomly to some pictures after applying horizontal or vertical symmetries: 2.8 Choosing the best augmentations As the number of pictures allowed could not exceed 10.000 elements, I had to know which transformations were providing the highest impact. I decided to benchmark them by comparing a baseline (a cleaned dataset with no transformation) and the individual performance of each of the augmentation technics (summary below): We can observe that the rotations, translations, and cropping were bringing a significant impact compared to others so I decided to focus on that ones. And “voilà”! As the process is stochastic (transformations are applied with a 50% probability and some random parameters), each iteration of the script will produce a unique combination of pictures. Many of my tests produced a performance of around 84% while the highest competitor had reached 86% (with a 64% baseline). Honorable I guess ? There would have been some additional tweaks to consider (like creating my own pictures and adding them to the dataset but I choose to only rely on the initial pictures provided). Some others probably gave it a try! A global overview of the competitors’ performance It is probably also worth mentioning that I have analyzed the performance of competitors during the first weeks of the challenge (until 26/08) and we can see how quickly most of the participants reached an acceptable performance, converging quickly towards 75% and above: Final words As mentioned earlier, after a proper review/cleaning of the data and a script executed in less than 30 seconds, you can easily outperform what state-of-the-art models could produce with noisy data! I really enjoyed participating in this challenge which, according to me, was more demanding of “fresh ideas” than “GPU power” and I am really looking forward to the next one! We had a lot of fun and rich interactions with other competitors and Lynn (from DeepLearning.ai) to share our views on this “first-of-its-kind” contest. Many participants were more seeking to share their views and findings along the way rather than being at the top of the leaderboard. I also know how difficult generating “average and noisy data” can be… so congratulations to the organizers for delivering such good material to work on! And, of course, I hope you liked the second part of this Deep-Dive on the Data-Centric Challenge from DeepLearning.ai! As promised, here is the link to the GitHub repository and feel free to share your experience and/or findings in the comments:

2026-08-11 李宇轩人工智能英-中

Information bottlenecks

Notes on speech processing, 2.9.2021 Information bottlenecks and dimensionality reduction in deep learning Autoencoders and other deep neural networks with information bottlenecks have become fashionable. The heuristic idea is that the dimensionality of the hidden layers is reduced such that the network is forced to focus on the important part of the data. Experiments have also demonstrated that autoencoders are efficient in this sense. I have however been left wondering whether the amount of information can be characterized in exact terms. How much information flows through the bottleneck? How would we even measure that? This short note is my attempt at characterizing and understanding the problem. I will start with some classical concepts of information theory and linear algebra and then discuss the extent to which such concepts are applicable in machine learning. A central result is that dimensionality of a hidden layer cannot alone be used as a measure of information content. Information content in discrete representations If a system has two states, A and B, then obviously we can represent the state by one bit. Four states can be represented by 2 bits, 8 states by 3 bits and in general, N states by log2(N) bits. We can therefore always easily determine the number of bits required for systems with a finite number of states. The amount of bits needed to describe the state is then a direct measure of the information content or entropy of the system. We can expand this to countable sets, such as integers, if we in additional have the access to probability of each state. Then we can make statements about average bitrate, that is, if we observe the system many times, how many bits do we on average need for representing the state? If the probability of state k is Pk, then the amount of bits needed to represent that state is log2 Pk. That bitrate, log2 Pk occurs at probability Pk, such that the average bitrate can be calculated as the sum, sum Pk log2 Pk, where the summation goes over all k. This applies also when k goes over an infinite but countable set. Information content in linear, continuous valued systems If the title is confusing, just think of linear algebra. How much information is there in a vector x of length N. Well, it is not really defined. What we do however know is that if we multiply it with a matrix A, as y=Ax, then if the matrix A is full rank, then all information is retained. In fact, then we can recover x from y by the inverse x=inv(A)y. No information is lost. Clearly the rank of A thus defines its capacity remove information. If rank(A)<N then information is lost and cannot be recovered from y. This is not yet the whole story though. In practical implementations of the inverse, we know that it is not only the rank which is important, but also the conditioning of A. If any of the singular values of A are close to zero, then A becomes ill-conditioned such that the recovery of x from y becomes numerically difficult. In the best case, we loose accuracy, such that x can be recovered only approximately, in severe cases information can be entirely lost. The information content is thus not only described by dimensionality, but also characterized by accuracy. As we shall see, I argue that it is more useful to characterize loss of information as a loss of accuracy rather than loss of dimensions. Diversion: Space filling curves If you have not heard about space-filling curves, start by watching the Numberphile video about them. The idea is an infinite recursion; you start with a simple shape which goes through a space. Then you add wiggles to that shape so that it spreads more over the space. Repeatedly adding more wiggles makes the curve spread out more and more, such that it converges to covering the whole space. The one-dimensional line thus covers the whole two-dimensional space (i.e. its Hausdorff dimension is 2). In terms of information content, now, the one-dimensional curve contains the information of the two-dimensional space. If we start with some particular point in 2D-space (x,y), we can convert that to a point d on the one-dimensional line, and then convert it back to the 2D-point (x,y). It is just that there is an infinite recursion involved, so this is not a practical algorithm. We can however, implement a finite number of recursions to get an approximation. In the example below, I have implemented an Hilbert-curve and plotted the curve for different number of recursions N. We can readily see that for each iteration, the accuracy with which the curve fills space is doubled (error is halved i.e. error energy is 1/4th). By accuracy I refer to the average distance from a random point in 2D space to the closest point on the curve. Each iteration, on the other hand, splits every segment into 4 sub-segments, at a cost of 2 bits. Halving the error thus comes at a cost of 2 bits. This results thus follows results of conventional lossy coding; halving error costs as many bits as we have dimensions. Now we have 2 dimensions so halving error costs 2 bits. Information content in autoencoders Observe that the above space-filling curve construction can be interpreted as an autoencoder. The 2-dimensional space is mapped (encoder) to a 1 dimensional space (bottleneck), which we can recover with the inverse (decoder). The curve is piecewise linear and could easily be implemented with a single layer of rectified linear units (RELUs). Each recursion consists of a subdivision into 4 parts, such that we can expect that the network can be implemented with 2^(2N) RELUs. Conversely, the error of the mapping is halved if the number of RELUs is quadrupled. A red herring One could easily be fooled to think that we can do some simpler space filling curve than Hilbert (or other equivalent curves). For example, we could draw zig-zag lines going end-to-end on dimension x and then takes a step 1/N on dimension y. This can be implemented with O(N) RELUs. The accuracy of this map would then be relative to 2^-N instead of 2^-(2N). However, we would then have error only on the y dimension and the x dimension could be always perfectly reconstructed. Our accuracy argument thus applies as before, we need 1 bit for each dimension to halve accuracy, when assuming that accuracy on each axis is equal. Reconstruction accuracy as a measure of information The pertinent consequence for autoencoders is that the dimensionality of the bottleneck does not alone define the amount of information that passes through. By exponentially increasing the number of non-linearities in the encoder and decoder, we gain a log-linear decrease in mean square error. Since we thus cannot measure information with the number of dimensions, we should therefore rather measure the amount of information in terms of reconstruction accuracy. This approach is in line also with conventional concepts in probability and statistics. For continuous valued variables x, we cannot define a probability, but only probability distributions, since there are an infinite number of possible values and any particular value would always have probability zero. In a similar fashion, for continuous-valued information bottlenecks, we cannot define absolute information content, but only relative information content, in terms of accuracy. That is, we can say that accuracy (and thus information content) is improved or reduced when changing the network structure, in particular with respect to the number of non-linearities. We can however not say how much information is passed through, but only compare relative amounts of information with different network structures. Vector quantization A particular form of autoencoders which have become fashionable is the VQ-VAE, or vector quantized variational autoencoder. I won’t be going into the ‘variational’ part here, but the vector quantized autoencoder refers to systems where the bottleneck is also quantized. In particular, vector quantizers have a fixed number of quantization levels such that the bitrate is well-defined. The above analysis is thus not directly applicable to such systems. Heuristically, I would argue (and guess) that the encoder complexity has to be sufficient, such that it can digest information into a form which the VQ can handle. Increasing the encoder complexity further would not improve reconstruction accuracy, since it is limited by the VQ accuracy. Conversely, if the encoder has a given structure, then the VQ bitrate has to be sufficient such that it can take full benefit of the embedding. From the space-filling curves above, you can appreciate that if the VQ bitrate is low, then it cannot model the complicated information contained in the high-recursion curves. In other words, the encoder structure and the VQ bitrate have to be jointly matched for optimal performance. Conclusion and to-do’s This is was my first, quick-and-dirty attempt of characterizing the information content in autoencoders. My own impression is that I’m on to something. Clearly a complex encoder can compress information into a narrow bottleneck such that it can be reconstructed with high accuracy. In fact, assuming perfect accuracy (no numerical round-off errors), then any vector could be compressed to a single real value and reconstructed with arbitrary accuracy, if the corresponding encoder and decoder are sufficiently complex. The magic is in the way the space-filling curve embeds infinities; two infinitely accurate signals can be interleaved together without loss of information. The above presentation does not have rigorous proofs and there’s plenty of hand-waving involved. For example, I detailed only the case where a 2D signal is mapped to a 1D signal (2D-to-1D), it can be easily extended to ND-to-1D, but a bit more reflection is needed to extend it to arbitrary width bottlenecks, ND-to-KD. I also did not properly define reconstruction accuracy, nor the number of RELUs in a space-filling curve and so on. I further would like to actually implement the space filling curve with something like pytorch as a demonstration. The VQ discussion was also superficial. I also haven’t done a literature study; let me know if you know of related work! Perhaps next time. In any case, this is a start for a theoretical discussion about information content in autoencoders and related deep neural networks.

2026-08-11 李宇轩人工智能英-中