<<返回 西安电子科技大学—学生翻译实践成果展
以下内容为西安电子科技大学学生最近一个月内的翻译实践成果
Efficient Data Valuation with Exact Shapley Values

Efficient Data Valuation with Exact Shapley Values Produce Better Models with Less Data. This post is an overview of the work done in the paper — ‘Efficient task-specific data valuation for nearest neighbour algorithms’. Does more data always produce better results? Given the trends in machine learning, you would expect as much. More data seems king given the recent rise in larger and larger machine learning models with exceptional performance. This pattern is exemplified in advances in Machine learning like OpenAIs GPT-3 model, trained on over 45 terabytes of data from books and the internet to support its 175 billion parameters. But this model is effectively dwarfed by Google’s new switch Transformer architecture with 1.6 trillion parameters. So it appears that bigger and bigger models with more and more data are inevitable. If data is so valuable, should we aim to gather more? Why shouldn’t we have more of what is so valuable? There is a problem with this approach. What if the additional data you gather is irrelevant to your problem. What if it is purely noise? Then any additional data will actively hurt the performance of your model. So what can you do to mitigate this problem? Well, for one, you can painstakingly look through your data. This approach is probably one of the most effective tools but is highly time-consuming when you have a lot of data. For computer vision tasks, there are a lot of different techniques that you can use. This post shows some exciting techniques for precisely that. For many cases, the applicable data is relatively easy to identify. However, for others, it is an unclear task at best. Often part of the problem is that the relationships between the data and the outcome are what we’re trying to model. Then identifying what data is most applicable is an arduous task. Fortunately, there is a new method that is perfect for this scenario based on Shapley values. SHAP Values SHapley Additive exPlanations (SHAP) is a game-theoretic approach to explain the output of any machine learning model. This method is fairly well known, but the attribution is based solely on the features. The method provides attribution values from features to the output of the model. SHAP values are based on the Shapley values, which determine how to distribute payment among multiple players in a game fairly. For SHAP values, the coalitions of players are based on the features. SHAP values are incredibly flexible. For example, in computer vision tasks, SHAP values represent the attribution of different pixels to the model’s output. There are many different methods to calculate SHAP values, including a KernelSHAP method which is model-independent. For each variation of SHAP values, the attributions are always related to the features of the models. However, there is an alternative. Use the instances as the players and calculate attribution for each instance. Data Valuation ‘Efficient task-specific data valuation for nearest neighbour algorithms’ is a recent paper providing novel algorithms to calculate exact Shapley values. For the rest of this post, I refer to the Shapley values produced for each instance as Data Shapley Values. Data Shapley Values are a recent innovation that utilizes Shapley values to determine the attribution of different data instances. The motivation of the research was inspired by optimizing records selected from a data market for privacy-preserving machine learning. The data market consists of many different medical records. Therefore, the data buyer chooses a subset of records to purchase from the data market. Because of the data cost, the buyer aim’s to select an optimal subset of patients for their model. The Shapley values in this problem configuration measure the marginal improvements of the utility attributed to each data point average overall possible data subsets. The most significant issue for computing Shapley values is the high degree of complexity. Generally, this is on the level of O(2^N) for exact calculations. However, the researchers have developed a novel algorithm for exact Shapley values designed for K-NN classifiers using a KNN utility. This algorithm relies on the fact that the KNN utility satisfies a piecewise utility difference property. I’ll leave the exact mathematical formulation for the paper and the curious readers. But here is the result. The algorithm performs exact computation with O(N log N) complexity. Experiments The structure of the experiments follows a simple format. First, the data is separated into training, validation, and test sets. Then, the exact Shapley values are calculated based on the attribution of the training instances to the validation instances. Then the training instances are ordered according to their Shapley values. This setup provides the user with the attributions of each instance to the validation set. Next, the user can select only those instances with the highest Shapley value, providing the user with a smaller subset of data. As this process selects data that contributes the most to the performance on the validation set, removing instances will low Shapley values often removes the noisiest instances within the data. And, at the same time, maintaining the most representative examples. The experiments use the diamonds dataset. This dataset contains almost 54,000 diamonds. The features include diamond attributes such as the carat of the diamond, the cut, the colour, and several other features. Some of the features are categorical, and for these experiments, I transform these features into boolean features for each category. The target is the price of the diamonds. I’ve taken 5000 instances for validation and testing. The remainder of the data is used for training. The aim is to produce a model to predict the price of diamonds using less data. I’ve evaluated the performance with a single decision tree regressor. The data is ordered based on the Shapley value, and the performance is measured with an R² score. Next, the Shapley values are calculated for the validation. The chunk of code also sequentially fits a model on smaller amounts of data. The data is considering according to decreasing Shapley Values. Once the model is trained on the subset of training data, the model is evaluated on the test dataset. The experiment results show that with less data, the model performs better on the test data. The peak of the test performance occurs when almost 50% of the training data is removed. What is also intriguing is that the model performance on the validation set peaks when over 60% of the training data is removed. Since the instances are removed based on the Shapley values on the validation set, this pattern makes some sense. Conclusion Despite the increasing availability of a massive amount of data, the quality of the data is crucial. Building your models on low-quality data produces low-quality models. Utilizing Shapley values on instances offer an alternative approach. You can create your models with fewer instances and get improved performance. The experiments show that less data can improve the model performance even with a dramatically smaller subset of data. Data is the new oil, but the quality of that oil matters. Consider using data Shapley values in your next machine learning model.

2026-08-12 李宇轩 人工智能 英-中
Explaining how I reached the top ranks of the new Data-Centric competition

Explaining how I reached the top ranks of the new Data-Centric competition Following the endorsement of Andrew NG himself(!) regarding my last article, it felt natural to share all tips (with code!) of how I handled DeepLearning.ai's new challenge. The competition explained… again! If you are not familiar yet with the new Data-Centric challenge launched by DeepLearning.ai a few weeks ago, you might have a look at the article I wrote a few weeks ago to describe this challenge. Not sure this is worth it? Just follow Andrew Ng advice ? : And if you are in a hurry, here is the long story short: the objective of the competition is to produce the best possible set of pictures to train a predefined model (ResNet50) to recognize roman numerals. The competition offers a “starting base” of approx. 3000 pictures, including noisy and mislabeled numbers, as you can observe below: Let’s the game begin! I am going to present the different steps to reach a good performance in a smooth way but this is obviously the result of many tests and trials I undertook to find the optimum combination! I have also created a dedicated repository on GitHub (link at the end of the article) if you want to explore my solution further. 1. Pictures review The first task is probably the most demanding: reviewing each picture to check a few criteria. Here are the ones I had used: Does this look like a roman number? (If not, we should remove it!) Is the picture correctly labeled? (ex. “II” in “III” folder or vice versa) What is the number quality? (rating from 1: good to 4:poor) What is the background quality? (same rating as above) What is the font style? (“Arial” or “Roman”) What is the exact format of the number? (“viii” or “VIII”) Could we apply symmetries? (horizontal or vertical symmetries are usually suiting “I, II, III, or X” numbers but not for “i”, “ii”, or “VII”) Here are three examples of my evaluations (stored in a tabular way): While reviewing the 3000 pictures (which took me approximately 2 or 3 hours ?), I sometimes had the feeling of “déjà vu” and I started to wonder whether some duplicates were hidden in the dataset? That would have not been very surprising so I had to take this also into consideration. I also designed a simple function to automatically check the content of the folders to evaluate the results of the different operations I would perform. Like all other functions I would use afterward in the notebook, I stored it in a dedicated “dcc_functions.py” (available on the GitHub repository). Here is the output on the initial dataset: 2. Dataset cleaning 2.1 Noise removal I started by removing all the pictures that I had identified as pure noise or, at least, too noisy to train properly the model. This is obviously a personal choice and each participant has probably ended up with a different selection. I identified approx. 260 pictures to be removed (the corresponding list is stored in an Excel file on the GitHub repo). 2.2 Duplicates removal As explained before, I had the feeling that some pictures were exactly the same but manually identifying them was impossible. There were a few technics I knew that could allow solving this issue: Pairing files with identical sizes… but a lot of false positives would arise Pairing files with identical sizes & configurations (like two “II” or “viii”) Pairing files according to their statistics (using, for ex., PIL’s ImageStat) Pairing files using Structure Similarity Index (some explanations here) Pairing files according to their “hash” number As it was not a “life or death” matter, I decided to use the second solution which was both easy and quick to implement. The script (available here) identified approx 200 pairs of twin pictures, out of which 53 were actually genuine duplicates (some examples below): 2.3 Moving some pictures in the right folders No need to spend a lot of time on that one: when a picture was mislabeled, I simply moved it back to the folder it belongs to. 2.4 Edgy or not edgy? Before we go further, I’d like to share an interesting finding: I reviewed the pictures twice: when I entered the competition and, a second time, when I had a better idea of what to look for in the pictures. When reviewing the original pictures for the first time, I had excluded a lot of “edgy cases” that seemed too ambiguous to train the model. But a few weeks after the competition started, I started to get used to these edgy cases and consider them differently, like: “Well, it could be good to include this one to teach to the model that this case might happen.” I ended up adding approx. 80 pictures to the dataset. Counter-intuitively, the performance was decreasing with this new selection, including more edgy pictures. How come? One of the participants, Mohamed Mohey, highlighted on the dedicated Discourse thread that the 32x32 transformation (applied to the dataset before the training) would sometimes completely denature the essence of the picture, as shown in the example below: We can observe that, due to this 32x32 transformation, an obvious “III” is becoming a plausible “II”, explaining why some edgy cases would not necessarily bring valuable information to the model. It would probably have been a good thing to review the pictures after a 32x32 transformation but I did not! 2.5 Using the “label book” pictures to train the model The organizers from DeepLearning.ai had provided a set of 52 pictures, not existing in the “train” or “validation” folders, to evaluate our model’s performance when the ResNet50 training was over. It was a good way to have a sense of how the model would be performing on the final and hidden dataset but I had also “guessed”, thanks to the scores displayed on the leaderboard, that the final evaluation on the hidden dataset was including 2420 pictures (see the corresponding notebook here). So 52 pictures were not very representative anyway! So I simply included these pictures in my training folder! The merrier, the funnier ? 2.6 Evaluating the impact of the augmentation technics As you might know, it is quite common to use augmentation technics on a dataset composed of pictures to help deep learning models identify the features that allow to properly infer the classes. I decided to consider a few of them: Horizontal and Vertical Symmetries Clockwise and Anti-clockwise rotations (10° and 20°) Horizontal and Vertical Translations Cropping the white areas in the pictures Adding synthetic “salt and pepper” noise Transfering noise of some pictures to some others 2.7 Implementing the customs functions The first functions are quite simple and easily implemented with PIL, OpenCV, or even “packaged solutions” such as ImgAug. I thought it would be more interesting to share some tips regarding some of the custom functions I had designed ? 2.7.1 Squared Cropping Function The cropping operation is an interesting one! As the picture will, ultimately, be converted to a 32x32 picture, it might be better to zoom in on the area where the number is located. However, if the number does not have a “squared” shape, the result could be distorted when converted to 32x32 (as shown below). I redesigned the function so that the cropped output will always have a square shape and avoid this distortion effect: 2.7.2 “Salt and Pepper” Function As the background is probably not always plain white on the final evaluation dataset, I tried to augment pictures by adding a synthetic background. I used the “salt & pepper” function which is, basically, adding “0” and “1” randomly into the NumPy arrays describing the pictures: 2.7.3 Background Noise Transfer Function I was not fully happy with the results of the “Salt and Pepper” function as the noise was always homogeneous so I imagined another way to add noise to the pictures. I recycled some of the pictures that I had originally considered as unreadable and made them become some “noisy background” basis. There were also some pictures with a “heavy background” for which I removed the number (as shown below) to get more samples. It provided me a “noisy backgrounds bank” of 10 pictures that I added randomly to some pictures after applying horizontal or vertical symmetries: 2.8 Choosing the best augmentations As the number of pictures allowed could not exceed 10.000 elements, I had to know which transformations were providing the highest impact. I decided to benchmark them by comparing a baseline (a cleaned dataset with no transformation) and the individual performance of each of the augmentation technics (summary below): We can observe that the rotations, translations, and cropping were bringing a significant impact compared to others so I decided to focus on that ones. And “voilà”! As the process is stochastic (transformations are applied with a 50% probability and some random parameters), each iteration of the script will produce a unique combination of pictures. Many of my tests produced a performance of around 84% while the highest competitor had reached 86% (with a 64% baseline). Honorable I guess ? There would have been some additional tweaks to consider (like creating my own pictures and adding them to the dataset but I choose to only rely on the initial pictures provided). Some others probably gave it a try! A global overview of the competitors’ performance It is probably also worth mentioning that I have analyzed the performance of competitors during the first weeks of the challenge (until 26/08) and we can see how quickly most of the participants reached an acceptable performance, converging quickly towards 75% and above: Final words As mentioned earlier, after a proper review/cleaning of the data and a script executed in less than 30 seconds, you can easily outperform what state-of-the-art models could produce with noisy data! I really enjoyed participating in this challenge which, according to me, was more demanding of “fresh ideas” than “GPU power” and I am really looking forward to the next one! We had a lot of fun and rich interactions with other competitors and Lynn (from DeepLearning.ai) to share our views on this “first-of-its-kind” contest. Many participants were more seeking to share their views and findings along the way rather than being at the top of the leaderboard. I also know how difficult generating “average and noisy data” can be… so congratulations to the organizers for delivering such good material to work on! And, of course, I hope you liked the second part of this Deep-Dive on the Data-Centric Challenge from DeepLearning.ai! As promised, here is the link to the GitHub repository and feel free to share your experience and/or findings in the comments:

2026-08-11 李宇轩 人工智能 英-中
Information bottlenecks

Notes on speech processing, 2.9.2021 Information bottlenecks and dimensionality reduction in deep learning Autoencoders and other deep neural networks with information bottlenecks have become fashionable. The heuristic idea is that the dimensionality of the hidden layers is reduced such that the network is forced to focus on the important part of the data. Experiments have also demonstrated that autoencoders are efficient in this sense. I have however been left wondering whether the amount of information can be characterized in exact terms. How much information flows through the bottleneck? How would we even measure that? This short note is my attempt at characterizing and understanding the problem. I will start with some classical concepts of information theory and linear algebra and then discuss the extent to which such concepts are applicable in machine learning. A central result is that dimensionality of a hidden layer cannot alone be used as a measure of information content. Information content in discrete representations If a system has two states, A and B, then obviously we can represent the state by one bit. Four states can be represented by 2 bits, 8 states by 3 bits and in general, N states by log2(N) bits. We can therefore always easily determine the number of bits required for systems with a finite number of states. The amount of bits needed to describe the state is then a direct measure of the information content or entropy of the system. We can expand this to countable sets, such as integers, if we in additional have the access to probability of each state. Then we can make statements about average bitrate, that is, if we observe the system many times, how many bits do we on average need for representing the state? If the probability of state k is Pk, then the amount of bits needed to represent that state is log2 Pk. That bitrate, log2 Pk occurs at probability Pk, such that the average bitrate can be calculated as the sum, sum Pk log2 Pk, where the summation goes over all k. This applies also when k goes over an infinite but countable set. Information content in linear, continuous valued systems If the title is confusing, just think of linear algebra. How much information is there in a vector x of length N. Well, it is not really defined. What we do however know is that if we multiply it with a matrix A, as y=Ax, then if the matrix A is full rank, then all information is retained. In fact, then we can recover x from y by the inverse x=inv(A)y. No information is lost. Clearly the rank of A thus defines its capacity remove information. If rank(A)<N then information is lost and cannot be recovered from y. This is not yet the whole story though. In practical implementations of the inverse, we know that it is not only the rank which is important, but also the conditioning of A. If any of the singular values of A are close to zero, then A becomes ill-conditioned such that the recovery of x from y becomes numerically difficult. In the best case, we loose accuracy, such that x can be recovered only approximately, in severe cases information can be entirely lost. The information content is thus not only described by dimensionality, but also characterized by accuracy. As we shall see, I argue that it is more useful to characterize loss of information as a loss of accuracy rather than loss of dimensions. Diversion: Space filling curves If you have not heard about space-filling curves, start by watching the Numberphile video about them. The idea is an infinite recursion; you start with a simple shape which goes through a space. Then you add wiggles to that shape so that it spreads more over the space. Repeatedly adding more wiggles makes the curve spread out more and more, such that it converges to covering the whole space. The one-dimensional line thus covers the whole two-dimensional space (i.e. its Hausdorff dimension is 2). In terms of information content, now, the one-dimensional curve contains the information of the two-dimensional space. If we start with some particular point in 2D-space (x,y), we can convert that to a point d on the one-dimensional line, and then convert it back to the 2D-point (x,y). It is just that there is an infinite recursion involved, so this is not a practical algorithm. We can however, implement a finite number of recursions to get an approximation. In the example below, I have implemented an Hilbert-curve and plotted the curve for different number of recursions N. We can readily see that for each iteration, the accuracy with which the curve fills space is doubled (error is halved i.e. error energy is 1/4th). By accuracy I refer to the average distance from a random point in 2D space to the closest point on the curve. Each iteration, on the other hand, splits every segment into 4 sub-segments, at a cost of 2 bits. Halving the error thus comes at a cost of 2 bits. This results thus follows results of conventional lossy coding; halving error costs as many bits as we have dimensions. Now we have 2 dimensions so halving error costs 2 bits. Information content in autoencoders Observe that the above space-filling curve construction can be interpreted as an autoencoder. The 2-dimensional space is mapped (encoder) to a 1 dimensional space (bottleneck), which we can recover with the inverse (decoder). The curve is piecewise linear and could easily be implemented with a single layer of rectified linear units (RELUs). Each recursion consists of a subdivision into 4 parts, such that we can expect that the network can be implemented with 2^(2N) RELUs. Conversely, the error of the mapping is halved if the number of RELUs is quadrupled. A red herring One could easily be fooled to think that we can do some simpler space filling curve than Hilbert (or other equivalent curves). For example, we could draw zig-zag lines going end-to-end on dimension x and then takes a step 1/N on dimension y. This can be implemented with O(N) RELUs. The accuracy of this map would then be relative to 2^-N instead of 2^-(2N). However, we would then have error only on the y dimension and the x dimension could be always perfectly reconstructed. Our accuracy argument thus applies as before, we need 1 bit for each dimension to halve accuracy, when assuming that accuracy on each axis is equal. Reconstruction accuracy as a measure of information The pertinent consequence for autoencoders is that the dimensionality of the bottleneck does not alone define the amount of information that passes through. By exponentially increasing the number of non-linearities in the encoder and decoder, we gain a log-linear decrease in mean square error. Since we thus cannot measure information with the number of dimensions, we should therefore rather measure the amount of information in terms of reconstruction accuracy. This approach is in line also with conventional concepts in probability and statistics. For continuous valued variables x, we cannot define a probability, but only probability distributions, since there are an infinite number of possible values and any particular value would always have probability zero. In a similar fashion, for continuous-valued information bottlenecks, we cannot define absolute information content, but only relative information content, in terms of accuracy. That is, we can say that accuracy (and thus information content) is improved or reduced when changing the network structure, in particular with respect to the number of non-linearities. We can however not say how much information is passed through, but only compare relative amounts of information with different network structures. Vector quantization A particular form of autoencoders which have become fashionable is the VQ-VAE, or vector quantized variational autoencoder. I won’t be going into the ‘variational’ part here, but the vector quantized autoencoder refers to systems where the bottleneck is also quantized. In particular, vector quantizers have a fixed number of quantization levels such that the bitrate is well-defined. The above analysis is thus not directly applicable to such systems. Heuristically, I would argue (and guess) that the encoder complexity has to be sufficient, such that it can digest information into a form which the VQ can handle. Increasing the encoder complexity further would not improve reconstruction accuracy, since it is limited by the VQ accuracy. Conversely, if the encoder has a given structure, then the VQ bitrate has to be sufficient such that it can take full benefit of the embedding. From the space-filling curves above, you can appreciate that if the VQ bitrate is low, then it cannot model the complicated information contained in the high-recursion curves. In other words, the encoder structure and the VQ bitrate have to be jointly matched for optimal performance. Conclusion and to-do’s This is was my first, quick-and-dirty attempt of characterizing the information content in autoencoders. My own impression is that I’m on to something. Clearly a complex encoder can compress information into a narrow bottleneck such that it can be reconstructed with high accuracy. In fact, assuming perfect accuracy (no numerical round-off errors), then any vector could be compressed to a single real value and reconstructed with arbitrary accuracy, if the corresponding encoder and decoder are sufficiently complex. The magic is in the way the space-filling curve embeds infinities; two infinitely accurate signals can be interleaved together without loss of information. The above presentation does not have rigorous proofs and there’s plenty of hand-waving involved. For example, I detailed only the case where a 2D signal is mapped to a 1D signal (2D-to-1D), it can be easily extended to ND-to-1D, but a bit more reflection is needed to extend it to arbitrary width bottlenecks, ND-to-KD. I also did not properly define reconstruction accuracy, nor the number of RELUs in a space-filling curve and so on. I further would like to actually implement the space filling curve with something like pytorch as a demonstration. The VQ discussion was also superficial. I also haven’t done a literature study; let me know if you know of related work! Perhaps next time. In any case, this is a start for a theoretical discussion about information content in autoencoders and related deep neural networks.

2026-08-11 李宇轩 人工智能 英-中
What does word2vec actually learn?

What does word2vec actually learn? And how to train embeddings from similarity functions Representing discrete objects by continuous vectors, the so-called embeddings, has been at the heart of many successful machine learning solutions. The superiority comes from the fact that, unlike the original discrete objects, the embedding vectors offer a compact representation that captures the similarity between the original objects. In this article, we consider the famous word2vec algorithm. Word2vec is simple and intuitive. At a high level, it says that words that appear frequently close to each other should have a similar vector representation. In particular, the example embedding(man) - embedding(king) ~ embedding(woman) - embedding(queen) has become the poster child for the ability of word embeddings to capture word semantics. However, the optimization objective cannot be presented by a well-defined quantity. For comparison, consider learning word embeddings using matrix factorization. Let D be a text corpus consisting of m documents and a vocabulary of n unique words. We compute the n-times-m word, document matrix M, where M[u,v] records how many times word u occurs in document v, see Figure 1. The matrix factorization is defined as In the following, we will slightly abuse notation and denote by u both a word u and its embedding vector. In this case, we know that for a word embedding u, and a document embedding v the inner product between u and v preserves the information how many times the word u occurs in the document v. The larger the embedding dimensionality d, the better the approximation. Unfortunately, there is no such clear formulation of the optimization objective for the word2vec model. What exactly does the inner product of two word vectors in word2vec preserve? And do the embeddings necessarily become better by increasing the dimensionality d? A research paper by Levy and Goldberg answers exactly this question [1]. In this article I present the theoretical results from [1] and later show how they can be used to design a more general class of embeddings. Training word embeddings: word2vec Let us briefly consider how word2vec with negative sampling works. For a more comprehensive description, we refer to this article. Let D be the corpus consisting of word, context pairs. In word2vec the context of word w is defined as the k words surrounding w where k is usually a small constant varying between 5 and 15. We want to learn word embeddings such that if two words frequently co-occur in the corpus, their inner product is large. Consider a word w and let c be a word in its context. For word pairs (w,c) occurring together in the corpus, we want the inner product of the embeddings to maximize the probability that (w,c) indeed appears in the corpus (denoted as D=1). The probability is modeled by a sigmoid function: The above has a trivial solution, we can simply make all inner products arbitrary large. Thus, we also introduce negative pairs, i.e. pairs that do not co-occur in the corpus, for which the objective is: The algorithm can be summarized as follows: We run the above algorithm for several epochs over the corpus in order to guarantee that the learning process converges in an optimum. The theoretical analysis Fix a word w and consider the objective for all pairs in which w appears. Let #(w,c) be the number of appearances of the pair (w,c) in the corpus. We can write the objective as where the second product is over the negative pairs we generate. By taking the logarithm of the objective and observing that each negative word cN has a chance to be sampled, we obtain: Let us explain the above. The word w is fixed, we consider all word context pairs (w,c) that appear in the corpus and we sample k negative pairs (w, cN) such that each word c is sampled with probability #(c)/|D|. We want for positive pairs the inner product to be a large positive number. For negative pairs we want the inner product to be a negative number with large absolute value. Observe that |D|, the number of pairs in the corpus, is constant. Thus, by dividing the above expression by |D| the objective becomes This already provides us with some intuition. The objective is to optimize the embeddings such that they reflect the probability for a positive pair to be sampled as opposed to a pair being sampled at random. Positive pairs are generated with probability And for negative pairs, two words are sampled independently at random, each with probability By setting the inner product as an unknown parameter and solving the corresponding optimization problem, we can find the optimal value for the inner product: In the above P(w,c) is the probability of occurrence of the pair (w,c), and P(w) is the marginal probability of occurrence of word w in the corpus. The above turns out to be a widely used word association measure in natural language processing, the pointwise-mutual information (PMI) measure. This is pretty amazing! It turns out that word2vec is essentially equivalent to matrix factorization where the matrix entries are the PMI scores between word pairs. And PMI as a distance measure was used for NLP-related tasks since the 80s [2], long before the emergence of the concept of word embeddings. Embeddings based on arbitrary similarity functions Now it is easy to see that we can simply replace the probability for sampling positive and negative pairs. We only need to update the second and third steps in the word2vec algorithm presented above: Why is this helpful? This gives us more freedom to assign importance to pairs. We can become creative and consider different similarity measures. For example, the Jaccard similarity between words is defined as follows: Thus, we can learn embeddings that optimize the objective that words w and c are similar to each other if the presence of w implies that it is likely that c also appears in the document, and vice versa. In this case, the pair (“keira”, “knightley”) will likely have a higher score than (“data”, “science”). The objective becomes: And we can also model the probability for generating negative pairs. For example, Pr(w) can be the uniform distribution where all words have the same probability of being selected, disregarding how often they appear. Sampling from a distribution If we could compute and store the similarity for all pairs (u, v), then sampling according to the similarity becomes trivial: just store the pairs with their similarity scores as weights and sample using an algorithm like numpy.random.choice. However, this might be computationally infeasible. There are different approaches to deal with the problem with a larger number of pairs. In general, we want to use as positive pairs only those that have a high similarity score. If your similarity measure is based mainly on counts, then a subsample of the data will preserve the most frequent pairs but many infrequent pairs will be filtered out. For example, we can consider only a subset of the documents in a corpus. Frequent word pairs such as (“data”, “science”) will likely survive. But this might not be the case with (“keira”, “knightley”). For each object, consider only the t nearest neighbors. For example, we might use the publicly available implementation from scikit-learn which uses algorithms like kd-trees to speed up similarity search. These algorithms work well for data that is not very high dimensional. Otherwise, one can consider approaches such as Locality-sensitive hashing that will generate similar words. This is especially true for measures like Jaccard similarity. A practical implementation For illustrative purposes, we implemented a simple solution for learning document embeddings from text corpora. The problem is orthogonal to the problem of training word embeddings: we train vector representations for documents based on the words they contain. We consider the IMDB sentiment analysis dataset. The dataset consists of movie reviews by users and each review is labeled with a positive or negative sentiment. After preprocessing the text, we transformed the documents to vectors by using a tf-idf encoding such that each document The parameter min_df says we consider only words that appear in at least 0.1% of the documents. Essentially, this prevents us from using very specific words that might appear just in a couple of documents. For each input vector, find its t nearest neighbors. This can be achieved using an off-the-shelf package such as scikit-learn’s K-NearestNeighbor which returns nearest neighbors for: Compute the similarities for the generated n*t positive pairs, sort them in an array, and sample according to their weight using numpy.random.choice(): Use a Keras generator to generate positive and negative pairs: Feed the generated pairs into a shallow neural network with an embeddings layer, a Dot layer computing the inner product, and an output layer with a sigmoid activation function: The above approach will train embeddings: Then we can extract the embedding layer for each word and cluster the documents (similarly to what is shown in the gif in Figure 1). We observe that the sentiment distribution in the two clusters is very different: Code The Python implementation for the above is publicly available at: https://github.com/konstantinkutzkov/sim2vec [1] Omer Levy, Yoav Goldberg. Neural Word Embedding as Implicit Matrix Factorization. NIPS 2014: 2177-2185 [2] Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22–29, 1990.

2026-08-11 李宇轩 人工智能 英-中
Face Landmark Detection using Python

Face Landmark Detection using Python The comparison between dlib and mediapipe library. Introduction Face landmark detection is a computer vision task where we want to detect and track keypoints from a human face. This task applies to many problems. For example, we can use the keypoints for detecting a human’s head pose position and rotation. With that, we can track whether a driver is paying attention or not. Also, we can use the keypoints for applying an augmented reality easier. And there are so many solutions that we can generate based on this task. Thankfully, we don’t have to understand the concepts of face landmark detection in detail. We can use the prebuilt library like dlib, OpenCV, and mediapipe. In this article, I will show you how to implement face landmark detection with dlib and mediapipe. Without further, let’s get started! Face Landmark Detection with Dlib Dlib is a library for applying machine learning and computer vision solutions. This library is based on the C++ language, but we can use a language like Python for using the library. One of the solutions that we can apply by using this library is face landmark detection. Now let’s get into the implementation. Install the library Installing a library can become a problem. If we don’t have a good guide, installing the library can take several days. Dlib is one of them. Because it uses C++ as the primary language, we have to install C++ tools for installing the library. There are several steps that we should do to install it. Here are the steps: First, install the CMake. You can download the software here. If you are using Windows, please locate the CMake file path first. Then, set the path to the executable path on the environment variable. Then, install Visual Studio with the C++ dependencies to it. You can download the software here. For the dependencies, you can look at this screenshot below: After you install the Visual Studio, the next step is to install the Python. To make your installation simpler, I recommend you for installing Anaconda. You can download it here. For the Python version, I recommend you for using the 3.6.6 version to avoid any errors. Lastly, install the CMake, dlib, and OpenCV library by using pip. Here is the command for doing that: Import the libraries After we’ve installed the libraries, the next step is to import them into our code. We will import OpenCV for retrieving inputs from the webcam, NumPy for numerical computation, and Dlib for detecting keypoints from a face. Here is the code for doing that: Initialize the objects Now let’s initialize several variables. There are three must need variables that we will initialize: A detector for detecting one or more faces. We set the dlib.get_frontal_face_detector function inside the variable. A predictor for detecting keypoints from faces. We set the dlib.shape_predictor function inside the variable. This function needs a pretrained model location as the parameter, which you can download here. The cv2.VideoCapture object for capturing images from the webcam. Also, we set a parameter with value 0 for capturing images from a webcam. Let’s write this code for initializing variables: Face landmark detection mechanism As you can see from above, we initialize the face landmark detector by using the pretrained model. The model is based on ensemble regression trees because the model will predict continuous numbers. You can read the details about the model here. That model is trained on the iBUG-300 W dataset, where it contains images and their corresponding 68 face landmark points. In general, those landmark points belong to the nose, the eyes, the mouth, and the edge of a face. You can download the dataset here. Here is the visualization of the face landmark locations below: Implement the face landmark detection Now you know how the face landmark detection algorithm works. Now let’s implement the algorithm. For implementing that, you can see the code below along with explanations on each line of code: By combining all the code as one, now let’s try the code! If the code doesn’t have any errors, the webcam will display the result along with the keypoints. In my case, here is the result: Face Landmark Detection with Mediapipe Mediapipe is a tool for implementing ML-based computer vision solutions. The tool is created by Google. This tool contains varieties computer vision solutions, such as face detection, pose estimation, object detection, and many more. The advantage of this library is that you can apply the solutions on many platforms, such as web, mobile, PC, and many more. I’ve already explained in the previous section to you how to implement face landmark detection using dlib. Now let’s implement the face landmark detection using Mediapipe. The mechanism The library uses the BlazeFace model for detecting face landmarks. BlazeFace is a deep learning model that is already optimized for low spec devices like smartphones. Therefore, we can use the model in real-time. BlazeFace contains two main steps. First, the model detects one or more faces on an image. Second, the image detects around 468 face keypoints by using regression. Different from the dlib library, this model detects 3D coordinates. Those x and y coordinates are normalized from the image scale. The z coordinate is retrieved by taking the relative calculation between the screen and the model x coordinates. You can read more details here. Here is the flattened mesh from a face with their corresponding indexes: Implementing the face landmark detection In general, the pipeline for implementing face landmark detection is the same as the dlib library. It starts from importing libraries, initializing objects, detect face and its landmarks, and done. Here is the code for doing that: If you implement the code correctly, the image will display on your computer. Here is the preview of my result: The comparison We have already take a walkthrough of face landmark detection libraries using dlib and mediapipe. We can say that both libraries are easy to use. Therefore, we can build our solution rapidly. However, there are differences between them. The dlib library needs C++ dependencies it. That’s why we need CMake and Visual Studio for installing the library. Also, this library needs a specific python library. Therefore, you have to create a virtual environment if you don’t have the supported Python version to run the library. On the other side, installing mediapipe is easier. All you need to do is to install from pip only. Therefore, you don’t have to worry about installation more while using the mediapipe. In the case of the solution, the dlib can detect only the 2D coordinates of the keypoints. On the other hand, the mediapipe can detect the 3D coordinates of the keypoints. Therefore, you can use those keypoints from the mediapipe library for estimating the head pose. Final Remarks Well done! Now you know how to implement face landmark detection using Python. I’ve shown you the libraries like dlib and mediapipe for implementing the solution. I hope it helps you in implementing a computer vision solution. Also, I hope it can become your foundation to build more complex applications. If you are interested in my articles, you can follow me on Medium for more articles like this. Also, if you have any questions, you can contact me on LinkedIn. Thank you for reading my article! References [1] https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/ [2] https://www.analyticsvidhya.com/blog/2021/07/facial-landmark-detection-simplified-with-opencv/

2026-08-11 李宇轩 人工智能 英-中
Topic Model Based Recommendation Systems

Topic Model Based Recommendation Systems A very quick and (hopefully) easy to follow introduction into the intuition (and very low level Maths) involved in Topic Model Based Recommendation Systems. Check out my GitHub for a working simple recommendation system based on Topic Modelling. In todays world, sometimes it feels like we are plagued with never ending decisions. Whether it be the Friday night movie or the next song to keep people dancing at an NYE party. So how do recommendation systems actually work? In this article I’m going to explain one approach based on Topic Modelling using a Latent Dirichlet Allocation (LDA). Topic Modelling Before we talk about how to model a topic, we need to first understand what a topic actually is. This is not an intuitive idea to think about so we will describe it in terms of collections of words. If we have a collection of documents randomly selected from a database, we can imagine that some of the words contained in these documents may be semantically similar, or be related to the same area. For example, if these documents were a collection of film reviews. We can imagine that we might be able to form groups of positive and negative reviews based on the words contained within them. Alternatively, we may wish to form collections of documents relating to Sci-Fi, Comedy, Romance etc. So, as we are starting to reorganise this collection of documents into many smaller collections. At the same time, we are starting to see that there are many layers of possibilities. Where each possibility is a selection of topics. You may then ask the question, how can we get a computer to organise these documents into topics and how do we know what topics it will pick? To answer this, we are going to think at a slightly deeper level… The Less General Idea In this article, I am going to describe one option for how to get a computer to perform topic modelling. However there are many other algorithms, approaches and methodologies out in the wild. Going back to our collection of documents and sticking with the approach of looking at the words contained within them, we can build up a vocabulary containing all of the unique words in the database. Say we have 1000 different unique words across 4 documents and we want to characterise each document by which words are contained within them (and by how many of each word). So now we can imagine that for each document, we have a vector with dimension 1000 (one dimension for each unique word). And at each position in each vector there is the count of how many times the word that this position corresponds to, appears in the document. For example, the first position in the vector corresponds to the first word in the vocabulary, which we will say is “Robot”. The first document is a film review about Terminator 23 (or whatever number we are on now…) and so the word “Robot” is mentioned 19 times. Therefore in the first position of the vector corresponding to the first document, we have (19,…). The second position corresponds to the word “Sport” and is mentioned zero times in the Terminator review which gives us (19, 0, …) and so on… The second document happens to be a review for a Tennis documentary and so for this document we have the vector (0, 10, …), since “Robot” is mentioned 0 times. Too many topics? Yes, way too many. I agree, as will your computer. At the moment we have essentially defined a “topic” for each word in the vocabulary. Which clearly is not ideal and will not provide us with many clearly separated topics to play with later on. The next step then is to find a middle ground where each of our reviews belong to a broader topic which is defined by a number of words in the vocabulary. Latent Dirichlet Allocation We are now looking to reduce the number of connections coming from each document by introducing a hidden layer of topics between the individual words in the vocabulary. This is exactly what we are going to use Latent Dirichlet Allocation (LDA) for. LDA requires us to define a required number of topics we want, this is what we call a hyperparameter (a parameter which is defined before the algorithm is run). In a real world we can imagine scanning over many numbers of topics to find the best outcome (aptly called a hyperparameter scan). Let’s say we are looking for 10 topics. This means we wish to add a hidden layer between the previous connections we had, which were linking documents to words (remembering our large vectors (19, 0, …) and (0, 10, …)). Mathematically and computationally this is very desirable for us, since we can replace these huge vectors describing each document with new vectors of size 10 (or however many topics you have chosen). To describe what the LDA is attempting to achieve, it is easiest to look at a matrix formulation below… So in our original perfect description of the documents, we had the matrix S. This matrix is the most complete picture we can have of all the documents, there is no information lost since each word is its own topic and if we ignore the orderings of the words, we are able to perfectly recreate each document. However, some of the information is too fine grained. For example, we don’t really need a separate topic for “Robot” and “Android”. These can just be combined into a coarser topic of “Sci-Fi” or whatever you want to name it. In this case, we can see that we have sacrificed some information. So if we were to recreate the document, there is no guarantee that we would get back the word “Robot”, since we only have information that a similar word from the same topic was mentioned. This is what matrices M and N are doing in this case. The Latent dimension K is our hidden layer of topics (i.e. 10 topics). And given this dimension K, the LDA is learning the matrices M and N in an attempt to best recreate the matrix S. It is not essential you understand the maths here to know what is going on, the takeaway points are: LDA creates coarser grained topics based on the documents given to it. As the LDA model does this, we lose specific information about the individual documents. If you think about how you would sort film reviews into 10 topics, this will hopefully start to make sense. Imagine if I asked you to summarise one of the topics you had created, you wouldn’t be able to recite every word of each document, but you’d probably be able to give a few of the most common words describing the overall topic. Hence, you have lost information. Why do we want to lose information? Losing information may sound like a bad thing, but it actually helps Machine Learning models find patterns in data that they would have missed otherwise. The trick is to not lose the useful information, which in some cases can be learnt in an algorithm or it must be controlled via a hyperparameter (as for LDA with the number of topics). Let’s look at an example… say we have a noisy set of data points loosely describing a quadratic function (plot A). In plot B, here we have tried to keep all the information we have to describe the data points, but does this look right? Probably not, our model is too specific and has not really caught the general trend. In plot C, we have lost too much information. The model is too simple for the data. In plot D, we have a good description of the data. We have found a good hyperparameter to be able to lose the less useful information and keep the important parts. This may all sound familiar because it is exactly the description of underfitting and overfitting data, just in the context of NLP and topic modelling. Back to Topics… Hopefully that brief interlude was useful. If not, sorry about that but we’re back on track now! So we have our LDA model which has sorted all these documents into a distribution of topics. Note that these are not hard clustered topics, they are distributions. So if 3 of the 10 topics we had were Sci-Fi, Documentary and Technology, we could have a film review for a RomCom between astronauts that would have a distribution of (0.2, 0.4, 0.3,…). On the diagram above, this corresponds to the top layer of connections between the documents and the topics. These distributions are actually called Embeddings. Since we have embedded information about the document into a usable mathematical format. Take our embedding from earlier, a = (0.2, 0.4, 0.3,…). Now if we have two more documents with embeddings b = (0.1, 0.5, 0.2,…) and c = (0.8, 0, 0.1,…). Is b or c more similar to embedding a? There are a few ways to answer this question but we will choose the simple (and very effective) cosine similarity. Which you may remember from Maths courses at school or college. This basically calculates the distance between the two embeddings and if they are closer together, they are more similar. In this case a and b are more similar than a and c. So if we were to ask the computer which one out of b and c would it recommend given a. We would expect that it may recommend to us the document b. This is how we can use topic modelling to create recommendations. Round up So we have looked at what topics are, then at what the LDA algorithm gives us and then finally how we can use these mathematical objects to produce recommendations. The key point for recommendations in topic modelling based on similarity is that we are assessing how similar the encoded information of each document is. As we have discussed, these recommendations may be completely terrible depending on how our topics are found and distributed (plot B or plot C from the crude explanation earlier). Or they might be fantastic and hit the sweet spot (plot D). The other point to note is that although we can control the number of topics, we have less control on what these topics are. This is determined from the data, which we are able to manipulate by removing unimportant words for example. But if we don’t have a representative sample of Sci-Fi film reviews in our database then the likelihood is that this topic will not exist. Another reason why its all about the DATA… This is a really short and low level insight into how these types of algorithms can be used to give recommendations. There is loads of much more extensive descriptions out there so I encourage you to read around. If you are interested, I have a working recommendation system code on my GitHub. References [1] — D. Blei, et. al., Latent Dirichlet Allocation (2003)

2026-08-11 李宇轩 人工智能 英-中
Churn prediction model

Churn prediction model Musing about a use case that’s been with me for a decade No company likes to lose valuable customers. In the beginning, a company typically focuses on acquiring new clients, then grows by offering additional products to existing clients or trying to get them to use their products more. If all is going well, there comes a point when the company is large enough that it must also choose a slightly more defensive strategy and focus on retaining existing customers. Despite the best user experience, there will always be a group of clients who are not satisfied and decide to leave. The company then faces the problem of how to prevent these (voluntary) departures as effectively as possible. This is where the churn model, among others, comes to the rescue. What is the churn model? It’s a predictive model that estimates — at the level of individual customers — the propensity (or susceptibility) they have to leave. For each customer at any given time, it tells us how high the risk is of losing them in the future. Technically, it’s a binary classifier that divides clients into two groups (classes) — those who leave and those who don’t. In addition to assigning them to one of the two groups, it will typically give us the probability with which the client belongs to that group. It is important to note that this is the probability of belonging to the group of clients who leave. Thus, it is the propensity to leave and not the probability of leaving. However, it is possible to estimate the probability through a churn model. What is it useful for? By knowing which clients are at the highest risk of leaving, we can better target our rescue efforts. For example, we can reach out to these clients with a marketing campaign, reminding them that they haven’t purchased from us in a while, or even offering them a benefit. In addition to knowing which clients to target, we can use the churn model to calculate the maximum benefit price that is still worthwhile. For example, if we know that the estimated probability of a particular client leaving is 10% and their annual revenue is $100, the expected value of future annual revenue is $90. Therefore, an offer that typically reduces the probability of leaving to 5% (the expected value of the revenue is then $95) will be worthwhile for this client, so long as it does not cost more than $5. What do we need for the churn model? Like any supervised machine learning model, a churn model needs training data with response (target) and explanatory variables (features). Based on this training data, the model learns to best capture the relationship between features and target. Typically, this is historical data, where we know which clients eventually left and which did not. Those who left have a positive target (yes, they left). Others have a negative target (no, they didn’t leave). Whilst features describe clients at a point in time when that outcome was not yet known. A properly defined target is fundamentally key. In many cases this is simple (e.g., cancellation of last product), sometimes less so (e.g., no transactions in the last three months). However, it is possible to apply the churn model to both contractual (e.g., bank) and non-contractual (e.g., e-shop) client relationships. Features include any data that can help identify clients who churn. Often this includes socio-demographic data, data on products owned, historical transactions, client-company interaction, e-commerce behaviour, and so on. It is also important to be careful about how far in advance we want to estimate the propensity to leave. In other words, how long is the time between the day we look at clients through the available features and the day we can tell if they have left? If that time is too short, we won’t have much time to make any kind of response. If, on the other hand, it is too long, the model will be less accurate and up to date. What does such a model look like? Modern churn models are often based on machine learning; specifically, on the binary classification algorithms mentioned above. There are a number of these algorithms, and it is necessary to test which one best fits a specific situation (specific training data, amount of data, etc.). Whether you use simple models such as logistic regression, more complex random forest or GBM, or venture into neural networks, you need to pay attention to the following two things. Classifiers have a variety of performance metrics. Since churn is very low for most companies, it is not enough to look at the accuracy of the churn model. For example, if the churn is 10% and the churn model for all clients says they will not leave, it will have 90% accuracy. But this is not useful. So, among other things, you need to look at sensitivity (how many of the clients who actually leave were detected by the model) and precision (how many of the clients identified by the model actually left). Furthermore, it is advisable to not use the resulting model as a black box. Rather, try to understand the parameters based on which decisions are made. Not only can this reveal flaws in the model or data, but it can also be very useful information for product and marketing teams. For example, if we know that the absolute amount of discount has less impact on churn than the relative amount of discount, we can use this to create more effective campaigns and pricing strategies. What next? Once you have the churn model ready, you need to plug it into the day-to-day running of the company. This involves monitoring, evaluating and updating it on an ongoing basis (whether that’s simply re-training it or even adding new features). Consequently, you can start to automatically detect events that tend to increase the propensity to leave that need to be responded to as quickly as possible. External data consultants can help you with both. But beware, it is crucial for the churn model (even more than for other data projects) to involve people with experience and a feel for the specific situation in the company and industry. The article was originally written in Czech and published on Bizztreat Blog. As ever, I’m indefinitely grateful to Chelsea Wilkinson for patiently shaping my thoughts into a publishable format. Thanks for reading! Please feel free to share your thoughts or opinions in the comments. Follow me on Medium, LinkedIn and Twitter.

2026-08-11 李宇轩 人工智能 英-中
XGBoost For Time Series Forecasting: Don’t Use It Blindly

XGBoost For Time Series Forecasting: Don’t Use It Blindly Forecasting techniques don’t work well with all time series When modelling a time series with a model such as ARIMA, we often pay careful attention to factors such as seasonality, trend, the appropriate time periods to use, among other factors. However, when it comes to using a machine learning model such as XGBoost to forecast a time series — all common sense seems to go out the window. Rather, we simply load the data into the model in a black-box like fashion and expect it to magically give us accurate output. A little known secret of time series analysis — not all time series can be forecast, no matter how good the model. Attempting to do so can often lead to spurious or misleading forecasts. To illustrate this point, let us see how XGBoost (specifically XGBRegressor) varies when it comes to forecasting 1) electricity consumption patterns for the Dublin City Council Civic Offices, Ireland and 2) quarterly condo sales for the Manhattan Valley. How XGBRegressor Forecasts Time Series XGBRegressor uses a number of gradient boosted trees (referred to as n_estimators in the model) to predict the value of a dependent variable. This is done through combining decision trees (which individually are weak learners) to form a combined strong learner. When forecasting a time series, the model uses what is known as a lookback period to forecast for a number of steps forward. For instance, if a lookback period of 1 is used, then the X_train (or independent variable) uses lagged values of the time series regressed against the time series at time t (Y_train) in order to forecast future values. Forecasting Electricity Consumption Let’s see how this works using the example of electricity consumption forecasting. The dataset in question is available from data.gov.ie. From this graph, we can see that a possible short-term seasonal factor could be present in the data, given that we are seeing significant fluctuations in consumption trends on a regular basis. Let’s use an autocorrelation function to investigate further. From this autocorrelation function, it is apparent that there is a strong correlation every 7 lags. Intuitively, this makes sense because we would expect that for a commercial building, consumption would peak on a weekday (most likely Monday), with consumption dropping at the weekends. When forecasting such a time series with XGBRegressor, this means that a value of 7 can be used as the lookback period. The model is run on the training data and the predictions are made: Let’s calculate the RMSE and compare it to the test mean (the lower the value of the former compared to the latter, the better). We see that the RMSE is quite low compared to the mean (11% of the size of the mean overall), which means that XGBoost did quite a good job at predicting the values of the test set. If you wish to view this example in more detail, further analysis is available here. Forecasting Manhattan Valley Condo Sales In the above example, we evidently had a weekly seasonal factor, and this meant that an appropriate lookback period could be used to make a forecast. However, there are many time series that do not have a seasonal factor. This makes it more difficult for any type of model to forecast such a time series — the lack of periodic fluctuations in the series causes significant issues in this regard. Here is a visual overview of quarterly condo sales in the Manhattan Valley from 2003 to 2015. The data was sourced from NYC Open Data, and the sale prices for Condos — Elevator Apartments across the Manhattan Valley were aggregated by quarter from 2003 to 2015. From the above, we can see that there are certain quarters where sales tend to reach a peak — but there does not seem to be a regular frequency by which this occurs. Again, let’s look at an autocorrelation function. From the autocorrelation, it looks as though there are small peaks in correlations every 9 lags — but these lie within the shaded region of the autocorrelation function and thus are not statistically significant. What if we tried to forecast quarterly sales using a lookback period of 9 for the XGBRegressor model? The same model as in the previous example is specified: Now, let’s calculate the RMSE and compare it to the mean value calculated across the test set: We can see that in this instance, the RMSE is quite sizable — accounting for 50% of the mean value as calculated across the test set. This indicates that the model does not have much predictive power in forecasting quarterly total sales of Manhattan Valley condos. Given that no seasonality seems to be present, how about if we shorten the lookback period? Let’s try a lookback period of 1, whereby only the immediate previous value is used. The size of the mean across the test set has decreased, since there are now more values included in the test set as a result of a lower lookback period. This has smoothed out the effects of the peaks in sales somewhat. However, we see that the size of the RMSE has not decreased that much, and the size of the error now accounts for over 60% of the total size of the mean. Therefore, using XGBRegressor (even with varying lookback periods) has not done a good job at forecasting non-seasonal data. Conclusion There are many types of time series that are simply too volatile or otherwise not suited to being forecasted outright. However, all too often, machine learning models like XGBoost are treated in a plug-and-play like manner, whereby the data is fed into the model without any consideration as to whether the data itself is suitable for analysis. Therefore, the main takeaway of this article is that whether you are using an XGBoost model — or any model for that matter — ensure that the time series itself is firstly analysed on its own merits. This means determining an overall trend and whether a seasonal pattern is present. The allure of XGBoost is that one can potentially use the model to forecast a time series without having to understand the technical components of that time series — and this is not the case. Many thanks for your time, and any questions or feedback are greatly appreciated. Disclaimer: This article is written on an “as is” basis and without warranty. It was written with the intention of providing an overview of data science concepts, and should not be interpreted as professional advice. The findings and interpretations in this article are those of the author and are not endorsed by or affiliated with any third-party mentioned in this article. The author has no relationship with any third parties mentioned in this article.

2026-08-11 李宇轩 人工智能 英-中
Personalized ASR Models from a Large and Diverse Disordered Speech Dataset

Posted by Katrin Tomanek, Software Engineer and Bob MacDonald, Technical Program Manager, Google Research Speech impairments affect millions of people, with underlying causes ranging from neurological or genetic conditions to physical impairment, brain damage or hearing loss. Similarly, the resulting speech patterns are diverse, including stuttering, dysarthria, apraxia, etc., and can have a detrimental impact on self-expression, participation in society and access to voice-enabled technologies. Automatic speech recognition (ASR) technologies have the potential to help individuals with such speech impairments by improving access to dictation and home automation and by enhancing communication. However, while the increased computational power of deep learning systems and the availability of large training datasets has improved the accuracy of ASR systems, their performance is still insufficient for many people with speech disorders, rendering the technology unusable for many of the speakers who could benefit the most. In 2019, we introduced Project Euphonia and discussed how we could use personalized ASR models of disordered speech to achieve accuracies on par with non-personalized ASR on typical speech. Today we share the results of two studies, presented at Interspeech 2021, that aim to expand the availability of personalized ASR models to more users. In “Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia”, we present a greatly expanded collection of disordered speech data, composed of over 1 million utterances. Then, in “Automatic Speech Recognition of Disordered Speech: Personalized models outperforming human listeners on short phrases”, we discuss our efforts to generate personalized ASR models based on this corpus. This approach leads to highly accurate models that can achieve up to 85% improvement to the word error rate (WER) in select domains compared to out-of-the-box speech models trained on typical speech. Impaired Speech Data Collection Since 2019, speakers with speech impairments of varying degrees of severity across a variety of conditions have provided voice samples to support Project Euphonia’s research mission. This effort has grown Euphonia’s corpus to over 1 million utterances, comprising over 1400 hours from 1330 speakers (as of August 2021). Distribution of severity of speech disorder and condition across all speakers with more than 300 utterances recorded. For conditions, only those with > 5 speakers are shown (all others aggregated into “OTHER” for k-anonymity). ALS = amyotrophic lateral sclerosis; DS = Down syndrome; PD = Parkinson’s disease; CP = cerebral palsy; HI = hearing impaired; MD = muscular dystrophy; MS = multiple sclerosis To simplify the data collection, participants used an at-home recording system on their personal hardware (laptop or phone, with and without headphones), instead of an idealized lab-based setting that would collect studio quality recordings. To reduce transcription cost, while still maintaining high transcript conformity, we prioritized scripted speech. Participants read prompts shown on a browser-based recording tool. Phrase prompts covered use-cases like home automation (“Turn on the TV.”), caregiver conversations (“I am hungry.”) and informal conversations (“How are you doing? Did you have a nice day?”). Most participants received a list of 1500 phrases, which included 1100 unique phrases along with 100 phrases that were each repeated four more times. Speech professionals conducted a comprehensive auditory-perceptual speech assessment while listening to a subset of utterances for every speaker providing the following speaker-level metadata: speech disorder type (e.g., stuttering, dysarthria, apraxia), rating of 24 features of abnormal speech (e.g., hypernasality, articulatory imprecision, dysprosody), as well as recording quality assessments of both technical (e.g., signal dropouts, segmentation problems) and acoustic (e.g., environmental noise, secondary speaker crosstalk) features. Personalized ASR Models This expanded impaired speech dataset is the foundation of our new approach to personalized ASR models for disordered speech. Each personalized model uses a standard end-to-end, RNN-Transducer (RNN-T) ASR model that is fine-tuned using data from the target speaker only. Architecture of RNN-Transducer. In our case, the encoder network consists of 8 layers and the predictor network consists of 2 layers of uni-directional LSTM cells. To accomplish this, we focus on adapting the encoder network, i.e. the part of the model dealing with the specific acoustics of a given speaker, as speech sound disorders were most common in our corpus. We found that only updating the bottom five (out of eight) encoder layers while freezing the top three encoder layers (as well as the joint layer and decoder layers) led to the best results and effectively avoided overfitting. To make these models more robust against background noise and other acoustic effects, we employ a configuration of SpecAugment specifically tuned to the prevailing characteristics of disordered speech. Further, we found that the choice of the pre-trained base model was critical. A base model trained on a large and diverse corpus of typical speech (multiple domains and acoustic conditions) proved to work best for our scenario. Results We trained personalized ASR models for ~430 speakers who recorded at least 300 utterances. 10% of utterances were held out as a test set (with no phrase overlap) on which we calculated the word error rate (WER) for the personalized model and the unadapted base model. Overall, our personalization approach yields significant improvements across all severity levels and conditions. Even for severely impaired speech, the median WER for short phrases from the home automation domain dropped from around 89% to 13%. Substantial accuracy improvements were also seen across other domains such as conversational and caregiver. WER of unadapted and personalized ASR models on home automation phrases. To understand when personalization does not work well, we analyzed several subgroups: HighWER and LowWER: Speakers with high and low personalized model WERs based on the 1st and 5th quintiles of the WER distribution. SurpHighWER: Speakers with a surprisingly high WER (participants with typical speech or mild speech impairment of the HighWER group). Different pathologies and speech disorder presentations are expected to impact ASR non-uniformly. The distribution of speech disorder types within the HighWER group indicates that dysarthria due to cerebral palsy was particularly difficult to model. Not surprisingly, median severity was also higher in this group. To identify the speaker-specific and technical factors that impact ASR accuracy, we examined the differences (Cohen's D) in the metadata between the participants that had poor (HighWER) and excellent (LowWER) ASR performance. As expected, overall speech severity was significantly lower in the LowWER group than in the HighWER group (p < 0.01). Intelligibility and severity were the most prominent atypical speech features in the HighWER group; however, other speech features also emerged, including abnormal prosody, articulation, and phonation. These speech features are known to degrade overall speech intelligibility. The SurpHighWER group had fewer training utterances and lower SNR compared with the LowWER group (p < 0.01) resulting in large (negative) effect sizes, with all other factors having small effect sizes, except fastness. In contrast, the HighWER group exhibited medium to large differences across all factors. Speech disorder and technical metadata effect sizes for the HighWER-vs-LowWER and SurpHighWER-vs-LowWER pairs. Positive effects indicated that the group values of the HighWER group were greater than LowWER groups. We then compared personalized ASR models to human listeners. Three speech professionals independently transcribed 30 utterances per speaker. We found that WERs were, on average, lower for personalized ASR models compared to the WERs of human listeners, with gains increasing by severity. Delta between the WERs of the personalized ASR models and the human listeners. Negative values indicate that personalized ASR performs better than human (expert) listeners. Conclusions With over 1 million utterances, Euphonia’s corpus is one of the largest and most diversely disordered speech corpora (in terms of disorder types and severities) and has enabled significant advances in ASR accuracy for these types of atypical speech. Our results demonstrate the efficacy of personalized ASR models for recognizing a wide range of speech impairments and severities, with potential for making ASR available to a wider population of users. Acknowledgements Key contributors to this project include Michael Brenner, Julie Cattiau, Richard Cave, Jordan Green, Rus Heywood, Pan-Pan Jiang, Anton Kast, Marilyn Ladewig, Bob MacDonald, Phil Nelson, Katie Seaver, Jimmy Tobin, and Katrin Tomanek. We gratefully acknowledge the support Project Euphonia received from members of many speech research teams across Google, including Françoise Beaufays, Fadi Biadsy, Dotan Emanuel, Khe Chai Sim, Pedro Moreno Mengibar, Arun Narayanan, Hasim Sak, Suzan Schwartz, Joel Shor, and many others. And most importantly, we wanted to say a huge thank you to the over 1300 participants who recorded speech samples and the many advocacy groups who helped us connect with these participants.

2026-08-09 李宇轩 人工智能 英-中
6 Common Metrics For Your Next Regression Project

6 Common Metrics For Your Next Regression Project Advantages, Disadvantages & Major Pitfalls So you trained a model, now what? Or, you’ve trained multiple models; how do you decide which one is best? Ok, let’s ask Google. Hmm, Google suggests many metrics that can be used to evaluate your model(s). But now, this becomes a meta-problem of what metric should I use to determine what model to use? So, here’s a list of some common metrics along with their advantages, disadvantages & nuances to start with: Mean absolute error (MAE) Mean squared error (MSE) Root mean squared error (RMSE) Normalized root mean squared error (NRMSE) Root mean squared log error (RMSLE) Mean absolute percentage error (MAPE) For the following sections, y is the true value, y-hat is the predicted value, n is the number of test instances, and i goes from 1 to n. Also, all metrics are evaluated on an unseen test set. Mean Absolute Error Mean absolute error is a very intuitive and, therefore, popular metric. It is simply the average distance between predicted and true values. To avoid errors canceling one another out, we take the absolute of every error that we compute. The best model is usually the one with the lowest MAE. However, there are a few features to consider when picking MAE as your metric. Although it is straightforward to interpret, MAE has a few disadvantages. For example, it doesn’t tell you whether your model tends to over-estimate or under-estimate since any direction information is destroyed by taking the absolute value. Also, the metric can be insensitive to large outliers. Take a look at the example below. On the left, the model is off by a bit here and there. However, on the right, the model misses the marks at the tail end by a much wider margin while being perfect at the beginning and in the middle. Yet, the MAEs for both are the same. If you decide to use this metric, it’s a good idea to plot the errors to see any outliers like case 2. All in all, if you want a metric that penalizes large errors, you’ll have better luck somewhere else. Mean Squared Error This brings me to mean squared errors. Like MAE, we’re destroying directional information when we square every computed error. The MSE is also always larger than or equal to 0. However, we are now able to distinguish between the two models above. Interestingly, MSE is related to the infamous bias-variance trade-off. It can be shown that the expected test MSE for a given test point can be written as [1]: where the 0 subscript is the index of a test data point, and ϵ is the irreducible or noise in the data. Variance refers to the amount by which y-hat changes when we change the training set. Usually, more flexible methods have a higher degree of variance. Variance is also dependent on how much data we have. The larger the training dataset, the lower the variance. Therefore, we can interpret the MSE for a given test data point in terms of the model’s bias and random noise in the limit of huge data. Bias occurs when we try to estimate a complicated relationship between predictors and targets with something simpler. For example, we often assume that x and y have linear or polynomial relationships because we know the form of these equations, which reduces the problem to a few parameters we can estimate. In reality, x and y might not have such a relationship. Finally, one major disadvantage to MSE is that the units of y are squared, which means it’s easy to misinterpret the final results. Roor Mean Squared Error How about we take the square root? RMSE and MSE are very similar, except that RMSE is more convenient because it has the same unit as whatever y has. However, have you ever tried to transform your targets to see if you get a better fit? Like taking the log of y, for example. Then you compute the RMSE of both approaches and see that one is higher than the other. This is not a fair comparison because the two values have different units. A way to get around this is to divide by some property of y to get a unit-less metric called normalized root mean squared error. Normalized Root Mean Squared Error I don’t have any equations here because there are many things you can divide RMS with. Some common choices are the mean of y, the difference between the max and min of y, standard deviation, and interquartile range. When to choose what is a subtle business, and [2] can provide a much more in-depth explanation if you’re interested. Root Mean Squared Log Error I think this metric was introduced by Kaggle a few years back [3]. This metric should be used when you want to add direction to how errors are penalized. In this case, we’re telling the metric to penalize underestimation more than overestimation. For example, suppose y=1. If our model gives y-hat = 0, then the error is [log(1/2)]² = 0.09. However, if the model gives y-hat=2, then the error is [log(3/2)]² = 0.03. If we used MSE, then our error would be 1 either way. Also, the metric takes into the relative scale of the true and predicted values. For example, if y=9 and y-hat=99, then the error is 1. If, on the other hand, y=99 and y-hat=999, then the error is still 1. So, this metric is handy when the range of your target values is high, and you don’t want to penalize large errors when both the predicted and true values are large. However, I haven’t been able to find a discussion on a few pitfalls to keep in mind here. If you look at the log(x) graph, you’ll see that x cannot be equal to or smaller than 0. This means that the argument inside our logarithm (y-hat+1 / y+1) can’t be equal to or smaller than 0. Then, there’s the added constraint that y can’t be -1. Otherwise, the whole thing blows up. Mean Absolute Percentage Error MAPE looks like a cousin of MAE, with the added benefit that it’s unitless. However, introducing division also comes with some major drawbacks. For example, with very small y_i or y_i = 0, MAPE can blow up or not be computable at all. It also inherits a problem from MAE, whether there isn’t an upper bound for how large each error can be (even though the name says ‘percentage’). Wrap-Up I hope you liked the list and learned something new. What’s next is just my opinion. I often see the mention of ‘interpretability’ when I read up on these metrics. I’m not exactly sure what ‘interpretability’ means or whether there’s even a consensus on it. My explanations of and how I understand these metrics are often based on how they behave in certain situations. All in all, how much should we emphasize the ‘interpretability’ of a metric when there are other metrics that behave more like the way we want but is harder to interpret? Stay Connected I like to write about data science & science. If you like this post, follow me on Medium and/or join my email list. See you next post! ? Sources [1] Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani. Introduction to Statistical Learning: With Applications in R (2013) [2] Otto, S.A. (2019, Jan.,7). How to normalize the RMSE [Blog post]. https://www.marinedatascience.co/blog/2019/01/07/normalizing-the-rmse/ [3] https://www.kaggle.com/carlolepelaars/understanding-the-metric-rmsle [4] https://en.wikipedia.org/wiki/Mean_absolute_percentage_error

2026-08-09 李宇轩 人工智能 英-中
From Inside a Hedge Fund: Revelations From Insiders

Hedge funds are the most off-limits financial institutions doing their best to keep the ways of making money off the competitors' radar. Generally, what happens within a fund, stays there. Former and current fund managers, traders, and analysts protect their portfolios and prevent data leaks. Some bits and pieces of insider information about the inner workings of hedge funds can usually be found on imageboards, as well as deep within Reddit and Quora. Here we have collected some interesting facts about what is happening in hedge funds based on open sources. “Gods and Slaves”: Ordinary Fund Employees Have 100% Risk, 5% Profit Traders and analysts working for funds should often make risky, non-obvious, and sometimes seemingly absurd decisions that defy explanation, especially when trying to justify them to clients. This was noted by Reddit user dascarescu, who provided moderators proofs of working in a hedge fund management company. ...You need to invest a lot in hand holding and explaining to people you know what you're doing - especially when you take on a contrarian trade (like the Greek REITS we are buying) that everyone thinks is insane...And especially if it drops after you buy it, since it's very hard to time the bottom on these things - you need to make sure you are right, and have the stamina to hold the position until it goes to a good sales price. There are no guarantees for small fund employees whose income directly depends on the performance of the entire fund. A good year can easily be followed by a loss-making one slashing incomes of even successful employees who have fulfilled their plans. According to Laurent Bernut, Former Analyst at Fidelity and Ward Ferry Management: ...HF eat what they kill. There is no guarantee there, no safety net. ...In small hedge funds, there are only 2 types of employees: gods, the partners and slaves the rest. Regular employees have 100% of the risk and 5% of the upside. In the 2nd HF, after the 800K bonus, we had a down year. I had a spectacular one but got paid a pittance. The principal kept his 2% of the 2–20 for himself. Retired Investors are Unable to Make Profit, While Managers Suffer from Burnout Caused by Broken Promises Working for a hedge fund is definitely not suitable for professionals who evaluate their work in terms of ethics, public duty, and morality. Phil Lord, Managing Partner at Vezno Capital, says that most investors bring their retirement savings bringing in another stress factor for decision-makers: Most accredited investors aren’t jetsetting billionaires. They’re regular people who need the 1–2M they saved up to retire. They worked their whole lives for that money, sacrificed time with their kids and friends. Lose the money, and you ruin their lives. It is no secret that the hedge fund performance has decreased significantly after the crisis of 2008, and the income of investors has fallen as well. According to Tom Groves, who has marked his Quora account as a “hedge fund partner,” it is easy to start hating what you do due to the inability to achieve the desired return for an investor: I actually retired because I hated it. I hated the feeling of not being able to give investors the returns they wanted… ...I hated most of all the fact that I felt like I could do "smart" things and guarantee failure, or take "dumb" risks that were bad for virtually everyone else (my partners, my investors, society in general). Tom also notes that working in a hedge fund is often accompanied by a toxic atmosphere and aggressive colleagues. This is why many get disappointed and quit. Tom notes that after leaving the hedge fund, he took up blogging and painting. Funds Cut Traders in Favor of Quantum Tools Fund managers can lay off traders if robots prove their worth in choosing stocks. As noted by an anonymous Reddit user, despite the fact that some fund partners do not fully trust quantum methods and question algorithmic trading, their fund may become 100% quantum in the future: I think quant is going to continue to dominate and if you have the desire and ability would encourage you to pursue a PhD in a math, science or comp sci field. Cameron E. Wild, former Vice President of Millennium Management with 17 years of trading experience, sums up career difficulties faced by traders leaving a hedge fund. After 8 years in the industry and 6 years as a hedge fund trader I couldn’t get another job. Though I didn’t try that hard since I was sick of it all (burnout). Technology is a massive destroyer of trading jobs. Cameron also confirmed that some funds set a goal to rid the trading process of the human factor within five years. Ready-made Software is Gradually Replacing Custom Applications and Excel Hedge funds still use custom applications (for example, the ones developed in C++) created for specific tasks. According to Scott Gosnell, Excel functionality remains ubiquitous. I have yet to see a firm that doesn’t take Excel out for a spin every day for basic modeling… Sometimes, the fancy prop models run on whatever software it is that the most junior analysts know how to run. Nevertheless, fund managers are currently opting for the market of ready-made software solutions allowing them to combine the experience of a development team with their own resources. For example, MetaTrader 5 for hedge funds works in conjunction with the powerful MQL5 algorithmic trading community and MQL5 Cloud Network resources allowing users to speed up testing strategies from a few days to 15 minutes. Richard Matthews, Ph.D. Computer Science and Engineering, notes that buying ready-made software enables fast and flexible fund structure deployment, as well as reduces the budget for creating an IT team: …Buying software from various quantitative trading software firms (like Backstop, Dynamo, ect) has the advantages of being fast to deploy because it's a ready made package and doesn't require extensive in house testing. Building software takes more time, more effort, and frequently the hiring of a team of quant developers.

2026-08-09 李宇轩 人工智能 英-中
Still Using the OS Module in Python? This Alternative is Remarkably Better

Still Using the OS Module in Python? This Alternative is Remarkably Better Python’s OS module is a nightmare for managing files and folders. You should try Pathlib. File and folder management with Python’s os module is a nightmare. Yet, it’s an essential part of every data science workflow. Saving reports, reading configuration files, you name it — there’s no way around it. Picture this — you spend weeks building an API around your model, and it works flawlessly, at least on your machine. Once deployed, it’s a whole different story. Your API fails in unexpected places or even won’t run, as absolute paths you’ve hardcoded simply don’t exist. There’s a no-brainer solution. The pathlib library comes by default with Python 3.4 and above. It’s by far the most humane way to work with files, folder, and their connection in your apps. The best thing is — today you’ll learn all about it. I’m not saying that the os module serves no purpose, just that pathlib is far superior in file and folder management. To start, create a new Python file and import the pathlib library: You’ll need to be inside a .py file for some functionalities to work. For example, you won’t have access to __file__ property in Jupyter Notebooks. Everything else should work perfectly. Here’s what you’ll learn today: Get a path to the current Python file Every so often, you need an absolute path to your working directory with the included Python file name. You can obtain it quickly with pathlib. Keep in mind, this command won’t work in Jupyter Notebooks, as you can’t access the __file__ property there. Skip this section if you’re a Notebook user. Anyhow, here’s how to get an absolute path + a file name of the Python script: Here’s how it looks on my machine: Easy, right? Right. There’s an easier solution if you don’t need the file name. Get a path to the current working directory This one is equivalent to executing pwd in the Unix shell. It will return a path to the directory you’re currently in, or where the running script is located. Here’s how to use it: And here’s what it prints on my machine: No file name, as you can see. But what if you need to access a file in a parent folder? That would be Desktop in my case. Let’s cover that next. Get a first parent folder path This one is easy. You only need to access the parent property of a current working directory. Here’s how it’s done: Here’s what it prints on my machine: Great! But what if one parent folder isn’t enough? Let’s see what your options are. Get an Nth parent folder path You have options. The first one is to call access the parent property multiple times like a crazy person. Here’s an example: The easier option is to access the parents property array and index it. For instance, here’s how you’d get a path to the second parent folder path: Here are the results: Array indexing starts at 0, so accessing parents[1] gets you to the second parent folder. You now have enough knowledge to start joining paths. Let’s see how next. Join paths Let’s say that a folder with sales reports is located two directories above your current location, and the report is called summer-sales.csv. Is there a way to access it with an absolute path? Of course there is. You already know how to access the Nth parent folder path. You’ll extend that functionality by calling joinpath() and providing the report name as an argument: Here’s what it prints on my machine: The joinpath() function is probably the one I use the most. It’s super useful. Create a directory if it doesn’t exist If I had a dollar every time something in production failed because I forgot to create a directory… It’s a pretty common mistake, and pathlib allows you to get around it without too much hassle. Let’s say you want to store sales reports in a reports folder located in your current working directory. You’ll have to create that folder before you can store files in it. You should create it only if it doesn’t exist. In a nutshell — use mkdir() to create a folder and exists() to check if a folder already exists. Here’s the complete code snippet: Executing the above code will, you’ve guessed it, create a reports folder: Neat. Let’s see how you can create files in that folder next. Create files You’d typically save reports through some third-party libraries. Nevertheless, you can also use pathlib to create empty files of any type. Here’s how to create both a CSV and TXT file inside the reports folder: The exist_ok=True parameter tells Python to overwrite a file if it already exists. Let’s see if the files were created: Works like a charm. Check if the path is a folder If you want to check if a given path is a folder, look no further than the is_dir() function. It returns a boolean. The following example uses the mentioned function both on a folder and on a file: Here’s what you should see printed: And that’s all there is to it! Check if the path is a file Similarly to the previous example, you can use the is_file() function to check if a given path results in a file. The example below uses it both on a folder and on a file: As you would imagine, you’ll get the exact opposite results this time: Let’s explore a few more useful functions before calling it a day. Get the name of the file You can access the name property if you need to extract a file name from an absolute path. Here’s a simple and not so useful example. It prints the file name of our summer-sales.csv file: Here’s what you should see in the console: Not much to it. Get the file extension Sometimes all you need is a file extension. Maybe you want to process different file types differently, and you don’t care too much about the file name. The suffix property has you covered. Here’s how you can get the file extension from the same summer-sales.csv file: Here’s what you should see printed out: And finally, let’s cover iteration. Iterate over files in a folder Let’s say you have a bunch of CSV reports in a single directory, and want to process them one by one. The iterdir() function is all you need. The entire process couldn’t be any simpler: Here’s what it prints on my machine: And that does it for today. You now have everything needed to never cause a stupid production mistake again. What are your thoughts on pathlib? Is it your favourite file and folder management library, or are you a fan of something else? Let me know in the comment section below. Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you. Stay connected Follow me on Medium for more stories like this Sign up for my newsletter Connect on LinkedIn

2026-08-07 李宇轩 人工智能 英-中

本校翻译实践排行榜

梁雯
211368字
范荣
146854字
孟焕蕊
122789字
4
王铅铅
17717字
5
李宇轩
15858字

400所高校都在用的翻译教学平台

试译宝所属母公司