Generalization error

Forsupervised learningapplications inmachine learningandstatistical learning theory,generalization error^[1](also known as theout-of-sample error^[2]or therisk) is a measure of how accurately an algorithm is able to predict outcome values for previously unseen data. Because learning algorithms are evaluated on finite samples, the evaluation of a learning algorithm may be sensitive tosampling error.As a result, measurements of prediction error on the current data may not provide much information about predictive ability on new data. Generalization error can be minimized by avoidingoverfittingin the learning algorithm. The performance of amachine learning algorithmis visualized by plots that show values ofestimatesof the generalization error through the learning process, which are calledlearning curves.

Definition

In a learning problem, the goal is to develop a function $f_{n}({\vec {x}})$ that predicts output values $y$ for each input datum ${\vec {x}}$ .The subscript $n$ indicates that the function $f_{n}$ is developed based on a data set of $n$ data points. Thegeneralization errororexpected lossorrisk $I[f]$ of a particular function $f$ over all possible values of ${\vec {x}}$ and $y$ is theexpected valueof theloss function $V(f)$ :^[1]

I[f]=\int _{X\times Y}V(f({\vec {x}}),y)\rho ({\vec {x}},y)d{\vec {x}}dy,

where $\rho ({\vec {x}},y)$ is the unknownjoint probability distributionfor ${\vec {x}}$ and $y$ .

Without knowing the joint probability distribution $\rho$ ,it is impossible to compute $I[f]$ .Instead, we can compute the error on sample data, which is calledempirical error(orempirical risk). Given $n$ data points, the empirical error of a candidate function $f$ is:

I_{n}[f]={\frac {1}{n}}\sum _{i=1}^{n}V(f({\vec {x}}_{i}),y_{i})

An algorithm is said to generalize if:

\lim _{n\rightarrow \infty }I[f]-I_{n}[f]=0

Of particular importance is thegeneralization error $I[f_{n}]$ of the data-dependent function $f_{n}$ that is found by a learning algorithm based on the sample. Again, for an unknown probability distribution, $I[f_{n}]$ cannot be computed. Instead, the aim of many problems in statistical learning theory is to bound or characterize the difference of the generalization error and the empirical error in probability:

P_{G}=P(I[f_{n}]-I_{n}[f_{n}]\leq \epsilon )\geq 1-\delta _{n}

That is, the goal is to characterize the probability $1-\delta _{n}$ that the generalization error is less than the empirical error plus some error bound $\epsilon$ (generally dependent on $\delta$ and $n$ ). For many types of algorithms, it has been shown that an algorithm has generalization bounds if it meets certainstabilitycriteria. Specifically, if an algorithm is symmetric (the order of inputs does not affect the result), has bounded loss and meets two stability conditions, it will generalize. The first stability condition,leave-one-out cross-validationstability, says that to be stable, the prediction error for each data point when leave-one-out cross validation is used must converge to zero as $n\rightarrow \infty$ .The second condition, expected-to-leave-one-out error stability (also known as hypothesis stability if operating in the $L_{1}$ norm) is met if the prediction on a left-out datapoint does not change when a single data point is removed from the training dataset.^[3]

These conditions can be formalized as:

Leave-one-out cross-validation Stability

An algorithm $L$ has $CVloo$ stability if for each $n$ ,there exists a $\beta _{CV}^{(n)}$ and $\delta _{CV}^{(n)}$ such that:

\forall i\in \{1,...,n\},\mathbb {P} _{S}\{|V(f_{S^{i}},z_{i})-V(f_{S},z_{i})|\leq \beta _{CV}^{(n)}\}\geq 1-\delta _{CV}^{(n)}

and $\beta _{CV}^{(n)}$ and $\delta _{CV}^{(n)}$ go to zero as $n$ goes to infinity.^[3]

Expected-leave-one-out error Stability

An algorithm $L$ has $Eloo_{err}$ stability if for each $n$ there exists a $\beta _{EL}^{m}$ and a $\delta _{EL}^{m}$ such that:

\forall i\in \{1,...,n\},\mathbb {P} _{S}\left\{\left|I[f_{S}]-{\frac {1}{n}}\sum _{i=1}^{N}V\left(f_{S^{i}},z_{i}\right)\right|\leq \beta _{EL}^{(n)}\right\}\geq 1-\delta _{EL}^{(n)}

with $\beta _{EL}^{(n)}$ and $\delta _{EL}^{(n)}$ going to zero for $n\rightarrow \infty$ .

For leave-one-out stability in the $L_{1}$ norm, this is the same as hypothesis stability:

\mathbb {E} _{S,z}[|V(f_{S},z)-V(f_{S^{i}},z)|]\leq \beta _{H}^{(n)}

with $\beta _{H}^{(n)}$ going to zero as $n$ goes to infinity.^[3]

Algorithms with proven stability

A number of algorithms have been proven to be stable and as a result have bounds on their generalization error. A list of these algorithms and the papers that proved stability is availablehere.

Relation to overfitting

This figure illustrates the relationship between overfitting and the generalization errorI[*f_n*] -*I_S*[*f_n*]. Data points were generated from the relationshipy=xwith white noise added to theyvalues. In the left column, a set of training points is shown in blue. A seventh order polynomial function was fit to the training data. In the right column, the function is tested on data sampled from the underlying joint probability distribution ofxandy.In the top row, the function is fit on a sample dataset of 10 datapoints. In the bottom row, the function is fit on a sample dataset of 100 datapoints. As we can see, for small sample sizes and complex functions, the error on the training set is small but error on the underlying distribution of data is large and we have overfit the data. As a result, generalization error is large. As the number of sample points increases, the prediction error on training and test data converges and generalization error goes to 0.

The concepts of generalization error and overfitting are closely related. Overfitting occurs when the learned function $f_{S}$ becomes sensitive to the noise in the sample. As a result, the function will perform well on the training set but not perform well on other data from the joint probability distribution of $x$ and $y$ .Thus, the more overfitting occurs, the larger the generalization error.

The amount of overfitting can be tested usingcross-validationmethods, that split the sample into simulated training samples and testing samples. The model is then trained on a training sample and evaluated on the testing sample. The testing sample is previously unseen by the algorithm and so represents a random sample from the joint probability distribution of $x$ and $y$ .This test sample allows us to approximate the expected error and as a result approximate a particular form of the generalization error.

Many algorithms exist to prevent overfitting. The minimization algorithm can penalize more complex functions (known as Tikhonovregularization), or the hypothesis space can be constrained, either explicitly in the form of the functions or by adding constraints to the minimization function (Ivanov regularization).

The approach to finding a function that does not overfit is at odds with the goal of finding a function that is sufficiently complex to capture the particular characteristics of the data. This is known as thebias–variance tradeoff.Keeping a function simple to avoid overfitting may introduce a bias in the resulting predictions, while allowing it to be more complex leads to overfitting and a higher variance in the predictions. It is impossible to minimize both simultaneously.

References

^^a ^bMohri, M., Rostamizadeh A., Talwakar A., (2018)Foundations of Machine learning,2nd ed., Boston: MIT Press
^Y S. Abu-Mostafa, M.Magdon-Ismail, and H.-T. Lin (2012) Learning from Data, AMLBook Press.ISBN 978-1600490064
^^a ^b ^cMukherjee, S.; Niyogi, P.; Poggio, T.; Rifkin., R. M. (2006)."Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization"(PDF).Adv. Comput. Math.25(1–3): 161–193.doi:10.1007/s10444-004-7634-z.S2CID 2240256.