Degrees of freedom (statistics)

For other uses, seeDegrees of freedom.

Instatistics,the number ofdegrees of freedomis the number of values in the final calculation of astatisticthat are free to vary.

Estimates ofstatistical parameterscan be based upon different amounts of information or data. The number of independent pieces of information that go into the estimate of a parameter is called the degrees of freedom. In general, the degrees of freedom of an estimate of a parameter are equal to the number of independentscoresthat go into the estimate minus the number of parameters used as intermediate steps in the estimation of the parameter itself. For example, if thevarianceis to be estimated from a random sample of${\textstyle N}$independent scores, then the degrees of freedom is equal to the number of independent scores (N) minus the number of parameters estimated as intermediate steps (one, namely, the sample mean) and is therefore equal to${\textstyle N-1}$.

Mathematically, degrees of freedom is the number ofdimensionsof the domain of arandom vector,or essentially the number of "free" components (how many components need to be known before the vector is fully determined).

The term is most often used in the context oflinear models(linear regression,analysis of variance), where certain random vectors are constrained to lie inlinear subspaces,and the number of degrees of freedom is the dimension of thesubspace.The degrees of freedom are also commonly associated with the squared lengths (or "sum of squares" of the coordinates) of such vectors, and the parameters ofchi-squaredand other distributions that arise in associated statistical testing problems.

While introductory textbooks may introduce degrees of freedom as distribution parameters or through hypothesis testing, it is the underlying geometry that defines degrees of freedom, and is critical to a proper understanding of the concept.

History

Although the basic concept of degrees of freedom was recognized as early as 1821 in the work of German astronomer and mathematicianCarl Friedrich Gauss,its modern definition and usage was first elaborated by English statisticianWilliam Sealy Gossetin his 1908Biometrikaarticle "The Probable Error of a Mean", published under the pen name "Student".While Gosset did not actually use the term 'degrees of freedom', he explained the concept in the course of developing what became known asStudent's t-distribution.The term itself was popularized by English statistician and biologistRonald Fisher,beginning with his 1922 work on chi squares.

Notation

In equations, the typical symbol for degrees of freedom isν(lowercaseGreek letter nu). In text and tables, the abbreviation "d.f." is commonly used.R. A. Fisherusednto symbolize degrees of freedom but modern usage typically reservesnfor sample size. When reporting the results ofstatistical tests,the degrees of freedom are typically noted beside the test statistic as either subscript or in parentheses.

Of random vectors

Geometrically, the degrees of freedom can be interpreted as the dimension of certain vector subspaces. As a starting point, suppose that we have a sample of independent normally distributed observations,

$X_{1},\dots,X_{n}.\,$

This can be represented as ann-dimensionalrandom vector:

${\begin{pmatrix}X_{1}\\\vdots \\X_{n}\end{pmatrix}}.$

Since this random vector can lie anywhere inn-dimensional space, it hasndegrees of freedom.

Now, let${\bar {X}}$be thesample mean.The random vector can be decomposed as the sum of the sample mean plus a vector of residuals:

${\begin{pmatrix}X_{1}\\\vdots \\X_{n}\end{pmatrix}}={\bar {X}}{\begin{pmatrix}1\\\vdots \\1\end{pmatrix}}+{\begin{pmatrix}X_{1}-{\bar {X}}\\\vdots \\X_{n}-{\bar {X}}\end{pmatrix}}.$

The first vector on the right-hand side is constrained to be a multiple of the vector of 1's, and the only free quantity is${\bar {X}}$.It therefore has 1 degree of freedom.

The second vector is constrained by the relation${\textstyle \sum _{i=1}^{n}(X_{i}-{\bar {X}})=0}$.The firstn− 1 components of this vector can be anything. However, once you know the firstn− 1 components, the constraint tells you the value of thenth component. Therefore, this vector hasn− 1 degrees of freedom.

Mathematically, the first vector is theoblique projectionof the data vector onto thesubspacespannedby the vector of 1's. The 1 degree of freedom is the dimension of this subspace. The second residual vector is the least-squares projection onto the (n− 1)-dimensionalorthogonal complementof this subspace, and hasn− 1 degrees of freedom.

In statistical testing applications, often one is not directly interested in the component vectors, but rather in their squared lengths. In the example above, theresidual sum-of-squaresis

$\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}={\begin{Vmatrix}X_{1}-{\bar {X}}\\\vdots \\X_{n}-{\bar {X}}\end{Vmatrix}}^{2}.$

If the data points$X_{i}$are normally distributed with mean 0 and variance$\sigma ^{2}$,then the residual sum of squares has a scaledchi-squared distribution(scaled by the factor$\sigma ^{2}$), withn− 1 degrees of freedom. The degrees-of-freedom, here a parameter of the distribution, can still be interpreted as the dimension of an underlying vector subspace.

Likewise, the one-samplet-teststatistic,

${\frac {{\sqrt {n}}({\bar {X}}-\mu _{0})}{\sqrt {\sum \limits _{i=1}^{n}(X_{i}-{\bar {X}})^{2}/(n-1)}}}$

follows aStudent's tdistribution withn− 1 degrees of freedom when the hypothesized mean$\mu _{0}$is correct. Again, the degrees-of-freedom arises from the residual vector in the denominator.

In structural equation models

When the results of structural equation models (SEM) are presented, they generally include one or more indices of overall model fit, the most common of which is aχ²statistic. This forms the basis for other indices that are commonly reported. Although it is these other statistics that are most commonly interpreted, thedegrees of freedomof theχ²are essential to understanding model fit as well as the nature of the model itself.

Degrees of freedom in SEM are computed as a difference between the number of unique pieces of information that are used as input into the analysis, sometimes called knowns, and the number of parameters that are uniquely estimated, sometimes called unknowns. For example, in a one-factor confirmatory factor analysis with 4 items, there are 10 knowns (the six unique covariances among the four items and the four item variances) and 8 unknowns (4 factor loadings and 4 error variances) for 2 degrees of freedom. Degrees of freedom are important to the understanding of model fit if for no other reason than that, all else being equal, the fewer degrees of freedom, the better indices such asχ²will be.

It has been shown that degrees of freedom can be used by readers of papers that contain SEMs to determine if the authors of those papers are in fact reporting the correct model fit statistics. In the organizational sciences, for example, nearly half of papers published in top journals report degrees of freedom that are inconsistent with the models described in those papers, leaving the reader to wonder which models were actually tested.

Of residuals

Further information:Residuals (statistics)

A common way to think of degrees of freedom is as the number of independent pieces of information available to estimate another piece of information. More concretely, the number of degrees of freedom is the number of independent observations in a sample of data that are available to estimate a parameter of the population from which that sample is drawn. For example, if we have two observations, when calculating the mean we have two independent observations; however, when calculating the variance, we have only one independent observation, since the two observations are equally distant from the sample mean.

In fitting statistical models to data, the vectors of residuals are constrained to lie in a space of smaller dimension than the number of components in the vector. That smaller dimension is the number ofdegrees of freedom for error,also calledresidual degrees of freedom.

Example

Perhaps the simplest example is this. Suppose

$X_{1},\dots,X_{n}$

arerandom variableseach withexpected valueμ,and let

${\overline {X}}_{n}={\frac {X_{1}+\cdots +X_{n}}{n}}$

be the "sample mean." Then the quantities

$X_{i}-{\overline {X}}_{n}$

are residuals that may be consideredestimatesof theerrorsX_i−μ.The sum of the residuals (unlike the sum of the errors) is necessarily 0. If one knows the values of anyn− 1 of the residuals, one can thus find the last one. That means they are constrained to lie in a space of dimensionn− 1. One says that there aren− 1 degrees of freedom for errors.

An example which is only slightly less simple is that ofleast squaresestimation ofaandbin the model

$Y_{i}=a+bx_{i}+e_{i}{\text{ for }}i=1,\dots,n$

wherex_iis given, but e_iand henceY_iare random. Let${\widehat {a}}$and${\widehat {b}}$be the least-squares estimates ofaandb.Then the residuals

${\widehat {e}}_{i}=y_{i}-({\widehat {a}}+{\widehat {b}}x_{i})$

are constrained to lie within the space defined by the two equations

${\widehat {e}}_{1}+\cdots +{\widehat {e}}_{n}=0,$

$x_{1}{\widehat {e}}_{1}+\cdots +x_{n}{\widehat {e}}_{n}=0.$

One says that there aren− 2 degrees of freedom for error.

Notationally, the capital letterYis used in specifying the model, while lower-caseyin the definition of the residuals; that is because the former are hypothesized random variables and the latter are actual data.

We can generalise this to multiple regression involvingpparameters and covariates (e.g.p− 1 predictors and one mean (=intercept in the regression)), in which case the cost indegrees of freedom of the fitisp,leavingn - pdegrees of freedom for errors

In linear models

The demonstration of thetand chi-squared distributions for one-sample problems above is the simplest example where degrees-of-freedom arise. However, similar geometry and vector decompositions underlie much of the theory oflinear models,includinglinear regressionandanalysis of variance.An explicit example based on comparison of three means is presented here; the geometry of linear models is discussed in more complete detail by Christensen (2002).

Suppose independent observations are made for three populations,$X_{1},\ldots,X_{n}$,$Y_{1},\ldots,Y_{n}$and$Z_{1},\ldots,Z_{n}$.The restriction to three groups and equal sample sizes simplifies notation, but the ideas are easily generalized.

The observations can be decomposed as

${\begin{aligned}X_{i}&={\bar {M}}+({\bar {X}}-{\bar {M}})+(X_{i}-{\bar {X}})\\Y_{i}&={\bar {M}}+({\bar {Y}}-{\bar {M}})+(Y_{i}-{\bar {Y}})\\Z_{i}&={\bar {M}}+({\bar {Z}}-{\bar {M}})+(Z_{i}-{\bar {Z}})\end{aligned}}$

where${\bar {X}},{\bar {Y}},{\bar {Z}}$are the means of the individual samples, and ${\bar {M}}=({\bar {X}}+{\bar {Y}}+{\bar {Z}})/3$is the mean of all 3nobservations. In vector notation this decomposition can be written as

${\begin{pmatrix}X_{1}\\\vdots \\X_{n}\\Y_{1}\\\vdots \\Y_{n}\\Z_{1}\\\vdots \\Z_{n}\end{pmatrix}}={\bar {M}}{\begin{pmatrix}1\\\vdots \\1\\1\\\vdots \\1\\1\\\vdots \\1\end{pmatrix}}+{\begin{pmatrix}{\bar {X}}-{\bar {M}}\\\vdots \\{\bar {X}}-{\bar {M}}\\{\bar {Y}}-{\bar {M}}\\\vdots \\{\bar {Y}}-{\bar {M}}\\{\bar {Z}}-{\bar {M}}\\\vdots \\{\bar {Z}}-{\bar {M}}\end{pmatrix}}+{\begin{pmatrix}X_{1}-{\bar {X}}\\\vdots \\X_{n}-{\bar {X}}\\Y_{1}-{\bar {Y}}\\\vdots \\Y_{n}-{\bar {Y}}\\Z_{1}-{\bar {Z}}\\\vdots \\Z_{n}-{\bar {Z}}\end{pmatrix}}.$

The observation vector, on the left-hand side, has 3ndegrees of freedom. On the right-hand side, the first vector has one degree of freedom (or dimension) for the overall mean. The second vector depends on three random variables,${\bar {X}}-{\bar {M}}$,${\bar {Y}}-{\bar {M}}$and${\overline {Z}}-{\overline {M}}$.However, these must sum to 0 and so are constrained; the vector therefore must lie in a 2-dimensional subspace, and has 2 degrees of freedom. The remaining 3n− 3 degrees of freedom are in the residual vector (made up ofn− 1 degrees of freedom within each of the populations).

In analysis of variance (ANOVA)

In statistical testing problems, one usually is not interested in the component vectors themselves, but rather in their squared lengths, or Sum of Squares. The degrees of freedom associated with a sum-of-squares is the degrees-of-freedom of the corresponding component vectors.

The three-population example above is an example ofone-way Analysis of Variance.The model, or treatment, sum-of-squares is the squared length of the second vector,

${\text{SST}}=n({\bar {X}}-{\bar {M}})^{2}+n({\bar {Y}}-{\bar {M}})^{2}+n({\bar {Z}}-{\bar {M}})^{2}$

with 2 degrees of freedom. The residual, or error, sum-of-squares is

${\text{SSE}}=\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}+\sum _{i=1}^{n}(Y_{i}-{\bar {Y}})^{2}+\sum _{i=1}^{n}(Z_{i}-{\bar {Z}})^{2}$

with 3(n−1) degrees of freedom. Of course, introductory books on ANOVA usually state formulae without showing the vectors, but it is this underlying geometry that gives rise to SS formulae, and shows how to unambiguously determine the degrees of freedom in any given situation.

Under the null hypothesis of no difference between population means (and assuming that standard ANOVA regularity assumptions are satisfied) the sums of squares have scaled chi-squared distributions, with the corresponding degrees of freedom. The F-test statistic is the ratio, after scaling by the degrees of freedom. If there is no difference between population means this ratio follows anF-distributionwith 2 and 3n− 3 degrees of freedom.

In some complicated settings, such as unbalancedsplit-plotdesigns, the sums-of-squares no longer have scaled chi-squared distributions. Comparison of sum-of-squares with degrees-of-freedom is no longer meaningful, and software may report certain fractional 'degrees of freedom' in these cases. Such numbers have no genuine degrees-of-freedom interpretation, but are simply providing anapproximatechi-squared distribution for the corresponding sum-of-squares. The details of such approximations are beyond the scope of this page.

In probability distributions

Several commonly encountered statistical distributions (Student'st,chi-squared,F) have parameters that are commonly referred to asdegrees of freedom.This terminology simply reflects that in many applications where these distributions occur, the parameter corresponds to the degrees of freedom of an underlying random vector, as in the preceding ANOVA example. Another simple example is: if$X_{i};i=1,\ldots,n$are independent normal$(\mu,\sigma ^{2})$random variables, the statistic

${\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{\sigma ^{2}}}$

follows a chi-squared distribution withn− 1 degrees of freedom. Here, the degrees of freedom arises from the residual sum-of-squares in the numerator, and in turn then− 1 degrees of freedom of the underlying residual vector$\{X_{i}-{\bar {X}}\}$.

In the application of these distributions to linear models, the degrees of freedom parameters can take onlyintegervalues. The underlying families of distributions allow fractional values for the degrees-of-freedom parameters, which can arise in more sophisticated uses. One set of examples is problems where chi-squared approximations based oneffective degrees of freedomare used. In other applications, such as modellingheavy-taileddata, a t orF-distribution may be used as an empirical model. In these cases, there is no particulardegrees of freedominterpretation to the distribution parameters, even though the terminology may continue to be used.

In non-standard regression

Many non-standard regression methods, includingregularized least squares(e.g.,ridge regression),linear smoothers,smoothing splines,andsemiparametric regression,are not based onordinary least squaresprojections, but rather onregularized(generalizedand/or penalized) least-squares, and so degrees of freedom defined in terms of dimensionality is generally not useful for these procedures. However, these procedures are still linear in the observations, and the fitted values of the regression can be expressed in the form

${\hat {y}}=Hy,$

where${\hat {y}}$is the vector of fitted values at each of the original covariate values from the fitted model,yis the original vector of responses, andHis thehat matrixor, more generally, smoother matrix.

For statistical inference, sums-of-squares can still be formed: the model sum-of-squares is$\|Hy\|^{2}$;the residual sum-of-squares is$\|y-Hy\|^{2}$.However, becauseHdoes not correspond to an ordinary least-squares fit (i.e. is not an orthogonal projection), these sums-of-squares no longer have (scaled, non-central) chi-squared distributions, and dimensionally defined degrees-of-freedom are not useful.

Theeffective degrees of freedomof the fit can be defined in various ways to implementgoodness-of-fit tests,cross-validation,and otherstatistical inferenceprocedures. Here one can distinguish betweenregression effective degrees of freedomandresidual effective degrees of freedom.

Regression effective degrees of freedom

For the regression effective degrees of freedom, appropriate definitions can include thetraceof the hat matrix,tr(H), the trace of the quadratic form of the hat matrix, tr(H'H), the form tr(2H–HH'), or theSatterthwaite approximation,tr(H'H)²/tr(H'HH'H). In the case of linear regression, the hat matrixHisX(X'X)⁻¹X ',and all these definitions reduce to the usual degrees of freedom. Notice that

$\operatorname {tr} (H)=\sum _{i}h_{ii}=\sum _{i}{\frac {\partial {\hat {y}}_{i}}{\partial y_{i}}},$

the regression (not residual) degrees of freedom in linear models are "the sum of the sensitivities of the fitted values with respect to the observed response values",i.e. the sum ofleverage scores.

One way to help to conceptualize this is to consider a simple smoothing matrix like aGaussian blur,used to mitigate data noise. In contrast to a simple linear or polynomial fit, computing the effective degrees of freedom of the smoothing function is not straightforward. In these cases, it is important to estimate the Degrees of Freedom permitted by the$H$matrix so that the residual degrees of freedom can then be used to estimate statistical tests such as$\chi ^{2}$.

Residual effective degrees of freedom

There are corresponding definitions of residual effective degrees-of-freedom (redf), withHreplaced byI−H.For example, if the goal is to estimate error variance, the redf would be defined as tr((I−H)'(I−H)), and the unbiased estimate is (with${\hat {r}}=y-Hy$),

${\hat {\sigma }}^{2}={\frac {\|{\hat {r}}\|^{2}}{\operatorname {tr} \left((I-H)'(I-H)\right)}},$

or:

${\hat {\sigma }}^{2}={\frac {\|{\hat {r}}\|^{2}}{n-\operatorname {tr} (2H-HH')}}={\frac {\|{\hat {r}}\|^{2}}{n-2\operatorname {tr} (H)+\operatorname {tr} (HH')}}$

${\hat {\sigma }}^{2}\approx {\frac {\|{\hat {r}}\|^{2}}{n-1.25\operatorname {tr} (H)+0.5}}.$

The last approximation abovereduces the computational cost fromO(n²) to onlyO(n). In general the numerator would be the objective function being minimized; e.g., if the hat matrix includes an observation covariance matrix, Σ, then$\|{\hat {r}}\|^{2}$becomes${\hat {r}}'\Sigma ^{-1}{\hat {r}}$.

General

Note that unlike in the original case, non-integer degrees of freedom are allowed, though the value must usually still be constrained between 0 andn.

Consider, as an example, thek-nearest neighboursmoother, which is the average of theknearest measured values to the given point. Then, at each of thenmeasured points, the weight of the original value on the linear combination that makes up the predicted value is just 1/k.Thus, the trace of the hat matrix isn/k.Thus the smooth costsn/keffective degrees of freedom.

As another example, consider the existence of nearly duplicated observations. Naive application of classical formula,n−p,would lead to over-estimation of the residuals degree of freedom, as if each observation were independent. More realistically, though, the hat matrixH=X(X' Σ⁻¹X)⁻¹X 'Σ⁻¹would involve an observation covariance matrix Σ indicating the non-zero correlation among observations.

The more general formulation of effective degree of freedom would result in a more realistic estimate for, e.g., the error variance σ²,which in its turn scales the unknown parameters'a posterioristandard deviation; the degree of freedom will also affect the expansion factor necessary to produce anerror ellipsefor a givenconfidence level.

Other formulations

Similar concepts are theequivalent degrees of freedominnon-parametric regression,thedegree of freedom of signalin atmospheric studies,and thenon-integer degree of freedomin geodesy.

The residual sum-of-squares$\|y-Hy\|^{2}$has ageneralized chi-squared distribution,and the theory associated with this distributionprovides an alternative route to the answers provided above.

See also

• Bessel's correction
• Chi-squared per degree of freedom
• Pooled degrees of freedom
• Replication (statistics)
• Sample size
• Statistical model
• Variance

Further reading

• Bowers, David (1982).Statistics for Economists.London: Macmillan. pp. 175–178.ISBN0-333-30110-2.
• Eisenhauer, J. G. (2008)."Degrees of Freedom".Teaching Statistics.30(3): 75–78.doi:10.1111/j.1467-9639.2008.00324.x.S2CID121982952.
• Good, I. J. (1973). "What Are Degrees of Freedom?".The American Statistician.27(5): 227–228.doi:10.1080/00031305.1973.10479042.JSTOR3087407.
• Walker, H. W. (1940). "Degrees of Freedom".Journal of Educational Psychology.31(4): 253–269.doi:10.1037/h0054588.Transcription by C Olsen with errata

External links

• Yu, Chong-ho (1997)Illustrating degrees of freedom in terms of sample size and dimensionality
• Dallal, GE. (2003)Degrees of Freedom