Standard error

Thestandard error(SE)^[1]of astatistic(usually an estimate of aparameter) is thestandard deviationof itssampling distribution^[2]or an estimate of that standard deviation. If the statistic is the sample mean, it is called thestandard error of the mean(SEM).^[1]The standard error is a key ingredient in producingconfidence intervals.^[3]

Thesampling distributionof a mean is generated by repeated sampling from the same population and recording of the sample means obtained. This forms a distribution of different means, and this distribution has its ownmeanandvariance.Mathematically, the variance of the sampling mean distribution obtained is equal to the variance of the population divided by the sample size. This is because as the sample size increases, sample means cluster more closely around the population mean.

Therefore, the relationship between the standard error of the mean and the standard deviation is such that, for a given sample size, the standard error of the mean equals the standard deviation divided by thesquare rootof the sample size.^[1]In other words, the standard error of the mean is a measure of the dispersion of sample means around the population mean.

Inregression analysis,the term "standard error" refers either to the square root of thereduced chi-squared statisticor the standard error for a particular regression coefficient (as used in, say,confidence intervals).

Standard error of the sample mean

Exact value

Suppose a statistically independent sample of $n$ observations $x_{1},x_{2},\ldots,x_{n}$ is taken from astatistical populationwith astandard deviationof $\sigma$ .The mean value calculated from the sample, ${\bar {x}}$ ,will have an associatedstandard error on the mean, ${\sigma }_{\bar {x}}$ ,given by:^[1] ${\sigma }_{\bar {x}}={\frac {\sigma }{\sqrt {n}}}.$

Practically this tells us that when trying to estimate the value of a population mean, due to the factor $1/{\sqrt {n}}$ ,reducing the error on the estimate by a factor of two requires acquiring four times as many observations in the sample; reducing it by a factor of ten requires a hundred times as many observations.

Estimate

The standard deviation $\sigma$ of the population being sampled is seldom known. Therefore, the standard error of the mean is usually estimated by replacing $\sigma$ with thesample standard deviation $\sigma _{x}$ instead: ${\sigma }_{\bar {x}}\ \approx {\frac {\sigma _{x}}{\sqrt {n}}}.$

As this is only anestimatorfor the true "standard error", it is common to see other notations here such as: ${\widehat {\sigma }}_{\bar {x}}:={\frac {\sigma _{x}}{\sqrt {n}}}\qquad {\text{ or }}\qquad {s}_{\bar {x}}\:={\frac {s}{\sqrt {n}}}.$

A common source of confusion occurs when failing to distinguish clearly between:

the standard deviation of thepopulation( $\sigma$ ),
the standard deviation of thesample( $\sigma _{x}$ ),
the standard deviation of themeanitself ( $\sigma _{\bar {x}}$ ,which is the standard error), and
theestimatorof the standard deviation of the mean ( ${\widehat {\sigma }}_{\bar {x}}$ ,which is the most often calculated quantity, and is also often colloquially called thestandard error).

Accuracy of the estimator

When the sample size is small, using the standard deviation of the sample instead of the true standard deviation of the population will tend to systematically underestimate the population standard deviation, and therefore also the standard error. Withn= 2, the underestimate is about 25%, but forn= 6, the underestimate is only 5%. Gurland and Tripathi (1971) provide a correction and equation for this effect.^[4]Sokal and Rohlf (1981) give an equation of the correction factor for small samples ofn< 20.^[5]Seeunbiased estimation of standard deviationfor further discussion.

Derivation

The standard error on the mean may be derived from thevarianceof a sum of independent random variables,^[6]given thedefinitionof variance and somepropertiesthereof. If $x_{1},x_{2},\ldots,x_{n}$ is a sample of $n$ independent observations from a population with mean ${\bar {x}}$ and standard deviation $\sigma$ ,then we can define the total $T=(x_{1}+x_{2}+\cdots +x_{n})$ which due to theBienaymé formula,will have variance $\operatorname {Var} (T)={\big (}\operatorname {Var} (x_{1})+\operatorname {Var} (x_{2})+\cdots +\operatorname {Var} (x_{n}){\big )}=n\sigma ^{2}.$ where we've approximated the standard deviations, i.e., the uncertainties, of the measurements themselves with the best value for the standard deviation of the population. The mean of these measurements ${\bar {x}}$ is given by ${\bar {x}}=T/n.$

The variance of the mean is then $\operatorname {Var} ({\bar {x}})=\operatorname {Var} \left({\frac {T}{n}}\right)={\frac {1}{n^{2}}}\operatorname {Var} (T)={\frac {1}{n^{2}}}n\sigma ^{2}={\frac {\sigma ^{2}}{n}}.$

The standard error is, by definition, the standard deviation of ${\bar {x}}$ which is the square root of the variance: $\sigma _{\bar {x}}={\sqrt {\frac {\sigma ^{2}}{n}}}={\frac {\sigma }{\sqrt {n}}}.$

For correlated random variables the sample variance needs to be computed according to theMarkov chain central limit theorem.

Independent and identically distributed random variables with random sample size

There are cases when a sample is taken without knowing, in advance, how many observations will be acceptable according to some criterion. In such cases, the sample size $N$ is a random variable whose variation adds to the variation of $X$ such that, $\operatorname {Var} (T)=\operatorname {E} (N)\operatorname {Var} (X)+\operatorname {Var} (N){\big (}\operatorname {E} (X){\big )}^{2}$ ^[7] which follows from thelaw of total variance.

If $N$ has aPoisson distribution,then $\operatorname {E} (N)=\operatorname {Var} (N)$ with estimator $n=N$ .Hence the estimator of $\operatorname {Var} (T)$ becomes $nS_{X}^{2}+n{\bar {X}}^{2}$ ,leading the following formula for standard error: $\operatorname {Standard~Error} ({\bar {X}})={\sqrt {\frac {S_{X}^{2}+{\bar {X}}^{2}}{n}}}$ (since the standard deviation is the square root of the variance).

Student approximation whenσvalue is unknown

In many practical applications, the true value ofσis unknown. As a result, we need to use a distribution that takes into account that spread of possibleσ's. When the true underlying distribution is known to be Gaussian, although with unknown σ, then the resulting estimated distribution follows the Student t-distribution. The standard error is the standard deviation of the Student t-distribution. T-distributions are slightly different from Gaussian, and vary depending on the size of the sample. Small samples are somewhat more likely to underestimate the population standard deviation and have a mean that differs from the true population mean, and the Student t-distribution accounts for the probability of these events with somewhat heavier tails compared to a Gaussian. To estimate the standard error of a Student t-distribution it is sufficient to use the sample standard deviation "s" instead ofσ,and we could use this value to calculate confidence intervals.

Note:TheStudent's probability distributionis approximated well by the Gaussian distribution when the sample size is over 100. For such samples one can use the latter distribution, which is much simpler. Also, even though the 'true' distribution of the population is unknown, assuming normality of the sampling distribution makes sense for a reasonable sample size, and under certain sampling conditions, seeCLT.If these conditions are not met, then using aBootstrap distributionto estimate the Standard Error is often a good workaround, but it can be computationally intensive.

Assumptions and usage

An example of how $\operatorname {SE}$ is used is to make confidence intervals of the unknown population mean. If the sampling distribution isnormally distributed,the sample mean, the standard error, and thequantilesof the normal distribution can be used to calculate confidence intervals for the true population mean. The following expressions can be used to calculate the upper and lower 95% confidence limits, where ${\bar {x}}$ is equal to the sample mean, $\operatorname {SE}$ is equal to the standard error for the sample mean, and1.96is the approximate value of the 97.5percentilepoint of thenormal distribution:

Upper 95% limit = ${\bar {x}}+(\operatorname {SE} \times 1.96)$ ,and
Lower 95% limit = ${\bar {x}}-(\operatorname {SE} \times 1.96)$ .

In particular, the standard error of asample statistic(such assample mean) is the actual or estimated standard deviation of the sample mean in the process by which it was generated. In other words, it is the actual or estimated standard deviation of thesampling distributionof the sample statistic. The notation for standard error can be any one of SE, SEM (for standard error ofmeasurementormean), or S_E.

Standard errors provide simple measures of uncertainty in a value and are often used because:

in many cases, if the standard error of several individual quantities is known then the standard error of somefunctionof the quantities can be easily calculated;
when theprobability distributionof the value is known, it can be used to calculate an exactconfidence interval;
when the probability distribution is unknown,Chebyshev's or theVysochanskiï–Petunin inequalitiescan be used to calculate a conservative confidence interval; and
as thesample sizetends to infinity thecentral limit theoremguarantees that the sampling distribution of the mean is asymptoticallynormal.

Standard error of mean versus standard deviation

In scientific and technical literature, experimental data are often summarized either using the mean and standard deviation of the sample data or the mean with the standard error. This often leads to confusion about their interchangeability. However, the mean and standard deviation aredescriptive statistics,whereas the standard error of the mean is descriptive of the random sampling process. The standard deviation of the sample data is a description of the variation in measurements, while the standard error of the mean is a probabilistic statement about how the sample size will provide a better bound on estimates of the population mean, in light of the central limit theorem.^[8]

Put simply, thestandard errorof the sample mean is an estimate of how far the sample mean is likely to be from the population mean, whereas thestandard deviationof the sample is the degree to which individuals within the sample differ from the sample mean.^[9]If the population standard deviation is finite, the standard error of the mean of the sample will tend to zero with increasing sample size, because the estimate of the population mean will improve, while the standard deviation of the sample will tend to approximate the population standard deviation as the sample size increases.

Extensions

Finite population correction (FPC)

The formula given above for the standard error assumes that the population is infinite. Nonetheless, it is often used for finite populations when people are interested in measuring the process that created the existing finite population (this is called ananalytic study). Though the above formula is not exactly correct when the population is finite, the difference between the finite- and infinite-population versions will be small whensampling fractionis small (e.g. a small proportion of a finite population is studied). In this case people often do not correct for the finite population, essentially treating it as an "approximately infinite" population.

If one is interested in measuring an existing finite population that will not change over time, then it is necessary to adjust for the population size (called anenumerative study). When thesampling fraction(often termedf) is large (approximately at 5% or more) in anenumerative study,the estimate of the standard error must be corrected by multiplying by a ''finite population correction'' (a.k.a.:FPC):^[10] ^[11] $\operatorname {FPC} ={\sqrt {\frac {N-n}{N-1}}}$ which, for largeN: $\operatorname {FPC} \approx {\sqrt {1-{\frac {n}{N}}}}={\sqrt {1-f}}$ to account for the added precision gained by sampling close to a larger percentage of the population. The effect of the FPC is that the error becomes zero when the sample sizenis equal to the population sizeN.

This happens insurvey methodologywhen samplingwithout replacement.If sampling with replacement, then FPC does not come into play.

Correction for correlation in the sample

Expected error in the mean ofAfor a sample ofndata points with sample bias coefficientρ.The unbiasedstandard errorplots as theρ= 0 diagonal line with log-log slope −1⁄2.

If values of the measured quantityAare not statistically independent but have been obtained from known locations in parameter spacex,an unbiased estimate of the true standard error of the mean (actually a correction on the standard deviation part) may be obtained by multiplying the calculated standard error of the sample by the factorf: $f={\sqrt {\frac {1+\rho }{1-\rho }}},$ where the sample bias coefficient ρ is the widely usedPrais–Winsten estimateof theautocorrelation-coefficient (a quantity between −1 and +1) for all sample point pairs. This approximate formula is for moderate to large sample sizes; the reference gives the exact formulas for any sample size, and can be applied to heavily autocorrelated time series like Wall Street stock quotes. Moreover, this formula works for positive and negative ρ alike.^[12]See alsounbiased estimation of standard deviationfor more discussion.

References

^^a ^b ^c ^dAltman, Douglas G; Bland, J Martin (2005-10-15)."Standard deviations and standard errors".BMJ: British Medical Journal.331(7521): 903.doi:10.1136/bmj.331.7521.903.ISSN 0959-8138.PMC1255808.PMID 16223828.
^Everitt, B. S. (2003).The Cambridge Dictionary of Statistics.Cambridge University Press.ISBN 978-0-521-81099-9.
^Wooldridge, Jeffrey M. (2023)."What is a standard error? (And how should we compute it?)".Journal of Econometrics.237(2, Part A).doi:10.1016/j.jeconom.2023.105517.ISSN 0304-4076.
^Gurland, J; Tripathi RC (1971). "A simple approximation for unbiased estimation of the standard deviation".American Statistician.25(4): 30–32.doi:10.2307/2682923.JSTOR 2682923.
^Sokal; Rohlf (1981).Biometry: Principles and Practice of Statistics in Biological Research(2nd ed.). p.53.ISBN 978-0-7167-1254-1.
^Hutchinson, T. P. (1993).Essentials of Statistical Methods, in 41 pages.Adelaide: Rumsby.ISBN 978-0-646-12621-0.
^Cornell, J R; Benjamin, C A (1970).Probability, Statistics, and Decisions for Civil Engineers.NY: McGraw-Hill. pp. 178–179.ISBN 0486796094.
^Barde, M. (2012)."What to use to express the variability of data: Standard deviation or standard error of mean?".Perspect. Clin. Res.3(3): 113–116.doi:10.4103/2229-3485.100662.PMC3487226.PMID 23125963.
^Wassertheil-Smoller, Sylvia(1995).Biostatistics and Epidemiology: A Primer for Health Professionals(Second ed.). New York: Springer. pp. 40–43.ISBN 0-387-94388-9.
^Isserlis, L. (1918)."On the value of a mean as calculated from a sample".Journal of the Royal Statistical Society.81(1): 75–81.doi:10.2307/2340569.JSTOR 2340569.(Equation 1)
^Bondy, Warren; Zlot, William (1976). "The Standard Error of the Mean and the Difference Between Means for Finite Populations".The American Statistician.30(2): 96–97.doi:10.1080/00031305.1976.10479149.JSTOR 2683803.(Equation 2)
^Bence, James R. (1995)."Analysis of Short Time Series: Correcting for Autocorrelation".Ecology.76(2): 628–639.doi:10.2307/1941218.JSTOR 1941218.

[:0-1] Altman, Douglas G; Bland, J Martin (2005-10-15)."Standard deviations and standard errors".BMJ: British Medical Journal.331(7521): 903.doi:10.1136/bmj.331.7521.903.ISSN 0959-8138.PMC1255808.PMID 16223828.

[2] Everitt, B. S. (2003).The Cambridge Dictionary of Statistics.Cambridge University Press.ISBN 978-0-521-81099-9.

[3] Wooldridge, Jeffrey M. (2023)."What is a standard error? (And how should we compute it?)".Journal of Econometrics.237(2, Part A).doi:10.1016/j.jeconom.2023.105517.ISSN 0304-4076.

[4] Gurland, J; Tripathi RC (1971). "A simple approximation for unbiased estimation of the standard deviation".American Statistician.25(4): 30–32.doi:10.2307/2682923.JSTOR 2682923.

[5] Sokal; Rohlf (1981).Biometry: Principles and Practice of Statistics in Biological Research(2nd ed.). p.53.ISBN 978-0-7167-1254-1.

[6] Hutchinson, T. P. (1993).Essentials of Statistical Methods, in 41 pages.Adelaide: Rumsby.ISBN 978-0-646-12621-0.

[7] Cornell, J R; Benjamin, C A (1970).Probability, Statistics, and Decisions for Civil Engineers.NY: McGraw-Hill. pp. 178–179.ISBN 0486796094.

[8] Barde, M. (2012)."What to use to express the variability of data: Standard deviation or standard error of mean?".Perspect. Clin. Res.3(3): 113–116.doi:10.4103/2229-3485.100662.PMC3487226.PMID 23125963.

[9] Wassertheil-Smoller, Sylvia(1995).Biostatistics and Epidemiology: A Primer for Health Professionals(Second ed.). New York: Springer. pp. 40–43.ISBN 0-387-94388-9.

[10] Isserlis, L. (1918)."On the value of a mean as calculated from a sample".Journal of the Royal Statistical Society.81(1): 75–81.doi:10.2307/2340569.JSTOR 2340569.(Equation 1)

[11] Bondy, Warren; Zlot, William (1976). "The Standard Error of the Mean and the Difference Between Means for Finite Populations".The American Statistician.30(2): 96–97.doi:10.1080/00031305.1976.10479149.JSTOR 2683803.(Equation 2)

[12] Bence, James R. (1995)."Analysis of Short Time Series: Correcting for Autocorrelation".Ecology.76(2): 628–639.doi:10.2307/1941218.JSTOR 1941218.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]