Likelihood function

Thelikelihood function(often simply called thelikelihood) is thejointprobability mass(orprobability density) ofobserved dataviewed as a function of theparametersof astatistical model.Intuitively, the likelihood function${\mathcal {L}}(\theta \mid x)$is the probability of observing data vector$x$assuming$\theta$is the actual vector of parameters.

Inmaximum likelihood estimation,thearg max(over the parameter$\theta$) of the likelihood function serves as apoint estimatefor$\theta$,while theFisher information(often approximated by the likelihood'sHessian matrix) indicates the estimate'sprecision.

In contrast, inBayesian statistics,parameter estimates are derived from theconverseof the likelihood, the so-calledposterior probability,which is calculated viaBayes' rule.

Definition

The likelihood function, parameterized by a (possibly multivariate) parameter$\theta$,is usually defined differently fordiscrete and continuousprobability distributions(a more general definition is discussed below). Given a probability density or mass function

$$x\mapsto f(x\mid \theta ),$$

where$x$is a realization of the random variable$X$,the likelihood function is $$\theta \mapsto f(x\mid \theta ),$$ often written $${\mathcal {L}}(\theta \mid x).$$

In other words, when$f(x\mid \theta )$is viewed as a function of$x$with$\theta$fixed, it is a probability density function, and when viewed as a function of$\theta$with$x$fixed, it is a likelihood function. In thefrequentist paradigm,the notation$f(x\mid \theta )$is often avoided and instead$f(x;\theta )$or$f(x,\theta )$are used to indicate that$\theta$is regarded as a fixed unknown quantity rather than as arandom variablebeing conditioned on.

The likelihood function doesnotspecify the probability that$\theta$is the truth, given the observed sample$X=x$.Such an interpretation is a common error, with potentially disastrous consequences (seeprosecutor's fallacy).

Discrete probability distribution

Let$X$be a discreterandom variablewithprobability mass function$p$depending on a parameter$\theta$.Then the function

$${\mathcal {L}}(\theta \mid x)=p_{\theta }(x)=P_{\theta }(X=x),$$

considered as a function of$\theta$,is thelikelihood function,given theoutcome$x$of the random variable$X$.Sometimes the probability of "the value$x$of$X$for the parameter value$\theta$"is written asP(X=x|θ)orP(X=x;θ).The likelihood is the probability that a particular outcome$x$is observed when the true value of the parameter is$\theta$,equivalent to the probability mass on$x$;it isnota probability density over the parameter$\theta$.The likelihood,${\mathcal {L}}(\theta \mid x)$,should not be confused with$P(\theta \mid x)$,which is the posterior probability of$\theta$given the data$x$.

Example

Figure 1. The likelihood function ($p_{\text{H}}^{2}$) for the probability of a coin landing heads-up (without prior knowledge of the coin's fairness), given that we have observed HH.

Figure 2. The likelihood function ($p_{\text{H}}^{2}(1-p_{\text{H}})$) for the probability of a coin landing heads-up (without prior knowledge of the coin's fairness), given that we have observed HHT.

Consider a simple statistical model of a coin flip: a single parameter$p_{\text{H}}$that expresses the "fairness" of the coin. The parameter is the probability that a coin lands heads up ( "H" ) when tossed.$p_{\text{H}}$can take on any value within the range 0.0 to 1.0. For a perfectlyfair coin,$p_{\text{H}}=0.5$.

Imagine flipping a fair coin twice, and observing two heads in two tosses ( "HH" ). Assuming that each successive coin flip isi.i.d.,then the probability of observing HH is

$$P({\text{HH}}\mid p_{\text{H}}=0.5)=0.5^{2}=0.25.$$

Equivalently, the likelihood of observing "HH" assuming$p_{\text{H}}=0.5$is

$${\mathcal {L}}(p_{\text{H}}=0.5\mid {\text{HH}})=0.25.$$

This is not the same as saying that$P(p_{\text{H}}=0.5\mid HH)=0.25$,a conclusion which could only be reached viaBayes' theoremgiven knowledge about the marginal probabilities$P(p_{\text{H}}=0.5)$and$P({\text{HH}})$.

Now suppose that the coin is not a fair coin, but instead that$p_{\text{H}}=0.3$.Then the probability of two heads on two flips is

$$P({\text{HH}}\mid p_{\text{H}}=0.3)=0.3^{2}=0.09.$$

Hence

$${\mathcal {L}}(p_{\text{H}}=0.3\mid {\text{HH}})=0.09.$$

More generally, for each value of$p_{\text{H}}$,we can calculate the corresponding likelihood. The result of such calculations is displayed in Figure 1. The integral of${\mathcal {L}}$over [0, 1] is 1/3; likelihoods need not integrate or sum to one over the parameter space.

Continuous probability distribution

Let$X$be arandom variablefollowing anabsolutely continuous probability distributionwithdensity function$f$(a function of$x$) which depends on a parameter$\theta$.Then the function

$${\mathcal {L}}(\theta \mid x)=f_{\theta }(x),$$

considered as a function of$\theta$,is thelikelihood function(of$\theta$,given theoutcome$X=x$). Again,${\mathcal {L}}$is not a probability density or mass function over$\theta$,despite being a function of$\theta$given the observation$X=x$.

Relationship between the likelihood and probability density functions

The use of theprobability densityin specifying the likelihood function above is justified as follows. Given an observation$x_{j}$,the likelihood for the interval$[x_{j},x_{j}+h]$,where$h>0$is a constant, is given by${\mathcal {L}}(\theta \mid x\in [x_{j},x_{j}+h])$.Observe that $$\mathop {\operatorname {arg\,max} } _{\theta }{\mathcal {L}}(\theta \mid x\in [x_{j},x_{j}+h])=\mathop {\operatorname {arg\,max} } _{\theta }{\frac {1}{h}}{\mathcal {L}}(\theta \mid x\in [x_{j},x_{j}+h]),$$ since$h$is positive and constant. Because $$\mathop {\operatorname {arg\,max} } _{\theta }{\frac {1}{h}}{\mathcal {L}}(\theta \mid x\in [x_{j},x_{j}+h])=\mathop {\operatorname {arg\,max} } _{\theta }{\frac {1}{h}}\Pr(x_{j}\leq x\leq x_{j}+h\mid \theta )=\mathop {\operatorname {arg\,max} } _{\theta }{\frac {1}{h}}\int _{x_{j}}^{x_{j}+h}f(x\mid \theta )\,dx,$$

where$f(x\mid \theta )$is the probability density function, it follows that

$$\mathop {\operatorname {arg\,max} } _{\theta }{\mathcal {L}}(\theta \mid x\in [x_{j},x_{j}+h])=\mathop {\operatorname {arg\,max} } _{\theta }{\frac {1}{h}}\int _{x_{j}}^{x_{j}+h}f(x\mid \theta )\,dx.$$

The firstfundamental theorem of calculusprovides that $$\lim _{h\to 0^{+}}{\frac {1}{h}}\int _{x_{j}}^{x_{j}+h}f(x\mid \theta )\,dx=f(x_{j}\mid \theta ).$$

Then $${\begin{aligned}&\mathop {\operatorname {arg\,max} } _{\theta }{\mathcal {L}}(\theta \mid x_{j})=\mathop {\operatorname {arg\,max} } _{\theta }\left[\lim _{h\to 0^{+}}{\mathcal {L}}(\theta \mid x\in [x_{j},x_{j}+h])\right]\\[4pt]={}&\mathop {\operatorname {arg\,max} } _{\theta }\left[\lim _{h\to 0^{+}}{\frac {1}{h}}\int _{x_{j}}^{x_{j}+h}f(x\mid \theta )\,dx\right]=\mathop {\operatorname {arg\,max} } _{\theta }f(x_{j}\mid \theta ).\end{aligned}}$$

Therefore, $$\mathop {\operatorname {arg\,max} } _{\theta }{\mathcal {L}}(\theta \mid x_{j})=\mathop {\operatorname {arg\,max} } _{\theta }f(x_{j}\mid \theta ),$$ and so maximizing the probability density at$x_{j}$amounts to maximizing the likelihood of the specific observation$x_{j}$.

In general

Inmeasure-theoretic probability theory,thedensity functionis defined as theRadon–Nikodym derivativeof the probability distribution relative to a common dominating measure.The likelihood function is this density interpreted as a function of the parameter, rather than the random variable.Thus, we can construct a likelihood function for any distribution, whether discrete, continuous, a mixture, or otherwise. (Likelihoods are comparable, e.g. for parameter estimation, only if they are Radon–Nikodym derivatives with respect to the same dominating measure.)

The above discussion of the likelihood for discrete random variables uses thecounting measure,under which the probability density at any outcome equals the probability of that outcome.

Likelihoods for mixed continuous–discrete distributions

The above can be extended in a simple way to allow consideration of distributions which contain both discrete and continuous components. Suppose that the distribution consists of a number of discrete probability masses$p_{k}(\theta )$and a density$f(x\mid \theta )$,where the sum of all the$p$'s added to the integral of$f$is always one. Assuming that it is possible to distinguish an observation corresponding to one of the discrete probability masses from one which corresponds to the density component, the likelihood function for an observation from the continuous component can be dealt with in the manner shown above. For an observation from the discrete component, the likelihood function for an observation from the discrete component is simply $${\mathcal {L}}(\theta \mid x)=p_{k}(\theta ),$$ where$k$is the index of the discrete probability mass corresponding to observation$x$,because maximizing the probability mass (or probability) at$x$amounts to maximizing the likelihood of the specific observation.

The fact that the likelihood function can be defined in a way that includes contributions that are not commensurate (the density and the probability mass) arises from the way in which the likelihood function is defined up to a constant of proportionality, where this "constant" can change with the observation$x$,but not with the parameter$\theta$.

Regularity conditions

In the context of parameter estimation, the likelihood function is usually assumed to obey certain conditions, known as regularity conditions. These conditions areassumedin various proofs involving likelihood functions, and need to be verified in each particular application. For maximum likelihood estimation, the existence of a global maximum of the likelihood function is of the utmost importance. By theextreme value theorem,it suffices that the likelihood function iscontinuouson acompactparameter space for the maximum likelihood estimator to exist.While the continuity assumption is usually met, the compactness assumption about the parameter space is often not, as the bounds of the true parameter values might be unknown. In that case,concavityof the likelihood function plays a key role.

More specifically, if the likelihood function is twice continuously differentiable on thek-dimensional parameter space$\Theta$assumed to be anopenconnectedsubset of$\mathbb {R} ^{k}\,,$there exists a unique maximum${\hat {\theta }}\in \Theta$if thematrix of second partials $$\mathbf {H} (\theta )\equiv \left[\,{\frac {\partial ^{2}L}{\,\partial \theta _{i}\,\partial \theta _{j}\,}}\,\right]_{i,j=1,1}^{n_{\mathrm {i} },n_{\mathrm {j} }}\;$$isnegative definitefor every$\,\theta \in \Theta \,$at which the gradient$\;\nabla L\equiv \left[\,{\frac {\partial L}{\,\partial \theta _{i}\,}}\,\right]_{i=1}^{n_{\mathrm {i} }}\;$vanishes, and if the likelihood function approaches a constant on theboundaryof the parameter space,$\;\partial \Theta \;,$i.e., $$\lim _{\theta \to \partial \Theta }L(\theta )=0\;,$$ which may include the points at infinity if$\,\Theta \,$is unbounded. Mäkeläinen and co-authors prove this result usingMorse theorywhile informally appealing to a mountain pass property.Mascarenhas restates their proof using themountain pass theorem.

In the proofs ofconsistencyand asymptotic normality of the maximum likelihood estimator, additional assumptions are made about the probability densities that form the basis of a particular likelihood function. These conditions were first established by Chanda.In particular, foralmost all$x$,and for all$\,\theta \in \Theta \,,$ $${\frac {\partial \log f}{\partial \theta _{r}}}\,,\quad {\frac {\partial ^{2}\log f}{\partial \theta _{r}\partial \theta _{s}}}\,,\quad {\frac {\partial ^{3}\log f}{\partial \theta _{r}\,\partial \theta _{s}\,\partial \theta _{t}}}\,$$ exist for all$\,r,s,t=1,2,\ldots,k\,$in order to ensure the existence of aTaylor expansion.Second, for almost all$x$and for every$\,\theta \in \Theta \,$it must be that $$\left|{\frac {\partial f}{\partial \theta _{r}}}\right|<F_{r}(x)\,,\quad \left|{\frac {\partial ^{2}f}{\partial \theta _{r}\,\partial \theta _{s}}}\right|<F_{rs}(x)\,,\quad \left|{\frac {\partial ^{3}f}{\partial \theta _{r}\,\partial \theta _{s}\,\partial \theta _{t}}}\right|<H_{rst}(x)$$ where$H$is such that$\,\int _{-\infty }^{\infty }H_{rst}(z)\mathrm {d} z\leq M<\infty \;.$This boundedness of the derivatives is needed to allow fordifferentiation under the integral sign.And lastly, it is assumed that theinformation matrix, $$\mathbf {I} (\theta )=\int _{-\infty }^{\infty }{\frac {\partial \log f}{\partial \theta _{r}}}\ {\frac {\partial \log f}{\partial \theta _{s}}}\ f\ \mathrm {d} z$$ ispositive definiteand$\,\left|\mathbf {I} (\theta )\right|\,$is finite. This ensures that thescorehas a finite variance.

The above conditions are sufficient, but not necessary. That is, a model that does not meet these regularity conditions may or may not have a maximum likelihood estimator of the properties mentioned above. Further, in case of non-independently or non-identically distributed observations additional properties may need to be assumed.

In Bayesian statistics, almost identical regularity conditions are imposed on the likelihood function in order to proof asymptotic normality of theposterior probability,and therefore to justify aLaplace approximationof the posterior in large samples.

Likelihood ratio and relative likelihood

See also:Pseudo-R-squared

Likelihood ratio

This section is about the likelihood ratio in general. For the use of likelihood ratios in interpreting diagnostic tests, seeLikelihood ratios in diagnostic testing.For the statistical test to compare goodness of fit, seeLikelihood-ratio test.

Alikelihood ratiois the ratio of any two specified likelihoods, frequently written as: $$\Lambda (\theta _{1}:\theta _{2}\mid x)={\frac {{\mathcal {L}}(\theta _{1}\mid x)}{{\mathcal {L}}(\theta _{2}\mid x)}}.$$

The likelihood ratio is central tolikelihoodist statistics:thelaw of likelihoodstates that degree to which data (considered as evidence) supports one parameter value versus another is measured by the likelihood ratio.

Infrequentist inference,the likelihood ratio is the basis for atest statistic,the so-calledlikelihood-ratio test.By theNeyman–Pearson lemma,this is the mostpowerfultest for comparing twosimple hypothesesat a givensignificance level.Numerous other tests can be viewed as likelihood-ratio tests or approximations thereof.The asymptotic distribution of the log-likelihood ratio, considered as a test statistic, is given byWilks' theorem.

The likelihood ratio is also of central importance inBayesian inference,where it is known as theBayes factor,and is used inBayes' rule.Stated in terms ofodds,Bayes' rule states that theposteriorodds of two alternatives,‍$A_{1}$‍and‍$A_{2}$‍,given an event‍$B$‍,is thepriorodds, times the likelihood ratio. As an equation: $$O(A_{1}:A_{2}\mid B)=O(A_{1}:A_{2})\cdot \Lambda (A_{1}:A_{2}\mid B).$$

The likelihood ratio is not directly used in AIC-based statistics. Instead, what is used is the relative likelihood of models (see below).

Inevidence-based medicine,likelihood ratiosare used in diagnostic testingto assess the value of performing adiagnostic test.

Relative likelihood function

See also:Relative likelihood

Since the actual value of the likelihood function depends on the sample, it is often convenient to work with a standardized measure. Suppose that themaximum likelihood estimatefor the parameterθis${\hat {\theta }}$.Relative plausibilities of otherθvalues may be found by comparing the likelihoods of those other values with the likelihood of${\hat {\theta }}$.Therelative likelihoodofθis defined to be $$R(\theta )={\frac {{\mathcal {L}}(\theta \mid x)}{{\mathcal {L}}({\hat {\theta }}\mid x)}}.$$ Thus, the relative likelihood is the likelihood ratio (discussed above) with the fixed denominator${\mathcal {L}}({\hat {\theta }})$.This corresponds to standardizing the likelihood to have a maximum of 1.

Likelihood region

Alikelihood regionis the set of all values ofθwhose relative likelihood is greater than or equal to a given threshold. In terms of percentages, ap% likelihood regionforθis defined to be

$$\left\{\theta:R(\theta )\geq {\frac {p}{100}}\right\}.$$

Ifθis a single real parameter, ap% likelihood region will usually comprise anintervalof real values. If the region does comprise an interval, then it is called alikelihood interval.

Likelihood intervals, and more generally likelihood regions, are used forinterval estimationwithin likelihoodist statistics: they are similar toconfidence intervalsin frequentist statistics andcredible intervalsin Bayesian statistics. Likelihood intervals are interpreted directly in terms of relative likelihood, not in terms ofcoverage probability(frequentism) orposterior probability(Bayesianism).

Given a model, likelihood intervals can be compared to confidence intervals. Ifθis a single real parameter, then under certain conditions, a 14.65% likelihood interval (about 1:7 likelihood) forθwill be the same as a 95% confidence interval (19/20 coverage probability).In a slightly different formulation suited to the use of log-likelihoods (seeWilks' theorem), the test statistic is twice the difference in log-likelihoods and the probability distribution of the test statistic is approximately achi-squared distributionwith degrees-of-freedom (df) equal to the difference in df's between the two models (therefore, thee⁻²likelihood interval is the same as the 0.954 confidence interval; assuming difference in df's to be 1).

Likelihoods that eliminate nuisance parameters

In many cases, the likelihood is a function of more than one parameter but interest focuses on the estimation of only one, or at most a few of them, with the others being considered asnuisance parameters.Several alternative approaches have been developed to eliminate such nuisance parameters, so that a likelihood can be written as a function of only the parameter (or parameters) of interest: the main approaches are profile, conditional, and marginal likelihoods.These approaches are also useful when a high-dimensional likelihood surface needs to be reduced to one or two parameters of interest in order to allow agraph.

Profile likelihood

It is possible to reduce the dimensions by concentrating the likelihood function for a subset of parameters by expressing the nuisance parameters as functions of the parameters of interest and replacing them in the likelihood function.In general, for a likelihood function depending on the parameter vector$\mathbf {\theta }$that can be partitioned into$\mathbf {\theta } =\left(\mathbf {\theta } _{1}:\mathbf {\theta } _{2}\right)$,and where a correspondence$\mathbf {\hat {\theta }} _{2}=\mathbf {\hat {\theta }} _{2}\left(\mathbf {\theta } _{1}\right)$can be determined explicitly, concentration reducescomputational burdenof the original maximization problem.

For instance, in alinear regressionwith normally distributed errors,$\mathbf {y} =\mathbf {X} \beta +u$,the coefficient vector could bepartitionedinto$\beta =\left[\beta _{1}:\beta _{2}\right]$(and consequently thedesign matrix$\mathbf {X} =\left[\mathbf {X} _{1}:\mathbf {X} _{2}\right]$). Maximizing with respect to$\beta _{2}$yields an optimal value function$\beta _{2}(\beta _{1})=\left(\mathbf {X} _{2}^{\mathsf {T}}\mathbf {X} _{2}\right)^{-1}\mathbf {X} _{2}^{\mathsf {T}}\left(\mathbf {y} -\mathbf {X} _{1}\beta _{1}\right)$.Using this result, the maximum likelihood estimator for$\beta _{1}$can then be derived as $${\hat {\beta }}_{1}=\left(\mathbf {X} _{1}^{\mathsf {T}}\left(\mathbf {I} -\mathbf {P} _{2}\right)\mathbf {X} _{1}\right)^{-1}\mathbf {X} _{1}^{\mathsf {T}}\left(\mathbf {I} -\mathbf {P} _{2}\right)\mathbf {y}$$ where$\mathbf {P} _{2}=\mathbf {X} _{2}\left(\mathbf {X} _{2}^{\mathsf {T}}\mathbf {X} _{2}\right)^{-1}\mathbf {X} _{2}^{\mathsf {T}}$is theprojection matrixof$\mathbf {X} _{2}$.This result is known as theFrisch–Waugh–Lovell theorem.

Since graphically the procedure of concentration is equivalent to slicing the likelihood surface along the ridge of values of the nuisance parameter$\beta _{2}$that maximizes the likelihood function, creating anisometricprofileof the likelihood function for a given$\beta _{1}$,the result of this procedure is also known asprofile likelihood.In addition to being graphed, the profile likelihood can also be used to computeconfidence intervalsthat often have better small-sample properties than those based on asymptoticstandard errorscalculated from the full likelihood.

Conditional likelihood

Sometimes it is possible to find asufficient statisticfor the nuisance parameters, and conditioning on this statistic results in a likelihood which does not depend on the nuisance parameters.

One example occurs in 2×2 tables, where conditioning on all four marginal totals leads to a conditional likelihood based on the non-centralhypergeometric distribution.This form of conditioning is also the basis forFisher's exact test.

Marginal likelihood

Main article:Marginal likelihood

Sometimes we can remove the nuisance parameters by considering a likelihood based on only part of the information in the data, for example by using the set of ranks rather than the numerical values. Another example occurs in linearmixed models,where considering a likelihood for the residuals only after fitting the fixed effects leads toresidual maximum likelihoodestimation of the variance components.

Partial likelihood

A partial likelihood is an adaption of the full likelihood such that only a part of the parameters (the parameters of interest) occur in it.It is a key component of theproportional hazards model:using a restriction on the hazard function, the likelihood does not contain the shape of the hazard over time.

Products of likelihoods

The likelihood, given two or moreindependentevents,is the product of the likelihoods of each of the individual events: $$\Lambda (A\mid X_{1}\land X_{2})=\Lambda (A\mid X_{1})\cdot \Lambda (A\mid X_{2}).$$ This follows from the definition of independence in probability: the probabilities of two independent events happening, given a model, is the product of the probabilities.

This is particularly important when the events are fromindependent and identically distributed random variables,such as independent observations orsampling with replacement.In such a situation, the likelihood function factors into a product of individual likelihood functions.

The empty product has value 1, which corresponds to the likelihood, given no event, being 1: before any data, the likelihood is always 1. This is similar to auniform priorin Bayesian statistics, but in likelihoodist statistics this is not animproper priorbecause likelihoods are not integrated.

Log-likelihood

See also:Log-probability

Log-likelihood functionis the logarithm of the likelihood function, often denoted by a lowercaselor‍$\ell$‍,to contrast with the uppercaseLor${\mathcal {L}}$for the likelihood. Because logarithms arestrictly increasingfunctions, maximizing the likelihood is equivalent to maximizing the log-likelihood. But for practical purposes it is more convenient to work with the log-likelihood function inmaximum likelihood estimation,in particular since most commonprobability distributions—notably theexponential family—are onlylogarithmically concave,andconcavityof theobjective functionplays a key role in themaximization.

Given the independence of each event, the overall log-likelihood of intersection equals the sum of the log-likelihoods of the individual events. This is analogous to the fact that the overalllog-probabilityis the sum of the log-probability of the individual events. In addition to the mathematical convenience from this, the adding process of log-likelihood has an intuitive interpretation, as often expressed as "support" from the data. When the parameters are estimated using the log-likelihood for themaximum likelihood estimation,each data point is used by being added to the total log-likelihood. As the data can be viewed as an evidence that support the estimated parameters, this process can be interpreted as "support from independent evidenceadds ",and the log-likelihood is the "weight of evidence". Interpreting negative log-probability asinformation contentorsurprisal,the support (log-likelihood) of a model, given an event, is the negative of the surprisal of the event, given the model: a model is supported by an event to the extent that the event is unsurprising, given the model.

A logarithm of a likelihood ratio is equal to the difference of the log-likelihoods: $$\log {\frac {{\mathcal {L}}(A)}{{\mathcal {L}}(B)}}=\log {\mathcal {L}}(A)-\log {\mathcal {L}}(B)=\ell (A)-\ell (B).$$

Just as the likelihood, given no event, being 1, the log-likelihood, given no event, is 0, which corresponds to the value of the empty sum: without any data, there is no support for any models.

Graph

Thegraphof the log-likelihood is called thesupport curve(in theunivariatecase). In the multivariate case, the concept generalizes into asupport surfaceover theparameter space. It has a relation to, but is distinct from, thesupport of a distribution.

The term was coined byA. W. F. Edwardsin the context ofstatistical hypothesis testing,i.e. whether or not the data "support" one hypothesis (or parameter value) being tested more than any other.

The log-likelihood function being plotted is used in the computation of thescore(the gradient of the log-likelihood) andFisher information(the curvature of the log-likelihood). Thus, the graph has a direct interpretation in the context ofmaximum likelihood estimationandlikelihood-ratio tests.

Likelihood equations

If the log-likelihood function issmooth,itsgradientwith respect to the parameter, known as thescoreand written$s_{n}(\theta )\equiv \nabla _{\theta }\ell _{n}(\theta )$,exists and allows for the application ofdifferential calculus.The basic way to maximize a differentiable function is to find thestationary points(the points where thederivativeis zero); since the derivative of a sum is just the sum of the derivatives, but the derivative of a product requires theproduct rule,it is easier to compute the stationary points of the log-likelihood of independent events than for the likelihood of independent events.

The equations defined by the stationary point of the score function serve asestimating equationsfor the maximum likelihood estimator. $$s_{n}(\theta )=\mathbf {0}$$ In that sense, the maximum likelihood estimator is implicitly defined by the value at$\mathbf {0}$of theinverse function$s_{n}^{-1}:\mathbb {E} ^{d}\to \Theta$,where$\mathbb {E} ^{d}$is thed-dimensionalEuclidean space,and$\Theta$is the parameter space. Using theinverse function theorem,it can be shown that$s_{n}^{-1}$iswell-definedin anopen neighborhoodabout$\mathbf {0}$with probability going to one, and${\hat {\theta }}_{n}=s_{n}^{-1}(\mathbf {0} )$is a consistent estimate of$\theta$.As a consequence there exists a sequence${\textstyle \left\{{\hat {\theta }}_{n}\right\}}$such that$s_{n}({\hat {\theta }}_{n})=\mathbf {0}$asymptoticallyalmost surely,and${\hat {\theta }}_{n}{\xrightarrow {\text{p}}}\theta _{0}$.A similar result can be established usingRolle's theorem.

The second derivative evaluated at${\hat {\theta }}$,known asFisher information,determines the curvature of the likelihood surface,and thus indicates theprecisionof the estimate.

Exponential families

Further information:Exponential family

The log-likelihood is also particularly useful forexponential familiesof distributions, which include many of the commonparametric probability distributions.The probability distribution function (and thus likelihood function) for exponential families contain products of factors involvingexponentiation.The logarithm of such a function is a sum of products, again easier to differentiate than the original function.

An exponential family is one whose probability density function is of the form (for some functions, writing$\langle -,-\rangle$for theinner product):

$$p(x\mid {\boldsymbol {\theta }})=h(x)\exp {\Big (}\langle {\boldsymbol {\eta }}({\boldsymbol {\theta }}),\mathbf {T} (x)\rangle -A({\boldsymbol {\theta }}){\Big )}.$$

Each of these terms has an interpretation,but simply switching from probability to likelihood and taking logarithms yields the sum:

$$\ell ({\boldsymbol {\theta }}\mid x)=\langle {\boldsymbol {\eta }}({\boldsymbol {\theta }}),\mathbf {T} (x)\rangle -A({\boldsymbol {\theta }})+\log h(x).$$

The${\boldsymbol {\eta }}({\boldsymbol {\theta }})$and$h(x)$each correspond to achange of coordinates,so in these coordinates, the log-likelihood of an exponential family is given by the simple formula:

$$\ell ({\boldsymbol {\eta }}\mid x)=\langle {\boldsymbol {\eta }},\mathbf {T} (x)\rangle -A({\boldsymbol {\eta }}).$$

In words, the log-likelihood of an exponential family is inner product of the natural parameter‍${\boldsymbol {\eta }}$‍and thesufficient statistic‍$\mathbf {T} (x)$‍,minus the normalization factor (log-partition function)‍$A({\boldsymbol {\eta }})$‍.Thus for example the maximum likelihood estimate can be computed by taking derivatives of the sufficient statisticTand the log-partition functionA.

Example: the gamma distribution

Thegamma distributionis an exponential family with two parameters,$\alpha$and$\beta$.The likelihood function is

$${\mathcal {L}}(\alpha,\beta \mid x)={\frac {\beta ^{\alpha }}{\Gamma (\alpha )}}x^{\alpha -1}e^{-\beta x}.$$

Finding the maximum likelihood estimate of$\beta$for a single observed value$x$looks rather daunting. Its logarithm is much simpler to work with:

$$\log {\mathcal {L}}(\alpha,\beta \mid x)=\alpha \log \beta -\log \Gamma (\alpha )+(\alpha -1)\log x-\beta x.\,$$

To maximize the log-likelihood, we first take thepartial derivativewith respect to$\beta$:

$${\frac {\partial \log {\mathcal {L}}(\alpha,\beta \mid x)}{\partial \beta }}={\frac {\alpha }{\beta }}-x.$$

If there are a number of independent observations$x_{1},\ldots,x_{n}$,then the joint log-likelihood will be the sum of individual log-likelihoods, and the derivative of this sum will be a sum of derivatives of each individual log-likelihood:

$${\begin{aligned}&{\frac {\partial \log {\mathcal {L}}(\alpha,\beta \mid x_{1},\ldots,x_{n})}{\partial \beta }}\\={}&{\frac {\partial \log {\mathcal {L}}(\alpha,\beta \mid x_{1})}{\partial \beta }}+\cdots +{\frac {\partial \log {\mathcal {L}}(\alpha,\beta \mid x_{n})}{\partial \beta }}={\frac {n\alpha }{\beta }}-\sum _{i=1}^{n}x_{i}.\end{aligned}}$$

To complete the maximization procedure for the joint log-likelihood, the equation is set to zero and solved for$\beta$:

$${\widehat {\beta }}={\frac {\alpha }{\bar {x}}}.$$

Here${\widehat {\beta }}$denotes the maximum-likelihood estimate, and$\textstyle {\bar {x}}={\frac {1}{n}}\sum _{i=1}^{n}x_{i}$is thesample meanof the observations.

Background and interpretation

Historical remarks

See also:History of statisticsandHistory of probability

The term "likelihood" has been in use in English since at least lateMiddle English.Its formal use to refer to a specificfunctionin mathematical statistics was proposed byRonald Fisher,in two research papers published in 1921and 1922.The 1921 paper introduced what is today called a "likelihood interval"; the 1922 paper introduced the term "method of maximum likelihood".Quoting Fisher:

[I]n 1922, I proposed the term 'likelihood,' in view of the fact that, with respect to [the parameter], it is not a probability, and does not obey the laws of probability, while at the same time it bears to the problem of rational choice among the possible values of [the parameter] a relation similar to that which probability bears to the problem of predicting events in games of chance.... Whereas, however, in relation to psychological judgment, likelihood has some resemblance to probability, the two concepts are wholly distinct.... "

The concept of likelihood should not be confused with probability as mentioned by Sir Ronald Fisher

I stress this because in spite of the emphasis that I have always laid upon the difference between probability and likelihood there is still a tendency to treat likelihood as though it were a sort of probability. The first result is thus that there are two different measures of rational belief appropriate to different cases. Knowing the population we can express our incomplete knowledge of, or expectation of, the sample in terms of probability; knowing the sample we can express our incomplete knowledge of the population in terms of likelihood.

Fisher's invention of statistical likelihood was in reaction against an earlier form of reasoning calledinverse probability.His use of the term "likelihood" fixed the meaning of the term within mathematical statistics.

A. W. F. Edwards(1972) established the axiomatic basis for use of the log-likelihood ratio as a measure of relative support for one hypothesis against another. Thesupport functionis then the natural logarithm of the likelihood function. Both terms are used inphylogenetics,but were not adopted in a general treatment of the topic of statistical evidence.

Interpretations under different foundations

Among statisticians, there is no consensus about what thefoundation of statisticsshould be. There are four main paradigms that have been proposed for the foundation:frequentism,Bayesianism,likelihoodism,andAIC-based.For each of the proposed foundations, the interpretation of likelihood is different. The four interpretations are described in the subsections below.

Bayesian interpretation

InBayesian inference,although one can speak about the likelihood of any proposition orrandom variablegiven another random variable: for example the likelihood of a parameter value or of astatistical model(seemarginal likelihood), given specified data or other evidence,the likelihood function remains the same entity, with the additional interpretations of (i) aconditional densityof the data given the parameter (since the parameter is then a random variable) and (ii) a measure or amount of information brought by the data about the parameter value or even the model.Due to the introduction of a probability structure on the parameter space or on the collection of models, it is possible that a parameter value or a statistical model have a large likelihood value for given data, and yet have a lowprobability,or vice versa.This is often the case in medical contexts.FollowingBayes' Rule,the likelihood when seen as a conditional density can be multiplied by theprior probabilitydensity of the parameter and then normalized, to give aposterior probabilitydensity.More generally, the likelihood of an unknown quantity$X$given another unknown quantity$Y$is proportional to theprobability of$Y$given$X$.

Likelihoodist interpretation

In frequentist statistics, the likelihood function is itself astatisticthat summarizes a single sample from a population, whose calculated value depends on a choice of several parametersθ₁...θ_p,wherepis the count of parameters in some already-selectedstatistical model.The value of the likelihood serves as a figure of merit for the choice used for the parameters, and the parameter set with maximum likelihood is the best choice, given the data available.

The specific calculation of the likelihood is the probability that the observed sample would be assigned, assuming that the model chosen and the values of the several parametersθgive an accurate approximation of thefrequency distributionof the population that the observed sample was drawn from. Heuristically, it makes sense that a good choice of parameters is those which render the sample actually observed the maximum possiblepost-hocprobability of having happened.Wilks' theoremquantifies the heuristic rule by showing that the difference in the logarithm of the likelihood generated by the estimate's parameter values and the logarithm of the likelihood generated by population's "true" (but unknown) parameter values is asymptoticallyχ²distributed.

Each independent sample's maximum likelihood estimate is a separate estimate of the "true" parameter set describing the population sampled. Successive estimates from many independent samples will cluster together with the population's "true" set of parameter values hidden somewhere in their midst. The difference in the logarithms of the maximum likelihood and adjacent parameter sets' likelihoods may be used to draw aconfidence regionon a plot whose co-ordinates are the parametersθ₁...θ_p.The region surrounds the maximum-likelihood estimate, and all points (parameter sets) within that region differ at most in log-likelihood by some fixed value. Theχ²distributiongiven byWilks' theoremconverts the region's log-likelihood differences into the "confidence" that the population's "true" parameter set lies inside. The art of choosing the fixed log-likelihood difference is to make the confidence acceptably high while keeping the region acceptably small (narrow range of estimates).

As more data are observed, instead of being used to make independent estimates, they can be combined with the previous samples to make a single combined sample, and that large sample may be used for a new maximum likelihood estimate. As the size of the combined sample increases, the size of the likelihood region with the same confidence shrinks. Eventually, either the size of the confidence region is very nearly a single point, or the entire population has been sampled; in both cases, the estimated parameter set is essentially the same as the population parameter set.

AIC-based interpretation

Under theAICparadigm, likelihood is interpreted within the context ofinformation theory.

See also

• Bayes factor
• Conditional entropy
• Conditional probability
• Empirical likelihood
• Likelihood principle
• Likelihood-ratio test
• Likelihoodist statistics
• Maximum likelihood
• Principle of maximum entropy
• Pseudolikelihood
• Score (statistics)

Further reading

• Azzalini, Adelchi (1996). "Likelihood".Statistical Inference Based on the Likelihood.Chapman and Hall. pp. 17–50.ISBN0-412-60650-X.
• Boos, Dennis D.; Stefanski, L. A. (2013). "Likelihood Construction and Estimation".Essential Statistical Inference: Theory and Methods.New York: Springer. pp. 27–124.doi:10.1007/978-1-4614-4818-1_2.ISBN978-1-4614-4817-4.
• Edwards, A. W. F.(1992) [1972].Likelihood(Expanded ed.).Johns Hopkins University Press.ISBN0-8018-4443-6.
• King, Gary(1989)."The Likelihood Model of Inference".Unifying Political Methodology: the Likehood Theory of Statistical Inference.Cambridge University Press. pp. 59–94.ISBN0-521-36697-6.
• Richard, Mark; Vecer, Jan (1 February 2021)."Efficiency Testing of Prediction Markets: Martingale Approach, Likelihood Ratio and Bayes Factor Analysis".Risks.9(2): 31.doi:10.3390/risks9020031.hdl:10419/258120.
• Lindsey, J. K. (1996)."Likelihood".Parametric Statistical Inference.Oxford University Press. pp. 69–139.ISBN0-19-852359-9.
• Rohde, Charles A. (2014).Introductory Statistical Inference with the Likelihood Function.Berlin: Springer.ISBN978-3-319-10460-7.
• Royall, Richard (1997).Statistical Evidence: A Likelihood Paradigm.London: Chapman & Hall.ISBN0-412-04411-0.
• Ward, Michael D.;Ahlquist, John S. (2018)."The Likelihood Function: A Deeper Dive".Maximum Likelihood for Social Science: Strategies for Analysis.Cambridge University Press.pp. 21–28.ISBN978-1-316-63682-4.

External links

• Likelihood function at Planetmath
• "Log-likelihood".Statlect.