Rate of convergence

Innumerical analysis,theorder of convergenceand therate of convergenceof aconvergent sequenceare quantities that represent how quickly the sequence approaches its limit. A sequence$(x_{n})$that converges to$L$is said to haveorder of convergence$q\geq 1$andrate of convergence$\mu$if

$\lim _{n\rightarrow \infty }{\frac {\left|x_{n+1}-L\right|}{\left|x_{n}-L\right|^{q}}}=\mu.$

The rate of convergence$\mu$is also called theasymptotic error constant. Note that this terminology is not standardized and some authors will useratewhere this article usesorder(e.g.,).

In practice, the rate and order of convergence provide useful insights when usingiterative methodsfor calculating numerical approximations. If the order of convergence is higher, then typically fewer iterations are necessary to yield a useful approximation. Strictly speaking, however, theasymptotic behaviorof a sequence does not give conclusive information about any finite part of the sequence.

Similar concepts are used fordiscretizationmethods. The solution of the discretized problem converges to the solution of the continuous problem as the grid size goes to zero, and the speed of convergence is one of the factors of the efficiency of the method. However, the terminology, in this case, is different from the terminology for iterative methods.

Series accelerationis a collection of techniques for improving the rate of convergence of a series discretization. Such acceleration is commonly accomplished withsequence transformations.

Convergence speed for iterative methods

Convergence definitions

Suppose that thesequence$(x_{k})$converges to the number$L$.The sequence is said toconverge with order$q$to$L$,and with arate of convergenceof$\mu$,if

$$\lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|^{q}}}=\mu$$(Definition 1)

for some positive constant$\mu \in (0,\infty )$if$q>1$,and$\mu \in (0,1)$if$q=1$.It is not necessary, however, that$q$be an integer. For example, thesecant method,when converging to a regular,simple root,has an order ofφ≈ 1.618.

Convergence with order

• $q=1$is calledlinear convergenceif$\mu \in (0,1)$,and the sequence is said toconverge Q-linearly to$L$.
• $q=2$is calledquadratic convergence.
• $q=3$is calledcubic convergence.
• etc.

Order estimation

A practical method to calculate the order of convergence for a sequence generated by a fixed point iteration is to calculate the following sequence, which converges to$q$: $$q\approx {\frac {\log \left|\displaystyle {\frac {x_{k+1}-x_{k}}{x_{k}-x_{k-1}}}\right|}{\log \left|\displaystyle {\frac {x_{k}-x_{k-1}}{x_{k-1}-x_{k-2}}}\right|}}.$$

For numerical approximation of an exact value through a numerical method of order q see

Q-convergence definitions

In addition to the previously defined Q-linear convergence, a few other Q-convergence definitions exist. Given Definition 1 defined above, the sequence is said toconverge Q-superlinearly to$L$(i.e. faster than linearly) in all the cases where$q>1$and also the case$q=1,\mu =0$.Given Definition 1, the sequence is said toconverge Q-sublinearly to$L$(i.e. slower than linearly) if$q=1,\mu =1$.The sequence$(x_{k})$converges logarithmically to$L$if the sequence converges sublinearly and additionally if $$\lim _{k\to \infty }{\frac {|x_{k+1}-x_{k}|}{|x_{k}-x_{k-1}|}}=1.$$ Note that unlike previous definitions, logarithmic convergence is not called "Q-logarithmic."

In the definitions above, the "Q-" stands for "quotient" because the terms are defined using the quotient between two successive terms.Often, however, the "Q-" is dropped and a sequence is simply said to havelinear convergence,quadratic convergence,etc.

R-convergence definition

The Q-convergence definitions have a shortcoming in that they do not include some sequences, such as the sequence$(b_{k})$below, which converge reasonably fast, but whose rate is variable. Therefore, the definition of rate of convergence is extended as follows.

Suppose that$(x_{k})$converges to$L$.The sequence is said toconverge R-linearly to$L$if there exists a sequence$(\varepsilon _{k})$such that $$|x_{k}-L|\leq \varepsilon _{k}\quad {\text{for all }}k\,,$$ and$(\varepsilon _{k})$converges Q-linearly to zero.The "R-" prefix stands for "root".

Examples

Consider the sequence $$(a_{k})=\left\{1,{\frac {1}{2}},{\frac {1}{4}},{\frac {1}{8}},{\frac {1}{16}},{\frac {1}{32}},\ldots,{\frac {1}{2^{k}}},\dots \right\}.$$ It can be shown that this sequence converges to$L=0$.To determine the type of convergence, we plug the sequence into the definition of Q-linear convergence, $$\lim _{k\to \infty }{\frac {\left|1/2^{k+1}-0\right|}{\left|1/2^{k}-0\right|}}=\lim _{k\to \infty }{\frac {2^{k}}{2^{k+1}}}={\frac {1}{2}}.$$ Thus, we find that$(a_{k})$converges Q-linearly and has a convergence rate of$\mu =1/2$. More generally, for any$c\in \mathbb {R},\mu \in (-1,1)$,the sequence$(c\mu ^{k})$converges linearly with rate$|\mu |$.

The sequence $$(b_{k})=\left\{1,1,{\frac {1}{4}},{\frac {1}{4}},{\frac {1}{16}},{\frac {1}{16}},\ldots,{\frac {1}{4^{\left\lfloor {\frac {k}{2}}\right\rfloor }}},\,\ldots \right\}$$ also converges linearly to 0 with rate 1/2 under the R-convergence definition, but not under the Q-convergence definition. (Note that$\lfloor x\rfloor$is thefloor function,which gives the largest integer that is less than or equal to$x$.)

The sequence $$(c_{k})=\left\{{\frac {1}{2}},{\frac {1}{4}},{\frac {1}{16}},{\frac {1}{256}},{\frac {1}{65,\!536}},\ldots,{\frac {1}{2^{2^{k}}}},\ldots \right\}$$ converges superlinearly. In fact, it is quadratically convergent.

Finally, the sequence $$(d_{k})=\left\{1,{\frac {1}{2}},{\frac {1}{3}},{\frac {1}{4}},{\frac {1}{5}},{\frac {1}{6}},\ldots,{\frac {1}{k+1}},\ldots \right\}$$ converges sublinearly and logarithmically.

Linear, linear, superlinear (quadratic), and sublinear rates of convergence

Convergence speed for discretization methods

A similar situation exists for discretization methods designed to approximate a function$y=f(x)$,which might be an integral being approximated bynumerical quadrature,or thesolution of an ordinary differential equation(see example below). The discretization method generates a sequence${y_{0},y_{1},y_{2},y_{3},...}$,where each successive$y_{j}$is a function of$y_{j-1},y_{j-2},...$along with the grid spacing$h$between successive values of the independent variable$x$.The important parameter here for the convergence speed to$y=f(x)$is the grid spacing$h$,inversely proportional to the number of grid points, i.e. the number of points in the sequence required to reach a given value of$x$.

In this case, the sequence$(y_{n})$is said to converge to the sequence$f(x_{n})$with orderqif there exists a constantCsuch that

$|y_{n}-f(x_{n})|<Ch^{q}{\text{ for all }}n.$

This is written as$|y_{n}-f(x_{n})|={\mathcal {O}}(h^{q})$usingbig O notation.

This is the relevant definition when discussing methods fornumerical quadratureor the solution of ordinary differential equations (ODEs).

A practical method to estimate the order of convergence for a discretization method is pick step sizes$h_{\text{new}}$and$h_{\text{old}}$and calculate the resulting errors$e_{\text{new}}$and$e_{\text{old}}$.The order of convergence is then approximated by the following formula:

$q\approx {\frac {\log(e_{\text{new}}/e_{\text{old}})}{\log(h_{\text{new}}/h_{\text{old}})}},$

which comes from writing the truncation error, at the old and new grid spacings, as

$e=|y_{n}-f(x_{n})|={\mathcal {O}}(h^{q}).$

The error$e$is, more specifically, aglobal truncation error(GTE), in that it represents a sum of errors accumulated over all$n$iterations, as opposed to alocal truncation error(LTE) over just one iteration.

Example of discretization methods

Consider the ordinary differential equation

${\frac {dy}{dx}}=-\kappa y$

with initial condition$y(0)=y_{0}$.We can solve this equation using theForward Euler scheme for numerical discretization:

${\frac {y_{n+1}-y_{n}}{h}}=-\kappa y_{n},$

which generates the sequence

$y_{n+1}=y_{n}(1-h\kappa ).$

In terms of$y(0)=y_{0}$,this sequence is as follows, from theBinomial theorem: $$y_{n}=y_{0}(1-h\kappa )^{n}=y_{0}\left(1-nh\kappa +n(n-1){\frac {h^{2}\kappa ^{2}}{2}}+....\right).$$

The exact solution to this ODE is$y=f(x)=y_{0}\exp(-\kappa x)$,corresponding to the followingTaylor expansionin$h\kappa$for$h\kappa \ll 1$: $$f(x_{n})=f(nh)=y_{0}\exp(-\kappa nh)=y_{0}\left[\exp(-\kappa h)\right]^{n}=y_{0}\left(1-h\kappa +{\frac {h^{2}\kappa ^{2}}{2}}+....\right)^{n}=y_{0}\left(1-nh\kappa +{\frac {n^{2}h^{2}\kappa ^{2}}{2}}+...\right).$$

In this case, the truncation error is

$e=|y_{n}-f(x_{n})|={\frac {nh^{2}\kappa ^{2}}{2}}={\mathcal {O}}(h^{2}),$

so$(y_{n})$converges to$f(x_{n})$with a convergence rate$q=2$.

Examples (continued)

The sequence$(d_{k})$with$d_{k}=1/(k+1)$was introduced above. This sequence converges with order 1 according to the convention for discretization methods.

The sequence$(a_{k})$with$a_{k}=2^{-k}$,which was also introduced above, converges with orderqfor every numberq.It is said to converge exponentially using the convention for discretization methods. However, it only converges linearly (that is, with order 1) using the convention for iterative methods.

Recurrent sequences and fixed points

The case of recurrent sequences$x_{n+1}:=f(x_{n})$which occurs indynamical systemsand in the context of variousfixed-point theoremsis of particular interest. Assuming that the relevant derivatives offare continuous, one can (easily) show that for a fixed point$f(p)=p$such that$|f'(p)|<1$,one has at least linear convergence for any starting value$x_{0}$sufficiently close top.If$|f'(p)|=0$and$|f''(p)|<1$,then one has at least quadratic convergence, and so on. If$|f'(p)|>1$,then one has arepulsive fixed pointand no starting value will produce a sequence converging top(unless one directly jumps to the pointpitself).

Acceleration of convergence

Main article:series acceleration

Many methods exist to increase the rate of convergence of a given sequence, i.e., totransform a given sequenceinto one converging faster to the same limit. Such techniques are in general known as "series acceleration".The goal is to reduce thecomputational costof approximating the limit of the transformed sequence. One example of series acceleration isAitken's delta-squared process.These methods in general (and in particular Aitken's method) do not increase the order of convergence, and are useful only if initially the convergence is not faster than linear:If$(x_{n})$convergences linearly, one gets a sequence$(a_{n})$that still converges linearly (except for pathologically designed special cases), but faster in the sense that$\lim(a_{n}-L)/(x_{n}-L)=0$.On the other hand, if the convergence is already of order ≥ 2, Aitken's method will bring no improvement.

Literature

The simple definition is used in

• Michelle Schatzman(2002),Numerical analysis: a mathematical introduction,Clarendon Press, Oxford.ISBN0-19-850279-6.

The extended definition is used in

• Walter Gautschi (1997),Numerical analysis: an introduction,Birkhäuser, Boston.ISBN0-8176-3895-4.
• Endre Süliand David Mayers (2003),An introduction to numerical analysis,Cambridge University Press.ISBN0-521-00794-1.

The Big O definition is used in

• Richard L. Burden and J. Douglas Faires (2001),Numerical Analysis(7th ed.), Brooks/Cole.ISBN0-534-38216-9

The termsQ-linearandR-linearare used in

• Nocedal, Jorge; Wright, Stephen J. (2006).Numerical Optimization(2nd ed.). Berlin, New York:Springer-Verlag.pp. 619+620.ISBN978-0-387-30303-1..