Sorting network

Incomputer science,comparator networksare abstract devices built up of a fixed number of "wires", carrying values, and comparator modules that connect pairs of wires, swapping the values on the wires if they are not in a desired order. Such networks are typically designed to performsortingon fixed numbers of values, in which case they are calledsorting networks.

Sorting networks differ from generalcomparison sortsin that they are not capable of handling arbitrarily large inputs, and in that their sequence of comparisons is set in advance, regardless of the outcome of previous comparisons. In order to sort larger amounts of inputs, new sorting networks must be constructed. This independence of comparison sequences is useful for parallel execution and for implementation inhardware.Despite the simplicity of sorting nets, their theory is surprisingly deep and complex. Sorting networks were first studied circa 1954 by Armstrong, Nelson and O'Connor,^[1]who subsequently patented the idea.^[2]

Sorting networks can be implemented either inhardwareor insoftware.Donald Knuthdescribes how the comparators for binary integers can be implemented as simple, three-state electronic devices.^[1]Batcher,in 1968, suggested using them to constructswitching networksfor computer hardware, replacing bothbusesand the faster, but more expensive,crossbar switches.^[3]Since the 2000s, sorting nets (especiallybitonic mergesort) are used by theGPGPUcommunity for constructing sorting algorithms to run ongraphics processing units.^[4]

Introduction

Demonstration of a comparator in a sorting network.

A sorting network consists of two types of items: comparators and wires. The wires are thought of as running from left to right, carrying values (one per wire) that traverse the network all at the same time. Each comparator connects two wires. When a pair of values, traveling through a pair of wires, encounter a comparator, the comparator swaps the valuesif and only ifthe top wire's value is greater or equal to the bottom wire's value.

In a formula, if the top wire carries $x$ and the bottom wire carries $y$ ,then after hitting a comparator the wires carry $x'=\min(x,y)$ and $y'=\max(x,y)$ ,respectively, so the pair of values is sorted.^[5]^: 635A network of wires and comparators that will correctly sort all possible inputs into ascending order is called a sorting network or Kruskal hub. By reflecting the network, it is also possible to sort all inputs into descending order.

The full operation of a simple sorting network is shown below. It is evident why this sorting network will correctly sort the inputs; note that the first four comparators will "sink" the largest value to the bottom and "float" the smallest value to the top. The final comparator sorts out the middle two wires.

Depth and efficiency

The efficiency of a sorting network can be measured by its total size, meaning the number of comparators in the network, or by itsdepth,defined (informally) as the largest number of comparators that any input value can encounter on its way through the network. Noting that sorting networks can perform certain comparisonsin parallel(represented in the graphical notation by comparators that lie on the same vertical line), and assuming all comparisons to take unit time, it can be seen that the depth of the network is equal to the number of time steps required to execute it.^[5]^{: 636–637}

Insertion and Bubble networks

We can easily construct a network of any size recursively using the principles of insertion and selection. Assuming we have a sorting network of sizen,we can construct a network of sizen+ 1by "inserting" an additional number into the already sorted subnet (using the principle underlyinginsertion sort). We can also accomplish the same thing by first "selecting" the lowest value from the inputs and then sort the remaining values recursively (using the principle underlyingbubble sort).

A sorting network constructed recursively that first sinks the largest value to the bottom and then sorts the remaining wires. Based onbubble sort

A sorting network constructed recursively that first sorts the first n wires, and then inserts the remaining value. Based oninsertion sort

The structure of these two sorting networks are very similar. A construction of the two different variants, which collapses together comparators that can be performed simultaneously shows that, in fact, they are identical.^[1]

When allowing for parallel comparators, bubble sort and insertion sort are identical

The insertion network (or equivalently, bubble network) has a depth of $2 n - 3$ ,^[1]where $n$ is the number of values. This is better than the $O (n log n)$ time needed byrandom-access machines,but it turns out that there are much more efficient sorting networks with a depth of just $O (log 2 n)$ ,as describedbelow.

Zero-one principle

While it is easy to prove the validity of some sorting networks (like the insertion/bubble sorter), it is not always so easy. There are $n!$ permutations of numbers in an $n$ -wire network, and to test all of them would take a significant amount of time, especially when $n$ is large. The number of test cases can be reduced significantly, to $2 n$ ,using the so-called zero-one principle. While still exponential, this is smaller than $n!$ for all $n \geq 4$ ,and the difference grows quite quickly with increasing $n$ .

The zero-one principle states that, if a sorting network can correctly sort all $2 n$ sequences of zeros and ones, then it is also valid for arbitrary ordered inputs. This not only drastically cuts down on the number of tests needed to ascertain the validity of a network, it is of great use in creating many constructions of sorting networks as well.

The principle can be proven by first observing the following fact about comparators: when amonotonically increasingfunction $f$ is applied to the inputs, i.e., $x$ and $y$ are replaced by $f (x)$ and $f (y)$ ,then the comparator produces $min(f (x), f (y)) = f (min(x, y))$ and $max(f (x), f (y)) = f (max(x, y))$ .Byinductionon the depth of the network, this result can be extended to alemmastating that if the network transforms the sequence $a 1,..., a n$ into $b 1,..., b n$ ,it will transform $f (a 1),..., f (a n)$ into $f (b 1),..., f (b n)$ .Suppose that some input $a 1,..., a n$ contains two items $a i < a j$ ,and the network incorrectly swaps these in the output. Then it will also incorrectly sort $f (a 1),..., f (a n)$ for the function

f(x)={\begin{cases}1\ &{\mbox{if }}x>a_{i}\\0\ &{\mbox{otherwise.}}\end{cases}}

This function is monotonic, so we have the zero-one principle as thecontrapositive.^[5]^{: 640–641}

Constructing sorting networks

Various algorithms exist to construct sorting networks of depth $O (log 2 n)$ (hence size $O (n log 2 n)$ ) such asBatcher odd–even mergesort,bitonic sort,Shell sort,and thePairwise sorting network.These networks are often used in practice.

It is also possible to construct networks of depth $O (log n)$ (hence size $O (n log n)$ ) using a construction called theAKS network,after its discoverersAjtai,Komlós,andSzemerédi.^[6]While an important theoretical discovery, the AKS network has very limited practical application because of the large linear constant hidden by theBig-O notation.^[5]^: 653These are partly due to a construction of anexpander graph.

A simplified version of the AKS network was described byPatersonin 1990, who noted that "the constants obtained for the depth bound still prevent the construction being of practical value".^[7]

A more recent construction called thezig-zag sorting networkof size $O (n log n)$ was discovered byGoodrichin 2014.^[8]While its size is much smaller than that of AKS networks, its depth $O (n log n)$ makes it unsuitable for a parallel implementation.

Optimal sorting networks

For small, fixed numbers of inputs $n$ ,optimalsorting networks can be constructed, with either minimal depth (for maximally parallel execution) or minimal size (number of comparators). These networks can be used to increase the performance of larger sorting networks resulting from therecursiveconstructions of, e.g., Batcher, by halting the recursion early and inserting optimal nets as base cases.^[9]The following table summarizes the optimality results for small networks for which the optimal depth is known:

$n$	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
Depth^[10]	0	1	3	3	5	5	6	6	7	7	8	8	9	9	9	9	10
Size, upper bound^[11]	0	1	3	5	9	12	16	19	25	29	35	39	45	51	56	60	71
Size, lower bound (if different)^[12]													43	47	51	55	60

For larger networks neither the optimal depth nor the optimal size are currently known. The bounds known so far are provided in the table below:

$n$	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32
Depth, upper bound^[10]^[13]^[14]	11	11	11	12	12	12	12	13	13	14	14	14	14	14	14
Depth, lower bound^[10]	10	10	10	10	10	10	10	10	10	10	10	10	10	10	10
Size, upper bound^[14]	77	85	91	99	107	114	120	131	139	148	155	165	172	180	185
Size, lower bound^[12]	65	70	75	80	85	90	95	100	105	110	115	120	125	130	135

The first sixteen depth-optimal networks are listed in Knuth'sArt of Computer Programming,^[1]and have been since the 1973 edition; however, while the optimality of the first eight was established byFloydand Knuth in the 1960s, this property wasn't proven for the final six until 2014^[15](the cases nine and ten having been decided in 1991^[9]).

For one to twelve inputs, minimal (i.e. size-optimal) sorting networks are known, and for higher values, lower bounds on their sizes $S (n)$ can be derived inductively using a lemma due to Van Voorhis^[1](p. 240): $S (n) \geq S (n - 1) + ⌈log 2 n ⌉$ .The first ten optimal networks have been known since 1969, with the first eight again being known as optimal since the work of Floyd and Knuth, but optimality of the cases $n = 9$ and $n = 10$ took until 2014 to be resolved.^[11] The optimality of the smallest known sorting networks for $n = 11$ and $n = 12$ was resolved in 2020.^[16]^[1]

Some work in designing optimal sorting network has been done usinggenetic algorithms:D. Knuth mentions that thesmallestknown sorting network for $n = 13$ was found by Hugues Juillé in 1995 "by simulating an evolutionary process of genetic breeding"^[1](p. 226), and that theminimum depthsorting networks for $n = 9$ and $n = 11$ were found by Loren Schwiebert in 2001 "using genetic methods"^[1](p. 229).

Complexity of testing sorting networks

UnlessP=NP,the problem of testing whether a candidate network is a sorting network is likely to remain difficult for networks of large sizes, due to the problem beingco-NP-complete.^[17]

References

^^a ^b ^c ^d ^e ^f ^g ^h ⁱKnuth, D. E.(1997).The Art of Computer Programming, Volume 3: Sorting and Searching(Second ed.). Addison–Wesley. pp. 219–247.ISBN 978-0-201-89685-5.Section 5.3.4: Networks for Sorting.
^US 3029413,O'Connor, Daniel G. & Nelson, Raymond J., "Sorting system with $n$ -line sorting switch ", published 10 April 1962
^Batcher, K. E. (1968).Sorting networks and their applications.Proc. AFIPS Spring Joint Computer Conference. pp. 307–314.
^Owens, J. D.; Houston, M.; Luebke, D.; Green, S.; Stone, J. E.; Phillips, J. C. (2008). "GPU Computing".Proceedings of the IEEE.96(5): 879–899.doi:10.1109/JPROC.2008.917757.S2CID 17091128.
^^a ^b ^c ^dCormen, Thomas H.;Leiserson, Charles E.;Rivest, Ronald L.(1990).Introduction to Algorithms(1st ed.). MIT Press and McGraw-Hill.ISBN 0-262-03141-8.
^Ajtai, M.;Komlós, J.;Szemerédi, E.(1983).An $O(n log n)$ sorting network.STOC'83.Proceedings of the fifteenth annual ACM symposium on Theory of computing.pp. 1–9.doi:10.1145/800061.808726.ISBN 0-89791-099-0.
^Paterson, M. S. (1990). "Improved sorting networks with $O (log N)$ depth ".Algorithmica.5(1–4): 75–92.doi:10.1007/BF01840378.S2CID 2064561.
^Goodrich, Michael(March 2014). "Zig-zag sort".Proceedings of the forty-sixth annual ACM symposium on Theory of computing.pp. 684–693.arXiv:1403.2777.doi:10.1145/2591796.2591830.ISBN 9781450327107.S2CID 947950.
^^a ^bParberry, Ian (1991)."A Computer Assisted Optimal Depth Lower Bound for Nine-Input Sorting Networks"(PDF).Mathematical Systems Theory.24:101–116.CiteSeerX10.1.1.712.219.doi:10.1007/bf02090393.S2CID 7077160.
^^a ^b ^cCodish, Michael; Cruz-Filipe, Luís; Ehlers, Thorsten; Müller, Mike; Schneider-Kamp, Peter (2015).Sorting Networks: to the End and Back Again.arXiv:1507.01428.Bibcode:2015arXiv150701428C.
^^a ^bCodish, Michael; Cruz-Filipe, Luís; Frank, Michael; Schneider-Kamp, Peter (2014).Twenty-Five Comparators is Optimal when Sorting Nine Inputs (and Twenty-Nine for Ten).Proc. Int'l Conf. Tools with AI (ICTAI). pp. 186–193.arXiv:1405.5754.Bibcode:2014arXiv1405.5754C.
^^a ^bObtained by Van Voorhis lemma and the value $S (11) = 35$
^Ehlers, Thorsten (February 2017). "Merging almost sorted sequences yields a 24-sorter".Information Processing Letters.118:17–20.doi:10.1016/j.ipl.2016.08.005.
^^a ^bDobbelaere, Bert."SorterHunter".GitHub.Retrieved2 Jan2022.
^Bundala, D.; Závodný, J. (2014). "Optimal Sorting Networks".Language and Automata Theory and Applications.Lecture Notes in Computer Science. Vol. 8370. pp. 236–247.arXiv:1310.6271.doi:10.1007/978-3-319-04921-2_19.ISBN 978-3-319-04920-5.S2CID 16860013.
^Harder, Jannis (2020). "An Answer to the Bose-Nelson Sorting Problem for 11 and 12 Channels".arXiv:2012.04400[cs.DS].
^Parberry, Ian (1991).On the Computational Complexity of Optimal Sorting Network Verification.Proc. PARLE '91: Parallel Architectures and Languages Europe, Volume I: Parallel Architectures and Algorithms, Eindhoven, the Netherlands.pp. 252–269.

Angel, O.; Holroyd, A. E.; Romik, D.; Virág, B. (2007)."Random sorting networks".Advances in Mathematics.215(2): 839–868.arXiv:math/0609538.doi:10.1016/j.aim.2007.05.019.

External links

List of smallest sorting networks for given number of inputs
Sorting Networks
CHAPTER 28: SORTING NETWORKS
Sorting Networks
Tool for generating and graphing sorting networks
Sorting networks and the END algorithm
Lipton, Richard J.;Regan, Ken (24 April 2014)."Galactic Sorting Networks".Gödel’s Lost Letter and P=NP.
Sorting Networks validity

[knuth-1] ^^a ^b ^c ^d ^e ^f ^g ^h ⁱKnuth, D. E.(1997).The Art of Computer Programming, Volume 3: Sorting and Searching(Second ed.). Addison–Wesley. pp. 219–247.ISBN 978-0-201-89685-5.Section 5.3.4: Networks for Sorting.

[2] US 3029413,O'Connor, Daniel G. & Nelson, Raymond J., "Sorting system with $n$ -line sorting switch ", published 10 April 1962

[3] Batcher, K. E. (1968).Sorting networks and their applications.Proc. AFIPS Spring Joint Computer Conference. pp. 307–314.

[4] Owens, J. D.; Houston, M.; Luebke, D.; Green, S.; Stone, J. E.; Phillips, J. C. (2008). "GPU Computing".Proceedings of the IEEE.96(5): 879–899.doi:10.1109/JPROC.2008.917757.S2CID 17091128.

[clrs-5] Cormen, Thomas H.;Leiserson, Charles E.;Rivest, Ronald L.(1990).Introduction to Algorithms(1st ed.). MIT Press and McGraw-Hill.ISBN 0-262-03141-8.

[6] Ajtai, M.;Komlós, J.;Szemerédi, E.(1983).An $O(n log n)$ sorting network.STOC'83.Proceedings of the fifteenth annual ACM symposium on Theory of computing.pp. 1–9.doi:10.1145/800061.808726.ISBN 0-89791-099-0.

[7] Paterson, M. S. (1990). "Improved sorting networks with $O (log N)$ depth ".Algorithmica.5(1–4): 75–92.doi:10.1007/BF01840378.S2CID 2064561.

[8] Goodrich, Michael(March 2014). "Zig-zag sort".Proceedings of the forty-sixth annual ACM symposium on Theory of computing.pp. 684–693.arXiv:1403.2777.doi:10.1145/2591796.2591830.ISBN 9781450327107.S2CID 947950.

[parberry91-9] Parberry, Ian (1991)."A Computer Assisted Optimal Depth Lower Bound for Nine-Input Sorting Networks"(PDF).Mathematical Systems Theory.24:101–116.CiteSeerX10.1.1.712.219.doi:10.1007/bf02090393.S2CID 7077160.

[codish_end_game-10] Codish, Michael; Cruz-Filipe, Luís; Ehlers, Thorsten; Müller, Mike; Schneider-Kamp, Peter (2015).Sorting Networks: to the End and Back Again.arXiv:1507.01428.Bibcode:2015arXiv150701428C.

[codish-11] Codish, Michael; Cruz-Filipe, Luís; Frank, Michael; Schneider-Kamp, Peter (2014).Twenty-Five Comparators is Optimal when Sorting Nine Inputs (and Twenty-Nine for Ten).Proc. Int'l Conf. Tools with AI (ICTAI). pp. 186–193.arXiv:1405.5754.Bibcode:2014arXiv1405.5754C.

[vvoorhis-12] Obtained by Van Voorhis lemma and the value $S (11) = 35$

[13] Ehlers, Thorsten (February 2017). "Merging almost sorted sequences yields a 24-sorter".Information Processing Letters.118:17–20.doi:10.1016/j.ipl.2016.08.005.

[sorterhunter-14] Dobbelaere, Bert."SorterHunter".GitHub.Retrieved2 Jan2022.

[bundala_zavodny-15] Bundala, D.; Závodný, J. (2014). "Optimal Sorting Networks".Language and Automata Theory and Applications.Lecture Notes in Computer Science. Vol. 8370. pp. 236–247.arXiv:1310.6271.doi:10.1007/978-3-319-04921-2_19.ISBN 978-3-319-04920-5.S2CID 16860013.

[harder-16] Harder, Jannis (2020). "An Answer to the Bose-Nelson Sorting Problem for 11 and 12 Channels".arXiv:2012.04400[cs.DS].

[parberry-17] Parberry, Ian (1991).On the Computational Complexity of Optimal Sorting Network Verification.Proc. PARLE '91: Parallel Architectures and Languages Europe, Volume I: Parallel Architectures and Algorithms, Eindhoven, the Netherlands.pp. 252–269.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

v t e Sorting algorithms
Theory	Computational complexity theory Big O notation Total order Lists Inplacement Stability Comparison sort Adaptive sort Sorting network Integer sorting X + Y sorting Transdichotomous model Quantum sort
Exchange sorts	Bubble sort Cocktail shaker sort Odd–even sort Comb sort Gnome sort Proportion extend sort Quicksort
Selection sorts	Selection sort Heapsort Smoothsort Cartesian tree sort Tournament sort Cycle sort Weak-heap sort
Insertion sorts	Insertion sort Shellsort Splaysort Tree sort Library sort Patience sorting
Merge sorts	Merge sort Cascade merge sort Oscillating merge sort Polyphase merge sort
Distribution sorts	American flag sort Bead sort Bucket sort Burstsort Counting sort Interpolation sort Pigeonhole sort Proxmap sort Radix sort Flashsort
Concurrent sorts	Bitonic sorter Batcher odd–even mergesort Pairwise sorting network Samplesort
Hybrid sorts	Block merge sort Kirkpatrick–Reisch sort Timsort Introsort Spreadsort Merge-insertion sort
Other	Topological sorting Pre-topological order Pancake sorting Spaghetti sort
Impractical sorts	Stooge sort Slowsort Bogosort