Wikipedia

Character encoding

(Redirected fromCode unit)

Character encodingis the process of assigning numbers to graphicalcharacters,especially the written characters of human language, allowing them to be stored, transmitted, and transformed using computers.[1]The numerical values that make up a character encoding are known ascode pointsand collectively comprise a code space, acode page,orcharacter map.

Punched tapewith the word "Wikipedia" encoded inASCII.Presence and absence of a hole represents 1 and 0, respectively; for example, W is encoded as1010111.

Early character codes associated with the optical or electricaltelegraphcould only represent a subset of the characters used inwritten languages,sometimes restricted toupper case letters,numeralsand somepunctuationonly. The advent of digital computer systems allows more elaborate encodings codes (such asUnicode) to support hundreds of written languages.[disputed–discuss]

Themost popular character encodingon theWorld Wide WebisUTF-8,which is used in 98.2% of surveyed web sites, as of May 2024.[2]Inapplication programsandoperating systemtasks, both UTF-8 andUTF-16are popular options.[3]

History

edit

The history of character codes illustrates the evolving need for machine-mediated character-based symbolic information over a distance, using once-novel electrical means. The earliest codes were based upon manual and hand-written encoding and cyphering systems, such asBacon's cipher,Braille,international maritime signal flags,and the 4-digit encoding of Chinese characters for aChinese telegraph code(Hans Schjellerup,1869). With the adoption of electrical and electro-mechanical techniques these earliest codes were adapted to the new capabilities and limitations of the early machines. The earliest well-known electrically transmitted character code,Morse code,introduced in the 1840s, used a system of four "symbols" (short signal, long signal, short space, long space) to generate codes of variable length. Though some commercial use of Morse code was via machinery, it was often used as a manual code, generated by hand on atelegraph keyand decipherable by ear, and persists inamateur radioandaeronauticaluse. Most codes are of fixed per-character length or variable-length sequences of fixed-length codes (e.g.Unicode).[4]

Common examples of character encoding systems include Morse code, theBaudot code,theAmerican Standard Code for Information Interchange(ASCII) and Unicode. Unicode, a well-defined and extensible encoding system, has replaced most earlier character encodings, but the path of code development to the present is fairly well known.

The Baudot code, a five-bitencoding, was created byÉmile Baudotin 1870, patented in 1874, modified by Donald Murray in 1901, and standardized by CCITT as International Telegraph Alphabet No. 2 (ITA2) in 1930. The namebaudothas been erroneously applied to ITA2 and its many variants. ITA2 suffered from many shortcomings and was often improved by many equipment manufacturers, sometimes creating compatibility issues. In 1959 the U.S. military defined itsFieldatacode, a six-or seven-bit code, introduced by the U.S. Army Signal Corps. While Fieldata addressed many of the then-modern issues (e.g. letter and digit codes arranged for machine collation), it fell short of its goals and was short-lived. In 1963 the first ASCII code was released (X3.4-1963) by the ASCII committee (which contained at least one member of the Fieldata committee, W. F. Leubbert), which addressed most of the shortcomings of Fieldata, using a simpler code. Many of the changes were subtle, such as collatable character sets within certain numeric ranges. ASCII63 was a success, widely adopted by industry, and with the follow-up issue of the 1967 ASCII code (which added lower-case letters and fixed some "control code" issues) ASCII67 was adopted fairly widely. ASCII67's American-centric nature was somewhat addressed in the EuropeanECMA-6standard.[5]

Hollerith 80-column punch card with EBCDIC character set

Herman Hollerithinvented punch card data encoding in the late 19th century to analyze census data. Initially, each hole position represented a different data element, but later, numeric information was encoded by numbering the lower rows 0 to 9, with a punch in a column representing its row number. Later Alpha betic data was encoded by allowing more than one punch per column. Electromechanicaltabulating machinesrepresented date internally by the timing of pulses relative to the motion of the cards through the machine. WhenIBMwent to electronic processing, starting with theIBM 603Electronic Multiplier, it used a variety of binary encoding schemes that were tied to the punch card code.

IBM used severalBinary Coded Decimal(BCD) six-bit character encoding schemes, starting as early as 1953 in its702[6]and704computers, and in its later7000 Seriesand1400 series,as well as in associated peripherals. Since the punched card code then in use only allowed digits, upper-case English letters and a few special characters, six bits were sufficient. These BCD encodings extended existing simple four-bit numeric encoding to include Alpha betic and special characters, mapping them easily to punch-card encoding which was already in widespread use. IBM's codes were used primarily with IBM equipment; other computer vendors of the era had their own character codes, often six-bit, but usually had the ability to read tapes produced on IBM equipment. These BCD encodings were the precursors of IBM'sExtended Binary-Coded Decimal Interchange Code(usually abbreviated as EBCDIC), an eight-bit encoding scheme developed in 1963 for theIBM System/360that featured a larger character set, including lower case letters.

In trying to develop universally interchangeable character encodings, researchers in the 1980s faced the dilemma that, on the one hand, it seemed necessary to add more bits to accommodate additional characters, but on the other hand, for the users of the relatively small character set of the Latin Alpha bet (who still constituted the majority of computer users), those additional bits were a colossal waste of then-scarce and expensive computing resources (as they would always be zeroed out for such users). In 1985, the average personal computer user'shard disk drivecould store only about 10 megabytes, and it cost approximately US$250 on the wholesale market (and much higher if purchased separately at retail),[7]so it was very important at the time to make every bit count.

The compromise solution that was eventually found anddeveloped into Unicode[vague]was to break the assumption (dating back to telegraph codes) that each character should always directly correspond to a particular sequence of bits. Instead, characters would first be mapped to a universal intermediate representation in the form of abstract numbers calledcode points.Code points would then be represented in a variety of ways and with various default numbers of bits per character (code units) depending on context. To encode code points higher than the length of the code unit, such as above 256 for eight-bit units, the solution was to implementvariable-length encodingswhere an escape sequence would signal that subsequent bits should be parsed as a higher code point.

Terminology

edit

Informally, the terms "character encoding", "character map", "character set" and "code page" are often used interchangeably.[8]Historically, the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units — usually with a single character per code unit. However, due to the emergence of more sophisticated character encodings, the distinction between these terms has become important.

  • Acharacteris a minimal unit of text that has semantic value.[8][9]
  • Acharacter setis a collection of elements used to represent text.[8][9]For example, theLatin Alpha betandGreek Alpha betare both character sets.
  • Acoded character setis a character set mapped to a set of unique numbers.[9]For historical reasons, this is also often referred to as acode page.[8]
  • Acharacter repertoireis the set of characters that can be represented by a particular coded character set.[9][10]The repertoire may be closed, meaning that no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series); or it may be open, allowing additions (as is the case with Unicode and to a limited extentWindows code pages).[10]
  • Acode pointis a value or position of a character in a coded character set.[9]
  • Acode spaceis the range of numerical values spanned by a coded character set.[9][11]
  • Acode unitis the minimum bit combination that can represent a character in a character encoding (incomputer scienceterms, it is thewordsize of the character encoding).[9][11]For example, common code units include 7-bit, 8-bit, 16-bit, and 32-bit. In some encodings, some characters are encoded using multiple code units; such an encoding is referred to as avariable-width encoding.

Code pages

edit
Main article:Code page

"Code page" is a historical name for a coded character set.

Originally, a code page referred to a specificpage numberin the IBM standard character set manual, which would define a particular character encoding.[12]Other vendors, includingMicrosoft,SAP,andOracle Corporation,also published their own sets of code pages; the most well-known code page suites are "Windows"(based on Windows-1252) and" IBM "/" DOS "(based oncode page 437).

Despite no longer referring to specific page numbers in a standard, many character encodings are still referred to by their code page number; likewise, the term "code page" is often still used to refer to character encodings in general.

The term "code page" is not used in Unix or Linux, where "charmap" is preferred, usually in the larger context of locales. IBM's Character Data Representation Architecture (CDRA) designates entities with coded character set identifiers (CCSIDs), each of which is variously called a "charset", "character set", "code page", or "CHARMAP".[11]

Code units

edit

The code unit size is equivalent to the bit measurement for the particular encoding:

Code points

edit

A code point is represented by a sequence of code units. The mapping is defined by the encoding. Thus, the number of code units required to represent a code point depends on the encoding:

  • UTF-8: code points map to a sequence of one, two, three or four code units.
  • UTF-16: code units are twice as long as 8-bit code units. Therefore, any code point with a scalar value less than U+10000 is encoded with a single code unit. Code points with a value U+10000 or higher require two code units each. These pairs of code units have a unique term in UTF-16:"Unicode surrogate pairs".
  • UTF-32: the 32-bit code unit is large enough that every code point is represented as a single code unit.
  • GB 18030: multiple code units per code point are common, because of the small code units. Code points are mapped to one, two, or four code units.[13]

Characters

edit
Main article:Character (computing)

Exactly what constitutes a character varies between character encodings.

For example, for letters withdiacritics,there are two distinct approaches that can be taken to encode them: they can be encoded either as a single unified character (known as a precomposed character), or as separate characters that combine into a singleglyph.The former simplifies the text handling system, but the latter allows any letter/diacritic combination to be used in text.Ligaturespose similar problems.

Exactly how to handleglyphvariants is a choice that must be made when constructing a particular character encoding. Some writing systems, such as Arabic and Hebrew, need to accommodate things likegraphemesthat are joined in different ways in different contexts, but represent the same semantic character.

Unicode encoding model

edit

Unicodeand its parallel standard, the ISO/IEC 10646Universal Character Set,together constitute a unified standard for character encoding. Rather than mapping characters directly tobytes,Unicode separately defines a coded character set that maps characters to unique natural numbers (code points), how those code points are mapped to a series of fixed-size natural numbers (code units), and finally how those units are encoded as a stream of octets (bytes). The purpose of this decomposition is to establish a universal set of characters that can be encoded in a variety of ways. To describe this model precisely, Unicode uses its own set of terminology to describe its process:[11]

Anabstract character repertoire(ACR) is the full set of abstract characters that a system supports. Unicode has an open repertoire, meaning that new characters will be added to the repertoire over time.

Acoded character set(CCS) is afunctionthat maps characters tocode points(each code point represents one character). For example, in a given repertoire, the capital letter "A" in the Latin Alpha bet might be represented by the code point 65, the character "B" by 66, and so on. Multiple coded character sets may share the same character repertoire; for exampleISO/IEC 8859-1and IBM code pages 037 and500all cover the same repertoire but map them to different code points.

Acharacter encoding form(CEF) is the mapping of code points tocode unitsto facilitate storage in a system that represents numbers as bit sequences of fixed length (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units can only directly represent code points 0 to 65,535 in each unit, but larger code points (say, 65,536 to 1.4 million) could be represented by using multiple 16-bit units. This correspondence is defined by a CEF.

Acharacter encoding scheme(CES) is the mapping of code units to a sequence of octets to facilitate storage on an octet-based file system or transmission over an octet-based network. Simple character encoding schemes includeUTF-8,UTF-16BE,UTF-32BE,UTF-16LE,andUTF-32LE;compound character encoding schemes, such asUTF-16,UTF-32andISO/IEC 2022,switch between several simple schemes by using abyte order markorescape sequences;compressing schemes try to minimize the number of bytes used per code unit (such asSCSUandBOCU).

AlthoughUTF-32BEandUTF-32LEare simpler CESes, most systems working with Unicode use eitherUTF-8,which isbackward compatiblewith fixed-length ASCII and maps Unicode code points to variable-length sequences of octets, orUTF-16BE,[citation needed]which isbackward compatiblewith fixed-length UCS-2BE and maps Unicode code points to variable-length sequences of 16-bit words. Seecomparison of Unicode encodingsfor a detailed discussion.

Finally, there may be ahigher-level protocolwhich supplies additional information to select the particular variant of aUnicodecharacter, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is theXMLattribute xml:lang.

The Unicode model uses the term "character map" for other systems which directly assign a sequence of characters to a sequence of bytes, covering all of the CCS, CEF and CES layers.[11]

Unicode code points

edit

In Unicode, a character can be referred to as 'U+' followed by its codepoint value in hexadecimal. The range of valid code points (the codespace) for the Unicode standard is U+0000 to U+10FFFF, inclusive, divided in 17planes,identified by the numbers 0 to 16. Characters in the range U+0000 to U+FFFF are in plane 0, called theBasic Multilingual Plane(BMP). This plane contains the most commonly-used characters. Characters in the range U+10000 to U+10FFFF in the other planes are calledsupplementary characters.

The following table shows examples of code point values:

Character Unicode code point Glyph
Latin A U+0041 Α
Latin sharp S U+00DF ß
Han for East U+6771 Đông
Ampersand U+0026 &
Inverted exclamation mark U+00A1 ¡
Section sign U+00A7 §

Example

edit

Consider astringof the letters "ab̲c𐐀" —that is, a string containing a Unicode combining character (U+0332̲COMBINING LOW LINE) as well as a supplementary character (U+10400𐐀DESERET CAPITAL LETTER LONG I). This string has several Unicode representations which are logically equivalent, yet while each is suited to a diverse set of circumstances or range of requirements:

  • Fourcomposed characters:
    a,b̲,c,𐐀
  • Fivegraphemes:
    a,b,_,c,𐐀
  • Five Unicodecode points:
    U+0061,U+0062,U+0332,U+0063,U+10400
  • Five UTF-32 code units (32-bit integer values):
    0x00000061,0x00000062,0x00000332,0x00000063,0x00010400
  • Six UTF-16 code units (16-bit integers)
    0x0061,0x0062,0x0332,0x0063,0xD801,0xDC00
  • Nine UTF-8 code units (8-bit values, orbytes)
    0x61,0x62,0xCC,0xB2,0x63,0xF0,0x90,0x90,0x80

Note in particular that 𐐀 is represented with either one 32-bit value (UTF-32), two 16-bit values (UTF-16), or four 8-bit values (UTF-8). Although each of those forms uses the same total number of bits (32) to represent the glyph, it is not obvious how the actual numeric byte values are related.

Transcoding

edit

As a result of having many character encoding methods in use (and the need for backward compatibility with archived data), many computer programs have been developed to translate data between character encoding schemes, a process known astranscoding.Some of these are cited below.

Cross-platform:

  • Web browsers– most modern web browsers feature automaticcharacter encoding detection.On Firefox 3, for example, see the View/Character Encoding submenu.
  • iconv– a program and standardized API to convert encodings
  • luit– a program that converts encoding of input and output to programs running interactively
  • International Components for Unicode– A set of C and Java libraries to perform charset conversion. uconv can be used from ICU4C.

Windows:

  • Encoding.Convert –.NET API[14]
  • MultiByteToWideChar/WideCharToMultiByte – to convert from ANSI to Unicode & Unicode to ANSI[15][16]

Common character encodings

edit
Main article:Popularity of text encodings

Themost used character encodingon thewebisUTF-8,used in 98.2% of surveyed web sites, as of May 2024.[2]Inapplication programsandoperating systemtasks, both UTF-8 andUTF-16are popular options.[3][17]

See also

edit

References

edit
  1. ^"Character Encoding Definition".The Tech Terms Dictionary.24 September 2010.
  2. ^ab"Usage Survey of Character Encodings broken down by Ranking".W3Techs.Retrieved29 April2024.
  3. ^ab"Charset".Android Developers.Retrieved2 January2021.Android note: The Android platform default is always UTF-8.
  4. ^Tom Henderson (17 April 2014)."Ancient Computer Character Code Tables – and Why They're Still Relevant".Smartbear. Archived fromthe originalon 30 April 2014.Retrieved29 April2014.
  5. ^Tom Jennings (20 April 2016)."An annotated history of some character codes".Sensitive Research.Retrieved1 November2018.
  6. ^"IBM Electronic Data-Processing Machines Type 702 Preliminary Manual of Information"(PDF).1954. p. 80. 22-6173-1.Archived(PDF)from the original on 9 October 2022 – via bitsavers.org.
  7. ^Strelho, Kevin (15 April 1985)."IBM Drives Hard Disks to New Standards".InfoWorld.Popular Computing Inc. pp. 29–33.Retrieved10 November2020.
  8. ^abcdShawn Steele (15 March 2005)."What's the difference between an Encoding, Code Page, Character Set and Unicode?".Microsoft Docs.
  9. ^abcdefg"Glossary of Unicode Terms".Unicode Consortium.
  10. ^ab"Chapter 3: Conformance".The Unicode Standard Version 15.0 – Core Specification(PDF).Unicode Consortium. September 2022.ISBN978-1-936213-32-0.
  11. ^abcdeWhistler, Ken; Freytag, Asmus (11 November 2022)."UTR#17: Unicode Character Encoding Model".Unicode Consortium.Retrieved12 August2023.
  12. ^"VT510 Video Terminal Programmer Information".Digital Equipment Corporation(DEC). 7.1. Character Sets - Overview.Archivedfrom the original on 26 January 2016.Retrieved15 February2017.In addition to traditionalDECandISOcharacter sets, which conform to the structure and rules ofISO 2022,theVT510supports a number of IBM PC code pages (page numbersin IBM's standard character set manual) inPCTermmode to emulate theconsole terminalof industry-standard PCs.
  13. ^"Terminology (The Java Tutorials)".Oracle.Retrieved25 March2018.
  14. ^"Encoding.Convert Method".Microsoft.NET Framework Class Library.
  15. ^"MultiByteToWideChar function (stringapiset.h)".Microsoft Docs.13 October 2021.
  16. ^"WideCharToMultiByte function (stringapiset.h)".Microsoft Docs.9 August 2022.
  17. ^Galloway, Matt (9 October 2012)."Character encoding for iOS developers. Or UTF-8 what now?".Matt Galloway.Retrieved2 January2021.in reality, you usually just assume UTF-8 since that is by far the most common encoding.

Further reading

edit
edit
Wikimedia Commons has media related toEncodings.
Wikiversity has learning resources aboutCharacter encoding
Retrieved from "https://en.wikipedia.org/w/index.php?title=Character_encoding&oldid=1258452884#Terminology"