UTF-32

UTF-32(32-bitUnicode Transformation Format) is a fixed-lengthencodingused to encode Unicodecode pointsthat uses exactly 32 bits (fourbytes) per code point (but a number of leading bits must be zero as there are far fewer than 2³²Unicode code points, needing actually only 21 bits).In contrast, all other Unicode transformation formats are variable-length encodings. Each 32-bit value in UTF-32 represents one Unicode code point and is exactly equal to that code point's numerical value.

The main advantage of UTF-32 is that the Unicode code points are directly indexed. Finding theNthcode point in a sequence of code points is aconstant-timeoperation. In contrast, avariable-length coderequireslinear-timeto countNcode points from the start of the string. This makes UTF-32 a simple replacement in code that usesintegersthat are incremented by one to examine each location in astring,as was commonly done forASCII.However, Unicode code points are rarely processed in complete isolation, such ascombining charactersequences and for emoji.

The main disadvantage of UTF-32 is that it is space-inefficient, using fourbytesper code point, including 11 bits that are always zero. Characters beyond theBMPare relatively rare in most texts (except, for example, in the case of texts with some popular emojis), and can typically be ignored for sizing estimates. This makes UTF-32 close to twice the size ofUTF-16.It can be up to four times the size ofUTF-8depending on how many of the characters are in theASCIIsubset.

History

The originalISO/IEC 10646standard defines a 32-bitencoding formcalledUCS-4,in which each code point in theUniversal Character Set(UCS) is represented by a 31-bit value from 0 to 0x7FFFFFFF (the sign bit was unused and zero). In November 2003, Unicode was restricted by RFC 3629 to match the constraints of theUTF-16encoding: explicitly prohibiting code points greater than U+10FFFF (and also the high and low surrogates U+D800 through U+DFFF). This limited subset defines UTF-32.Although the ISO standard had (as of 1998 in Unicode 2.1) "reserved for private use" 0xE00000 to 0xFFFFFF, and 0x60000000 to 0x7FFFFFFFthese areas were removed in later versions. Because the Principles and Procedures document ofISO/IEC JTC 1/SC 2Working Group 2 states that all future assignments of code points will be constrained to the Unicode range, UTF-32 will be able to represent all UCS code points and UTF-32 and UCS-4 are identical.

Utility of fixed width

A fixed number of bytes per code point has theoretical advantages, but each of these has problems in reality:

• Truncation becomes easier, but not significantly so compared toUTF-8andUTF-16(both of which can search backwards for the point to truncate by looking at 2–4 code units at most).
• Finding theNth characterin a string. For fixed width, this is simply aO(1) problem,while it isO(n) problemin a variable-width encoding. Novice programmers often vastly overestimate how useful this is.Also what a user might call a "character" is still variable-width, for instance thecombining characterácould be 2 code points, the emoji👨‍🦲is three,and the ligatureffis one.
• Quickly knowing the "width" of a string. However even "fixed width" fonts have varying width, oftenCJK ideographsare twice as wide,plus the already-mentioned problems with the number of code points not being equal to the number of characters.

Use

The main use of UTF-32 is in internal APIs where the data is single code points orglyphs,rather than strings of characters. For instance, in modern text rendering, it is commonthat the last step is to build a list of structures each containingcoordinates (x,y),attributes, and a single UTF-32 code point identifying the glyph to draw. Often non-Unicode information is stored in the "unused" 11 bits of each word.

Use of UTF-32 strings on Windows (wherewchar_tis 16 bits) is almost non-existent. On Unix systems, UTF-32 strings are sometimes, but rarely, used internally by applications, due to the typewchar_tbeing defined as 32 bit.Pythonversions up to 3.2 can be compiled to use them instead ofUTF-16;from version 3.3 onward all Unicode strings are stored in UTF-32 but with leading zero bytes optimized away "depending on the [code point] with the largest Unicode ordinal (1, 2, or 4 bytes)" to make all code points that size.Seed7andLassoprogramming languages encode all strings with UTF-32, in the belief that direct inde xing is important, whereas theJuliaprogramming language moved away from builtin UTF-32 support with its 1.0 release, simplifying the language to having only UTF-8 strings (with all the other encodings considered legacy and moved out of the standard library to package) following the "UTF-8 Everywhere Manifesto".

Variants

Though technically invalid, the surrogate halves are often encoded and allowed. This allows invalid UTF-16 (such as Windows filenames) to be translated to UTF-32, similar to how theWTF-8variant of UTF-8 works. Sometimes paired surrogates are encoded instead of non-BMP characters, similar toCESU-8.Due to the large number of unused 32-bit values, it is also possible to preserve invalid UTF-8 by using non-Unicode values to encode UTF-8 errors, though there is no standard for this.

See also

• Comparison of Unicode encodings

External links

• The Unicode Standard 5.0.0, chapter 3– formally defines UTF-32 in § 3.9, D90 (PDF page 40) and § 3.10, D99-D101 (PDF page 45)
• Unicode Standard Annex #19– formally defined UTF-32 for Unicode 3.x (March 2001; last updated March 2002)
• Registration of new charsets: UTF-32, UTF-32BE, UTF-32LE– announcement of UTF-32 being added to the IANA charset registry (April 2002)