UTF-7
Language(s) | International |
---|---|
Standard | RFC2152 |
Classification | Unicode Transformation Format,ASCII armor,variable-width encoding,stateful encoding |
Transforms / Encodes | ISO/IEC 10646(Unicode) |
Preceded by | HZ-GB-2312 |
Succeeded by | UTF-8over8BITMIME |
UTF-7(7-bitUnicode Transformation Format) is an obsolete variable-length character encoding for representingUnicodetext using a stream ofASCIIcharacters. It was originally intended to provide a means of encodingUnicodetext for use inInternetE-mailmessages that was more efficient than the combination ofUTF-8withquoted-printable.
UTF-7 (according to its RFC) isn't a "Unicode Transformation Format",as the definition can only encode code points in theBMP(the first 65536 Unicode code points, which does not includeemojisand many other characters). However if a UTF-7 translator is to/fromUTF-16then it can (and probably does)[citation needed]encode each surrogate half as though it was a 16-bit code point, and thus can encode all code points. It is unclear if other UTF-7 software (such as translators to UTF-32 or UTF-8) support this.
UTF-7 has never been an official standard of theUnicode Consortium.It is known to have security issues, which is why software has been changed to disable its use.[1]It is prohibited inHTML 5.[2][3]
Motivation
[edit]MIME,the modern standard for e-mail formats, forbids encoding ofheadersusing byte values above the ASCII range. Although MIME allows encoding the message body in variouscharacter sets(broader than ASCII), the underlying transmission infrastructure (SMTP,the main E-mail transfer standard) is still not guaranteed to be8-bit clean.Therefore, a non-trivial content transfer encoding has to be applied in case of doubt. UnfortunatelyBase64has a disadvantage of making evenASCIIcharacters unreadable in non-MIME clients. On the other hand, UTF-8 combined withquoted-printableproduces a very size-inefficient format requiring 6–9 bytes for non-ASCII characters from theBMPand 12 bytes for characters outside the BMP.
Provided certain rules are followed during encoding, UTF-7 can be sent in e-mail without using an underlying MIMEtransfer encoding,but still must be explicitly identified as the text character set. In addition, if used within e-mail headers such as "Subject:", UTF-7 must be contained in MIMEencoded wordsidentifying the character set. Since encoded words force use of eitherquoted-printableorBase64,UTF-7 was designed to avoid using the = sign as an escape character to avoid double escaping when it is combined with quoted-printable (or its variant, the RFC 2047/1522 "Q" -encoding of headers).
UTF-7 is generally not used as a native representation within applications as it is very awkward to process. Despite its size advantage over the combination of UTF-8 with either quoted-printable or Base64, the now defunctInternet Mail Consortiumrecommended against its use.[4]
8BITMIMEhas also been introduced, which reduces the need to encode message bodies in a 7-bit format.
A modified form of UTF-7 (sometimes dubbed 'mUTF-7'[5]) was used in theInternet Message Access Protocol (IMAP)e-mail retrieval protocol, version 4 rev 1, for "international" mailbox names.[6] The following version, IMAP version 4 rev 2, uses UTF-8 instead.[7]
Description
[edit]UTF-7 was first proposed as an experimental protocol in RFC 1642,A Mail-Safe Transformation Format of Unicode.ThisRFChas been made obsolete by RFC 2152, an informational RFC which never became a standard. As RFC 2152 clearly states, the RFC "does not specify an Internet standard of any kind". Despite this, RFC 2152 is quoted as the definition of UTF-7 in the IANA's list of charsets. Neither is UTF-7 a Unicode Standard.The Unicode Standard 5.0only lists UTF-8, UTF-16 and UTF-32. There is also a modified version, specified in RFC 2060, which is sometimes identified as UTF-7.
Some characters can be represented directly as single ASCII bytes. The first group is known as "direct characters" and contains 62 Alpha numeric characters and 9 symbols:' ( ), -. /:?
.The direct characters are safe to include literally. The other main group, known as "optional direct characters", contains all other printable characters in the rangeU+0020–U+007E except~ \ +
and space (the characters\
and~
being excluded due to being redefined in "variants of ASCII" such asJIS-Roman). Using the optional direct characters reduces size and enhances human readability but also increases the chance of breakage by things like badly designed mail gateways and may require extra escaping when used in encoded words for header fields.
Space, tab, carriage return and line feed may also be represented directly as single ASCII bytes. However, if the encoded text is to be used in e-mail, care is needed to ensure that these characters are used in ways that do not require further content transfer encoding to be suitable for e-mail. The plus sign (+
)maybe encoded as+-
.
Other characters must be encoded inUTF-16(hence U+10000 and higher would be encoded into two surrogates), and then inmodified Base64.The start of these blocks of modified Base64-encoded UTF-16 is indicated by a+
sign. The end is indicated by any character not in the modified Base64 set. If the character after the modified Base64 is a-
(ASCIIhyphen-minus) then it is consumed by the decoder and decoding resumes with the next character. Otherwise decoding resumes with the character after the Base64.
Examples
[edit]- "
Hello, World!
"is encoded as"Hello, World+ACE-
" - "
1 + 1 = 2
"is encoded as"1 +- 1 +AD0- 2
" - "
£1
"is encoded as"+AKM-1
".The Unicode code point for thepound signis U+00A3 which converts intomodified Base64as in the table below. There are two bits left over, which are padded to 0.
Hex digit | 0 | 0 | A | 3 | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Bit pattern | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 |
Index | 0 | 10 | 12 | |||||||||||||||
Base64-Encoded | A | K | M |
Algorithm for encoding and decoding
[edit]Encoding
[edit]First, an encoder must decide which characters to represent directly in ASCII form, which+
have to be escaped as+-
,and which to place in blocks of Unicode characters. The expansion cost of UTF-7 can be high: for example, the character sequence U+10FFFF U+0077 U+10FFFF is 9 bytes in UTF-8, but 17 bytes in UTF-7. (At worst, treating every codepoint as a sequence in its own right produces the maximum expansion of 5x, e.g. when encoding@@
as+AEA-+AEA-
.) Each Unicode sequence must be encoded using the following procedure, then surrounded by the appropriate delimiters.
Using the £† (U+00A3 U+2020) character sequence as an example:
- Express the character's Unicode numbers (UTF-16) in binary:
- 0x00A3 → 0000 0000 1010 0011
- 0x2020 → 0010 0000 0010 0000
- Concatenate the binary sequences:
0000 0000 1010 0011 and 0010 0000 0010 0000 → 0000 0000 1010 0011 0010 0000 0010 0000 - Regroup the binary into groups of six bits, starting from the left:
0000 0000 1010 0011 0010 0000 0010 0000 → 000000 001010 001100 100000 001000 00 - If the last group has fewer than six bits, add trailing zeros:
000000 001010 001100 100000 001000 00 → 000000 001010 001100 100000 001000 000000 - Replace each group of six bits with a respective Base64 code:
000000 001010 001100 100000 001000 000000 → AKMgIA
Decoding
[edit]First an encoded data must be separated into plain ASCII text chunks (including+es followed by a dash) and nonempty Unicode blocks as mentioned in the description section. Once this is done, each Unicode block must be decoded with the following procedure (using the result of the encoding example above as our example)
- Express each Base64 code as the bit sequence it represents:
AKMgIA → 000000 001010 001100 100000 001000 000000 - Regroup the binary into groups of sixteen bits, starting from the left:
000000 001010 001100 100000 001000 000000 → 0000000010100011 0010000000100000 0000 - If there is an incomplete group at the end containing only zeros, discard it (if the incomplete group contains any ones, the code is invalid):
0000000010100011 0010000000100000 - Each group of 16 bits is a character's Unicode (UTF-16) number and can be expressed in other forms:
0000 0000 1010 0011 ≡ 0x00A3 ≡ 16310
Byte order mark
[edit]A byte order mark (BOM) is an optional special byte sequence at the very start of a stream or file that, without being data itself, indicates the encoding used for the data that follows; it can be used in the absence of metadata that denotes the encoding. For a given encoding scheme, it's that scheme's representation of Unicode code pointU+FEFF
.[8]
While it's typically a single, fixed byte sequence, in UTF-7 four variations may appear, because the last 2 bits of the 4th byte of the UTF-7 encoding ofU+FEFF
belong to thefollowingcharacter, resulting in 4 possible bit patterns and therefore 4 different possible bytes in the 4th position. See the UTF-7 entry in thetable of Unicode byte order marks.[9]
Security
[edit]UTF-7 allows multiple representations of the same source string. In particular, ASCII characters can be represented as part of Unicode blocks. As such, if standard ASCII-based escaping or validation processes are used on strings that may be later interpreted as UTF-7, then Unicode blocks may be used to slip malicious strings past them. To mitigate this problem, systems should perform decoding before validation and should avoid attempting to autodetect UTF-7.
Older versions ofInternet Explorercan be tricked into interpreting the page as UTF-7. This can be used for across-site scriptingattack as the<
and>
marks can be encoded as+ADw-
and+AD4-
in UTF-7, which most validators let through as simple text.[10]
UTF-7 is considered obsolete, at least for Microsoft software (.NET), with code paths previously supporting it intentionally broken (to prevent security issues) in.NET 5, in 2020.[1]
See also
[edit]References
[edit]- ^ab"Breaking change: UTF-7 code paths are obsolete".docs.microsoft.Retrieved8 January2021.
- ^"8.2.2.3. Character encodings".HTML 5.1 Standard.W3C.
- ^"12.2.3.3 Character encodings".HTML Living Standard.WHATWG.
- ^"Using International Characters in Internet Mail".Internet Mail Consortium.1 August 1998. Archived fromthe originalon 7 September 2015.
- ^"Configuration Manual".Dovecot Documentation.8 February 2023. Sec. "Mail Location Settings".Retrieved28 February2023.
Store mailbox names on disk using UTF-8 instead of modified UTF-7 (mUTF-7).
- ^Crispin, Mark (March 2003).INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1.Network Working Group.doi:10.17487/RFC3501.RFC3501.Proposed Standard.sec. 5.1.3 "Mailbox International Naming Convention". Obsoleted byRFC9051.Updated byRFC7817,8437,8474,4551,4469,5182,4466,5032and5738.ObsoletesRFC2060.
In modified UTF-7, printableUS-ASCIIcharacters, except for "&", represent themselves…. The character "&" (0x26) is represented by the two-octet sequence "&-". All other characters… are represented in modified BASE64….
- ^Melnikov, Alexey; Leiba, Barry (August 2021).Internet Message Access Protocol (IMAP) - Version 4rev2.IETF.doi:10.17487/RFC9051.ISSN2070-1721.RFC9051.Proposed Standard.sec. 5.1. "Mailbox Naming". ObsoletesRFC3501
In IMAP4rev2, mailbox names are encoded in Net-Unicode (this differs from IMAP4rev1).
- ^"FAQ – UTF-8, UTF-16, UTF-32 & BOM".
- ^"Clarify guidance for use of a BOM as a UTF-8 encoding signature"(PDF).Retrieved17 January2024.
- ^"ArticleUtf7 - doctype-mirror - UTF-7: the case of the missing charset - Mirror of Google Doctype - Google Project Hosting".14 October 2011.Retrieved29 June2012.