Delimiter

This article is about delimiters in computing. For delimiters in human use, seeWord divideranddigit grouping.

A stylistic depiction of values inside of a so-namedcomma-separated values(CSV) text file. The commas (shown in red) are used as field delimiters.

Adelimiteris a sequence of one or morecharactersfor specifying the boundary between separate, independent regions inplain text,mathematical expressionsor otherdata streams.An example of a delimiter is thecommacharacter, which acts as afield delimiterin a sequence ofcomma-separated values.Another example of a delimiter is the time gap used to separate letters and words in the transmission ofMorse code.

Inmathematics,delimiters are often used to specify the scope of anoperation,and can occur both as isolated symbols (e.g.,colonin "$1:4$") and as a pair of opposing-looking symbols (e.g.,angled bracketsin$\langle a,b\rangle$).

Delimiters represent one of various means of specifying boundaries in adata stream.Declarative notation,for example, is an alternate method (without the use of delimiters) that uses a length field at the start of a data stream to specify the number of characters that the data stream contains.

Overview

Delimiters may be characterized as field and record delimiters, or as bracket delimiters.

Field and record delimiters

Field delimiters separate data fields. Record delimiters separate groups of fields.

For example, theCSV formatuses a comma as the delimiter betweenfields,and anend-of-lineindicator as the delimiter betweenrecords:

fname,lname,age,salary
nancy,davolio,33,$30000
erin,borakova,28,$25250
tony,raphael,35,$28700

This specifies a simpleflat-file databasetableusing the CSV file format.

Bracket delimiters

Bracket delimiters, also called block delimiters, region delimiters, or balanced delimiters, mark both the start and end of a region of text.

Common examples of bracket delimiters include:

Delimiters	Description
()	Parentheses.TheLispprogramming language syntax is cited as recognizable primarily by its use of parentheses.
{}	Braces (also calledcurly brackets).
[]	Brackets (commonly used to denote a subscript).
<>	Angle brackets.
""	commonly used to denotestring literals.
''	commonly used to denotecharacter literals.
<??>	used to indicate XMLprocessing instructions.
/**/	used to denotecommentsin some programming languages.
<%%>	used in someweb templatesto specify language boundaries.

Conventions

Historically, computing platforms have used certain delimiters by convention.The following tables depict a few examples for comparison.

Programming languages (See also,Comparison of programming languages (syntax)).

	String Literal	End of Statement
Pascal	singlequote	semicolon
Python	doublequote, singlequote	end of line(EOL)

Field and Record delimiters(See also,ASCII,Control character).

	End of Field	End of Record	End of File
Unix-likesystems includingmacOS,AmigaOS	Tab	LF	none
Windows,MS-DOS,OS/2,CP/M	Tab	CRLF	none (except in CP/M),Control-Z
Classic Mac OS,Apple DOS,ProDOS,GS/OS	Tab	CR	none
ASCII/Unicode	UNIT SEPARATOR Position 31 (U+001F)	RECORD SEPARATOR Position 30 (U+001E)	FILE SEPARATOR Position 28 (U+001C)

Delimiter collision

Delimiter collisionis a problem that occurs when an author or programmer introduces delimiters into text without actually intending them to be interpreted as boundaries between separate regions.In the case of XML, for example, this can occur whenever an author attempts to specify anangle bracketcharacter.

In most file types there is both a field delimiter and a record delimiter, both of which are subject to collision. In the case ofcomma-separated valuesfiles, for example, field collision can occur whenever an author attempts to include a comma as part of a field value (e.g., salary = "$30,000" ), and record delimiter collision would occur whenever a field contained multiple lines. Both record and field delimiter collision occur frequently in text files.

In some contexts, a malicious user or attacker may seek to exploit this problem intentionally. Consequently, delimiter collision can be the source of securityvulnerabilitiesandexploits.Malicious users can take advantage of delimiter collision in languages such asSQLandHTMLto deploy such well-known attacks asSQL injectionandcross-site scripting,respectively.

Solutions

Because delimiter collision is a very common problem, various methods for avoiding it have been invented. Some authors may attempt to avoid the problem by choosing a delimiter character (or sequence of characters) that is not likely to appear in the data stream itself. Thisad hocapproach may be suitable, but it necessarily depends on a correct guess of what will appear in the data stream, and offers no security against malicious collisions. Other, more formal conventions are therefore applied as well.

ASCII delimited text

The ASCII and Unicode character sets were designed to solve this problem by the provision of non-printing characters that can be used as delimiters. These are the range from ASCII 28 to 31.

ASCIIDec	Symbol	Unicode Name	Common Name	Usage
28	␜	INFORMATION SEPARATOR FOUR	file separator	End of file. Or between a concatenation of what might otherwise be separate files.
29	␝	INFORMATION SEPARATOR THREE	group separator	Between sections of data. Not needed in simple data files.
30	␞	INFORMATION SEPARATOR TWO	record separator	End of a record or row.
31	␟	INFORMATION SEPARATOR ONE	unit separator	Between fields of a record, or members of a row.

The use of ASCII 31Unit separatoras a field separator and ASCII 30Record separatorsolves the problem of both field and record delimiters that appear in a text data stream.

Escape character

One method for avoiding delimiter collision is to useescape characters.From a language design standpoint, these are adequate, but they have drawbacks:

• text can be rendered unreadable when littered with numerous escape characters, a problem referred to asleaning toothpick syndrome(due to use of \ to escape / inPerlregular expressions,leading to sequences such as "\/\/" );
• text becomes difficult to parse through regular expression
• they require a mechanism to "escape the escapes" when not intended as escape characters; and
• although easy to type, they can be cryptic to someone unfamiliar with the language.
• they do not protect against injection attacks

Escape sequence

Escape sequences are similar to escape characters, except they usually consist of some kind of mnemonic instead of just a single character. One use is instring literalsthat include a doublequote ( ") character. For example inPerl,the code:

print"Nancy said \x22Hello World!\x22 to the crowd.";### use \x22

produces the same output as:

print"Nancy said \" Hello World!\ "to the crowd.";### use escape char

One drawback of escape sequences, when used by people, is the need to memorize the codes that represent individual characters (see also:character entity reference,numeric character reference).

Dual quoting delimiters

In contrast to escape sequences and escape characters, dual delimiters provide yet another way to avoid delimiter collision. Some languages, for example, allow the use of either a single quote (') or a double quote ( ") to specify a string literal. For example, inPerl:

print'Nancy said "Hello World!" to the crowd.';

produces the desired output without requiring escapes. This approach, however, only works when the string does not containbothtypes of quotation marks.

Padding quoting delimiters

In contrast to escape sequences and escape characters, padding delimiters provide yet another way to avoid delimiter collision.Visual Basic,for example, uses double quotes as delimiters. This is similar to escaping the delimiter.

print"Nancy said" "Hello World!" "to the crowd."

produces the desired output without requiring escapes. Like regular escaping it can, however, become confusing when many quotes are used. The code to print the above source code would look more confusing:

print"print" "Nancy said" "" "Hello World!" "" "to the crowd." ""

Configurable alternative quoting delimiters

In contrast to dual delimiters, multiple delimiters are even more flexible for avoiding delimiter collision.

For example, inPerl:

printqq^Nancy doesn't want to say "Hello World!" anymore.^;
printqq@Nancy doesn't want to say "Hello World!" anymore.@;
printqq(Nancy doesn't want to say "Hello World!" anymore.);

all produce the desired output through use ofquote operators,which allow any convenient character to act as a delimiter. Although this method is more flexible, few languages support it. Perl andRubyare two that do.

Content boundary

Acontent boundaryis a special type of delimiter that is specifically designed to resist delimiter collision. It works by allowing the author to specify a sequence of characters that is guaranteed to always indicate a boundary between parts in a multi-part message, with no other possible interpretation.

The delimiter is frequently generated from a random sequence of characters that is statistically improbable to occur in the content. This may be followed by an identifying mark such as aUUID,atimestamp,or some other distinguishing mark. Alternatively, the content may be scanned to guarantee that a delimiter does not appear in the text. This may allow the delimiter to be shorter or simpler, and increase the human readability of the document. (See e.g.,MIME,Here documents).

Whitespace or indentation

Some programming and computer languages allow the use ofwhitespace delimitersorindentationas a means of specifying boundaries between independent regions in text.

Regular expression syntax

See also:Regular expression examples

In specifying aregular expression,alternate delimiters may also be used to simplify the syntax formatchandsubstitutionoperations inPerl.

For example, a simple match operation may be specified in Perl with the following syntax:

$string1='Nancy said "Hello World!" to the crowd.';# specify a target string
print$string1=~m/[aeiou]+/;# match one or more vowels

The syntax is flexible enough to specify match operations with alternate delimiters, making it easy to avoid delimiter collision:

$string1='Nancy said "http://Hello/World.htm" is not a valid address.';# target string

print$string1=~m@http://@;# match using alternate regular expression delimiter
print$string1=~m{http://};# same as previous, but different delimiter
print$string1=~m!http://!;# same as previous, but different delimiter.

Here document

AHere documentallows the inclusion of arbitrary content by describing a special end sequence. Many languages support this includingPHP,bash scripts,rubyandperl.A here document starts by describing what the end sequence will be and continues until that sequence is seen at the start of a new line.

Here is an example in perl:

print<<ENDOFHEREDOC;
It's very hard to encode a string with "certain characters".

Newlines, commas, and other characters can cause delimiter collisions.
ENDOFHEREDOC

This code would print:

It's very hard to encode a string with "certain characters".

Newlines, commas, and other characters can cause delimiter collisions.

By using a special end sequence all manner of characters are allowed in the string.

ASCII armor

Although principally used as a mechanism for text encoding of binary data, ASCII armoringis a programming and systems administration technique that also helps to avoid delimiter collision in some circumstances.This technique is contrasted from the other approaches described above because it is more complicated, and therefore not suitable for small applications and simple data storage formats. The technique employs a special encoding scheme, such asbase64,to ensure that delimiter or other significant characters do not appear in transmitted data. The purpose is to prevent multilayeredescaping,i.e. fordoublequotes.

This technique is used, for example, inMicrosoft'sASP.NETweb development technology, and is closely associated with the "VIEWSTATE" component of that system.

Example

The following simplified example demonstrates how this technique works in practice.

The first code fragment shows a simpleHTML tagin which the VIEWSTATE value contains characters that are incompatible with the delimiters of the HTML tag itself:

<inputtype="hidden"name="__VIEWSTATE"value="BookTitle:Nancy doesn't say"HelloWorld! "anymore. "/>

This first code fragment is notwell-formed,and would therefore not work properly in a "real world" deployed system.

To store arbitrary text in an HTML attribute,HTML entitiescan be used. In this case """ stands in for the double-quote:

<inputtype="hidden"name="__VIEWSTATE"value="BookTitle:Nancy doesn't say &quot;Hello World!&quot; anymore."/>

Alternatively, any encoding could be used that doesn't include characters that have special meaning in the context, such as base64:

<inputtype="hidden"name="__VIEWSTATE"value="Qm9va1RpdGxlOk5hbmN5IGRvZXNuJ3Qgc2F5ICJIZWxsbyBXb3JsZCEiIGFueW1vcmUu"/>

Orpercent-encoding:

<inputtype="hidden"name="__VIEWSTATE"value="BookTitle:Nancy%20doesn%27t%20say%20%22Hello%20World!%22%20anymore."/>

This prevents delimiter collision and ensures that incompatible characters will not appear inside the HTML code, regardless of what characters appear in the original (decoded) text.

See also

• CDATA
• Decimal separator
• Delimiter-separated values
• Escape sequence
• String literal
• Tab-separated values

External links

• Data File MetaformatsfromThe Art of Unix ProgrammingbyEric Steven Raymond