Query understanding

Query understandingis the process of inferring theintentof asearch engineuser by extracting semantic meaning from the searcher’s keywords.Query understanding methods generally take place before the search engineretrievesandranksresults. It is related tonatural language processingbut specifically focused on the understanding of search queries. Query understanding is at the heart of technologies likeAmazon Alexa,Apple'sSiri.Google Assistant,IBM'sWatson,andMicrosoft'sCortana.

Methods

Tokenization

Tokenizationis the process of breaking up atext stringinto words or other meaningful elements called tokens. Typically, tokenization occurs at the word level. However, it is sometimes difficult to define what is meant by a "word". Often a tokenizer relies on simple heuristics, such as splitting the string on punctuation andwhitespace characters.Tokenization is more challenging in languages without spaces between words, such asChineseandJapanese.Tokenizing text in these languages requires the use ofword segmentationalgorithms.

Spelling correction

Spelling correctionis the process of automatically detecting and correcting spelling errors in search queries. Most spelling correction algorithms are based on alanguage model,which determines thea priori probabilityof an intended query, and an error model (typically anoisy channel model), which determines the probability of a particular misspelling, given an intended query.

Stemming and lemmatization

Many, but not all, languagesinflectwords to reflect their role in the utterance they appear in: a word such as *care* may appear as, besides the base form. as *cares*, *cared*, *caring*, and others. The variation between various forms of a word is likely to be of little importance for the relatively coarse-grained model of meaning involved in a retrieval system, and for this reason the task of conflating the various forms of a word is a potentially useful technique to increase recall of a retrieval system.

The languages of the world vary in how much morphological variation they exhibit, and for some languages there are simple methods to reduce a word in query to itslemmaorrootform or itsstem.For some other languages, this operation involves non-trivial string processing. A noun in English typically appears in four variants: *cat* *cat's* *cats* *cats'* or *child* *child´s* *children* *children's*. Other languages have more variation.Finnish,e.g., potentially exhibits about 5000 forms for a noun,and for many languages the inflectional forms are not limited toaffixesbut change the core of the word itself.

Stemming algorithms, also known as stemmers, typically use a collection of simple rules to removesuffixesintended to model the language’s inflection rules.

More advanced methods,lemmatisationmethods, group together the inflected forms of a word through more complex rule sets based on a word’spart of speechor its record in alexical database,transforming an inflected word through lookup or a series of transformations to its lemma. For a long time, it was taken to be proven that morphological normalisation by and large did not help retrieval performance.

Once the attention of the information retrieval field moved to languages other than English, it was found that for some languages there were obvious gains to be found.

Entity recognition

Entity recognition is the process of locating and classifying entities within a text string.Named-entity recognitionspecifically focuses onnamed entities,such as names of people, places, and organizations. In addition, entity recognition includes identifying concepts in queries that may be represented by multi-word phrases. Entity recognition systems typically use grammar-based linguistic techniques or statisticalmachine learningmodels.

Query rewriting

Query rewriting is the process of automatically reformulating a search query to more accurately capture its intent.Query expansionadds additional query terms, such as synonyms, in order to retrieve more documents and thereby increase recall. Query relaxation removes query terms to reduce the requirements for a document to match the query, thereby also increasingrecall.Other forms of query rewriting, such as automatically converting consecutive query terms intophrasesand restricting query terms to specificfields,aim to increaseprecision.Apache Lucene search engineuses query rewrite to transform complex queries to more primitive queries, such as expressions with wildcards (e.g. quer*) into a boolean query of the matching terms from the index (such as query OR queries).

See also

• Daniel Tunkelang's blog on Query Understanding
• ACM SIGIR 2010 Workshop Report on Query Representation and Understanding
• Proceedings of ACM SIGIR 2011 Workshop on Query Representation and Understanding
• ACM WSDM 2016 Workshop on Query Understanding for Search on All Devices
• Query Understanding for Search Engines (Yi Chang and Hongbo Deng, Eds.)