A token is a string of characters, categorized according to the rules as a symbol (e.g., IDENTIFIER, NUMBER, COMMA). The process of forming tokens from an input stream of characters is called tokenization, and the lexer categorizes them according to a symbol type. A token can look like anything that is useful for processing an input text stream or text file.
A lexical analyzer generally does nothing with combinations of tokens, a task left for a parser. For example, a typical lexical analyzer recognizes parentheses as tokens, but does nothing to ensure that each "(" is matched with a ")".
Consider this expression in the C programming language:
sum=3+2;
Tokenized in the following table:
| Lexeme | Token type |
|---|---|
| sum | Identifier |
| = | Assignment operator |
| 3 | Integer literal |
| + | Addition operator |
| 2 | Integer literal |
| ; | End of statement |
Tokens are frequently defined by regular expressions, which are understood by a lexical analyzer generator such as lex. The lexical analyzer (either generated automatically by a tool like lex, or hand-crafted) reads in a stream of characters, identifies the lexemes in the stream, and categorizes them into tokens. This is called "tokenizing." If the lexer finds an invalid token, it will report an error.
Following tokenizing is parsing. From there, the interpreted data may be loaded into data structures for general use, interpretation, or compiling.
Read more about this topic: Lexical Analysis
Famous quotes containing the word token:
“He made for the door, automatically resuming his glasses and leaving in front of her, on the floor, his right slipper in token of his speedy return. Then, his desire exposed and his eyes wicked behind their strong lenses, he attempted to push her toward the bed.”
—Vladimir Nabokov (1899–1977)
““With the gracious consent of the audience, you will be made to don the red tophat”Ma token phrase that the courts had evolved, whose true meaning was known to every schoolboy.”
—Vladimir Nabokov (1899–1977)
“To become a token woman—whether you win the Nobel Prize or merely get tenure at the cost of denying your sisters—is to become something less than a man ... since men are loyal at least to their own world-view, their laws of brotherhood and self-interest.”
—Adrienne Rich (b. 1929)