A token is a string of characters, categorized according to the rules as a symbol (e.g., IDENTIFIER, NUMBER, COMMA). The process of forming tokens from an input stream of characters is called tokenization, and the lexer categorizes them according to a symbol type. A token can look like anything that is useful for processing an input text stream or text file.
A lexical analyzer generally does nothing with combinations of tokens, a task left for a parser. For example, a typical lexical analyzer recognizes parentheses as tokens, but does nothing to ensure that each "(" is matched with a ")".
Consider this expression in the C programming language:
sum=3+2;
Tokenized in the following table:
| Lexeme | Token type |
|---|---|
| sum | Identifier |
| = | Assignment operator |
| 3 | Integer literal |
| + | Addition operator |
| 2 | Integer literal |
| ; | End of statement |
Tokens are frequently defined by regular expressions, which are understood by a lexical analyzer generator such as lex. The lexical analyzer (either generated automatically by a tool like lex, or hand-crafted) reads in a stream of characters, identifies the lexemes in the stream, and categorizes them into tokens. This is called "tokenizing." If the lexer finds an invalid token, it will report an error.
Following tokenizing is parsing. From there, the interpreted data may be loaded into data structures for general use, interpretation, or compiling.
Read more about this topic: Lexical Analysis
Famous quotes containing the word token:
“To become a token woman—whether you win the Nobel Prize or merely get tenure at the cost of denying your sisters—is to become something less than a man ... since men are loyal at least to their own world-view, their laws of brotherhood and self-interest.”
—Adrienne Rich (b. 1929)
“It doesn’t make any difference how much money a father earns, his name is always Dad-Can-I.... Like all other children, my five have one great talent: they are gifted beggars. Not one of them ever ran into the room, looked up at me, and said, “I’m really happy that you’re my father, and as a tangible token of my appreciation, here’s a dollar.””
—Bill Cosby (20th century)
“A woman with cut hair is a filthy spectacle, and much like a monster ... it being natural and comely to women to nourish their hair, which even God and nature have given them for a covering, a token of subjection, and a natural badge to distinguish them from men.”
—William Prynne (1600–1669)