A token is a string of characters, categorized according to the rules as a symbol (e.g., IDENTIFIER, NUMBER, COMMA). The process of forming tokens from an input stream of characters is called tokenization, and the lexer categorizes them according to a symbol type. A token can look like anything that is useful for processing an input text stream or text file.
A lexical analyzer generally does nothing with combinations of tokens, a task left for a parser. For example, a typical lexical analyzer recognizes parentheses as tokens, but does nothing to ensure that each "(" is matched with a ")".
Consider this expression in the C programming language:
sum=3+2;
Tokenized in the following table:
| Lexeme | Token type |
|---|---|
| sum | Identifier |
| = | Assignment operator |
| 3 | Integer literal |
| + | Addition operator |
| 2 | Integer literal |
| ; | End of statement |
Tokens are frequently defined by regular expressions, which are understood by a lexical analyzer generator such as lex. The lexical analyzer (either generated automatically by a tool like lex, or hand-crafted) reads in a stream of characters, identifies the lexemes in the stream, and categorizes them into tokens. This is called "tokenizing." If the lexer finds an invalid token, it will report an error.
Following tokenizing is parsing. From there, the interpreted data may be loaded into data structures for general use, interpretation, or compiling.
Read more about this topic: Lexical Analysis
Famous quotes containing the word token:
“He made for the door, automatically resuming his glasses and leaving in front of her, on the floor, his right slipper in token of his speedy return. Then, his desire exposed and his eyes wicked behind their strong lenses, he attempted to push her toward the bed.”
—Vladimir Nabokov (1899–1977)
“A woman with cut hair is a filthy spectacle, and much like a monster ... it being natural and comely to women to nourish their hair, which even God and nature have given them for a covering, a token of subjection, and a natural badge to distinguish them from men.”
—William Prynne (1600–1669)
“It doesn’t make any difference how much money a father earns, his name is always Dad-Can-I.... Like all other children, my five have one great talent: they are gifted beggars. Not one of them ever ran into the room, looked up at me, and said, “I’m really happy that you’re my father, and as a tangible token of my appreciation, here’s a dollar.””
—Bill Cosby (20th century)