Lexer Generator
Lexical analysis can often be performed in a single pass if reading is done a character at a time. Single-pass lexers can be generated by tools such as flex.
The lex/flex family of generators uses a table-driven approach which is much less efficient than the directly coded approach. With the latter approach the generator produces an engine that directly jumps to follow-up states via goto statements. Tools like re2c and Quex have proven (e.g. RE2C - A More Versatile Scanner Generator (1994) to produce engines that are between two to three times faster than flex produced engines. It is in general difficult to hand-write analyzers that perform better than engines generated by these latter tools.
The simple utility of using a scanner generator should not be discounted, especially in the developmental phase, when a language specification might change daily. The ability to express lexical constructs as regular expressions facilitates the description of a lexical analyzer. Some tools offer the specification of pre- and post-conditions which are hard to program by hand. In that case, using a scanner generator may save a lot of development time.
Read more about this topic: Lexical Analysis
Famous quotes containing the word generator:
“He admired the terrible recreative power of his memory. It was only with the weakening of this generator whose fecundity diminishes with age that he could hope for his torture to be appeased. But it appeared that the power to make him suffer of one of Odette’s statements seemed exhausted, then one of these statements on which Swann’s spirit had until then not dwelled, an almost new word relayed the others and struck him with new vigor.”
—Marcel Proust (1871–1922)