| This is flex.info, produced by makeinfo version 4.13 from flex.texi. |
| |
| INFO-DIR-SECTION Programming |
| START-INFO-DIR-ENTRY |
| * flex: (flex). Fast lexical analyzer generator (lex replacement). |
| END-INFO-DIR-ENTRY |
| |
| The flex manual is placed under the same licensing conditions as the |
| rest of flex: |
| |
| Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2012 The |
| Flex Project. |
| |
| Copyright (C) 1990, 1997 The Regents of the University of California. |
| All rights reserved. |
| |
| This code is derived from software contributed to Berkeley by Vern |
| Paxson. |
| |
| The United States Government has rights in this work pursuant to |
| contract no. DE-AC03-76SF00098 between the United States Department of |
| Energy and the University of California. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are |
| met: |
| |
| 1. Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| |
| 2. Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the |
| distribution. |
| |
| Neither the name of the University nor the names of its contributors |
| may be used to endorse or promote products derived from this software |
| without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED |
| WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF |
| MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. |
| |
| |
| File: flex.info, Node: Top, Next: Copyright, Prev: (dir), Up: (dir) |
| |
| flex |
| **** |
| |
| This manual describes `flex', a tool for generating programs that |
| perform pattern-matching on text. The manual includes both tutorial and |
| reference sections. |
| |
| This edition of `The flex Manual' documents `flex' version 2.5.37. |
| It was last updated on 22 July 2012. |
| |
| This manual was written by Vern Paxson, Will Estes and John Millaway. |
| |
| * Menu: |
| |
| * Copyright:: |
| * Reporting Bugs:: |
| * Introduction:: |
| * Simple Examples:: |
| * Format:: |
| * Patterns:: |
| * Matching:: |
| * Actions:: |
| * Generated Scanner:: |
| * Start Conditions:: |
| * Multiple Input Buffers:: |
| * EOF:: |
| * Misc Macros:: |
| * User Values:: |
| * Yacc:: |
| * Scanner Options:: |
| * Performance:: |
| * Cxx:: |
| * Reentrant:: |
| * Lex and Posix:: |
| * Memory Management:: |
| * Serialized Tables:: |
| * Diagnostics:: |
| * Limitations:: |
| * Bibliography:: |
| * FAQ:: |
| * Appendices:: |
| * Indices:: |
| |
| --- The Detailed Node Listing --- |
| |
| Format of the Input File |
| |
| * Definitions Section:: |
| * Rules Section:: |
| * User Code Section:: |
| * Comments in the Input:: |
| |
| Scanner Options |
| |
| * Options for Specifying Filenames:: |
| * Options Affecting Scanner Behavior:: |
| * Code-Level And API Options:: |
| * Options for Scanner Speed and Size:: |
| * Debugging Options:: |
| * Miscellaneous Options:: |
| |
| Reentrant C Scanners |
| |
| * Reentrant Uses:: |
| * Reentrant Overview:: |
| * Reentrant Example:: |
| * Reentrant Detail:: |
| * Reentrant Functions:: |
| |
| The Reentrant API in Detail |
| |
| * Specify Reentrant:: |
| * Extra Reentrant Argument:: |
| * Global Replacement:: |
| * Init and Destroy Functions:: |
| * Accessor Methods:: |
| * Extra Data:: |
| * About yyscan_t:: |
| |
| Memory Management |
| |
| * The Default Memory Management:: |
| * Overriding The Default Memory Management:: |
| * A Note About yytext And Memory:: |
| |
| Serialized Tables |
| |
| * Creating Serialized Tables:: |
| * Loading and Unloading Serialized Tables:: |
| * Tables File Format:: |
| |
| FAQ |
| |
| * When was flex born?:: |
| * How do I expand backslash-escape sequences in C-style quoted strings?:: |
| * Why do flex scanners call fileno if it is not ANSI compatible?:: |
| * Does flex support recursive pattern definitions?:: |
| * How do I skip huge chunks of input (tens of megabytes) while using flex?:: |
| * Flex is not matching my patterns in the same order that I defined them.:: |
| * My actions are executing out of order or sometimes not at all.:: |
| * How can I have multiple input sources feed into the same scanner at the same time?:: |
| * Can I build nested parsers that work with the same input file?:: |
| * How can I match text only at the end of a file?:: |
| * How can I make REJECT cascade across start condition boundaries?:: |
| * Why cant I use fast or full tables with interactive mode?:: |
| * How much faster is -F or -f than -C?:: |
| * If I have a simple grammar cant I just parse it with flex?:: |
| * Why doesn't yyrestart() set the start state back to INITIAL?:: |
| * How can I match C-style comments?:: |
| * The period isn't working the way I expected.:: |
| * Can I get the flex manual in another format?:: |
| * Does there exist a "faster" NDFA->DFA algorithm?:: |
| * How does flex compile the DFA so quickly?:: |
| * How can I use more than 8192 rules?:: |
| * How do I abandon a file in the middle of a scan and switch to a new file?:: |
| * How do I execute code only during initialization (only before the first scan)?:: |
| * How do I execute code at termination?:: |
| * Where else can I find help?:: |
| * Can I include comments in the "rules" section of the file?:: |
| * I get an error about undefined yywrap().:: |
| * How can I change the matching pattern at run time?:: |
| * How can I expand macros in the input?:: |
| * How can I build a two-pass scanner?:: |
| * How do I match any string not matched in the preceding rules?:: |
| * I am trying to port code from AT&T lex that uses yysptr and yysbuf.:: |
| * Is there a way to make flex treat NULL like a regular character?:: |
| * Whenever flex can not match the input it says "flex scanner jammed".:: |
| * Why doesn't flex have non-greedy operators like perl does?:: |
| * Memory leak - 16386 bytes allocated by malloc.:: |
| * How do I track the byte offset for lseek()?:: |
| * How do I use my own I/O classes in a C++ scanner?:: |
| * How do I skip as many chars as possible?:: |
| * deleteme00:: |
| * Are certain equivalent patterns faster than others?:: |
| * Is backing up a big deal?:: |
| * Can I fake multi-byte character support?:: |
| * deleteme01:: |
| * Can you discuss some flex internals?:: |
| * unput() messes up yy_at_bol:: |
| * The | operator is not doing what I want:: |
| * Why can't flex understand this variable trailing context pattern?:: |
| * The ^ operator isn't working:: |
| * Trailing context is getting confused with trailing optional patterns:: |
| * Is flex GNU or not?:: |
| * ERASEME53:: |
| * I need to scan if-then-else blocks and while loops:: |
| * ERASEME55:: |
| * ERASEME56:: |
| * ERASEME57:: |
| * Is there a repository for flex scanners?:: |
| * How can I conditionally compile or preprocess my flex input file?:: |
| * Where can I find grammars for lex and yacc?:: |
| * I get an end-of-buffer message for each character scanned.:: |
| * unnamed-faq-62:: |
| * unnamed-faq-63:: |
| * unnamed-faq-64:: |
| * unnamed-faq-65:: |
| * unnamed-faq-66:: |
| * unnamed-faq-67:: |
| * unnamed-faq-68:: |
| * unnamed-faq-69:: |
| * unnamed-faq-70:: |
| * unnamed-faq-71:: |
| * unnamed-faq-72:: |
| * unnamed-faq-73:: |
| * unnamed-faq-74:: |
| * unnamed-faq-75:: |
| * unnamed-faq-76:: |
| * unnamed-faq-77:: |
| * unnamed-faq-78:: |
| * unnamed-faq-79:: |
| * unnamed-faq-80:: |
| * unnamed-faq-81:: |
| * unnamed-faq-82:: |
| * unnamed-faq-83:: |
| * unnamed-faq-84:: |
| * unnamed-faq-85:: |
| * unnamed-faq-86:: |
| * unnamed-faq-87:: |
| * unnamed-faq-88:: |
| * unnamed-faq-90:: |
| * unnamed-faq-91:: |
| * unnamed-faq-92:: |
| * unnamed-faq-93:: |
| * unnamed-faq-94:: |
| * unnamed-faq-95:: |
| * unnamed-faq-96:: |
| * unnamed-faq-97:: |
| * unnamed-faq-98:: |
| * unnamed-faq-99:: |
| * unnamed-faq-100:: |
| * unnamed-faq-101:: |
| * What is the difference between YYLEX_PARAM and YY_DECL?:: |
| * Why do I get "conflicting types for yylex" error?:: |
| * How do I access the values set in a Flex action from within a Bison action?:: |
| |
| Appendices |
| |
| * Makefiles and Flex:: |
| * Bison Bridge:: |
| * M4 Dependency:: |
| * Common Patterns:: |
| |
| Indices |
| |
| * Concept Index:: |
| * Index of Functions and Macros:: |
| * Index of Variables:: |
| * Index of Data Types:: |
| * Index of Hooks:: |
| * Index of Scanner Options:: |
| |
| |
| File: flex.info, Node: Copyright, Next: Reporting Bugs, Prev: Top, Up: Top |
| |
| 1 Copyright |
| *********** |
| |
| The flex manual is placed under the same licensing conditions as the |
| rest of flex: |
| |
| Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2012 The |
| Flex Project. |
| |
| Copyright (C) 1990, 1997 The Regents of the University of California. |
| All rights reserved. |
| |
| This code is derived from software contributed to Berkeley by Vern |
| Paxson. |
| |
| The United States Government has rights in this work pursuant to |
| contract no. DE-AC03-76SF00098 between the United States Department of |
| Energy and the University of California. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are |
| met: |
| |
| 1. Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| |
| 2. Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the |
| distribution. |
| |
| Neither the name of the University nor the names of its contributors |
| may be used to endorse or promote products derived from this software |
| without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED |
| WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF |
| MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. |
| |
| |
| File: flex.info, Node: Reporting Bugs, Next: Introduction, Prev: Copyright, Up: Top |
| |
| 2 Reporting Bugs |
| **************** |
| |
| If you find a bug in `flex', please report it using the SourceForge Bug |
| Tracking facilities which can be found on flex's SourceForge Page |
| (http://sourceforge.net/projects/flex). |
| |
| |
| File: flex.info, Node: Introduction, Next: Simple Examples, Prev: Reporting Bugs, Up: Top |
| |
| 3 Introduction |
| ************** |
| |
| `flex' is a tool for generating "scanners". A scanner is a program |
| which recognizes lexical patterns in text. The `flex' program reads |
| the given input files, or its standard input if no file names are |
| given, for a description of a scanner to generate. The description is |
| in the form of pairs of regular expressions and C code, called "rules". |
| `flex' generates as output a C source file, `lex.yy.c' by default, |
| which defines a routine `yylex()'. This file can be compiled and |
| linked with the flex runtime library to produce an executable. When |
| the executable is run, it analyzes its input for occurrences of the |
| regular expressions. Whenever it finds one, it executes the |
| corresponding C code. |
| |
| |
| File: flex.info, Node: Simple Examples, Next: Format, Prev: Introduction, Up: Top |
| |
| 4 Some Simple Examples |
| ********************** |
| |
| First some simple examples to get the flavor of how one uses `flex'. |
| |
| The following `flex' input specifies a scanner which, when it |
| encounters the string `username' will replace it with the user's login |
| name: |
| |
| %% |
| username printf( "%s", getlogin() ); |
| |
| By default, any text not matched by a `flex' scanner is copied to |
| the output, so the net effect of this scanner is to copy its input file |
| to its output with each occurrence of `username' expanded. In this |
| input, there is just one rule. `username' is the "pattern" and the |
| `printf' is the "action". The `%%' symbol marks the beginning of the |
| rules. |
| |
| Here's another simple example: |
| |
| int num_lines = 0, num_chars = 0; |
| |
| %% |
| \n ++num_lines; ++num_chars; |
| . ++num_chars; |
| |
| %% |
| |
| int main() |
| { |
| yylex(); |
| printf( "# of lines = %d, # of chars = %d\n", |
| num_lines, num_chars ); |
| } |
| |
| This scanner counts the number of characters and the number of lines |
| in its input. It produces no output other than the final report on the |
| character and line counts. The first line declares two globals, |
| `num_lines' and `num_chars', which are accessible both inside `yylex()' |
| and in the `main()' routine declared after the second `%%'. There are |
| two rules, one which matches a newline (`\n') and increments both the |
| line count and the character count, and one which matches any character |
| other than a newline (indicated by the `.' regular expression). |
| |
| A somewhat more complicated example: |
| |
| /* scanner for a toy Pascal-like language */ |
| |
| %{ |
| /* need this for the call to atof() below */ |
| #include <math.h> |
| %} |
| |
| DIGIT [0-9] |
| ID [a-z][a-z0-9]* |
| |
| %% |
| |
| {DIGIT}+ { |
| printf( "An integer: %s (%d)\n", yytext, |
| atoi( yytext ) ); |
| } |
| |
| {DIGIT}+"."{DIGIT}* { |
| printf( "A float: %s (%g)\n", yytext, |
| atof( yytext ) ); |
| } |
| |
| if|then|begin|end|procedure|function { |
| printf( "A keyword: %s\n", yytext ); |
| } |
| |
| {ID} printf( "An identifier: %s\n", yytext ); |
| |
| "+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext ); |
| |
| "{"[\^{}}\n]*"}" /* eat up one-line comments */ |
| |
| [ \t\n]+ /* eat up whitespace */ |
| |
| . printf( "Unrecognized character: %s\n", yytext ); |
| |
| %% |
| |
| int main( int argc, char **argv ) |
| { |
| ++argv, --argc; /* skip over program name */ |
| if ( argc > 0 ) |
| yyin = fopen( argv[0], "r" ); |
| else |
| yyin = stdin; |
| |
| yylex(); |
| } |
| |
| This is the beginnings of a simple scanner for a language like |
| Pascal. It identifies different types of "tokens" and reports on what |
| it has seen. |
| |
| The details of this example will be explained in the following |
| sections. |
| |
| |
| File: flex.info, Node: Format, Next: Patterns, Prev: Simple Examples, Up: Top |
| |
| 5 Format of the Input File |
| ************************** |
| |
| The `flex' input file consists of three sections, separated by a line |
| containing only `%%'. |
| |
| definitions |
| %% |
| rules |
| %% |
| user code |
| |
| * Menu: |
| |
| * Definitions Section:: |
| * Rules Section:: |
| * User Code Section:: |
| * Comments in the Input:: |
| |
| |
| File: flex.info, Node: Definitions Section, Next: Rules Section, Prev: Format, Up: Format |
| |
| 5.1 Format of the Definitions Section |
| ===================================== |
| |
| The "definitions section" contains declarations of simple "name" |
| definitions to simplify the scanner specification, and declarations of |
| "start conditions", which are explained in a later section. |
| |
| Name definitions have the form: |
| |
| name definition |
| |
| The `name' is a word beginning with a letter or an underscore (`_') |
| followed by zero or more letters, digits, `_', or `-' (dash). The |
| definition is taken to begin at the first non-whitespace character |
| following the name and continuing to the end of the line. The |
| definition can subsequently be referred to using `{name}', which will |
| expand to `(definition)'. For example, |
| |
| DIGIT [0-9] |
| ID [a-z][a-z0-9]* |
| |
| Defines `DIGIT' to be a regular expression which matches a single |
| digit, and `ID' to be a regular expression which matches a letter |
| followed by zero-or-more letters-or-digits. A subsequent reference to |
| |
| {DIGIT}+"."{DIGIT}* |
| |
| is identical to |
| |
| ([0-9])+"."([0-9])* |
| |
| and matches one-or-more digits followed by a `.' followed by |
| zero-or-more digits. |
| |
| An unindented comment (i.e., a line beginning with `/*') is copied |
| verbatim to the output up to the next `*/'. |
| |
| Any _indented_ text or text enclosed in `%{' and `%}' is also copied |
| verbatim to the output (with the %{ and %} symbols removed). The %{ |
| and %} symbols must appear unindented on lines by themselves. |
| |
| A `%top' block is similar to a `%{' ... `%}' block, except that the |
| code in a `%top' block is relocated to the _top_ of the generated file, |
| before any flex definitions (1). The `%top' block is useful when you |
| want certain preprocessor macros to be defined or certain files to be |
| included before the generated code. The single characters, `{' and |
| `}' are used to delimit the `%top' block, as show in the example below: |
| |
| %top{ |
| /* This code goes at the "top" of the generated file. */ |
| #include <stdint.h> |
| #include <inttypes.h> |
| } |
| |
| Multiple `%top' blocks are allowed, and their order is preserved. |
| |
| ---------- Footnotes ---------- |
| |
| (1) Actually, `yyIN_HEADER' is defined before the `%top' block. |
| |
| |
| File: flex.info, Node: Rules Section, Next: User Code Section, Prev: Definitions Section, Up: Format |
| |
| 5.2 Format of the Rules Section |
| =============================== |
| |
| The "rules" section of the `flex' input contains a series of rules of |
| the form: |
| |
| pattern action |
| |
| where the pattern must be unindented and the action must begin on |
| the same line. *Note Patterns::, for a further description of patterns |
| and actions. |
| |
| In the rules section, any indented or %{ %} enclosed text appearing |
| before the first rule may be used to declare variables which are local |
| to the scanning routine and (after the declarations) code which is to be |
| executed whenever the scanning routine is entered. Other indented or |
| %{ %} text in the rule section is still copied to the output, but its |
| meaning is not well-defined and it may well cause compile-time errors |
| (this feature is present for POSIX compliance. *Note Lex and Posix::, |
| for other such features). |
| |
| Any _indented_ text or text enclosed in `%{' and `%}' is copied |
| verbatim to the output (with the %{ and %} symbols removed). The %{ |
| and %} symbols must appear unindented on lines by themselves. |
| |
| |
| File: flex.info, Node: User Code Section, Next: Comments in the Input, Prev: Rules Section, Up: Format |
| |
| 5.3 Format of the User Code Section |
| =================================== |
| |
| The user code section is simply copied to `lex.yy.c' verbatim. It is |
| used for companion routines which call or are called by the scanner. |
| The presence of this section is optional; if it is missing, the second |
| `%%' in the input file may be skipped, too. |
| |
| |
| File: flex.info, Node: Comments in the Input, Prev: User Code Section, Up: Format |
| |
| 5.4 Comments in the Input |
| ========================= |
| |
| Flex supports C-style comments, that is, anything between `/*' and `*/' |
| is considered a comment. Whenever flex encounters a comment, it copies |
| the entire comment verbatim to the generated source code. Comments may |
| appear just about anywhere, but with the following exceptions: |
| |
| * Comments may not appear in the Rules Section wherever flex is |
| expecting a regular expression. This means comments may not appear |
| at the beginning of a line, or immediately following a list of |
| scanner states. |
| |
| * Comments may not appear on an `%option' line in the Definitions |
| Section. |
| |
| If you want to follow a simple rule, then always begin a comment on a |
| new line, with one or more whitespace characters before the initial |
| `/*'). This rule will work anywhere in the input file. |
| |
| All the comments in the following example are valid: |
| |
| %{ |
| /* code block */ |
| %} |
| |
| /* Definitions Section */ |
| %x STATE_X |
| |
| %% |
| /* Rules Section */ |
| ruleA /* after regex */ { /* code block */ } /* after code block */ |
| /* Rules Section (indented) */ |
| <STATE_X>{ |
| ruleC ECHO; |
| ruleD ECHO; |
| %{ |
| /* code block */ |
| %} |
| } |
| %% |
| /* User Code Section */ |
| |
| |
| File: flex.info, Node: Patterns, Next: Matching, Prev: Format, Up: Top |
| |
| 6 Patterns |
| ********** |
| |
| The patterns in the input (see *note Rules Section::) are written using |
| an extended set of regular expressions. These are: |
| |
| `x' |
| match the character 'x' |
| |
| `.' |
| any character (byte) except newline |
| |
| `[xyz]' |
| a "character class"; in this case, the pattern matches either an |
| 'x', a 'y', or a 'z' |
| |
| `[abj-oZ]' |
| a "character class" with a range in it; matches an 'a', a 'b', any |
| letter from 'j' through 'o', or a 'Z' |
| |
| `[^A-Z]' |
| a "negated character class", i.e., any character but those in the |
| class. In this case, any character EXCEPT an uppercase letter. |
| |
| `[^A-Z\n]' |
| any character EXCEPT an uppercase letter or a newline |
| |
| `[a-z]{-}[aeiou]' |
| the lowercase consonants |
| |
| `r*' |
| zero or more r's, where r is any regular expression |
| |
| `r+' |
| one or more r's |
| |
| `r?' |
| zero or one r's (that is, "an optional r") |
| |
| `r{2,5}' |
| anywhere from two to five r's |
| |
| `r{2,}' |
| two or more r's |
| |
| `r{4}' |
| exactly 4 r's |
| |
| `{name}' |
| the expansion of the `name' definition (*note Format::). |
| |
| `"[xyz]\"foo"' |
| the literal string: `[xyz]"foo' |
| |
| `\X' |
| if X is `a', `b', `f', `n', `r', `t', or `v', then the ANSI-C |
| interpretation of `\x'. Otherwise, a literal `X' (used to escape |
| operators such as `*') |
| |
| `\0' |
| a NUL character (ASCII code 0) |
| |
| `\123' |
| the character with octal value 123 |
| |
| `\x2a' |
| the character with hexadecimal value 2a |
| |
| `(r)' |
| match an `r'; parentheses are used to override precedence (see |
| below) |
| |
| `(?r-s:pattern)' |
| apply option `r' and omit option `s' while interpreting pattern. |
| Options may be zero or more of the characters `i', `s', or `x'. |
| |
| `i' means case-insensitive. `-i' means case-sensitive. |
| |
| `s' alters the meaning of the `.' syntax to match any single byte |
| whatsoever. `-s' alters the meaning of `.' to match any byte |
| except `\n'. |
| |
| `x' ignores comments and whitespace in patterns. Whitespace is |
| ignored unless it is backslash-escaped, contained within `""'s, or |
| appears inside a character class. |
| |
| The following are all valid: |
| |
| (?:foo) same as (foo) |
| (?i:ab7) same as ([aA][bB]7) |
| (?-i:ab) same as (ab) |
| (?s:.) same as [\x00-\xFF] |
| (?-s:.) same as [^\n] |
| (?ix-s: a . b) same as ([Aa][^\n][bB]) |
| (?x:a b) same as ("ab") |
| (?x:a\ b) same as ("a b") |
| (?x:a" "b) same as ("a b") |
| (?x:a[ ]b) same as ("a b") |
| (?x:a |
| /* comment */ |
| b |
| c) same as (abc) |
| |
| `(?# comment )' |
| omit everything within `()'. The first `)' character encountered |
| ends the pattern. It is not possible to for the comment to contain |
| a `)' character. The comment may span lines. |
| |
| `rs' |
| the regular expression `r' followed by the regular expression `s'; |
| called "concatenation" |
| |
| `r|s' |
| either an `r' or an `s' |
| |
| `r/s' |
| an `r' but only if it is followed by an `s'. The text matched by |
| `s' is included when determining whether this rule is the longest |
| match, but is then returned to the input before the action is |
| executed. So the action only sees the text matched by `r'. This |
| type of pattern is called "trailing context". (There are some |
| combinations of `r/s' that flex cannot match correctly. *Note |
| Limitations::, regarding dangerous trailing context.) |
| |
| `^r' |
| an `r', but only at the beginning of a line (i.e., when just |
| starting to scan, or right after a newline has been scanned). |
| |
| `r$' |
| an `r', but only at the end of a line (i.e., just before a |
| newline). Equivalent to `r/\n'. |
| |
| Note that `flex''s notion of "newline" is exactly whatever the C |
| compiler used to compile `flex' interprets `\n' as; in particular, |
| on some DOS systems you must either filter out `\r's in the input |
| yourself, or explicitly use `r/\r\n' for `r$'. |
| |
| `<s>r' |
| an `r', but only in start condition `s' (see *note Start |
| Conditions:: for discussion of start conditions). |
| |
| `<s1,s2,s3>r' |
| same, but in any of start conditions `s1', `s2', or `s3'. |
| |
| `<*>r' |
| an `r' in any start condition, even an exclusive one. |
| |
| `<<EOF>>' |
| an end-of-file. |
| |
| `<s1,s2><<EOF>>' |
| an end-of-file when in start condition `s1' or `s2' |
| |
| Note that inside of a character class, all regular expression |
| operators lose their special meaning except escape (`\') and the |
| character class operators, `-', `]]', and, at the beginning of the |
| class, `^'. |
| |
| The regular expressions listed above are grouped according to |
| precedence, from highest precedence at the top to lowest at the bottom. |
| Those grouped together have equal precedence (see special note on the |
| precedence of the repeat operator, `{}', under the documentation for |
| the `--posix' POSIX compliance option). For example, |
| |
| foo|bar* |
| |
| is the same as |
| |
| (foo)|(ba(r*)) |
| |
| since the `*' operator has higher precedence than concatenation, and |
| concatenation higher than alternation (`|'). This pattern therefore |
| matches _either_ the string `foo' _or_ the string `ba' followed by |
| zero-or-more `r''s. To match `foo' or zero-or-more repetitions of the |
| string `bar', use: |
| |
| foo|(bar)* |
| |
| And to match a sequence of zero or more repetitions of `foo' and |
| `bar': |
| |
| (foo|bar)* |
| |
| In addition to characters and ranges of characters, character classes |
| can also contain "character class expressions". These are expressions |
| enclosed inside `[': and `:]' delimiters (which themselves must appear |
| between the `[' and `]' of the character class. Other elements may |
| occur inside the character class, too). The valid expressions are: |
| |
| [:alnum:] [:alpha:] [:blank:] |
| [:cntrl:] [:digit:] [:graph:] |
| [:lower:] [:print:] [:punct:] |
| [:space:] [:upper:] [:xdigit:] |
| |
| These expressions all designate a set of characters equivalent to the |
| corresponding standard C `isXXX' function. For example, `[:alnum:]' |
| designates those characters for which `isalnum()' returns true - i.e., |
| any alphabetic or numeric character. Some systems don't provide |
| `isblank()', so flex defines `[:blank:]' as a blank or a tab. |
| |
| For example, the following character classes are all equivalent: |
| |
| [[:alnum:]] |
| [[:alpha:][:digit:]] |
| [[:alpha:][0-9]] |
| [a-zA-Z0-9] |
| |
| A word of caution. Character classes are expanded immediately when |
| seen in the `flex' input. This means the character classes are |
| sensitive to the locale in which `flex' is executed, and the resulting |
| scanner will not be sensitive to the runtime locale. This may or may |
| not be desirable. |
| |
| * If your scanner is case-insensitive (the `-i' flag), then |
| `[:upper:]' and `[:lower:]' are equivalent to `[:alpha:]'. |
| |
| * Character classes with ranges, such as `[a-Z]', should be used with |
| caution in a case-insensitive scanner if the range spans upper or |
| lowercase characters. Flex does not know if you want to fold all |
| upper and lowercase characters together, or if you want the |
| literal numeric range specified (with no case folding). When in |
| doubt, flex will assume that you meant the literal numeric range, |
| and will issue a warning. The exception to this rule is a |
| character range such as `[a-z]' or `[S-W]' where it is obvious |
| that you want case-folding to occur. Here are some examples with |
| the `-i' flag enabled: |
| |
| Range Result Literal Range Alternate Range |
| `[a-t]' ok `[a-tA-T]' |
| `[A-T]' ok `[a-tA-T]' |
| `[A-t]' ambiguous `[A-Z\[\\\]_`a-t]' `[a-tA-T]' |
| `[_-{]' ambiguous `[_`a-z{]' `[_`a-zA-Z{]' |
| `[@-C]' ambiguous `[@ABC]' `[@A-Z\[\\\]_`abc]' |
| |
| * A negated character class such as the example `[^A-Z]' above |
| _will_ match a newline unless `\n' (or an equivalent escape |
| sequence) is one of the characters explicitly present in the |
| negated character class (e.g., `[^A-Z\n]'). This is unlike how |
| many other regular expression tools treat negated character |
| classes, but unfortunately the inconsistency is historically |
| entrenched. Matching newlines means that a pattern like `[^"]*' |
| can match the entire input unless there's another quote in the |
| input. |
| |
| Flex allows negation of character class expressions by prepending |
| `^' to the POSIX character class name. |
| |
| [:^alnum:] [:^alpha:] [:^blank:] |
| [:^cntrl:] [:^digit:] [:^graph:] |
| [:^lower:] [:^print:] [:^punct:] |
| [:^space:] [:^upper:] [:^xdigit:] |
| |
| Flex will issue a warning if the expressions `[:^upper:]' and |
| `[:^lower:]' appear in a case-insensitive scanner, since their |
| meaning is unclear. The current behavior is to skip them entirely, |
| but this may change without notice in future revisions of flex. |
| |
| * The `{-}' operator computes the difference of two character |
| classes. For example, `[a-c]{-}[b-z]' represents all the |
| characters in the class `[a-c]' that are not in the class `[b-z]' |
| (which in this case, is just the single character `a'). The `{-}' |
| operator is left associative, so `[abc]{-}[b]{-}[c]' is the same |
| as `[a]'. Be careful not to accidentally create an empty set, |
| which will never match. |
| |
| * The `{+}' operator computes the union of two character classes. |
| For example, `[a-z]{+}[0-9]' is the same as `[a-z0-9]'. This |
| operator is useful when preceded by the result of a difference |
| operation, as in, `[[:alpha:]]{-}[[:lower:]]{+}[q]', which is |
| equivalent to `[A-Zq]' in the "C" locale. |
| |
| * A rule can have at most one instance of trailing context (the `/' |
| operator or the `$' operator). The start condition, `^', and |
| `<<EOF>>' patterns can only occur at the beginning of a pattern, |
| and, as well as with `/' and `$', cannot be grouped inside |
| parentheses. A `^' which does not occur at the beginning of a |
| rule or a `$' which does not occur at the end of a rule loses its |
| special properties and is treated as a normal character. |
| |
| * The following are invalid: |
| |
| foo/bar$ |
| <sc1>foo<sc2>bar |
| |
| Note that the first of these can be written `foo/bar\n'. |
| |
| * The following will result in `$' or `^' being treated as a normal |
| character: |
| |
| foo|(bar$) |
| foo|^bar |
| |
| If the desired meaning is a `foo' or a |
| `bar'-followed-by-a-newline, the following could be used (the |
| special `|' action is explained below, *note Actions::): |
| |
| foo | |
| bar$ /* action goes here */ |
| |
| A similar trick will work for matching a `foo' or a |
| `bar'-at-the-beginning-of-a-line. |
| |
| |
| File: flex.info, Node: Matching, Next: Actions, Prev: Patterns, Up: Top |
| |
| 7 How the Input Is Matched |
| ************************** |
| |
| When the generated scanner is run, it analyzes its input looking for |
| strings which match any of its patterns. If it finds more than one |
| match, it takes the one matching the most text (for trailing context |
| rules, this includes the length of the trailing part, even though it |
| will then be returned to the input). If it finds two or more matches of |
| the same length, the rule listed first in the `flex' input file is |
| chosen. |
| |
| Once the match is determined, the text corresponding to the match |
| (called the "token") is made available in the global character pointer |
| `yytext', and its length in the global integer `yyleng'. The "action" |
| corresponding to the matched pattern is then executed (*note |
| Actions::), and then the remaining input is scanned for another match. |
| |
| If no match is found, then the "default rule" is executed: the next |
| character in the input is considered matched and copied to the standard |
| output. Thus, the simplest valid `flex' input is: |
| |
| %% |
| |
| which generates a scanner that simply copies its input (one |
| character at a time) to its output. |
| |
| Note that `yytext' can be defined in two different ways: either as a |
| character _pointer_ or as a character _array_. You can control which |
| definition `flex' uses by including one of the special directives |
| `%pointer' or `%array' in the first (definitions) section of your flex |
| input. The default is `%pointer', unless you use the `-l' lex |
| compatibility option, in which case `yytext' will be an array. The |
| advantage of using `%pointer' is substantially faster scanning and no |
| buffer overflow when matching very large tokens (unless you run out of |
| dynamic memory). The disadvantage is that you are restricted in how |
| your actions can modify `yytext' (*note Actions::), and calls to the |
| `unput()' function destroys the present contents of `yytext', which can |
| be a considerable porting headache when moving between different `lex' |
| versions. |
| |
| The advantage of `%array' is that you can then modify `yytext' to |
| your heart's content, and calls to `unput()' do not destroy `yytext' |
| (*note Actions::). Furthermore, existing `lex' programs sometimes |
| access `yytext' externally using declarations of the form: |
| |
| extern char yytext[]; |
| |
| This definition is erroneous when used with `%pointer', but correct |
| for `%array'. |
| |
| The `%array' declaration defines `yytext' to be an array of `YYLMAX' |
| characters, which defaults to a fairly large value. You can change the |
| size by simply #define'ing `YYLMAX' to a different value in the first |
| section of your `flex' input. As mentioned above, with `%pointer' |
| yytext grows dynamically to accommodate large tokens. While this means |
| your `%pointer' scanner can accommodate very large tokens (such as |
| matching entire blocks of comments), bear in mind that each time the |
| scanner must resize `yytext' it also must rescan the entire token from |
| the beginning, so matching such tokens can prove slow. `yytext' |
| presently does _not_ dynamically grow if a call to `unput()' results in |
| too much text being pushed back; instead, a run-time error results. |
| |
| Also note that you cannot use `%array' with C++ scanner classes |
| (*note Cxx::). |
| |
| |
| File: flex.info, Node: Actions, Next: Generated Scanner, Prev: Matching, Up: Top |
| |
| 8 Actions |
| ********* |
| |
| Each pattern in a rule has a corresponding "action", which can be any |
| arbitrary C statement. The pattern ends at the first non-escaped |
| whitespace character; the remainder of the line is its action. If the |
| action is empty, then when the pattern is matched the input token is |
| simply discarded. For example, here is the specification for a program |
| which deletes all occurrences of `zap me' from its input: |
| |
| %% |
| "zap me" |
| |
| This example will copy all other characters in the input to the |
| output since they will be matched by the default rule. |
| |
| Here is a program which compresses multiple blanks and tabs down to a |
| single blank, and throws away whitespace found at the end of a line: |
| |
| %% |
| [ \t]+ putchar( ' ' ); |
| [ \t]+$ /* ignore this token */ |
| |
| If the action contains a `{', then the action spans till the |
| balancing `}' is found, and the action may cross multiple lines. |
| `flex' knows about C strings and comments and won't be fooled by braces |
| found within them, but also allows actions to begin with `%{' and will |
| consider the action to be all the text up to the next `%}' (regardless |
| of ordinary braces inside the action). |
| |
| An action consisting solely of a vertical bar (`|') means "same as |
| the action for the next rule". See below for an illustration. |
| |
| Actions can include arbitrary C code, including `return' statements |
| to return a value to whatever routine called `yylex()'. Each time |
| `yylex()' is called it continues processing tokens from where it last |
| left off until it either reaches the end of the file or executes a |
| return. |
| |
| Actions are free to modify `yytext' except for lengthening it |
| (adding characters to its end-these will overwrite later characters in |
| the input stream). This however does not apply when using `%array' |
| (*note Matching::). In that case, `yytext' may be freely modified in |
| any way. |
| |
| Actions are free to modify `yyleng' except they should not do so if |
| the action also includes use of `yymore()' (see below). |
| |
| There are a number of special directives which can be included |
| within an action: |
| |
| `ECHO' |
| copies yytext to the scanner's output. |
| |
| `BEGIN' |
| followed by the name of a start condition places the scanner in the |
| corresponding start condition (see below). |
| |
| `REJECT' |
| directs the scanner to proceed on to the "second best" rule which |
| matched the input (or a prefix of the input). The rule is chosen |
| as described above in *note Matching::, and `yytext' and `yyleng' |
| set up appropriately. It may either be one which matched as much |
| text as the originally chosen rule but came later in the `flex' |
| input file, or one which matched less text. For example, the |
| following will both count the words in the input and call the |
| routine `special()' whenever `frob' is seen: |
| |
| int word_count = 0; |
| %% |
| |
| frob special(); REJECT; |
| [^ \t\n]+ ++word_count; |
| |
| Without the `REJECT', any occurrences of `frob' in the input would |
| not be counted as words, since the scanner normally executes only |
| one action per token. Multiple uses of `REJECT' are allowed, each |
| one finding the next best choice to the currently active rule. For |
| example, when the following scanner scans the token `abcd', it will |
| write `abcdabcaba' to the output: |
| |
| %% |
| a | |
| ab | |
| abc | |
| abcd ECHO; REJECT; |
| .|\n /* eat up any unmatched character */ |
| |
| The first three rules share the fourth's action since they use the |
| special `|' action. |
| |
| `REJECT' is a particularly expensive feature in terms of scanner |
| performance; if it is used in _any_ of the scanner's actions it |
| will slow down _all_ of the scanner's matching. Furthermore, |
| `REJECT' cannot be used with the `-Cf' or `-CF' options (*note |
| Scanner Options::). |
| |
| Note also that unlike the other special actions, `REJECT' is a |
| _branch_. Code immediately following it in the action will _not_ |
| be executed. |
| |
| `yymore()' |
| tells the scanner that the next time it matches a rule, the |
| corresponding token should be _appended_ onto the current value of |
| `yytext' rather than replacing it. For example, given the input |
| `mega-kludge' the following will write `mega-mega-kludge' to the |
| output: |
| |
| %% |
| mega- ECHO; yymore(); |
| kludge ECHO; |
| |
| First `mega-' is matched and echoed to the output. Then `kludge' |
| is matched, but the previous `mega-' is still hanging around at the |
| beginning of `yytext' so the `ECHO' for the `kludge' rule will |
| actually write `mega-kludge'. |
| |
| Two notes regarding use of `yymore()'. First, `yymore()' depends on |
| the value of `yyleng' correctly reflecting the size of the current |
| token, so you must not modify `yyleng' if you are using `yymore()'. |
| Second, the presence of `yymore()' in the scanner's action entails a |
| minor performance penalty in the scanner's matching speed. |
| |
| `yyless(n)' returns all but the first `n' characters of the current |
| token back to the input stream, where they will be rescanned when the |
| scanner looks for the next match. `yytext' and `yyleng' are adjusted |
| appropriately (e.g., `yyleng' will now be equal to `n'). For example, |
| on the input `foobar' the following will write out `foobarbar': |
| |
| %% |
| foobar ECHO; yyless(3); |
| [a-z]+ ECHO; |
| |
| An argument of 0 to `yyless()' will cause the entire current input |
| string to be scanned again. Unless you've changed how the scanner will |
| subsequently process its input (using `BEGIN', for example), this will |
| result in an endless loop. |
| |
| Note that `yyless()' is a macro and can only be used in the flex |
| input file, not from other source files. |
| |
| `unput(c)' puts the character `c' back onto the input stream. It |
| will be the next character scanned. The following action will take the |
| current token and cause it to be rescanned enclosed in parentheses. |
| |
| { |
| int i; |
| /* Copy yytext because unput() trashes yytext */ |
| char *yycopy = strdup( yytext ); |
| unput( ')' ); |
| for ( i = yyleng - 1; i >= 0; --i ) |
| unput( yycopy[i] ); |
| unput( '(' ); |
| free( yycopy ); |
| } |
| |
| Note that since each `unput()' puts the given character back at the |
| _beginning_ of the input stream, pushing back strings must be done |
| back-to-front. |
| |
| An important potential problem when using `unput()' is that if you |
| are using `%pointer' (the default), a call to `unput()' _destroys_ the |
| contents of `yytext', starting with its rightmost character and |
| devouring one character to the left with each call. If you need the |
| value of `yytext' preserved after a call to `unput()' (as in the above |
| example), you must either first copy it elsewhere, or build your |
| scanner using `%array' instead (*note Matching::). |
| |
| Finally, note that you cannot put back `EOF' to attempt to mark the |
| input stream with an end-of-file. |
| |
| `input()' reads the next character from the input stream. For |
| example, the following is one way to eat up C comments: |
| |
| %% |
| "/*" { |
| register int c; |
| |
| for ( ; ; ) |
| { |
| while ( (c = input()) != '*' && |
| c != EOF ) |
| ; /* eat up text of comment */ |
| |
| if ( c == '*' ) |
| { |
| while ( (c = input()) == '*' ) |
| ; |
| if ( c == '/' ) |
| break; /* found the end */ |
| } |
| |
| if ( c == EOF ) |
| { |
| error( "EOF in comment" ); |
| break; |
| } |
| } |
| } |
| |
| (Note that if the scanner is compiled using `C++', then `input()' is |
| instead referred to as yyinput(), in order to avoid a name clash with |
| the `C++' stream by the name of `input'.) |
| |
| `YY_FLUSH_BUFFER;' flushes the scanner's internal buffer so that the |
| next time the scanner attempts to match a token, it will first refill |
| the buffer using `YY_INPUT()' (*note Generated Scanner::). This action |
| is a special case of the more general `yy_flush_buffer;' function, |
| described below (*note Multiple Input Buffers::) |
| |
| `yyterminate()' can be used in lieu of a return statement in an |
| action. It terminates the scanner and returns a 0 to the scanner's |
| caller, indicating "all done". By default, `yyterminate()' is also |
| called when an end-of-file is encountered. It is a macro and may be |
| redefined. |
| |
| |
| File: flex.info, Node: Generated Scanner, Next: Start Conditions, Prev: Actions, Up: Top |
| |
| 9 The Generated Scanner |
| *********************** |
| |
| The output of `flex' is the file `lex.yy.c', which contains the |
| scanning routine `yylex()', a number of tables used by it for matching |
| tokens, and a number of auxiliary routines and macros. By default, |
| `yylex()' is declared as follows: |
| |
| int yylex() |
| { |
| ... various definitions and the actions in here ... |
| } |
| |
| (If your environment supports function prototypes, then it will be |
| `int yylex( void )'.) This definition may be changed by defining the |
| `YY_DECL' macro. For example, you could use: |
| |
| #define YY_DECL float lexscan( a, b ) float a, b; |
| |
| to give the scanning routine the name `lexscan', returning a float, |
| and taking two floats as arguments. Note that if you give arguments to |
| the scanning routine using a K&R-style/non-prototyped function |
| declaration, you must terminate the definition with a semi-colon (;). |
| |
| `flex' generates `C99' function definitions by default. However flex |
| does have the ability to generate obsolete, er, `traditional', function |
| definitions. This is to support bootstrapping gcc on old systems. |
| Unfortunately, traditional definitions prevent us from using any |
| standard data types smaller than int (such as short, char, or bool) as |
| function arguments. For this reason, future versions of `flex' may |
| generate standard C99 code only, leaving K&R-style functions to the |
| historians. Currently, if you do *not* want `C99' definitions, then |
| you must use `%option noansi-definitions'. |
| |
| Whenever `yylex()' is called, it scans tokens from the global input |
| file `yyin' (which defaults to stdin). It continues until it either |
| reaches an end-of-file (at which point it returns the value 0) or one |
| of its actions executes a `return' statement. |
| |
| If the scanner reaches an end-of-file, subsequent calls are undefined |
| unless either `yyin' is pointed at a new input file (in which case |
| scanning continues from that file), or `yyrestart()' is called. |
| `yyrestart()' takes one argument, a `FILE *' pointer (which can be |
| NULL, if you've set up `YY_INPUT' to scan from a source other than |
| `yyin'), and initializes `yyin' for scanning from that file. |
| Essentially there is no difference between just assigning `yyin' to a |
| new input file or using `yyrestart()' to do so; the latter is available |
| for compatibility with previous versions of `flex', and because it can |
| be used to switch input files in the middle of scanning. It can also |
| be used to throw away the current input buffer, by calling it with an |
| argument of `yyin'; but it would be better to use `YY_FLUSH_BUFFER' |
| (*note Actions::). Note that `yyrestart()' does _not_ reset the start |
| condition to `INITIAL' (*note Start Conditions::). |
| |
| If `yylex()' stops scanning due to executing a `return' statement in |
| one of the actions, the scanner may then be called again and it will |
| resume scanning where it left off. |
| |
| By default (and for purposes of efficiency), the scanner uses |
| block-reads rather than simple `getc()' calls to read characters from |
| `yyin'. The nature of how it gets its input can be controlled by |
| defining the `YY_INPUT' macro. The calling sequence for `YY_INPUT()' |
| is `YY_INPUT(buf,result,max_size)'. Its action is to place up to |
| `max_size' characters in the character array `buf' and return in the |
| integer variable `result' either the number of characters read or the |
| constant `YY_NULL' (0 on Unix systems) to indicate `EOF'. The default |
| `YY_INPUT' reads from the global file-pointer `yyin'. |
| |
| Here is a sample definition of `YY_INPUT' (in the definitions |
| section of the input file): |
| |
| %{ |
| #define YY_INPUT(buf,result,max_size) \ |
| { \ |
| int c = getchar(); \ |
| result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \ |
| } |
| %} |
| |
| This definition will change the input processing to occur one |
| character at a time. |
| |
| When the scanner receives an end-of-file indication from YY_INPUT, it |
| then checks the `yywrap()' function. If `yywrap()' returns false |
| (zero), then it is assumed that the function has gone ahead and set up |
| `yyin' to point to another input file, and scanning continues. If it |
| returns true (non-zero), then the scanner terminates, returning 0 to |
| its caller. Note that in either case, the start condition remains |
| unchanged; it does _not_ revert to `INITIAL'. |
| |
| If you do not supply your own version of `yywrap()', then you must |
| either use `%option noyywrap' (in which case the scanner behaves as |
| though `yywrap()' returned 1), or you must link with `-lfl' to obtain |
| the default version of the routine, which always returns 1. |
| |
| For scanning from in-memory buffers (e.g., scanning strings), see |
| *note Scanning Strings::. *Note Multiple Input Buffers::. |
| |
| The scanner writes its `ECHO' output to the `yyout' global (default, |
| `stdout'), which may be redefined by the user simply by assigning it to |
| some other `FILE' pointer. |
| |
| |
| File: flex.info, Node: Start Conditions, Next: Multiple Input Buffers, Prev: Generated Scanner, Up: Top |
| |
| 10 Start Conditions |
| ******************* |
| |
| `flex' provides a mechanism for conditionally activating rules. Any |
| rule whose pattern is prefixed with `<sc>' will only be active when the |
| scanner is in the "start condition" named `sc'. For example, |
| |
| <STRING>[^"]* { /* eat up the string body ... */ |
| ... |
| } |
| |
| will be active only when the scanner is in the `STRING' start |
| condition, and |
| |
| <INITIAL,STRING,QUOTE>\. { /* handle an escape ... */ |
| ... |
| } |
| |
| will be active only when the current start condition is either |
| `INITIAL', `STRING', or `QUOTE'. |
| |
| Start conditions are declared in the definitions (first) section of |
| the input using unindented lines beginning with either `%s' or `%x' |
| followed by a list of names. The former declares "inclusive" start |
| conditions, the latter "exclusive" start conditions. A start condition |
| is activated using the `BEGIN' action. Until the next `BEGIN' action |
| is executed, rules with the given start condition will be active and |
| rules with other start conditions will be inactive. If the start |
| condition is inclusive, then rules with no start conditions at all will |
| also be active. If it is exclusive, then _only_ rules qualified with |
| the start condition will be active. A set of rules contingent on the |
| same exclusive start condition describe a scanner which is independent |
| of any of the other rules in the `flex' input. Because of this, |
| exclusive start conditions make it easy to specify "mini-scanners" |
| which scan portions of the input that are syntactically different from |
| the rest (e.g., comments). |
| |
| If the distinction between inclusive and exclusive start conditions |
| is still a little vague, here's a simple example illustrating the |
| connection between the two. The set of rules: |
| |
| %s example |
| %% |
| |
| <example>foo do_something(); |
| |
| bar something_else(); |
| |
| is equivalent to |
| |
| %x example |
| %% |
| |
| <example>foo do_something(); |
| |
| <INITIAL,example>bar something_else(); |
| |
| Without the `<INITIAL,example>' qualifier, the `bar' pattern in the |
| second example wouldn't be active (i.e., couldn't match) when in start |
| condition `example'. If we just used `<example>' to qualify `bar', |
| though, then it would only be active in `example' and not in `INITIAL', |
| while in the first example it's active in both, because in the first |
| example the `example' start condition is an inclusive `(%s)' start |
| condition. |
| |
| Also note that the special start-condition specifier `<*>' matches |
| every start condition. Thus, the above example could also have been |
| written: |
| |
| %x example |
| %% |
| |
| <example>foo do_something(); |
| |
| <*>bar something_else(); |
| |
| The default rule (to `ECHO' any unmatched character) remains active |
| in start conditions. It is equivalent to: |
| |
| <*>.|\n ECHO; |
| |
| `BEGIN(0)' returns to the original state where only the rules with |
| no start conditions are active. This state can also be referred to as |
| the start-condition `INITIAL', so `BEGIN(INITIAL)' is equivalent to |
| `BEGIN(0)'. (The parentheses around the start condition name are not |
| required but are considered good style.) |
| |
| `BEGIN' actions can also be given as indented code at the beginning |
| of the rules section. For example, the following will cause the scanner |
| to enter the `SPECIAL' start condition whenever `yylex()' is called and |
| the global variable `enter_special' is true: |
| |
| int enter_special; |
| |
| %x SPECIAL |
| %% |
| if ( enter_special ) |
| BEGIN(SPECIAL); |
| |
| <SPECIAL>blahblahblah |
| ...more rules follow... |
| |
| To illustrate the uses of start conditions, here is a scanner which |
| provides two different interpretations of a string like `123.456'. By |
| default it will treat it as three tokens, the integer `123', a dot |
| (`.'), and the integer `456'. But if the string is preceded earlier in |
| the line by the string `expect-floats' it will treat it as a single |
| token, the floating-point number `123.456': |
| |
| %{ |
| #include <math.h> |
| %} |
| %s expect |
| |
| %% |
| expect-floats BEGIN(expect); |
| |
| <expect>[0-9]+.[0-9]+ { |
| printf( "found a float, = %f\n", |
| atof( yytext ) ); |
| } |
| <expect>\n { |
| /* that's the end of the line, so |
| * we need another "expect-number" |
| * before we'll recognize any more |
| * numbers |
| */ |
| BEGIN(INITIAL); |
| } |
| |
| [0-9]+ { |
| printf( "found an integer, = %d\n", |
| atoi( yytext ) ); |
| } |
| |
| "." printf( "found a dot\n" ); |
| |
| Here is a scanner which recognizes (and discards) C comments while |
| maintaining a count of the current input line. |
| |
| %x comment |
| %% |
| int line_num = 1; |
| |
| "/*" BEGIN(comment); |
| |
| <comment>[^*\n]* /* eat anything that's not a '*' */ |
| <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ |
| <comment>\n ++line_num; |
| <comment>"*"+"/" BEGIN(INITIAL); |
| |
| This scanner goes to a bit of trouble to match as much text as |
| possible with each rule. In general, when attempting to write a |
| high-speed scanner try to match as much possible in each rule, as it's |
| a big win. |
| |
| Note that start-conditions names are really integer values and can |
| be stored as such. Thus, the above could be extended in the following |
| fashion: |
| |
| %x comment foo |
| %% |
| int line_num = 1; |
| int comment_caller; |
| |
| "/*" { |
| comment_caller = INITIAL; |
| BEGIN(comment); |
| } |
| |
| ... |
| |
| <foo>"/*" { |
| comment_caller = foo; |
| BEGIN(comment); |
| } |
| |
| <comment>[^*\n]* /* eat anything that's not a '*' */ |
| <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ |
| <comment>\n ++line_num; |
| <comment>"*"+"/" BEGIN(comment_caller); |
| |
| Furthermore, you can access the current start condition using the |
| integer-valued `YY_START' macro. For example, the above assignments to |
| `comment_caller' could instead be written |
| |
| comment_caller = YY_START; |
| |
| Flex provides `YYSTATE' as an alias for `YY_START' (since that is |
| what's used by AT&T `lex'). |
| |
| For historical reasons, start conditions do not have their own |
| name-space within the generated scanner. The start condition names are |
| unmodified in the generated scanner and generated header. *Note |
| option-header::. *Note option-prefix::. |
| |
| Finally, here's an example of how to match C-style quoted strings |
| using exclusive start conditions, including expanded escape sequences |
| (but not including checking for a string that's too long): |
| |
| %x str |
| |
| %% |
| char string_buf[MAX_STR_CONST]; |
| char *string_buf_ptr; |
| |
| |
| \" string_buf_ptr = string_buf; BEGIN(str); |
| |
| <str>\" { /* saw closing quote - all done */ |
| BEGIN(INITIAL); |
| *string_buf_ptr = '\0'; |
| /* return string constant token type and |
| * value to parser |
| */ |
| } |
| |
| <str>\n { |
| /* error - unterminated string constant */ |
| /* generate error message */ |
| } |
| |
| <str>\\[0-7]{1,3} { |
| /* octal escape sequence */ |
| int result; |
| |
| (void) sscanf( yytext + 1, "%o", &result ); |
| |
| if ( result > 0xff ) |
| /* error, constant is out-of-bounds */ |
| |
| *string_buf_ptr++ = result; |
| } |
| |
| <str>\\[0-9]+ { |
| /* generate error - bad escape sequence; something |
| * like '\48' or '\0777777' |
| */ |
| } |
| |
| <str>\\n *string_buf_ptr++ = '\n'; |
| <str>\\t *string_buf_ptr++ = '\t'; |
| <str>\\r *string_buf_ptr++ = '\r'; |
| <str>\\b *string_buf_ptr++ = '\b'; |
| <str>\\f *string_buf_ptr++ = '\f'; |
| |
| <str>\\(.|\n) *string_buf_ptr++ = yytext[1]; |
| |
| <str>[^\\\n\"]+ { |
| char *yptr = yytext; |
| |
| while ( *yptr ) |
| *string_buf_ptr++ = *yptr++; |
| } |
| |
| Often, such as in some of the examples above, you wind up writing a |
| whole bunch of rules all preceded by the same start condition(s). Flex |
| makes this a little easier and cleaner by introducing a notion of start |
| condition "scope". A start condition scope is begun with: |
| |
| <SCs>{ |
| |
| where `SCs' is a list of one or more start conditions. Inside the |
| start condition scope, every rule automatically has the prefix `SCs>' |
| applied to it, until a `}' which matches the initial `{'. So, for |
| example, |
| |
| <ESC>{ |
| "\\n" return '\n'; |
| "\\r" return '\r'; |
| "\\f" return '\f'; |
| "\\0" return '\0'; |
| } |
| |
| is equivalent to: |
| |
| <ESC>"\\n" return '\n'; |
| <ESC>"\\r" return '\r'; |
| <ESC>"\\f" return '\f'; |
| <ESC>"\\0" return '\0'; |
| |
| Start condition scopes may be nested. |
| |
| The following routines are available for manipulating stacks of |
| start conditions: |
| |
| -- Function: void yy_push_state ( int `new_state' ) |
| pushes the current start condition onto the top of the start |
| condition stack and switches to `new_state' as though you had used |
| `BEGIN new_state' (recall that start condition names are also |
| integers). |
| |
| -- Function: void yy_pop_state () |
| pops the top of the stack and switches to it via `BEGIN'. |
| |
| -- Function: int yy_top_state () |
| returns the top of the stack without altering the stack's contents. |
| |
| The start condition stack grows dynamically and so has no built-in |
| size limitation. If memory is exhausted, program execution aborts. |
| |
| To use start condition stacks, your scanner must include a `%option |
| stack' directive (*note Scanner Options::). |
| |
| |
| File: flex.info, Node: Multiple Input Buffers, Next: EOF, Prev: Start Conditions, Up: Top |
| |
| 11 Multiple Input Buffers |
| ************************* |
| |
| Some scanners (such as those which support "include" files) require |
| reading from several input streams. As `flex' scanners do a large |
| amount of buffering, one cannot control where the next input will be |
| read from by simply writing a `YY_INPUT()' which is sensitive to the |
| scanning context. `YY_INPUT()' is only called when the scanner reaches |
| the end of its buffer, which may be a long time after scanning a |
| statement such as an `include' statement which requires switching the |
| input source. |
| |
| To negotiate these sorts of problems, `flex' provides a mechanism |
| for creating and switching between multiple input buffers. An input |
| buffer is created by using: |
| |
| -- Function: YY_BUFFER_STATE yy_create_buffer ( FILE *file, int size ) |
| |
| which takes a `FILE' pointer and a size and creates a buffer |
| associated with the given file and large enough to hold `size' |
| characters (when in doubt, use `YY_BUF_SIZE' for the size). It returns |
| a `YY_BUFFER_STATE' handle, which may then be passed to other routines |
| (see below). The `YY_BUFFER_STATE' type is a pointer to an opaque |
| `struct yy_buffer_state' structure, so you may safely initialize |
| `YY_BUFFER_STATE' variables to `((YY_BUFFER_STATE) 0)' if you wish, and |
| also refer to the opaque structure in order to correctly declare input |
| buffers in source files other than that of your scanner. Note that the |
| `FILE' pointer in the call to `yy_create_buffer' is only used as the |
| value of `yyin' seen by `YY_INPUT'. If you redefine `YY_INPUT()' so it |
| no longer uses `yyin', then you can safely pass a NULL `FILE' pointer to |
| `yy_create_buffer'. You select a particular buffer to scan from using: |
| |
| -- Function: void yy_switch_to_buffer ( YY_BUFFER_STATE new_buffer ) |
| |
| The above function switches the scanner's input buffer so subsequent |
| tokens will come from `new_buffer'. Note that `yy_switch_to_buffer()' |
| may be used by `yywrap()' to set things up for continued scanning, |
| instead of opening a new file and pointing `yyin' at it. If you are |
| looking for a stack of input buffers, then you want to use |
| `yypush_buffer_state()' instead of this function. Note also that |
| switching input sources via either `yy_switch_to_buffer()' or |
| `yywrap()' does _not_ change the start condition. |
| |
| -- Function: void yy_delete_buffer ( YY_BUFFER_STATE buffer ) |
| |
| is used to reclaim the storage associated with a buffer. (`buffer' |
| can be NULL, in which case the routine does nothing.) You can also |
| clear the current contents of a buffer using: |
| |
| -- Function: void yypush_buffer_state ( YY_BUFFER_STATE buffer ) |
| |
| This function pushes the new buffer state onto an internal stack. |
| The pushed state becomes the new current state. The stack is maintained |
| by flex and will grow as required. This function is intended to be used |
| instead of `yy_switch_to_buffer', when you want to change states, but |
| preserve the current state for later use. |
| |
| -- Function: void yypop_buffer_state ( ) |
| |
| This function removes the current state from the top of the stack, |
| and deletes it by calling `yy_delete_buffer'. The next state on the |
| stack, if any, becomes the new current state. |
| |
| -- Function: void yy_flush_buffer ( YY_BUFFER_STATE buffer ) |
| |
| This function discards the buffer's contents, so the next time the |
| scanner attempts to match a token from the buffer, it will first fill |
| the buffer anew using `YY_INPUT()'. |
| |
| -- Function: YY_BUFFER_STATE yy_new_buffer ( FILE *file, int size ) |
| |
| is an alias for `yy_create_buffer()', provided for compatibility |
| with the C++ use of `new' and `delete' for creating and destroying |
| dynamic objects. |
| |
| `YY_CURRENT_BUFFER' macro returns a `YY_BUFFER_STATE' handle to the |
| current buffer. It should not be used as an lvalue. |
| |
| Here are two examples of using these features for writing a scanner |
| which expands include files (the `<<EOF>>' feature is discussed below). |
| |
| This first example uses yypush_buffer_state and yypop_buffer_state. |
| Flex maintains the stack internally. |
| |
| /* the "incl" state is used for picking up the name |
| * of an include file |
| */ |
| %x incl |
| %% |
| include BEGIN(incl); |
| |
| [a-z]+ ECHO; |
| [^a-z\n]*\n? ECHO; |
| |
| <incl>[ \t]* /* eat the whitespace */ |
| <incl>[^ \t\n]+ { /* got the include file name */ |
| yyin = fopen( yytext, "r" ); |
| |
| if ( ! yyin ) |
| error( ... ); |
| |
| yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE )); |
| |
| BEGIN(INITIAL); |
| } |
| |
| <<EOF>> { |
| yypop_buffer_state(); |
| |
| if ( !YY_CURRENT_BUFFER ) |
| { |
| yyterminate(); |
| } |
| } |
| |
| The second example, below, does the same thing as the previous |
| example did, but manages its own input buffer stack manually (instead |
| of letting flex do it). |
| |
| /* the "incl" state is used for picking up the name |
| * of an include file |
| */ |
| %x incl |
| |
| %{ |
| #define MAX_INCLUDE_DEPTH 10 |
| YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH]; |
| int include_stack_ptr = 0; |
| %} |
| |
| %% |
| include BEGIN(incl); |
| |
| [a-z]+ ECHO; |
| [^a-z\n]*\n? ECHO; |
| |
| <incl>[ \t]* /* eat the whitespace */ |
| <incl>[^ \t\n]+ { /* got the include file name */ |
| if ( include_stack_ptr >= MAX_INCLUDE_DEPTH ) |
| { |
| fprintf( stderr, "Includes nested too deeply" ); |
| exit( 1 ); |
| } |
| |
| include_stack[include_stack_ptr++] = |
| YY_CURRENT_BUFFER; |
| |
| yyin = fopen( yytext, "r" ); |
| |
| if ( ! yyin ) |
| error( ... ); |
| |
| yy_switch_to_buffer( |
| yy_create_buffer( yyin, YY_BUF_SIZE ) ); |
| |
| BEGIN(INITIAL); |
| } |
| |
| <<EOF>> { |
| if ( --include_stack_ptr 0 ) |
| { |
| yyterminate(); |
| } |
| |
| else |
| { |
| yy_delete_buffer( YY_CURRENT_BUFFER ); |
| yy_switch_to_buffer( |
| include_stack[include_stack_ptr] ); |
| } |
| } |
| |
| The following routines are available for setting up input buffers for |
| scanning in-memory strings instead of files. All of them create a new |
| input buffer for scanning the string, and return a corresponding |
| `YY_BUFFER_STATE' handle (which you should delete with |
| `yy_delete_buffer()' when done with it). They also switch to the new |
| buffer using `yy_switch_to_buffer()', so the next call to `yylex()' |
| will start scanning the string. |
| |
| -- Function: YY_BUFFER_STATE yy_scan_string ( const char *str ) |
| scans a NUL-terminated string. |
| |
| -- Function: YY_BUFFER_STATE yy_scan_bytes ( const char *bytes, int |
| len ) |
| scans `len' bytes (including possibly `NUL's) starting at location |
| `bytes'. |
| |
| Note that both of these functions create and scan a _copy_ of the |
| string or bytes. (This may be desirable, since `yylex()' modifies the |
| contents of the buffer it is scanning.) You can avoid the copy by |
| using: |
| |
| -- Function: YY_BUFFER_STATE yy_scan_buffer (char *base, yy_size_t |
| size) |
| which scans in place the buffer starting at `base', consisting of |
| `size' bytes, the last two bytes of which _must_ be |
| `YY_END_OF_BUFFER_CHAR' (ASCII NUL). These last two bytes are not |
| scanned; thus, scanning consists of `base[0]' through |
| `base[size-2]', inclusive. |
| |
| If you fail to set up `base' in this manner (i.e., forget the final |
| two `YY_END_OF_BUFFER_CHAR' bytes), then `yy_scan_buffer()' returns a |
| NULL pointer instead of creating a new input buffer. |
| |
| -- Data type: yy_size_t |
| is an integral type to which you can cast an integer expression |
| reflecting the size of the buffer. |
| |
| |
| File: flex.info, Node: EOF, Next: Misc Macros, Prev: Multiple Input Buffers, Up: Top |
| |
| 12 End-of-File Rules |
| ******************** |
| |
| The special rule `<<EOF>>' indicates actions which are to be taken when |
| an end-of-file is encountered and `yywrap()' returns non-zero (i.e., |
| indicates no further files to process). The action must finish by |
| doing one of the following things: |
| |
| * assigning `yyin' to a new input file (in previous versions of |
| `flex', after doing the assignment you had to call the special |
| action `YY_NEW_FILE'. This is no longer necessary.) |
| |
| * executing a `return' statement; |
| |
| * executing the special `yyterminate()' action. |
| |
| * or, switching to a new buffer using `yy_switch_to_buffer()' as |
| shown in the example above. |
| |
| <<EOF>> rules may not be used with other patterns; they may only be |
| qualified with a list of start conditions. If an unqualified <<EOF>> |
| rule is given, it applies to _all_ start conditions which do not |
| already have <<EOF>> actions. To specify an <<EOF>> rule for only the |
| initial start condition, use: |
| |
| <INITIAL><<EOF>> |
| |
| These rules are useful for catching things like unclosed comments. |
| An example: |
| |
| %x quote |
| %% |
| |
| ...other rules for dealing with quotes... |
| |
| <quote><<EOF>> { |
| error( "unterminated quote" ); |
| yyterminate(); |
| } |
| <<EOF>> { |
| if ( *++filelist ) |
| yyin = fopen( *filelist, "r" ); |
| else |
| yyterminate(); |
| } |
| |
| |
| File: flex.info, Node: Misc Macros, Next: User Values, Prev: EOF, Up: Top |
| |
| 13 Miscellaneous Macros |
| *********************** |
| |
| The macro `YY_USER_ACTION' can be defined to provide an action which is |
| always executed prior to the matched rule's action. For example, it |
| could be #define'd to call a routine to convert yytext to lower-case. |
| When `YY_USER_ACTION' is invoked, the variable `yy_act' gives the |
| number of the matched rule (rules are numbered starting with 1). |
| Suppose you want to profile how often each of your rules is matched. |
| The following would do the trick: |
| |
| #define YY_USER_ACTION ++ctr[yy_act] |
| |
| where `ctr' is an array to hold the counts for the different rules. |
| Note that the macro `YY_NUM_RULES' gives the total number of rules |
| (including the default rule), even if you use `-s)', so a correct |
| declaration for `ctr' is: |
| |
| int ctr[YY_NUM_RULES]; |
| |
| The macro `YY_USER_INIT' may be defined to provide an action which |
| is always executed before the first scan (and before the scanner's |
| internal initializations are done). For example, it could be used to |
| call a routine to read in a data table or open a logging file. |
| |
| The macro `yy_set_interactive(is_interactive)' can be used to |
| control whether the current buffer is considered "interactive". An |
| interactive buffer is processed more slowly, but must be used when the |
| scanner's input source is indeed interactive to avoid problems due to |
| waiting to fill buffers (see the discussion of the `-I' flag in *note |
| Scanner Options::). A non-zero value in the macro invocation marks the |
| buffer as interactive, a zero value as non-interactive. Note that use |
| of this macro overrides `%option always-interactive' or `%option |
| never-interactive' (*note Scanner Options::). `yy_set_interactive()' |
| must be invoked prior to beginning to scan the buffer that is (or is |
| not) to be considered interactive. |
| |
| The macro `yy_set_bol(at_bol)' can be used to control whether the |
| current buffer's scanning context for the next token match is done as |
| though at the beginning of a line. A non-zero macro argument makes |
| rules anchored with `^' active, while a zero argument makes `^' rules |
| inactive. |
| |
| The macro `YY_AT_BOL()' returns true if the next token scanned from |
| the current buffer will have `^' rules active, false otherwise. |
| |
| In the generated scanner, the actions are all gathered in one large |
| switch statement and separated using `YY_BREAK', which may be |
| redefined. By default, it is simply a `break', to separate each rule's |
| action from the following rule's. Redefining `YY_BREAK' allows, for |
| example, C++ users to #define YY_BREAK to do nothing (while being very |
| careful that every rule ends with a `break' or a `return'!) to avoid |
| suffering from unreachable statement warnings where because a rule's |
| action ends with `return', the `YY_BREAK' is inaccessible. |
| |
| |
| File: flex.info, Node: User Values, Next: Yacc, Prev: Misc Macros, Up: Top |
| |
| 14 Values Available To the User |
| ******************************* |
| |
| This chapter summarizes the various values available to the user in the |
| rule actions. |
| |
| `char *yytext' |
| holds the text of the current token. It may be modified but not |
| lengthened (you cannot append characters to the end). |
| |
| If the special directive `%array' appears in the first section of |
| the scanner description, then `yytext' is instead declared `char |
| yytext[YYLMAX]', where `YYLMAX' is a macro definition that you can |
| redefine in the first section if you don't like the default value |
| (generally 8KB). Using `%array' results in somewhat slower |
| scanners, but the value of `yytext' becomes immune to calls to |
| `unput()', which potentially destroy its value when `yytext' is a |
| character pointer. The opposite of `%array' is `%pointer', which |
| is the default. |
| |
| You cannot use `%array' when generating C++ scanner classes (the |
| `-+' flag). |
| |
| `int yyleng' |
| holds the length of the current token. |
| |
| `FILE *yyin' |
| is the file which by default `flex' reads from. It may be |
| redefined but doing so only makes sense before scanning begins or |
| after an EOF has been encountered. Changing it in the midst of |
| scanning will have unexpected results since `flex' buffers its |
| input; use `yyrestart()' instead. Once scanning terminates |
| because an end-of-file has been seen, you can assign `yyin' at the |
| new input file and then call the scanner again to continue |
| scanning. |
| |
| `void yyrestart( FILE *new_file )' |
| may be called to point `yyin' at the new input file. The |
| switch-over to the new file is immediate (any previously |
| buffered-up input is lost). Note that calling `yyrestart()' with |
| `yyin' as an argument thus throws away the current input buffer |
| and continues scanning the same input file. |
| |
| `FILE *yyout' |
| is the file to which `ECHO' actions are done. It can be reassigned |
| by the user. |
| |
| `YY_CURRENT_BUFFER' |
| returns a `YY_BUFFER_STATE' handle to the current buffer. |
| |
| `YY_START' |
| returns an integer value corresponding to the current start |
| condition. You can subsequently use this value with `BEGIN' to |
| return to that start condition. |
| |
| |
| File: flex.info, Node: Yacc, Next: Scanner Options, Prev: User Values, Up: Top |
| |
| 15 Interfacing with Yacc |
| ************************ |
| |
| One of the main uses of `flex' is as a companion to the `yacc' |
| parser-generator. `yacc' parsers expect to call a routine named |
| `yylex()' to find the next input token. The routine is supposed to |
| return the type of the next token as well as putting any associated |
| value in the global `yylval'. To use `flex' with `yacc', one specifies |
| the `-d' option to `yacc' to instruct it to generate the file `y.tab.h' |
| containing definitions of all the `%tokens' appearing in the `yacc' |
| input. This file is then included in the `flex' scanner. For example, |
| if one of the tokens is `TOK_NUMBER', part of the scanner might look |
| like: |
| |
| %{ |
| #include "y.tab.h" |
| %} |
| |
| %% |
| |
| [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER; |
| |
| |
| File: flex.info, Node: Scanner Options, Next: Performance, Prev: Yacc, Up: Top |
| |
| 16 Scanner Options |
| ****************** |
| |
| The various `flex' options are categorized by function in the following |
| menu. If you want to lookup a particular option by name, *Note Index of |
| Scanner Options::. |
| |
| * Menu: |
| |
| * Options for Specifying Filenames:: |
| * Options Affecting Scanner Behavior:: |
| * Code-Level And API Options:: |
| * Options for Scanner Speed and Size:: |
| * Debugging Options:: |
| * Miscellaneous Options:: |
| |
| Even though there are many scanner options, a typical scanner might |
| only specify the following options: |
| |
| %option 8bit reentrant bison-bridge |
| %option warn nodefault |
| %option yylineno |
| %option outfile="scanner.c" header-file="scanner.h" |
| |
| The first line specifies the general type of scanner we want. The |
| second line specifies that we are being careful. The third line asks |
| flex to track line numbers. The last line tells flex what to name the |
| files. (The options can be specified in any order. We just divided |
| them.) |
| |
| `flex' also provides a mechanism for controlling options within the |
| scanner specification itself, rather than from the flex command-line. |
| This is done by including `%option' directives in the first section of |
| the scanner specification. You can specify multiple options with a |
| single `%option' directive, and multiple directives in the first |
| section of your flex input file. |
| |
| Most options are given simply as names, optionally preceded by the |
| word `no' (with no intervening whitespace) to negate their meaning. |
| The names are the same as their long-option equivalents (but without the |
| leading `--' ). |
| |
| `flex' scans your rule actions to determine whether you use the |
| `REJECT' or `yymore()' features. The `REJECT' and `yymore' options are |
| available to override its decision as to whether you use the options, |
| either by setting them (e.g., `%option reject)' to indicate the feature |
| is indeed used, or unsetting them to indicate it actually is not used |
| (e.g., `%option noyymore)'. |
| |
| A number of options are available for lint purists who want to |
| suppress the appearance of unneeded routines in the generated scanner. |
| Each of the following, if unset (e.g., `%option nounput'), results in |
| the corresponding routine not appearing in the generated scanner: |
| |
| input, unput |
| yy_push_state, yy_pop_state, yy_top_state |
| yy_scan_buffer, yy_scan_bytes, yy_scan_string |
| |
| yyget_extra, yyset_extra, yyget_leng, yyget_text, |
| yyget_lineno, yyset_lineno, yyget_in, yyset_in, |
| yyget_out, yyset_out, yyget_lval, yyset_lval, |
| yyget_lloc, yyset_lloc, yyget_debug, yyset_debug |
| |
| (though `yy_push_state()' and friends won't appear anyway unless you |
| use `%option stack)'. |
| |
| |
| File: flex.info, Node: Options for Specifying Filenames, Next: Options Affecting Scanner Behavior, Prev: Scanner Options, Up: Scanner Options |
| |
| 16.1 Options for Specifying Filenames |
| ===================================== |
| |
| `--header-file=FILE, `%option header-file="FILE"'' |
| instructs flex to write a C header to `FILE'. This file contains |
| function prototypes, extern variables, and types used by the |
| scanner. Only the external API is exported by the header file. |
| Many macros that are usable from within scanner actions are not |
| exported to the header file. This is due to namespace problems and |
| the goal of a clean external API. |
| |
| While in the header, the macro `yyIN_HEADER' is defined, where `yy' |
| is substituted with the appropriate prefix. |
| |
| The `--header-file' option is not compatible with the `--c++' |
| option, since the C++ scanner provides its own header in |
| `yyFlexLexer.h'. |
| |
| `-oFILE, --outfile=FILE, `%option outfile="FILE"'' |
| directs flex to write the scanner to the file `FILE' instead of |
| `lex.yy.c'. If you combine `--outfile' with the `--stdout' option, |
| then the scanner is written to `stdout' but its `#line' directives |
| (see the `-l' option above) refer to the file `FILE'. |
| |
| `-t, --stdout, `%option stdout'' |
| instructs `flex' to write the scanner it generates to standard |
| output instead of `lex.yy.c'. |
| |
| `-SFILE, --skel=FILE' |
| overrides the default skeleton file from which `flex' constructs |
| its scanners. You'll never need this option unless you are doing |
| `flex' maintenance or development. |
| |
| `--tables-file=FILE' |
| Write serialized scanner dfa tables to FILE. The generated scanner |
| will not contain the tables, and requires them to be loaded at |
| runtime. *Note serialization::. |
| |
| `--tables-verify' |
| This option is for flex development. We document it here in case |
| you stumble upon it by accident or in case you suspect some |
| inconsistency in the serialized tables. Flex will serialize the |
| scanner dfa tables but will also generate the in-code tables as it |
| normally does. At runtime, the scanner will verify that the |
| serialized tables match the in-code tables, instead of loading |
| them. |
| |
| |
| |
| File: flex.info, Node: Options Affecting Scanner Behavior, Next: Code-Level And API Options, Prev: Options for Specifying Filenames, Up: Scanner Options |
| |
| 16.2 Options Affecting Scanner Behavior |
| ======================================= |
| |
| `-i, --case-insensitive, `%option case-insensitive'' |
| instructs `flex' to generate a "case-insensitive" scanner. The |
| case of letters given in the `flex' input patterns will be ignored, |
| and tokens in the input will be matched regardless of case. The |
| matched text given in `yytext' will have the preserved case (i.e., |
| it will not be folded). For tricky behavior, see *note case and |
| character ranges::. |
| |
| `-l, --lex-compat, `%option lex-compat'' |
| turns on maximum compatibility with the original AT&T `lex' |
| implementation. Note that this does not mean _full_ compatibility. |
| Use of this option costs a considerable amount of performance, and |
| it cannot be used with the `--c++', `--full', `--fast', `-Cf', or |
| `-CF' options. For details on the compatibilities it provides, see |
| *note Lex and Posix::. This option also results in the name |
| `YY_FLEX_LEX_COMPAT' being `#define''d in the generated scanner. |
| |
| `-B, --batch, `%option batch'' |
| instructs `flex' to generate a "batch" scanner, the opposite of |
| _interactive_ scanners generated by `--interactive' (see below). |
| In general, you use `-B' when you are _certain_ that your scanner |
| will never be used interactively, and you want to squeeze a |
| _little_ more performance out of it. If your goal is instead to |
| squeeze out a _lot_ more performance, you should be using the |
| `-Cf' or `-CF' options, which turn on `--batch' automatically |
| anyway. |
| |
| `-I, --interactive, `%option interactive'' |
| instructs `flex' to generate an interactive scanner. An |
| interactive scanner is one that only looks ahead to decide what |
| token has been matched if it absolutely must. It turns out that |
| always looking one extra character ahead, even if the scanner has |
| already seen enough text to disambiguate the current token, is a |
| bit faster than only looking ahead when necessary. But scanners |
| that always look ahead give dreadful interactive performance; for |
| example, when a user types a newline, it is not recognized as a |
| newline token until they enter _another_ token, which often means |
| typing in another whole line. |
| |
| `flex' scanners default to `interactive' unless you use the `-Cf' |
| or `-CF' table-compression options (*note Performance::). That's |
| because if you're looking for high-performance you should be using |
| one of these options, so if you didn't, `flex' assumes you'd |
| rather trade off a bit of run-time performance for intuitive |
| interactive behavior. Note also that you _cannot_ use |
| `--interactive' in conjunction with `-Cf' or `-CF'. Thus, this |
| option is not really needed; it is on by default for all those |
| cases in which it is allowed. |
| |
| You can force a scanner to _not_ be interactive by using `--batch' |
| |
| `-7, --7bit, `%option 7bit'' |
| instructs `flex' to generate a 7-bit scanner, i.e., one which can |
| only recognize 7-bit characters in its input. The advantage of |
| using `--7bit' is that the scanner's tables can be up to half the |
| size of those generated using the `--8bit'. The disadvantage is |
| that such scanners often hang or crash if their input contains an |
| 8-bit character. |
| |
| Note, however, that unless you generate your scanner using the |
| `-Cf' or `-CF' table compression options, use of `--7bit' will |
| save only a small amount of table space, and make your scanner |
| considerably less portable. `Flex''s default behavior is to |
| generate an 8-bit scanner unless you use the `-Cf' or `-CF', in |
| which case `flex' defaults to generating 7-bit scanners unless |
| your site was always configured to generate 8-bit scanners (as will |
| often be the case with non-USA sites). You can tell whether flex |
| generated a 7-bit or an 8-bit scanner by inspecting the flag |
| summary in the `--verbose' output as described above. |
| |
| Note that if you use `-Cfe' or `-CFe' `flex' still defaults to |
| generating an 8-bit scanner, since usually with these compression |
| options full 8-bit tables are not much more expensive than 7-bit |
| tables. |
| |
| `-8, --8bit, `%option 8bit'' |
| instructs `flex' to generate an 8-bit scanner, i.e., one which can |
| recognize 8-bit characters. This flag is only needed for scanners |
| generated using `-Cf' or `-CF', as otherwise flex defaults to |
| generating an 8-bit scanner anyway. |
| |
| See the discussion of `--7bit' above for `flex''s default behavior |
| and the tradeoffs between 7-bit and 8-bit scanners. |
| |
| `--default, `%option default'' |
| generate the default rule. |
| |
| `--always-interactive, `%option always-interactive'' |
| instructs flex to generate a scanner which always considers its |
| input _interactive_. Normally, on each new input file the scanner |
| calls `isatty()' in an attempt to determine whether the scanner's |
| input source is interactive and thus should be read a character at |
| a time. When this option is used, however, then no such call is |
| made. |
| |
| `--never-interactive, `--never-interactive'' |
| instructs flex to generate a scanner which never considers its |
| input interactive. This is the opposite of `always-interactive'. |
| |
| `-X, --posix, `%option posix'' |
| turns on maximum compatibility with the POSIX 1003.2-1992 |
| definition of `lex'. Since `flex' was originally designed to |
| implement the POSIX definition of `lex' this generally involves |
| very few changes in behavior. At the current writing the known |
| differences between `flex' and the POSIX standard are: |
| |
| * In POSIX and AT&T `lex', the repeat operator, `{}', has lower |
| precedence than concatenation (thus `ab{3}' yields `ababab'). |
| Most POSIX utilities use an Extended Regular Expression (ERE) |
| precedence that has the precedence of the repeat operator |
| higher than concatenation (which causes `ab{3}' to yield |
| `abbb'). By default, `flex' places the precedence of the |
| repeat operator higher than concatenation which matches the |
| ERE processing of other POSIX utilities. When either |
| `--posix' or `-l' are specified, `flex' will use the |
| traditional AT&T and POSIX-compliant precedence for the |
| repeat operator where concatenation has higher precedence |
| than the repeat operator. |
| |
| `--stack, `%option stack'' |
| enables the use of start condition stacks (*note Start |
| Conditions::). |
| |
| `--stdinit, `%option stdinit'' |
| if set (i.e., %option stdinit) initializes `yyin' and `yyout' to |
| `stdin' and `stdout', instead of the default of `NULL'. Some |
| existing `lex' programs depend on this behavior, even though it is |
| not compliant with ANSI C, which does not require `stdin' and |
| `stdout' to be compile-time constant. In a reentrant scanner, |
| however, this is not a problem since initialization is performed |
| in `yylex_init' at runtime. |
| |
| `--yylineno, `%option yylineno'' |
| directs `flex' to generate a scanner that maintains the number of |
| the current line read from its input in the global variable |
| `yylineno'. This option is implied by `%option lex-compat'. In a |
| reentrant C scanner, the macro `yylineno' is accessible regardless |
| of the value of `%option yylineno', however, its value is not |
| modified by `flex' unless `%option yylineno' is enabled. |
| |
| `--yywrap, `%option yywrap'' |
| if unset (i.e., `--noyywrap)', makes the scanner not call |
| `yywrap()' upon an end-of-file, but simply assume that there are no |
| more files to scan (until the user points `yyin' at a new file and |
| calls `yylex()' again). |
| |
| |
| |
| File: flex.info, Node: Code-Level And API Options, Next: Options for Scanner Speed and Size, Prev: Options Affecting Scanner Behavior, Up: Scanner Options |
| |
| 16.3 Code-Level And API Options |
| =============================== |
| |
| `--ansi-definitions, `%option ansi-definitions'' |
| instruct flex to generate ANSI C99 definitions for functions. |
| This option is enabled by default. If `%option |
| noansi-definitions' is specified, then the obsolete style is |
| generated. |
| |
| `--ansi-prototypes, `%option ansi-prototypes'' |
| instructs flex to generate ANSI C99 prototypes for functions. |
| This option is enabled by default. If `noansi-prototypes' is |
| specified, then prototypes will have empty parameter lists. |
| |
| `--bison-bridge, `%option bison-bridge'' |
| instructs flex to generate a C scanner that is meant to be called |
| by a `GNU bison' parser. The scanner has minor API changes for |
| `bison' compatibility. In particular, the declaration of `yylex' |
| is modified to take an additional parameter, `yylval'. *Note |
| Bison Bridge::. |
| |
| `--bison-locations, `%option bison-locations'' |
| instruct flex that `GNU bison' `%locations' are being used. This |
| means `yylex' will be passed an additional parameter, `yylloc'. |
| This option implies `%option bison-bridge'. *Note Bison Bridge::. |
| |
| `-L, --noline, `%option noline'' |
| instructs `flex' not to generate `#line' directives. Without this |
| option, `flex' peppers the generated scanner with `#line' |
| directives so error messages in the actions will be correctly |
| located with respect to either the original `flex' input file (if |
| the errors are due to code in the input file), or `lex.yy.c' (if |
| the errors are `flex''s fault - you should report these sorts of |
| errors to the email address given in *note Reporting Bugs::). |
| |
| `-R, --reentrant, `%option reentrant'' |
| instructs flex to generate a reentrant C scanner. The generated |
| scanner may safely be used in a multi-threaded environment. The |
| API for a reentrant scanner is different than for a non-reentrant |
| scanner *note Reentrant::). Because of the API difference between |
| reentrant and non-reentrant `flex' scanners, non-reentrant flex |
| code must be modified before it is suitable for use with this |
| option. This option is not compatible with the `--c++' option. |
| |
| The option `--reentrant' does not affect the performance of the |
| scanner. |
| |
| `-+, --c++, `%option c++'' |
| specifies that you want flex to generate a C++ scanner class. |
| *Note Cxx::, for details. |
| |
| `--array, `%option array'' |
| specifies that you want yytext to be an array instead of a char* |
| |
| `--pointer, `%option pointer'' |
| specify that `yytext' should be a `char *', not an array. This |
| default is `char *'. |
| |
| `-PPREFIX, --prefix=PREFIX, `%option prefix="PREFIX"'' |
| changes the default `yy' prefix used by `flex' for all |
| globally-visible variable and function names to instead be |
| `PREFIX'. For example, `--prefix=foo' changes the name of |
| `yytext' to `footext'. It also changes the name of the default |
| output file from `lex.yy.c' to `lex.foo.c'. Here is a partial |
| list of the names affected: |
| |
| yy_create_buffer |
| yy_delete_buffer |
| yy_flex_debug |
| yy_init_buffer |
| yy_flush_buffer |
| yy_load_buffer_state |
| yy_switch_to_buffer |
| yyin |
| yyleng |
| yylex |
| yylineno |
| yyout |
| yyrestart |
| yytext |
| yywrap |
| yyalloc |
| yyrealloc |
| yyfree |
| |
| (If you are using a C++ scanner, then only `yywrap' and |
| `yyFlexLexer' are affected.) Within your scanner itself, you can |
| still refer to the global variables and functions using either |
| version of their name; but externally, they have the modified name. |
| |
| This option lets you easily link together multiple `flex' programs |
| into the same executable. Note, though, that using this option |
| also renames `yywrap()', so you now _must_ either provide your own |
| (appropriately-named) version of the routine for your scanner, or |
| use `%option noyywrap', as linking with `-lfl' no longer provides |
| one for you by default. |
| |
| `--main, `%option main'' |
| directs flex to provide a default `main()' program for the |
| scanner, which simply calls `yylex()'. This option implies |
| `noyywrap' (see below). |
| |
| `--nounistd, `%option nounistd'' |
| suppresses inclusion of the non-ANSI header file `unistd.h'. This |
| option is meant to target environments in which `unistd.h' does |
| not exist. Be aware that certain options may cause flex to |
| generate code that relies on functions normally found in |
| `unistd.h', (e.g. `isatty()', `read()'.) If you wish to use these |
| functions, you will have to inform your compiler where to find |
| them. *Note option-always-interactive::. *Note option-read::. |
| |
| `--yyclass=NAME, `%option yyclass="NAME"'' |
| only applies when generating a C++ scanner (the `--c++' option). |
| It informs `flex' that you have derived `NAME' as a subclass of |
| `yyFlexLexer', so `flex' will place your actions in the member |
| function `foo::yylex()' instead of `yyFlexLexer::yylex()'. It |
| also generates a `yyFlexLexer::yylex()' member function that emits |
| a run-time error (by invoking `yyFlexLexer::LexerError())' if |
| called. *Note Cxx::. |
| |
| |
| |
| File: flex.info, Node: Options for Scanner Speed and Size, Next: Debugging Options, Prev: Code-Level And API Options, Up: Scanner Options |
| |
| 16.4 Options for Scanner Speed and Size |
| ======================================= |
| |
| `-C[aefFmr]' |
| controls the degree of table compression and, more generally, |
| trade-offs between small scanners and fast scanners. |
| |
| `-C' |
| A lone `-C' specifies that the scanner tables should be |
| compressed but neither equivalence classes nor |
| meta-equivalence classes should be used. |
| |
| `-Ca, --align, `%option align'' |
| ("align") instructs flex to trade off larger tables in the |
| generated scanner for faster performance because the elements |
| of the tables are better aligned for memory access and |
| computation. On some RISC architectures, fetching and |
| manipulating longwords is more efficient than with |
| smaller-sized units such as shortwords. This option can |
| quadruple the size of the tables used by your scanner. |
| |
| `-Ce, --ecs, `%option ecs'' |
| directs `flex' to construct "equivalence classes", i.e., sets |
| of characters which have identical lexical properties (for |
| example, if the only appearance of digits in the `flex' input |
| is in the character class "[0-9]" then the digits '0', '1', |
| ..., '9' will all be put in the same equivalence class). |
| Equivalence classes usually give dramatic reductions in the |
| final table/object file sizes (typically a factor of 2-5) and |
| are pretty cheap performance-wise (one array look-up per |
| character scanned). |
| |
| `-Cf' |
| specifies that the "full" scanner tables should be generated - |
| `flex' should not compress the tables by taking advantages of |
| similar transition functions for different states. |
| |
| `-CF' |
| specifies that the alternate fast scanner representation |
| (described above under the `--fast' flag) should be used. |
| This option cannot be used with `--c++'. |
| |
| `-Cm, --meta-ecs, `%option meta-ecs'' |
| directs `flex' to construct "meta-equivalence classes", which |
| are sets of equivalence classes (or characters, if equivalence |
| classes are not being used) that are commonly used together. |
| Meta-equivalence classes are often a big win when using |
| compressed tables, but they have a moderate performance |
| impact (one or two `if' tests and one array look-up per |
| character scanned). |
| |
| `-Cr, --read, `%option read'' |
| causes the generated scanner to _bypass_ use of the standard |
| I/O library (`stdio') for input. Instead of calling |
| `fread()' or `getc()', the scanner will use the `read()' |
| system call, resulting in a performance gain which varies |
| from system to system, but in general is probably negligible |
| unless you are also using `-Cf' or `-CF'. Using `-Cr' can |
| cause strange behavior if, for example, you read from `yyin' |
| using `stdio' prior to calling the scanner (because the |
| scanner will miss whatever text your previous reads left in |
| the `stdio' input buffer). `-Cr' has no effect if you define |
| `YY_INPUT()' (*note Generated Scanner::). |
| |
| The options `-Cf' or `-CF' and `-Cm' do not make sense together - |
| there is no opportunity for meta-equivalence classes if the table |
| is not being compressed. Otherwise the options may be freely |
| mixed, and are cumulative. |
| |
| The default setting is `-Cem', which specifies that `flex' should |
| generate equivalence classes and meta-equivalence classes. This |
| setting provides the highest degree of table compression. You can |
| trade off faster-executing scanners at the cost of larger tables |
| with the following generally being true: |
| |
| slowest & smallest |
| -Cem |
| -Cm |
| -Ce |
| -C |
| -C{f,F}e |
| -C{f,F} |
| -C{f,F}a |
| fastest & largest |
| |
| Note that scanners with the smallest tables are usually generated |
| and compiled the quickest, so during development you will usually |
| want to use the default, maximal compression. |
| |
| `-Cfe' is often a good compromise between speed and size for |
| production scanners. |
| |
| `-f, --full, `%option full'' |
| specifies "fast scanner". No table compression is done and |
| `stdio' is bypassed. The result is large but fast. This option |
| is equivalent to `--Cfr' |
| |
| `-F, --fast, `%option fast'' |
| specifies that the _fast_ scanner table representation should be |
| used (and `stdio' bypassed). This representation is about as fast |
| as the full table representation `--full', and for some sets of |
| patterns will be considerably smaller (and for others, larger). In |
| general, if the pattern set contains both _keywords_ and a |
| catch-all, _identifier_ rule, such as in the set: |
| |
| "case" return TOK_CASE; |
| "switch" return TOK_SWITCH; |
| ... |
| "default" return TOK_DEFAULT; |
| [a-z]+ return TOK_ID; |
| |
| then you're better off using the full table representation. If |
| only the _identifier_ rule is present and you then use a hash |
| table or some such to detect the keywords, you're better off using |
| `--fast'. |
| |
| This option is equivalent to `-CFr'. It cannot be used with |
| `--c++'. |
| |
| |
| |
| File: flex.info, Node: Debugging Options, Next: Miscellaneous Options, Prev: Options for Scanner Speed and Size, Up: Scanner Options |
| |
| 16.5 Debugging Options |
| ====================== |
| |
| `-b, --backup, `%option backup'' |
| Generate backing-up information to `lex.backup'. This is a list of |
| scanner states which require backing up and the input characters on |
| which they do so. By adding rules one can remove backing-up |
| states. If _all_ backing-up states are eliminated and `-Cf' or |
| `-CF' is used, the generated scanner will run faster (see the |
| `--perf-report' flag). Only users who wish to squeeze every last |
| cycle out of their scanners need worry about this option. (*note |
| Performance::). |
| |
| `-d, --debug, `%option debug'' |
| makes the generated scanner run in "debug" mode. Whenever a |
| pattern is recognized and the global variable `yy_flex_debug' is |
| non-zero (which is the default), the scanner will write to |
| `stderr' a line of the form: |
| |
| -accepting rule at line 53 ("the matched text") |
| |
| The line number refers to the location of the rule in the file |
| defining the scanner (i.e., the file that was fed to flex). |
| Messages are also generated when the scanner backs up, accepts the |
| default rule, reaches the end of its input buffer (or encounters a |
| NUL; at this point, the two look the same as far as the scanner's |
| concerned), or reaches an end-of-file. |
| |
| `-p, --perf-report, `%option perf-report'' |
| generates a performance report to `stderr'. The report consists of |
| comments regarding features of the `flex' input file which will |
| cause a serious loss of performance in the resulting scanner. If |
| you give the flag twice, you will also get comments regarding |
| features that lead to minor performance losses. |
| |
| Note that the use of `REJECT', and variable trailing context |
| (*note Limitations::) entails a substantial performance penalty; |
| use of `yymore()', the `^' operator, and the `--interactive' flag |
| entail minor performance penalties. |
| |
| `-s, --nodefault, `%option nodefault'' |
| causes the _default rule_ (that unmatched scanner input is echoed |
| to `stdout)' to be suppressed. If the scanner encounters input |
| that does not match any of its rules, it aborts with an error. |
| This option is useful for finding holes in a scanner's rule set. |
| |
| `-T, --trace, `%option trace'' |
| makes `flex' run in "trace" mode. It will generate a lot of |
| messages to `stderr' concerning the form of the input and the |
| resultant non-deterministic and deterministic finite automata. |
| This option is mostly for use in maintaining `flex'. |
| |
| `-w, --nowarn, `%option nowarn'' |
| suppresses warning messages. |
| |
| `-v, --verbose, `%option verbose'' |
| specifies that `flex' should write to `stderr' a summary of |
| statistics regarding the scanner it generates. Most of the |
| statistics are meaningless to the casual `flex' user, but the |
| first line identifies the version of `flex' (same as reported by |
| `--version'), and the next line the flags used when generating the |
| scanner, including those that are on by default. |
| |
| `--warn, `%option warn'' |
| warn about certain things. In particular, if the default rule can |
| be matched but no default rule has been given, the flex will warn |
| you. We recommend using this option always. |
| |
| |
| |
| File: flex.info, Node: Miscellaneous Options, Prev: Debugging Options, Up: Scanner Options |
| |
| 16.6 Miscellaneous Options |
| ========================== |
| |
| `-c' |
| A do-nothing option included for POSIX compliance. |
| |
| `-h, -?, --help' |
| generates a "help" summary of `flex''s options to `stdout' and |
| then exits. |
| |
| `-n' |
| Another do-nothing option included for POSIX compliance. |
| |
| `-V, --version' |
| prints the version number to `stdout' and exits. |
| |
| |
| |
| File: flex.info, Node: Performance, Next: Cxx, Prev: Scanner Options, Up: Top |
| |
| 17 Performance Considerations |
| ***************************** |
| |
| The main design goal of `flex' is that it generate high-performance |
| scanners. It has been optimized for dealing well with large sets of |
| rules. Aside from the effects on scanner speed of the table compression |
| `-C' options outlined above, there are a number of options/actions |
| which degrade performance. These are, from most expensive to least: |
| |
| REJECT |
| arbitrary trailing context |
| |
| pattern sets that require backing up |
| %option yylineno |
| %array |
| |
| %option interactive |
| %option always-interactive |
| |
| ^ beginning-of-line operator |
| yymore() |
| |
| with the first two all being quite expensive and the last two being |
| quite cheap. Note also that `unput()' is implemented as a routine call |
| that potentially does quite a bit of work, while `yyless()' is a |
| quite-cheap macro. So if you are just putting back some excess text you |
| scanned, use `yyless()'. |
| |
| `REJECT' should be avoided at all costs when performance is |
| important. It is a particularly expensive option. |
| |
| There is one case when `%option yylineno' can be expensive. That is |
| when your patterns match long tokens that could _possibly_ contain a |
| newline character. There is no performance penalty for rules that can |
| not possibly match newlines, since flex does not need to check them for |
| newlines. In general, you should avoid rules such as `[^f]+', which |
| match very long tokens, including newlines, and may possibly match your |
| entire file! A better approach is to separate `[^f]+' into two rules: |
| |
| %option yylineno |
| %% |
| [^f\n]+ |
| \n+ |
| |
| The above scanner does not incur a performance penalty. |
| |
| Getting rid of backing up is messy and often may be an enormous |
| amount of work for a complicated scanner. In principal, one begins by |
| using the `-b' flag to generate a `lex.backup' file. For example, on |
| the input: |
| |
| %% |
| foo return TOK_KEYWORD; |
| foobar return TOK_KEYWORD; |
| |
| the file looks like: |
| |
| State #6 is non-accepting - |
| associated rule line numbers: |
| 2 3 |
| out-transitions: [ o ] |
| jam-transitions: EOF [ \001-n p-\177 ] |
| |
| State #8 is non-accepting - |
| associated rule line numbers: |
| 3 |
| out-transitions: [ a ] |
| jam-transitions: EOF [ \001-` b-\177 ] |
| |
| State #9 is non-accepting - |
| associated rule line numbers: |
| 3 |
| out-transitions: [ r ] |
| jam-transitions: EOF [ \001-q s-\177 ] |
| |
| Compressed tables always back up. |
| |
| The first few lines tell us that there's a scanner state in which it |
| can make a transition on an 'o' but not on any other character, and |
| that in that state the currently scanned text does not match any rule. |
| The state occurs when trying to match the rules found at lines 2 and 3 |
| in the input file. If the scanner is in that state and then reads |
| something other than an 'o', it will have to back up to find a rule |
| which is matched. With a bit of headscratching one can see that this |
| must be the state it's in when it has seen `fo'. When this has |
| happened, if anything other than another `o' is seen, the scanner will |
| have to back up to simply match the `f' (by the default rule). |
| |
| The comment regarding State #8 indicates there's a problem when |
| `foob' has been scanned. Indeed, on any character other than an `a', |
| the scanner will have to back up to accept "foo". Similarly, the |
| comment for State #9 concerns when `fooba' has been scanned and an `r' |
| does not follow. |
| |
| The final comment reminds us that there's no point going to all the |
| trouble of removing backing up from the rules unless we're using `-Cf' |
| or `-CF', since there's no performance gain doing so with compressed |
| scanners. |
| |
| The way to remove the backing up is to add "error" rules: |
| |
| %% |
| foo return TOK_KEYWORD; |
| foobar return TOK_KEYWORD; |
| |
| fooba | |
| foob | |
| fo { |
| /* false alarm, not really a keyword */ |
| return TOK_ID; |
| } |
| |
| Eliminating backing up among a list of keywords can also be done |
| using a "catch-all" rule: |
| |
| %% |
| foo return TOK_KEYWORD; |
| foobar return TOK_KEYWORD; |
| |
| [a-z]+ return TOK_ID; |
| |
| This is usually the best solution when appropriate. |
| |
| Backing up messages tend to cascade. With a complicated set of rules |
| it's not uncommon to get hundreds of messages. If one can decipher |
| them, though, it often only takes a dozen or so rules to eliminate the |
| backing up (though it's easy to make a mistake and have an error rule |
| accidentally match a valid token. A possible future `flex' feature |
| will be to automatically add rules to eliminate backing up). |
| |
| It's important to keep in mind that you gain the benefits of |
| eliminating backing up only if you eliminate _every_ instance of |
| backing up. Leaving just one means you gain nothing. |
| |
| _Variable_ trailing context (where both the leading and trailing |
| parts do not have a fixed length) entails almost the same performance |
| loss as `REJECT' (i.e., substantial). So when possible a rule like: |
| |
| %% |
| mouse|rat/(cat|dog) run(); |
| |
| is better written: |
| |
| %% |
| mouse/cat|dog run(); |
| rat/cat|dog run(); |
| |
| or as |
| |
| %% |
| mouse|rat/cat run(); |
| mouse|rat/dog run(); |
| |
| Note that here the special '|' action does _not_ provide any |
| savings, and can even make things worse (*note Limitations::). |
| |
| Another area where the user can increase a scanner's performance (and |
| one that's easier to implement) arises from the fact that the longer the |
| tokens matched, the faster the scanner will run. This is because with |
| long tokens the processing of most input characters takes place in the |
| (short) inner scanning loop, and does not often have to go through the |
| additional work of setting up the scanning environment (e.g., `yytext') |
| for the action. Recall the scanner for C comments: |
| |
| %x comment |
| %% |
| int line_num = 1; |
| |
| "/*" BEGIN(comment); |
| |
| <comment>[^*\n]* |
| <comment>"*"+[^*/\n]* |
| <comment>\n ++line_num; |
| <comment>"*"+"/" BEGIN(INITIAL); |
| |
| This could be sped up by writing it as: |
| |
| %x comment |
| %% |
| int line_num = 1; |
| |
| "/*" BEGIN(comment); |
| |
| <comment>[^*\n]* |
| <comment>[^*\n]*\n ++line_num; |
| <comment>"*"+[^*/\n]* |
| <comment>"*"+[^*/\n]*\n ++line_num; |
| <comment>"*"+"/" BEGIN(INITIAL); |
| |
| Now instead of each newline requiring the processing of another |
| action, recognizing the newlines is distributed over the other rules to |
| keep the matched text as long as possible. Note that _adding_ rules |
| does _not_ slow down the scanner! The speed of the scanner is |
| independent of the number of rules or (modulo the considerations given |
| at the beginning of this section) how complicated the rules are with |
| regard to operators such as `*' and `|'. |
| |
| A final example in speeding up a scanner: suppose you want to scan |
| through a file containing identifiers and keywords, one per line and |
| with no other extraneous characters, and recognize all the keywords. A |
| natural first approach is: |
| |
| %% |
| asm | |
| auto | |
| break | |
| ... etc ... |
| volatile | |
| while /* it's a keyword */ |
| |
| .|\n /* it's not a keyword */ |
| |
| To eliminate the back-tracking, introduce a catch-all rule: |
| |
| %% |
| asm | |
| auto | |
| break | |
| ... etc ... |
| volatile | |
| while /* it's a keyword */ |
| |
| [a-z]+ | |
| .|\n /* it's not a keyword */ |
| |
| Now, if it's guaranteed that there's exactly one word per line, then |
| we can reduce the total number of matches by a half by merging in the |
| recognition of newlines with that of the other tokens: |
| |
| %% |
| asm\n | |
| auto\n | |
| break\n | |
| ... etc ... |
| volatile\n | |
| while\n /* it's a keyword */ |
| |
| [a-z]+\n | |
| .|\n /* it's not a keyword */ |
| |
| One has to be careful here, as we have now reintroduced backing up |
| into the scanner. In particular, while _we_ know that there will never |
| be any characters in the input stream other than letters or newlines, |
| `flex' can't figure this out, and it will plan for possibly needing to |
| back up when it has scanned a token like `auto' and then the next |
| character is something other than a newline or a letter. Previously it |
| would then just match the `auto' rule and be done, but now it has no |
| `auto' rule, only a `auto\n' rule. To eliminate the possibility of |
| backing up, we could either duplicate all rules but without final |
| newlines, or, since we never expect to encounter such an input and |
| therefore don't how it's classified, we can introduce one more |
| catch-all rule, this one which doesn't include a newline: |
| |
| %% |
| asm\n | |
| auto\n | |
| break\n | |
| ... etc ... |
| volatile\n | |
| while\n /* it's a keyword */ |
| |
| [a-z]+\n | |
| [a-z]+ | |
| .|\n /* it's not a keyword */ |
| |
| Compiled with `-Cf', this is about as fast as one can get a `flex' |
| scanner to go for this particular problem. |
| |
| A final note: `flex' is slow when matching `NUL's, particularly when |
| a token contains multiple `NUL's. It's best to write rules which match |
| _short_ amounts of text if it's anticipated that the text will often |
| include `NUL's. |
| |
| Another final note regarding performance: as mentioned in *note |
| Matching::, dynamically resizing `yytext' to accommodate huge tokens is |
| a slow process because it presently requires that the (huge) token be |
| rescanned from the beginning. Thus if performance is vital, you should |
| attempt to match "large" quantities of text but not "huge" quantities, |
| where the cutoff between the two is at about 8K characters per token. |
| |
| |
| File: flex.info, Node: Cxx, Next: Reentrant, Prev: Performance, Up: Top |
| |
| 18 Generating C++ Scanners |
| ************************** |
| |
| *IMPORTANT*: the present form of the scanning class is _experimental_ |
| and may change considerably between major releases. |
| |
| `flex' provides two different ways to generate scanners for use with |
| C++. The first way is to simply compile a scanner generated by `flex' |
| using a C++ compiler instead of a C compiler. You should not encounter |
| any compilation errors (*note Reporting Bugs::). You can then use C++ |
| code in your rule actions instead of C code. Note that the default |
| input source for your scanner remains `yyin', and default echoing is |
| still done to `yyout'. Both of these remain `FILE *' variables and not |
| C++ _streams_. |
| |
| You can also use `flex' to generate a C++ scanner class, using the |
| `-+' option (or, equivalently, `%option c++)', which is automatically |
| specified if the name of the `flex' executable ends in a '+', such as |
| `flex++'. When using this option, `flex' defaults to generating the |
| scanner to the file `lex.yy.cc' instead of `lex.yy.c'. The generated |
| scanner includes the header file `FlexLexer.h', which defines the |
| interface to two C++ classes. |
| |
| The first class, `FlexLexer', provides an abstract base class |
| defining the general scanner class interface. It provides the |
| following member functions: |
| |
| `const char* YYText()' |
| returns the text of the most recently matched token, the |
| equivalent of `yytext'. |
| |
| `int YYLeng()' |
| returns the length of the most recently matched token, the |
| equivalent of `yyleng'. |
| |
| `int lineno() const' |
| returns the current input line number (see `%option yylineno)', or |
| `1' if `%option yylineno' was not used. |
| |
| `void set_debug( int flag )' |
| sets the debugging flag for the scanner, equivalent to assigning to |
| `yy_flex_debug' (*note Scanner Options::). Note that you must |
| build the scanner using `%option debug' to include debugging |
| information in it. |
| |
| `int debug() const' |
| returns the current setting of the debugging flag. |
| |
| Also provided are member functions equivalent to |
| `yy_switch_to_buffer()', `yy_create_buffer()' (though the first |
| argument is an `istream*' object pointer and not a `FILE*)', |
| `yy_flush_buffer()', `yy_delete_buffer()', and `yyrestart()' (again, |
| the first argument is a `istream*' object pointer). |
| |
| The second class defined in `FlexLexer.h' is `yyFlexLexer', which is |
| derived from `FlexLexer'. It defines the following additional member |
| functions: |
| |
| `yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 )' |
| constructs a `yyFlexLexer' object using the given streams for input |
| and output. If not specified, the streams default to `cin' and |
| `cout', respectively. |
| |
| `virtual int yylex()' |
| performs the same role is `yylex()' does for ordinary `flex' |
| scanners: it scans the input stream, consuming tokens, until a |
| rule's action returns a value. If you derive a subclass `S' from |
| `yyFlexLexer' and want to access the member functions and variables |
| of `S' inside `yylex()', then you need to use `%option |
| yyclass="S"' to inform `flex' that you will be using that subclass |
| instead of `yyFlexLexer'. In this case, rather than generating |
| `yyFlexLexer::yylex()', `flex' generates `S::yylex()' (and also |
| generates a dummy `yyFlexLexer::yylex()' that calls |
| `yyFlexLexer::LexerError()' if called). |
| |
| `virtual void switch_streams(istream* new_in = 0, ostream* new_out = 0)' |
| reassigns `yyin' to `new_in' (if non-null) and `yyout' to |
| `new_out' (if non-null), deleting the previous input buffer if |
| `yyin' is reassigned. |
| |
| `int yylex( istream* new_in, ostream* new_out = 0 )' |
| first switches the input streams via `switch_streams( new_in, |
| new_out )' and then returns the value of `yylex()'. |
| |
| In addition, `yyFlexLexer' defines the following protected virtual |
| functions which you can redefine in derived classes to tailor the |
| scanner: |
| |
| `virtual int LexerInput( char* buf, int max_size )' |
| reads up to `max_size' characters into `buf' and returns the |
| number of characters read. To indicate end-of-input, return 0 |
| characters. Note that `interactive' scanners (see the `-B' and |
| `-I' flags in *note Scanner Options::) define the macro |
| `YY_INTERACTIVE'. If you redefine `LexerInput()' and need to take |
| different actions depending on whether or not the scanner might be |
| scanning an interactive input source, you can test for the |
| presence of this name via `#ifdef' statements. |
| |
| `virtual void LexerOutput( const char* buf, int size )' |
| writes out `size' characters from the buffer `buf', which, while |
| `NUL'-terminated, may also contain internal `NUL's if the |
| scanner's rules can match text with `NUL's in them. |
| |
| `virtual void LexerError( const char* msg )' |
| reports a fatal error message. The default version of this |
| function writes the message to the stream `cerr' and exits. |
| |
| Note that a `yyFlexLexer' object contains its _entire_ scanning |
| state. Thus you can use such objects to create reentrant scanners, but |
| see also *note Reentrant::. You can instantiate multiple instances of |
| the same `yyFlexLexer' class, and you can also combine multiple C++ |
| scanner classes together in the same program using the `-P' option |
| discussed above. |
| |
| Finally, note that the `%array' feature is not available to C++ |
| scanner classes; you must use `%pointer' (the default). |
| |
| Here is an example of a simple C++ scanner: |
| |
| // An example of using the flex C++ scanner class. |
| |
| %{ |
| #include <iostream> |
| using namespace std; |
| int mylineno = 0; |
| %} |
| |
| %option noyywrap |
| |
| string \"[^\n"]+\" |
| |
| ws [ \t]+ |
| |
| alpha [A-Za-z] |
| dig [0-9] |
| name ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])* |
| num1 [-+]?{dig}+\.?([eE][-+]?{dig}+)? |
| num2 [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)? |
| number {num1}|{num2} |
| |
| %% |
| |
| {ws} /* skip blanks and tabs */ |
| |
| "/*" { |
| int c; |
| |
| while((c = yyinput()) != 0) |
| { |
| if(c == '\n') |
| ++mylineno; |
| |
| else if(c == '*') |
| { |
| if((c = yyinput()) == '/') |
| break; |
| else |
| unput(c); |
| } |
| } |
| } |
| |
| {number} cout << "number " << YYText() << '\n'; |
| |
| \n mylineno++; |
| |
| {name} cout << "name " << YYText() << '\n'; |
| |
| {string} cout << "string " << YYText() << '\n'; |
| |
| %% |
| |
| int main( int /* argc */, char** /* argv */ ) |
| { |
| FlexLexer* lexer = new yyFlexLexer; |
| while(lexer->yylex() != 0) |
| ; |
| return 0; |
| } |
| |
| If you want to create multiple (different) lexer classes, you use the |
| `-P' flag (or the `prefix=' option) to rename each `yyFlexLexer' to |
| some other `xxFlexLexer'. You then can include `<FlexLexer.h>' in your |
| other sources once per lexer class, first renaming `yyFlexLexer' as |
| follows: |
| |
| #undef yyFlexLexer |
| #define yyFlexLexer xxFlexLexer |
| #include <FlexLexer.h> |
| |
| #undef yyFlexLexer |
| #define yyFlexLexer zzFlexLexer |
| #include <FlexLexer.h> |
| |
| if, for example, you used `%option prefix="xx"' for one of your |
| scanners and `%option prefix="zz"' for the other. |
| |
| |
| File: flex.info, Node: Reentrant, Next: Lex and Posix, Prev: Cxx, Up: Top |
| |
| 19 Reentrant C Scanners |
| *********************** |
| |
| `flex' has the ability to generate a reentrant C scanner. This is |
| accomplished by specifying `%option reentrant' (`-R') The generated |
| scanner is both portable, and safe to use in one or more separate |
| threads of control. The most common use for reentrant scanners is from |
| within multi-threaded applications. Any thread may create and execute |
| a reentrant `flex' scanner without the need for synchronization with |
| other threads. |
| |
| * Menu: |
| |
| * Reentrant Uses:: |
| * Reentrant Overview:: |
| * Reentrant Example:: |
| * Reentrant Detail:: |
| * Reentrant Functions:: |
| |
| |
| File: flex.info, Node: Reentrant Uses, Next: Reentrant Overview, Prev: Reentrant, Up: Reentrant |
| |
| 19.1 Uses for Reentrant Scanners |
| ================================ |
| |
| However, there are other uses for a reentrant scanner. For example, you |
| could scan two or more files simultaneously to implement a `diff' at |
| the token level (i.e., instead of at the character level): |
| |
| /* Example of maintaining more than one active scanner. */ |
| |
| do { |
| int tok1, tok2; |
| |
| tok1 = yylex( scanner_1 ); |
| tok2 = yylex( scanner_2 ); |
| |
| if( tok1 != tok2 ) |
| printf("Files are different."); |
| |
| } while ( tok1 && tok2 ); |
| |
| Another use for a reentrant scanner is recursion. (Note that a |
| recursive scanner can also be created using a non-reentrant scanner and |
| buffer states. *Note Multiple Input Buffers::.) |
| |
| The following crude scanner supports the `eval' command by invoking |
| another instance of itself. |
| |
| /* Example of recursive invocation. */ |
| |
| %option reentrant |
| |
| %% |
| "eval(".+")" { |
| yyscan_t scanner; |
| YY_BUFFER_STATE buf; |
| |
| yylex_init( &scanner ); |
| yytext[yyleng-1] = ' '; |
| |
| buf = yy_scan_string( yytext + 5, scanner ); |
| yylex( scanner ); |
| |
| yy_delete_buffer(buf,scanner); |
| yylex_destroy( scanner ); |
| } |
| ... |
| %% |
| |
| |
| File: flex.info, Node: Reentrant Overview, Next: Reentrant Example, Prev: Reentrant Uses, Up: Reentrant |
| |
| 19.2 An Overview of the Reentrant API |
| ===================================== |
| |
| The API for reentrant scanners is different than for non-reentrant |
| scanners. Here is a quick overview of the API: |
| |
| `%option reentrant' must be specified. |
| |
| * All functions take one additional argument: `yyscanner' |
| |
| * All global variables are replaced by their macro equivalents. (We |
| tell you this because it may be important to you during debugging.) |
| |
| * `yylex_init' and `yylex_destroy' must be called before and after |
| `yylex', respectively. |
| |
| * Accessor methods (get/set functions) provide access to common |
| `flex' variables. |
| |
| * User-specific data can be stored in `yyextra'. |
| |
| |
| File: flex.info, Node: Reentrant Example, Next: Reentrant Detail, Prev: Reentrant Overview, Up: Reentrant |
| |
| 19.3 Reentrant Example |
| ====================== |
| |
| First, an example of a reentrant scanner: |
| /* This scanner prints "//" comments. */ |
| |
| %option reentrant stack noyywrap |
| %x COMMENT |
| |
| %% |
| |
| "//" yy_push_state( COMMENT, yyscanner); |
| .|\n |
| |
| <COMMENT>\n yy_pop_state( yyscanner ); |
| <COMMENT>[^\n]+ fprintf( yyout, "%s\n", yytext); |
| |
| %% |
| |
| int main ( int argc, char * argv[] ) |
| { |
| yyscan_t scanner; |
| |
| yylex_init ( &scanner ); |
| yylex ( scanner ); |
| yylex_destroy ( scanner ); |
| return 0; |
| } |
| |
| |
| File: flex.info, Node: Reentrant Detail, Next: Reentrant Functions, Prev: Reentrant Example, Up: Reentrant |
| |
| 19.4 The Reentrant API in Detail |
| ================================ |
| |
| Here are the things you need to do or know to use the reentrant C API of |
| `flex'. |
| |
| * Menu: |
| |
| * Specify Reentrant:: |
| * Extra Reentrant Argument:: |
| * Global Replacement:: |
| * Init and Destroy Functions:: |
| * Accessor Methods:: |
| * Extra Data:: |
| * About yyscan_t:: |
| |
| |
| File: flex.info, Node: Specify Reentrant, Next: Extra Reentrant Argument, Prev: Reentrant Detail, Up: Reentrant Detail |
| |
| 19.4.1 Declaring a Scanner As Reentrant |
| --------------------------------------- |
| |
| %option reentrant (-reentrant) must be specified. |
| |
| Notice that `%option reentrant' is specified in the above example |
| (*note Reentrant Example::. Had this option not been specified, `flex' |
| would have happily generated a non-reentrant scanner without |
| complaining. You may explicitly specify `%option noreentrant', if you |
| do _not_ want a reentrant scanner, although it is not necessary. The |
| default is to generate a non-reentrant scanner. |
| |
| |
| File: flex.info, Node: Extra Reentrant Argument, Next: Global Replacement, Prev: Specify Reentrant, Up: Reentrant Detail |
| |
| 19.4.2 The Extra Argument |
| ------------------------- |
| |
| All functions take one additional argument: `yyscanner'. |
| |
| Notice that the calls to `yy_push_state' and `yy_pop_state' both |
| have an argument, `yyscanner' , that is not present in a non-reentrant |
| scanner. Here are the declarations of `yy_push_state' and |
| `yy_pop_state' in the reentrant scanner: |
| |
| static void yy_push_state ( int new_state , yyscan_t yyscanner ) ; |
| static void yy_pop_state ( yyscan_t yyscanner ) ; |
| |
| Notice that the argument `yyscanner' appears in the declaration of |
| both functions. In fact, all `flex' functions in a reentrant scanner |
| have this additional argument. It is always the last argument in the |
| argument list, it is always of type `yyscan_t' (which is typedef'd to |
| `void *') and it is always named `yyscanner'. As you may have guessed, |
| `yyscanner' is a pointer to an opaque data structure encapsulating the |
| current state of the scanner. For a list of function declarations, see |
| *note Reentrant Functions::. Note that preprocessor macros, such as |
| `BEGIN', `ECHO', and `REJECT', do not take this additional argument. |
| |
| |
| File: flex.info, Node: Global Replacement, Next: Init and Destroy Functions, Prev: Extra Reentrant Argument, Up: Reentrant Detail |
| |
| 19.4.3 Global Variables Replaced By Macros |
| ------------------------------------------ |
| |
| All global variables in traditional flex have been replaced by macro |
| equivalents. |
| |
| Note that in the above example, `yyout' and `yytext' are not plain |
| variables. These are macros that will expand to their equivalent lvalue. |
| All of the familiar `flex' globals have been replaced by their macro |
| equivalents. In particular, `yytext', `yyleng', `yylineno', `yyin', |
| `yyout', `yyextra', `yylval', and `yylloc' are macros. You may safely |
| use these macros in actions as if they were plain variables. We only |
| tell you this so you don't expect to link to these variables |
| externally. Currently, each macro expands to a member of an internal |
| struct, e.g., |
| |
| #define yytext (((struct yyguts_t*)yyscanner)->yytext_r) |
| |
| One important thing to remember about `yytext' and friends is that |
| `yytext' is not a global variable in a reentrant scanner, you can not |
| access it directly from outside an action or from other functions. You |
| must use an accessor method, e.g., `yyget_text', to accomplish this. |
| (See below). |
| |
| |
| File: flex.info, Node: Init and Destroy Functions, Next: Accessor Methods, Prev: Global Replacement, Up: Reentrant Detail |
| |
| 19.4.4 Init and Destroy Functions |
| --------------------------------- |
| |
| `yylex_init' and `yylex_destroy' must be called before and after |
| `yylex', respectively. |
| |
| int yylex_init ( yyscan_t * ptr_yy_globals ) ; |
| int yylex_init_extra ( YY_EXTRA_TYPE user_defined, yyscan_t * ptr_yy_globals ) ; |
| int yylex ( yyscan_t yyscanner ) ; |
| int yylex_destroy ( yyscan_t yyscanner ) ; |
| |
| The function `yylex_init' must be called before calling any other |
| function. The argument to `yylex_init' is the address of an |
| uninitialized pointer to be filled in by `yylex_init', overwriting any |
| previous contents. The function `yylex_init_extra' may be used instead, |
| taking as its first argument a variable of type `YY_EXTRA_TYPE'. See |
| the section on yyextra, below, for more details. |
| |
| The value stored in `ptr_yy_globals' should thereafter be passed to |
| `yylex' and `yylex_destroy'. Flex does not save the argument passed to |
| `yylex_init', so it is safe to pass the address of a local pointer to |
| `yylex_init' so long as it remains in scope for the duration of all |
| calls to the scanner, up to and including the call to `yylex_destroy'. |
| |
| The function `yylex' should be familiar to you by now. The reentrant |
| version takes one argument, which is the value returned (via an |
| argument) by `yylex_init'. Otherwise, it behaves the same as the |
| non-reentrant version of `yylex'. |
| |
| Both `yylex_init' and `yylex_init_extra' returns 0 (zero) on success, |
| or non-zero on failure, in which case errno is set to one of the |
| following values: |
| |
| * ENOMEM Memory allocation error. *Note memory-management::. |
| |
| * EINVAL Invalid argument. |
| |
| The function `yylex_destroy' should be called to free resources used |
| by the scanner. After `yylex_destroy' is called, the contents of |
| `yyscanner' should not be used. Of course, there is no need to destroy |
| a scanner if you plan to reuse it. A `flex' scanner (both reentrant |
| and non-reentrant) may be restarted by calling `yyrestart'. |
| |
| Below is an example of a program that creates a scanner, uses it, |
| then destroys it when done: |
| |
| int main () |
| { |
| yyscan_t scanner; |
| int tok; |
| |
| yylex_init(&scanner); |
| |
| while ((tok=yylex(scanner)) > 0) |
| printf("tok=%d yytext=%s\n", tok, yyget_text(scanner)); |
| |
| yylex_destroy(scanner); |
| return 0; |
| } |
| |
| |
| File: flex.info, Node: Accessor Methods, Next: Extra Data, Prev: Init and Destroy Functions, Up: Reentrant Detail |
| |
| 19.4.5 Accessing Variables with Reentrant Scanners |
| -------------------------------------------------- |
| |
| Accessor methods (get/set functions) provide access to common `flex' |
| variables. |
| |
| Many scanners that you build will be part of a larger project. |
| Portions of your project will need access to `flex' values, such as |
| `yytext'. In a non-reentrant scanner, these values are global, so |
| there is no problem accessing them. However, in a reentrant scanner, |
| there are no global `flex' values. You can not access them directly. |
| Instead, you must access `flex' values using accessor methods (get/set |
| functions). Each accessor method is named `yyget_NAME' or `yyset_NAME', |
| where `NAME' is the name of the `flex' variable you want. For example: |
| |
| /* Set the last character of yytext to NULL. */ |
| void chop ( yyscan_t scanner ) |
| { |
| int len = yyget_leng( scanner ); |
| yyget_text( scanner )[len - 1] = '\0'; |
| } |
| |
| The above code may be called from within an action like this: |
| |
| %% |
| .+\n { chop( yyscanner );} |
| |
| You may find that `%option header-file' is particularly useful for |
| generating prototypes of all the accessor functions. *Note |
| option-header::. |
| |
| |
| File: flex.info, Node: Extra Data, Next: About yyscan_t, Prev: Accessor Methods, Up: Reentrant Detail |
| |
| 19.4.6 Extra Data |
| ----------------- |
| |
| User-specific data can be stored in `yyextra'. |
| |
| In a reentrant scanner, it is unwise to use global variables to |
| communicate with or maintain state between different pieces of your |
| program. However, you may need access to external data or invoke |
| external functions from within the scanner actions. Likewise, you may |
| need to pass information to your scanner (e.g., open file descriptors, |
| or database connections). In a non-reentrant scanner, the only way to |
| do this would be through the use of global variables. `Flex' allows |
| you to store arbitrary, "extra" data in a scanner. This data is |
| accessible through the accessor methods `yyget_extra' and `yyset_extra' |
| from outside the scanner, and through the shortcut macro `yyextra' from |
| within the scanner itself. They are defined as follows: |
| |
| #define YY_EXTRA_TYPE void* |
| YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner ); |
| void yyset_extra ( YY_EXTRA_TYPE arbitrary_data , yyscan_t scanner); |
| |
| In addition, an extra form of `yylex_init' is provided, |
| `yylex_init_extra'. This function is provided so that the yyextra value |
| can be accessed from within the very first yyalloc, used to allocate |
| the scanner itself. |
| |
| By default, `YY_EXTRA_TYPE' is defined as type `void *'. You may |
| redefine this type using `%option extra-type="your_type"' in the |
| scanner: |
| |
| /* An example of overriding YY_EXTRA_TYPE. */ |
| %{ |
| #include <sys/stat.h> |
| #include <unistd.h> |
| %} |
| %option reentrant |
| %option extra-type="struct stat *" |
| %% |
| |
| __filesize__ printf( "%ld", yyextra->st_size ); |
| __lastmod__ printf( "%ld", yyextra->st_mtime ); |
| %% |
| void scan_file( char* filename ) |
| { |
| yyscan_t scanner; |
| struct stat buf; |
| FILE *in; |
| |
| in = fopen( filename, "r" ); |
| stat( filename, &buf ); |
| |
| yylex_init_extra( buf, &scanner ); |
| yyset_in( in, scanner ); |
| yylex( scanner ); |
| yylex_destroy( scanner ); |
| |
| fclose( in ); |
| } |
| |
| |
| File: flex.info, Node: About yyscan_t, Prev: Extra Data, Up: Reentrant Detail |
| |
| 19.4.7 About yyscan_t |
| --------------------- |
| |
| `yyscan_t' is defined as: |
| |
| typedef void* yyscan_t; |
| |
| It is initialized by `yylex_init()' to point to an internal |
| structure. You should never access this value directly. In particular, |
| you should never attempt to free it (use `yylex_destroy()' instead.) |
| |
| |
| File: flex.info, Node: Reentrant Functions, Prev: Reentrant Detail, Up: Reentrant |
| |
| 19.5 Functions and Macros Available in Reentrant C Scanners |
| =========================================================== |
| |
| The following Functions are available in a reentrant scanner: |
| |
| char *yyget_text ( yyscan_t scanner ); |
| int yyget_leng ( yyscan_t scanner ); |
| FILE *yyget_in ( yyscan_t scanner ); |
| FILE *yyget_out ( yyscan_t scanner ); |
| int yyget_lineno ( yyscan_t scanner ); |
| YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner ); |
| int yyget_debug ( yyscan_t scanner ); |
| |
| void yyset_debug ( int flag, yyscan_t scanner ); |
| void yyset_in ( FILE * in_str , yyscan_t scanner ); |
| void yyset_out ( FILE * out_str , yyscan_t scanner ); |
| void yyset_lineno ( int line_number , yyscan_t scanner ); |
| void yyset_extra ( YY_EXTRA_TYPE user_defined , yyscan_t scanner ); |
| |
| There are no "set" functions for yytext and yyleng. This is |
| intentional. |
| |
| The following Macro shortcuts are available in actions in a reentrant |
| scanner: |
| |
| yytext |
| yyleng |
| yyin |
| yyout |
| yylineno |
| yyextra |
| yy_flex_debug |
| |
| In a reentrant C scanner, support for yylineno is always present |
| (i.e., you may access yylineno), but the value is never modified by |
| `flex' unless `%option yylineno' is enabled. This is to allow the user |
| to maintain the line count independently of `flex'. |
| |
| The following functions and macros are made available when `%option |
| bison-bridge' (`--bison-bridge') is specified: |
| |
| YYSTYPE * yyget_lval ( yyscan_t scanner ); |
| void yyset_lval ( YYSTYPE * yylvalp , yyscan_t scanner ); |
| yylval |
| |
| The following functions and macros are made available when `%option |
| bison-locations' (`--bison-locations') is specified: |
| |
| YYLTYPE *yyget_lloc ( yyscan_t scanner ); |
| void yyset_lloc ( YYLTYPE * yyllocp , yyscan_t scanner ); |
| yylloc |
| |
| Support for yylval assumes that `YYSTYPE' is a valid type. Support |
| for yylloc assumes that `YYSLYPE' is a valid type. Typically, these |
| types are generated by `bison', and are included in section 1 of the |
| `flex' input. |
| |
| |
| File: flex.info, Node: Lex and Posix, Next: Memory Management, Prev: Reentrant, Up: Top |
| |
| 20 Incompatibilities with Lex and Posix |
| *************************************** |
| |
| `flex' is a rewrite of the AT&T Unix _lex_ tool (the two |
| implementations do not share any code, though), with some extensions and |
| incompatibilities, both of which are of concern to those who wish to |
| write scanners acceptable to both implementations. `flex' is fully |
| compliant with the POSIX `lex' specification, except that when using |
| `%pointer' (the default), a call to `unput()' destroys the contents of |
| `yytext', which is counter to the POSIX specification. In this section |
| we discuss all of the known areas of incompatibility between `flex', |
| AT&T `lex', and the POSIX specification. `flex''s `-l' option turns on |
| maximum compatibility with the original AT&T `lex' implementation, at |
| the cost of a major loss in the generated scanner's performance. We |
| note below which incompatibilities can be overcome using the `-l' |
| option. `flex' is fully compatible with `lex' with the following |
| exceptions: |
| |
| * The undocumented `lex' scanner internal variable `yylineno' is not |
| supported unless `-l' or `%option yylineno' is used. |
| |
| * `yylineno' should be maintained on a per-buffer basis, rather than |
| a per-scanner (single global variable) basis. |
| |
| * `yylineno' is not part of the POSIX specification. |
| |
| * The `input()' routine is not redefinable, though it may be called |
| to read characters following whatever has been matched by a rule. |
| If `input()' encounters an end-of-file the normal `yywrap()' |
| processing is done. A "real" end-of-file is returned by `input()' |
| as `EOF'. |
| |
| * Input is instead controlled by defining the `YY_INPUT()' macro. |
| |
| * The `flex' restriction that `input()' cannot be redefined is in |
| accordance with the POSIX specification, which simply does not |
| specify any way of controlling the scanner's input other than by |
| making an initial assignment to `yyin'. |
| |
| * The `unput()' routine is not redefinable. This restriction is in |
| accordance with POSIX. |
| |
| * `flex' scanners are not as reentrant as `lex' scanners. In |
| particular, if you have an interactive scanner and an interrupt |
| handler which long-jumps out of the scanner, and the scanner is |
| subsequently called again, you may get the following message: |
| |
| fatal flex scanner internal error--end of buffer missed |
| |
| To reenter the scanner, first use: |
| |
| yyrestart( yyin ); |
| |
| Note that this call will throw away any buffered input; usually |
| this isn't a problem with an interactive scanner. *Note |
| Reentrant::, for `flex''s reentrant API. |
| |
| * Also note that `flex' C++ scanner classes _are_ reentrant, so if |
| using C++ is an option for you, you should use them instead. |
| *Note Cxx::, and *note Reentrant:: for details. |
| |
| * `output()' is not supported. Output from the ECHO macro is done |
| to the file-pointer `yyout' (default `stdout)'. |
| |
| * `output()' is not part of the POSIX specification. |
| |
| * `lex' does not support exclusive start conditions (%x), though they |
| are in the POSIX specification. |
| |
| * When definitions are expanded, `flex' encloses them in parentheses. |
| With `lex', the following: |
| |
| NAME [A-Z][A-Z0-9]* |
| %% |
| foo{NAME}? printf( "Found it\n" ); |
| %% |
| |
| will not match the string `foo' because when the macro is expanded |
| the rule is equivalent to `foo[A-Z][A-Z0-9]*?' and the precedence |
| is such that the `?' is associated with `[A-Z0-9]*'. With `flex', |
| the rule will be expanded to `foo([A-Z][A-Z0-9]*)?' and so the |
| string `foo' will match. |
| |
| * Note that if the definition begins with `^' or ends with `$' then |
| it is _not_ expanded with parentheses, to allow these operators to |
| appear in definitions without losing their special meanings. But |
| the `<s>', `/', and `<<EOF>>' operators cannot be used in a `flex' |
| definition. |
| |
| * Using `-l' results in the `lex' behavior of no parentheses around |
| the definition. |
| |
| * The POSIX specification is that the definition be enclosed in |
| parentheses. |
| |
| * Some implementations of `lex' allow a rule's action to begin on a |
| separate line, if the rule's pattern has trailing whitespace: |
| |
| %% |
| foo|bar<space here> |
| { foobar_action();} |
| |
| `flex' does not support this feature. |
| |
| * The `lex' `%r' (generate a Ratfor scanner) option is not |
| supported. It is not part of the POSIX specification. |
| |
| * After a call to `unput()', _yytext_ is undefined until the next |
| token is matched, unless the scanner was built using `%array'. |
| This is not the case with `lex' or the POSIX specification. The |
| `-l' option does away with this incompatibility. |
| |
| * The precedence of the `{,}' (numeric range) operator is different. |
| The AT&T and POSIX specifications of `lex' interpret `abc{1,3}' as |
| match one, two, or three occurrences of `abc'", whereas `flex' |
| interprets it as "match `ab' followed by one, two, or three |
| occurrences of `c'". The `-l' and `--posix' options do away with |
| this incompatibility. |
| |
| * The precedence of the `^' operator is different. `lex' interprets |
| `^foo|bar' as "match either 'foo' at the beginning of a line, or |
| 'bar' anywhere", whereas `flex' interprets it as "match either |
| `foo' or `bar' if they come at the beginning of a line". The |
| latter is in agreement with the POSIX specification. |
| |
| * The special table-size declarations such as `%a' supported by |
| `lex' are not required by `flex' scanners.. `flex' ignores them. |
| |
| * The name `FLEX_SCANNER' is `#define''d so scanners may be written |
| for use with either `flex' or `lex'. Scanners also include |
| `YY_FLEX_MAJOR_VERSION', `YY_FLEX_MINOR_VERSION' and |
| `YY_FLEX_SUBMINOR_VERSION' indicating which version of `flex' |
| generated the scanner. For example, for the 2.5.22 release, these |
| defines would be 2, 5 and 22 respectively. If the version of |
| `flex' being used is a beta version, then the symbol `FLEX_BETA' |
| is defined. |
| |
| * The symbols `[[' and `]]' in the code sections of the input may |
| conflict with the m4 delimiters. *Note M4 Dependency::. |
| |
| |
| The following `flex' features are not included in `lex' or the POSIX |
| specification: |
| |
| * C++ scanners |
| |
| * %option |
| |
| * start condition scopes |
| |
| * start condition stacks |
| |
| * interactive/non-interactive scanners |
| |
| * yy_scan_string() and friends |
| |
| * yyterminate() |
| |
| * yy_set_interactive() |
| |
| * yy_set_bol() |
| |
| * YY_AT_BOL() <<EOF>> |
| |
| * <*> |
| |
| * YY_DECL |
| |
| * YY_START |
| |
| * YY_USER_ACTION |
| |
| * YY_USER_INIT |
| |
| * #line directives |
| |
| * %{}'s around actions |
| |
| * reentrant C API |
| |
| * multiple actions on a line |
| |
| * almost all of the `flex' command-line options |
| |
| The feature "multiple actions on a line" refers to the fact that |
| with `flex' you can put multiple actions on the same line, separated |
| with semi-colons, while with `lex', the following: |
| |
| foo handle_foo(); ++num_foos_seen; |
| |
| is (rather surprisingly) truncated to |
| |
| foo handle_foo(); |
| |
| `flex' does not truncate the action. Actions that are not enclosed |
| in braces are simply terminated at the end of the line. |
| |
| |
| File: flex.info, Node: Memory Management, Next: Serialized Tables, Prev: Lex and Posix, Up: Top |
| |
| 21 Memory Management |
| ******************** |
| |
| This chapter describes how flex handles dynamic memory, and how you can |
| override the default behavior. |
| |
| * Menu: |
| |
| * The Default Memory Management:: |
| * Overriding The Default Memory Management:: |
| * A Note About yytext And Memory:: |
| |
| |
| File: flex.info, Node: The Default Memory Management, Next: Overriding The Default Memory Management, Prev: Memory Management, Up: Memory Management |
| |
| 21.1 The Default Memory Management |
| ================================== |
| |
| Flex allocates dynamic memory during initialization, and once in a |
| while from within a call to yylex(). Initialization takes place during |
| the first call to yylex(). Thereafter, flex may reallocate more memory |
| if it needs to enlarge a buffer. As of version 2.5.9 Flex will clean up |
| all memory when you call `yylex_destroy' *Note faq-memory-leak::. |
| |
| Flex allocates dynamic memory for four purposes, listed below (1) |
| |
| 16kB for the input buffer. |
| Flex allocates memory for the character buffer used to perform |
| pattern matching. Flex must read ahead from the input stream and |
| store it in a large character buffer. This buffer is typically |
| the largest chunk of dynamic memory flex consumes. This buffer |
| will grow if necessary, doubling the size each time. Flex frees |
| this memory when you call yylex_destroy(). The default size of |
| this buffer (16384 bytes) is almost always too large. The ideal |
| size for this buffer is the length of the longest token expected, |
| in bytes, plus a little more. Flex will allocate a few extra |
| bytes for housekeeping. Currently, to override the size of the |
| input buffer you must `#define YY_BUF_SIZE' to whatever number of |
| bytes you want. We don't plan to change this in the near future, |
| but we reserve the right to do so if we ever add a more robust |
| memory management API. |
| |
| 64kb for the REJECT state. This will only be allocated if you use REJECT. |
| The size is large enough to hold the same number of states as |
| characters in the input buffer. If you override the size of the |
| input buffer (via `YY_BUF_SIZE'), then you automatically override |
| the size of this buffer as well. |
| |
| 100 bytes for the start condition stack. |
| Flex allocates memory for the start condition stack. This is the |
| stack used for pushing start states, i.e., with yy_push_state(). |
| It will grow if necessary. Since the states are simply integers, |
| this stack doesn't consume much memory. This stack is not present |
| if `%option stack' is not specified. You will rarely need to tune |
| this buffer. The ideal size for this stack is the maximum depth |
| expected. The memory for this stack is automatically destroyed |
| when you call yylex_destroy(). *Note option-stack::. |
| |
| 40 bytes for each YY_BUFFER_STATE. |
| Flex allocates memory for each YY_BUFFER_STATE. The buffer state |
| itself is about 40 bytes, plus an additional large character |
| buffer (described above.) The initial buffer state is created |
| during initialization, and with each call to yy_create_buffer(). |
| You can't tune the size of this, but you can tune the character |
| buffer as described above. Any buffer state that you explicitly |
| create by calling yy_create_buffer() is _NOT_ destroyed |
| automatically. You must call yy_delete_buffer() to free the |
| memory. The exception to this rule is that flex will delete the |
| current buffer automatically when you call yylex_destroy(). If you |
| delete the current buffer, be sure to set it to NULL. That way, |
| flex will not try to delete the buffer a second time (possibly |
| crashing your program!) At the time of this writing, flex does not |
| provide a growable stack for the buffer states. You have to |
| manage that yourself. *Note Multiple Input Buffers::. |
| |
| 84 bytes for the reentrant scanner guts |
| Flex allocates about 84 bytes for the reentrant scanner structure |
| when you call yylex_init(). It is destroyed when the user calls |
| yylex_destroy(). |
| |
| |
| ---------- Footnotes ---------- |
| |
| (1) The quantities given here are approximate, and may vary due to |
| host architecture, compiler configuration, or due to future |
| enhancements to flex. |
| |
| |
| File: flex.info, Node: Overriding The Default Memory Management, Next: A Note About yytext And Memory, Prev: The Default Memory Management, Up: Memory Management |
| |
| 21.2 Overriding The Default Memory Management |
| ============================================= |
| |
| Flex calls the functions `yyalloc', `yyrealloc', and `yyfree' when it |
| needs to allocate or free memory. By default, these functions are |
| wrappers around the standard C functions, `malloc', `realloc', and |
| `free', respectively. You can override the default implementations by |
| telling flex that you will provide your own implementations. |
| |
| To override the default implementations, you must do two things: |
| |
| 1. Suppress the default implementations by specifying one or more of |
| the following options: |
| |
| * `%option noyyalloc' |
| |
| * `%option noyyrealloc' |
| |
| * `%option noyyfree'. |
| |
| 2. Provide your own implementation of the following functions: (1) |
| |
| // For a non-reentrant scanner |
| void * yyalloc (size_t bytes); |
| void * yyrealloc (void * ptr, size_t bytes); |
| void yyfree (void * ptr); |
| |
| // For a reentrant scanner |
| void * yyalloc (size_t bytes, void * yyscanner); |
| void * yyrealloc (void * ptr, size_t bytes, void * yyscanner); |
| void yyfree (void * ptr, void * yyscanner); |
| |
| |
| In the following example, we will override all three memory |
| routines. We assume that there is a custom allocator with garbage |
| collection. In order to make this example interesting, we will use a |
| reentrant scanner, passing a pointer to the custom allocator through |
| `yyextra'. |
| |
| %{ |
| #include "some_allocator.h" |
| %} |
| |
| /* Suppress the default implementations. */ |
| %option noyyalloc noyyrealloc noyyfree |
| %option reentrant |
| |
| /* Initialize the allocator. */ |
| #define YY_EXTRA_TYPE struct allocator* |
| #define YY_USER_INIT yyextra = allocator_create(); |
| |
| %% |
| .|\n ; |
| %% |
| |
| /* Provide our own implementations. */ |
| void * yyalloc (size_t bytes, void* yyscanner) { |
| return allocator_alloc (yyextra, bytes); |
| } |
| |
| void * yyrealloc (void * ptr, size_t bytes, void* yyscanner) { |
| return allocator_realloc (yyextra, bytes); |
| } |
| |
| void yyfree (void * ptr, void * yyscanner) { |
| /* Do nothing -- we leave it to the garbage collector. */ |
| } |
| |
| ---------- Footnotes ---------- |
| |
| (1) It is not necessary to override all (or any) of the memory |
| management routines. You may, for example, override `yyrealloc', but |
| not `yyfree' or `yyalloc'. |
| |
| |
| File: flex.info, Node: A Note About yytext And Memory, Prev: Overriding The Default Memory Management, Up: Memory Management |
| |
| 21.3 A Note About yytext And Memory |
| =================================== |
| |
| When flex finds a match, `yytext' points to the first character of the |
| match in the input buffer. The string itself is part of the input |
| buffer, and is _NOT_ allocated separately. The value of yytext will be |
| overwritten the next time yylex() is called. In short, the value of |
| yytext is only valid from within the matched rule's action. |
| |
| Often, you want the value of yytext to persist for later processing, |
| i.e., by a parser with non-zero lookahead. In order to preserve yytext, |
| you will have to copy it with strdup() or a similar function. But this |
| introduces some headache because your parser is now responsible for |
| freeing the copy of yytext. If you use a yacc or bison parser, |
| (commonly used with flex), you will discover that the error recovery |
| mechanisms can cause memory to be leaked. |
| |
| To prevent memory leaks from strdup'd yytext, you will have to track |
| the memory somehow. Our experience has shown that a garbage collection |
| mechanism or a pooled memory mechanism will save you a lot of grief |
| when writing parsers. |
| |
| |
| File: flex.info, Node: Serialized Tables, Next: Diagnostics, Prev: Memory Management, Up: Top |
| |
| 22 Serialized Tables |
| ******************** |
| |
| A `flex' scanner has the ability to save the DFA tables to a file, and |
| load them at runtime when needed. The motivation for this feature is |
| to reduce the runtime memory footprint. Traditionally, these tables |
| have been compiled into the scanner as C arrays, and are sometimes |
| quite large. Since the tables are compiled into the scanner, the |
| memory used by the tables can never be freed. This is a waste of |
| memory, especially if an application uses several scanners, but none of |
| them at the same time. |
| |
| The serialization feature allows the tables to be loaded at runtime, |
| before scanning begins. The tables may be discarded when scanning is |
| finished. |
| |
| * Menu: |
| |
| * Creating Serialized Tables:: |
| * Loading and Unloading Serialized Tables:: |
| * Tables File Format:: |
| |
| |
| File: flex.info, Node: Creating Serialized Tables, Next: Loading and Unloading Serialized Tables, Prev: Serialized Tables, Up: Serialized Tables |
| |
| 22.1 Creating Serialized Tables |
| =============================== |
| |
| You may create a scanner with serialized tables by specifying: |
| |
| %option tables-file=FILE |
| or |
| --tables-file=FILE |
| |
| These options instruct flex to save the DFA tables to the file FILE. |
| The tables will _not_ be embedded in the generated scanner. The scanner |
| will not function on its own. The scanner will be dependent upon the |
| serialized tables. You must load the tables from this file at runtime |
| before you can scan anything. |
| |
| If you do not specify a filename to `--tables-file', the tables will |
| be saved to `lex.yy.tables', where `yy' is the appropriate prefix. |
| |
| If your project uses several different scanners, you can concatenate |
| the serialized tables into one file, and flex will find the correct set |
| of tables, using the scanner prefix as part of the lookup key. An |
| example follows: |
| |
| $ flex --tables-file --prefix=cpp cpp.l |
| $ flex --tables-file --prefix=c c.l |
| $ cat lex.cpp.tables lex.c.tables > all.tables |
| |
| The above example created two scanners, `cpp', and `c'. Since we did |
| not specify a filename, the tables were serialized to `lex.c.tables' and |
| `lex.cpp.tables', respectively. Then, we concatenated the two files |
| together into `all.tables', which we will distribute with our project. |
| At runtime, we will open the file and tell flex to load the tables from |
| it. Flex will find the correct tables automatically. (See next |
| section). |
| |
| |
| File: flex.info, Node: Loading and Unloading Serialized Tables, Next: Tables File Format, Prev: Creating Serialized Tables, Up: Serialized Tables |
| |
| 22.2 Loading and Unloading Serialized Tables |
| ============================================ |
| |
| If you've built your scanner with `%option tables-file', then you must |
| load the scanner tables at runtime. This can be accomplished with the |
| following function: |
| |
| -- Function: int yytables_fload (FILE* FP [, yyscan_t SCANNER]) |
| Locates scanner tables in the stream pointed to by FP and loads |
| them. Memory for the tables is allocated via `yyalloc'. You must |
| call this function before the first call to `yylex'. The argument |
| SCANNER only appears in the reentrant scanner. This function |
| returns `0' (zero) on success, or non-zero on error. |
| |
| The loaded tables are *not* automatically destroyed (unloaded) when |
| you call `yylex_destroy'. The reason is that you may create several |
| scanners of the same type (in a reentrant scanner), each of which needs |
| access to these tables. To avoid a nasty memory leak, you must call |
| the following function: |
| |
| -- Function: int yytables_destroy ([yyscan_t SCANNER]) |
| Unloads the scanner tables. The tables must be loaded again before |
| you can scan any more data. The argument SCANNER only appears in |
| the reentrant scanner. This function returns `0' (zero) on |
| success, or non-zero on error. |
| |
| *The functions `yytables_fload' and `yytables_destroy' are not |
| thread-safe.* You must ensure that these functions are called exactly |
| once (for each scanner type) in a threaded program, before any thread |
| calls `yylex'. After the tables are loaded, they are never written to, |
| and no thread protection is required thereafter - until you destroy |
| them. |
| |
| |
| File: flex.info, Node: Tables File Format, Prev: Loading and Unloading Serialized Tables, Up: Serialized Tables |
| |
| 22.3 Tables File Format |
| ======================= |
| |
| This section defines the file format of serialized `flex' tables. |
| |
| The tables format allows for one or more sets of tables to be |
| specified, where each set corresponds to a given scanner. Scanners are |
| indexed by name, as described below. The file format is as follows: |
| |
| TABLE SET 1 |
| +-------------------------------+ |
| Header | uint32 th_magic; | |
| | uint32 th_hsize; | |
| | uint32 th_ssize; | |
| | uint16 th_flags; | |
| | char th_version[]; | |
| | char th_name[]; | |
| | uint8 th_pad64[]; | |
| +-------------------------------+ |
| Table 1 | uint16 td_id; | |
| | uint16 td_flags; | |
| | uint32 td_hilen; | |
| | uint32 td_lolen; | |
| | void td_data[]; | |
| | uint8 td_pad64[]; | |
| +-------------------------------+ |
| Table 2 | | |
| . . . |
| . . . |
| . . . |
| . . . |
| Table n | | |
| +-------------------------------+ |
| TABLE SET 2 |
| . |
| . |
| . |
| TABLE SET N |
| |
| The above diagram shows that a complete set of tables consists of a |
| header followed by multiple individual tables. Furthermore, multiple |
| complete sets may be present in the same file, each set with its own |
| header and tables. The sets are contiguous in the file. The only way to |
| know if another set follows is to check the next four bytes for the |
| magic number (or check for EOF). The header and tables sections are |
| padded to 64-bit boundaries. Below we describe each field in detail. |
| This format does not specify how the scanner will expand the given |
| data, i.e., data may be serialized as int8, but expanded to an int32 |
| array at runtime. This is to reduce the size of the serialized data |
| where possible. Remember, _all integer values are in network byte |
| order_. |
| |
| Fields of a table header: |
| |
| `th_magic' |
| Magic number, always 0xF13C57B1. |
| |
| `th_hsize' |
| Size of this entire header, in bytes, including all fields plus |
| any padding. |
| |
| `th_ssize' |
| Size of this entire set, in bytes, including the header, all |
| tables, plus any padding. |
| |
| `th_flags' |
| Bit flags for this table set. Currently unused. |
| |
| `th_version[]' |
| Flex version in NULL-terminated string format. e.g., `2.5.13a'. |
| This is the version of flex that was used to create the serialized |
| tables. |
| |
| `th_name[]' |
| Contains the name of this table set. The default is `yytables', |
| and is prefixed accordingly, e.g., `footables'. Must be |
| NULL-terminated. |
| |
| `th_pad64[]' |
| Zero or more NULL bytes, padding the entire header to the next |
| 64-bit boundary as calculated from the beginning of the header. |
| |
| Fields of a table: |
| |
| `td_id' |
| Specifies the table identifier. Possible values are: |
| `YYTD_ID_ACCEPT (0x01)' |
| `yy_accept' |
| |
| `YYTD_ID_BASE (0x02)' |
| `yy_base' |
| |
| `YYTD_ID_CHK (0x03)' |
| `yy_chk' |
| |
| `YYTD_ID_DEF (0x04)' |
| `yy_def' |
| |
| `YYTD_ID_EC (0x05)' |
| `yy_ec ' |
| |
| `YYTD_ID_META (0x06)' |
| `yy_meta' |
| |
| `YYTD_ID_NUL_TRANS (0x07)' |
| `yy_NUL_trans' |
| |
| `YYTD_ID_NXT (0x08)' |
| `yy_nxt'. This array may be two dimensional. See the |
| `td_hilen' field below. |
| |
| `YYTD_ID_RULE_CAN_MATCH_EOL (0x09)' |
| `yy_rule_can_match_eol' |
| |
| `YYTD_ID_START_STATE_LIST (0x0A)' |
| `yy_start_state_list'. This array is handled specially |
| because it is an array of pointers to structs. See the |
| `td_flags' field below. |
| |
| `YYTD_ID_TRANSITION (0x0B)' |
| `yy_transition'. This array is handled specially because it |
| is an array of structs. See the `td_lolen' field below. |
| |
| `YYTD_ID_ACCLIST (0x0C)' |
| `yy_acclist' |
| |
| `td_flags' |
| Bit flags describing how to interpret the data in `td_data'. The |
| data arrays are one-dimensional by default, but may be two |
| dimensional as specified in the `td_hilen' field. |
| |
| `YYTD_DATA8 (0x01)' |
| The data is serialized as an array of type int8. |
| |
| `YYTD_DATA16 (0x02)' |
| The data is serialized as an array of type int16. |
| |
| `YYTD_DATA32 (0x04)' |
| The data is serialized as an array of type int32. |
| |
| `YYTD_PTRANS (0x08)' |
| The data is a list of indexes of entries in the expanded |
| `yy_transition' array. Each index should be expanded to a |
| pointer to the corresponding entry in the `yy_transition' |
| array. We count on the fact that the `yy_transition' array |
| has already been seen. |
| |
| `YYTD_STRUCT (0x10)' |
| The data is a list of yy_trans_info structs, each of which |
| consists of two integers. There is no padding between struct |
| elements or between structs. The type of each member is |
| determined by the `YYTD_DATA*' bits. |
| |
| `td_hilen' |
| If `td_hilen' is non-zero, then the data is a two-dimensional |
| array. Otherwise, the data is a one-dimensional array. `td_hilen' |
| contains the number of elements in the higher dimensional array, |
| and `td_lolen' contains the number of elements in the lowest |
| dimension. |
| |
| Conceptually, `td_data' is either `sometype td_data[td_lolen]', or |
| `sometype td_data[td_hilen][td_lolen]', where `sometype' is |
| specified by the `td_flags' field. It is possible for both |
| `td_lolen' and `td_hilen' to be zero, in which case `td_data' is a |
| zero length array, and no data is loaded, i.e., this table is |
| simply skipped. Flex does not currently generate tables of zero |
| length. |
| |
| `td_lolen' |
| Specifies the number of elements in the lowest dimension array. If |
| this is a one-dimensional array, then it is simply the number of |
| elements in this array. The element size is determined by the |
| `td_flags' field. |
| |
| `td_data[]' |
| The table data. This array may be a one- or two-dimensional array, |
| of type `int8', `int16', `int32', `struct yy_trans_info', or |
| `struct yy_trans_info*', depending upon the values in the |
| `td_flags', `td_hilen', and `td_lolen' fields. |
| |
| `td_pad64[]' |
| Zero or more NULL bytes, padding the entire table to the next |
| 64-bit boundary as calculated from the beginning of this table. |
| |
| |
| File: flex.info, Node: Diagnostics, Next: Limitations, Prev: Serialized Tables, Up: Top |
| |
| 23 Diagnostics |
| ************** |
| |
| The following is a list of `flex' diagnostic messages: |
| |
| * `warning, rule cannot be matched' indicates that the given rule |
| cannot be matched because it follows other rules that will always |
| match the same text as it. For example, in the following `foo' |
| cannot be matched because it comes after an identifier "catch-all" |
| rule: |
| |
| [a-z]+ got_identifier(); |
| foo got_foo(); |
| |
| Using `REJECT' in a scanner suppresses this warning. |
| |
| * `warning, -s option given but default rule can be matched' means |
| that it is possible (perhaps only in a particular start condition) |
| that the default rule (match any single character) is the only one |
| that will match a particular input. Since `-s' was given, |
| presumably this is not intended. |
| |
| * `reject_used_but_not_detected undefined' or |
| `yymore_used_but_not_detected undefined'. These errors can occur |
| at compile time. They indicate that the scanner uses `REJECT' or |
| `yymore()' but that `flex' failed to notice the fact, meaning that |
| `flex' scanned the first two sections looking for occurrences of |
| these actions and failed to find any, but somehow you snuck some in |
| (via a #include file, for example). Use `%option reject' or |
| `%option yymore' to indicate to `flex' that you really do use |
| these features. |
| |
| * `flex scanner jammed'. a scanner compiled with `-s' has |
| encountered an input string which wasn't matched by any of its |
| rules. This error can also occur due to internal problems. |
| |
| * `token too large, exceeds YYLMAX'. your scanner uses `%array' and |
| one of its rules matched a string longer than the `YYLMAX' |
| constant (8K bytes by default). You can increase the value by |
| #define'ing `YYLMAX' in the definitions section of your `flex' |
| input. |
| |
| * `scanner requires -8 flag to use the character 'x''. Your scanner |
| specification includes recognizing the 8-bit character `'x'' and |
| you did not specify the -8 flag, and your scanner defaulted to |
| 7-bit because you used the `-Cf' or `-CF' table compression |
| options. See the discussion of the `-7' flag, *note Scanner |
| Options::, for details. |
| |
| * `flex scanner push-back overflow'. you used `unput()' to push back |
| so much text that the scanner's buffer could not hold both the |
| pushed-back text and the current token in `yytext'. Ideally the |
| scanner should dynamically resize the buffer in this case, but at |
| present it does not. |
| |
| * `input buffer overflow, can't enlarge buffer because scanner uses |
| REJECT'. the scanner was working on matching an extremely large |
| token and needed to expand the input buffer. This doesn't work |
| with scanners that use `REJECT'. |
| |
| * `fatal flex scanner internal error--end of buffer missed'. This can |
| occur in a scanner which is reentered after a long-jump has jumped |
| out (or over) the scanner's activation frame. Before reentering |
| the scanner, use: |
| yyrestart( yyin ); |
| or, as noted above, switch to using the C++ scanner class. |
| |
| * `too many start conditions in <> construct!' you listed more start |
| conditions in a <> construct than exist (so you must have listed at |
| least one of them twice). |
| |
| |
| File: flex.info, Node: Limitations, Next: Bibliography, Prev: Diagnostics, Up: Top |
| |
| 24 Limitations |
| ************** |
| |
| Some trailing context patterns cannot be properly matched and generate |
| warning messages (`dangerous trailing context'). These are patterns |
| where the ending of the first part of the rule matches the beginning of |
| the second part, such as `zx*/xy*', where the 'x*' matches the 'x' at |
| the beginning of the trailing context. (Note that the POSIX draft |
| states that the text matched by such patterns is undefined.) For some |
| trailing context rules, parts which are actually fixed-length are not |
| recognized as such, leading to the abovementioned performance loss. In |
| particular, parts using `|' or `{n}' (such as `foo{3}') are always |
| considered variable-length. Combining trailing context with the |
| special `|' action can result in _fixed_ trailing context being turned |
| into the more expensive _variable_ trailing context. For example, in |
| the following: |
| |
| %% |
| abc | |
| xyz/def |
| |
| Use of `unput()' invalidates yytext and yyleng, unless the `%array' |
| directive or the `-l' option has been used. Pattern-matching of `NUL's |
| is substantially slower than matching other characters. Dynamic |
| resizing of the input buffer is slow, as it entails rescanning all the |
| text matched so far by the current (generally huge) token. Due to both |
| buffering of input and read-ahead, you cannot intermix calls to |
| `<stdio.h>' routines, such as, getchar(), with `flex' rules and expect |
| it to work. Call `input()' instead. The total table entries listed by |
| the `-v' flag excludes the number of table entries needed to determine |
| what rule has been matched. The number of entries is equal to the |
| number of DFA states if the scanner does not use `REJECT', and somewhat |
| greater than the number of states if it does. `REJECT' cannot be used |
| with the `-f' or `-F' options. |
| |
| The `flex' internal algorithms need documentation. |
| |
| |
| File: flex.info, Node: Bibliography, Next: FAQ, Prev: Limitations, Up: Top |
| |
| 25 Additional Reading |
| ********************* |
| |
| You may wish to read more about the following programs: |
| * lex |
| |
| * yacc |
| |
| * sed |
| |
| * awk |
| |
| The following books may contain material of interest: |
| |
| John Levine, Tony Mason, and Doug Brown, _Lex & Yacc_, O'Reilly and |
| Associates. Be sure to get the 2nd edition. |
| |
| M. E. Lesk and E. Schmidt, _LEX - Lexical Analyzer Generator_ |
| |
| Alfred Aho, Ravi Sethi and Jeffrey Ullman, _Compilers: Principles, |
| Techniques and Tools_, Addison-Wesley (1986). Describes the |
| pattern-matching techniques used by `flex' (deterministic finite |
| automata). |
| |
| |
| File: flex.info, Node: FAQ, Next: Appendices, Prev: Bibliography, Up: Top |
| |
| FAQ |
| *** |
| |
| From time to time, the `flex' maintainer receives certain questions. |
| Rather than repeat answers to well-understood problems, we publish them |
| here. |
| |
| * Menu: |
| |
| * When was flex born?:: |
| * How do I expand backslash-escape sequences in C-style quoted strings?:: |
| * Why do flex scanners call fileno if it is not ANSI compatible?:: |
| * Does flex support recursive pattern definitions?:: |
| * How do I skip huge chunks of input (tens of megabytes) while using flex?:: |
| * Flex is not matching my patterns in the same order that I defined them.:: |
| * My actions are executing out of order or sometimes not at all.:: |
| * How can I have multiple input sources feed into the same scanner at the same time?:: |
| * Can I build nested parsers that work with the same input file?:: |
| * How can I match text only at the end of a file?:: |
| * How can I make REJECT cascade across start condition boundaries?:: |
| * Why cant I use fast or full tables with interactive mode?:: |
| * How much faster is -F or -f than -C?:: |
| * If I have a simple grammar cant I just parse it with flex?:: |
| * Why doesn't yyrestart() set the start state back to INITIAL?:: |
| * How can I match C-style comments?:: |
| * The period isn't working the way I expected.:: |
| * Can I get the flex manual in another format?:: |
| * Does there exist a "faster" NDFA->DFA algorithm?:: |
| * How does flex compile the DFA so quickly?:: |
| * How can I use more than 8192 rules?:: |
| * How do I abandon a file in the middle of a scan and switch to a new file?:: |
| * How do I execute code only during initialization (only before the first scan)?:: |
| * How do I execute code at termination?:: |
| * Where else can I find help?:: |
| * Can I include comments in the "rules" section of the file?:: |
| * I get an error about undefined yywrap().:: |
| * How can I change the matching pattern at run time?:: |
| * How can I expand macros in the input?:: |
| * How can I build a two-pass scanner?:: |
| * How do I match any string not matched in the preceding rules?:: |
| * I am trying to port code from AT&T lex that uses yysptr and yysbuf.:: |
| * Is there a way to make flex treat NULL like a regular character?:: |
| * Whenever flex can not match the input it says "flex scanner jammed".:: |
| * Why doesn't flex have non-greedy operators like perl does?:: |
| * Memory leak - 16386 bytes allocated by malloc.:: |
| * How do I track the byte offset for lseek()?:: |
| * How do I use my own I/O classes in a C++ scanner?:: |
| * How do I skip as many chars as possible?:: |
| * deleteme00:: |
| * Are certain equivalent patterns faster than others?:: |
| * Is backing up a big deal?:: |
| * Can I fake multi-byte character support?:: |
| * deleteme01:: |
| * Can you discuss some flex internals?:: |
| * unput() messes up yy_at_bol:: |
| * The | operator is not doing what I want:: |
| * Why can't flex understand this variable trailing context pattern?:: |
| * The ^ operator isn't working:: |
| * Trailing context is getting confused with trailing optional patterns:: |
| * Is flex GNU or not?:: |
| * ERASEME53:: |
| * I need to scan if-then-else blocks and while loops:: |
| * ERASEME55:: |
| * ERASEME56:: |
| * ERASEME57:: |
| * Is there a repository for flex scanners?:: |
| * How can I conditionally compile or preprocess my flex input file?:: |
| * Where can I find grammars for lex and yacc?:: |
| * I get an end-of-buffer message for each character scanned.:: |
| * unnamed-faq-62:: |
| * unnamed-faq-63:: |
| * unnamed-faq-64:: |
| * unnamed-faq-65:: |
| * unnamed-faq-66:: |
| * unnamed-faq-67:: |
| * unnamed-faq-68:: |
| * unnamed-faq-69:: |
| * unnamed-faq-70:: |
| * unnamed-faq-71:: |
| * unnamed-faq-72:: |
| * unnamed-faq-73:: |
| * unnamed-faq-74:: |
| * unnamed-faq-75:: |
| * unnamed-faq-76:: |
| * unnamed-faq-77:: |
| * unnamed-faq-78:: |
| * unnamed-faq-79:: |
| * unnamed-faq-80:: |
| * unnamed-faq-81:: |
| * unnamed-faq-82:: |
| * unnamed-faq-83:: |
| * unnamed-faq-84:: |
| * unnamed-faq-85:: |
| * unnamed-faq-86:: |
| * unnamed-faq-87:: |
| * unnamed-faq-88:: |
| * unnamed-faq-90:: |
| * unnamed-faq-91:: |
| * unnamed-faq-92:: |
| * unnamed-faq-93:: |
| * unnamed-faq-94:: |
| * unnamed-faq-95:: |
| * unnamed-faq-96:: |
| * unnamed-faq-97:: |
| * unnamed-faq-98:: |
| * unnamed-faq-99:: |
| * unnamed-faq-100:: |
| * unnamed-faq-101:: |
| * What is the difference between YYLEX_PARAM and YY_DECL?:: |
| * Why do I get "conflicting types for yylex" error?:: |
| * How do I access the values set in a Flex action from within a Bison action?:: |
| |
| |
| File: flex.info, Node: When was flex born?, Next: How do I expand backslash-escape sequences in C-style quoted strings?, Up: FAQ |
| |
| When was flex born? |
| =================== |
| |
| Vern Paxson took over the `Software Tools' lex project from Jef |
| Poskanzer in 1982. At that point it was written in Ratfor. Around |
| 1987 or so, Paxson translated it into C, and a legend was born :-). |
| |
| |
| File: flex.info, Node: How do I expand backslash-escape sequences in C-style quoted strings?, Next: Why do flex scanners call fileno if it is not ANSI compatible?, Prev: When was flex born?, Up: FAQ |
| |
| How do I expand backslash-escape sequences in C-style quoted strings? |
| ===================================================================== |
| |
| A key point when scanning quoted strings is that you cannot (easily) |
| write a single rule that will precisely match the string if you allow |
| things like embedded escape sequences and newlines. If you try to |
| match strings with a single rule then you'll wind up having to rescan |
| the string anyway to find any escape sequences. |
| |
| Instead you can use exclusive start conditions and a set of rules, |
| one for matching non-escaped text, one for matching a single escape, |
| one for matching an embedded newline, and one for recognizing the end |
| of the string. Each of these rules is then faced with the question of |
| where to put its intermediary results. The best solution is for the |
| rules to append their local value of `yytext' to the end of a "string |
| literal" buffer. A rule like the escape-matcher will append to the |
| buffer the meaning of the escape sequence rather than the literal text |
| in `yytext'. In this way, `yytext' does not need to be modified at all. |
| |
| |
| File: flex.info, Node: Why do flex scanners call fileno if it is not ANSI compatible?, Next: Does flex support recursive pattern definitions?, Prev: How do I expand backslash-escape sequences in C-style quoted strings?, Up: FAQ |
| |
| Why do flex scanners call fileno if it is not ANSI compatible? |
| ============================================================== |
| |
| Flex scanners call `fileno()' in order to get the file descriptor |
| corresponding to `yyin'. The file descriptor may be passed to |
| `isatty()' or `read()', depending upon which `%options' you specified. |
| If your system does not have `fileno()' support, to get rid of the |
| `read()' call, do not specify `%option read'. To get rid of the |
| `isatty()' call, you must specify one of `%option always-interactive' or |
| `%option never-interactive'. |
| |
| |
| File: flex.info, Node: Does flex support recursive pattern definitions?, Next: How do I skip huge chunks of input (tens of megabytes) while using flex?, Prev: Why do flex scanners call fileno if it is not ANSI compatible?, Up: FAQ |
| |
| Does flex support recursive pattern definitions? |
| ================================================ |
| |
| e.g., |
| |
| %% |
| block "{"({block}|{statement})*"}" |
| |
| No. You cannot have recursive definitions. The pattern-matching |
| power of regular expressions in general (and therefore flex scanners, |
| too) is limited. In particular, regular expressions cannot "balance" |
| parentheses to an arbitrary degree. For example, it's impossible to |
| write a regular expression that matches all strings containing the same |
| number of '{'s as '}'s. For more powerful pattern matching, you need a |
| parser, such as `GNU bison'. |
| |
| |
| File: flex.info, Node: How do I skip huge chunks of input (tens of megabytes) while using flex?, Next: Flex is not matching my patterns in the same order that I defined them., Prev: Does flex support recursive pattern definitions?, Up: FAQ |
| |
| How do I skip huge chunks of input (tens of megabytes) while using flex? |
| ======================================================================== |
| |
| Use `fseek()' (or `lseek()') to position yyin, then call `yyrestart()'. |
| |
| |
| File: flex.info, Node: Flex is not matching my patterns in the same order that I defined them., Next: My actions are executing out of order or sometimes not at all., Prev: How do I skip huge chunks of input (tens of megabytes) while using flex?, Up: FAQ |
| |
| Flex is not matching my patterns in the same order that I defined them. |
| ======================================================================= |
| |
| `flex' picks the rule that matches the most text (i.e., the longest |
| possible input string). This is because `flex' uses an entirely |
| different matching technique ("deterministic finite automata") that |
| actually does all of the matching simultaneously, in parallel. (Seems |
| impossible, but it's actually a fairly simple technique once you |
| understand the principles.) |
| |
| A side-effect of this parallel matching is that when the input |
| matches more than one rule, `flex' scanners pick the rule that matched |
| the _most_ text. This is explained further in the manual, in the |
| section *Note Matching::. |
| |
| If you want `flex' to choose a shorter match, then you can work |
| around this behavior by expanding your short rule to match more text, |
| then put back the extra: |
| |
| data_.* yyless( 5 ); BEGIN BLOCKIDSTATE; |
| |
| Another fix would be to make the second rule active only during the |
| `<BLOCKIDSTATE>' start condition, and make that start condition |
| exclusive by declaring it with `%x' instead of `%s'. |
| |
| A final fix is to change the input language so that the ambiguity for |
| `data_' is removed, by adding characters to it that don't match the |
| identifier rule, or by removing characters (such as `_') from the |
| identifier rule so it no longer matches `data_'. (Of course, you might |
| also not have the option of changing the input language.) |
| |
| |
| File: flex.info, Node: My actions are executing out of order or sometimes not at all., Next: How can I have multiple input sources feed into the same scanner at the same time?, Prev: Flex is not matching my patterns in the same order that I defined them., Up: FAQ |
| |
| My actions are executing out of order or sometimes not at all. |
| ============================================================== |
| |
| Most likely, you have (in error) placed the opening `{' of the action |
| block on a different line than the rule, e.g., |
| |
| ^(foo|bar) |
| { <<<--- WRONG! |
| |
| } |
| |
| `flex' requires that the opening `{' of an action associated with a |
| rule begin on the same line as does the rule. You need instead to |
| write your rules as follows: |
| |
| ^(foo|bar) { // CORRECT! |
| |
| } |
| |
| |
| File: flex.info, Node: How can I have multiple input sources feed into the same scanner at the same time?, Next: Can I build nested parsers that work with the same input file?, Prev: My actions are executing out of order or sometimes not at all., Up: FAQ |
| |
| How can I have multiple input sources feed into the same scanner at the same time? |
| ================================================================================== |
| |
| If ... |
| * your scanner is free of backtracking (verified using `flex''s `-b' |
| flag), |
| |
| * AND you run your scanner interactively (`-I' option; default |
| unless using special table compression options), |
| |
| * AND you feed it one character at a time by redefining `YY_INPUT' |
| to do so, |
| |
| then every time it matches a token, it will have exhausted its input |
| buffer (because the scanner is free of backtracking). This means you |
| can safely use `select()' at the point and only call `yylex()' for |
| another token if `select()' indicates there's data available. |
| |
| That is, move the `select()' out from the input function to a point |
| where it determines whether `yylex()' gets called for the next token. |
| |
| With this approach, you will still have problems if your input can |
| arrive piecemeal; `select()' could inform you that the beginning of a |
| token is available, you call `yylex()' to get it, but it winds up |
| blocking waiting for the later characters in the token. |
| |
| Here's another way: Move your input multiplexing inside of |
| `YY_INPUT'. That is, whenever `YY_INPUT' is called, it `select()''s to |
| see where input is available. If input is available for the scanner, |
| it reads and returns the next byte. If input is available from another |
| source, it calls whatever function is responsible for reading from that |
| source. (If no input is available, it blocks until some input is |
| available.) I've used this technique in an interpreter I wrote that |
| both reads keyboard input using a `flex' scanner and IPC traffic from |
| sockets, and it works fine. |
| |
| |
| File: flex.info, Node: Can I build nested parsers that work with the same input file?, Next: How can I match text only at the end of a file?, Prev: How can I have multiple input sources feed into the same scanner at the same time?, Up: FAQ |
| |
| Can I build nested parsers that work with the same input file? |
| ============================================================== |
| |
| This is not going to work without some additional effort. The reason is |
| that `flex' block-buffers the input it reads from `yyin'. This means |
| that the "outermost" `yylex()', when called, will automatically slurp |
| up the first 8K of input available on yyin, and subsequent calls to |
| other `yylex()''s won't see that input. You might be tempted to work |
| around this problem by redefining `YY_INPUT' to only return a small |
| amount of text, but it turns out that that approach is quite difficult. |
| Instead, the best solution is to combine all of your scanners into one |
| large scanner, using a different exclusive start condition for each. |
| |
| |
| File: flex.info, Node: How can I match text only at the end of a file?, Next: How can I make REJECT cascade across start condition boundaries?, Prev: Can I build nested parsers that work with the same input file?, Up: FAQ |
| |
| How can I match text only at the end of a file? |
| =============================================== |
| |
| There is no way to write a rule which is "match this text, but only if |
| it comes at the end of the file". You can fake it, though, if you |
| happen to have a character lying around that you don't allow in your |
| input. Then you redefine `YY_INPUT' to call your own routine which, if |
| it sees an `EOF', returns the magic character first (and remembers to |
| return a real `EOF' next time it's called). Then you could write: |
| |
| <COMMENT>(.|\n)*{EOF_CHAR} /* saw comment at EOF */ |
| |
| |
| File: flex.info, Node: How can I make REJECT cascade across start condition boundaries?, Next: Why cant I use fast or full tables with interactive mode?, Prev: How can I match text only at the end of a file?, Up: FAQ |
| |
| How can I make REJECT cascade across start condition boundaries? |
| ================================================================ |
| |
| You can do this as follows. Suppose you have a start condition `A', and |
| after exhausting all of the possible matches in `<A>', you want to try |
| matches in `<INITIAL>'. Then you could use the following: |
| |
| %x A |
| %% |
| <A>rule_that_is_long ...; REJECT; |
| <A>rule ...; REJECT; /* shorter rule */ |
| <A>etc. |
| ... |
| <A>.|\n { |
| /* Shortest and last rule in <A>, so |
| * cascaded REJECTs will eventually |
| * wind up matching this rule. We want |
| * to now switch to the initial state |
| * and try matching from there instead. |
| */ |
| yyless(0); /* put back matched text */ |
| BEGIN(INITIAL); |
| } |
| |
| |
| File: flex.info, Node: Why cant I use fast or full tables with interactive mode?, Next: How much faster is -F or -f than -C?, Prev: How can I make REJECT cascade across start condition boundaries?, Up: FAQ |
| |
| Why can't I use fast or full tables with interactive mode? |
| ========================================================== |
| |
| One of the assumptions flex makes is that interactive applications are |
| inherently slow (they're waiting on a human after all). It has to do |
| with how the scanner detects that it must be finished scanning a token. |
| For interactive scanners, after scanning each character the current |
| state is looked up in a table (essentially) to see whether there's a |
| chance of another input character possibly extending the length of the |
| match. If not, the scanner halts. For non-interactive scanners, the |
| end-of-token test is much simpler, basically a compare with 0, so no |
| memory bus cycles. Since the test occurs in the innermost scanning |
| loop, one would like to make it go as fast as possible. |
| |
| Still, it seems reasonable to allow the user to choose to trade off |
| a bit of performance in this area to gain the corresponding |
| flexibility. There might be another reason, though, why fast scanners |
| don't support the interactive option. |
| |
| |
| File: flex.info, Node: How much faster is -F or -f than -C?, Next: If I have a simple grammar cant I just parse it with flex?, Prev: Why cant I use fast or full tables with interactive mode?, Up: FAQ |
| |
| How much faster is -F or -f than -C? |
| ==================================== |
| |
| Much faster (factor of 2-3). |
| |
| |
| File: flex.info, Node: If I have a simple grammar cant I just parse it with flex?, Next: Why doesn't yyrestart() set the start state back to INITIAL?, Prev: How much faster is -F or -f than -C?, Up: FAQ |
| |
| If I have a simple grammar can't I just parse it with flex? |
| =========================================================== |
| |
| Is your grammar recursive? That's almost always a sign that you're |
| better off using a parser/scanner rather than just trying to use a |
| scanner alone. |
| |
| |
| File: flex.info, Node: Why doesn't yyrestart() set the start state back to INITIAL?, Next: How can I match C-style comments?, Prev: If I have a simple grammar cant I just parse it with flex?, Up: FAQ |
| |
| Why doesn't yyrestart() set the start state back to INITIAL? |
| ============================================================ |
| |
| There are two reasons. The first is that there might be programs that |
| rely on the start state not changing across file changes. The second |
| is that beginning with `flex' version 2.4, use of `yyrestart()' is no |
| longer required, so fixing the problem there doesn't solve the more |
| general problem. |
| |
| |
| File: flex.info, Node: How can I match C-style comments?, Next: The period isn't working the way I expected., Prev: Why doesn't yyrestart() set the start state back to INITIAL?, Up: FAQ |
| |
| How can I match C-style comments? |
| ================================= |
| |
| You might be tempted to try something like this: |
| |
| "/*".*"*/" // WRONG! |
| |
| or, worse, this: |
| |
| "/*"(.|\n)"*/" // WRONG! |
| |
| The above rules will eat too much input, and blow up on things like: |
| |
| /* a comment */ do_my_thing( "oops */" ); |
| |
| Here is one way which allows you to track line information: |
| |
| <INITIAL>{ |
| "/*" BEGIN(IN_COMMENT); |
| } |
| <IN_COMMENT>{ |
| "*/" BEGIN(INITIAL); |
| [^*\n]+ // eat comment in chunks |
| "*" // eat the lone star |
| \n yylineno++; |
| } |
| |
| |
| File: flex.info, Node: The period isn't working the way I expected., Next: Can I get the flex manual in another format?, Prev: How can I match C-style comments?, Up: FAQ |
| |
| The '.' isn't working the way I expected. |
| ========================================= |
| |
| Here are some tips for using `.': |
| |
| * A common mistake is to place the grouping parenthesis AFTER an |
| operator, when you really meant to place the parenthesis BEFORE |
| the operator, e.g., you probably want this `(foo|bar)+' and NOT |
| this `(foo|bar+)'. |
| |
| The first pattern matches the words `foo' or `bar' any number of |
| times, e.g., it matches the text `barfoofoobarfoo'. The second |
| pattern matches a single instance of `foo' or a single instance of |
| `bar' followed by one or more `r's, e.g., it matches the text |
| `barrrr' . |
| |
| * A `.' inside `[]''s just means a literal`.' (period), and NOT "any |
| character except newline". |
| |
| * Remember that `.' matches any character EXCEPT `\n' (and `EOF'). |
| If you really want to match ANY character, including newlines, |
| then use `(.|\n)' Beware that the regex `(.|\n)+' will match your |
| entire input! |
| |
| * Finally, if you want to match a literal `.' (a period), then use |
| `[.]' or `"."' |
| |
| |
| File: flex.info, Node: Can I get the flex manual in another format?, Next: Does there exist a "faster" NDFA->DFA algorithm?, Prev: The period isn't working the way I expected., Up: FAQ |
| |
| Can I get the flex manual in another format? |
| ============================================ |
| |
| The `flex' source distribution includes a texinfo manual. You are free |
| to convert that texinfo into whatever format you desire. The `texinfo' |
| package includes tools for conversion to a number of formats. |
| |
| |
| File: flex.info, Node: Does there exist a "faster" NDFA->DFA algorithm?, Next: How does flex compile the DFA so quickly?, Prev: Can I get the flex manual in another format?, Up: FAQ |
| |
| Does there exist a "faster" NDFA->DFA algorithm? |
| ================================================ |
| |
| There's no way around the potential exponential running time - it can |
| take you exponential time just to enumerate all of the DFA states. In |
| practice, though, the running time is closer to linear, or sometimes |
| quadratic. |
| |
| |
| File: flex.info, Node: How does flex compile the DFA so quickly?, Next: How can I use more than 8192 rules?, Prev: Does there exist a "faster" NDFA->DFA algorithm?, Up: FAQ |
| |
| How does flex compile the DFA so quickly? |
| ========================================= |
| |
| There are two big speed wins that `flex' uses: |
| |
| 1. It analyzes the input rules to construct equivalence classes for |
| those characters that always make the same transitions. It then |
| rewrites the NFA using equivalence classes for transitions instead |
| of characters. This cuts down the NFA->DFA computation time |
| dramatically, to the point where, for uncompressed DFA tables, the |
| DFA generation is often I/O bound in writing out the tables. |
| |
| 2. It maintains hash values for previously computed DFA states, so |
| testing whether a newly constructed DFA state is equivalent to a |
| previously constructed state can be done very quickly, by first |
| comparing hash values. |
| |
| |
| File: flex.info, Node: How can I use more than 8192 rules?, Next: How do I abandon a file in the middle of a scan and switch to a new file?, Prev: How does flex compile the DFA so quickly?, Up: FAQ |
| |
| How can I use more than 8192 rules? |
| =================================== |
| |
| `Flex' is compiled with an upper limit of 8192 rules per scanner. If |
| you need more than 8192 rules in your scanner, you'll have to recompile |
| `flex' with the following changes in `flexdef.h': |
| |
| < #define YY_TRAILING_MASK 0x2000 |
| < #define YY_TRAILING_HEAD_MASK 0x4000 |
| -- |
| > #define YY_TRAILING_MASK 0x20000000 |
| > #define YY_TRAILING_HEAD_MASK 0x40000000 |
| |
| This should work okay as long as your C compiler uses 32 bit |
| integers. But you might want to think about whether using such a huge |
| number of rules is the best way to solve your problem. |
| |
| The following may also be relevant: |
| |
| With luck, you should be able to increase the definitions in |
| flexdef.h for: |
| |
| #define JAMSTATE -32766 /* marks a reference to the state that always jams */ |
| #define MAXIMUM_MNS 31999 |
| #define BAD_SUBSCRIPT -32767 |
| |
| recompile everything, and it'll all work. Flex only has these |
| 16-bit-like values built into it because a long time ago it was |
| developed on a machine with 16-bit ints. I've given this advice to |
| others in the past but haven't heard back from them whether it worked |
| okay or not... |
| |
| |
| File: flex.info, Node: How do I abandon a file in the middle of a scan and switch to a new file?, Next: How do I execute code only during initialization (only before the first scan)?, Prev: How can I use more than 8192 rules?, Up: FAQ |
| |
| How do I abandon a file in the middle of a scan and switch to a new file? |
| ========================================================================= |
| |
| Just call `yyrestart(newfile)'. Be sure to reset the start state if you |
| want a "fresh start, since `yyrestart' does NOT reset the start state |
| back to `INITIAL'. |
| |
| |
| File: flex.info, Node: How do I execute code only during initialization (only before the first scan)?, Next: How do I execute code at termination?, Prev: How do I abandon a file in the middle of a scan and switch to a new file?, Up: FAQ |
| |
| How do I execute code only during initialization (only before the first scan)? |
| ============================================================================== |
| |
| You can specify an initial action by defining the macro `YY_USER_INIT' |
| (though note that `yyout' may not be available at the time this macro |
| is executed). Or you can add to the beginning of your rules section: |
| |
| %% |
| /* Must be indented! */ |
| static int did_init = 0; |
| |
| if ( ! did_init ){ |
| do_my_init(); |
| did_init = 1; |
| } |
| |
| |
| File: flex.info, Node: How do I execute code at termination?, Next: Where else can I find help?, Prev: How do I execute code only during initialization (only before the first scan)?, Up: FAQ |
| |
| How do I execute code at termination? |
| ===================================== |
| |
| You can specify an action for the `<<EOF>>' rule. |
| |
| |
| File: flex.info, Node: Where else can I find help?, Next: Can I include comments in the "rules" section of the file?, Prev: How do I execute code at termination?, Up: FAQ |
| |
| Where else can I find help? |
| =========================== |
| |
| You can find the flex homepage on the web at |
| `http://flex.sourceforge.net/'. See that page for details about flex |
| mailing lists as well. |
| |
| |
| File: flex.info, Node: Can I include comments in the "rules" section of the file?, Next: I get an error about undefined yywrap()., Prev: Where else can I find help?, Up: FAQ |
| |
| Can I include comments in the "rules" section of the file? |
| ========================================================== |
| |
| Yes, just about anywhere you want to. See the manual for the specific |
| syntax. |
| |
| |
| File: flex.info, Node: I get an error about undefined yywrap()., Next: How can I change the matching pattern at run time?, Prev: Can I include comments in the "rules" section of the file?, Up: FAQ |
| |
| I get an error about undefined yywrap(). |
| ======================================== |
| |
| You must supply a `yywrap()' function of your own, or link to `libfl.a' |
| (which provides one), or use |
| |
| %option noyywrap |
| |
| in your source to say you don't want a `yywrap()' function. |
| |
| |
| File: flex.info, Node: How can I change the matching pattern at run time?, Next: How can I expand macros in the input?, Prev: I get an error about undefined yywrap()., Up: FAQ |
| |
| How can I change the matching pattern at run time? |
| ================================================== |
| |
| You can't, it's compiled into a static table when flex builds the |
| scanner. |
| |
| |
| File: flex.info, Node: How can I expand macros in the input?, Next: How can I build a two-pass scanner?, Prev: How can I change the matching pattern at run time?, Up: FAQ |
| |
| How can I expand macros in the input? |
| ===================================== |
| |
| The best way to approach this problem is at a higher level, e.g., in |
| the parser. |
| |
| However, you can do this using multiple input buffers. |
| |
| %% |
| macro/[a-z]+ { |
| /* Saw the macro "macro" followed by extra stuff. */ |
| main_buffer = YY_CURRENT_BUFFER; |
| expansion_buffer = yy_scan_string(expand(yytext)); |
| yy_switch_to_buffer(expansion_buffer); |
| } |
| |
| <<EOF>> { |
| if ( expansion_buffer ) |
| { |
| // We were doing an expansion, return to where |
| // we were. |
| yy_switch_to_buffer(main_buffer); |
| yy_delete_buffer(expansion_buffer); |
| expansion_buffer = 0; |
| } |
| else |
| yyterminate(); |
| } |
| |
| You probably will want a stack of expansion buffers to allow nested |
| macros. From the above though hopefully the idea is clear. |
| |
| |
| File: flex.info, Node: How can I build a two-pass scanner?, Next: How do I match any string not matched in the preceding rules?, Prev: How can I expand macros in the input?, Up: FAQ |
| |
| How can I build a two-pass scanner? |
| =================================== |
| |
| One way to do it is to filter the first pass to a temporary file, then |
| process the temporary file on the second pass. You will probably see a |
| performance hit, due to all the disk I/O. |
| |
| When you need to look ahead far forward like this, it almost always |
| means that the right solution is to build a parse tree of the entire |
| input, then walk it after the parse in order to generate the output. |
| In a sense, this is a two-pass approach, once through the text and once |
| through the parse tree, but the performance hit for the latter is |
| usually an order of magnitude smaller, since everything is already |
| classified, in binary format, and residing in memory. |
| |
| |
| File: flex.info, Node: How do I match any string not matched in the preceding rules?, Next: I am trying to port code from AT&T lex that uses yysptr and yysbuf., Prev: How can I build a two-pass scanner?, Up: FAQ |
| |
| How do I match any string not matched in the preceding rules? |
| ============================================================= |
| |
| One way to assign precedence, is to place the more specific rules |
| first. If two rules would match the same input (same sequence of |
| characters) then the first rule listed in the `flex' input wins, e.g., |
| |
| %% |
| foo[a-zA-Z_]+ return FOO_ID; |
| bar[a-zA-Z_]+ return BAR_ID; |
| [a-zA-Z_]+ return GENERIC_ID; |
| |
| Note that the rule `[a-zA-Z_]+' must come *after* the others. It |
| will match the same amount of text as the more specific rules, and in |
| that case the `flex' scanner will pick the first rule listed in your |
| scanner as the one to match. |
| |
| |
| File: flex.info, Node: I am trying to port code from AT&T lex that uses yysptr and yysbuf., Next: Is there a way to make flex treat NULL like a regular character?, Prev: How do I match any string not matched in the preceding rules?, Up: FAQ |
| |
| I am trying to port code from AT&T lex that uses yysptr and yysbuf. |
| =================================================================== |
| |
| Those are internal variables pointing into the AT&T scanner's input |
| buffer. I imagine they're being manipulated in user versions of the |
| `input()' and `unput()' functions. If so, what you need to do is |
| analyze those functions to figure out what they're doing, and then |
| replace `input()' with an appropriate definition of `YY_INPUT'. You |
| shouldn't need to (and must not) replace `flex''s `unput()' function. |
| |
| |
| File: flex.info, Node: Is there a way to make flex treat NULL like a regular character?, Next: Whenever flex can not match the input it says "flex scanner jammed"., Prev: I am trying to port code from AT&T lex that uses yysptr and yysbuf., Up: FAQ |
| |
| Is there a way to make flex treat NULL like a regular character? |
| ================================================================ |
| |
| Yes, `\0' and `\x00' should both do the trick. Perhaps you have an |
| ancient version of `flex'. The latest release is version 2.5.37. |
| |
| |
| File: flex.info, Node: Whenever flex can not match the input it says "flex scanner jammed"., Next: Why doesn't flex have non-greedy operators like perl does?, Prev: Is there a way to make flex treat NULL like a regular character?, Up: FAQ |
| |
| Whenever flex can not match the input it says "flex scanner jammed". |
| ==================================================================== |
| |
| You need to add a rule that matches the otherwise-unmatched text, e.g., |
| |
| %option yylineno |
| %% |
| [[a bunch of rules here]] |
| |
| . printf("bad input character '%s' at line %d\n", yytext, yylineno); |
| |
| See `%option default' for more information. |
| |
| |
| File: flex.info, Node: Why doesn't flex have non-greedy operators like perl does?, Next: Memory leak - 16386 bytes allocated by malloc., Prev: Whenever flex can not match the input it says "flex scanner jammed"., Up: FAQ |
| |
| Why doesn't flex have non-greedy operators like perl does? |
| ========================================================== |
| |
| A DFA can do a non-greedy match by stopping the first time it enters an |
| accepting state, instead of consuming input until it determines that no |
| further matching is possible (a "jam" state). This is actually easier |
| to implement than longest leftmost match (which flex does). |
| |
| But it's also much less useful than longest leftmost match. In |
| general, when you find yourself wishing for non-greedy matching, that's |
| usually a sign that you're trying to make the scanner do some parsing. |
| That's generally the wrong approach, since it lacks the power to do a |
| decent job. Better is to either introduce a separate parser, or to |
| split the scanner into multiple scanners using (exclusive) start |
| conditions. |
| |
| You might have a separate start state once you've seen the `BEGIN'. |
| In that state, you might then have a regex that will match `END' (to |
| kick you out of the state), and perhaps `(.|\n)' to get a single |
| character within the chunk ... |
| |
| This approach also has much better error-reporting properties. |
| |
| |
| File: flex.info, Node: Memory leak - 16386 bytes allocated by malloc., Next: How do I track the byte offset for lseek()?, Prev: Why doesn't flex have non-greedy operators like perl does?, Up: FAQ |
| |
| Memory leak - 16386 bytes allocated by malloc. |
| ============================================== |
| |
| UPDATED 2002-07-10: As of `flex' version 2.5.9, this leak means that |
| you did not call `yylex_destroy()'. If you are using an earlier version |
| of `flex', then read on. |
| |
| The leak is about 16426 bytes. That is, (8192 * 2 + 2) for the |
| read-buffer, and about 40 for `struct yy_buffer_state' (depending upon |
| alignment). The leak is in the non-reentrant C scanner only (NOT in the |
| reentrant scanner, NOT in the C++ scanner). Since `flex' doesn't know |
| when you are done, the buffer is never freed. |
| |
| However, the leak won't multiply since the buffer is reused no |
| matter how many times you call `yylex()'. |
| |
| If you want to reclaim the memory when you are completely done |
| scanning, then you might try this: |
| |
| /* For non-reentrant C scanner only. */ |
| yy_delete_buffer(YY_CURRENT_BUFFER); |
| yy_init = 1; |
| |
| Note: `yy_init' is an "internal variable", and hasn't been tested in |
| this situation. It is possible that some other globals may need |
| resetting as well. |
| |
| |
| File: flex.info, Node: How do I track the byte offset for lseek()?, Next: How do I use my own I/O classes in a C++ scanner?, Prev: Memory leak - 16386 bytes allocated by malloc., Up: FAQ |
| |
| How do I track the byte offset for lseek()? |
| =========================================== |
| |
| > We thought that it would be possible to have this number through the |
| > evaluation of the following expression: |
| > |
| > seek_position = (no_buffers)*YY_READ_BUF_SIZE + yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf |
| |
| While this is the right idea, it has two problems. The first is that |
| it's possible that `flex' will request less than `YY_READ_BUF_SIZE' |
| during an invocation of `YY_INPUT' (or that your input source will |
| return less even though `YY_READ_BUF_SIZE' bytes were requested). The |
| second problem is that when refilling its internal buffer, `flex' keeps |
| some characters from the previous buffer (because usually it's in the |
| middle of a match, and needs those characters to construct `yytext' for |
| the match once it's done). Because of this, `yy_c_buf_p - |
| YY_CURRENT_BUFFER->yy_ch_buf' won't be exactly the number of characters |
| already read from the current buffer. |
| |
| An alternative solution is to count the number of characters you've |
| matched since starting to scan. This can be done by using |
| `YY_USER_ACTION'. For example, |
| |
| #define YY_USER_ACTION num_chars += yyleng; |
| |
| (You need to be careful to update your bookkeeping if you use |
| `yymore('), `yyless()', `unput()', or `input()'.) |
| |
| |
| File: flex.info, Node: How do I use my own I/O classes in a C++ scanner?, Next: How do I skip as many chars as possible?, Prev: How do I track the byte offset for lseek()?, Up: FAQ |
| |
| How do I use my own I/O classes in a C++ scanner? |
| ================================================= |
| |
| When the flex C++ scanning class rewrite finally happens, then this |
| sort of thing should become much easier. |
| |
| You can do this by passing the various functions (such as |
| `LexerInput()' and `LexerOutput()') NULL `iostream*''s, and then |
| dealing with your own I/O classes surreptitiously (i.e., stashing them |
| in special member variables). This works because the only assumption |
| about the lexer regarding what's done with the iostream's is that |
| they're ultimately passed to `LexerInput()' and `LexerOutput', which |
| then do whatever is necessary with them. |
| |
| |
| File: flex.info, Node: How do I skip as many chars as possible?, Next: deleteme00, Prev: How do I use my own I/O classes in a C++ scanner?, Up: FAQ |
| |
| How do I skip as many chars as possible? |
| ======================================== |
| |
| How do I skip as many chars as possible - without interfering with the |
| other patterns? |
| |
| In the example below, we want to skip over characters until we see |
| the phrase "endskip". The following will _NOT_ work correctly (do you |
| see why not?) |
| |
| /* INCORRECT SCANNER */ |
| %x SKIP |
| %% |
| <INITIAL>startskip BEGIN(SKIP); |
| ... |
| <SKIP>"endskip" BEGIN(INITIAL); |
| <SKIP>.* ; |
| |
| The problem is that the pattern .* will eat up the word "endskip." |
| The simplest (but slow) fix is: |
| |
| <SKIP>"endskip" BEGIN(INITIAL); |
| <SKIP>. ; |
| |
| The fix involves making the second rule match more, without making |
| it match "endskip" plus something else. So for example: |
| |
| <SKIP>"endskip" BEGIN(INITIAL); |
| <SKIP>[^e]+ ; |
| <SKIP>. ;/* so you eat up e's, too */ |
| |
| |
| File: flex.info, Node: deleteme00, Next: Are certain equivalent patterns faster than others?, Prev: How do I skip as many chars as possible?, Up: FAQ |
| |
| deleteme00 |
| ========== |
| |
| QUESTION: |
| When was flex born? |
| |
| Vern Paxson took over |
| the Software Tools lex project from Jef Poskanzer in 1982. At that point it |
| was written in Ratfor. Around 1987 or so, Paxson translated it into C, and |
| a legend was born :-). |
| |
| |
| File: flex.info, Node: Are certain equivalent patterns faster than others?, Next: Is backing up a big deal?, Prev: deleteme00, Up: FAQ |
| |
| Are certain equivalent patterns faster than others? |
| =================================================== |
| |
| To: Adoram Rogel <adoram@orna.hybridge.com> |
| Subject: Re: Flex 2.5.2 performance questions |
| In-reply-to: Your message of Wed, 18 Sep 96 11:12:17 EDT. |
| Date: Wed, 18 Sep 96 10:51:02 PDT |
| From: Vern Paxson <vern> |
| |
| [Note, the most recent flex release is 2.5.4, which you can get from |
| ftp.ee.lbl.gov. It has bug fixes over 2.5.2 and 2.5.3.] |
| |
| > 1. Using the pattern |
| > ([Ff](oot)?)?[Nn](ote)?(\.)? |
| > instead of |
| > (((F|f)oot(N|n)ote)|((N|n)ote)|((N|n)\.)|((F|f)(N|n)(\.))) |
| > (in a very complicated flex program) caused the program to slow from |
| > 300K+/min to 100K/min (no other changes were done). |
| |
| These two are not equivalent. For example, the first can match "footnote." |
| but the second can only match "footnote". This is almost certainly the |
| cause in the discrepancy - the slower scanner run is matching more tokens, |
| and/or having to do more backing up. |
| |
| > 2. Which of these two are better: [Ff]oot or (F|f)oot ? |
| |
| From a performance point of view, they're equivalent (modulo presumably |
| minor effects such as memory cache hit rates; and the presence of trailing |
| context, see below). From a space point of view, the first is slightly |
| preferable. |
| |
| > 3. I have a pattern that look like this: |
| > pats {p1}|{p2}|{p3}|...|{p50} (50 patterns ORd) |
| > |
| > running yet another complicated program that includes the following rule: |
| > <snext>{and}/{no4}{bb}{pats} |
| > |
| > gets me to "too complicated - over 32,000 states"... |
| |
| I can't tell from this example whether the trailing context is variable-length |
| or fixed-length (it could be the latter if {and} is fixed-length). If it's |
| variable length, which flex -p will tell you, then this reflects a basic |
| performance problem, and if you can eliminate it by restructuring your |
| scanner, you will see significant improvement. |
| |
| > so I divided {pats} to {pats1}, {pats2},..., {pats5} each consists of about |
| > 10 patterns and changed the rule to be 5 rules. |
| > This did compile, but what is the rule of thumb here ? |
| |
| The rule is to avoid trailing context other than fixed-length, in which for |
| a/b, either the 'a' pattern or the 'b' pattern have a fixed length. Use |
| of the '|' operator automatically makes the pattern variable length, so in |
| this case '[Ff]oot' is preferred to '(F|f)oot'. |
| |
| > 4. I changed a rule that looked like this: |
| > <snext8>{and}{bb}/{ROMAN}[^A-Za-z] { BEGIN... |
| > |
| > to the next 2 rules: |
| > <snext8>{and}{bb}/{ROMAN}[A-Za-z] { ECHO;} |
| > <snext8>{and}{bb}/{ROMAN} { BEGIN... |
| > |
| > Again, I understand the using [^...] will cause a great performance loss |
| |
| Actually, it doesn't cause any sort of performance loss. It's a surprising |
| fact about regular expressions that they always match in linear time |
| regardless of how complex they are. |
| |
| > but are there any specific rules about it ? |
| |
| See the "Performance Considerations" section of the man page, and also |
| the example in MISC/fastwc/. |
| |
| Vern |
| |
| |
| File: flex.info, Node: Is backing up a big deal?, Next: Can I fake multi-byte character support?, Prev: Are certain equivalent patterns faster than others?, Up: FAQ |
| |
| Is backing up a big deal? |
| ========================= |
| |
| To: Adoram Rogel <adoram@hybridge.com> |
| Subject: Re: Flex 2.5.2 performance questions |
| In-reply-to: Your message of Thu, 19 Sep 96 10:16:04 EDT. |
| Date: Thu, 19 Sep 96 09:58:00 PDT |
| From: Vern Paxson <vern> |
| |
| > a lot about the backing up problem. |
| > I believe that there lies my biggest problem, and I'll try to improve |
| > it. |
| |
| Since you have variable trailing context, this is a bigger performance |
| problem. Fixing it is usually easier than fixing backing up, which in a |
| complicated scanner (yours seems to fit the bill) can be extremely |
| difficult to do correctly. |
| |
| You also don't mention what flags you are using for your scanner. |
| -f makes a large speed difference, and -Cfe buys you nearly as much |
| speed but the resulting scanner is considerably smaller. |
| |
| > I have an | operator in {and} and in {pats} so both of them are variable |
| > length. |
| |
| -p should have reported this. |
| |
| > Is changing one of them to fixed-length is enough ? |
| |
| Yes. |
| |
| > Is it possible to change the 32,000 states limit ? |
| |
| Yes. I've appended instructions on how. Before you make this change, |
| though, you should think about whether there are ways to fundamentally |
| simplify your scanner - those are certainly preferable! |
| |
| Vern |
| |
| To increase the 32K limit (on a machine with 32 bit integers), you increase |
| the magnitude of the following in flexdef.h: |
| |
| #define JAMSTATE -32766 /* marks a reference to the state that always jams */ |
| #define MAXIMUM_MNS 31999 |
| #define BAD_SUBSCRIPT -32767 |
| #define MAX_SHORT 32700 |
| |
| Adding a 0 or two after each should do the trick. |
| |
| |
| File: flex.info, Node: Can I fake multi-byte character support?, Next: deleteme01, Prev: Is backing up a big deal?, Up: FAQ |
| |
| Can I fake multi-byte character support? |
| ======================================== |
| |
| To: Heeman_Lee@hp.com |
| Subject: Re: flex - multi-byte support? |
| In-reply-to: Your message of Thu, 03 Oct 1996 17:24:04 PDT. |
| Date: Fri, 04 Oct 1996 11:42:18 PDT |
| From: Vern Paxson <vern> |
| |
| > I assume as long as my *.l file defines the |
| > range of expected character code values (in octal format), flex will |
| > scan the file and read multi-byte characters correctly. But I have no |
| > confidence in this assumption. |
| |
| Your lack of confidence is justified - this won't work. |
| |
| Flex has in it a widespread assumption that the input is processed |
| one byte at a time. Fixing this is on the to-do list, but is involved, |
| so it won't happen any time soon. In the interim, the best I can suggest |
| (unless you want to try fixing it yourself) is to write your rules in |
| terms of pairs of bytes, using definitions in the first section: |
| |
| X \xfe\xc2 |
| ... |
| %% |
| foo{X}bar found_foo_fe_c2_bar(); |
| |
| etc. Definitely a pain - sorry about that. |
| |
| By the way, the email address you used for me is ancient, indicating you |
| have a very old version of flex. You can get the most recent, 2.5.4, from |
| ftp.ee.lbl.gov. |
| |
| Vern |
| |
| |
| File: flex.info, Node: deleteme01, Next: Can you discuss some flex internals?, Prev: Can I fake multi-byte character support?, Up: FAQ |
| |
| deleteme01 |
| ========== |
| |
| To: moleary@primus.com |
| Subject: Re: Flex / Unicode compatibility question |
| In-reply-to: Your message of Tue, 22 Oct 1996 10:15:42 PDT. |
| Date: Tue, 22 Oct 1996 11:06:13 PDT |
| From: Vern Paxson <vern> |
| |
| Unfortunately flex at the moment has a widespread assumption within it |
| that characters are processed 8 bits at a time. I don't see any easy |
| fix for this (other than writing your rules in terms of double characters - |
| a pain). I also don't know of a wider lex, though you might try surfing |
| the Plan 9 stuff because I know it's a Unicode system, and also the PCCT |
| toolkit (try searching say Alta Vista for "Purdue Compiler Construction |
| Toolkit"). |
| |
| Fixing flex to handle wider characters is on the long-term to-do list. |
| But since flex is a strictly spare-time project these days, this probably |
| won't happen for quite a while, unless someone else does it first. |
| |
| Vern |
| |
| |
| File: flex.info, Node: Can you discuss some flex internals?, Next: unput() messes up yy_at_bol, Prev: deleteme01, Up: FAQ |
| |
| Can you discuss some flex internals? |
| ==================================== |
| |
| To: Johan Linde <jl@theophys.kth.se> |
| Subject: Re: translation of flex |
| In-reply-to: Your message of Sun, 10 Nov 1996 09:16:36 PST. |
| Date: Mon, 11 Nov 1996 10:33:50 PST |
| From: Vern Paxson <vern> |
| |
| > I'm working for the Swedish team translating GNU program, and I'm currently |
| > working with flex. I have a few questions about some of the messages which |
| > I hope you can answer. |
| |
| All of the things you're wondering about, by the way, concerning flex |
| internals - probably the only person who understands what they mean in |
| English is me! So I wouldn't worry too much about getting them right. |
| That said ... |
| |
| > #: main.c:545 |
| > msgid " %d protos created\n" |
| > |
| > Does proto mean prototype? |
| |
| Yes - prototypes of state compression tables. |
| |
| > #: main.c:539 |
| > msgid " %d/%d (peak %d) template nxt-chk entries created\n" |
| > |
| > Here I'm mainly puzzled by 'nxt-chk'. I guess it means 'next-check'. (?) |
| > However, 'template next-check entries' doesn't make much sense to me. To be |
| > able to find a good translation I need to know a little bit more about it. |
| |
| There is a scheme in the Aho/Sethi/Ullman compiler book for compressing |
| scanner tables. It involves creating two pairs of tables. The first has |
| "base" and "default" entries, the second has "next" and "check" entries. |
| The "base" entry is indexed by the current state and yields an index into |
| the next/check table. The "default" entry gives what to do if the state |
| transition isn't found in next/check. The "next" entry gives the next |
| state to enter, but only if the "check" entry verifies that this entry is |
| correct for the current state. Flex creates templates of series of |
| next/check entries and then encodes differences from these templates as a |
| way to compress the tables. |
| |
| > #: main.c:533 |
| > msgid " %d/%d base-def entries created\n" |
| > |
| > The same problem here for 'base-def'. |
| |
| See above. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unput() messes up yy_at_bol, Next: The | operator is not doing what I want, Prev: Can you discuss some flex internals?, Up: FAQ |
| |
| unput() messes up yy_at_bol |
| =========================== |
| |
| To: Xinying Li <xli@npac.syr.edu> |
| Subject: Re: FLEX ? |
| In-reply-to: Your message of Wed, 13 Nov 1996 17:28:38 PST. |
| Date: Wed, 13 Nov 1996 19:51:54 PST |
| From: Vern Paxson <vern> |
| |
| > "unput()" them to input flow, question occurs. If I do this after I scan |
| > a carriage, the variable "YY_CURRENT_BUFFER->yy_at_bol" is changed. That |
| > means the carriage flag has gone. |
| |
| You can control this by calling yy_set_bol(). It's described in the manual. |
| |
| > And if in pre-reading it goes to the end of file, is anything done |
| > to control the end of curren buffer and end of file? |
| |
| No, there's no way to put back an end-of-file. |
| |
| > By the way I am using flex 2.5.2 and using the "-l". |
| |
| The latest release is 2.5.4, by the way. It fixes some bugs in 2.5.2 and |
| 2.5.3. You can get it from ftp.ee.lbl.gov. |
| |
| Vern |
| |
| |
| File: flex.info, Node: The | operator is not doing what I want, Next: Why can't flex understand this variable trailing context pattern?, Prev: unput() messes up yy_at_bol, Up: FAQ |
| |
| The | operator is not doing what I want |
| ======================================= |
| |
| To: Alain.ISSARD@st.com |
| Subject: Re: Start condition with FLEX |
| In-reply-to: Your message of Mon, 18 Nov 1996 09:45:02 PST. |
| Date: Mon, 18 Nov 1996 10:41:34 PST |
| From: Vern Paxson <vern> |
| |
| > I am not able to use the start condition scope and to use the | (OR) with |
| > rules having start conditions. |
| |
| The problem is that if you use '|' as a regular expression operator, for |
| example "a|b" meaning "match either 'a' or 'b'", then it must *not* have |
| any blanks around it. If you instead want the special '|' *action* (which |
| from your scanner appears to be the case), which is a way of giving two |
| different rules the same action: |
| |
| foo | |
| bar matched_foo_or_bar(); |
| |
| then '|' *must* be separated from the first rule by whitespace and *must* |
| be followed by a new line. You *cannot* write it as: |
| |
| foo | bar matched_foo_or_bar(); |
| |
| even though you might think you could because yacc supports this syntax. |
| The reason for this unfortunately incompatibility is historical, but it's |
| unlikely to be changed. |
| |
| Your problems with start condition scope are simply due to syntax errors |
| from your use of '|' later confusing flex. |
| |
| Let me know if you still have problems. |
| |
| Vern |
| |
| |
| File: flex.info, Node: Why can't flex understand this variable trailing context pattern?, Next: The ^ operator isn't working, Prev: The | operator is not doing what I want, Up: FAQ |
| |
| Why can't flex understand this variable trailing context pattern? |
| ================================================================= |
| |
| To: Gregory Margo <gmargo@newton.vip.best.com> |
| Subject: Re: flex-2.5.3 bug report |
| In-reply-to: Your message of Sat, 23 Nov 1996 16:50:09 PST. |
| Date: Sat, 23 Nov 1996 17:07:32 PST |
| From: Vern Paxson <vern> |
| |
| > Enclosed is a lex file that "real" lex will process, but I cannot get |
| > flex to process it. Could you try it and maybe point me in the right direction? |
| |
| Your problem is that some of the definitions in the scanner use the '/' |
| trailing context operator, and have it enclosed in ()'s. Flex does not |
| allow this operator to be enclosed in ()'s because doing so allows undefined |
| regular expressions such as "(a/b)+". So the solution is to remove the |
| parentheses. Note that you must also be building the scanner with the -l |
| option for AT&T lex compatibility. Without this option, flex automatically |
| encloses the definitions in parentheses. |
| |
| Vern |
| |
| |
| File: flex.info, Node: The ^ operator isn't working, Next: Trailing context is getting confused with trailing optional patterns, Prev: Why can't flex understand this variable trailing context pattern?, Up: FAQ |
| |
| The ^ operator isn't working |
| ============================ |
| |
| To: Thomas Hadig <hadig@toots.physik.rwth-aachen.de> |
| Subject: Re: Flex Bug ? |
| In-reply-to: Your message of Tue, 26 Nov 1996 14:35:01 PST. |
| Date: Tue, 26 Nov 1996 11:15:05 PST |
| From: Vern Paxson <vern> |
| |
| > In my lexer code, i have the line : |
| > ^\*.* { } |
| > |
| > Thus all lines starting with an astrix (*) are comment lines. |
| > This does not work ! |
| |
| I can't get this problem to reproduce - it works fine for me. Note |
| though that if what you have is slightly different: |
| |
| COMMENT ^\*.* |
| %% |
| {COMMENT} { } |
| |
| then it won't work, because flex pushes back macro definitions enclosed |
| in ()'s, so the rule becomes |
| |
| (^\*.*) { } |
| |
| and now that the '^' operator is not at the immediate beginning of the |
| line, it's interpreted as just a regular character. You can avoid this |
| behavior by using the "-l" lex-compatibility flag, or "%option lex-compat". |
| |
| Vern |
| |
| |
| File: flex.info, Node: Trailing context is getting confused with trailing optional patterns, Next: Is flex GNU or not?, Prev: The ^ operator isn't working, Up: FAQ |
| |
| Trailing context is getting confused with trailing optional patterns |
| ==================================================================== |
| |
| To: Adoram Rogel <adoram@hybridge.com> |
| Subject: Re: Flex 2.5.4 BOF ??? |
| In-reply-to: Your message of Tue, 26 Nov 1996 16:10:41 PST. |
| Date: Wed, 27 Nov 1996 10:56:25 PST |
| From: Vern Paxson <vern> |
| |
| > Organization(s)?/[a-z] |
| > |
| > This matched "Organizations" (looking in debug mode, the trailing s |
| > was matched with trailing context instead of the optional (s) in the |
| > end of the word. |
| |
| That should only happen with lex. Flex can properly match this pattern. |
| (That might be what you're saying, I'm just not sure.) |
| |
| > Is there a way to avoid this dangerous trailing context problem ? |
| |
| Unfortunately, there's no easy way. On the other hand, I don't see why |
| it should be a problem. Lex's matching is clearly wrong, and I'd hope |
| that usually the intent remains the same as expressed with the pattern, |
| so flex's matching will be correct. |
| |
| Vern |
| |
| |
| File: flex.info, Node: Is flex GNU or not?, Next: ERASEME53, Prev: Trailing context is getting confused with trailing optional patterns, Up: FAQ |
| |
| Is flex GNU or not? |
| =================== |
| |
| To: Cameron MacKinnon <mackin@interlog.com> |
| Subject: Re: Flex documentation bug |
| In-reply-to: Your message of Mon, 02 Dec 1996 00:07:08 PST. |
| Date: Sun, 01 Dec 1996 22:29:39 PST |
| From: Vern Paxson <vern> |
| |
| > I'm not sure how or where to submit bug reports (documentation or |
| > otherwise) for the GNU project stuff ... |
| |
| Well, strictly speaking flex isn't part of the GNU project. They just |
| distribute it because no one's written a decent GPL'd lex replacement. |
| So you should send bugs directly to me. Those sent to the GNU folks |
| sometimes find there way to me, but some may drop between the cracks. |
| |
| > In GNU Info, under the section 'Start Conditions', and also in the man |
| > page (mine's dated April '95) is a nice little snippet showing how to |
| > parse C quoted strings into a buffer, defined to be MAX_STR_CONST in |
| > size. Unfortunately, no overflow checking is ever done ... |
| |
| This is already mentioned in the manual: |
| |
| Finally, here's an example of how to match C-style quoted |
| strings using exclusive start conditions, including expanded |
| escape sequences (but not including checking for a string |
| that's too long): |
| |
| The reason for not doing the overflow checking is that it will needlessly |
| clutter up an example whose main purpose is just to demonstrate how to |
| use flex. |
| |
| The latest release is 2.5.4, by the way, available from ftp.ee.lbl.gov. |
| |
| Vern |
| |
| |
| File: flex.info, Node: ERASEME53, Next: I need to scan if-then-else blocks and while loops, Prev: Is flex GNU or not?, Up: FAQ |
| |
| ERASEME53 |
| ========= |
| |
| To: tsv@cs.UManitoba.CA |
| Subject: Re: Flex (reg).. |
| In-reply-to: Your message of Thu, 06 Mar 1997 23:50:16 PST. |
| Date: Thu, 06 Mar 1997 15:54:19 PST |
| From: Vern Paxson <vern> |
| |
| > [:alpha:] ([:alnum:] | \\_)* |
| |
| If your rule really has embedded blanks as shown above, then it won't |
| work, as the first blank delimits the rule from the action. (It wouldn't |
| even compile ...) You need instead: |
| |
| [:alpha:]([:alnum:]|\\_)* |
| |
| and that should work fine - there's no restriction on what can go inside |
| of ()'s except for the trailing context operator, '/'. |
| |
| Vern |
| |
| |
| File: flex.info, Node: I need to scan if-then-else blocks and while loops, Next: ERASEME55, Prev: ERASEME53, Up: FAQ |
| |
| I need to scan if-then-else blocks and while loops |
| ================================================== |
| |
| To: "Mike Stolnicki" <mstolnic@ford.com> |
| Subject: Re: FLEX help |
| In-reply-to: Your message of Fri, 30 May 1997 13:33:27 PDT. |
| Date: Fri, 30 May 1997 10:46:35 PDT |
| From: Vern Paxson <vern> |
| |
| > We'd like to add "if-then-else", "while", and "for" statements to our |
| > language ... |
| > We've investigated many possible solutions. The one solution that seems |
| > the most reasonable involves knowing the position of a TOKEN in yyin. |
| |
| I strongly advise you to instead build a parse tree (abstract syntax tree) |
| and loop over that instead. You'll find this has major benefits in keeping |
| your interpreter simple and extensible. |
| |
| That said, the functionality you mention for get_position and set_position |
| have been on the to-do list for a while. As flex is a purely spare-time |
| project for me, no guarantees when this will be added (in particular, it |
| for sure won't be for many months to come). |
| |
| Vern |
| |
| |
| File: flex.info, Node: ERASEME55, Next: ERASEME56, Prev: I need to scan if-then-else blocks and while loops, Up: FAQ |
| |
| ERASEME55 |
| ========= |
| |
| To: Colin Paul Adams <colin@colina.demon.co.uk> |
| Subject: Re: Flex C++ classes and Bison |
| In-reply-to: Your message of 09 Aug 1997 17:11:41 PDT. |
| Date: Fri, 15 Aug 1997 10:48:19 PDT |
| From: Vern Paxson <vern> |
| |
| > #define YY_DECL int yylex (YYSTYPE *lvalp, struct parser_control |
| > *parm) |
| > |
| > I have been trying to get this to work as a C++ scanner, but it does |
| > not appear to be possible (warning that it matches no declarations in |
| > yyFlexLexer, or something like that). |
| > |
| > Is this supposed to be possible, or is it being worked on (I DID |
| > notice the comment that scanner classes are still experimental, so I'm |
| > not too hopeful)? |
| |
| What you need to do is derive a subclass from yyFlexLexer that provides |
| the above yylex() method, squirrels away lvalp and parm into member |
| variables, and then invokes yyFlexLexer::yylex() to do the regular scanning. |
| |
| Vern |
| |
| |
| File: flex.info, Node: ERASEME56, Next: ERASEME57, Prev: ERASEME55, Up: FAQ |
| |
| ERASEME56 |
| ========= |
| |
| To: Mikael.Latvala@lmf.ericsson.se |
| Subject: Re: Possible mistake in Flex v2.5 document |
| In-reply-to: Your message of Fri, 05 Sep 1997 16:07:24 PDT. |
| Date: Fri, 05 Sep 1997 10:01:54 PDT |
| From: Vern Paxson <vern> |
| |
| > In that example you show how to count comment lines when using |
| > C style /* ... */ comments. My question is, shouldn't you take into |
| > account a scenario where end of a comment marker occurs inside |
| > character or string literals? |
| |
| The scanner certainly needs to also scan character and string literals. |
| However it does that (there's an example in the man page for strings), the |
| lexer will recognize the beginning of the literal before it runs across the |
| embedded "/*". Consequently, it will finish scanning the literal before it |
| even considers the possibility of matching "/*". |
| |
| Example: |
| |
| '([^']*|{ESCAPE_SEQUENCE})' |
| |
| will match all the text between the ''s (inclusive). So the lexer |
| considers this as a token beginning at the first ', and doesn't even |
| attempt to match other tokens inside it. |
| |
| I thinnk this subtlety is not worth putting in the manual, as I suspect |
| it would confuse more people than it would enlighten. |
| |
| Vern |
| |
| |
| File: flex.info, Node: ERASEME57, Next: Is there a repository for flex scanners?, Prev: ERASEME56, Up: FAQ |
| |
| ERASEME57 |
| ========= |
| |
| To: "Marty Leisner" <leisner@sdsp.mc.xerox.com> |
| Subject: Re: flex limitations |
| In-reply-to: Your message of Sat, 06 Sep 1997 11:27:21 PDT. |
| Date: Mon, 08 Sep 1997 11:38:08 PDT |
| From: Vern Paxson <vern> |
| |
| > %% |
| > [a-zA-Z]+ /* skip a line */ |
| > { printf("got %s\n", yytext); } |
| > %% |
| |
| What version of flex are you using? If I feed this to 2.5.4, it complains: |
| |
| "bug.l", line 5: EOF encountered inside an action |
| "bug.l", line 5: unrecognized rule |
| "bug.l", line 5: fatal parse error |
| |
| Not the world's greatest error message, but it manages to flag the problem. |
| |
| (With the introduction of start condition scopes, flex can't accommodate |
| an action on a separate line, since it's ambiguous with an indented rule.) |
| |
| You can get 2.5.4 from ftp.ee.lbl.gov. |
| |
| Vern |
| |
| |
| File: flex.info, Node: Is there a repository for flex scanners?, Next: How can I conditionally compile or preprocess my flex input file?, Prev: ERASEME57, Up: FAQ |
| |
| Is there a repository for flex scanners? |
| ======================================== |
| |
| Not that we know of. You might try asking on comp.compilers. |
| |
| |
| File: flex.info, Node: How can I conditionally compile or preprocess my flex input file?, Next: Where can I find grammars for lex and yacc?, Prev: Is there a repository for flex scanners?, Up: FAQ |
| |
| How can I conditionally compile or preprocess my flex input file? |
| ================================================================= |
| |
| Flex doesn't have a preprocessor like C does. You might try using m4, |
| or the C preprocessor plus a sed script to clean up the result. |
| |
| |
| File: flex.info, Node: Where can I find grammars for lex and yacc?, Next: I get an end-of-buffer message for each character scanned., Prev: How can I conditionally compile or preprocess my flex input file?, Up: FAQ |
| |
| Where can I find grammars for lex and yacc? |
| =========================================== |
| |
| In the sources for flex and bison. |
| |
| |
| File: flex.info, Node: I get an end-of-buffer message for each character scanned., Next: unnamed-faq-62, Prev: Where can I find grammars for lex and yacc?, Up: FAQ |
| |
| I get an end-of-buffer message for each character scanned. |
| ========================================================== |
| |
| This will happen if your LexerInput() function returns only one |
| character at a time, which can happen either if you're scanner is |
| "interactive", or if the streams library on your platform always |
| returns 1 for yyin->gcount(). |
| |
| Solution: override LexerInput() with a version that returns whole |
| buffers. |
| |
| |
| File: flex.info, Node: unnamed-faq-62, Next: unnamed-faq-63, Prev: I get an end-of-buffer message for each character scanned., Up: FAQ |
| |
| unnamed-faq-62 |
| ============== |
| |
| To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE |
| Subject: Re: Flex maximums |
| In-reply-to: Your message of Mon, 17 Nov 1997 17:16:06 PST. |
| Date: Mon, 17 Nov 1997 17:16:15 PST |
| From: Vern Paxson <vern> |
| |
| > I took a quick look into the flex-sources and altered some #defines in |
| > flexdefs.h: |
| > |
| > #define INITIAL_MNS 64000 |
| > #define MNS_INCREMENT 1024000 |
| > #define MAXIMUM_MNS 64000 |
| |
| The things to fix are to add a couple of zeroes to: |
| |
| #define JAMSTATE -32766 /* marks a reference to the state that always jams */ |
| #define MAXIMUM_MNS 31999 |
| #define BAD_SUBSCRIPT -32767 |
| #define MAX_SHORT 32700 |
| |
| and, if you get complaints about too many rules, make the following change too: |
| |
| #define YY_TRAILING_MASK 0x200000 |
| #define YY_TRAILING_HEAD_MASK 0x400000 |
| |
| - Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-63, Next: unnamed-faq-64, Prev: unnamed-faq-62, Up: FAQ |
| |
| unnamed-faq-63 |
| ============== |
| |
| To: jimmey@lexis-nexis.com (Jimmey Todd) |
| Subject: Re: FLEX question regarding istream vs ifstream |
| In-reply-to: Your message of Mon, 08 Dec 1997 15:54:15 PST. |
| Date: Mon, 15 Dec 1997 13:21:35 PST |
| From: Vern Paxson <vern> |
| |
| > stdin_handle = YY_CURRENT_BUFFER; |
| > ifstream fin( "aFile" ); |
| > yy_switch_to_buffer( yy_create_buffer( fin, YY_BUF_SIZE ) ); |
| > |
| > What I'm wanting to do, is pass the contents of a file thru one set |
| > of rules and then pass stdin thru another set... It works great if, I |
| > don't use the C++ classes. But since everything else that I'm doing is |
| > in C++, I thought I'd be consistent. |
| > |
| > The problem is that 'yy_create_buffer' is expecting an istream* as it's |
| > first argument (as stated in the man page). However, fin is a ifstream |
| > object. Any ideas on what I might be doing wrong? Any help would be |
| > appreciated. Thanks!! |
| |
| You need to pass &fin, to turn it into an ifstream* instead of an ifstream. |
| Then its type will be compatible with the expected istream*, because ifstream |
| is derived from istream. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-64, Next: unnamed-faq-65, Prev: unnamed-faq-63, Up: FAQ |
| |
| unnamed-faq-64 |
| ============== |
| |
| To: Enda Fadian <fadiane@piercom.ie> |
| Subject: Re: Question related to Flex man page? |
| In-reply-to: Your message of Tue, 16 Dec 1997 15:17:34 PST. |
| Date: Tue, 16 Dec 1997 14:17:09 PST |
| From: Vern Paxson <vern> |
| |
| > Can you explain to me what is ment by a long-jump in relation to flex? |
| |
| Using the longjmp() function while inside yylex() or a routine called by it. |
| |
| > what is the flex activation frame. |
| |
| Just yylex()'s stack frame. |
| |
| > As far as I can see yyrestart will bring me back to the sart of the input |
| > file and using flex++ isnot really an option! |
| |
| No, yyrestart() doesn't imply a rewind, even though its name might sound |
| like it does. It tells the scanner to flush its internal buffers and |
| start reading from the given file at its present location. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-65, Next: unnamed-faq-66, Prev: unnamed-faq-64, Up: FAQ |
| |
| unnamed-faq-65 |
| ============== |
| |
| To: hassan@larc.info.uqam.ca (Hassan Alaoui) |
| Subject: Re: Need urgent Help |
| In-reply-to: Your message of Sat, 20 Dec 1997 19:38:19 PST. |
| Date: Sun, 21 Dec 1997 21:30:46 PST |
| From: Vern Paxson <vern> |
| |
| > /usr/lib/yaccpar: In function `int yyparse()': |
| > /usr/lib/yaccpar:184: warning: implicit declaration of function `int yylex(...)' |
| > |
| > ld: Undefined symbol |
| > _yylex |
| > _yyparse |
| > _yyin |
| |
| This is a known problem with Solaris C++ (and/or Solaris yacc). I believe |
| the fix is to explicitly insert some 'extern "C"' statements for the |
| corresponding routines/symbols. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-66, Next: unnamed-faq-67, Prev: unnamed-faq-65, Up: FAQ |
| |
| unnamed-faq-66 |
| ============== |
| |
| To: mc0307@mclink.it |
| Cc: gnu@prep.ai.mit.edu |
| Subject: Re: [mc0307@mclink.it: Help request] |
| In-reply-to: Your message of Fri, 12 Dec 1997 17:57:29 PST. |
| Date: Sun, 21 Dec 1997 22:33:37 PST |
| From: Vern Paxson <vern> |
| |
| > This is my definition for float and integer types: |
| > . . . |
| > NZD [1-9] |
| > ... |
| > I've tested my program on other lex version (on UNIX Sun Solaris an HP |
| > UNIX) and it work well, so I think that my definitions are correct. |
| > There are any differences between Lex and Flex? |
| |
| There are indeed differences, as discussed in the man page. The one |
| you are probably running into is that when flex expands a name definition, |
| it puts parentheses around the expansion, while lex does not. There's |
| an example in the man page of how this can lead to different matching. |
| Flex's behavior complies with the POSIX standard (or at least with the |
| last POSIX draft I saw). |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-67, Next: unnamed-faq-68, Prev: unnamed-faq-66, Up: FAQ |
| |
| unnamed-faq-67 |
| ============== |
| |
| To: hassan@larc.info.uqam.ca (Hassan Alaoui) |
| Subject: Re: Thanks |
| In-reply-to: Your message of Mon, 22 Dec 1997 16:06:35 PST. |
| Date: Mon, 22 Dec 1997 14:35:05 PST |
| From: Vern Paxson <vern> |
| |
| > Thank you very much for your help. I compile and link well with C++ while |
| > declaring 'yylex ...' extern, But a little problem remains. I get a |
| > segmentation default when executing ( I linked with lfl library) while it |
| > works well when using LEX instead of flex. Do you have some ideas about the |
| > reason for this ? |
| |
| The one possible reason for this that comes to mind is if you've defined |
| yytext as "extern char yytext[]" (which is what lex uses) instead of |
| "extern char *yytext" (which is what flex uses). If it's not that, then |
| I'm afraid I don't know what the problem might be. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-68, Next: unnamed-faq-69, Prev: unnamed-faq-67, Up: FAQ |
| |
| unnamed-faq-68 |
| ============== |
| |
| To: "Bart Niswonger" <NISWONGR@almaden.ibm.com> |
| Subject: Re: flex 2.5: c++ scanners & start conditions |
| In-reply-to: Your message of Tue, 06 Jan 1998 10:34:21 PST. |
| Date: Tue, 06 Jan 1998 19:19:30 PST |
| From: Vern Paxson <vern> |
| |
| > The problem is that when I do this (using %option c++) start |
| > conditions seem to not apply. |
| |
| The BEGIN macro modifies the yy_start variable. For C scanners, this |
| is a static with scope visible through the whole file. For C++ scanners, |
| it's a member variable, so it only has visible scope within a member |
| function. Your lexbegin() routine is not a member function when you |
| build a C++ scanner, so it's not modifying the correct yy_start. The |
| diagnostic that indicates this is that you found you needed to add |
| a declaration of yy_start in order to get your scanner to compile when |
| using C++; instead, the correct fix is to make lexbegin() a member |
| function (by deriving from yyFlexLexer). |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-69, Next: unnamed-faq-70, Prev: unnamed-faq-68, Up: FAQ |
| |
| unnamed-faq-69 |
| ============== |
| |
| To: "Boris Zinin" <boris@ippe.rssi.ru> |
| Subject: Re: current position in flex buffer |
| In-reply-to: Your message of Mon, 12 Jan 1998 18:58:23 PST. |
| Date: Mon, 12 Jan 1998 12:03:15 PST |
| From: Vern Paxson <vern> |
| |
| > The problem is how to determine the current position in flex active |
| > buffer when a rule is matched.... |
| |
| You will need to keep track of this explicitly, such as by redefining |
| YY_USER_ACTION to count the number of characters matched. |
| |
| The latest flex release, by the way, is 2.5.4, available from ftp.ee.lbl.gov. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-70, Next: unnamed-faq-71, Prev: unnamed-faq-69, Up: FAQ |
| |
| unnamed-faq-70 |
| ============== |
| |
| To: Bik.Dhaliwal@bis.org |
| Subject: Re: Flex question |
| In-reply-to: Your message of Mon, 26 Jan 1998 13:05:35 PST. |
| Date: Tue, 27 Jan 1998 22:41:52 PST |
| From: Vern Paxson <vern> |
| |
| > That requirement involves knowing |
| > the character position at which a particular token was matched |
| > in the lexer. |
| |
| The way you have to do this is by explicitly keeping track of where |
| you are in the file, by counting the number of characters scanned |
| for each token (available in yyleng). It may prove convenient to |
| do this by redefining YY_USER_ACTION, as described in the manual. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-71, Next: unnamed-faq-72, Prev: unnamed-faq-70, Up: FAQ |
| |
| unnamed-faq-71 |
| ============== |
| |
| To: Vladimir Alexiev <vladimir@cs.ualberta.ca> |
| Subject: Re: flex: how to control start condition from parser? |
| In-reply-to: Your message of Mon, 26 Jan 1998 05:50:16 PST. |
| Date: Tue, 27 Jan 1998 22:45:37 PST |
| From: Vern Paxson <vern> |
| |
| > It seems useful for the parser to be able to tell the lexer about such |
| > context dependencies, because then they don't have to be limited to |
| > local or sequential context. |
| |
| One way to do this is to have the parser call a stub routine that's |
| included in the scanner's .l file, and consequently that has access ot |
| BEGIN. The only ugliness is that the parser can't pass in the state |
| it wants, because those aren't visible - but if you don't have many |
| such states, then using a different set of names doesn't seem like |
| to much of a burden. |
| |
| While generating a .h file like you suggests is certainly cleaner, |
| flex development has come to a virtual stand-still :-(, so a workaround |
| like the above is much more pragmatic than waiting for a new feature. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-72, Next: unnamed-faq-73, Prev: unnamed-faq-71, Up: FAQ |
| |
| unnamed-faq-72 |
| ============== |
| |
| To: Barbara Denny <denny@3com.com> |
| Subject: Re: freebsd flex bug? |
| In-reply-to: Your message of Fri, 30 Jan 1998 12:00:43 PST. |
| Date: Fri, 30 Jan 1998 12:42:32 PST |
| From: Vern Paxson <vern> |
| |
| > lex.yy.c:1996: parse error before `=' |
| |
| This is the key, identifying this error. (It may help to pinpoint |
| it by using flex -L, so it doesn't generate #line directives in its |
| output.) I will bet you heavy money that you have a start condition |
| name that is also a variable name, or something like that; flex spits |
| out #define's for each start condition name, mapping them to a number, |
| so you can wind up with: |
| |
| %x foo |
| %% |
| ... |
| %% |
| void bar() |
| { |
| int foo = 3; |
| } |
| |
| and the penultimate will turn into "int 1 = 3" after C preprocessing, |
| since flex will put "#define foo 1" in the generated scanner. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-73, Next: unnamed-faq-74, Prev: unnamed-faq-72, Up: FAQ |
| |
| unnamed-faq-73 |
| ============== |
| |
| To: Maurice Petrie <mpetrie@infoscigroup.com> |
| Subject: Re: Lost flex .l file |
| In-reply-to: Your message of Mon, 02 Feb 1998 14:10:01 PST. |
| Date: Mon, 02 Feb 1998 11:15:12 PST |
| From: Vern Paxson <vern> |
| |
| > I am curious as to |
| > whether there is a simple way to backtrack from the generated source to |
| > reproduce the lost list of tokens we are searching on. |
| |
| In theory, it's straight-forward to go from the DFA representation |
| back to a regular-expression representation - the two are isomorphic. |
| In practice, a huge headache, because you have to unpack all the tables |
| back into a single DFA representation, and then write a program to munch |
| on that and translate it into an RE. |
| |
| Sorry for the less-than-happy news ... |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-74, Next: unnamed-faq-75, Prev: unnamed-faq-73, Up: FAQ |
| |
| unnamed-faq-74 |
| ============== |
| |
| To: jimmey@lexis-nexis.com (Jimmey Todd) |
| Subject: Re: Flex performance question |
| In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST. |
| Date: Thu, 19 Feb 1998 08:48:51 PST |
| From: Vern Paxson <vern> |
| |
| > What I have found, is that the smaller the data chunk, the faster the |
| > program executes. This is the opposite of what I expected. Should this be |
| > happening this way? |
| |
| This is exactly what will happen if your input file has embedded NULs. |
| From the man page: |
| |
| A final note: flex is slow when matching NUL's, particularly |
| when a token contains multiple NUL's. It's best to write |
| rules which match short amounts of text if it's anticipated |
| that the text will often include NUL's. |
| |
| So that's the first thing to look for. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-75, Next: unnamed-faq-76, Prev: unnamed-faq-74, Up: FAQ |
| |
| unnamed-faq-75 |
| ============== |
| |
| To: jimmey@lexis-nexis.com (Jimmey Todd) |
| Subject: Re: Flex performance question |
| In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST. |
| Date: Thu, 19 Feb 1998 15:42:25 PST |
| From: Vern Paxson <vern> |
| |
| So there are several problems. |
| |
| First, to go fast, you want to match as much text as possible, which |
| your scanners don't in the case that what they're scanning is *not* |
| a <RN> tag. So you want a rule like: |
| |
| [^<]+ |
| |
| Second, C++ scanners are particularly slow if they're interactive, |
| which they are by default. Using -B speeds it up by a factor of 3-4 |
| on my workstation. |
| |
| Third, C++ scanners that use the istream interface are slow, because |
| of how poorly implemented istream's are. I built two versions of |
| the following scanner: |
| |
| %% |
| .*\n |
| .* |
| %% |
| |
| and the C version inhales a 2.5MB file on my workstation in 0.8 seconds. |
| The C++ istream version, using -B, takes 3.8 seconds. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-76, Next: unnamed-faq-77, Prev: unnamed-faq-75, Up: FAQ |
| |
| unnamed-faq-76 |
| ============== |
| |
| To: "Frescatore, David (CRD, TAD)" <frescatore@exc01crdge.crd.ge.com> |
| Subject: Re: FLEX 2.5 & THE YEAR 2000 |
| In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT. |
| Date: Wed, 03 Jun 1998 10:22:26 PDT |
| From: Vern Paxson <vern> |
| |
| > I am researching the Y2K problem with General Electric R&D |
| > and need to know if there are any known issues concerning |
| > the above mentioned software and Y2K regardless of version. |
| |
| There shouldn't be, all it ever does with the date is ask the system |
| for it and then print it out. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-77, Next: unnamed-faq-78, Prev: unnamed-faq-76, Up: FAQ |
| |
| unnamed-faq-77 |
| ============== |
| |
| To: "Hans Dermot Doran" <htd@ibhdoran.com> |
| Subject: Re: flex problem |
| In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT. |
| Date: Tue, 21 Jul 1998 14:23:34 PDT |
| From: Vern Paxson <vern> |
| |
| > To overcome this, I gets() the stdin into a string and lex the string. The |
| > string is lexed OK except that the end of string isn't lexed properly |
| > (yy_scan_string()), that is the lexer dosn't recognise the end of string. |
| |
| Flex doesn't contain mechanisms for recognizing buffer endpoints. But if |
| you use fgets instead (which you should anyway, to protect against buffer |
| overflows), then the final \n will be preserved in the string, and you can |
| scan that in order to find the end of the string. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-78, Next: unnamed-faq-79, Prev: unnamed-faq-77, Up: FAQ |
| |
| unnamed-faq-78 |
| ============== |
| |
| To: soumen@almaden.ibm.com |
| Subject: Re: Flex++ 2.5.3 instance member vs. static member |
| In-reply-to: Your message of Mon, 27 Jul 1998 02:10:04 PDT. |
| Date: Tue, 28 Jul 1998 01:10:34 PDT |
| From: Vern Paxson <vern> |
| |
| > %{ |
| > int mylineno = 0; |
| > %} |
| > ws [ \t]+ |
| > alpha [A-Za-z] |
| > dig [0-9] |
| > %% |
| > |
| > Now you'd expect mylineno to be a member of each instance of class |
| > yyFlexLexer, but is this the case? A look at the lex.yy.cc file seems to |
| > indicate otherwise; unless I am missing something the declaration of |
| > mylineno seems to be outside any class scope. |
| > |
| > How will this work if I want to run a multi-threaded application with each |
| > thread creating a FlexLexer instance? |
| |
| Derive your own subclass and make mylineno a member variable of it. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-79, Next: unnamed-faq-80, Prev: unnamed-faq-78, Up: FAQ |
| |
| unnamed-faq-79 |
| ============== |
| |
| To: Adoram Rogel <adoram@hybridge.com> |
| Subject: Re: More than 32K states change hangs |
| In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT. |
| Date: Tue, 04 Aug 1998 22:28:45 PDT |
| From: Vern Paxson <vern> |
| |
| > Vern Paxson, |
| > |
| > I followed your advice, posted on Usenet bu you, and emailed to me |
| > personally by you, on how to overcome the 32K states limit. I'm running |
| > on Linux machines. |
| > I took the full source of version 2.5.4 and did the following changes in |
| > flexdef.h: |
| > #define JAMSTATE -327660 |
| > #define MAXIMUM_MNS 319990 |
| > #define BAD_SUBSCRIPT -327670 |
| > #define MAX_SHORT 327000 |
| > |
| > and compiled. |
| > All looked fine, including check and bigcheck, so I installed. |
| |
| Hmmm, you shouldn't increase MAX_SHORT, though looking through my email |
| archives I see that I did indeed recommend doing so. Try setting it back |
| to 32700; that should suffice that you no longer need -Ca. If it still |
| hangs, then the interesting question is - where? |
| |
| > Compiling the same hanged program with a out-of-the-box (RedHat 4.2 |
| > distribution of Linux) |
| > flex 2.5.4 binary works. |
| |
| Since Linux comes with source code, you should diff it against what |
| you have to see what problems they missed. |
| |
| > Should I always compile with the -Ca option now ? even short and simple |
| > filters ? |
| |
| No, definitely not. It's meant to be for those situations where you |
| absolutely must squeeze every last cycle out of your scanner. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-80, Next: unnamed-faq-81, Prev: unnamed-faq-79, Up: FAQ |
| |
| unnamed-faq-80 |
| ============== |
| |
| To: "Schmackpfeffer, Craig" <Craig.Schmackpfeffer@usa.xerox.com> |
| Subject: Re: flex output for static code portion |
| In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT. |
| Date: Mon, 17 Aug 1998 23:57:42 PDT |
| From: Vern Paxson <vern> |
| |
| > I would like to use flex under the hood to generate a binary file |
| > containing the data structures that control the parse. |
| |
| This has been on the wish-list for a long time. In principle it's |
| straight-forward - you redirect mkdata() et al's I/O to another file, |
| and modify the skeleton to have a start-up function that slurps these |
| into dynamic arrays. The concerns are (1) the scanner generation code |
| is hairy and full of corner cases, so it's easy to get surprised when |
| going down this path :-( ; and (2) being careful about buffering so |
| that when the tables change you make sure the scanner starts in the |
| correct state and reading at the right point in the input file. |
| |
| > I was wondering if you know of anyone who has used flex in this way. |
| |
| I don't - but it seems like a reasonable project to undertake (unlike |
| numerous other flex tweaks :-). |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-81, Next: unnamed-faq-82, Prev: unnamed-faq-80, Up: FAQ |
| |
| unnamed-faq-81 |
| ============== |
| |
| Received: from 131.173.17.11 (131.173.17.11 [131.173.17.11]) |
| by ee.lbl.gov (8.9.1/8.9.1) with ESMTP id AAA03838 |
| for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 00:47:57 -0700 (PDT) |
| Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2]) |
| by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694 |
| for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 09:47:55 +0200 |
| Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200 |
| From: Georg Rehm <georg@hal.cl-ki.uni-osnabrueck.de> |
| Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de> |
| Subject: "flex scanner push-back overflow" |
| To: vern@ee.lbl.gov |
| Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST) |
| Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE |
| X-NoJunk: Do NOT send commercial mail, spam or ads to this address! |
| X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/ |
| X-Mailer: ELM [version 2.4ME+ PL28 (25)] |
| MIME-Version: 1.0 |
| Content-Type: text/plain; charset=US-ASCII |
| Content-Transfer-Encoding: 7bit |
| |
| Hi Vern, |
| |
| Yesterday, I encountered a strange problem: I use the macro processor m4 |
| to include some lengthy lists into a .l file. Following is a flex macro |
| definition that causes some serious pain in my neck: |
| |
| AUTHOR ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...]) |
| |
| The complete list contains about 10kB. When I try to "flex" this file |
| (on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased |
| some of the predefined values in flexdefs.h) I get the error: |
| |
| myflex/flex -8 sentag.tmp.l |
| flex scanner push-back overflow |
| |
| When I remove the slashes in the macro definition everything works fine. |
| As I understand it, the double quotes escape the slash-character so it |
| really means "/" and not "trailing context". Furthermore, I tried to |
| escape the slashes with backslashes, but with no use, the same error message |
| appeared when flexing the code. |
| |
| Do you have an idea what's going on here? |
| |
| Greetings from Germany, |
| Georg |
| -- |
| Georg Rehm georg@cl-ki.uni-osnabrueck.de |
| Institute for Semantic Information Processing, University of Osnabrueck, FRG |
| |
| |
| File: flex.info, Node: unnamed-faq-82, Next: unnamed-faq-83, Prev: unnamed-faq-81, Up: FAQ |
| |
| unnamed-faq-82 |
| ============== |
| |
| To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE |
| Subject: Re: "flex scanner push-back overflow" |
| In-reply-to: Your message of Thu, 20 Aug 1998 09:47:54 PDT. |
| Date: Thu, 20 Aug 1998 07:05:35 PDT |
| From: Vern Paxson <vern> |
| |
| > myflex/flex -8 sentag.tmp.l |
| > flex scanner push-back overflow |
| |
| Flex itself uses a flex scanner. That scanner is running out of buffer |
| space when it tries to unput() the humongous macro you've defined. When |
| you remove the '/'s, you make it small enough so that it fits in the buffer; |
| removing spaces would do the same thing. |
| |
| The fix is to either rethink how come you're using such a big macro and |
| perhaps there's another/better way to do it; or to rebuild flex's own |
| scan.c with a larger value for |
| |
| #define YY_BUF_SIZE 16384 |
| |
| - Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-83, Next: unnamed-faq-84, Prev: unnamed-faq-82, Up: FAQ |
| |
| unnamed-faq-83 |
| ============== |
| |
| To: Jan Kort <jan@research.techforce.nl> |
| Subject: Re: Flex |
| In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200. |
| Date: Sat, 05 Sep 1998 00:59:49 PDT |
| From: Vern Paxson <vern> |
| |
| > %% |
| > |
| > "TEST1\n" { fprintf(stderr, "TEST1\n"); yyless(5); } |
| > ^\n { fprintf(stderr, "empty line\n"); } |
| > . { } |
| > \n { fprintf(stderr, "new line\n"); } |
| > |
| > %% |
| > -- input --------------------------------------- |
| > TEST1 |
| > -- output -------------------------------------- |
| > TEST1 |
| > empty line |
| > ------------------------------------------------ |
| |
| IMHO, it's not clear whether or not this is in fact a bug. It depends |
| on whether you view yyless() as backing up in the input stream, or as |
| pushing new characters onto the beginning of the input stream. Flex |
| interprets it as the latter (for implementation convenience, I'll admit), |
| and so considers the newline as in fact matching at the beginning of a |
| line, as after all the last token scanned an entire line and so the |
| scanner is now at the beginning of a new line. |
| |
| I agree that this is counter-intuitive for yyless(), given its |
| functional description (it's less so for unput(), depending on whether |
| you're unput()'ing new text or scanned text). But I don't plan to |
| change it any time soon, as it's a pain to do so. Consequently, |
| you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak |
| your scanner into the behavior you desire. |
| |
| Sorry for the less-than-completely-satisfactory answer. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-84, Next: unnamed-faq-85, Prev: unnamed-faq-83, Up: FAQ |
| |
| unnamed-faq-84 |
| ============== |
| |
| To: Patrick Krusenotto <krusenot@mac-info-link.de> |
| Subject: Re: Problems with restarting flex-2.5.2-generated scanner |
| In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT. |
| Date: Thu, 24 Sep 1998 23:28:43 PDT |
| From: Vern Paxson <vern> |
| |
| > I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately |
| > trying to make my scanner restart with a new file after my parser stops |
| > with a parse error. When my compiler restarts, the parser always |
| > receives the token after the token (in the old file!) that caused the |
| > parser error. |
| |
| I suspect the problem is that your parser has read ahead in order |
| to attempt to resolve an ambiguity, and when it's restarted it picks |
| up with that token rather than reading a fresh one. If you're using |
| yacc, then the special "error" production can sometimes be used to |
| consume tokens in an attempt to get the parser into a consistent state. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-85, Next: unnamed-faq-86, Prev: unnamed-faq-84, Up: FAQ |
| |
| unnamed-faq-85 |
| ============== |
| |
| To: Henric Jungheim <junghelh@pe-nelson.com> |
| Subject: Re: flex 2.5.4a |
| In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST. |
| Date: Tue, 27 Oct 1998 16:50:14 PST |
| From: Vern Paxson <vern> |
| |
| > This brings up a feature request: How about a command line |
| > option to specify the filename when reading from stdin? That way one |
| > doesn't need to create a temporary file in order to get the "#line" |
| > directives to make sense. |
| |
| Use -o combined with -t (per the man page description of -o). |
| |
| > P.S., Is there any simple way to use non-blocking IO to parse multiple |
| > streams? |
| |
| Simple, no. |
| |
| One approach might be to return a magic character on EWOULDBLOCK and |
| have a rule |
| |
| .*<magic-character> // put back .*, eat magic character |
| |
| This is off the top of my head, not sure it'll work. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-86, Next: unnamed-faq-87, Prev: unnamed-faq-85, Up: FAQ |
| |
| unnamed-faq-86 |
| ============== |
| |
| To: "Repko, Billy D" <billy.d.repko@intel.com> |
| Subject: Re: Compiling scanners |
| In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST. |
| Date: Thu, 14 Jan 1999 00:25:30 PST |
| From: Vern Paxson <vern> |
| |
| > It appears that maybe it cannot find the lfl library. |
| |
| The Makefile in the distribution builds it, so you should have it. |
| It's exceedingly trivial, just a main() that calls yylex() and |
| a yyrap() that always returns 1. |
| |
| > %% |
| > \n ++num_lines; ++num_chars; |
| > . ++num_chars; |
| |
| You can't indent your rules like this - that's where the errors are coming |
| from. Flex copies indented text to the output file, it's how you do things |
| like |
| |
| int num_lines_seen = 0; |
| |
| to declare local variables. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-87, Next: unnamed-faq-88, Prev: unnamed-faq-86, Up: FAQ |
| |
| unnamed-faq-87 |
| ============== |
| |
| To: Erick Branderhorst <Erick.Branderhorst@asml.nl> |
| Subject: Re: flex input buffer |
| In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST. |
| Date: Tue, 09 Feb 1999 21:03:37 PST |
| From: Vern Paxson <vern> |
| |
| > In the flex.skl file the size of the default input buffers is set. Can you |
| > explain why this size is set and why it is such a high number. |
| |
| It's large to optimize performance when scanning large files. You can |
| safely make it a lot lower if needed. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-88, Next: unnamed-faq-90, Prev: unnamed-faq-87, Up: FAQ |
| |
| unnamed-faq-88 |
| ============== |
| |
| To: "Guido Minnen" <guidomi@cogs.susx.ac.uk> |
| Subject: Re: Flex error message |
| In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST. |
| Date: Thu, 25 Feb 1999 00:11:31 PST |
| From: Vern Paxson <vern> |
| |
| > I'm extending a larger scanner written in Flex and I keep running into |
| > problems. More specifically, I get the error message: |
| > "flex: input rules are too complicated (>= 32000 NFA states)" |
| |
| Increase the definitions in flexdef.h for: |
| |
| #define JAMSTATE -32766 /* marks a reference to the state that always j |
| ams */ |
| #define MAXIMUM_MNS 31999 |
| #define BAD_SUBSCRIPT -32767 |
| |
| recompile everything, and it should all work. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-90, Next: unnamed-faq-91, Prev: unnamed-faq-88, Up: FAQ |
| |
| unnamed-faq-90 |
| ============== |
| |
| To: "Dmitriy Goldobin" <gold@ems.chel.su> |
| Subject: Re: FLEX trouble |
| In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT. |
| Date: Tue, 01 Jun 1999 00:15:07 PDT |
| From: Vern Paxson <vern> |
| |
| > I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20 |
| > but rule "/*"(.|\n)*"*/" don't work ? |
| |
| The second of these will have to scan the entire input stream (because |
| "(.|\n)*" matches an arbitrary amount of any text) in order to see if |
| it ends with "*/", terminating the comment. That potentially will overflow |
| the input buffer. |
| |
| > More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error |
| > 'unrecognized rule'. |
| |
| You can't use the '/' operator inside parentheses. It's not clear |
| what "(a/b)*" actually means. |
| |
| > I now use workaround with state <comment>, but single-rule is |
| > better, i think. |
| |
| Single-rule is nice but will always have the problem of either setting |
| restrictions on comments (like not allowing multi-line comments) and/or |
| running the risk of consuming the entire input stream, as noted above. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-91, Next: unnamed-faq-92, Prev: unnamed-faq-90, Up: FAQ |
| |
| unnamed-faq-91 |
| ============== |
| |
| Received: from mc-qout4.whowhere.com (mc-qout4.whowhere.com [209.185.123.18]) |
| by ee.lbl.gov (8.9.3/8.9.3) with SMTP id IAA05100 |
| for <vern@ee.lbl.gov>; Tue, 15 Jun 1999 08:56:06 -0700 (PDT) |
| Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999 |
| To: vern@ee.lbl.gov |
| Date: Tue, 15 Jun 1999 08:55:43 -0700 |
| From: "Aki Niimura" <neko@my-deja.com> |
| Message-ID: <KNONDOHDOBGAEAAA@my-deja.com> |
| Mime-Version: 1.0 |
| Cc: |
| X-Sent-Mail: on |
| Reply-To: |
| X-Mailer: MailCity Service |
| Subject: A question on flex C++ scanner |
| X-Sender-Ip: 12.72.207.61 |
| Organization: My Deja Email (http://www.my-deja.com:80) |
| Content-Type: text/plain; charset=us-ascii |
| Content-Transfer-Encoding: 7bit |
| |
| Dear Dr. Paxon, |
| |
| I have been using flex for years. |
| It works very well on many projects. |
| Most case, I used it to generate a scanner on C language. |
| However, one project I needed to generate a scanner |
| on C++ lanuage. Thanks to your enhancement, flex did |
| the job. |
| |
| Currently, I'm working on enhancing my previous project. |
| I need to deal with multiple input streams (recursive |
| inclusion) in this scanner (C++). |
| I did similar thing for another scanner (C) as you |
| explained in your documentation. |
| |
| The generated scanner (C++) has necessary methods: |
| - switch_to_buffer(struct yy_buffer_state *b) |
| - yy_create_buffer(istream *is, int sz) |
| - yy_delete_buffer(struct yy_buffer_state *b) |
| |
| However, I couldn't figure out how to access current |
| buffer (yy_current_buffer). |
| |
| yy_current_buffer is a protected member of yyFlexLexer. |
| I can't access it directly. |
| Then, I thought yy_create_buffer() with is = 0 might |
| return current stream buffer. But it seems not as far |
| as I checked the source. (flex 2.5.4) |
| |
| I went through the Web in addition to Flex documentation. |
| However, it hasn't been successful, so far. |
| |
| It is not my intention to bother you, but, can you |
| comment about how to obtain the current stream buffer? |
| |
| Your response would be highly appreciated. |
| |
| Best regards, |
| Aki Niimura |
| |
| --== Sent via Deja.com http://www.deja.com/ ==-- |
| Share what you know. Learn what you don't. |
| |
| |
| File: flex.info, Node: unnamed-faq-92, Next: unnamed-faq-93, Prev: unnamed-faq-91, Up: FAQ |
| |
| unnamed-faq-92 |
| ============== |
| |
| To: neko@my-deja.com |
| Subject: Re: A question on flex C++ scanner |
| In-reply-to: Your message of Tue, 15 Jun 1999 08:55:43 PDT. |
| Date: Tue, 15 Jun 1999 09:04:24 PDT |
| From: Vern Paxson <vern> |
| |
| > However, I couldn't figure out how to access current |
| > buffer (yy_current_buffer). |
| |
| Derive your own subclass from yyFlexLexer. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-93, Next: unnamed-faq-94, Prev: unnamed-faq-92, Up: FAQ |
| |
| unnamed-faq-93 |
| ============== |
| |
| To: "Stones, Darren" <Darren.Stones@nectech.co.uk> |
| Subject: Re: You're the man to see? |
| In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT. |
| Date: Wed, 23 Jun 1999 09:01:40 PDT |
| From: Vern Paxson <vern> |
| |
| > I hope you can help me. I am using Flex and Bison to produce an interpreted |
| > language. However all goes well until I try to implement an IF statement or |
| > a WHILE. I cannot get this to work as the parser parses all the conditions |
| > eg. the TRUE and FALSE conditons to check for a rule match. So I cannot |
| > make a decision!! |
| |
| You need to use the parser to build a parse tree (= abstract syntax trwee), |
| and when that's all done you recursively evaluate the tree, binding variables |
| to values at that time. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-94, Next: unnamed-faq-95, Prev: unnamed-faq-93, Up: FAQ |
| |
| unnamed-faq-94 |
| ============== |
| |
| To: Petr Danecek <petr@ics.cas.cz> |
| Subject: Re: flex - question |
| In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT. |
| Date: Fri, 02 Jul 1999 16:52:13 PDT |
| From: Vern Paxson <vern> |
| |
| > file, it takes an enormous amount of time. It is funny, because the |
| > source code has only 12 rules!!! I think it looks like an exponencial |
| > growth. |
| |
| Right, that's the problem - some patterns (those with a lot of |
| ambiguity, where yours has because at any given time the scanner can |
| be in the middle of all sorts of combinations of the different |
| rules) blow up exponentially. |
| |
| For your rules, there is an easy fix. Change the ".*" that comes fater |
| the directory name to "[^ ]*". With that in place, the rules are no |
| longer nearly so ambiguous, because then once one of the directories |
| has been matched, no other can be matched (since they all require a |
| leading blank). |
| |
| If that's not an acceptable solution, then you can enter a start state |
| to pick up the .*\n after each directory is matched. |
| |
| Also note that for speed, you'll want to add a ".*" rule at the end, |
| otherwise rules that don't match any of the patterns will be matched |
| very slowly, a character at a time. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-95, Next: unnamed-faq-96, Prev: unnamed-faq-94, Up: FAQ |
| |
| unnamed-faq-95 |
| ============== |
| |
| To: Tielman Koekemoer <tielman@spi.co.za> |
| Subject: Re: Please help. |
| In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT. |
| Date: Thu, 08 Jul 1999 08:20:39 PDT |
| From: Vern Paxson <vern> |
| |
| > I was hoping you could help me with my problem. |
| > |
| > I tried compiling (gnu)flex on a Solaris 2.4 machine |
| > but when I ran make (after configure) I got an error. |
| > |
| > -------------------------------------------------------------- |
| > gcc -c -I. -I. -g -O parse.c |
| > ./flex -t -p ./scan.l >scan.c |
| > sh: ./flex: not found |
| > *** Error code 1 |
| > make: Fatal error: Command failed for target `scan.c' |
| > ------------------------------------------------------------- |
| > |
| > What's strange to me is that I'm only |
| > trying to install flex now. I then edited the Makefile to |
| > and changed where it says "FLEX = flex" to "FLEX = lex" |
| > ( lex: the native Solaris one ) but then it complains about |
| > the "-p" option. Is there any way I can compile flex without |
| > using flex or lex? |
| > |
| > Thanks so much for your time. |
| |
| You managed to step on the bootstrap sequence, which first copies |
| initscan.c to scan.c in order to build flex. Try fetching a fresh |
| distribution from ftp.ee.lbl.gov. (Or you can first try removing |
| ".bootstrap" and doing a make again.) |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-96, Next: unnamed-faq-97, Prev: unnamed-faq-95, Up: FAQ |
| |
| unnamed-faq-96 |
| ============== |
| |
| To: Tielman Koekemoer <tielman@spi.co.za> |
| Subject: Re: Please help. |
| In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT. |
| Date: Fri, 09 Jul 1999 00:27:20 PDT |
| From: Vern Paxson <vern> |
| |
| > First I removed .bootstrap (and ran make) - no luck. I downloaded the |
| > software but I still have the same problem. Is there anything else I |
| > could try. |
| |
| Try: |
| |
| cp initscan.c scan.c |
| touch scan.c |
| make scan.o |
| |
| If this last tries to first build scan.c from scan.l using ./flex, then |
| your "make" is broken, in which case compile scan.c to scan.o by hand. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-97, Next: unnamed-faq-98, Prev: unnamed-faq-96, Up: FAQ |
| |
| unnamed-faq-97 |
| ============== |
| |
| To: Sumanth Kamenani <skamenan@crl.nmsu.edu> |
| Subject: Re: Error |
| In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT. |
| Date: Tue, 20 Jul 1999 00:18:26 PDT |
| From: Vern Paxson <vern> |
| |
| > I am getting a compilation error. The error is given as "unknown symbol- yylex". |
| |
| The parser relies on calling yylex(), but you're instead using the C++ scanning |
| class, so you need to supply a yylex() "glue" function that calls an instance |
| scanner of the scanner (e.g., "scanner->yylex()"). |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-98, Next: unnamed-faq-99, Prev: unnamed-faq-97, Up: FAQ |
| |
| unnamed-faq-98 |
| ============== |
| |
| To: daniel@synchrods.synchrods.COM (Daniel Senderowicz) |
| Subject: Re: lex |
| In-reply-to: Your message of Mon, 22 Nov 1999 11:19:04 PST. |
| Date: Tue, 23 Nov 1999 15:54:30 PST |
| From: Vern Paxson <vern> |
| |
| Well, your problem is the |
| |
| switch (yybgin-yysvec-1) { /* witchcraft */ |
| |
| at the beginning of lex rules. "witchcraft" == "non-portable". It's |
| assuming knowledge of the AT&T lex's internal variables. |
| |
| For flex, you can probably do the equivalent using a switch on YYSTATE. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-99, Next: unnamed-faq-100, Prev: unnamed-faq-98, Up: FAQ |
| |
| unnamed-faq-99 |
| ============== |
| |
| To: archow@hss.hns.com |
| Subject: Re: Regarding distribution of flex and yacc based grammars |
| In-reply-to: Your message of Sun, 19 Dec 1999 17:50:24 +0530. |
| Date: Wed, 22 Dec 1999 01:56:24 PST |
| From: Vern Paxson <vern> |
| |
| > When we provide the customer with an object code distribution, is it |
| > necessary for us to provide source |
| > for the generated C files from flex and bison since they are generated by |
| > flex and bison ? |
| |
| For flex, no. I don't know what the current state of this is for bison. |
| |
| > Also, is there any requrirement for us to neccessarily provide source for |
| > the grammar files which are fed into flex and bison ? |
| |
| Again, for flex, no. |
| |
| See the file "COPYING" in the flex distribution for the legalese. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-100, Next: unnamed-faq-101, Prev: unnamed-faq-99, Up: FAQ |
| |
| unnamed-faq-100 |
| =============== |
| |
| To: Martin Gallwey <gallweym@hyperion.moe.ul.ie> |
| Subject: Re: Flex, and self referencing rules |
| In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST. |
| Date: Sat, 19 Feb 2000 18:33:16 PST |
| From: Vern Paxson <vern> |
| |
| > However, I do not use unput anywhere. I do use self-referencing |
| > rules like this: |
| > |
| > UnaryExpr ({UnionExpr})|("-"{UnaryExpr}) |
| |
| You can't do this - flex is *not* a parser like yacc (which does indeed |
| allow recursion), it is a scanner that's confined to regular expressions. |
| |
| Vern |
| |
| |
| File: flex.info, Node: unnamed-faq-101, Next: What is the difference between YYLEX_PARAM and YY_DECL?, Prev: unnamed-faq-100, Up: FAQ |
| |
| unnamed-faq-101 |
| =============== |
| |
| To: slg3@lehigh.edu (SAMUEL L. GULDEN) |
| Subject: Re: Flex problem |
| In-reply-to: Your message of Thu, 02 Mar 2000 12:29:04 PST. |
| Date: Thu, 02 Mar 2000 23:00:46 PST |
| From: Vern Paxson <vern> |
| |
| If this is exactly your program: |
| |
| > digit [0-9] |
| > digits {digit}+ |
| > whitespace [ \t\n]+ |
| > |
| > %% |
| > "[" { printf("open_brac\n");} |
| > "]" { printf("close_brac\n");} |
| > "+" { printf("addop\n");} |
| > "*" { printf("multop\n");} |
| > {digits} { printf("NUMBER = %s\n", yytext);} |
| > whitespace ; |
| |
| then the problem is that the last rule needs to be "{whitespace}" ! |
| |
| Vern |
| |
| |
| File: flex.info, Node: What is the difference between YYLEX_PARAM and YY_DECL?, Next: Why do I get "conflicting types for yylex" error?, Prev: unnamed-faq-101, Up: FAQ |
| |
| What is the difference between YYLEX_PARAM and YY_DECL? |
| ======================================================= |
| |
| YYLEX_PARAM is not a flex symbol. It is for Bison. It tells Bison to |
| pass extra params when it calls yylex() from the parser. |
| |
| YY_DECL is the Flex declaration of yylex. The default is similar to |
| this: |
| |
| #define int yy_lex () |
| |
| |
| File: flex.info, Node: Why do I get "conflicting types for yylex" error?, Next: How do I access the values set in a Flex action from within a Bison action?, Prev: What is the difference between YYLEX_PARAM and YY_DECL?, Up: FAQ |
| |
| Why do I get "conflicting types for yylex" error? |
| ================================================= |
| |
| This is a compiler error regarding a generated Bison parser, not a Flex |
| scanner. It means you need a prototype of yylex() in the top of the |
| Bison file. Be sure the prototype matches YY_DECL. |
| |
| |
| File: flex.info, Node: How do I access the values set in a Flex action from within a Bison action?, Prev: Why do I get "conflicting types for yylex" error?, Up: FAQ |
| |
| How do I access the values set in a Flex action from within a Bison action? |
| =========================================================================== |
| |
| With $1, $2, $3, etc. These are called "Semantic Values" in the Bison |
| manual. See *note Top: (bison)Top. |
| |
| |
| File: flex.info, Node: Appendices, Next: Indices, Prev: FAQ, Up: Top |
| |
| Appendix A Appendices |
| ********************* |
| |
| * Menu: |
| |
| * Makefiles and Flex:: |
| * Bison Bridge:: |
| * M4 Dependency:: |
| * Common Patterns:: |
| |
| |
| File: flex.info, Node: Makefiles and Flex, Next: Bison Bridge, Prev: Appendices, Up: Appendices |
| |
| A.1 Makefiles and Flex |
| ====================== |
| |
| In this appendix, we provide tips for writing Makefiles to build your |
| scanners. |
| |
| In a traditional build environment, we say that the `.c' files are |
| the sources, and the `.o' files are the intermediate files. When using |
| `flex', however, the `.l' files are the sources, and the generated `.c' |
| files (along with the `.o' files) are the intermediate files. This |
| requires you to carefully plan your Makefile. |
| |
| Modern `make' programs understand that `foo.l' is intended to |
| generate `lex.yy.c' or `foo.c', and will behave accordingly(1)(2). The |
| following Makefile does not explicitly instruct `make' how to build |
| `foo.c' from `foo.l'. Instead, it relies on the implicit rules of the |
| `make' program to build the intermediate file, `scan.c': |
| |
| # Basic Makefile -- relies on implicit rules |
| # Creates "myprogram" from "scan.l" and "myprogram.c" |
| # |
| LEX=flex |
| myprogram: scan.o myprogram.o |
| scan.o: scan.l |
| |
| For simple cases, the above may be sufficient. For other cases, you |
| may have to explicitly instruct `make' how to build your scanner. The |
| following is an example of a Makefile containing explicit rules: |
| |
| # Basic Makefile -- provides explicit rules |
| # Creates "myprogram" from "scan.l" and "myprogram.c" |
| # |
| LEX=flex |
| myprogram: scan.o myprogram.o |
| $(CC) -o $@ $(LDFLAGS) $^ |
| |
| myprogram.o: myprogram.c |
| $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^ |
| |
| scan.o: scan.c |
| $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^ |
| |
| scan.c: scan.l |
| $(LEX) $(LFLAGS) -o $@ $^ |
| |
| clean: |
| $(RM) *.o scan.c |
| |
| Notice in the above example that `scan.c' is in the `clean' target. |
| This is because we consider the file `scan.c' to be an intermediate |
| file. |
| |
| Finally, we provide a realistic example of a `flex' scanner used |
| with a `bison' parser(3). There is a tricky problem we have to deal |
| with. Since a `flex' scanner will typically include a header file |
| (e.g., `y.tab.h') generated by the parser, we need to be sure that the |
| header file is generated BEFORE the scanner is compiled. We handle this |
| case in the following example: |
| |
| # Makefile example -- scanner and parser. |
| # Creates "myprogram" from "scan.l", "parse.y", and "myprogram.c" |
| # |
| LEX = flex |
| YACC = bison -y |
| YFLAGS = -d |
| objects = scan.o parse.o myprogram.o |
| |
| myprogram: $(objects) |
| scan.o: scan.l parse.c |
| parse.o: parse.y |
| myprogram.o: myprogram.c |
| |
| In the above example, notice the line, |
| |
| scan.o: scan.l parse.c |
| |
| , which lists the file `parse.c' (the generated parser) as a |
| dependency of `scan.o'. We want to ensure that the parser is created |
| before the scanner is compiled, and the above line seems to do the |
| trick. Feel free to experiment with your specific implementation of |
| `make'. |
| |
| For more details on writing Makefiles, see *note Top: (make)Top. |
| |
| ---------- Footnotes ---------- |
| |
| (1) GNU `make' and GNU `automake' are two such programs that provide |
| implicit rules for flex-generated scanners. |
| |
| (2) GNU `automake' may generate code to execute flex in |
| lex-compatible mode, or to stdout. If this is not what you want, then |
| you should provide an explicit rule in your Makefile.am |
| |
| (3) This example also applies to yacc parsers. |
| |
| |
| File: flex.info, Node: Bison Bridge, Next: M4 Dependency, Prev: Makefiles and Flex, Up: Appendices |
| |
| A.2 C Scanners with Bison Parsers |
| ================================= |
| |
| This section describes the `flex' features useful when integrating |
| `flex' with `GNU bison'(1). Skip this section if you are not using |
| `bison' with your scanner. Here we discuss only the `flex' half of the |
| `flex' and `bison' pair. We do not discuss `bison' in any detail. For |
| more information about generating `bison' parsers, see *note Top: |
| (bison)Top. |
| |
| A compatible `bison' scanner is generated by declaring `%option |
| bison-bridge' or by supplying `--bison-bridge' when invoking `flex' |
| from the command line. This instructs `flex' that the macro `yylval' |
| may be used. The data type for `yylval', `YYSTYPE', is typically |
| defined in a header file, included in section 1 of the `flex' input |
| file. For a list of functions and macros available, *Note |
| bison-functions::. |
| |
| The declaration of yylex becomes, |
| |
| int yylex ( YYSTYPE * lvalp, yyscan_t scanner ); |
| |
| If `%option bison-locations' is specified, then the declaration |
| becomes, |
| |
| int yylex ( YYSTYPE * lvalp, YYLTYPE * llocp, yyscan_t scanner ); |
| |
| Note that the macros `yylval' and `yylloc' evaluate to pointers. |
| Support for `yylloc' is optional in `bison', so it is optional in |
| `flex' as well. The following is an example of a `flex' scanner that is |
| compatible with `bison'. |
| |
| /* Scanner for "C" assignment statements... sort of. */ |
| %{ |
| #include "y.tab.h" /* Generated by bison. */ |
| %} |
| |
| %option bison-bridge bison-locations |
| % |
| |
| [[:digit:]]+ { yylval->num = atoi(yytext); return NUMBER;} |
| [[:alnum:]]+ { yylval->str = strdup(yytext); return STRING;} |
| "="|";" { return yytext[0];} |
| . {} |
| % |
| |
| As you can see, there really is no magic here. We just use `yylval' |
| as we would any other variable. The data type of `yylval' is generated |
| by `bison', and included in the file `y.tab.h'. Here is the |
| corresponding `bison' parser: |
| |
| /* Parser to convert "C" assignments to lisp. */ |
| %{ |
| /* Pass the argument to yyparse through to yylex. */ |
| #define YYPARSE_PARAM scanner |
| #define YYLEX_PARAM scanner |
| %} |
| %locations |
| %pure_parser |
| %union { |
| int num; |
| char* str; |
| } |
| %token <str> STRING |
| %token <num> NUMBER |
| %% |
| assignment: |
| STRING '=' NUMBER ';' { |
| printf( "(setf %s %d)", $1, $3 ); |
| } |
| ; |
| |
| ---------- Footnotes ---------- |
| |
| (1) The features described here are purely optional, and are by no |
| means the only way to use flex with bison. We merely provide some glue |
| to ease development of your parser-scanner pair. |
| |
| |
| File: flex.info, Node: M4 Dependency, Next: Common Patterns, Prev: Bison Bridge, Up: Appendices |
| |
| A.3 M4 Dependency |
| ================= |
| |
| The macro processor `m4'(1) must be installed wherever flex is |
| installed. `flex' invokes `m4', found by searching the directories in |
| the `PATH' environment variable. Any code you place in section 1 or in |
| the actions will be sent through m4. Please follow these rules to |
| protect your code from unwanted `m4' processing. |
| |
| * Do not use symbols that begin with, `m4_', such as, `m4_define', |
| or `m4_include', since those are reserved for `m4' macro names. If |
| for some reason you need m4_ as a prefix, use a preprocessor |
| #define to get your symbol past m4 unmangled. |
| |
| * Do not use the strings `[[' or `]]' anywhere in your code. The |
| former is not valid in C, except within comments and strings, but |
| the latter is valid in code such as `x[y[z]]'. The solution is |
| simple. To get the literal string `"]]"', use `"]""]"'. To get the |
| array notation `x[y[z]]', use `x[y[z] ]'. Flex will attempt to |
| detect these sequences in user code, and escape them. However, |
| it's best to avoid this complexity where possible, by removing |
| such sequences from your code. |
| |
| |
| `m4' is only required at the time you run `flex'. The generated |
| scanner is ordinary C or C++, and does _not_ require `m4'. |
| |
| ---------- Footnotes ---------- |
| |
| (1) The use of m4 is subject to change in future revisions of flex. |
| It is not part of the public API of flex. Do not depend on it. |
| |
| |
| File: flex.info, Node: Common Patterns, Prev: M4 Dependency, Up: Appendices |
| |
| A.4 Common Patterns |
| =================== |
| |
| This appendix provides examples of common regular expressions you might |
| use in your scanner. |
| |
| * Menu: |
| |
| * Numbers:: |
| * Identifiers:: |
| * Quoted Constructs:: |
| * Addresses:: |
| |
| |
| File: flex.info, Node: Numbers, Next: Identifiers, Up: Common Patterns |
| |
| A.4.1 Numbers |
| ------------- |
| |
| C99 decimal constant |
| `([[:digit:]]{-}[0])[[:digit:]]*' |
| |
| C99 hexadecimal constant |
| `0[xX][[:xdigit:]]+' |
| |
| C99 octal constant |
| `0[01234567]*' |
| |
| C99 floating point constant |
| {dseq} ([[:digit:]]+) |
| {dseq_opt} ([[:digit:]]*) |
| {frac} (({dseq_opt}"."{dseq})|{dseq}".") |
| {exp} ([eE][+-]?{dseq}) |
| {exp_opt} ({exp}?) |
| {fsuff} [flFL] |
| {fsuff_opt} ({fsuff}?) |
| {hpref} (0[xX]) |
| {hdseq} ([[:xdigit:]]+) |
| {hdseq_opt} ([[:xdigit:]]*) |
| {hfrac} (({hdseq_opt}"."{hdseq})|({hdseq}".")) |
| {bexp} ([pP][+-]?{dseq}) |
| {dfc} (({frac}{exp_opt}{fsuff_opt})|({dseq}{exp}{fsuff_opt})) |
| {hfc} (({hpref}{hfrac}{bexp}{fsuff_opt})|({hpref}{hdseq}{bexp}{fsuff_opt})) |
| |
| {c99_floating_point_constant} ({dfc}|{hfc}) |
| |
| See C99 section 6.4.4.2 for the gory details. |
| |
| |
| |
| File: flex.info, Node: Identifiers, Next: Quoted Constructs, Prev: Numbers, Up: Common Patterns |
| |
| A.4.2 Identifiers |
| ----------------- |
| |
| C99 Identifier |
| ucn ((\\u([[:xdigit:]]{4}))|(\\U([[:xdigit:]]{8}))) |
| nondigit [_[:alpha:]] |
| c99_id ([_[:alpha:]]|{ucn})([_[:alnum:]]|{ucn})* |
| |
| Technically, the above pattern does not encompass all possible C99 |
| identifiers, since C99 allows for "implementation-defined" |
| characters. In practice, C compilers follow the above pattern, |
| with the addition of the `$' character. |
| |
| UTF-8 Encoded Unicode Code Point |
| [\x09\x0A\x0D\x20-\x7E]|[\xC2-\xDF][\x80-\xBF]|\xE0[\xA0-\xBF][\x80-\xBF]|[\xE1-\xEC\xEE\xEF]([\x80-\xBF]{2})|\xED[\x80-\x9F][\x80-\xBF]|\xF0[\x90-\xBF]([\x80-\xBF]{2})|[\xF1-\xF3]([\x80-\xBF]{3})|\xF4[\x80-\x8F]([\x80-\xBF]{2}) |
| |
| |
| |
| File: flex.info, Node: Quoted Constructs, Next: Addresses, Prev: Identifiers, Up: Common Patterns |
| |
| A.4.3 Quoted Constructs |
| ----------------------- |
| |
| C99 String Literal |
| `L?\"([^\"\\\n]|(\\['\"?\\abfnrtv])|(\\([0123456]{1,3}))|(\\x[[:xdigit:]]+)|(\\u([[:xdigit:]]{4}))|(\\U([[:xdigit:]]{8})))*\"' |
| |
| C99 Comment |
| `("/*"([^*]|"*"[^/])*"*/")|("/"(\\\n)*"/"[^\n]*)' |
| |
| Note that in C99, a `//'-style comment may be split across lines, |
| and, contrary to popular belief, does not include the trailing |
| `\n' character. |
| |
| A better way to scan `/* */' comments is by line, rather than |
| matching possibly huge comments all at once. This will allow you |
| to scan comments of unlimited length, as long as line breaks |
| appear at sane intervals. This is also more efficient when used |
| with automatic line number processing. *Note option-yylineno::. |
| |
| <INITIAL>{ |
| "/*" BEGIN(COMMENT); |
| } |
| <COMMENT>{ |
| "*/" BEGIN(0); |
| [^*\n]+ ; |
| "*"[^/] ; |
| \n ; |
| } |
| |
| |
| |
| File: flex.info, Node: Addresses, Prev: Quoted Constructs, Up: Common Patterns |
| |
| A.4.4 Addresses |
| --------------- |
| |
| IPv4 Address |
| dec-octet [0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5] |
| IPv4address {dec-octet}\.{dec-octet}\.{dec-octet}\.{dec-octet} |
| |
| IPv6 Address |
| h16 [0-9A-Fa-f]{1,4} |
| ls32 {h16}:{h16}|{IPv4address} |
| IPv6address ({h16}:){6}{ls32}| |
| ::({h16}:){5}{ls32}| |
| ({h16})?::({h16}:){4}{ls32}| |
| (({h16}:){0,1}{h16})?::({h16}:){3}{ls32}| |
| (({h16}:){0,2}{h16})?::({h16}:){2}{ls32}| |
| (({h16}:){0,3}{h16})?::{h16}:{ls32}| |
| (({h16}:){0,4}{h16})?::{ls32}| |
| (({h16}:){0,5}{h16})?::{h16}| |
| (({h16}:){0,6}{h16})?:: |
| |
| See RFC 2373 (http://www.ietf.org/rfc/rfc2373.txt) for details. |
| Note that you have to fold the definition of `IPv6address' into one |
| line and that it also matches the "unspecified address" "::". |
| |
| URI |
| `(([^:/?#]+):)?("//"([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?' |
| |
| This pattern is nearly useless, since it allows just about any |
| character to appear in a URI, including spaces and control |
| characters. See RFC 2396 (http://www.ietf.org/rfc/rfc2396.txt) |
| for details. |
| |
| |
| |
| File: flex.info, Node: Indices, Prev: Appendices, Up: Top |
| |
| Indices |
| ******* |
| |
| * Menu: |
| |
| * Concept Index:: |
| * Index of Functions and Macros:: |
| * Index of Variables:: |
| * Index of Data Types:: |
| * Index of Hooks:: |
| * Index of Scanner Options:: |
| |