Parser-5 : Getting tokens with lex

 In this example, we will see how to use lex as a lexical analyser that will return tokens, and how to get these tokens in a C program.

The program is borken into 3 source files:

tokens.h: Include file defining the mapping values for tokens.

tokens.lex: Lex file that defines rules for each token. This will generate the lexical analyser that will allow us to get tokens.

parser5.c: C program using the lexical analyser generated from tokens.lex to get tokens one by one.

More detailed explanations will be given below.

Returning tokens with LEX

 Lex, as we saw it in the first four examples (parser[1-4]), can be used as a lexical AND syntactical analyser. However, its primary function is to define (and generate) a lexical analyser that will get and return tokens one by one.

 Each time yylex() is called it continues processing tokens from where it last left off until it either reaches the end the file or executes a return. In the examples we have seen, we were processing the entire input before stopping. In this example, each time a token will be found, we will return a corresponding integer value. Doing this, each call of yylex() in the C program (parser5.c) will return the next token.
 

 In tokens.lex, we define rules for tokens and associate each one with the action of returning the integer value corresponding to the token.

 Each time yylex() will be called by the C program, it will return an integer value (corresponding to the token found) and stop. Then, if called again, it will get the next token, and return again the corresponding integer value, and so on...

Getting a token and its corresponding text

 When the lexical analyser is generated with flex, it produces a C file, lex.yy.c that contains several things including the following important ones:

 yylex(): a function that implements the lexical analyser and returns the next token and yytext: a global variable that contains the text of the last token scanned with yylex()

 yylex() is used in the C program to get the next token. If we want to be able to access the value of yytext, we have to tell the compiler that we want to access a variable that is externally defined (because defined in lex.yy.c). This is done in parser5.c by using extern yytext;