Tools for Lexical Analysis


5th lecture, May 6, 2016

Language Theory and Compilers

Martin J. Dürst


© 2005-16 Martin J. Dürst 青山学院大学

Today's Schedule


Summary up to Now

These all have the same power, describe/recognize regular languages, and can be converted into each other.


Compilation Stages

  1. Lexical analysis
  2. Parsing (syntax analysis)
  3. Semantic analysis
  4. Optimization (or 5)
  5. Code generation (or 4)


Compiler Structure


Implementing Lexical Analysis

Main points:



Automating Lexical Analysis


Overview of flex


How to Use Cygwin



Cygwin and Harddisks



How to Use flex

  1. Create an input file for flex (a (f)lex file), with the extension .l (example: test.l)
  2. Use flex to convert test.l to a C program:
    $ flex test.l
    (the output file is named lex.yy.c)
  3. Compile lex.yy.c with a C compiler (maybe together with other files)
  4. Execute the compiled program


Two Ways to Use flex

  1. Independent file processing (use regular expressions to recognize or change parts of a file):

    Call the yylex() function once from the main function

  2. Calling the lexical analyzer from the parser:

    Repeatedly call yylex() from the parser, and return a token with return

In today's exercises and homework, we will use 1.

In the second half of this cours, we will use 2. together with bison


Example of flex Input Format

        int num_lines = 0, num_chars = 0;
\n      ++num_lines; ++num_chars;
 .       ++num_chars;
%% main() { yylex(); printf( "# of lines = %d, # of chars = %d\n", num_lines, num_chars ); } int yywrap () { return 1; }


flex Exercise 1

Process and execute the flex program on the previous slide

  1. Create a file test.l and copy the contents of the previous slide to the file
  2. Create the file lex.yy.c with
    $ flex test.l
  3. Create the executable file a.exe with
    $ gcc lex.yy.c
  4. Execute the program with some input from standard input
    $ ./a <file


The Skeleton of a flex Input File

declarations,... (C program language)
declarations,... (C program language)
regexp statement (C program language)
regexp statement (C program language)
functions,... (C program language)
functions,... (C program language)


Structure of flex Input

Mixture of flex-specific instructions and C program fragments

Three main parts, separated by two %%:

  1. Preparation/setup part:
    C #includes, definition and initialization of global variables, definition of regular expression components
  2. Regular expressions to be recognized (lexical rules), and the program fragments executed on recognition
  3. Rest of C program (functions,...)

Newlines and indent can be significant!


How to Study flex


Basic Behavior of flex


How flex Processes its Input


flex Exercise 2

The table below shows how to escape various characters in XML
Create a program in flex for this conversion, and for the reverse conversion

Raw text XML escapes
' &apos;
" &quot;
& &amp;
< &lt;
> &gt;


flex Exercise 3: Detect Numbers

Create a program with flex to output the input without changes, except that numbers are enclosed with >>> and <<<

Example input:


Example output:


Hint: The string recognized by a regular expression is available with the variable yytext


flex Exercise 4 (Homework):
General Rules

Deadline: May 19, 2016 (Thursday), 19:00

Where to submit: Box in front of room O-529 (building O, 5th floor)


Collaboration: The same rules as for Computer Practice I apply


flex Exercise 4 (Homework):
Lexical Analysis for C

Using flex, Create a program for lexical analysis of C programs. Output one token per line.

Process the following tokens:


flex Exercise 4 (Homework):
Example Input and Output

Simple example input:

if (xyz*3 > 5) abc = 'c';

Example output:

keyword: if
parenthesis: (
identifier: xyz
operator: *
integer constant: 3
operator: <
integer constant: 5
parenthesis: )
identifier: abc
operator: =
character constant: 'c'
semicolon: ;

More examples: input, output, input for bonus problem, output for bonus problem


Frequent Problems with flex


Hints for Homework



There will be a minitest (ca. 30 minutes) next week. Please prepare well.



lexical analyzer
lexical analyzer generator
字句解析器生成系 (生成器)
parser generator
構文解析器生成系 (生成器)
integer literal
character literal