Tools for Lexical Analysis

(字句解析ツール)

5th lecture, May 6, 2016

Language Theory and Compilers

http://www.sw.it.aoyama.ac.jp/2016/Compiler/lecture5.html

Martin J. Dürst

© 2005-16 Martin J. Dürst 青山学院大学

Today's Schedule

• Homework from last lecture
• Remainders from last lecture
• Automating lexical analysis
• How to use cygwin
• Overview of flex
• flex exercises

Summary up to Now

• (Non-)Deterministic Finite Automata
• (Left|Right) Linear Grammars
• Regular Expressions

These all have the same power, describe/recognize regular languages, and can be converted into each other.

Compilation Stages

1. Lexical analysis
2. Parsing (syntax analysis)
3. Semantic analysis
4. Optimization (or 5)
5. Code generation (or 4)

Compiler Structure

• Parsing is the core of the front end (analysis) or of the whole compiler
• The parser repeatedly obtains the next token from the lexical analyzer
  (with a function such as getNextToken())
• The parser calls semantic analysis,... when needed

Implementing Lexical Analysis

Main points:

• Detailled processing for each character → speed/efficiency is important
• Using a buffer for actual input

Choices:

• Write a lexical analyzer by hand
• Use a tool to create a lexical analyzer

Automating Lexical Analysis

• Lexical analysis is the first stage of a compiler
• It analyzes and extracts the "words" of a programming language
• Regular expressions are best suited to describe lexical units
• An efficient implementation is possible by converting
  regular expressions to NFAs and then to DFAs
• Doing this by hand is very tedious and error-prone
  → automation

Overview of flex

• Lexical analyzer generator
• Opensource version of lex, with various extensions
• lex: Lexical analyzer generator available with Unix (creator: Mike Lesk)
• Easy to write/create lexical analyzers e.g. for compilers
• Works well with parser generator bison
• (do not confuse with Adobe Flex)

How to Use Cygwin

(reminder)

• ls: list the files in a directory
• mkdir: create a new directory
• cd: change (working) directory
• pwd: print (current) working directory
• gcc: compile a C program
• ./a: execute a compiled program

Cygwin and Harddisks

(reminder)

• Assumption: cygwin is installed in C:\cygwin
• Usually, only directories below C:\cygwin can be reached
• The user's home directory is e.g. C:\cygwin\home\user1
• This can be displayed as /home/user1 with pwd
• How to escape C:\cygwin:
  cd /cygdrive/c (change directory to the C drive of MS Windows)

How to Use flex

1. Create an input file for flex (a (f)lex file), with the extension .l (example: test.l)
2. Use flex to convert test.l to a C program:
   $ flex test.l
   (the output file is named lex.yy.c)
3. Compile lex.yy.c with a C compiler (maybe together with other files)
4. Execute the compiled program

Two Ways to Use flex

1. Independent file processing (use regular expressions to recognize or change parts of a file):

   Call the yylex() function once from the main function

2. Calling the lexical analyzer from the parser:

   Repeatedly call yylex() from the parser, and return a token with return

In today's exercises and homework, we will use 1.

In the second half of this cours, we will use 2. together with bison

Example of flex Input Format

        int num_lines = 0, num_chars = 0;
%%
\n      ++num_lines; ++num_chars;
 .       ++num_chars;

%%

main()
{
        yylex();
        printf( "# of lines = %d, # of chars = %d\n",
                num_lines, num_chars );
}

int yywrap () { return 1; }

flex Exercise 1

Process and execute the flex program on the previous slide

1. Create a file test.l and copy the contents of the previous slide to the file
2. Create the file lex.yy.c with
   $ flex test.l
3. Create the executable file a.exe with
   $ gcc lex.yy.c
4. Execute the program with some input from standard input
   $ ./a <file

The Skeleton of a flex Input File

declarations,... (C program language)
declarations,... (C program language) 
%%
regexp    statement (C program language)
regexp    statement (C program language)
%%
functions,... (C program language)
functions,... (C program language)

Structure of flex Input

Mixture of flex-specific instructions and C program fragments

Three main parts, separated by two %%:

1. Preparation/setup part:
   C #includes, definition and initialization of global variables, definition of regular expression components
2. Regular expressions to be recognized (lexical rules), and the program fragments executed on recognition
3. Rest of C program (functions,...)

Newlines and indent can be significant!

How to Study flex

• Read the manual: English, Japanese
  To complete the homework, the use of the manual is necessary

  Caution: A manual is not a novel.

• Read the output of flex (lex.yy.c)
• Compare the output of flex for different inputs
• Read the source code of flex (flex also uses lexical analyzis, which is written using flex)
• Use options to output internal information (example: flex -v)

Basic Behavior of flex

• When calling flex:
  • Process the regular expressions, and combine them into a single DFA
  • Create and initialize the necessary tables
  • Copy the program that executes the DFA
  • Copy the C fragments in the .l file
• When executing:
  • Read the input character by character
  • Match the longest possible input string with a regular expression
  • If multiple regular expressions match the same length input string, use the earliest in the .l file
  • If a match is found, execute the corresponding C fragment
  • If no match is found:
    • Output the first input character
    • Start processing again with the next input character

How flex Processes its Input

• Convert each regular expression to an NFA,
  and associate each accepting state with the corresponding C fragment
• Combine all NFAs into a single large NFA
• Convert this NFA to a DFA
• Minimize this DFA
• If multiple C fragments are available at an accepting state, keep only the one that comes earliest in the .l file
• When working on input, always remember the last accepting state passed and the input up to that state; use this state/input when there is no next state

flex Exercise 2

The table below shows how to escape various characters in XML
Create a program in flex for this conversion, and for the reverse conversion

flex Exercise 3: Detect Numbers

Create a program with flex to output the input without changes, except that numbers are enclosed with >>> and <<<

Example input:

abc123def345gh

Example output:

abc>>>123<<<def>>>345<<<gh

Hint: The string recognized by a regular expression is available with the variable yytext

flex Exercise 4 (Homework):
General Rules

Deadline: May 19, 2016 (Thursday), 19:00

Where to submit: Box in front of room O-529 (building O, 5th floor)

Format:

• A4 double-sided printout of .l filesingle page
  
• Stapled in upper if more than one page
• NO cover page
• NO wrapping lines
• Name (kanji and kana) and student number as a comment at the top right

Collaboration: The same rules as for Computer Practice I apply

flex Exercise 4 (Homework):
Lexical Analysis for C

Using flex, Create a program for lexical analysis of C programs. Output one token per line.

Process the following tokens:

• Integer literal
• Identifiers
• Operators (can be limited to the 20 most frequently used)
• Keywords (can be limited to the 10 most frequently used)
• Semicolon, parentheses, brackets,...
• Character literals
• Character string literals (bonus problem)
• Floating point literals (bonus problem)
• Comments (bonus problem)

flex Exercise 4 (Homework):
Example Input and Output

Simple example input:

if (xyz*3 > 5) abc = 'c';

Example output:

keyword: if
parenthesis: (
identifier: xyz
operator: *
integer constant: 3
operator: <
integer constant: 5
parenthesis: )
identifier: abc
operator: =
character constant: 'c'
semicolon: ;

More examples: input, output, input for bonus problem, output for bonus problem

Frequent Problems with flex

• If there is an error in the .l file, flex may still run without errors

  Solution: Always start with flex:
  > flex file.l && gcc lex.yy.c && ./a <input.txt

• In the first part of the .l file (before the first %%), C program fragments have to be indented by at least one space
• Don't forget int yywrap () { return 1; }

Hints for Homework

• This homework will take considerably more time than the previous homeworks
• You can ask questions about this homework next week
• You can assume that the input is restricted to US-ASCII
• In some cases, the order of the regular expressions (rules) is important
• The text matched with a regular expression is available in the variable yytext
  Example: putchar(yytext[0]); (output the first character of the matched text)
• Metacharacters in regular expressions have to be escaped with\ or quoted within ""

Announcement

There will be a minitest (ca. 30 minutes) next week. Please prepare well.

Glossary

automate

自動化する

parser

構文解析器

lexical analyzer

字句解析器

lexical analyzer generator

字句解析器生成系 (生成器)

parser generator

構文解析器生成系 (生成器)

extension

拡張子

skeleton

骨格

definition

定義

initialization

初期化

integer literal

整数定数

character literal

文字定数