Tools for Lexical Analysis

(字句解析ツール)

5th lecture, May 19, 2023

Language Theory and Compilers

https://www.sw.it.aoyama.ac.jp/2023/Compiler/lecture5.html

Martin J. Dürst

© 2005-23 Martin J. Dürst 青山学院大学

Today's Schedule

• Homework from last lecture
• Leftovers from last lecture
• flex example
• Automating lexical analysis
• Overview of flex
• How to use cygwin
• flex exercises

Last Week's Homework

1. Construct the state transition diagram for the NFA corresponding to the following grammar
   S → xB | yA | yC, A → xC | z | yS, B → zA | zC | xB | y, C →yA | aD | z
   

2. Convert the automaton defined by the following transition table to a right linear grammar

   	a	b
   →T	G	M
   *G	K	H
   H	L	G
   K	H	-
   *L	M	T
   M	L	G

    
   Result:
   

   T → aG | bM | a
   G → aK | bH
   H → aL | bG | a | b
   K → aH
   L → aM | bT
   M → aL | bG | a | b

3. Construct the state transition diagram for the regular expression pr|x*t
   Write down two versions:
   1. The result of the full procedure (with all ε transitions)
   2. A version that is as simple as possible
4. Bring your notebook PC with you next lecture (May 19).
   Make sure you can use flex, bison, gcc, make, diff, and m4 (no need to submit)

Leftovers from Last Lecture

Practical Regular Expressions

Summary for Regular Languages

We learned about:

• (Non-)Deterministic Finite Automata
• Right (/Left) Linear Grammars
• Regular Expressions

These all:

• Can be converted into each other
• Have the same power
  
• Describe/recognize regular languages
  

Compilation Stages

1. Lexical analysis
2. Parsing (syntax analysis)
3. Semantic analysis
4. Optimization (or 5)
5. Code generation (or 4)

Compiler Structure

• Parsing is the core of the front end (analysis)
  or of the whole compiler
• The parser repeatedly obtains the next token from the lexical analyzer
  (with a function such as getNextToken())
• The parser calls semantic analysis,...
  when needed

Implementing Lexical Analysis

• Lexical analysis is the first stage of a compiler
• Analyzing/extracting the "words" (tokens) of a programming language
• Processing is character by character
  → speed/efficiency is important
• Structure of tokens is simple
  → Regular language is powerful enough
• Efficient implementation is possible
  (regular expressions → NFA → DFA
  → minimized DFA)

Choices for Implementing
Lexical Analysis

• Write by hand
  (tedious and error-prone)
• Use a tool
  to automate creation of lexical analyzer
  (example: flex)

Example of flex Input

    int num_lines = 0, num_chars = 0;
%%
\n  ++num_lines; ++num_chars;
 .   ++num_chars;

%%
int main(void)
{
    yylex();
    printf( "%d lines, %d characters\n",
        num_lines, num_chars );
}

int yywrap () { return 1; }

flex Exercise 1

Process the flex program on the previous slide using cygwin

1. Download and save the file test.l
   
2. Create lex.yy.c with
   $ flex test.l
3. Create the executable a.exe with
   $ gcc lex.yy.c
4. Execute the program with input from stdin
   $ ./a <file

Overview of flex

• Lexical analyzer generator
• Opensource version of lex,
  with various extensions
• lex: Lexical analyzer generator
  available with Unix (creator: Mike Lesk)
• Easy to write/create lexical analyzers
  e.g. for compilers
• Works well with parser generator bison

How to Use Cygwin

• ls: list the files in a directory
• mkdir: create a new directory
• cd: change (working) directory
• pwd: print (current) working directory
• gcc: compile a C program
• ./a: execute a compiled program (a.exe)

Cygwin and Harddisks

• Assumption: cygwin is installed in C:\cygwin
• Usually, only directories below C:\cygwin
  can be reached
• Home directory of user1
  is C:\cygwin\home\user1
• This can be displayed as /home/user1
  with pwd
• Change directory to Windows C drive:
  cd /cygdrive/c

flex Usage Steps

1. Create an input file for flex (a (f)lex file),
   with the extension .l (example: test.l)
2. Use flex to convert test.l to a C program:
   $ flex test.l
   (the output file is named lex.yy.c)
3. Compile lex.yy.c with a C compiler, e.g. gcc
   (maybe together with other files)
4. Execute the compiled program

Two Methods to Use flex

1. Independent file processing (use regular expressions to recognize/change parts of a file):

   Call the yylex() function
   once from the main function

2. Combination with parser:

   Repeatedly call yylex() from the parser,
   and return a token with return

In today's exercises and homework, we use method 1.

In later lectures, we will use method 2 together with bison.

Example of flex Input

    int num_lines = 0, num_chars = 0;
%%
\n  ++num_lines; ++num_chars;
 .   ++num_chars;

%%
int main(void)
{
    yylex();
    printf("%d lines, %d characters\n",
        num_lines, num_chars );
}

int yywrap () { return 1; }

Skeleton of flex Input Format

declarations,... (C program language)
declarations,... (C program language) 
%%
regexp    statement (C program language)
regexp    statement (C program language)
%%
functions,... (C program language)
functions,... (C program language)

Structure of flex Input Format

Mixture of flex-specific instructions and C program fragments

Three main parts, separated by two %%:

1. Preparation/setup part:
   • C #includes, #defines
   • Definition and initialization of global variables
   • Definition of regular expression components
2. Flex rules:
   • Left side: Regular expressions to be recognized (lexical rules)
   • Right side: Program fragments
     executed on recognition
3. Rest of C program (functions,...)

Newlines and indent can be significant!

How to Study flex

• Read manual: English, Japanese
  For homework, use of manual is necessary

  Caution: Manual, not novel
  Caution: Old, but good enough for homework

• Read the output of flex (lex.yy.c)
• Compare the output of flex for different inputs
• Read flex source code (flex itself uses
  lexical analysis, written with flex)
• Use options to output internal information (example: flex -v (verbose))

How the Program
Created by flex Works

• Read the input character by character
• Match the longest possible input string with a regular expression
• If multiple regular expressions match
  the same input string,
  use the earliest regular expression in the .l file
• Always remember the last accepting state passed and the input up to that state; use this state/input when there is no next state
• If a match is found, execute
  the corresponding C fragment
• If no match is found, output the first input character
• Repeat starting with the first character after a match or the next character after a non-match

How flex Works

• Convert each regular expression to an NFA,
  and associate each accepting state with the corresponding C fragment
• Combine all NFAs into a single large NFA
  (if multiple C fragments are available
  at an accepting state,
  use the earliest fragment in the .l file)
• Convert this NFA to a DFA
• Minimize the DFA
• Create and initialize the necessary tables
• Copy the program that executes the DFA
• Copy the C fragments in the .l file

flex Exercise 2

The table below shows how to
escape various characters in XML
Create a program in flex
for this conversion and for the reverse conversion

flex Exercise 3: Detect Numbers

Create a program with flex to output the input without changes, except that numbers are enclosed with >*> and <*<

Example input: abc123def345gh

Example output: abc>*>123<*<def>*>345<*<gh

Hint: The string recognized by a regular expression is available with the variable yytext

flex Exercise 4 (Homework):
General Rules

Deadline: June 1, 2023 (Thursday), 22:00

Where to submit: Moodle

Important: This homework requires significantly more time than other homeworks.
Start early, so that you can ask questions on May 26 (Friday) and in Moodle (Q&A Forum)

Submission: flex input file (.l file), name (kanji and kana) and student number as a comment at the top
(make sure comment is in UTF-8, and processing works even after adding comment (use only /* */, not //))

Collaboration: The same rules as for
Projects in Information Technology II apply!

flex Exercise 4 (Homework):
Lexical Analysis for C

Using flex, Create a program for lexical analysis of C programs. Output one token per line.

Process the following tokens:

• Integer literal
• Identifiers
• Operators (can be limited to the 20 most frequently used)
• Keywords (can be limited to the 10 most frequently used)
• Semicolon, parentheses, brackets,...
• Character literals
• Character string literals (bonus problem)
• Floating point literals (bonus problem)
• Comments (bonus problem)

flex Exercise 4 (Homework):
Example Input and Output

Simple example input:

if (xyz*3 > 15) abc = 'c';

Example output:

keyword: if
parenthesis: (
identifier: xyz
operator: *
integer constant: 3
operator: <
integer constant: 15
parenthesis: )
identifier: abc
operator: =
character constant: 'c'
semicolon: ;

Frequent Problems with flex

• If there is an error in the .l file, flex may still run without errors

  Solution: Always start with flex:
  > flex file.l && gcc lex.yy.c && ./a <input.txt

• In the first part of the .l file (before the first %%), C program fragments have to be indented by at least one space
• Don't forget int yywrap () { return 1; }

Hints for Homework

• This homework will take much more time than the previous homeworks
• You can ask questions about this homework
  next week
• You can ask questions about this homework on Moodle (deadline: Wednesday, May 31, 22:00)
• You can assume that the input is only US-ASCII
• In some cases, the order of the regular expressions (rules) is important
• The text matched with a regular expression is available in the variable yytext
  Example: putchar(yytext[0]); will output the first character of the matched text
• Metacharacters in regular expressions have to be escaped with\ or quoted within ""

Announcement: Minitest

There will be a minitest (30 minutes) next week

Please prepare well!

Glossary

automate

自動化する

parser

構文解析器

lexical analyzer

字句解析器

lexical analyzer generator

字句解析器生成系 (生成器)

parser generator

構文解析器生成系 (生成器)

extension

拡張子

skeleton

骨格

definition

定義

initialization

初期化

integer literal

整数定数

character literal

文字定数