Principles of Bottom-Up Parsing

(上向き構文解析の原理)

9th lecture, June 14, 2019

Language Theory and Compilers

http://www.sw.it.aoyama.ac.jp/2019/Compiler/lecture9.html

Martin J. Dürst

Today's Schedule

Summary and homework from last lecture
Left recursion and EBNF
Limits of top-down parsing
Bottom-up parsing
bison

Summary of Last Lecture

General top-down parsing: Backtracking
Recursive descent parsing
- Concept: A function for each nonterminal symbol
- Main problem: Left recursion

Last Week's Homework 1

都合により削除

Last Week's Homework 2

都合により削除

Problems with Top-Down Parsing

For efficient parsing, the grammar has to be restricted

Grammars where lookahead is necessary are difficult to deal with
Example:

Factor → FunctionCall | Variable | ArrayElement
FunctionCall → Identifier '(' Parameters ')'
Variable → Identifier
ArrayElement → Identifier '[' Expression ']'

(it is unclear which rule should be selected in Factor when the next token is Identifier)

Possible solutions: Longer lookahead, grammar rewriting

Additional Requirements for Grammars

Depending on parsing method, additional requirements become necessary:

Efficiency
Recognition power
Guarantee to terminate (example: no left recursion)
No ambiguity

⇒ A grammar that just produces/recognizes 'words' is not enough

Limits of Top-Down Parsing

With recursive descent parsing, it is easily possible to convert grammar rules to a parser by hand
However, this requires changes and limitations to the grammar
A parsing method with less limitations is desirable
Automatic implementation is desirable

⇒ Bottom-up parsing

Top-Down and Bottom-Up Parsing

Parsing Direction	Top-Down	Bottom-Up
Start	top (start symbol)	bottom (terminal symbols)
Parse tree(s)	single tree, some branches incomplete	multiple small trees
General method	backtracking	dynamic programming (CYK algorithm)
Practical method	recursive descent	LR parsing
Grammar type	(E)BNF (incl. repetitions)	no explicit repetitions
Tools	usually by hand; ANTLR,...	yacc, bison,...
Main problem	left recursion	shift/... conflicts

`bison` Overview

yacc: Yet Another Compiler Compiler (1975)
Became popular together with Unix
Compiler Compiler: A compiler that constructs a compiler
Yet Another: one more
(it was very popular to create compiler compilers, and difficult to find a good name)
(due to yacc, names starting with YA.. became widely used)
bison is the gnu free software version of yacc

Exercise: A Simple Pocket Calculator

Files to start: makefile, calc.y, calc.lex

Use of `make`

It is difficult not to forget flex, bison, and gcc
The make command follows the instructions in the makefile, executing a minimum of commants to bring everything up to date
If necessary, please add the make command to your cygwin installation
Syntax of makefile (→: tabulator; other whitespace characters will not work!):
target: input1 input2 input3 ... →command to construct target
Running make without arguments will construct the first target in the makefile
Running make target will construct target
Caution: The file name has to be exactly makefile, without extension
(makefile.txt will not work!)

Example of `bison` Input Format

%{
#include <stdio.h>
#define YYSTYPE double
int yylex (void);
void yyerror (char const *);
%}
%token NUM PLUS

%%
statement: exp { printf ("Result is %g\n", $1); }
;

exp:       exp PLUS exp { $$ = $1 + $3; }
         | NUM { $$ = $1; }
;

%%
int main (void)
{    return yyparse ();  }
     
void yyerror (char const *s)
{    fprintf (stderr, "%s\n", s);  }

Structure of `bison` Input Format

bison 関係の宣言など
%{ C 言語の宣言など %}
%%
書換規則    { 実行文 (C 言語) }
書換規則    { 実行文 (C 言語) }
書換規則    { 実行文 (C 言語) }
%%
関数など (C 言語)
関数など (C 言語)

Details of `bison` Input Format

Mixture of bison-specific directives and C program fragments

There are three main parts, separated by %%:

Preparation/settings:
Settings for using bison
C #include statements, definition/initialization of global variables,...
C parts have to be surrounded by %{ ... %}
Rewriting rules and program fragments (in { ... }) that get executed when a rule is matched
(the first nonterminal symbol is the start symbol)
Rest of C program (functions,...)

Newlines and indentation can be significant

Details of `bison` Rewriting Rules

exp: exp PLUS exp { $$ = $1 + $3; }

   | NUM { $$ = $1; }

Left hand side (single nonterminal) starts at start of line, followed by ':' (instead of →)
Right hand sides for the same left hand side separated by '|', terminated by ';'
Each right hand side (sequence of nonterminals and terminals (tokens)) may be followed by a C program fragment
The C program fragments make the grammar an attributed grammar
By default, the start symbol is the left-hand side of the first rewriting rule
In general, the order of left-hand sides and right-hand sides is not significant
Exceptions: Start symbol, shift-shift conflict

Attribute(d) Grammar

Simple parsing will only accept or reject the input
Attribute grammars add attributes to each (non)terminal symbol
For each rewriting rule, rules for how to calculate attribute values are defined
Order of calculating attribute values:
Usually: bottom-up
A combination of both directions is also possible
Example rewriting rule: E₀ → E₁ '-' E₃
Rule for calculating attribute values:
S(E₀) = S(E₁) - S(E₃)
Notation in bison: $$ = $1 - $3
Typical attribute types:
- Hight of tree, length of input
- Result of expression evaluation
- Partial parse tree (or syntax tree)
- Generated code

`bison` Manual

English:
http://www.gnu.org/software/bison/manual/
Japanese:
http://guppy.eng.kagawa-u.ac.jp/2019/Compiler/bison-1.2.8/bison-ja_toc.html

Hints for Developping `bison` Programs

Prepare a very simple input file for testing
Prepare a correct output file for testing
Automatically compile/execute the program and compare the output using the diff command
If there is no output from diff, the test is successful
Test First: Add a new test, check that it fails, then work on making the test successful

Homework

Deadline: June 20, 2019 (Thursday), 19:00

Where to submit: Box in front of room O-529 (building O, 5th floor)

Format: A4 double-sided printout of parser program. Stapled in upper left if more than one page, no cover page, no wrapping lines, legible font size, non-proportional font, portrait (not landscape), formatted (indents,...) for easy visibility, name (kanji and kana) and student number as a comment at the top right

Collaboration: The same rules as for Computer Practice I (計算機実習 I) apply

Complete calc.y so that it can be used as a calculator for the four basic arithmetic operations (+-*/), including prefix minus for negative numbers and parentheses for grouping. See test.in example input, and test.check example output. Use the grammar to define priorities and associativities (do NOT use %left, %right,...); submit calc.y only.

Hint: If there are differences with newlines, make sure that all files use Unix line ending convention
(In Notepad2, choose File → Line Endings → Unix (LF))

Glossary

factor: 因子
tabulator (tab character): タブ文字
attribute(d) grammar: 属性文法