Principles of Bottom-Up Parsing

(上向き構文解析の原理)

10th lecture, June 23, 2023

Language Theory and Compilers

https://www.sw.it.aoyama.ac.jp/2023/Compiler/lecture10.html

Martin J. Dürst

© 2005-23 Martin J. Dürst 青山学院大学

Today's Schedule

• About last lecture's homework
• Summary of last lecture
• How to write a grammar: Priorities, associativity, repetition (lists), parentheses, unary operations, other constructs
• How bison works:
  • Orders of derivation
  • LALR parsing
  • How to debug bison grammars
  • How to read the .output file

Last Lecture's Homework

Complete calc.y so that it can be used as a calculator for the four basic arithmetic operations (+-*/), including prefix minus for negative numbers and parentheses for grouping. Create your own test.in for example input, and test.check for example output. Use the grammar to define priorities and associativities (do NOT use %left, %right,...); submit calc.y only.

Summary of Last Lecture

• Creating a program with bison and flex needs many steps, so using make is important
• The input format for bison is very similar to the input format for flex, but there are also some differences
• bison uses attribute grammars to calculate the result of parsing
• The attributes are referenced as $$ (left hand side) and $1, $2,... (right hand side) in the C program fragments
• Priority and associativity of operators are expressed in the rewriting rules of the grammar

How to Write a Grammar

• Priorities
• Associativity
• Repetition (lists)
• Parentheses, unary operations
• Other constructs

Grammar Patterns: Priority

(priority: (lowest) low_exp < middle_exp < high_exp (highest); assuming left associative)

low_exp: low_exp low_pri_op middle_exp
       | middle_exp
;
middle_exp: middle_exp mid_pri_op high_exp
          | high_exp
;

How to Express Priorities

• Use a separate nonterminal symbol for each priority level
• Write the grammar starting with the lowest priority (outside)
• On the right hand side of the rewriting rule for a given priority's nonterminal,
  use nonterminals of the same priority or one level higher priority
  
• How to select names for nonterminals:
  • Use mathematical terms (用語): term (項), factor (因子)
  • Use the type of operator, or one representative operator
    (shift_expression, mulExpression,...)
  • Finding good names is difficult, but helps a lot

Grammar Patterns: Associativity

Left associative:

low_exp: low_exp left_assoc_op high_exp
       | high_exp
;

Right associative:

low_exp: high_exp right_assoc_op low_exp
       | high_exp
;

How to Express Associativity

• Left associative:
  • Use the same nonterminal on the left hand side of the rewriting rule and to the left of the operator on the right hand side
  • Use the higher priority nonterminal to the right of the operator
  • (unless this operator is required) Also create a rewriting rule with only the higher priority nonterminal as the right hand side
• Right associative:
  • Exchange the nonterminals on the right and the left of the operator (same nonterminal is to the right of the operator)

Grammar Patterns: Repetition

Zero or more times:

items: items item
     |
;

One or more times:

items: items item
     | item 
;

Instead of "items item", "item items" is also possible, but bison's stack may become a problem

How to Express Repetition (Lists)

• Example: A list of statements (statementList or statements)
• Two rewriting rules are needed
• Base rule:
  • Repetition of 0 or more times: A rewriting rule with an empty right hand side
  • Repetition of 1 or more times: A rewriting rule with a single element (e.g. statement) on the right hand side
• Inductive rule:
  • Right hand side uses both list and single element nonterminals
  • If associativity is important, it determines the order of the two nonterminals
  • If associativity is not important, there are two choices:
    • List first: Left recursion; advantage: smaller stack
    • Element first: Right recursion

Grammar Patterns: Parentheses and Unary Expressions

Expressions with parentheses are at the innermost (highest precedence) level (high_exp)

Inside parentheses, any expression can be used, therefore lowest precedence (low_exp)

high_exp: open_paren low_exp close_paren
        | literal
;

Unary expressions (e.g. prefix hyphen for negative numbers) are also at the innermost level, but refer to the innermost level on the right hand side, too:

high_exp: HYPHEN high_exp
;

How to Express Other Constructs (e.g. if statement)

Write as is, carefully distinguishing alternatives and terminal/non-terminal symbols

if_statement : IF '(' cond ')' statement
             | IF '(' cond ')' statement
               ELSE statement
;

(statement is a generic term including blocks ({}, compound statements)

How bison Works

• Order of derivation
• LALR parsing
• How to debug bison grammars
• How to read the .output file

Order of Derivation: Leftmost and Rightmost Derivation

Leftmost derivation: Leftmost nonterminal in syntax tree always expanded first

Rightmost derivation: Rightmost nonterminal in syntax tree always expanded first

Simple example grammar:

E → E '-' T | T
T → n

Example of input: 5 - 7 - 3

Derivation Choices

Different choices may generate:

• Different words:
  This is necessary to be able to process different inputs
• Different syntax trees (same word):
  Ambiguous grammar, try to avoid
• Different orders (leftmost/rightmost/... derivation; same syntax tree):
  Different parsing algorithms

Analysis Methods

• LL: Read input from the left, use leftmost derivation (used in top-down parsing)
• LR: Read input from the left, use rightmost derivation, in reverse order)
• LL(1): LL, with one token lookahead
• LR(1): LR, with one token lookahead
• LALR: A kind of LR(1), used widely in yacc and bison

The labels are also used for grammars:
grammar g is LL(1) ⇔ grammar g can be used with an LL(1) parser.

Understanding bison: Debuging

#define YYDEBUG 1 switches on debugging

The output shows how bison works:

• A (pushdown) stack is used to store:
  • States (of an automaton)
  • Already read terminals and reduced nonterminals
• The automaton state and the next input token decide the action to be taken
• There are three possible actions:
  • shift: Read a token and put it on the stack (together with a state)
  • reduce: Convert some (non)terminals on the stack to a single nonterminal using a rewriting rule
    (a reduce action is always followed by a goto to another state)
  • accept: Stop processing and accept the input

Understanding bison: The .output File

bison -v creates a file with extension .output, containing the following interesting details:

• [Problems: Unused terminal symbols, conflicts]
• Grammar: Numbered rewriting rules; rule number 0 is $accept: start_symbol $end)
• Terminals: Numbered terminal symbols; numbers are ASCII codes or ≧256)
• Nonterminals, with numbers of rules where they appear
• States, with the following information for each state:
  • Rewriting rules (. shows current position)
  • Terminal symbols (or $default) and the action (shift, reduce, goto) if this symbol is the next symbol in the input
  • Nonterminal symbols: goal state of transition after reduction

Conflicts and Ambiguous Grammars

• When running bison, it may show some conflicts:
  • shift/reduce conflicts: Both shift and reduce are possible; shift is always choosen
  • reduce/reduce conflicts: There is more than one way to reduce
• bison just chooses one of the selections:
  • If this is the right selection, we are fine
    (but we may want to fix the grammar anyway)
  • If this is the wrong selection, we need to fix the grammar
• Grammar example: E → E '-' E | integer
For 5 - 3 - 7, this grammar allows two interpretations:
  (5-3) - 7 and 5 - (3-7)

Another Example of Ambiguity

The grammar for if-else is a famous example of ambiguity:

if (...) if (...) ...; else ...;

can be parsed in two ways:

if (...) {
    if (...)  ...;
    else      ...;
}

or

if (...) {
    if (...)  ...;
}
else  ...;

This creates a shift-reduce conflict.

The first way of parsing is correct (for C), and is choosen by bison because in a shift-reduce conflict, shift is selected.

Grammar of bison Rewriting Rules (META!)

rewritingRule → nonterminalSymbol ":" rightHandList ";"
rightHandList → rightHand | rightHand "|" rightHandList
rightHand → symbolList "{" CFragment "}"
symbolList → ε | symbol symbolList
symbol → nonterminalSymbol | terminalSymbol

How to Combine flex and bison

• In the .y file, list all token types:
  %token NUM PLUS ASTERISK ...
• In the .y file, define the type of the attributes
  #define YYSTYPE int
• In the .l file, define one or more rules for each token type
  For each rule, return the appropriate token type
• In the .y file, define the rewriting rules of the grammar
• In the .y file, write the program fragments to calculate attribute values
• Process (with flex/bison), compile, and test

Advantages and Problems of Bottom-Up Parsing

• Advantages:
  • No problems with left recursion
  • Wider range of grammars
  • Automatic creation of parser
• Problems:
  • Very hard to create parser by hand
  • Ambiguities needs attention

Homework

Deadline: July 6, 2023 (Thursday in two weeks), 22:00

Where to submit: Moodle (rational.y)

Important: This homework requires significantly more time than other homeworks.
Start early, so that you can ask questions next week (June 30, Friday) or in Moodle (Q&A Forum)

Important: Files need to compile with flex/bison/gcc, otherwise this homework will not be counted

Important: Make sure you have #define YYDEBUG 0 in rational.y

Submission: rational.l and rational.y, name (kanji and kana) and student number as a comment at the top right of the file; no overlong (>80 char) lines

Collaboration: The same rules as for
Projects in Information Technology II apply!

Homework: A Calculator for Rational Numbers

Change the simple calculator of calc.y to a calculator that can handle integers and rationals.

• Statements are separated with ;.
• Rationals are expressed as [numerator, denominator] .
• Inside [], division is not allowed. Make sure this is checked by the grammar.
• Nesting of [] is not allowed. Make sure this is checked by the grammar.
• Calculations are exact, not using floating point numbers.
• Print out the result of each statement.
• Results are given as irreducible fractions, with a minus sign on the numerator if applicable.
• Example result: Result is -5/3
• Bad example result: Result is 15/-6
• Use the grammar to define priorities and associativities (do NOT use %left, %right,...).

Hints for Homework

• This is a significant homework that will require serious time and effort
• Start from the calc.l and calc.y files, but change the file names
• Use the makefile (change file names)
• Use YYSTYPE to define a type (struct) that can handle rationals and integers (both in .l and .y)
• Define the necessary additional tokens in .y (%token rule)
• If you have a shift/reduce or reduce/reduce conflict:
  • Use the -Wcounterexamples option on the bison command
  • Check the .output file
  • Check using debugging (YYDEBUG 1)
• Expand your grammar little by little, always carefully testing
• Build up your own test file, and expand it together with expansions to the grammar
• Create tests that check important aspects of the grammar (priorities, associativity, errors)
• Save test outputs and use them for automatic comparison

Additional Homework

(no need to submit)

Use calc.output and #define YYDEBUG 1 to understand how bison works

Announcement:

• There will be a 20~30 minutes written test next week
• Topics: All topics of this lecture until now,
  but biased towards parsing

Glossary

unary (operator)

単項 (演算子)

rightmost derivation

最右導出

reverse order

逆順

lookahead

先読み

non-proportional font

等幅のフォント

rational number

有理数

numerator

分子

denominator

分母

sign

符号

irreducible fraction

既約分数