# Principles of top-down parsing

(下向き構文解析の実装)

## Language Theory and Compilers

http://www.sw.it.aoyama.ac.jp/2016/Compiler/lecture8.html

### Martin J. Dürst

© 2005-16 Martin J. Dürst 青山学院大学

# Today's Schedule

• Remainders and summary from last lecture
• Top-down parsing
• Recursive descent parsing
• Implementation of recursive descent parsing
• How to deal with various problems in a grammar:
• Limitation of number of operations
• Priority
• Left/right associativity
• Left recursion

# Summary of Last Lecture

• `flex` homework: Lexical analysis/regular expressions for C
• Regular expression for C comments
• Grammars for various languages
• How to construct a grammar
• Results of parsing: Parse tree and abstract syntax tree

# General Top-Down Parsing

• Create parse tree starting with the start symbol of the grammar
• Expand parse tree depth-first from the left
• If there is a choice (of rewriting rules), try each one in turn
• Once the parse tree reaches a terminal symbol, compare with input
• If there is a match, continue
• If there is no match, give up and try another choice (backtracking)

# Main Points of Backtracking

Backtracking may be very slow, but this can be improved:

• Change the grammar so that backtracking is reduced or eliminated
(ideally, the next token should be enough to select a single rewriting rule)
• Remember intermediate results (packrat parser)
Example implementation: treetop (for Ruby)

# Recursive Descent Parsing

• Create a function for each non-terminal symbol of the grammar
• In the function, proceed along the right side of the rewriting rule(s):
• For a terminal symbol, compare with input
• For a non-terminal symbol, call the corresponding function
• Use branching (`if`,...) for a choice (`|`) in the grammar
• Use repetition (`while`,...) for repetition in BNF
• Works well for hand-written parsers
• Reason for name: Recursive grammar rules
Example: `A → variable '=' A | integer` (assignment expression)

# Recursive Descent Parsing: Simple Hand-Written Parser

Program files: scanner.h, scanner.c, parser1.c

How to complie: `gcc scanner.c parser.c && ./a`

# Details of Recursive Descent Parsing: Lexical Analysis

(see scanner.c)

• Invariant:
• The next character to be looked at is always in `nextChar` (one-character lookahead)
• As soon as a character is processed, the next character is read into `nextChar`
• `nextChar` is a global variable, but this can be changed to a function parameter
• How to use from parser:
• Initialize with `initScanner`
• Read tokens with `getNextToken`
• Implementation of `getNextToken`:
• One-character tokens: Direct decision
• Multiple-character tokens: Decide on first character, read the rest with a dedicated function

# Details of Recursive Descent Parsing: Parsing

(see parser1.c)

• Invariant (same as for lexical analysis):
• The next token to be looked at is always in `nextToken` (one-token lookahead)
• As soon as a token is processed, the next token is read into `nextToken`
• `nextToken` is a global variable, but this can be changed to a function parameter
• Overall usage:
• Initialize using `initScanner` and `getNextToken`
• Call the function corresponding to the start symbol of the grammar (e.g. `Expression()`)
• Further process the returned value (abstract syntax tree or result of evaluation)

# Details of Recursive Descent Parsing: Non-Terminal Symbols

• Create a function for each non-terminal symbol of the grammar
• In the function, deal with all rewriting rules for this non-terminal symbol
• For each non-terminal on the right-hand side, call the corresponding function
• For each terminal on the right-hand side, compare with `nextToken`

# How to Deal with Left Recursion

Example of left recursion:

`E → E '-' integer | integer`

Wrong solution (change of associativity):

`E → integer '-' E | integer`

Correct solution:

`E → integer EE`

`EE → '-' integer EE | ε`

In (E)BNF:

`E → integer {'-' integer}`

# Differences between Grammars and Regular Expressions

Grammar:

• Multiple rules
• Non-terminal symbols, derivation from left-hand side to right-hand side
• For simple grammars, *, (), and | are not available

Regular Expression:

• Limited to one single rule
• No non-terminal symbols, only right-hand side
• For practical regular expressions, lots of metacharacters/functionality besides *, (), and | available

A simple regular expression corresponds to a single rewriting rule in an (BNF,...) grammar

# Homework

Deadline: June 2, 2016 (Thursday), 19:00

Where to submit: Box in front of room O-529 (building O, 5th floor)

Format: A4 single page (using both sides is okay; NO cover page, staple in top left corner if more than one page is necessary), printout (no wrapping lines), name (kanji and kana) and student number in comment at the top right

1. Expand the top-down parser of parser1.c to correctly deal with the four basic arithmetic operations.
(`scanner.h`/`c` do not change, so no need to submit them)
2. (bonus problem) Add more operations to the top-down parser.
(If you solve this problem, also submit the `scanner.h`/`c` files, but only one `parser.c` file for both problems.)
3. Bring your notebook computer to the next lecture. Check again that `flex`, `bison`, `make`, and `gcc` are installed.

# Glossary

recursive descent parsing

depth-first

backtracking
バックトラック
ambiguous grammar

right associative

invariant

left recursion