# Algorithm Design Strategies

(アルゴリズムの設計方法)

## Data Structures and Algorithms

### 13th lecture, January 9, 2020

http://www.sw.it.aoyama.ac.jp/2019/DA/lecture13.html

### Martin J. Dürst © 2009-20 Martin J. Dürst 青山学院大学

# Today's Schedule

• Remaining schedule
• Leftovers/summary of last lecture
• Algorithm design strategies
• Examples of 'difficult' problems

# Remaining Schedule

• January 9 (today): 13th lecture (algorithm design strategies)
• January 16: 14th lecture (NP-completenes, reducibility; makeup class)
Important: Contents of makeup class is part of final exam!
• January 23: 15th lecture (approximation algorithms)
• January 30, 09:30-10:55: Term Final Exam

# Summary of Last Lecture

• Dynamic programming is an algorithm design strategy
• Dynamic programming is suited for problems where the overall (optimal) solution can be obtained from solutions for subproblems, but the subproblems overlap
• The time complexity of dynamic programming depends on the structure of the actual problem
• Dynamic programming is closely related to memoization

# Algorithm Design Strategies/Methods

• Simple/simplistic algorithms
• Brute force algorithms
• Greedy algorithms
Find overall optimal solution by selecting locally optimal solutions
• Divide and conquer
Divide problem into nonoverlapping subproblems
• Dynamic programming
Find overall optimal solution from solutions for overlapping subproblems

# The Knapsack Problem

• A knapsack with capacity c (weight or volume) and
• n items s1,..., sn (each with a weight/volume, and in some variations a value)
• Goal: Find the best way to pack the knapsack

Depending on the details of the problem, the best algorithm design strategy is different

# Variations of the Knapsack Problem

1. All items are the same; how many items fit in?
2. Pack as many items as possible
3. Use as much capacity as possible (integer version)
4. Maximise value

# All Items are the Same

Example: Capacity c = 20kg, weight per item: 3.5kg

'Algorithm': Divide the capacity by the weight per item, round down

Design strategy: simplistic 'algorithm'

# Simplistic Algorithm

• Sometimes too simple to be called 'algorithm'
• Examples:
• Select the third-smallest element from a sorted array
• Obtain the surface of a rectangle from the length of its sides
• Closed formula of a number sequence
• Often forgotten because computers are so fast now

# Pack as Many Items as Possible

Example: Capacity c = 20kg, weight of items: 8kg, 2kg, 4kg, 7kg, 2kg, 1kg, 5kg, 12kg

Algorithm: Sort items by increasing weight, pack starting with lightest item

Answer for example: 5 items (e.g. 1kg, 2kg, 2kg, 4kg, 5kg)

Design strategy: Greedy algorithm

Time complexity: O(n log n)

# Greedy Algorithm

• Develop solution step-by-step
• Consider only locally optimal solutions
• Optimal substructure is a precondition, but the structure is different than for dynamic programming
• Time complexities of O(n) and O(n log n) are frequent
• Examples: Calculating change, many planning/scheduling problems, some algorithms on graphs (e.g. minimum spanning tree,...),...

# Use as Much Capacity as Possible (Integer Version)

Example: Capacity c = 20kg, weight of items (ws1,...,wsn): 8kg, 2kg, 4kg, 7kg, 2kg, 1kg, 5kg, 12kg

Algorithm: Consider subproblems with capacity c'c using only the first k items s1,..., sk (kn)

Design strategy: Dynamic programming

Recursive problem description:

max_weight(0, ...) = 0 = max_weight(x, {})

max_weight(c1, {s1,..., sk}) = max(max_weight(c1-wsk, {s1,..., sk-1})+wsk,
max_weight(c1,      {s1,..., sk-1}))

# Example Solution

 k↓ c→ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 (8kg) 0 ×0 ×0 ×0 ×0 ×0 ×0 ×0 ○8 ○8 ○8 ○8 ○8 ○8 ○8 ○8 ○8 ○8 ○8 ○8 ○8 2 (2kg) 0 ×0 ○2 ○2 ○2 ○2 ○2 ○2 ×8 ×8 ○10 ○10 ○10 ○10 ○10 ○10 ○10 ○10 ○10 ○10 ○10 3 (4kg) 0 ×0 ×2 ×2 ○4 ○4 ○6 ○6 ×8 ×8 ×10 ×10 ○12 ○12 ○14 ○14 ○14 ○14 ○14 ○14 ○14 4 (7kg) 0 ×0 ×2 ×2 ×4 ×4 ×6 ○7 ×8 ○9 ×10 ○11 ×12 ○13 ×14 ○15 ○15 ○17 ○17 ○19 ○19 5 (2kg) 0 ×0 ×2 ×2 ×4 ×4 ×6 ×7 ×8 ×9 ×10 ×11 ×12 ×13 ×14 ×15 ○16 ×17 ×17 ×19 ×19 6 (1kg) 0 ○1 ×2 ○3 ×4 ○5 ×6 ×7 ×8 ×9 ×10 ×11 ×12 ×13 ×14 ×15 ×16 ×17 ○18 ×19 ○20 7 (5kg) 0 ×1 ×2 ×3 ×4 ×5 ×6 ×7 ×8 ×9 ×10 ×11 ×12 ×13 ×14 ×15 ×16 ×17 ×18 ×19 ×20 8 (12kg) 0 ×1 ×2 ×3 ×4 ×5 ×6 ×7 ×8 ×9 ×10 ×11 ×12 ×13 ×14 ×15 ×16 ×17 ×18 ×19 ×20

Table rows: Items to add (or not) to knapsack

Table columns: Total capacity

Table entries: ×: Item in this row is not used; ○: Item in this row is used; number: max weight possible

How to find the solution: See next slide

# How to Find the Solution

• Start at bottom right corner of the table
• If there is a ×, do not include the item on this row, and move up one row.
• If there is a ○:
1. Do include the item on this row
2. Move left w columns, where w is the weight of the item included
3. Move up one row
• Repeat until you arrive at a row or column that is 0

Answer for example: 1kg, 7kg, 4kg, 8kg (total: 20kg; other solutions possible)

Time complexity: O(cn)

# Value Maximization

Example: Capacity c = 20kg, weight and value of items: 8kg, 500¥; 2kg, 2000¥; 4kg, 235¥; 7kg, 700¥; 2kg, 400¥; 1kg, 100¥; 5kg, 450¥; 12kg, 650¥

Algorithm: Try all possible solutions

Design strategy: Brute force

Implementation: Dknapsack.rb (of various algorithms, only brute force finds optimal solution if weights and values can be non-integers)

# Brute Force Algorithm

• Try all solution possibilities (combinations, permutations,...)
• Choose the best solution
• May be very slow
• May be improved by shortcuts (e.g. compare with best solution found up to now)
• For more examples, see next lecture

# Variations of the Knapsack Problem (Summary)

1. All items are the same; how many items fit in?
Solution: Divide capacity by item weight
Design strategy: Simplistic 'algorithm'
2. Pack as many items as possible
Design strategy: Greedy algorithm
3. Use as much capacity as possible (integer version)
Solution: Consider subproblems with capacity c'c and items s1,..., sk (kn)
Design Strategy: Dynamic programming
4. Maximize value
Design Strategy: Brute force

# Algorithm Design Strategies

• Useful when developing algorithms
• Consider design strategies one-by-one
• For some problems, some strategies can be excluded quickly
• Depending on the details of the problem, the best strategy may be different
• For the same problem and the same strategy, there may be several algorithms

# Goal for Remaining Time

• Be able to distinguish between 'simple' and 'difficult' problems
• Expand your view, look at the algorithm universe

# Example Problem 1: 3-SAT

• n binary variables
• A logical formula using these variables
• The formula is a conjunction of disjunctions (of negations)
• All disjunctions use exactly 3 terms
• Problem: Find values for each variable so that the overall formula becomes true
• Problem variant: Decide whether a solution is possible or not
• Example (' indicates negation):
(x1x2x4) ∧ (x1'∨x3x4') ∧ (x2x3'∨x4) ∧ (x1'∨x2'∨x3')
• Number of possible answers: 2n
• Time to check all possible answers: O(n2n)
• Currently, no faster general algorithm is known
• Currently, there is no proof that there is no faster general algorithm

# Example Problem 2: Independent Set

• Graph with n vertices
• If two vertices are connected by an edge, there is a conflict
• Problem: Find the largest independent set (i.e. subset of vertices without conflicts)
• Problem variant: Decide whether there is an independent set of size ≧k
• Number of possible answers: 2n
• Time to check all possible answers: O(n2n)
• Currently, no faster general algorithm is known
• Currently, there is no proof that there is no faster general algorithm

# Example Problem 3: Traveling Salesman

• n towns
• Distances (or time or cost) between each pair of towns
• Problem: Find the shortest (fastest/cheapest) tour that visits all cities exactly once
• Problem variant: Decide whether there is a tour of size (duration/cost) ≦k
• Number of possible answers: n!
• Time to check all possible answers: O(n!)
• Currently, no faster general algorithm is known
• Currently, there is no proof that there is no faster general algorithm

# Homework

(no need to submit)

• Review this lecture
• Review Dknapsack.rb
• Find commonalities of 3-SAT, independent set, and traveling salesman problems
• Prepare for term final exam

# Glossary

NP-completeness
NP-完全性
reducibility

approximation algorithms

brute force

greedy algorithm

knapsack problem
ナップサック問題
capacity

closed formula
「閉じた式」
number sequence

minimum spanning tree (problem)

independent set

conflict

traveling salesman