(クイックソート、平均計算量)

http://www.sw.it.aoyama.ac.jp/2019/DA/lecture7.html

© 2009-19 Martin J. Dürst 青山学院大学

- Leftovers, summary of last lecture
- Quicksort:

Concept, implementation, optimizations - Average time complexity
- Sorting in C, Ruby, ...
- Comparing sorting algorithms using animation

- Sorting is a very important operation for Information Technology
- Simple sorting algorithms are all
`O`(`n`^{2}) - Merge sort is
`O`(`n`log`n`) and needs double memory - Heap sort is
`O`(`n`log`n`) and uses a large number of comparisons and exchanges

- 218341.368 seconds (⇒about 61 hours)
- 6
^{1010}·10^{3·1010}(units? way too big) `O`(40000) (how many seconds could this be)- Calulation of actual time backwards from big-
`O`notation: 1second/operation,`n`=5000,`O`(`n`^{2}) ⇒ 25'000'000 seconds? - A
`O`(`n`) algorithm (example: "5 seconds per page") - For 12 people, having only one person work towards the end of the algorithm
- For humans, binary sorting is constraining (sorting into 3~10 parts is better)
- Using bubble sort (868 days without including breaks or sleep)
- Prepare 10
^{10}boxes (problem: space, cost, distance for walking) - Forgetting time for preparation, cleanup, breaks,...
- Submitting just a program
- Report too short

Using quicksort as an example, understand

- Different ways to use divide-and-conquer for sorting
- Move from algorithmic concept to efficient implementation
- Average time complexity

- Invented by C. A. R. Hoare in 1959
- Researched in great detail
- Extremely widely used

- Heap sort: The highest priority item of the overall tree is the highest priority item of the two subtrees
- Merge sort: Split into equal-length parts, recurse, merge

- Quicksort: Use an arbitrary boundary element to partition the data, recursively

- Select one element as the
*partitioning element*(`pivot`) - Split elements so that:
- Elements smaller than the pivot go to the left, and
- Elements larger than the pivot go to the right

- Apply quicksort recursively to the data on the left and right sides of the pivot

Ruby pseudocode/implementation: `conceptual_quick_sort`

in 7qsort.rb

Both algorithms use the same split-recurse-merge pattern, but there are important differences:

Mergesort | Quicksort | |
---|---|---|

split | equal size | size unpredictable |

work done | on merge | on split |

no work needed | on split | on merge |

- Use e.g. the rightmost element as the pivot
- Starting from the left, find an element larger than the pivot
- Starting from the right, find an element smaller than the pivot
- Exchange the elements found in steps 2. and 3.
- Repeat steps 2.-4. until no further exchanges are needed
- Exchange the pivot with the element in the middle
- Recurse on both sides

Ruby pseudocode/implementation: `simple_quick_sort`

in 7qsort.rb

- What happens if the largest (or the smallest) element is always choosen as the pivot?
- The time complexity is Q
_{w}(`n`) =`n`+ Q_{w}(`n`-1) = Σ^{n}_{i=1}`i`

⇒`O`(`n`^{2}) - This is the
*worst case complexity*(*worst case running time*) for quick sort - This complexity is the same as the complexity of the simple sorting algorithms
- This worst case can easily happen if the input is already sorted

- Q
_{B}(1) = 0 - Q
_{B}(`n`) =`n`+ 1 + 2 Q_{B}(`n`/2) - Same as merge sort
- ⇒
`O`(`n`log`n`^{}) - Unclear whether this is relevant

For most algorithms (but there are exceptions):

- Worst case complexity is very important
- Best case complexity is mostly irrelevant

- Assumption: All permutations of the input values have the same
probability
→for the pivot, each position is equally probably

- Q
_{A}(1) = 0 - Q
_{A}(`n`) =`n`+ 1 + 1/`n`Σ_{1≤k≤n}(Q_{A}(`k`-1)+Q_{A}(`n`-`k`))

[1] Q_{A}(`n`) = `n`
+ 1 + 1/`n` Σ_{1≤k≤n}
(Q_{A}(`k`-1)+Q_{A}(`n`-`k`))

[2] Q_{A}(0) + ... +
Q_{A}(`n`-2) + Q_{A}(`n`-1) =

= Q_{A}(`n`-1) + Q_{A}(`n`-2) + ... +
Q_{A}(0)

[3] Q_{A}(`n`) = `n`
+ 1 + 2/`n` Σ_{1≤k≤n}
Q_{A}(`k`-1) [use [2] in [1]]

[4] `n` Q_{A}(`n`) =
`n` (`n` + 1) + 2`
Σ`_{1≤k≤n}
Q_{A}(`k`-1) [multiply [3] by
`n`]

[5] (`n`-1)
Q_{A}(`n`-1) = (`n`-1) `n` + 2`
Σ`_{1≤k≤n-1}
Q_{A}(`k`-1) [[4], with `n`
replaced by `n`-1]

[6] `n` Q_{A}(`n`) -
(`n`-1) Q_{A}(`n`-1) = `n` (`n`+1)
- (`n`-1) `n` + 2 Q_{A}(`n`-1) [[4]-[5]]

[7] `n` Q_{A}(`n`) =
(`n`+1) Q_{A}(`n`-1) + 2`n` [simplifying [6]]

[8]
Q_{A}(`n`)/(`n`+1) =
Q_{A}(`n`-1)/`n` + 2/(` n` + 1)
[dividing [7] by

`Q`_{A}(`n`)/(`n`+1) =

= Q_{A}(`n`-1)/`n` + 2/(` n` + 1)
= [repeatedly expand right side of [8] by using [8]]

= Q

= Q

= Q

Q_{A}(`n`)/(`n`+1) ≈ 2
Σ_{1≤k≤n} 2/`k` ≈
2∫_{1}` ^{n}` 1/

Q_{A}(`n`) ≈ 2`n` ln `n` ≈ 1.39
`n` log_{2} `n`

⇒ `O`(`n` log `n`^{})

⇒ The number of comparisons on average is ~1.39 times the optimal number of comparisons in an optimal decision tree

- A good average complexity is not enough if the worst case is frequently reached
- For Q
_{A}, it can be shown that the standard deviation is about 0.65`n` - This means that the probability of deviation from the average very quickly gets extremely small
- That means that assuming a normal distribution:
~68% is within 1.39

`n`log_{2}`n`±0.65`n`~95% is within 1.39

`n`log_{2}`n`±1.3`n`~99.7% is within 1.39

`n`log_{2}`n`±1.95`n`~99.993% is within 1.39

`n`log_{2}`n`±2.6`n`~99.999'94% is within 1.39

`n`log_{2}`n`±3.25`n`~99.999'999'8% is within 1.39

`n`log_{2}`n`±3.9`n`

Question: What is the complexity of sorting (as a problem)?

- Many good sorting algorithms are
`O`(`n`log`n`) - The basic operations for sorting are comparison and movement
- For
`n`data items, the number of different sorting orders (permutations) is`n`! - With each comparision, in the best case, we can reduce the number of sorting orders to half
- The mimimum number of comparisions necessary for sorting is log (
`n`!) ≈ O(`n`log`n`) - Sorting using pairwise comparison is
`Ω`(`n`log`n`)

- The efficiency of quicksort strongly depends on the selection of the pivot
- Some solutions:
- Select rightmost element

(dangerous!) - Use the median of three values
- Use the value at a random location

(this is an example of a*randomized algorithm*)

- Select rightmost element

- Comparison of indices

→ Use a sentinel to remove one comparision

- Stack overflow for deep recursion

→ When splitting into two, use recursion for the smaller part, and tail recursion or a loop for the larger part - Low efficiency of quicksort for short arrays/parts

→ For parts smaller than a given size, change to a simple sort algorithm

→ With insertion sort, it is possible to do this in one go at the very end

(this needs care when testing)

→ Quicksort gets about 10% faster if change is made at an array size of about 10 - Duplicate keys

→ Split in three rather than two

Ruby pseudocode/implementation (excluding split in three):
`quick_sort`

in 7qsort.rb

- Uses Web technology: SVG (2D vector graphics) and JavaScript
- Uses special library (Narrative JavaScript) for timing adjustments
- Comparisons are shown in yellow (except for insertion sort), exchanges in blue

Watch animation: sort.svg

- Definition: A sorting algorithm is
`stable`if it retains the original order for two data items with the same key value - Used for sorting with multiple criteria (e.g. sort by year and
prefecture):

- First, sort using the lower priority criterion (e.g. prefecture)
- Then, sort using the higher priority criterion (e.g. year)

- The simple sorting algorithms and merge sort can easily be made stable
- Heap sort and quicksort are not stable

→ Solution 1: Sort multiple criteria together

→ Solution 2: Use the original position as a lower priority criterion

- Sorting is provided as a library function or method
- Implementation is often based on quicksort
- Comparison of data items depends on type of data and purpose of
sorting

→ Use comparison function as a function argument - If comparison is slow

→ Precompute a value that can be used for sorting - If exchange of data items is slow (e.g. very large data items)

→ Sort/exchange references (pointers) only

`qsort`

Functionvoid qsort( void *base, // start of array size_t nel, // number of elements in array size_t width, // element size int (*compar)( // comparison function const void *, const void *) );

`Array#sort`

(`Klass#method`

denotes instance method `method`

of
class `Klass`

)

`array.sort`

uses `<=>`

for comparison

`array.sort { |a, b| a.length <=> b.length }`

This example sorts (e.g. strings) by length

The code block (between `{`

and `}`

) is used as a
comparison function

`<=>`

Operator(also called spaceship operator, similar to `strcmp`

in C)

Relationship between `a` and `b` |
return value of `a <=> b` |

`a < b` |
`-1` (or other integer smaller than 0) |

`a = b` |
`0` |

`a > b` |
`+1` (or other integer greater than 0) |

`Array#sort_by`

`array.sort_by { |str| str.length }`

or ```
array.sort_by
&:length
```

(sorting strings by length)

```
array.sort_by { |stu| [stu.year, stu.prefecture]
}
```

(sorting students by year and prefecture)

This calculates the values for the sort criterion for each array element in advance

- Quicksort is another application of divide and conquer
- Quicksort is a very famous algorithm, and a good example to learn about algorithms and their implementation
- Quicksort has been carefully researched and widely implemented and used
- Quicksort is a classical example of the importance of average time complexity
- Quicksort is our first example of a randomized algorithm
- Sorting based on pairwise comparison is
`Θ`(`n`log`n`)

- Think about inputs for which
`conceptual_quick_sort`

will fail - Watch the animations carefully (>20 times) to deepen your understanding of sorting algorithms

- quicksort
- クイックソート
- partition
- 分割
- partitioning element (pivot)
- 分割要素
- worst case complexity (running time)
- 最悪時の計算量
- best case complexity (running time)
- 最善時の計算量
- average complexity (running time)
- 平均計算量
- standard deviation
- 標準偏差
- randomized algorithm
- ランドム化アルゴリズム
- median
- 中央値
- decision tree
- 決定木
- tail recursion
- 末尾再帰
- in one go
- 一括
- stable sorting
- 安定な整列法
- criterion (plural criteria)
- 基準
- block
- ブロック