From Emperors to Zombies:
Ruby from Unicode 10 to 12.1

https://www.sw.it.aoyama.ac.jp/2019/pub/RubyKaigiLight/

Ruby Kaigi 2019 Lightning Talk, Fukuoka, Japan, April 19, 2019

Martin J. DÜRST (マーティンと呼んでください)

duerst@it.aoyama.ac.jp, Aoyama Gakuin University

© 2019 Martin J. Dürst, Aoyama Gakuin University

What is Your Ruby Version?

What is Your Unicode Version?

What is Your Unicode Version?

RbConfig::CONFIG["UNICODE_VERSION"]

⇒ '12.1.0'

What is Your Unicode Version?

Who cares?
どうでもいいのではないか?

Does Your Unicode Version Matter?

New Scripts and Characters

Does Your Unicode Version Matter?

New Scripts and Characters
New Emoji

Does Your Unicode Version Matter?

New Scripts and Characters
New Emoji
Reiwa (令和) support

Ruby and Unicode Versions

U = Y - 2007 = 10R-15              R = (Y-1992) · 0.1 = 0.1U + 1.5      

Not so Fast!

Not so Fast!

Faster!!!

Ruby and Unicode Versions

Faster Indeed!

Faster But Why?

Unicode 12.1.0 is still in beta

The Easiest: Unicode 12.0.0

This is what an easy (I'd wish to say typical) upgrade looks like:

• Check whether there are changes that affect Ruby → None
• Change version in common.mk
• Run make up; …
• Run all tests with make check
• Commit (r67169), pray that CI will work, NEWS (r67173), cleanup (r67174)
• Wait for Yui Naruse to release (THANKS!)

The Weirdest: Unicode 11.0.0

• Feature #14802
• Check whether there are changes that affect Ruby:
  • New Definition of Extended Grapheme Cluster (Feature #15182)
    (Unicode Standard Annex #29: Unicode Text Segmentation)
  • Gregorian MTAVRULI (Feature #14839)

Extended   
Grapheme
Cluster
WHAT?


/\x/


/\x/
WHAT?

/\x/
read as backslash-x

Extended Grapheme Clusters   

Example: Flag of Wales

flag_of_Wales = "\u{1F3F4 E0067 E0062 E0077 E006C E0073 E007F}"

flag_of_Wales.length ⇒ 7 (characters)

flag_of_Wales.bytes.length ⇒ 28 (bytes)

flag_of_Wales.grapheme_clusters.length ⇒ 1
(extended grapheme clusters)

"A#{flag_of_Wales}Z".match? /A\xZ/ ⇒ true

How is \x Implemented?

• Function node_extended_grapheme_cluster in regparse.c
• Called every time the Regexp parser sees a \x
• Very long (~700 lines) and highly repetitive
• Builds up a tree of nodes
• Needs to be updated for Unicode 11.0.0
• Unclear how it matches to current (10.0.0) spec,
  because spec itself defines breaks, not runs

How to Understand Code?

Rewrite it!

Rewritten!

700 lines, 1 function
⇒ 300 lines, 5 functions
Very Different Code Style

Questions Remain

No way to reuse node tree?
No way to convert subexpression to node tree?

How to Rewrite Safely?

Test First!

The Zombie Bug

• Not just a few lonely asserts!
• Use test data from Unicode:
  enc/unicode/data/emoji/11.0.0/emoji-sequences.txt,… (4 total)
• Create/run tests:

  ruby test/runner.rb test/ruby/enc/test_emoji_breaks.rb

  4 tests, 81777 assertions, 0 failures, 0 errors, 0 skips

• Bug #15343: String#each_grapheme_cluster wrongly splits some emoji (genie, zombie, wrestling)

The Zombie Bug Explained

• Ruby 2.5 thinks that zombies can have skin colors

• Unicode says that zombies canNOT have skin colors
  
  
  

  Zombie Skin Color

Unicode 12.1.0 Background

• Crown Prince Naruhito will become Japanese Emperor on May 1st
• Japanese dates work by era: Today is April 19, Heisei 31
• Each modern era name has a short, square character:
  U+337B (㍻, Heisei), U+337C (㍼, Showa),
  U+337D (㍽, Taisei), and U+337E (㍾, Meiji)
• The character for the new era name will be at U+32FF

The Fastest: Unicode 12.1.0, Part 1

• Preparation: Make Ruby work with Unicode beta versions

  Test on phony beta:

  U+32FF SQUARE ERA NAME SAYUU (左右, left-right)

• The annoucement: April 1st, ~11:40 Japan time: 令和 / Reiwa
• Sent mail to Unicode Consortium (also: official notice)

  From: Martin_J. Dürst <duerst@it.aoyama.ac.jp>
Subject: New Japanese Era Name
Date: Mon, 1 Apr 2019 11:43:24 +0900

The new Japanese era name has been announced a few minutes ago: 令和 U+4EE4 U+548C; Reading: REIWA
This is the information that's needed for Unicode 12.1.0. Does anybody know what's the schedule for the official approval of this new version?

Reiwa Background

• 梅花の宴: Plum Flower viewing party held in Dazaifu
• Dazaifu: Very close to Fukuoka city, worth a trip if you have time
  (sorry, too late for plum flowers)
• Manyoshu (万葉集): Everleaf Japanese poem collection
• 初春の令月にして、気淑く風和ぎ、梅は鏡前の粉を披き、蘭は珮後の香を薫らす

The Fastest: Unicode 12.1.0, Part 2

• Question: Why is Unicode 12.1.0 released on May 7?

  Answer: Because they are in a meeting the week before :-)

• Real beta released April 2
• Mail received from Unicode expert:

  From: Ken Whistler <xxxxxx@yyyyy.zzz>
Date: Sun, 31 Mar 2019 21:51:03 -0700

If all you need is just the code point, name, and decomposition, then by all means, start prepping your updates as soon as you can.

The Fastest: Unicode 12.1.0, Part 3

• April 5: Commit (r67441)
• Errors on CI
• Aaron Paterson: I am a puts debugger
• Me: I am a puts commit (commit puts?) debugger
  (some puts: r67445, more puts: r67447, removed: r67451)
• Help from Nobu and k0kubun (THANKS!)
• April 6: cleanup (r67453)
• April 9: NEWS (r67476)
• April 17: Ruby 2.6.3 Released with Unicode 12.1.0 (Reiwa / 令和)
• Faster than most other programming languages
• Even faster if we didn't wait for RubyKaigi :-)

New in Ruby 2.6.3

In 2.6.2:

"\u32FF".match? /\p{age=12.1}/
⇒ Systax Error: invalid character property name

In 2.6.3:

"\u32FF".unicode_normalize :nfkc
⇒ "令和"

"\u32FF".match? /\p{age=12.1}/
⇒ true

New in Ruby 2.6.3

"\u32FF".unicode_normalize :nfkc
⇒ "令和"

"\u32FF".match? /\p{age=12.1}/
⇒ true

Enjoy!