Design Considerations for
Internationalization in Ruby 2.2

http://www.sw.it.aoyama.ac.jp/2014/pub/RubyI18N/

IUC 38, Santa Clara, CA, U.S.A., 4 November 2014

Martin J. DÜRST

duerst@it.aoyama.ac.jp

Aoyama Gakuin University

© 2014 Martin J. Dürst, Aoyama Gakuin University

Introductions

• Programming experience?
• Programming with Ruby?
• Internationalization/Globalization experience?
• Unicode knowledge?

Overview

• Ruby
• Ruby I18N
• Case studies:
  • Normalization
  • Case conversion
  • Properties
  • Break Iterators
• Conclusions

Conventions Used

(Ruby) code is green, monospace
puts "Hello Ruby!"

Variable parts are orange
puts "some string"

Encoding is indicated with a _subscript

• "Юに코δ"_UTF-8
• "ユニコード"_SJIS

Results are indicated with "⇒": 1 + 1 ⇒ 2

Ruby

• Created by Yukihiro Matsumoto (Matz)
• Easy for beginners,
  deep for experts
• Object-oriented throughout, but not obtrusive
• (internal) DSLs
• Metaprogramming
• irb: Interactive Ruby Shell
• 'killer' application: Ruby on Rails Web framework

Ruby Implementations

• MRI (Matz's Ruby Implementation), aka C-Ruby
  available on many platforms (download for Windows)
• JRuby: Ruby on the JVM
• Rubinius: Ruby (mostly) in Ruby
• mruby: Lightweight Ruby for embedding
• MagLev: Ruby on GemStone/S JIT
• IronRuby: Ruby on Microsoft CLR
• ...more

Basic Ruby

5.times { puts "Hello Ruby!" }

• Everything is an object
• Methods can take blocks ({ } or do end)
• Method arguments don't need parentheses
• No semicolons at end of line

Ruby Library Hierarchy

• Core:
  direct use (String, Array, Hash,...)
• Standard Library:
  preinstalled, needs require
• Gems: Huge collection at rubygems.org
  easy download, needs require
• Other code
  needs installation and require

Object-Orientation

• Big classes:
  Ruby: String
  [Java: String, StringBuffer, StringBuilder, byte[], char[]]
• Open classes:
  Methods can be added/changed at any time
• Unobtrusive
  main object as file context
• Duck typing
  Methods as interfaces, inheritance for implementation

Current I18N Support

(since Ruby 1.9)

1. Strings are sequences of characters (codepoints)
   "Юに코δ".length ⇒ 4
   
2. Strings tagged with encoding
   "Юに코δ"_UTF-8.encoding ⇒ UTF-8
   
3. Encoding conversion
   "abcδ".encode 'SJIS' ⇒ "abcδ"_SJIS
4. Regular expressions:
   "Юに코δ".match /δ/
5. Source files are UTF-8 by default
   (since Ruby 2.1, US-ASCII before)

Point 2 is fundamentally different from other programming languages, which in general use just a single encoding internally (Java, JavaScript: UTF-16; Python: 8-bit/UTF-16/UTF-32; Perl: 8-bit/UTF-8; C/C++: mb/wc)

Important Encodings

In Ruby, all encodings are equal, but some encodings are more equal than others:

[Adapted from George Orwell's Animal Farm.]

• US-ASCII
  • Common denominator
  • Various optimizations
• ASCII-8BIT (alias BINARY)
• UTF-8
  • Preferred encoding for Unicode data
  • Pivot for encoding conversions
  • Dedicated optimizations
  • Encoding of the future

Missing I18N Support

• Normalization
• Case conversion
• Number/date formatting
• Character properties
• Collation
• Grapheme clusters, break iterators
• ...

External Libraries

Most I18N support is currently available via various libraries:

• Pure Ruby:
  • UnicodeUtils
  • ActiveSupport::Multibyte
  • TwitterCLDR
• C extensions:
  • ICU as a gem: icu, ffi-icu

Design Considerations

• Unobtrusive
• Backwards compatible
• Match with object system
• Match with encoding tagging
• Portability among implementations
• Efficiency: Performance, Memory

Case Studies

• Normalization
• Case conversion
• Properties
• Break Iterators

Case Study: Normalization

Unicode Normalization Forms NFC, NFD, NFKC, NFKD

Defined in UTS #15, yesterday's tutorial

Example: NFC of R + ̣ + is Ṝ

Normalization in Other Languages

• Java
  import java.text.Normalizer
Normalizer.normalize(string, Normalizer.Form.NFC)
• C#
  using System.Text;
string.Normalize(NormalizationForm.FormKD);
• Perl (Unicode::Normalize)
  require Unicode::Normalize;
Unicode::Normalize->import(qw(NFD NFC NFKD NFKC));
NFKD($string);
• Python (unicodedata.normalize)
  import unicodedata
unicodedata.normalize('NFKD', string)

Normalization in Ruby Libraries

• UnicodeUtils
  

  require "unicode_utils/nfc"
  require "unicode_utils/nfkd"
  require "unicode_utils/nfkc"
  require "unicode_utils/canonical_decomposition"
  UnicodeUtils.nfkc string

• ActiveSupport::Multibyte

  require 'active_support/multibyte/chars'
  ActiveSupport::Multibyte::Chars.new(string).normalize :kc

• TwitterCLDR
  

  require 'twitter_cldr'
  TwitterCldr::Normalization::NFKC.normalize string

Observations

• require/using/import necessary
  
• (mostly) separate class/module
• (mostly) function rather than method

We can do better!

I18N Functionality Where It Belongs

• Normalization: Method on String
• Case conversion: Method on String
• Number formatting: Method(s) on number class(es)
• Date formatting: Method(s) on Date/Time
• Character properties: Method on String

Monkey Patching

• Name: Guerilla patching → gorilla patching → monkey patching
• Adding or extending methods on existing classes
• Okay for short-lived scripts
• Frowned upon for libraries

⇒ Libraries are reluctant to patch core classes

⇒ Adding functionality to language core

Memory Footprint

• I18N operations can be very memory-intensive
• Normalization is not an exception
• Not every script needs normalization
• Avoid memory footprint where possible

⇒ Dynamic loading

Dynamic Loading to the Rescue (0)

Starting point: Add method to class String

class String
  def unicode_normalize(form)
    # implementation goes here
  end
end

Dynamic Loading to the Rescue (1)

Step 1: Separate invocation and implementation

class String
  def unicode_normalize(form)
    UnicodeNormalize.normalize(self, form)
  end
end

Implementation in separate module String::UnicodeNormalize (including internal methods)

Dynamic Loading to the Rescue (2)

Step 2: Conditionally require implementation (only on first call)

class String #reopen class String to add new methods
  def unicode_normalize(form = :nfc)
    unless defined? UnicodeNormalize
      require 'unicode_normalize/normalize.rb'
    end
    UnicodeNormalize.normalize(self, form)
  end
end

Dynamic Loading to the Rescue (3)

Speed optimization: Dynamically redefine method without require

class String #reopen class String to add new methods
  def unicode_normalize(form = :nfc)
    unless defined? UnicodeNormalize
      require 'unicode_normalize/normalize.rb'
    end
    String.send(:define_method,
      :unicode_normalize, ->(form = :nfc) do
        UnicodeNormalize.normalize(self, form)
      end
    )
    UnicodeNormalize.normalize(self, form)
  end
end

• :define_method: Same as (1), but using metaprogramming
• In practice, not used because actual speed gain minimal

We have to use metaprogramming (send, define_method) to redefine the method because Ruby doesn't allow to directly define a method inside a method.

Three New Methods

• Creating a new normalized String
string.unicode_normalize(form)
• 'Destructively' normalizing string
• string.unicode_normalize!(form)
• Checking string for normalization
  string.unicode_normalized?(form)

Why the unicode_ prefix?

• Added by Matz
• There are many kinds of normalization
• Normalization isn't a terribly frequent operation
• Normalization only works on Unicode encodings
  (UTF-8, UTF-16(BE|LE),...) and US-ASCII

Symbols for Normalization Forms

• Symbol:
  
  • Start with :
  • Very few operations
  • Interned (Lisp atoms)
  • Fast comparison for equality
  • Used to denote identifiers
• Normalization Forms:
  • :nfc, :nfd, :nfkc, :nfkd

Pure Ruby vs. Extension

• Pure Ruby: written only in Ruby
  • Easier to write
  • Portable among Ruby implementations
• Extension C (Java for JRuby,...):
  • Faster
  • Less memory

Efficient Pure Ruby

• Reanalyze algorithm
• Use built-in functionality
  
  • Global substitution (gsub)
  • Regular expressions
  • Hash
• Eprun (talk last year):
  
  • Up to 100 times faster than simple Ruby implementation
  • As fast as straightforward C implementation

Normalization in Various Languages

• Java
  import java.text.Normalizer
Normalizer.normalize(string, Normalizer.Form.NFC)
• C#
  using System.Text;
string.Normalize(NormalizationForm.FormKD);
• Perl (Unicode::Normalize)
  require Unicode::Normalize;
Unicode::Normalize->import(qw(NFD NFC NFKD NFKC));
NFKD($string);
• Python (unicodedata.normalize)
  import unicodedata
unicodedata.normalize('NFKD', string)
• Ruby
  string.unicode_normalize :nfkd

Ruby clearly is shorter. But it's not shortness for shortness' sake, it just feels better. Ruby is a programmer's best friend!

Case Study: Casing

• Besides upper case and lower case, also titlecase
  (upper: DŽ, lower: dž, title: Dž)
• Case may be language/locale-dependent
  (e.g. English: i↔I; Turkish: i↔İ, ı↔I)
• Case may depend on typographic tradition
  (accents on French uppercase letters,...)
• Case may not be character-to-character
  (e.g. ß↔SS)
• Case may be context-dependent
  e.g. σ↔Σ, but at the end of a word: ς↔Σ
• Case folding vs. case mapping
  Folding: aggressive, for search
  Mapping: Default+tailorings

Casing in Ruby Now

• Available methods:
  'Hello'.upcase ⇒ 'HELLO'
'Hello'.downcase ⇒ 'hello'
'hello'.capitalize ⇒ 'Hello'
'Hello'.swapcase ⇒ 'hELLO'
'Hello'.casecmp 'hello' ⇒ true +destructive variants
• Work on ASCII characters only, others get left as is
  ('München'.upcase ⇒ 'MüNCHEN')

Widening Method Interface

• Current methods don't take/need arguments
• Use argument to select details
• Symbols for specific conversions

  'München'.upcase :unicode ⇒ 'MÜNCHEN'

• Strings for language codes (BCP 47)
  'Türkiye'.upcase 'tr' ⇒ 'TÜRKİYE'
• Blocks for additional functionality
• Locale object in future versions?
  'Türkiye'.upcase tr_locale ⇒ 'TÜRKİYE'

In languages with strict typing (C++, Java,...), this can be achieved by method overloading.

Implementation Reuse

• Regular expressions already have casing data for /regexp/i option
• Looking at reusing that data
• Early stage
• Means most of implementation may be in C

One Encoding or All

• Matz would be okay to cover only UTF-8 (or Unicode encodings)
• Makes more sense to apply to all encodings than normalization

Default: To Unicode or not to Unicode

• Should 'München'.upcase (no argument) be
  'MÜNCHEN' or 'MüNCHEN'?
• Backwards compatibility suggests 'MüNCHEN'
  (default is ASCII-only)
• Matz would be okay with 'MÜNCHEN'
  (default is Unicode default case mapping)
• Should be okay because deliberate ASCII-only upcase should be very rare

Case Study: Properties

• Unicode provides a wide range of character properties
• Large part available in Ruby regular expressions
  (e.g. \p{Hiragana})
• Finds characters with give properties in String
• Currently impossible to ask for specific property of specific character
• Could be implemented because the data is (mostly) there

Case Study: Iterators

• Unicode provides various kinds of text segmentation (USA #29)
• Could be implemented as iterators on strings

Existing String Iterators

Strings can be cut up, and iterated over, in four ways:

Lines: "Юに코δ".lines.to_a
⇒ ["Юに코δ"]

Characters: "Юに코δ".chars.to_a
⇒ ["Ю", "に", "코", "δ"]

Codepoints: "Юに코δ".codepoints.to_a
⇒ [1070, 12395, 46020, 948]

Bytes: "Юに코δ".bytes.to_a
⇒ [208, 174, 227, 129, 171, 235, 143, 132, 206, 180]

As the result of chars shows, Ruby treats characters as Strings of length 1. They were integers up to Ruby 1.8. Using the same class for both strings and characters avoids the distinction between characters and strings of length 1. This matches Ruby's "big classes" policy. It also leaves the door open for 'characters' other than single codepoints.

each_line, each_char are older names for lines, chars, codepoints, and bytes. The methods lines, chars, codepoints, and bytes return Enumerators. Here we just use to_a to produce arrays. Enumerators can be used directly for iteration with each, or with separate iterators for mapping with map, selection with select/reject.

The need for several enumerators on a single object, resulting form the change of string/character representation between Ruby 1.8 and 1.9, was one of the main motivators for introducing enumerators into Ruby. This is an interesting example of how internationalization concerns can affect more 'fundamental' language features. The orthogonality resulting from separating what to enumerate over and how to iterate results in very expressive code.

Implementation Status

• Ruby 2.2.0
  • Preview1 released on Sept. 18
  • Preview2: Planned for mid-November:
    Includes string.unicode_normalize
  • Release candidate in early December?
  • Release planned for Christmas
• Ruby 2.3
  next year?

A note about Ruby versions and Unicode versions: The Ruby core team is very conservative (in my view too conservative) in introducing new Unicode versions as bug fixes. Update to new Unicode versions therefore only happens for new Ruby versions.

Conclusions

• More Unicode/I18N is coming to Ruby
• Trying to leverage Ruby's strong points
• Trying to leverage existing data
• Tell me what you want most

Q & A

Send questions and comments to Martin Dürst, duerst@it.aoyama.ac.jp

The latest version of this presentation is available at:

http://www.sw.it.aoyama.ac.jp/2014/pub/RubyI18N/

Acknowledgments

Ayumu Nojima (野島 歩) and Kimihito Matsui (松井 仁人) for help with research and implementations. Yui Naruse (成瀬 ゆい), Nobu Nakada (中田 伸悦) and many other Ruby committers for help and support. Matz (まつもと ゆきひろ) for Ruby, a programmer's best friend.

The IME Pad for facilitating character input.

Image credits: Christmas tree: Frits Ahlefeldt-Laurvig