Tutorial

Internationalization and Localization
in Ruby and Ruby on Rails

http://www.sw.it.aoyama.ac.jp/2015/pub/RubyRails/

IUC 39, Santa Clara, CA, U.S.A., 26 October 2015

Martin J. DÜRST

duerst@it.aoyama.ac.jp

Ruby Programming Language

© 2012-5 Martin J. Dürst, Aoyama Gakuin University

Overview and Introduction

 

Overview

Abstract

Ruby is a purely object-oriented scripting language designed to make programming fun and efficient. Ruby on Rails is the groundbreaking web application framework built using the programming language Ruby. This tutorial will help you understand the basics for internationalization and localization in Ruby and Ruby on Rails.

The tutorial will start with a discussion of how character encoding works in Ruby and how to make the best use of it both in throw-away scripts and in long-running applications. We will show how in Ruby, all character encodings are equal, but UTF-8 is more equal than others, and should be used with preference.

Ruby on Rails also preferably uses UTF-8, because this is the best choice for web applications. Ruby on Rails comes with its own internationalization and localization framework. As is typical for Ruby on Rails, this framework is very simple but easily extensible. We will show discuss both the basics framework as will as several helpful extensions, e.g. for handling timeliness or for translating user interface texts.

The tutorial assumes that participants have some experience with programming and Web applications. Experience with Ruby and/or Ruby on Rails is a plus, but is not a precondition for attending.

[Text appearing in gray are comments not showing up in presentation mode. The best way to view the slides as they were presented is with Opera, pressing F11.]

 

Introductions

 

Goals

 

Conventions Used

 

Frequent Example Юに코δ

 

Ruby Basics

 

Ruby

Ruby - A Programmer's Best Friend

 

Basic Ruby

5.times { puts "Hello Ruby!" }

 

Ruby Implementations

This tutorial is about MRI/C-Ruby, the reference implementation

 

Ruby Versions

A note about Ruby versions and Unicode versions: The Ruby core team is very conservative in introducing new Unicode versions as bug fixes. New versions therefore only get added on a minor version upgrade (e.g. 2.0, 2.1 (Unicode 6.0), 2.2 (Unicode 7.0), ...).

 

Up and Running

 

Basic Ruby Internationalization

 

String Basics

Using the same class for both strings and characters avoids the distinction between characters and strings of length 1. This matches Ruby's "big classes" policy. It also leaves the door open for 'characters' other than single codepoints. Strings are not Arrays, but where it makes sense, operations work the same for both classes. This is called duck typing.

 

Operations on Strings

Internationalized:

 

Operations on Strings

Not yet internationalized:

 

Four Ways to Cut up Strings

Lines: "Unicode\nЮに코δ".lines.to_a
["Unicode\n", "Юに코δ"]

Characters: "Юに코δ".chars.to_a
["Ю", "に", "코", "δ"]

Codepoints: "Юに코δ".codepoints.map {|cp|
cp+1}.pack 'U*'
"Яぬ콕ε" (rotation by 1)

Bytes: "Юに코δ".bytes.inject {|m,b| m+b}
1865 (sum of byte values)

each_line, each_char, each_codepoint, and each_byte are older names for lines, chars, codepoints, and bytes. The methods lines, chars, codepoints, and bytes return Enumerators. Here we just use to_a to produce arrays. Enumerators can be used directly for iteration with each, or with separate iterators for mapping with map, selection with select/reject.

The need for several enumerators on a single object was one of the main motivators for introducing enumerators into Ruby. This is an interesting example of how internationalization concerns can affect more 'fundamental' language features. The orthogonality resulting from separating what to enumerate over and how to iterate produces very expressive code.

 

Escaping Syntax

(all for "Юに코δ")

 

Non-ASCII Identifiers

π = 3.14...

Possible, but not recommended (possible exceptions: basic education, special terminology)

Problems:

 

Regular Expressions

 

Regular Expression Details

UTS #18 recommends: \s, \p{space}, \p{Whitespace} → Unicode whitespace

In Ruby: \s → ASCII whitespace; \p{space}, \p{Whitespace} → Unicode whitespace

Examples:

"abc def" =~ /\s/3 (i.e. found)
"abc\u00A0def" =~ /\s/nil
"abc\u00A0def" =~ /\p{space}/3(i.e. found)
"abc\u00A0def" =~ /\p{Whitespace}/3(i.e. found)

Keeping \s to mean ASCII whitespace only was done for backwards compatibility. This can be explained as follows: If somebody wrote a script doing some processing where they wanted to match ASCII whitespace characters, they used \s. If Ruby would change \s to suddenly match more characters than before, the meaning of that program would change. Maybe it would change just in the right way. But there's also a good chance that it will change in ways not intended by the programmer. (See also https://bugs.ruby-lang.org/issues/7154.)

 

Tips on Character Processing

 

Speedup with Hash, Regexp, and gsub

Example: Convert German to ASCII-only

  1. Create Hash:
    german_hash = {'Ä'=>'Ae', 'ä'=>'ae',
    'Ö'=>'Oe', 'ö'=>'oe', 'Ü'=>'Ue',

    'ü'=>'ue', 'ß'=>'ss' }
  2. Create regular expression:
    german_regexp = /[ÄäÖöÜüß]/
  3. Use global substitution:
    my_german.gsub german_regexp, german_hash

 

Application of Speedup

See also: Implementing Normalization in Pure Ruby - the Fast and Easy Way, Martin J. Dürst, IUC 37, Santa Clara, Oct 2013

 

Character Encodings

 

Choices to Handle Other Encodings

  1. Convert to/from UTF-8 on input/outpt
  2. Work in other encoding throughout
    (helpful for quick scripts)
  3. Work with multiple encodings internally
    (not recommended)

 

Transcoding on Input/Output

 

More Details about String

Java, JavaScript: UTF-16; Python: 8-bit/UTF-16/UTF-32; Perl: 8-bit/UTF-8; C/C++: mb/wc

 

Why CSI

(Matz pushing for moving to UTF-8)

 

Other Objects with Encoding

 

Setting an Encoding

(Very simplified, more details later)

Unfortunately, force_encoding is destructive, no non-destructive equivalent.

 

Terminology: Transcoding

 

Encoding Clashes

"Юに코δ"UTF-8 + "Юに코δ"UTF-8
"Юに코δЮに코̃"UTF-8

"Юに코δ"UTF-8 + "Юに코δ"UTF-16
Encoding::CompatibilityError

Other examples:
"Dürst"ISO-8859-1 == "Dürst"ISO-8859-2false

πUTF-8πSJIS

Trying to combine strings with different encodings, as here with concatenation (+), leads to an exception. There are some exceptions (sic!) to this rule that we will look at later. The reasoning for the error here is that transcoding should not happen without the programmer being aware of it.

Trying to compare two character-by-character identical strings in different encodings will produce false, even if these strings are, as in the above example, also byte-for-byte identical. Again, the reason for the result is that encoding mismatches should be detected early. In addition, a simple byte-for-byte comparison could produce false positives.

 

Encodings

 

Important Encodings

In Ruby, all encodings are equal, but some encodings are more equal than others.

[Adapted from George Orwell's Animal Farm.]

The important encodings are:

 

US-ASCII Encoding

 

ASCII-Only Data

ASCII-only data very frequent in programs

Special treatment, to reduce errors and improve performance

This also makes it easy for programmers who don't think about encoding when working only with US-ASCII. Internally, Ruby caches whether a string is ASCII-only or not, to increase performance.

"Data"SJIS.ascii_only?true

"データ"SJIS.ascii_only?false

 

ASCII-only does not Clash

ASCII-only data does not cause Encoding::CompatibilityErrors,
even if encodings clash

"Юに코δ"UTF-8 + "Data"SJIS
"Юに코δData"UTF-8

"Data"SJIS + "Юに코δ"UTF-8
"DataЮに코δ"UTF-8

Similar behavior for other string operations

 

Testing for Clashes

Test with Encoding.compatible? st1, st2

Encoding.compatible? "Юに코δ"UTF-8, "Data"SJIS ⇒ UTF-8

 

ASCII-8BIT

Alias: BINARY

Used for binary data

Use when high-bit bytes' semantics are unknown (but 7-bit bytes are ASCII or can be treated as such)

Special shortcut (since Ruby 2.0):

string.b
string.dup.force_encoding('ASCII-8BIT')

 

UTF-8

 

Encodings as Objects

String to encoding:
e = Encoding.find "WinDOwS-1252"

Encoding to string: e.to_s"Windows-1252"

Methods most often accept both Strings and Encodings to indicate encodings

Constants for Encodings: Encoding::Windows_1252, Encoding::WINDOWS_1252

(We see that encoding names are case-insensitive, and are canonicalized. The internal canonicalization is specific to each encoding, and can start with a lower-case letter. Encodings are also exposed as constants in the Encoding module. There is always a constant with all letters in uppercase. There is also a constant that is the same or close to the internal canonical name, but starts with an uppercase letter. For the constants, hyphens are converted to underscores.)

 

Information about Encodings

List all encodings:

Encoding.list.join("\n")

Count the encodings supported:

Encoding.list.length

Ruby currently supports around 100 encodings

 

Encoding Aliases

Aliases are only separate names (not separate encodings)

To list all encoding names:
Encoding.name_list.join "\n"

To list all (active) aliases:
Encoding.aliases.each { |a, e|
puts "#{a} => #{e}" }

 

ASCII-Compatible Encodings

Most, but not all encodings are ASCII-compatible:
Encoding::UTF_8.ascii_compatible?
true
Encoding::UTF_16.ascii_compatible?
false

ASCII-compatible: ASCII characters≡ ASCII bytes
Encoding::Shift_JIS.ascii_compatible?
true
Encoding::ISO_2022_JP.ascii_compatible?
false

 

Non-ASCII-Compatible Encodings

List of non-ASCII-compatible encodings:

Encoding.list.reject &:ascii_compatible?

UTF-16BE, UTF-16LE, UTF-16
UTF-32BE, UTF-32LE, UTF-32
UTF-7
ISO-2022-JP, ISO-2022-JP-2, ISO-2022-JP-KDDI
CP50220, CP50221

 

Almost ASCII-Compatible Encodings

 

Dummy Encodings

List of dummy encodings:

Encoding.list.select &:dummy?

UTF-16[LE|BE], UTF-32[LE|BE], UTF-7
ISO-2022-JP, ISO-2022-JP-2, ISO-2022-JP-KDDI
CP50220, CP50221

Dummy encodings are treated as opaque byte sequences

Dummy encodings are encodings which are treated as opaque byte sequences. But they are labeled, and may be transcoded to some other encodings.

 

Exceptional Encodings

 

System Encodings

 

Source Encoding

 

Locale Encoding

 

External encoding

 

Internal encoding

 

Operations on IOs

 

Command-line Options

 

Filesystem Encoding

 

Encoding Integrity Check

"abc\xFE"UTF-8.valid_encoding?false
"a\xC0\x80"UTF-8.valid_encoding?false
"abc\xC2\x80"UTF-8.valid_encoding?true
"abc\u03A2"UTF-8.valid_encoding?true(there is no uppercase final Sigma (yet!?))

Checks for code structure, not unassigned codepoints

 

How to Avoid Invalid Data

 

Transcoding Details

 

Transcoding String to String

 

Explicit to and from Encodings

Example: Force UTF-8 "double-encoding" (not recommended)

"Юに코δ"UTF-8.force_encoding('iso-8859-1')
.encode('UTF-8')

"Юã\u0081«ì½\u0094δ"

Equivalent:

"Юに코δ"UTF-8.encode('UTF-8', 'iso-8859-1')

Note order of arguments: to from

(I always disliked the order of arguments in iconv, but it was unavoidable here, because usually, just the 'to' encoding is needed, so that has to come first.)

 

Transcoding Failures

 

Fallback Options

Additional options for newline conversion and XML escaping

 

Adding an Encoding or Transcoding

 

Encoding Models

 

Encoding Models for Applications

An encoding model describes which encoding(s) can be used in what part of an application (e.g. externally, internally). It defines the conditions and restrictions with respect to string processing that the application (programmer) has to maintain. When creating a Ruby application, it is important to choose the appropriate encoding model.

 

Any Single Encoding

 

One Single Encoding

For applications that use more character semantics outside the ASCII range, or that keep data for a long time, the ideal solution is to use only a single encoding. This should be UTF-8 because that covers all of Unicode and works best with Ruby.

 

One Single Internal Encoding

If data (e.g. files) in other encodings also have to be handled, it will in most cases be best to adopt the model of most other programming languages: Use Unicode (i.e. UTF-8 for Ruby) inside, and convert on input/output.

 

Mixed Encodings

Ruby would also allow the creation of an application using many different encodings internally at the same time. However, this is not what the encoding model of Ruby was created for, and it should be avoided if at all possible.

 

Encoding Models for Libraries

Important: Don't touch system encodings
(except maybe for a framework)

 

Libraries for Unicode Support

Ruby leaves open some important i18n support:

Various librairies are available for such tasks, mostly in the form of Ruby Gems

 

UnicodeUtils

 

Twitter_CLDR

 

ActiveSupport

 

Ruby on Rails

 

Rails and Encodings

 

Rails I18n Framework - Message Translation

 

Text Replacement: Templates

In templates, replace
<h1>Hello Rails!</h1>

with
<h1><%= t 'welcome.rails' %></h1>

welcome.rails is a structured key used for looking up the translated string

Tedious but simple!

 

Text Replacement: Backends

Where to get your text replacements from:

 

Locale Information

 

Ongoing/Future Work

 

Q & A

Send questions and comments to Martin Dürst, duerst@it.aoyama.ac.jp

The latest version of this tutorial is available at:

http://www.sw.it.aoyama.ac.jp/2015/pub/RubyRails/