Internationalization and Localization
in Ruby and Ruby on Rails


IUC 39, Santa Clara, CA, U.S.A., 26 October 2015

Martin J. DÜRST


Ruby Programming Language

© 2012-5 Martin J. Dürst, Aoyama Gakuin University

Overview and Introduction




Ruby is a purely object-oriented scripting language designed to make programming fun and efficient. Ruby on Rails is the groundbreaking web application framework built using the programming language Ruby. This tutorial will help you understand the basics for internationalization and localization in Ruby and Ruby on Rails.

The tutorial will start with a discussion of how character encoding works in Ruby and how to make the best use of it both in throw-away scripts and in long-running applications. We will show how in Ruby, all character encodings are equal, but UTF-8 is more equal than others, and should be used with preference.

Ruby on Rails also preferably uses UTF-8, because this is the best choice for web applications. Ruby on Rails comes with its own internationalization and localization framework. As is typical for Ruby on Rails, this framework is very simple but easily extensible. We will show discuss both the basics framework as will as several helpful extensions, e.g. for handling timeliness or for translating user interface texts.

The tutorial assumes that participants have some experience with programming and Web applications. Experience with Ruby and/or Ruby on Rails is a plus, but is not a precondition for attending.

[Text appearing in gray are comments not showing up in presentation mode. The best way to view the slides as they were presented is with Opera, pressing F11.]






Conventions Used


Frequent Example Юに코δ


Ruby Basics



Ruby - A Programmer's Best Friend


Basic Ruby

5.times { puts "Hello Ruby!" }


Ruby Implementations

This tutorial is about MRI/C-Ruby, the reference implementation


Ruby Versions

A note about Ruby versions and Unicode versions: The Ruby core team is very conservative in introducing new Unicode versions as bug fixes. New versions therefore only get added on a minor version upgrade (e.g. 2.0, 2.1 (Unicode 6.0), 2.2 (Unicode 7.0), ...).


Up and Running


Basic Ruby Internationalization


String Basics

Using the same class for both strings and characters avoids the distinction between characters and strings of length 1. This matches Ruby's "big classes" policy. It also leaves the door open for 'characters' other than single codepoints. Strings are not Arrays, but where it makes sense, operations work the same for both classes. This is called duck typing.


Operations on Strings



Operations on Strings

Not yet internationalized:


Four Ways to Cut up Strings

Lines: "Unicode\nЮに코δ".lines.to_a
["Unicode\n", "Юに코δ"]

Characters: "Юに코δ".chars.to_a
["Ю", "に", "코", "δ"]

Codepoints: "Юに코δ".codepoints.map {|cp|
cp+1}.pack 'U*'
"Яぬ콕ε" (rotation by 1)

Bytes: "Юに코δ".bytes.inject {|m,b| m+b}
1865 (sum of byte values)

each_line, each_char, each_codepoint, and each_byte are older names for lines, chars, codepoints, and bytes. The methods lines, chars, codepoints, and bytes return Enumerators. Here we just use to_a to produce arrays. Enumerators can be used directly for iteration with each, or with separate iterators for mapping with map, selection with select/reject.

The need for several enumerators on a single object was one of the main motivators for introducing enumerators into Ruby. This is an interesting example of how internationalization concerns can affect more 'fundamental' language features. The orthogonality resulting from separating what to enumerate over and how to iterate produces very expressive code.


Escaping Syntax

(all for "Юに코δ")


Non-ASCII Identifiers

π = 3.14...

Possible, but not recommended (possible exceptions: basic education, special terminology)



Regular Expressions


Regular Expression Details

UTS #18 recommends: \s, \p{space}, \p{Whitespace} → Unicode whitespace

In Ruby: \s → ASCII whitespace; \p{space}, \p{Whitespace} → Unicode whitespace


"abc def" =~ /\s/3 (i.e. found)
"abc\u00A0def" =~ /\s/nil
"abc\u00A0def" =~ /\p{space}/3(i.e. found)
"abc\u00A0def" =~ /\p{Whitespace}/3(i.e. found)

Keeping \s to mean ASCII whitespace only was done for backwards compatibility. This can be explained as follows: If somebody wrote a script doing some processing where they wanted to match ASCII whitespace characters, they used \s. If Ruby would change \s to suddenly match more characters than before, the meaning of that program would change. Maybe it would change just in the right way. But there's also a good chance that it will change in ways not intended by the programmer. (See also https://bugs.ruby-lang.org/issues/7154.)


Tips on Character Processing


Speedup with Hash, Regexp, and gsub

Example: Convert German to ASCII-only

  1. Create Hash:
    german_hash = {'Ä'=>'Ae', 'ä'=>'ae',
    'Ö'=>'Oe', 'ö'=>'oe', 'Ü'=>'Ue',

    'ü'=>'ue', 'ß'=>'ss' }
  2. Create regular expression:
    german_regexp = /[ÄäÖöÜüß]/
  3. Use global substitution:
    my_german.gsub german_regexp, german_hash


Application of Speedup

See also: Implementing Normalization in Pure Ruby - the Fast and Easy Way, Martin J. Dürst, IUC 37, Santa Clara, Oct 2013


Character Encodings


Choices to Handle Other Encodings

  1. Convert to/from UTF-8 on input/outpt
  2. Work in other encoding throughout
    (helpful for quick scripts)
  3. Work with multiple encodings internally
    (not recommended)


Transcoding on Input/Output


More Details about String

Java, JavaScript: UTF-16; Python: 8-bit/UTF-16/UTF-32; Perl: 8-bit/UTF-8; C/C++: mb/wc



(Matz pushing for moving to UTF-8)


Other Objects with Encoding


Setting an Encoding

(Very simplified, more details later)

Unfortunately, force_encoding is destructive, no non-destructive equivalent.


Terminology: Transcoding


Encoding Clashes

"Юに코δ"UTF-8 + "Юに코δ"UTF-8

"Юに코δ"UTF-8 + "Юに코δ"UTF-16

Other examples:
"Dürst"ISO-8859-1 == "Dürst"ISO-8859-2false


Trying to combine strings with different encodings, as here with concatenation (+), leads to an exception. There are some exceptions (sic!) to this rule that we will look at later. The reasoning for the error here is that transcoding should not happen without the programmer being aware of it.

Trying to compare two character-by-character identical strings in different encodings will produce false, even if these strings are, as in the above example, also byte-for-byte identical. Again, the reason for the result is that encoding mismatches should be detected early. In addition, a simple byte-for-byte comparison could produce false positives.




Important Encodings

In Ruby, all encodings are equal, but some encodings are more equal than others.

[Adapted from George Orwell's Animal Farm.]

The important encodings are:


US-ASCII Encoding


ASCII-Only Data

ASCII-only data very frequent in programs

Special treatment, to reduce errors and improve performance

This also makes it easy for programmers who don't think about encoding when working only with US-ASCII. Internally, Ruby caches whether a string is ASCII-only or not, to increase performance.




ASCII-only does not Clash

ASCII-only data does not cause Encoding::CompatibilityErrors,
even if encodings clash

"Юに코δ"UTF-8 + "Data"SJIS

"Data"SJIS + "Юに코δ"UTF-8

Similar behavior for other string operations


Testing for Clashes

Test with Encoding.compatible? st1, st2

Encoding.compatible? "Юに코δ"UTF-8, "Data"SJIS ⇒ UTF-8




Used for binary data

Use when high-bit bytes' semantics are unknown (but 7-bit bytes are ASCII or can be treated as such)

Special shortcut (since Ruby 2.0):





Encodings as Objects

String to encoding:
e = Encoding.find "WinDOwS-1252"

Encoding to string: e.to_s"Windows-1252"

Methods most often accept both Strings and Encodings to indicate encodings

Constants for Encodings: Encoding::Windows_1252, Encoding::WINDOWS_1252

(We see that encoding names are case-insensitive, and are canonicalized. The internal canonicalization is specific to each encoding, and can start with a lower-case letter. Encodings are also exposed as constants in the Encoding module. There is always a constant with all letters in uppercase. There is also a constant that is the same or close to the internal canonical name, but starts with an uppercase letter. For the constants, hyphens are converted to underscores.)


Information about Encodings

List all encodings:


Count the encodings supported:


Ruby currently supports around 100 encodings


Encoding Aliases

Aliases are only separate names (not separate encodings)

To list all encoding names:
Encoding.name_list.join "\n"

To list all (active) aliases:
Encoding.aliases.each { |a, e|
puts "#{a} => #{e}" }


ASCII-Compatible Encodings

Most, but not all encodings are ASCII-compatible:

ASCII-compatible: ASCII characters≡ ASCII bytes


Non-ASCII-Compatible Encodings

List of non-ASCII-compatible encodings:

Encoding.list.reject &:ascii_compatible?

UTF-16BE, UTF-16LE, UTF-16
UTF-32BE, UTF-32LE, UTF-32
ISO-2022-JP, ISO-2022-JP-2, ISO-2022-JP-KDDI
CP50220, CP50221


Almost ASCII-Compatible Encodings


Dummy Encodings

List of dummy encodings:

Encoding.list.select &:dummy?

UTF-16[LE|BE], UTF-32[LE|BE], UTF-7
ISO-2022-JP, ISO-2022-JP-2, ISO-2022-JP-KDDI
CP50220, CP50221

Dummy encodings are treated as opaque byte sequences

Dummy encodings are encodings which are treated as opaque byte sequences. But they are labeled, and may be transcoded to some other encodings.


Exceptional Encodings


System Encodings


Source Encoding


Locale Encoding


External encoding


Internal encoding


Operations on IOs


Command-line Options


Filesystem Encoding


Encoding Integrity Check

"abc\u03A2"UTF-8.valid_encoding?true(there is no uppercase final Sigma (yet!?))

Checks for code structure, not unassigned codepoints


How to Avoid Invalid Data


Transcoding Details


Transcoding String to String


Explicit to and from Encodings

Example: Force UTF-8 "double-encoding" (not recommended)




"Юに코δ"UTF-8.encode('UTF-8', 'iso-8859-1')

Note order of arguments: to from

(I always disliked the order of arguments in iconv, but it was unavoidable here, because usually, just the 'to' encoding is needed, so that has to come first.)


Transcoding Failures


Fallback Options

Additional options for newline conversion and XML escaping


Adding an Encoding or Transcoding


Encoding Models


Encoding Models for Applications

An encoding model describes which encoding(s) can be used in what part of an application (e.g. externally, internally). It defines the conditions and restrictions with respect to string processing that the application (programmer) has to maintain. When creating a Ruby application, it is important to choose the appropriate encoding model.


Any Single Encoding


One Single Encoding

For applications that use more character semantics outside the ASCII range, or that keep data for a long time, the ideal solution is to use only a single encoding. This should be UTF-8 because that covers all of Unicode and works best with Ruby.


One Single Internal Encoding

If data (e.g. files) in other encodings also have to be handled, it will in most cases be best to adopt the model of most other programming languages: Use Unicode (i.e. UTF-8 for Ruby) inside, and convert on input/output.


Mixed Encodings

Ruby would also allow the creation of an application using many different encodings internally at the same time. However, this is not what the encoding model of Ruby was created for, and it should be avoided if at all possible.


Encoding Models for Libraries

Important: Don't touch system encodings
(except maybe for a framework)


Libraries for Unicode Support

Ruby leaves open some important i18n support:

Various librairies are available for such tasks, mostly in the form of Ruby Gems








Ruby on Rails


Rails and Encodings


Rails I18n Framework - Message Translation


Text Replacement: Templates

In templates, replace
<h1>Hello Rails!</h1>

<h1><%= t 'welcome.rails' %></h1>

welcome.rails is a structured key used for looking up the translated string

Tedious but simple!


Text Replacement: Backends

Where to get your text replacements from:


Locale Information


Ongoing/Future Work


Q & A

Send questions and comments to Martin Dürst, duerst@it.aoyama.ac.jp

The latest version of this tutorial is available at: