Ruby 1.9: Convert byte array to string with multibyte UTF-8 characters

Question

I am trying to find a way in Ruby to take a UTF-8 byte array and transform it back to a string.

In irb (Ruby 1.9.2 preview 3) I can create the correct byte array from UTF-8 string:

ruby-1.9.2-preview3 > 'Café'.bytes.to_a
 => [67, 97, 102, 195, 169]

However, I can't find a way to roundtrip from bytes back to an array. I tried to use Array.pack with the U* option, but that doesn't work for multibyte characters.

ruby-1.9.2-preview3 > [67, 97, 102, 195, 169].pack('U*')
 => "CafÃ©"

Does anybody know a way to take a UTF-8 byte array with multibyte characters and convert it back to a string?

Thanks.

Jakob Borg · Accepted Answer · 2010-12-13 20:27:33Z

34

This has to do with how pack interprets its input data. The U* in your example causes it to convert the input data (assumed to be in a default character set, I assume; I really couldn't find any documentation of this) to UTF-8, thus the double encoding. Instead, just pack the bytes and interpret as UTF-8:

irb(main):010:0> [67, 97, 102, 195, 169].pack('C*').force_encoding('utf-8')
=> "Café"

answered Dec 13, 2010 at 20:27

Jakob Borg

24.7k6 gold badges49 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

steenslag · Accepted Answer · 2010-12-13 20:39:54Z

11

You specifically ask about a byte array, but maybe codepoints are more suitable:

ar = 'Café'.codepoints.to_a
# => [67, 97, 102, 233]
ar.pack('U*')
# => Café

answered Dec 13, 2010 at 20:39

steenslag

80.2k16 gold badges144 silver badges174 bronze badges

1 Comment

Charlie Over a year ago

I might use that on another project, but for this one I start out with the byte array, and have to work back to the string.

Collectives™ on Stack Overflow

Ruby 1.9: Convert byte array to string with multibyte UTF-8 characters

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related