Ruby 2.0.0 String#Match ArgumentError: invalid byte sequence in UTF-8

Question

I see this a lot and haven't figured out a graceful solution. If user input contains invalid byte sequences, I need to be able to have it not raise an exception. For example:

# @raw_response comes from user and contains invalid UTF-8
# for example: @raw_response = "\xBF"  
regex.match(@raw_response)
ArgumentError: invalid byte sequence in UTF-8

Numerous similar questions have been asked and the result appears to be encoding or force encoding the string. Neither of these work for me however:

regex.match(@raw_response.force_encoding("UTF-8"))
ArgumentError: invalid byte sequence in UTF-8

or

regex.match(@raw_response.encode("UTF-8", :invalid=>:replace, :replace=>"?"))
ArgumentError: invalid byte sequence in UTF-8

Is this a bug with Ruby 2.0.0 or am I missing something?

What is strange is it appear to be encoding correctly, but match continues to raise an exception:

@raw_response.encode("UTF-8", :invalid=>:replace, :replace=>"?").encoding
 => #<Encoding:UTF-8>

You tag this as ruby-on-rails, but normally any rails form will post utf-8 to the server. How does your form look? Or what is the origin of your strings. — nathanvda
– nathanvda, Commented Jun 4, 2014 at 12:35
This happens several different ways in our app: user agent strings and meta information on files. — Tom Rossi
– Tom Rossi, Commented Jun 4, 2014 at 12:36
mdesantis sure, just use any invalid string: @raw_response = "\xBF" — Tom Rossi
– Tom Rossi, Commented Jun 4, 2014 at 12:39

matt · Accepted Answer · 2014-06-04 12:47:22Z

51

In Ruby 2.0 the encode method is a no-op when encoding a string to its current encoding:

Please note that conversion from an encoding enc to the same encoding enc is a no-op, i.e. the receiver is returned without any changes, and no exceptions are raised, even if there are invalid bytes.

This changed in 2.1, which also added the scrub method as an easier way to do this.

If you are unable to upgrade to 2.1, you’ll have to encode into a different encoding and back in order to remove invalid bytes, something like:

if ! s.valid_encoding?
  s = s.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
end

answered Jun 4, 2014 at 12:47

matt

80k8 gold badges169 silver badges199 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Tom Rossi Over a year ago

Thanks @matt, thats exactly what I needed! A bit of a hack, but I'll take it.

Aleksey Over a year ago

@matt thank you! It can be any encoding, not exactly UTF-16be? How should I choose one?

Yogh · Accepted Answer · 2014-11-12 19:14:42Z

11

Since you're using Rails and not just Ruby you can also use tidy_bytes. This works with Ruby 2.0 and also will probably give you back sensible data instead of just replacement characters.

answered Nov 12, 2014 at 19:14

Yogh

5916 silver badges17 bronze badges

2 Comments

dancow Over a year ago

I feel compelled to give you a big belated thank you. Knowing about "tidy_bytes" earlier would've saved me many hours of frustration...can't believe I hadn't heard of it until just now.

Tom Lehman Over a year ago

tidy_bytes is amazing! I wish I knew about it earlier too lol.

Collectives™ on Stack Overflow

Ruby 2.0.0 String#Match ArgumentError: invalid byte sequence in UTF-8

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related