`scan': invalid byte sequence in UTF-8 (ArgumentError)

Question

I'm trying to read a .txt file in ruby and split the text line-by-line.

Here is my code:

def file_read(filename)
  File.open(filename, 'r').read
end

puts f = file_read('alice_in_wonderland.txt')

This works perfectly. But when I add the method line_cutter like this:

def file_read(filename)
  File.open(filename, 'r').read
end

def line_cutter(file)
  file.scan(/\w/)
end

puts f = line_cutter(file_read('alice_in_wonderland.txt'))

I get an error:

`scan': invalid byte sequence in UTF-8 (ArgumentError)

I found this online for untrusted website and tried to use it for my own code but it's not working. How can I remove this error?

Link to the file: File

The linked article was written in 2006, you are not using Ruby 1.8, are you? — Stefan
– Stefan, Commented Mar 18, 2016 at 14:59
It is better to do File.read(filename). That won't mess with the file kept open. — sawa
– sawa, Commented Mar 18, 2016 at 15:01

cremno · Accepted Answer · 2016-03-18 16:37:25Z

7

The linked text file contains the following line:

Character set encoding: ISO-8859-1

If converting it isn't desired or possible then you have to tell Ruby that this file is ISO-8859-1 encoded. Otherwise the default external encoding is used (UTF-8 in your case). A possible way to do that is:

s = File.read('alice_in_wonderland.txt', encoding: 'ISO-8859-1')
s.encoding  # => #<Encoding:ISO-8859-1>

Or even like this if you prefer your string UTF-8 encoded (see utf8everywhere.org):

s = File.read('alice_in_wonderland.txt', encoding: 'ISO-8859-1:UTF-8')
s.encoding  # => #<Encoding:UTF-8>

edited Mar 18, 2016 at 16:37

answered Mar 18, 2016 at 16:29

cremno

4,9871 gold badge20 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

JLB · Accepted Answer · 2016-03-18 16:22:41Z

2

It seems to work if you read the file directly from the page, maybe there's something funny about the local copy you have. Try this:

require 'net/http'

uri = 'http://www.ccs.neu.edu/home/vip/teach/Algorithms/7_hash_RBtree_simpleDS/hw_hash_RBtree/alice_in_wonderland.txt'
scanned = Net::HTTP.get_response(URI.parse(uri)).body.scan(/\w/)

answered Mar 18, 2016 at 16:22

JLB

3233 silver badges8 bronze badges

Collectives™ on Stack Overflow

`scan': invalid byte sequence in UTF-8 (ArgumentError)

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related