Parsing separated text file in Scala

Question

When i trying parse txt file at 200k line in file i have this error:

java.nio.charset.UnmappableCharacterException: Input length = 1

After i got error my program break:

val bufferedSource = io.Source.fromFile( path)
for (line <- bufferedSource.getLines.drop(1)) {
    line.split('|').toList.drop(1)
    }

If I understand correctly, the error is in io.Source.fromFile( path). How i can skip bad rows ?

Possible duplicate of How to resolve java.nio.charset.UnmappableCharacterException in Scala 2.8.0? — Simon
– Simon, Commented Feb 12, 2017 at 21:20
i have this error not at first row. I have error at 200k row — Kostiantyn Palianychka
– Kostiantyn Palianychka, Commented Feb 12, 2017 at 21:23
It still seems to be a character encoding issue. Either you need to handle non-ascii characters or make sure that your file only has acii values. — Simon
– Simon, Commented Feb 12, 2017 at 21:56
My solution is :import scala.io.Codec implicit val codec = Codec("cp1251") codec.onMalformedInput(CodingErrorAction.REPLACE) codec.onUnmappableCharacter(CodingErrorAction.REPLACE) — Kostiantyn Palianychka
– Kostiantyn Palianychka, Commented Feb 12, 2017 at 22:07

Sami Badawi · Accepted Answer · 2017-02-12 22:13:33Z

1

Unfortunately you have to deal with encodings issues yourself. One of these 2 encodings often work for me:

val bufferedSource = io.Source.fromFile( path, enc = Codec.UTF8.name)
for (line <- bufferedSource.getLines.drop(1)) {
    line.split('|').toList.drop(1)
    }

or

val bufferedSource = io.Source.fromFile( path, enc = Codec.ISO8859.name)

answered Feb 12, 2017 at 22:13

Sami Badawi

1,0321 gold badge14 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Parsing separated text file in Scala

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related