Convert text file to sequence array format in Spark Scala

Question

i have sample.txt:

how to convert this to become sequence array same as

(Seq( Array(Array(1), Array(2), Array(3)),
      Array(Array(1), Array(3), Array(2), Array(1), Array(2)),
      Array(Array(1), Array(2), Array(5)),
      Array(Array(6) )

i want to try using text file for prefixSpan mllib Spark, check this

Shadowlands · Accepted Answer · 2016-01-04 10:53:55Z

3

Try:

val file = new java.io.File("path/to/sample.txt")
Source.fromFile(file).getLines().map(_.split(' ').map(s => Array(s.toInt)))

This will actually produce an iterator (of type Iterator[Array[Array[Int]]]), but that can be converted to a sequence using .toSeq or .toList or similar.

Working with Spark (which I am not in a position to check with right now), this should become something like:

val data = sc.textFile("...")
data.map(_.split(' ').map(s => Array(s.toInt)))

edited Jan 4, 2016 at 10:53

answered Jan 4, 2016 at 10:18

Shadowlands

15.1k4 gold badges47 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Muhammad Nur Akbar Over a year ago

i have try like this : val data = sc.textFile("src/data/sample_prefixspan.txt") val transactions = data.map(s => s.trim.split(' ')).map(Array(_) its work, but the result same as (Seq( Array(1,2,3)), Array(1,3,2,1,2), Array(1,2,5), Array(6)) do you have any advice?

Shadowlands Over a year ago

Sorry, I'm not sure what you are saying in your comment - it looks like you are on the right track, but I can't tell quite what the remaining problem is.

Muhammad Nur Akbar Over a year ago

I mean, I have to try and have a result like this Seq( Array(Array(1,2,3)), not Seq( Array(Array(1), Array(2), Array(3)),

Muhammad Nur Akbar Over a year ago

I just need to change a little bit of code and its works thanks

Shadowlands Over a year ago

Ah, that is different from what you show in your question - just change s => Array(s.toInt) to _.toInt.

Collectives™ on Stack Overflow

Convert text file to sequence array format in Spark Scala

i have sample.txt:

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

i have sample.txt:

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related