Array to Tuple in Spark with many input variables

Question

Lets say I am importing a flat file from HDFS into spark using something like the following:

val data = sc.textFile("hdfs://name_of_file.tsv").map(_.split('\t'))

This will produce an Array[Array[String]]. If I wanted an array of tuples I could do as referenced in this solution and map the elements to a tuple.

val dataToTuple = data.map{ case Array(x,y) => (x,y) }

But what if my input data has say, 100 columns? Is there a way in scala using some sort of wildcard to say

val dataToTuple = data.map{ case Array(x,y, ... ) => (x,y, ...) }

without having to write out 100 variable to match on?

I tried doing something like

val dataToTuple = data.map{ case Array(_) => (_) }

but that didn't seem to make much sense.

Why would you want a tuple with 100 elements? Just use the array that split produces? — The Archetypal Paul
– The Archetypal Paul, Commented May 12, 2016 at 19:43
if you really need that - you can use Shapeless library: stackoverflow.com/a/19901310/1809978, but be aware that maximum size of tuple is limited in scala to 22 (last time I was checking it) + I believe, you still have to specify type per column. Besides, it might not be what you actually need — dk14
– dk14, Commented May 12, 2016 at 19:48

Community · Accepted Answer · 2017-05-23 11:59:50Z

1

If your data-columns are homogenous (like Array of Strings) - tuple may not be a best solution to improve type-safety. All you can do is to fix the size of your array using sized list from Shapeless library:

How to require typesafe constant-size array in scala?

This is a right approach if your column's are unnamed. For instance, your row might be a representation of a vector in Euclidean space.

Otherwise (named columns, maybe different types), it's better to model it with a case class, but be aware of size restriction. This might help you to quickly map array (or its parts) to ADT: https://stackoverflow.com/a/19901310/1809978

edited May 23, 2017 at 11:59

CommunityBot

11 silver badge

answered May 12, 2016 at 20:08

dk14

22.4k4 gold badges55 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Array to Tuple in Spark with many input variables

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related