scala-spark Array mapping

Question

I have a question about mapping Array in scala. I have the following Array:

Array[(scala.collection.immutable.Set[String], com.trends.City, com.trends.State)]

Basically, I want to map the Array such that each String in the Set will have com.trends.City and State attached to it. The result should look something like:

Array[(String, com.trends.City, com.trends.State)]

Which is like flatMap, but I want the com.trends to be in there.

I could also convert the Array into a RDD if needed and use flatMapValues, but I am concerned for the efficiency, could someone tell me what is the best way to go?

abalcerek · Accepted Answer · 2015-05-06 15:15:57Z

3

You can use flatMap on scala array like this:

class City
class State
val array: Array[(scala.collection.immutable.Set[String], City, State)] = Array()
array.flatMap(p => p._1.map(q => (q, p._2, p._3)))

answered May 6, 2015 at 15:15

abalcerek

1,8191 gold badge23 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Carlos Vilchez Over a year ago

I don't think this version works. It doesn't search if the string belongs to the Set, so it will always give a result, even if it shouldn't.

abalcerek Over a year ago

@CarlosVilchez can u explain what do u mean?

Carlos Vilchez Over a year ago

The Array[(scala.collection.immutable.Set[String], com.trends.City, com.trends.State)] has a Set[String] that determines if the String should be associated with that City and State. In your code you are iterating through the list of strings in the Set and then creating each possible result. I know it sounds complex, but bear with me. Nothing better than an example for this situation. Lets suppose we have this data: val cities = Array ("a", "b", "c") val cityArray = Array ((Set("x", "y"), "b", "c"), (Set("z"), "b2", "c2")) The result would be: Array((x,b,c), (y,b,c), (z,b2,c2))

abalcerek Over a year ago

@CarlosVilchez I used your example and got the same result with my solution

Carlos Vilchez Over a year ago

my bad. I misunderstood the problem. Your solution is correct.

Collectives™ on Stack Overflow

scala-spark Array mapping

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related