How do you convert RDD[Array[Byte]] into Array[Byte]

Question

I am a bit stuck trying to convert RDD[Array[Byte]] into Array[Byte]. I have the following where I am first extracting the RDD[Array[Byte]] from a previously defined tuple in the form of RDD(String, Array[Byte]):

val extractArrayFromRDD: RDD[Array[Byte]] = rdd.map(t => t._2)

I then can really only get the first element of the array, as follows:

val rddToBytes: Array[Byte] = extractArrayFromRDD.first()

However, I indeed need to be returned with the entire Array, but I do not seem to find a way to do it. Any idea?

Thank you

Berthur · Accepted Answer · 2018-10-30 23:44:43Z

1

Not sure I understood you correctly, but if you want to collect your RDD you can obtain it as an Array using

rdd.collect()

which should return an Array[Array[Byte]]. And if you want the arrays combined into an Array[Byte], you can use flatten() or whatever suits your needs.

edited Oct 30, 2018 at 23:44

answered Oct 30, 2018 at 23:37

Berthur

4,5142 gold badges20 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

panza Over a year ago

thank you for your comment. Yeah I was not sure how to move from Array[Array[Byte]] to the desiderd data structure Array[Byte]. I am indeed fairly new to RDD manipulation and Scala programming in general.

Berthur Over a year ago

Glad you found it helpful. Notice however the implications of collecting: You're losing the benefits of distribution of the data, and moreover you have to make sure there is not too much data in the RDD to be collected to an array on the master. Ae a consequence, you should generally perform as much of the work as possible before collecting.

Collectives™ on Stack Overflow

How do you convert RDD[Array[Byte]] into Array[Byte]

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related