Spark : converting Array[Byte] data to RDD or DataFrame

Question

I have data in the form of Array[Byte] which I want to convert into Spark RDD or DataFrame so that I can write my data directly into a Google bucket in the form of a file. I am not able to write Array[Byte] data into Google bucket directly. So looking for this conversion.

My below code is able to write data into Local FS, but not Google bucket

val encrypted = encrypt(original, readPublicKey(pubKey), outFile, true, true)
val dfis = new FileOutputStream(outFile)
dfis.write(encrypted)
dfis.close()

def encrypt(clearData: Array[Byte], encKey: PGPPublicKey, fileName: String, withIntegrityCheck: Boolean, armor: Boolean): Array[Byte] = {
...
}

So how can I convert Array[Byte] data to RDD or DataFrame? I am using Scala.

chlebek · Accepted Answer · 2019-11-01 20:29:47Z

2

just use .toDF() or .toDF().rdd

scala> val arr: Array[Byte] = Array(192.toByte, 168.toByte, 1.toByte, 4.toByte)
arr: Array[Byte] = Array(-64, -88, 1, 4)

scala> val df = arr.toSeq.toDF()
df: org.apache.spark.sql.DataFrame = [value: tinyint]

scala> df.show()
+-----+
|value|
+-----+
|  -64|
|  -88|
|    1|
|    4|
+-----+


scala> df.printSchema()
root
 |-- value: byte (nullable = false)

answered Nov 1, 2019 at 20:29

chlebek

2,4511 gold badge10 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2531569 Over a year ago

Thanks a lot for your input.scala> val arr: Array[Byte] = Array(192.toByte, 168.toByte, 1.toByte, 4.toByte) arr: Array[Byte] = Array(-64, -88, 1, 4) scala> val df = arr.toSeq.toDF() <console>:19: error: value toDF is not a member of Seq[Byte] val df = arr.toSeq.toDF() ^ When tried on my env facing this issue. Any idea how can I get rid of this?

Collectives™ on Stack Overflow

Spark : converting Array[Byte] data to RDD or DataFrame

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related