how to transform each column of a dataframe from binary to byte array

Question

I have a dataset ds1 with following schema

root
 |-- binary_col1: binary (nullable = true)

which i transform as needed using

val ds2 = ds1.map(row => row.getAs[Array[Byte]]("binary_col1"))

But how do I transform the dataset when it has two columns of binary type ?

root
 |-- binary_col1: binary (nullable = true)
  -- binary_col2: binary (nullable = false)

I want to create new dataset with 2 columns
( binary_col1.toByteArray , binary_col2.toByteArray)

mck · Accepted Answer · 2021-03-25 08:41:40Z

1

You can use as on the dataframe/dataset, and provide a tuple2 type:

val ds2 = ds1.as[(Array[Byte], Array[Byte])]

This is better than using map because it preserves column names.

Of course, you can also use map, e.g.

val ds2 = ds1.map(row => (row.getAs[Array[Byte]]("binary_col1"), row.getAs[Array[Byte]]("binary_col2")))

answered Mar 25, 2021 at 8:41

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1