Use content of binary as string in DataFrame in pyspark

Question

I created a DataFrame from an existing table where two columns are stored as binary. For the further processing I need the binary-content as string.

Example:

+--------------------+--------------------+
|              DB_KEY|          PARENT_KEY|
+--------------------+--------------------+
|[00 50 56 88 0A]    |[00 50 56 88 12]    |

Schema:
root
 |-- DB_KEY: binary (nullable = true)
 |-- PARENT_KEY: binary (nullable = true)

Content of binary should be used as string like:

DB_KEY = "005056880A"
PARENT_KEY = "0050568812"

Can you please give me any hints how to do this?

mazaneicha · Accepted Answer · 2020-06-15 18:35:26Z

4

Have you tried to hex() your binary fields?

scala> val df = spark.sql("select unhex('005056880A') as db_key")
df: org.apache.spark.sql.DataFrame = [db_key: binary]

scala> df.withColumn("db_key_string", hex($"db_key")).show(false)
+----------------+-------------+
|db_key          |db_key_string|
+----------------+-------------+
|[00 50 56 88 0A]|005056880A   |
+----------------+-------------+

edited Jun 15, 2020 at 18:35

answered Jun 15, 2020 at 18:28

mazaneicha

9,5604 gold badges38 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Use content of binary as string in DataFrame in pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related