Exclude column from json string?

Question

I have a field in dataframe that contains json. Is there a way to exclude some column from json?

Input:

{"column1":"data", "column2":"data"}

Expected output:

{"column1":"data"}

What version of Spark are you using?

Nithish
– Nithish

2022-01-12 10:38:52 +00:00
Commented Jan 12, 2022 at 10:38 — Nithish
– Nithish, Commented Jan 12, 2022 at 10:38

blackbishop · Accepted Answer · 2022-01-12 13:00:52Z

1

You can convert the json into MapType with from_json function, filter the map to exclude the columns you want, then convert back to json using to_json:

import pyspark.sql.functions as F

df = spark.createDataFrame([('{"column1":"data", "column2":"data"}',)], ["json_col"])

cols_to_exclude = ["column2"]

df1 = df.withColumn(
    "json_col",
    F.from_json("json_col", "map<string,string>")
).withColumn(
    "json_col",
    F.to_json(
        F.map_filter("json_col", lambda k, v: ~k.isin(cols_to_exclude))
    )
)

df1.show()
#+------------------+
#|          json_col|
#+------------------+
#|{"column1":"data"}|
#+------------------+

answered Jan 12, 2022 at 13:00

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Exclude column from json string?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related