Convert DataFrame of JSON Strings

Question

Is it possible to convert a DataFrame containing JSON strings to a DataFrame containing a typed representation of the JSON strings using Spark 2.4?

For example: given the definition below, I'd like to convert the single column in jsonDF using a schema that is inferred from the JSON string.

val jsonDF = spark.sparkContext.parallelize(Seq("""{"a": 1, "b": 2}""")).toDF

If people who downvote would explain why they are down-voting, that would be helpful and encourage a better community. — thestephenstanton
– thestephenstanton, Commented May 6, 2019 at 14:08

ollik1 · Accepted Answer · 2019-05-03 14:28:46Z

1

DataFrameReader can read JSON from string data sets. For example using toDS instead of toDF

val jsonDS = Seq("""{"a": 1, "b": 2}""").toDS
spark.read.json(jsonDS).show()

Output:

+---+---+
|  a|  b|
+---+---+
|  1|  2|
+---+---+

answered May 3, 2019 at 14:28

ollik1

4,5601 gold badge12 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1