2 values in an Array to 2 columns in my DataFrame

Question

Using Apache Spark in Java, I have:

root
 |-- datasetid: string (nullable = true)
 |-- fields: struct (nullable = true)
...
 |    |-- latlon: array (nullable = true)
 |    |    |-- element: double (containsNull = true)

This is based on this JSON fragment:

"fields":{
  "latlon":[
    35.9543748,
    -78.9944911
  ],

I was trying to extract the data to a column using:

df = df.withColumn("lat", df.col("fields.latlon[0]"));
df = df.withColumn("lon", df.col("fields.latlon[1]"));

(I hope you appreciate the neatness of the syntax). However, I must admit it does not really work:

No such struct field latlon[1] in

I tried a few other things without much luck...

Rockie Yang · Accepted Answer · 2016-07-05 05:04:02Z

1

Use code

df.withColumn("lat", $"fields.latlon".getItem(0))
  .withColumn("lon", $"fields.latlon".getItem(1))

Use sql

df.registerTempTable("geo")
latlon = sqlContext.sql("select fields.latlon[0] as lat, fields.latlon[1] as lon from geo")

edited Jul 5, 2016 at 5:04

answered Jul 5, 2016 at 4:40

Rockie Yang

4,94533 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jgp Over a year ago

Awesome @Rockie! I just changed the notation to be more Java and less Scala using the static method col(). Thanks for your help!

Collectives™ on Stack Overflow

2 values in an Array to 2 columns in my DataFrame

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related