PySpark issue loading json data with schema

Question

I have json data that looks like this (1 object per row):

{
  "id": "c428c2e2-c30c-4864-8c12-458ead4b17f5",
  "weight": 73,
  "topics": {
    "type": 1,
    "values": [
      1,
      2,
      3
    ]
  }
}

When I read in the data without a specified schema, Spark infers topics.values to be an ArrayType but I need it to be a VectorUDT for doing ML tasks. So I am trying to read in the data set using a schema as follows:

    schema = StructType([
        StructField("id", StringType()),
        StructField("weight", IntegerType()),
        StructField("topics", StructType([
            StructField("type", IntegerType()),
            StructField("values", VectorUDT())
        ]))
    ])

When I do this I see the type (using dtype) of the data frame as follows:

[('id', 'string'), ('weight', 'int'), ('topics', 'struct<type:int,values:vector>')]

But there seems to be no actual data in the data frame, as show by using first():

Row(id=None, weight=None, topics=None)

And when I write the data frame to disk, I just see empty braces on each line. Seems odd! What am I doing wrong?

It is not odd. You pass schema which is not applicable for JSON document. — user6022341
– user6022341, Commented Aug 31, 2016 at 17:52
@LostInOverflow Can you elaborate? Obviously I am here asking a question because I don't know that. — Evan Zamir
– Evan Zamir, Commented Aug 31, 2016 at 18:00
@LostInOverflow Well, your comment did make me realize how to do this correctly. So thanks for that. — Evan Zamir
– Evan Zamir, Commented Aug 31, 2016 at 18:04
Glad it was helpful and sorry I didn't have more definitive suggestion. — user6022341
– user6022341, Commented Aug 31, 2016 at 18:40

Evan Zamir · Accepted Answer · 2016-08-31 18:05:17Z

1

Well, I figured it out:

Just needed to change the schema a bit:

schema = StructType([
                StructField("id", StringType()),
                StructField("weight", DoubleType()),
                StructField("topics", VectorUDT())
            ])

Now it makes sense.

answered Aug 31, 2016 at 18:05

Evan Zamir

8,52114 gold badges63 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Arjjun Over a year ago

whY VectorUDT? Can you explain logic behind it? I checked the documentation here: spark.apache.org/docs/1.5.0/api/java/org/apache/spark/mllib/…

Collectives™ on Stack Overflow

PySpark issue loading json data with schema

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related