How to create schema Array in data frame with spark

Question

I have code to create data frame and this works fine if there is no array in my input data.

I tried using Json data which doen't have array and it runs successfully. My code is

val vals = sc.parallelize(
  """{"id":"1","name":"alex"}""" ::
  Nil
)

val schema = (new StructType)
      .add("id", StringType)
      .add("name", StringType)


  sqlContext.read.schema(schema).json(vals).select($"*").printSchema()

My question is, if I have input data with array like below then how to create schema?

     val vals = sc.parallelize(
  """{"id":"1","name":"alex","score":[{"keyword":"read","point":10}]}""" ::
  Nil
)


val schema = (new StructType)
      .add("id", StringType)
      .add("name", StringType)

Thanks.

there are so many articles on it on the web. what have you tried? — Samar
– Samar, Commented Sep 14, 2016 at 8:17

RJK · Accepted Answer · 2016-09-14 08:46:32Z

7

Oke, I could have solution in my code.

Create schema in array in data frame spark you can this code.

val vals = sc.parallelize(
  """{"id":"1","name":"alex","score":[{"keyword":"read","point":10}]}""" ::
  Nil
)

val schema = StructType(
      Array(
        StructField("id", StringType),
        StructField("name", StringType),
        StructField("score", ArrayType(StructType(Array(
          StructField("keyword", StringType),
          StructField("point", IntegerType)
        ))))
      )
    )

and you print schema

sqlContext.read.schema(schema).json(vals).select($"*").printSchema()

Thanks is resolved

answered Sep 14, 2016 at 8:46

RJK

2392 gold badges3 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to create schema Array in data frame with spark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related