StructType from Array

Question

What I need to do?

Create schema for a DataFrame that should look like this:

root
 |-- doubleColumn: double (nullable = false)
 |-- longColumn: long (nullable = false)
 |-- col0: double (nullable = true)
 |-- col1: double (nullable = true)
...

Columns with prefix col can vary in number. Their names are stored in an array ar: Array[String].

My attempt

val schema = StructType(
    StructField("doubleColumn", DoubleType, false) ::
    StructField("longColumn", LongType, false) ::
    ar.map(item => StructField(item, DoubleType, true)) // how to reduce it?
    Nil
)

I have a problem with the commented line (4), I don't know, how to pass this array.

would having a case class like this help: case class MySchema( doubleColumn:Double, longColumn:Long, ar:List[ArrayData] ) case class ArrayData(col:Double) — Achilleus
– Achilleus, Commented Feb 11, 2019 at 21:45
Is it really needed, if I'm only using it in this one place? — Andronicus
– Andronicus, Commented Feb 12, 2019 at 5:03

2 revs user11047646 · Accepted Answer · 2019-02-11 21:36:47Z

2

There is no need to reduce anything. You can just perpend a list of known columns: val

val schema = StructType(Seq(
    StructField("doubleColumn", DoubleType, false),
    StructField("longColumn", LongType, false)
  ) ++ ar.map(item => StructField(item, DoubleType, true))
)

You might also

ar.foldLeft(StructType(Seq(
  StructField("doubleColumn", DoubleType, false), 
  StructField("longColumn", LongType, false)
)))((acc, name) => acc.add(name, DoubleType, true))

Collectives™ on Stack Overflow

StructType from Array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related