I want to convert List[Map] to a spark dataframe,the keys of Map are sname,the keys of Map are DataFrame's columns
2 Answers
If you already have res which is a List[Map[String,String]] :
res: List[Map[String,String]] = List(Map(A -> a1, B -> b1, C -> c1), Map(A -> a2, B -> b2, C -> c2))
You can do this to create you dataframe:
//create your rows
val rows = res.map(m => Row(m.values.toSeq:_*))
//create the schema from the header
val header = res.head.keys.toList
val schema = StructType(header.map(fieldName => StructField(fieldName, StringType, true)))
//create your rdd
val rdd = sc.parallelize(rows)
//create your dataframe using
val df = spark.createDataFrame(rdd, schema)
You can output the result with df.show() :
+---+---+---+
| A| B| C|
+---+---+---+
| a1| b1| c1|
| a2| b2| c2|
+---+---+---+
Note that you can also create your schema in this way:
val schema = StructType(
List(
StructField("A", StringType, true),
StructField("B", StringType, true),
StructField("C", StringType, true)
)
)