There are many examples on how to create empty dataframe/Dataset using Spark Scala/Python. But I would like to know how to create an empty dataframe/Dataset in Java Spark.
I have to create an empty dataframe with just one column with header as Column_1 and type String.
Another approach to create an empty dataset with specified schema in Java is given in this answer.
Once you created the schema of type StructType, use
Dataset<Row> emptyDataSet = spark.createDataFrame(new ArrayList<>(), schema);
empty dataframe with the user defined schema// alternative - 1
StructType s = new StructType()
.add(new StructField("Column_1", DataTypes.StringType, true, Metadata.empty()));
Dataset<Row> csv = spark.read().schema(s).csv(spark.emptyDataset(Encoders.STRING()));
csv.show(false);
csv.printSchema();
/**
* +--------+
* |Column_1|
* +--------+
* +--------+
*
* root
* |-- Column_1: string (nullable = true)
*/
null value and user defined schema Dataset<Row> df4 = spark.sql("select cast(null as string) Column_1");
df4.show(false);
df4.printSchema();
/**
* +--------+
* |Column_1|
* +--------+
* |null |
* +--------+
*
* root
* |-- Column_1: string (nullable = true)
*/
ClassTag<Row> rowTag = scala.reflect.ClassTag$.MODULE$.apply(Row.class);
Dataset<Row> df5 = spark.createDataFrame(spark.sparkContext().emptyRDD(rowTag),
new StructType()
.add(new StructField("Column_1", DataTypes.StringType, true, Metadata.empty())));
df5.show(false);
df5.printSchema();
/**
* +--------+
* |Column_1|
* +--------+
* +--------+
*
* root
* |-- Column_1: string (nullable = true)
*/
spark.emptyDataframe to create dataframe without any column and value Dataset<Row> rowDataset = spark.emptyDataFrame();
rowDataset.show(false);
rowDataset.printSchema();
/**
* ++
* ||
* ++
* ++
*
* root
*/
