Creating empty Spark dataframe and adding arbitrary values

Question

I have created an empty dataframe and then trying to add columns and values to the dataframe.

Dataset<Row> runMetadata = sparkSession.emptyDataFrame();
runMetadata = runMetadata.withColumn("record_count", lit(count));
runMetadata = 
         runMetadata.withColumn("start_time",lit(currTimestamp));
runMetadata.show();

count is long and currTimestamp is a timestamp.

But 'runMetadata.show()' is only showing the column names and no values. Also i am trying to save this dataframe in bigquery, there also no value is getting appended, also i am not seeing any errors in logs.

danielsepulvedab · Accepted Answer · 2022-02-01 12:51:49Z

2

When you use withColumn, you're defining a transformation that it's going to be applied to all the rows of your DataFrame. Your DataFrame has no rows, so there's no rows apply the new columns to.

If you want a DataFrame with just one row you can create it using the createDataFrame method from the SparkSession passing a list with only one element.

answered Feb 1, 2022 at 12:51

danielsepulvedab

6741 gold badge11 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Creating empty Spark dataframe and adding arbitrary values

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related