How to add multidimensional array to an existing Spark DataFrame

Question

If I understand correctly, ArrayType can be added as Spark DataFrame columns. I am trying to add a multidimensional array to an existing Spark DataFrame by using the withColumn method. My idea is to have this array available with each DataFrame row in order to use it to send back information from the map function.

The error I get says that the withColumn function is looking for a Column type but it is getting an array. Are there any other functions that will allow adding an ArrayType?

    object TestDataFrameWithMultiDimArray {
  val nrRows = 1400
  val nrCols = 500

  /** Our main function where the action happens */
  def main(args: Array[String]) {

    // Create a SparkContext using every core of the local machine, named RatingsCounter
    val sc = new SparkContext("local[*]", "TestDataFrameWithMultiDimArray")  
    val sqlContext = new SQLContext(sc)

    val PropertiesDF = sqlContext.read
    .format("com.crealytics.spark.excel")
    .option("location", "C:/Users/tjoha/Desktop/Properties.xlsx")
    .option("useHeader", "true")
    .option("treatEmptyValuesAsNulls", "true")
    .option("inferSchema", "true")
    .option("addColorColumns", "False")
    .option("sheetName", "Sheet1")
    .load()

    PropertiesDF.show()
    PropertiesDF.printSchema()

    val PropertiesDFPlusMultiDimArray = PropertiesDF.withColumn("ArrayCol", Array.ofDim[Any](nrRows,nrCols))

  }

Thanks for your help.

Kind regards,

Johann

Fermat's Little Student · Accepted Answer · 2018-10-31 12:33:14Z

1

There are 2 problems in your code

the 2nd argument to withColumn needs to be a Column. you can wrap constant value with function col
Spark cant take Any as its column type, you need to use a specific supported type.

val PropertiesDFPlusMultiDimArray = PropertiesDF.withColumn("ArrayCol", lit(Array.ofDim[Int](nrRows,nrCols)))

will do the trick

edited Oct 31, 2018 at 12:33

answered Oct 30, 2018 at 20:06

Fermat's Little Student

6,3697 gold badges53 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

TJVR Over a year ago

Hi Will, thanks for your answer. What does lit() do? My end-goal is to calculate multiple rows and columns of values for each line of data in the DataFrame and to return it as an array. There will be multiple different types of data in the array, including strings, integers and floating point numbers. Do you have any idea as to achieve this type of functionality. Also, is the array you set up above compatible with a multi dimensional array? Once I have answers I will create a new post as the answer led to new questions.

TJVR Over a year ago

I see lit() creates a column of literal value.

Fermat's Little Student Over a year ago

@TJVR array in statically typed language always have the same type for all elements. Maybe you want to either create a new StructType for this column, or add all array as a column each instead?

WestCoastProjects Over a year ago

wow how did you dig up that ability to specify a type within the literal function?

Collectives™ on Stack Overflow

How to add multidimensional array to an existing Spark DataFrame

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related