Load CSV data in to Dataframe and convert to Array using Apache Spark (Java)

Question

I have a CSV file with below data :

1,2,5  
2,4  
2,3

I want to load them into a Dataframe having schema of string of array

The output should be like below.

[1, 2, 5]  
[2, 4]  
[2, 3]

This has been answered using scala here: Spark: Convert column of string to an array

I want to make it happen in Java.
Please help

The question you have attached uses DSL. It'll be almost similar in Java as well. Did you try writing anything? If yes, what error did you get? — philantrovert
– philantrovert, Commented Dec 7, 2017 at 6:26
I was trying to load it a RDD and attach schema to it as below : JavaRDD<Row> rowRDD = sparkSession.read().textFile("D:\\sanjaya\\OAWorkspace\\spark-basics\\src\\main\\resources\\marketbasketdata.csv") .javaRDD().map((Function<String, Row>) record -> { String[] attributes = record.split(","); return RowFactory.create(Arrays.asList(attributes)); <br> — sanjaya
– sanjaya, Commented Dec 7, 2017 at 6:42

abaghel · Accepted Answer · 2017-12-09 07:54:00Z

4

Below is the sample code in Java. You need to read your file using spark.read().text(String path) method and then call the split function.

import static org.apache.spark.sql.functions.split;

public class SparkSample {
    public static void main(String[] args) {
        SparkSession spark = SparkSession
                .builder()
                .appName("SparkSample")
                .master("local[*]")
                .getOrCreate();
        //Read file
        Dataset<Row> ds = spark.read().text("c://tmp//sample.csv").toDF("value");
        ds.show(false);     
        Dataset<Row> ds1 = ds.select(split(ds.col("value"), ",")).toDF("new_value");
        ds1.show(false);
        ds1.printSchema();
    }
}

edited Dec 9, 2017 at 7:54

answered Dec 9, 2017 at 7:40

abaghel

15.4k2 gold badges56 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Matko Soric · Accepted Answer · 2019-01-03 12:54:33Z

0

you can use VectorAssembler class to create as array of features, which is particulary useful with pipelines:

val assembler = new VectorAssembler()
  .setInputCols(Array("city", "status", "vendor"))
  .setOutputCol("features")

https://spark.apache.org/docs/2.2.0/ml-features.html#vectorassembler

answered Jan 3, 2019 at 12:54

Matko Soric

1174 silver badges16 bronze badges

Collectives™ on Stack Overflow

Load CSV data in to Dataframe and convert to Array using Apache Spark (Java)

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related