How to create a Row from a List or Array in Spark using java

Question

In Java, I use RowFactory.create() to create a Row:

Row row = RowFactory.create(record.getLong(1), record.getInt(2), record.getString(3));

where "record" is a record from a database, but I cannot know the length of "record" in advance, so I want to use a List or an Array to create the "row". In Scala, I can use Row.fromSeq() to create a Row from a List or an Array, but how can I achieve that in Java?

Paul Roub · Accepted Answer · 2017-12-02 14:58:39Z

16

We often need to create Datasets or Dataframes in real world applications. Here is an example of how to create Rows and Dataset in a Java application:

// initialize first SQLContext
SQLContext sqlContext = ... 
StructType schemata = DataTypes.createStructType(
        new StructField[]{
                createStructField("NAME", StringType, false),
                createStructField("STRING_VALUE", StringType, false),
                createStructField("NUM_VALUE", IntegerType, false),
        });
Row r1 = RowFactory.create("name1", "value1", 1);
Row r2 = RowFactory.create("name2", "value2", 2);
List<Row> rowList = ImmutableList.of(r1, r2);
Dataset<Row> data = sqlContext.createDataFrame(rowList, schemata);

+-----+------------+---------+
| NAME|STRING_VALUE|NUM_VALUE|
+-----+------------+---------+
|name1|      value1|        1|
|name2|      value2|        2|
+-----+------------+---------+

edited Dec 2, 2017 at 14:58

Paul Roub

36.5k27 gold badges88 silver badges95 bronze badges

answered Dec 2, 2017 at 14:52

Andrushenko Alexander

2,03322 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BdEngineer Over a year ago

@thank you , in scala we will do sc.paralallize(List((x,y),(a,b))).toDF("col1","col2"), it is so simple , why these Row , JavaRDD and etc ? any simple way like that ?

Borja Over a year ago

You are saying you need to create Dataset in real world applications and making a hard definition of the all variables. Does not make any sense. In the real world everything has to be parameterizable and beforehand you do not know the values.

abaghel · Accepted Answer · 2016-09-26 11:52:15Z

12

I am not sure if I get your question correctly but you can use the RowFactory to create Row from ArrayList in java.

List<MyData> mlist = new ArrayList<MyData>();
    mlist.add(d1);
    mlist.add(d2);

Row row = RowFactory.create(mlist.toArray());

edited Sep 26, 2016 at 11:52

answered Sep 26, 2016 at 7:48

abaghel

15.4k2 gold badges56 silver badges69 bronze badges

4 Comments

user2736706 Over a year ago

hi, when I use your method, I found spark regard mlist as a whole object: Row row = RowFactory.create(mlist); System.out.println("row number:" + row.length()); System.out.println("mlist number:" + mlist.size()); I got: row number:1 mlist number:2

abaghel Over a year ago

Yes but Row will have both records.You can try printing System.out.println("row number:" + row.toSeq());

user2736706 Over a year ago

hi, thanks so much! And you can try this: Object[] rowArray = {obj1, obj2, ....} Row row = RowFactory.create(rowArray); System.out.println("row number:" + row.length()); You will get - row number:6

abaghel Over a year ago

Thanks. I updated my answer. I checked the source code for RowFactory and GenericRow class.-"An internal row implementation that uses an array of objects as the underlying storage."

Sanjay Singh · Accepted Answer · 2019-10-21 18:24:54Z

0

//Create a a list of DTO

List<MyDTO> dtoList = Arrays.asList(.....));

//Create a Dataset of DTO

Dataset<MyDTO> dtoSet = sparkSession.createDataset(dtoList,
                Encoders.bean(MyDTO.class));

//If you need dataset of Row

Dataset<Row> rowSet= dtoSet .select("col1","col2","col3");

answered Oct 21, 2019 at 18:24

Sanjay Singh

1,05711 silver badges8 bronze badges

Comments

Alex Stanovsky · Accepted Answer · 2019-03-07 11:59:54Z

-1

For simple list values you can use Encoders:

 List<Row> rows = ImmutableList.of(RowFactory.create(new Timestamp(currentTime)));
 Dataset<Row> input = sparkSession.createDataFrame(rows, Encoders.TIMESTAMP().schema());

answered Mar 7, 2019 at 11:59

Alex Stanovsky

1,3962 gold badges15 silver badges29 bronze badges

Collectives™ on Stack Overflow

How to create a Row from a List or Array in Spark using java

4 Answers 4

2 Comments

4 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related