Spark Java: Passing variable number of arguments to a function

Question

Please see section "Programmatically Specifying the Schema". Java section.

The example works. But I have a question about this particular code snippet.

JavaRDD<Row> rowRDD = people.map(
new Function<String, Row>() {
public Row call(String record) throws Exception {
String[] fields = record.split(",");
  return Row.create(fields[0], fields[1].trim());
}

The Row create method is being called with a static number of objects determined at compile time.

However, in my code, I need to call the Row.create method for a dynamic number of arguments.

I will only know the number of fields at run time

For example, it may be one of:

return Row.create(fields[0], fields[1].trim(), fields[2]);

or

return Row.create(fields[0]);

or

return Row.create(fields[0],fields[1].trim(), fields[2], fields[3],fields[4]);

How do I do it?

Some people have suggested that "three dots after the argument" will solve my problem. It wont. The method i am calling already accepts a dynamic number of arguments. The problem is in calling. I wont know how many arguments to pass to it at compile time. I will know it at runtime. The problem is not the function. — Sit
– Sit, Commented Oct 7, 2014 at 7:26

Bibudesh Rohit · Accepted Answer · 2015-01-20 13:24:58Z

1

Here is how you can do it. Worked for me.

JavaRDD<Row> rowRDD = people.map(
  new Function<String, Row>() {
   public Row call(String record) throws Exception {
     String[] fields = record.split(",");         
    //return Row.create(fields[0], fields[1].trim());
      Object[] fields_converted = fields;
      return Row.create(fields_converted);
      }
      });

answered Jan 20, 2015 at 13:24

Bibudesh Rohit

111 bronze badge

Sign up to request clarification or add additional context in comments.

Comments

Qadir Hussain · Accepted Answer · 2014-10-07 06:01:06Z

0

Try using elipsis in your implemented method as below.

public static void create(String ...arg) { ... }

Elipsis accept n number of arguments.

answered Oct 7, 2014 at 6:01

Qadir Hussain

1,2611 gold badge10 silver badges26 bronze badges

Comments

SamTebbs33 · Accepted Answer · 2014-10-07 06:04:21Z

0

You can specify a method to take multiple arguments by using three dots after the argument, for example:

public static <return_type> create(String...args){
    // Yoo can now use the String[] args
}

Replace with your desired return type. Please change the signature of your call method as you have not specified a return type for it!

answered Oct 7, 2014 at 6:04

SamTebbs33

5,6473 gold badges24 silver badges47 bronze badges

Comments

Notinlist · Accepted Answer · 2014-12-16 16:20:28Z

0

Here is what I did in the same situation

new Function<String, Row>(String s) {
    public Row call(String s){
        int n = /* width of actual schema */
        Object rec[] = new Object[n];
        for( int i = 0; i < n; ++i )
            rec[i] = /* Something that aligns with the type of #i field */
        return Row.create( rec );
    }
}

There might be dragons here. My version compiles, looks good, not tested yet.

answered Dec 16, 2014 at 16:20

Notinlist

16.7k11 gold badges59 silver badges104 bronze badges

Collectives™ on Stack Overflow

Spark Java: Passing variable number of arguments to a function

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related