Java - Spark SQL DataFrame map function is not working

Question

In Spark SQL when I tried to use map function on DataFrame then I am getting below error.

The method map(Function1, ClassTag) in the type DataFrame is not applicable for the arguments (new Function(){})

I am following spark 1.3 documentation as well. https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection Have any one solution?

Here is my testing code.

   // SQL can be run over RDDs that have been registered as tables.
DataFrame teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19");

List<String> teenagerNames = teenagers.map(
            new Function<Row, String>() {
          public String call(Row row) {
            return "Name: " + row.getString(0);
          }
        }).collect();

could you please provide your full code? btw which version os SPARK are you using? (>1.3.0) — Fabio Fantoni
– Fabio Fantoni, Commented Apr 22, 2015 at 8:46

econn · Accepted Answer · 2015-05-05 17:44:22Z

12

Change this to:

Java 6 & 7

List<String> teenagerNames = teenagers.javaRDD().map(
    new Function<Row, String>() {
    public String call(Row row) {
        return "Name: " + row.getString(0);
    }
}).collect();

Java 8

List<String> t2 = teenagers.javaRDD().map(
    row -> "Name: " + row.getString(0)
).collect();

Once you call javaRDD() it works just like any other RDD map function.

This works with Spark 1.3.0 and up.

answered May 5, 2015 at 17:44

econn

1212 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

GiCo Over a year ago

What happens if you transform it in an RDD? Is the transform lazy? Is the memory moved into a new structure? Can the execution still be optimize?

Vijay Anantharamu · Accepted Answer · 2017-12-23 03:55:38Z

2

No need to convert to RDD, its delays the execution it can be done as below

`public static void mapMethod() { // Read the data from file, where the file is in the classpath. Dataset df = sparkSession.read().json("file1.json");

// Prior to java 1.8 
Encoder<String> encoder = Encoders.STRING();
    List<String> rowsList = df.map((new MapFunction<Row, String>() {
        private static final long serialVersionUID = 1L;

        @Override
        public String call(Row row) throws Exception {
            return "string:>" + row.getString(0).toString() + "<";
        }
    }), encoder).collectAsList();

// from java 1.8 onwards
List<String> rowsList1 = df.map((row -> "string >" + row.getString(0) + "<" ), encoder).collectAsList();
System.out.println(">>> " + rowsList);
System.out.println(">>> " + rowsList1);

}`

answered Dec 23, 2017 at 3:55

Vijay Anantharamu

313 bronze badges

1 Comment

Amu Over a year ago

Which spark version are you using for this?

urug · Accepted Answer · 2015-04-22 13:38:55Z

0

Do you have the correct dependency set in your pom. Set this and try

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.10</artifactId>
        <version>1.3.1</version>
    </dependency>

answered Apr 22, 2015 at 13:38

urug

4157 silver badges18 bronze badges

2 Comments

user3206330 Over a year ago

I am using below dependencies with Java 1.7 [code] <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>1.3.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.11</artifactId> <version>1.3.1</version> </dependency> [/code]

urug Over a year ago

Documentation says you can run all the normal functions of JavaRDD against DataFrames but that does not appear to be the case here. I was able to reproduce your problem. map() method of DataFrame class expects 2 arguments. May be explicitly convert the Dataframe to RDD as teenagers.javaRDD() then apply the map.

Yassine Jouini · Accepted Answer · 2015-06-04 10:01:12Z

0

try this:

// SQL can be run over RDDs that have been registered as tables.
DataFrame teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19");

List<String> teenagerNames = teenagers.toJavaRDD().map(
        new Function<Row, String>() {
      public String call(Row row) {
        return "Name: " + row.getString(0);
      }
    }).collect();

you have to transforme your DataFrame to javaRDD

answered Jun 4, 2015 at 10:01

Yassine Jouini

313 bronze badges

Comments

Swaminathan S · Accepted Answer · 2016-02-04 11:02:47Z

0

check if you are using the correct import for

Row(import org.apache.spark.sql.Row) Remove any other imports related to Row.otherwise ur syntax is correct

answered Feb 4, 2016 at 11:02

Swaminathan S

2353 silver badges11 bronze badges

Comments

ankitbeohar90 · Accepted Answer · 2016-04-12 06:45:34Z

0

Please check your input file's data and your dataframe sql query same thing I am facing and when I look back to the data so it was not matching with my query. So probably same issue you are facing. toJavaRDD and JavaRDD both are working.

answered Apr 12, 2016 at 6:45

ankitbeohar90

1072 silver badges16 bronze badges

Collectives™ on Stack Overflow

Java - Spark SQL DataFrame map function is not working

6 Answers 6

1 Comment

1 Comment

2 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

1 Comment

1 Comment

2 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related