Java Spark Dataset MapFunction - Task not serializable without any reference to class

Question

I have a following class that reads csv data into Spark's Dataset. Everything works fine if I just simply read and return the data.

However, if I apply a MapFunction to the data before returning from function, I get

Exception in thread "main" org.apache.spark.SparkException: Task not serializable

Caused by: java.io.NotSerializableException: com.Workflow.

I know Spark's working and its need to serialize objects for distributed processing, however, I'm NOT using any reference to Workflow class in my mapping logic. I'm not calling any Workflow class function in my mapping logic. So why is Spark trying to serialize Workflow class? Any help will be appreciated.

public class Workflow {

    private final SparkSession spark;   

    public Dataset<Row> readData(){
        final StructType schema = new StructType()
            .add("text", "string", false)
            .add("category", "string", false);

        Dataset<Row> data = spark.read()
            .schema(schema)
            .csv(dataPath);

        /* 
         * works fine till here if I call
         * return data;
         */

        Dataset<Row> cleanedData = data.map(new MapFunction<Row, Row>() {
            public Row call(Row row){
                /* some mapping logic */
                return row;
            }
        }, RowEncoder.apply(schema));

        cleanedData.printSchema();
        /* .... ERROR .... */
        cleanedData.show();

        return cleanedData;
    }
}

see if this helps : stackoverflow.com/questions/40818001/… — Nikunj Kakadiya
– Nikunj Kakadiya, Commented Feb 17, 2021 at 4:33

badger · Accepted Answer · 2021-06-15 11:58:52Z

1

anonymous inner classes have a hidden/implicit reference to enclosing class. use Lambda expression or go with Roma Anankin's solution

edited Jun 15, 2021 at 11:58

answered Jun 10, 2021 at 17:11

badger

3,3161 gold badge18 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Roma Anankin · Accepted Answer · 2021-02-17 08:21:28Z

0

you could make Workflow implement Serializeble and SparkSession as @transient

answered Feb 17, 2021 at 8:21

Roma Anankin

561 bronze badge

1 Comment

F Baig Over a year ago

while this may work, it does not answer the question i.e. why is Spark trying to serialize Workflow when I'm not making any reference to it in the map function.

Collectives™ on Stack Overflow

Java Spark Dataset MapFunction - Task not serializable without any reference to class

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related