Task not Serializable - Spark Java

Question

I'm getting the Task not serializable error in Spark. I've searched and tried to use a static function as suggested in some posts, but it still gives the same error.

Code is as below:

public class Rating implements Serializable {
    private SparkSession spark;
    private SparkConf sparkConf;
    private JavaSparkContext jsc;
    private static Function<String, Rating> mapFunc;

    public Rating() {
        mapFunc = new Function<String, Rating>() {
            public Rating call(String str) {
                return Rating.parseRating(str);
            }
        };
    }

    public void runProcedure() { 
        sparkConf = new SparkConf().setAppName("Filter Example").setMaster("local");
        jsc = new JavaSparkContext(sparkConf);
        SparkSession spark = SparkSession.builder().master("local").appName("Word Count")
            .config("spark.some.config.option", "some-value").getOrCreate();        

        JavaRDD<Rating> ratingsRDD = spark.read().textFile("sample_movielens_ratings.txt")
                .javaRDD()
                .map(mapFunc);
    }

    public static void main(String[] args) {
        Rating newRating = new Rating();
        newRating.runProcedure();
    }
}

The error gives:

How do I solve this error? Thanks in advance.

Christian Neverdal · Accepted Answer · 2017-10-19 09:24:26Z

15

Clearly Rating cannot be Serializable, because it contains references to Spark structures (i.e. SparkSession, SparkConf, etc.) as attributes.

The problem here is in

JavaRDD<Rating> ratingsRD = spark.read().textFile("sample_movielens_ratings.txt")
            .javaRDD()
            .map(mapFunc);

If you look at the definition of mapFunc, you're returning a Rating object.

mapFunc = new Function<String, Rating>() {
    public Rating call(String str) {
        return Rating.parseRating(str);
    }
};

This function is used inside a map (a transformation in Spark terms). Because the transformations are executed directly into the worker nodes and not in the driver node, their code must be serializable. This forces Spark to try serialize the Rating class, but it is not possible.

Try to extract the features you need from Rating, and placing them in a different class that does not own any Spark structure. Finally, use this new class as return type of your mapFunc function.

edited Oct 19, 2017 at 9:24

Christian Neverdal

5,4357 gold badges41 silver badges97 bronze badges

answered Nov 8, 2016 at 6:38

riccardo.cardin

8,4037 gold badges70 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Fleur Over a year ago

Separating Rating and the procedure into two classes worked! Thanks :)

Christian Mata · Accepted Answer · 2018-03-27 10:51:02Z

5

In addition you have to be sure to not include non-serializable variables in your class like JavaSparkContext and SparkSession. if you need to include them you should define like this:

private transient JavaSparkContext sparkCtx;
private transient SparkSession spark;

Good luck.

answered Mar 27, 2018 at 10:51

Christian Mata

511 silver badge2 bronze badges

1 Comment

Amy Lashley Over a year ago

This little nugget just saved me. Until seeing this, I just couldn't get a clear idea of what was going wrong. So after working in circles for a good day, I finally get it. Thanks

Collectives™ on Stack Overflow

Task not Serializable - Spark Java

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related