java spark parallelize method parameters

Question

I'm writing a test program in java and would like to parallelize a list object.

SparkSession spark = SparkSession
      .builder()
      .master("local[*]")
      .appName("JavaWordCount")
      .getOrCreate();

System.out.println("hello");
List<String> l = new ArrayList<>(5);
l.add("view.txt");
spark.sparkContext().parallelize(l,1,"test");

The method parallelize(Seq, int, ClassTag) in the type SparkContext is not applicable for the arguments (List, int, String)

I'm not sure what would be the third parameter - class Tag

user8902160 · Accepted Answer · 2017-11-07 18:57:17Z

5

In general when working with Java you should prefer JavaSparkContext methods:

import org.apache.spark.api.java.JavaSparkContext;

JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());

JavaRDD<String> rdd = jsc.parallelize(l, 1);

SparkContext is intended mostly for Scala usage.

answered Nov 7, 2017 at 18:57

user8902160

661 bronze badge

Sign up to request clarification or add additional context in comments.

Comments

Thomas Jungblut · Accepted Answer · 2017-11-07 18:56:46Z

0

Give you want to parallelize a list of String, this should do:

ClassTag.apply(String.class)

answered Nov 7, 2017 at 18:56

Thomas Jungblut

21k6 gold badges71 silver badges92 bronze badges

Collectives™ on Stack Overflow

java spark parallelize method parameters

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related