Spark error: missing parameter type in map()

Question

I am trying to learn Spark GraphX on Windows 10 by replicating the code here. The code is developed using an older version of Spark and I'm not able to find a solution to create a vertex. The following is the code

import scala.util.MurmurHash
import org.apache.spark._
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD

val path = "F:/Soft/spark/2008.csv"
val df_1 = spark.read.option("header", true).csv(path)

val flightsFromTo = df_1.select($"Origin",$"Dest")
val airportCodes = df_1.select($"Origin", $"Dest").flatMap(x => Iterable(x(0).toString, x(1).toString))

// error caused by the following line
val airportVertices: RDD[(VertexId, String)] = airportCodes.distinct().map(x => (MurmurHash.stringHash(x), x))

The following is the error message:

<console>:57: error: missing parameter type
       val airportVertices: RDD[(VertexId, String)] = airportCodes.distinct().map(x => (MurmurHash.stringHash(x), x))
                                                                                  ^

I think the syntax is obsolete and I tried to find the latest syntax on official documents but it was of no help. The data set can be downloaded from here.

UPDATE:

Basically, I'm trying to create a Vertex and Edge, to finally create a graph as shown in the tutorial. I'm also new to the Map-Reduce paradigm.

SoakingHummer · Accepted Answer · 2017-03-28 06:56:09Z

4

The following lines of code worked for me.

// imported latest library - works without this too, just gives a warning
import scala.util.hashing.MurmurHash3

// datasets are set to rdd - this is the cause of the error
val flightsFromTo = df_1.select($"Origin",$"Dest").rdd
val airportCodes = df_1.select($"Origin", $"Dest").flatMap(x => Iterable(x(0).toString, x(1).toString)).rdd

val airportVertices: RDD[(VertexId, String)] = airportCodes.distinct().map(x => (MurmurHash3.stringHash(x), x))

answered Mar 28, 2017 at 6:56

SoakingHummer

5821 gold badge7 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Wang · Accepted Answer · 2017-03-27 13:42:55Z

1

You can try: val airportVertices: RDD[(VertexId, String)] = airportCodes.distinct().map(x => (MurmurHash.stringHash(x(0)), x(1)))

answered Mar 27, 2017 at 13:42

Wang

1655 bronze badges

1 Comment

SoakingHummer Over a year ago

It still gives the same error:

<console>:37: error: missing parameter type        val airportVertices: RDD[(VertexId, String)] = airportCodes.distinct().map(x => (MurmurHash.stringHash(x(0)), x(1)))

curiouscoder · Accepted Answer · 2017-09-13 06:57:05Z

1

// In order to apply map(), just try to convert the variable to RDD.

val airportVertices: RDD[(VertexId, String)] = airportCodes.rdd.distinct().map(x => (MurmurHash3.stringHash(x), x))

answered Sep 13, 2017 at 6:57

curiouscoder

317 bronze badges

Collectives™ on Stack Overflow

Spark error: missing parameter type in map()

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related