How to map a RDD of type org.apache.spark.rdd.RDD[Array[String]]?

Question

I am new to Spark and Scala. I have an RDD that is of type org.apache.spark.rdd.RDD[Array[String]].

Here is a listing from myRdd.take(3).

Array(Array(1, 2524474, CBSGPRS, 1, 2015-09-09 10:42:03, 0, 47880, 302001131103734, NAT, "", 502161081073570, "", BLANK, UNK, "", "", "", MV_PVC, BLANK, 1, "", 0, 475078439, 41131;0;0, "", 102651;0;0, 3|3), Array(2, 2524516, CBSGPRS, 1, 2015-09-09 23:42:14, 0, 1260, 302001131104272, NAT, "", 502161081074085, "", BLANK, UNK, "", "", "", MV_PVC, BLANK, 1, "", 0, 2044745984, 3652;0;0, "", 8636;0;0, 3|3), Array(3, 2524545, CBSGPRS, 1, 2015-09-09 14:56:55, 0, 32886, 302001131101629, NAT, "", 502161081071599, "", BLANK, UNK, "", "", "", MV_PVC, BLANK, 1, "", 0, 1956194307, 14164657;0;0, "", 18231194;0;0, 3|3))

I am trying to map it as follows ..

var gprsMap = frows.collect().map{ tuple =>
// bind variables to the tuple
var (recKey, origRecKey, recTypeId, durSpanId, timestamp, prevConvDur, convDur,
    msisdn, callType, aPtyCellId, aPtyImsi, aPtyMsrn, bPtyNbr, bPtyNbrTypeId,
    bPtyCellId, bPtyImsi, bPtyMsrn, inTrgId, outTrgId, callStatusId, suppSvcId, provChgAmt,
    genFld1, genFld2, genFld3, genFld4, genFld5) = tuple

var dtm = timestamp.split(" ");
var idx = timestamp indexOf ' '
var dt = timestamp slice(0, idx)
var tm = timestamp slice(idx + 1, timestamp.length)

// return the results tuple
((dtm(0), msisdn, callType, recTypeId, provChgAmt), (convDur))
}

I keep getting error:

error: object Tuple27 is not a member of package scala.

I am not sure what the error is. Can someone help?

Till Rohrmann · Accepted Answer · 2015-09-15 14:54:22Z

3

The problem is that Scala only supports tuples with up to 22 fields. Additionally, your frows: RDD[Array[String]] contains Array[String] as elements. Thus, also your tuple variable in the map function is of type Array[String]. Therefore, it is not possible to unapply the variable tuple into a tuple.

But what you can do is accessing the elements of the array directly via indices.

val recKey = tuple(0)
val timestamp = tuple(4)
...

answered Sep 15, 2015 at 14:54

Till Rohrmann

13.4k2 gold badges27 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to map a RDD of type org.apache.spark.rdd.RDD[Array[String]]?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related