How to access object's members from an array of object tuples

Question

Object class is

class VertexAttributes(val m: Boolean, n: Any){

        val rootParentCustNumber: String = if(n == null) "Was Null" else n.toString
        val firstMsgFlg = m

}

I have an RDD of this object's type:

scala> myGraph.vertices
res92: org.apache.spark.graphx.VertexRDD[VertexAttributes] = VertexRDDImpl[2280] at RDD at VertexRDD.scala:57

Filtering on the RDD, I get the following:

scala> res92.filter{case(k,m) => k == 964088677}.collect
res94: Array[(org.apache.spark.graphx.VertexId, VertexAttributes)] = Array((964088677,VertexAttributes@2612b83f))

How can I access [email protected] in Array((964088677,VertexAttributes@2612b83f))

I have tried res92.filter{case(k,m) => k == 964088677}.map{case Array(k,m)=> m.rootParentCustNumber}

But I get the following error:

<console>:243: error: pattern type is incompatible with expected type;
 found   : Array[T]
 required: (org.apache.spark.graphx.VertexId, VertexAttributes)
    (which expands to)  (Long, VertexAttributes)
       res92.filter{case(k,m) => k == 964088677}.map{case Array(k,m)=> m.rootParentCustNumber}
                                                               ^

Same as the filter stage: .map{ case (k, m) => m.rootParentCustNumber } — Xavier Guihot
– Xavier Guihot, Commented Mar 18, 2018 at 19:15
You sill can filter before. The filtering stage doesn't change the type of the RDD. So you can pipe the returned RDD of the filter stage with a map stage: res92.filter{ case(k, m) => k == 964088677 }.map{ case (k, m) => m.rootParentCustNumber } — Xavier Guihot
– Xavier Guihot, Commented Mar 18, 2018 at 19:18
Thank you. Could you please post that answer, so I can select? — Haris Irshad
– Haris Irshad, Commented Mar 18, 2018 at 19:20

Xavier Guihot · Accepted Answer · 2018-03-18 19:26:11Z

1

The filtering stage doesn't change the type of the RDD ( which is RDD[(Long, VertexAttributes)]).

So you can pipe the returned RDD of the filter stage with a map stage and work with each record the same way you did in the filtering stage:

res92
  .filter{ case (k, m) => k == 964088677 }
  .map{ case (k, m) => m.rootParentCustNumber }

I think you've been misled by the collect stage which transforms the RDD into an Array.

edited Mar 18, 2018 at 19:26

answered Mar 18, 2018 at 19:23

Xavier Guihot

62.8k26 gold badges320 silver badges202 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to access object's members from an array of object tuples

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related