Scala Counter in a Loop causing issues at run time

Question

I have the following function that worked fine in the REPL, essentially what it is doing is checking the datatype of a schema and matching it to the column when I flatten the file out later and zipWithIndex:

 //Match a Schema to a Column value
    def schemaMatch(x: Array[String]) = {
      var accum = 0
      for(i <- 0 until x.length) {
        val convert = x(i).toString.toUpperCase
        println(convert)
        val split = convert.split(' ')
        println(split.mkString(" "))
        matchTest(split(1), accum)
        accum += 1
      }
      def matchTest(y:String, z:Int) = y match{

        case "STRING" => strBuf += z
        case "INTEGER" => decimalBuf += z
        case "DECIMAL" => decimalBuf += z
        case "DATE" => dateBuf += z
      }
    }


    schemaMatch(schema1)

The error I am getting:

Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.IntRef.create(I)Lscala/runtime/IntRef;
    at com.capitalone.DTS.dataProfiling$.schemaMatch$1(dataProfiling.scala:112)
    at com.capitalone.DTS.dataProfiling$.main(dataProfiling.scala:131)
    at com.capitalone.DTS.dataProfiling.main(dataProfiling.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Line 112:

var accum = 0

Any ideas why its no longer working when compiled but worked in the REPL, and how to correct it?

Maybe you could give us the output from when it worked in the REPL ? — Peter Neyens
– Peter Neyens, Commented Jun 23, 2015 at 15:51

Peter Neyens · Accepted Answer · 2015-06-24 14:12:35Z

1

It is difficult to see the underlying problem causing the NoSuchMethodError, with the code you have provided us, but your method to extract the column types and the corresponding indices can be simplified:

def schemaMatch(schema: Array[String]) : Map[String,List[Int]] =
  schema
    // get the 2nd word (column type) in upper cases
    .map(columnDescr => columnDescr.split(' ')(1).toUpperCase)
    // combine column type with index
    .zipWithIndex
    // group by column type
    .groupBy{ case (colType, index) => colType }
    // keep only the indices
    .mapValues( columsIndices => columsIndices.map(_._2).toList )

Which can be used as:

val columns = Array("x string", "1 integer", "2 decimal", "2015 date") 
val columnTypeMap = schemaMatch(columns)
//Map(DATE -> List(3), STRING -> List(0), DECIMAL -> List(2), INTEGER -> List(1))

val strIndices = columnTypeMap.getOrElse("STRING", Nil)
// List(0)

val decimalIndices = columnTypeMap.getOrElse("INTEGER", Nil) :::
  columnTypeMap.getOrElse("DECIMAL", Nil)
// List(1, 2)

edited Jun 24, 2015 at 14:12

answered Jun 23, 2015 at 16:52

Peter Neyens

9,82029 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

theMadKing Over a year ago

That worked like a charm and is cool to see a better way to work through this problem!

Collectives™ on Stack Overflow

Scala Counter in a Loop causing issues at run time

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related