Scala class inheriting from a Java generic class

Question

I am trying to write a Hadoop mapper class in Scala. As a starting point, I have taken a Java example from the book "Hadoop: the Definitive Guide" and tried to port it to Scala.

The original Java class extends org.apache.hadoop.mapreduce.Mapper:

public class MaxTemperatureMapper
    extends Mapper<LongWritable, Text, Text, IntWritable>

and overrides the method

public void map(LongWritable key, Text value, Context context)
    throws IOException, InterruptedException

This methods gets called and works properly (I tested using a unit test and then run it with yarn).

My attempt at a Scala port is:

class MaxTemperatureMapperS extends Mapper[LongWritable, Text, Text, IntWritable]

and then the method

@throws(classOf[IOException])
@throws(classOf[InterruptedException])
override def map(key: LongWritable, value: Text, context: Context): Unit = 
{
    ...
}

but the Scala compiler issues an error:

error: method map overrides nothing.

So I thought the two methods had the same signature in Scala and Java, but apparently I am missing something. Can you give me some hint?

Mikel San Vicente · Accepted Answer · 2018-02-08 19:16:57Z

7

Sometimes the best way to do this is by letting your IDE work for you:

class Test extends Mapper[LongWritable, Text, Text, IntWritable] {
  override def map(key: LongWritable, value: Text, context: Mapper[LongWritable, Text, Text, IntWritable]#Context): Unit = ???
}

In this case the problem is that the definition of the class Context "lives" inside the class Mapper so you need to use the # syntax

answered Feb 8, 2018 at 19:16

Mikel San Vicente

3,8632 gold badges24 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Giorgio Over a year ago

So the Context that was found by the compiler was probably some other class that was in scope! It makes sense. Thanks a lot!

Giorgio Over a year ago

It works now. I have ported the whole Hadoop example application from Java to Scala and it runs without a glitch. Thanks a lot.

Pramod Kumar · Accepted Answer · 2020-01-29 15:11:37Z

0

For reference providing code for overriding both map and reduce method in Mapper and Reducer class respectively in scala

Mapper Example :

class MaxTemperatureMapper extends Mapper[LongWritable, Text, AvroKey[Integer], AvroValue[GenericRecord]] {

  val parser = new NcdcRecordParser()
  val record = new GenericData.Record(AvroSchema.SCHEMA)

  @throws(classOf[IOException])
  @throws(classOf[InterruptedException])
  override def map(key: LongWritable, value: Text, context:Mapper[LongWritable, Text, AvroKey[Integer], AvroValue[GenericRecord]]#Context) = {

Reducer Example :

class MaxTemperatureReducer extends  Reducer[AvroKey[Integer],AvroValue[GenericRecord],AvroKey[GenericRecord],NullWritable]{

  @throws(classOf[IOException])
  @throws(classOf[InterruptedException])
  override def reduce(key:AvroKey[Integer],values:java.lang.Iterable[AvroValue[GenericRecord]],
      context:Reducer[AvroKey[Integer],AvroValue[GenericRecord],AvroKey[GenericRecord],NullWritable]#Context) = {

edited Jan 29, 2020 at 15:11

answered Jan 29, 2020 at 9:22

Pramod Kumar

413 bronze badges

2 Comments

Borja Over a year ago

Please do not post images of code, copy your code into the question with block code format: stackoverflow.com/help/how-to-ask

Pramod Kumar Over a year ago

Edited as required.Thanks.

Collectives™ on Stack Overflow

Scala class inheriting from a Java generic class

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related