Hadoop FAILED Error: java.lang.ArrayIndexOutOfBoundsException: 5

Question

I am using Hadoop MapReduce to calculate each year's min and max value, but when I run the program, I get the error: FAILED Error: java.lang.ArrayIndexOutOfBoundsException: 5

I think this is because there are null values in my data because the program runs fine when there is no null value.

Therefore, in my map function, I write if statement to check whether there is a header and if there is a null value:

 public static class ExposureMapper
        extends Mapper<Object, Text, Text, MinMaxExposure> {

    private Text year = new Text();
    private double minexposure;
    private Double maxexposure;

    private MinMaxExposure outPut = new MinMaxExposure();

    public void map(Object key, Text value, Context context
    ) throws IOException, InterruptedException {
        try {
            //Some condition satisfying it is header
            if (value.toString().contains("Product")) {
                return;
            } else if(value.toString()==null) {
               return;
            }
            else{
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        String[] solarFields = value.toString().split(",");

        year.set(solarFields[2]);
        minexposure = Double.parseDouble(solarFields[5]);
        maxexposure = Double.parseDouble(solarFields[5]);

        try {
            outPut.setMinExposure(minexposure);
            outPut.setMaxExposure(maxexposure);
            context.write(year, outPut);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

But the same error happens... Is it because value.toString()==null is not a proper way to check null value?

EDIT:

19/06/07 00:14:30 INFO mapreduce.Job: Task Id : attempt_1527224104960_0812_m_000000_1, Status : FAILED
Error: java.lang.ArrayIndexOutOfBoundsException: 5
    at com.mycompany.hw1.SolarMinMax$ExposureMapper.map(SolarMinMax.java:50)
    at com.mycompany.hw1.SolarMinMax$ExposureMapper.map(SolarMinMax.java:23)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

A stack trace will be useful Shin. Also you should use value.toString().isEmpty to check null values. — Milan Desai
– Milan Desai, Commented Jun 6, 2019 at 15:19
Add the string you are splitting. My guess is that that there are not as many results as you think! — Micromuncher
– Micromuncher, Commented Jun 6, 2019 at 15:21
@MilanDesai I try value.toString().isEmpty, but it's not work :( — Shin Yu Wu
– Shin Yu Wu, Commented Jun 6, 2019 at 15:26
@Micromuncher What do you mean Add the string you are splitting? thank you. — Shin Yu Wu
– Shin Yu Wu, Commented Jun 6, 2019 at 15:27
Can you please post stacktrace. it will get us understand at which point you see ArrayIndexOutOfBoundsException — Milan Desai
– Milan Desai, Commented Jun 6, 2019 at 15:27

Data Man · Accepted Answer · 2019-06-06 15:32:09Z

1

If value.toString().split(","); has has fewer than six elements, solarFields[5] will not be an element, and so you'll see a ArrayIndexOutOfBoundsException.

Immediately after creating solarFields you should check its length:

if (solarFields == null || solarFields.length < 6) {
    return;
}

You also want to make sure that Double.parseDouble(solarFields[5]); doesn't throw a NumberFormatException:

Double exposure;
try {
    exposure = Double.parseDouble(solarFields[5]);
} catch (NumberFormatException e) {
    return;
}

answered Jun 6, 2019 at 15:32

Data Man

5,5596 gold badges46 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Hadoop FAILED Error: java.lang.ArrayIndexOutOfBoundsException: 5

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related