0

i am following the book(definitive guide hadoop).

and while trying to execute an example in the book with localhost, i have faced the error,

14/06/13 22:24:57 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
****hdfs://localhost/usr/kim/input/ncdc
14/06/13 22:24:57 INFO input.FileInputFormat: Total input paths to process : 3
14/06/13 22:24:57 WARN snappy.LoadSnappy: Snappy native library is available
14/06/13 22:24:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/06/13 22:24:57 INFO snappy.LoadSnappy: Snappy native library loaded
14/06/13 22:24:57 INFO mapred.JobClient: Running job: job_201406132100_0011
14/06/13 22:24:58 INFO mapred.JobClient:  map 0% reduce 0%
14/06/13 22:25:11 INFO mapred.JobClient: Task Id : attempt_201406132100_0011_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: mapred.MaxTemperatureMapper_v1
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
    at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mapred.MaxTemperatureMapper_v1
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
    ... 8 more

i executed the job using command line, hadoop jar MaxTemperatureDriver.jar mapred.MaxTemperatureDriver -conf /hadoop_conf/Hadoop-localhost.xml /input/ncdc /max-temp

there are 2 folder META-INF, mapred in the jar file and there are 5 classes in the mapred folder(these classes are in mapred package)

  1. MaxTemeratureReducer.class
  2. MaxTemperatureDriver.class
  3. MaxTemperatureMapper_v1.class
  4. MaxTemperatureMapper_v1$Temperature.class
  5. NcdcRecordParser.class

those are the configuration file, MaxTemperatureDriver and MaxTemperatureMapper_v1

<?xml version="1.0"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost/</value>
    </property>

    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:8021</value>
    </property>

    <property>
        <name>dfs.replication</name>    
        <value>1</value>
    </property>
</configuration>

.

public class MaxTemperatureDriver extends Configured implements Tool{
    @Override
    public int run(String[] args) throws Exception{
        if(args.length != 2){
            System.err.printf("Usage: %s [generic options] <input> <output>\n", getClass().getSimpleName());
            ToolRunner.printGenericCommandUsage(System.err);
            return -1;
        }

        Job job = new Job(getConf(), "Max Temperature");
        job.setJarByClass(getClass());

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job,  new Path(args[1]));

        job.setReducerClass(MaxTemeratureReducer.class);
        job.setMapperClass(MaxTemperatureMapper_v1.class);
        job.setCombinerClass(MaxTemeratureReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        return job.waitForCompletion(true) ? 0 : 1;
    }
}

.

public class MaxTemperatureMapper_v1 extends Mapper<LongWritable, Text, Text, IntWritable>{
    enum Temperature{
        OVER_100
    }

    private NcdcRecordParser parser = new NcdcRecordParser();

    @Override
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
        parser.parse(value);    //Text.toString()
        if(parser.isValidTemperature()){
            int airTemperature = parser.getAirTemperature();
            if(airTemperature > 1000){
                System.err.println("Temperature over 100 degrees for input: " + value);
                context.setStatus("Detected possibly corrupt record: see logs.");
                context.getCounter(Temperature.OVER_100).increment(1);
            }
            context.write(new Text(parser.getYear()), new IntWritable(parser.getAirTemperature()));
        }
    }
}

2 Answers 2

2

set jar by class name sometime its give error and also set the inputformat and output format I hope this will help you..

job.setJarByClass(MaxTemperatureDriver.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
Sign up to request clarification or add additional context in comments.

1 Comment

I think you meant job.setJarByClass(MaxTemperatureMapper.class);
0

job.setJarByClass("Main class");

Main class means that have main() method

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.