I am new to Hadoop. I want to access a command line argument from main function(Java program) inside the map function of the mapper class. Please suggest ways to do this.
Add a comment
|
3 Answers
Hadoop 0.20, introduced new MR API, there is not much functionality difference between the new (o.a.h.mapreduce package) and old MR API (o.a.h.mapred) except that data can be pulled within the mappers and the reducers using the new API. What Arnon is mentioned is with the old API.
Check this article for passing the parameters using the new and old API.
3 Comments
Pooja N Babu
Thank you very much for the link. It helped solve the problem.
Praveen Sripati
BTW, I am the author of the blog :)
Sudhir Belagali
@PraveenSripati , Hadoop 0.20 is not existed [ 404 ], ,Please change it
You can pass parameters by hanging them on the Configuration
JobConf job = new JobConf(new Configuration(), TheJob.class);
job.setLong("Param Name",longValue)
The Configuration class has few set methods (Long, Int, Strings etc.) so you can pass parameters of several types. In the map job you can get the configuration from the Context (getConfiguration)
3 Comments
Nihar Sarangi
Is it possible to pass parameters of ArrayList type?
Fgblanch
@NiharSarangi No that way but you can pass an array of strings using the setStrings Method described here hadoop.apache.org/docs/r0.20.2/api/org/apache/hadoop/conf/…, java.lang.String...)
Steven Magana-Zook
@ArnonRotem-Gal-Oz +1 I always like answers that stand on their own (no dependence on external sites for information since links often break or content changes).
In recent Hadoop (e.g. >=0.2 up to 2.4+) you would set this kind of options during the job configuration:
conf = new JobConf(MyJarClass);
conf.set("myStringOption", "myStringValue");
conf.set("myIntOption", 42);
And retrieve those options in the configure() method ofmapper/reducer classes:
public static class MyMapper extends MapReduceBase implements Mapper<...> {
Integer myIntegerOption;
String myStringOption;
@Override
public void configure(JobConf job) {
super.configure(job);
myIntegerOption = job.getInt("myIntOption", -1);
// nb: last arg is the default value if option is not set
myStringOption = job.get("myStringOption", "notSet");
}
@Override
public void map(... key, ... value,
OutputCollector<..> output, Reporter reporter) throws IOException {
// here you can use the options in your processing
processRecord(key, value, myIntOption, myStringOption);
}
}
Note that configure() will be called once before any records are passed to the map or reduce.
