Running hdfs -text command within mapper and using output

Question

Is there a way to run hadoop commands in the mapper?

For example I want to run

hdfs dfs -text /path/to/file

and I will use that output for my mapper logic. Or is there an alternative?

Such as running the command hdfs dfs -text /path/to/file as a subprocess with Java

Jeremy Beard · Accepted Answer · 2015-03-27 00:46:38Z

1

Better is to use the Java HDFS API to open an HDFS file and read the contents into a variable that you can use.

For example:

InputStream is = FileSystem.get(yourConfigurationObject).open(new Path("/path/to/file"));

answered Mar 27, 2015 at 0:46

Jeremy Beard

2,7251 gold badge21 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Liondancer Over a year ago

How could I perform an -ls? I want to store all the paths to a file. There is so much documentation. I am not sure where to look and how to look for it.

Jeremy Beard Over a year ago

fs.listStatus(new Path("/your/path")) will return an array of FileStatus, from which each has a getPath() method.

Collectives™ on Stack Overflow

Running hdfs -text command within mapper and using output

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related