How to resolve java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2?

Question

I am trying to execute NLTK in Hadoop environment. Following is the command which i used for execution.

bin/hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.0.4.jar -input /user/nltk/input/ -output /user/nltk/output1/ -file /home/hduser/softwares/NLTK/unsupervised_sentiment-master.zip -mapper /home/hduser/softwares/NLTK/unsupervised_sentiment-master/sentiment.py

unsupervised_sentiment-master.zip --- contains all the dependent files required for sentiment.py

I am getting

java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

Any help would be greatly appreciated!!!

You should check the logs using the jobtracker in the HadoopUI they will help you out — viper
– viper, Commented Jul 15, 2013 at 16:44

Community · Accepted Answer · 2021-01-08 07:58:58Z

10

Could you pls post the python files? My guess is that, you need to add #!/usr/bin/python to the top of your py file. This was the case when I was streaming using python.

edited Jan 8, 2021 at 7:58

CommunityBot

11 silver badge

answered Jun 8, 2013 at 14:33

quiet_penguin

8507 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user3886907 · Accepted Answer · 2014-10-05 06:18:03Z

5

Adding below line to the top of python script made the code work for me.

#!/usr/bin/python

answered Oct 5, 2014 at 6:18

user3886907

3556 silver badges7 bronze badges

1 Comment

schiegg Over a year ago

In my case #!/usr/local/bin/python was correct, one may also check this out.

Alex Hamburgers · Accepted Answer · 2017-11-17 18:50:56Z

2

In your sentiment.py file, add the following line to the top:

```

!/usr/bin/env python

```

This worked for me.

answered Nov 17, 2017 at 18:50

Alex Hamburgers

212 bronze badges

Comments

Brian Dolan · Accepted Answer · 2014-07-31 04:32:38Z

1

I can't say exactly what your error is, but mine was that I had an unresolved dependency in my python script. Namely statsmodels.

answered Jul 31, 2014 at 4:32

Brian Dolan

3,1362 gold badges28 silver badges36 bronze badges

Collectives™ on Stack Overflow

How to resolve java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2?

4 Answers 4

Comments

1 Comment

!/usr/bin/env python

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

1 Comment

!/usr/bin/env python

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related