Adding pyspark python path in oozie

Question

I'm trying to run a simple python script on Oozie using Hue. I'm using anaconda parcels installed so I've also add in Cloudera manager, spark configuration (Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh)

if [ -z "${PYSPARK_PYTHON}" ]; then
export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python
fi

When running the job, i've a python error ImportError: No module named pandas.io.json , meaning that the PYSPARK_PYTHON doesn't seems to take the one from anaconda.

I've tried to add an arguments with

PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python

on the spark action via hue, but doesn't seems to work.

If I run the scripts via CLI and spark-submit it works. If I run other python scripts on Oozie via Hue (without packages from anaconda) it works.

What am I missing ? :/

Mariusz · Accepted Answer · 2018-09-12 14:05:47Z

4

When using spark via Oozie you need to tell what environment variables should be set on launcher container (the one that starts spark session).

Try adding a new property of spark action with key oozie.launcher.mapreduce.map.env and value PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python and it should work as expected.

edited Sep 12, 2018 at 14:05

answered Oct 21, 2017 at 10:23

Mariusz

14k3 gold badges66 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Rohan Over a year ago

Hi Can I get the property for oozie spark action such that spark job is submitted as "user" not as "YARN"

Mariusz Over a year ago

This feature is called "impersonation" and as far as I know it is not configurable for action but for whole oozie servers configuration.

Antoine Pointeau Over a year ago

you save my day !

Samson Scharfrichter Over a year ago

Nit: "mapred" properties are deprecated since Hadoop V2, and may be ignored in V3 => oozie.launcher.mapreduce.map.env

Collectives™ on Stack Overflow

Adding pyspark python path in oozie

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related