spark-submit python file and getting No module Found

Question

 spark-submit --master yarn-cluster --deploy-mode cluster test.py

end up with error

import pandas as pd ImportError: No module named pandas

this is the only error I see.

using anaconda python distribution 2.7 PYSPARK_VENV]/lib/python2.7/site-packages/ location has pandas.

Can you elaborate what is your error in details, please ? Maybe an error warning snippet and the version of python and spark you are using would be helpful. — james.bondu
– james.bondu, Commented Jan 25, 2019 at 18:19

Sincole Brans · Accepted Answer · 2020-03-05 22:28:20Z

1

Setting PYSPARK_PYTHON path should solve this:

check the pyspark path using: which pyspark

export PYSPARK_PYTHON=/pyspark/path/from/above

answered Mar 5, 2020 at 22:28

Sincole Brans

3042 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

james.bondu · Accepted Answer · 2019-01-25 18:19:52Z

0

You can check whether pandas installed in [PYSPARK_VENV]/lib/python2.7/site-packages/ folder. Looks like you are executing your pyspark application on another python interpreter. Please ensure that you have installed pandas package for that interpreter.

You can use Anaconda for managing python packages in these knida situations.

answered Jan 25, 2019 at 18:19

james.bondu

1,1782 gold badges18 silver badges28 bronze badges

1 Comment

Narendra Pinnaka Over a year ago

pandas are installed, I have no problem using pysaprk and I have issues with python interpreter.

Collectives™ on Stack Overflow

spark-submit python file and getting No module Found

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related