TypeError: 'JavaPackage' object is not callable

Question

when I code the spark sql API hiveContext.sql()

from pyspark import SparkConf,SparkContext
from pyspark.sql import SQLContext,HiveContext

conf = SparkConf().setAppName("spark_sql")

sc = SparkContext(conf = conf)
hc = HiveContext(sc)

#rdd = sc.textFile("test.txt")
sqlContext = SQLContext(sc)
res = hc.sql("use teg_uee_app")
#for each in res.collect():
#    print(each[0])
sc.stop()

I got the following error:

enFile "spark_sql.py", line 23, in <module>
res = hc.sql("use teg_uee_app")
File "/spark/python/pyspark/sql/context.py", line 580, in sql
    return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
File "/spark/python/pyspark/sql/context.py", line 683, in _ssql_ctx
    self._scala_HiveContext = self._get_hive_ctx()
File "/spark/python/pyspark/sql/context.py", line 692, in _get_hive_ctx
return self._jvm.HiveContext(self._jsc.sc())
  TypeError: 'JavaPackage' object is not callable

how do I add SPARK_CLASSPATH or SparkContext.addFile?I don't have idea.

Christian Z. · Accepted Answer · 2016-09-02 11:45:47Z

4

Maybe this will help you: When using HiveContext I have to add three jars to the spark-submit arguments:

spark-submit --jars /usr/lib/spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/lib/spark/lib/datanucleus-core-3.2.10.jar,/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar ...

Of course the paths and versions depend on your cluster setup.

answered Sep 2, 2016 at 11:45

Christian Z.

4515 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dskrvk · Accepted Answer · 2017-06-13 17:22:55Z

1

In my case this turned out to be a classpath issue - I had a Hadoop jar on the classpath that was a wrong version of Hadoop than I was running.

Make sure you only set the executor and/or driver classpaths in one place and that there's no system-wide default applied somewhere such as .bashrc or Spark's conf/spark-env.sh.

answered Jun 13, 2017 at 17:22

dskrvk

1,38817 silver badges24 bronze badges

Collectives™ on Stack Overflow

TypeError: 'JavaPackage' object is not callable

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related