unable to submit spark python script

Question

I'm using the following script to submit a python script

#!/usr/bin/python

from pyspark.mllib.classification import LogisticRegressionWithSGD
from pyspark.mllib.regression import LabeledPoint
from numpy import array
from pyspark import SparkContext as sc, SparkConf

data = sc.textFile("hdfs:/dataset/parkinsons.data")

got this error:

data = sc.textFile("hdfs:/dataset/parkinsons.data")
TypeError: unbound method textFile() must be called with SparkContext instance as first argument (got str instance instead)

artemdevel · Accepted Answer · 2015-04-22 11:35:44Z

1

You must create a SparkContext at first, for example:

from pyspark import SparkContext

sc = SparkContext(appName="TestApp")
data = sc.textFile("hdfs:/dataset/parkinsons.data")

answered Apr 22, 2015 at 11:35

artemdevel

6411 gold badge10 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

unable to submit spark python script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related