Launch Spark-Submit with restful service in Python

Question

following this tutorial I've made a restful service in python. Using this service I want to call an other python script with spark-submit, but it doens't work.

Here my service.py:

import pickle
import subprocess
from flask import Flask, request
from flask_restful import Resource, Api
from json import dumps
from flask_jsonpify import jsonify

app = Flask(__name__)
api = Api(app)

class Test(Resource):
    def post(self):
        imageID = request.form.get('imageID')
        tags = request.form.get('tags')

        return subprocess.call("spark-submit NaiveBayesClassifier.py",shell=True,stderr=subprocess.STDOUT)


api.add_resource(Test, '/test') 

if __name__ == '__main__':
    app.run(port=5002)

This service is made with virtualenv started using this:

source venv/bin/activate
python service.py

But when the script run ubprocess.call("spark-submit NaiveBayesClassifier.py",shell=True,stderr=subprocess.STDOUT) it return me this error:

Running on http://127.0.0.1:5002/ (Press CTRL+C to quit)
OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory 
file:
   34475
 Try using the -Djava.io.tmpdir= option to select an alternate temp location.

 OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:
   34462
Try using the -Djava.io.tmpdir= option to select an alternate temp location.

Traceback (most recent call last):
  File "/home/usertest/project/NaiveBayesClassifier.py", line 2, in <module>
    import numpy
ImportError: No module named numpy
127.0.0.1 - - [23/Feb/2018 14:36:06] "POST /test HTTP/1.1" 200 -

Any ideas about the problem? I'm using Spark 1.6.1

Aso Strife · Accepted Answer · 2018-02-23 19:24:52Z

1

I see three problems.

First in your code, you should use Popen in this way:

class Test(Resource):
    def post(self):
        imageID = request.form.get('imageID')
        tags = request.form.get('tags')

        p = subprocess.Popen(["spark-submit", "NaiveBayesClassifier.py"], stdout=subprocess.PIPE)
        return p.communicate()

Second in your virtualenv you should install pip

pip install numpy

or using sudo if you get an errors

sudo pip install numpy

Third this message means that there's no more space in your HDD. Try to delete some large file or incrase your partition if you can.

warning: Insufficient space for shared memory file: 34475

answered Feb 23, 2018 at 19:24

Aso Strife

1,1393 gold badges15 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Launch Spark-Submit with restful service in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related