How to run script in Pyspark and drop into IPython shell when done?

Question

I want to run a spark script and drop into an IPython shell to interactively examine data.

Running both:

$ IPYTHON=1 pyspark --master local[2] myscript.py

and

$ IPYTHON=1 spark-submit --master local[2] myscript.py

both exit out of IPython once done.

This seems really simple, but can't find how to do it anywhere.

elyase · Accepted Answer · 2014-11-25 14:54:12Z

3

If you launch the iPython shell with:

$ IPYTHON=1 pyspark --master local[2]

you can do:

 >>> %run myscript.py

and all variables will stay in the workspace. You can also debug step by step with:

>>> %run -d myscript.py

answered Nov 25, 2014 at 14:54

elyase

41.2k12 gold badges121 silver badges123 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

trailblazer Over a year ago

That doesn't seem to pass the context variable though. sc (default). Thoughts ?

Alaa Ali · Accepted Answer · 2014-11-13 02:44:59Z

1

Launch the IPython shell using IPYTHON=1 pyspark, then run execfile('/path/to/myscript.py'), that should run your script inside the shell and return back to it.

answered Nov 13, 2014 at 2:44

Alaa Ali

9261 gold badge12 silver badges25 bronze badges

Collectives™ on Stack Overflow

How to run script in Pyspark and drop into IPython shell when done?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related