Running python in python block by block capturing the output

Question

As a pet project, I want to build something similar to the Jupyter notebook. Given an array of strings, each of which is a peace of python code, I would like to run each piece one by one in a single python process and then associate blocks of output with each piece of code. I would also like to manage it all in another (parent) python process.

To make the problem tangible, let's say I have a list of strings, each is a piece of python code. One string use variables from the preceding piece of code, i.e. they should all be run in a single process. Now I want to run one piece of code, wait until it finishes, capture the output, then run the next piece, and so on.

Unfortunately, googling around only gave me an example, where I can run peace of code, using subprocess.Popen('python', stdout=PIPE, ...), but with this approach, it will start executing my command after I close stdin, effectively closing the whole python process.

What is the point of isolating each line to a separate process if you also want the processes to be able to "share" variables? — jordanm
– jordanm, Commented Feb 20, 2020 at 18:42
How about simply executing each string sequentially with eval(...)? If required, in a single subprocess? — Seb
– Seb, Commented Feb 20, 2020 at 18:52
@jordanm, I don't really want to isolate every piece, I just want to know what output one piece gives and what output I get from another. I.e. I want to associate blocks of output with each piece of code. — Andrew Slabko
– Andrew Slabko, Commented Feb 20, 2020 at 20:08
@Seb, I didn't think about eval(...) honestly, but how can I capture output of eval? — Andrew Slabko
– Andrew Slabko, Commented Feb 20, 2020 at 20:19
I think I missed important point, that I want to associate blocks of output with each block of code. Added that to the question. — Andrew Slabko
– Andrew Slabko, Commented Feb 20, 2020 at 20:21

Seb · Accepted Answer · 2020-02-20 23:33:35Z

1

You can use contextlib.redirect_stdout from the standard library to capture the output of exec() calls. With that, your idea of code blocks (as I understand them) is straightforward to implement:

import io
from contextlib import redirect_stdout

class Block:
    def __init__(self, code=''):
        self.code = code
        self.stdout = io.StringIO()

    def run(self):
        with redirect_stdout(self.stdout):
            exec(self.code, globals()) # Pass global variable dict to allow modification

    @property
    def output(self):
        return self.stdout.getvalue()

>>> b1 = Block('a = 42; print(a)')
>>> b2 = Block('print(1/a)')
>>> b1.run()
>>> b2.run()
>>> b1.output
'42\n'
>>> b2.output
'0.023809523809523808\n'

edited Feb 20, 2020 at 23:33

answered Feb 20, 2020 at 21:17

Seb

4,60516 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Andrew Slabko Over a year ago

Thank you very much, this is the direction where I wanted someone to guide me. Together with multiprocessing it solves the problem nicely!

Collectives™ on Stack Overflow

Running python in python block by block capturing the output

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related