0

I am writing a microservice in Haskell and it seems that we'll need to call into a Python library. I know how to create and configure a process to do that from Haskell, but my Python is rusty. Here's the logic I am trying to implement:

  1. The Haskell application initializes by creating a persistent subprocess (lifetime of the subprocess = lifetime of the parent process) running a minimized application serving the Python library.
  2. The Haskell application receives a network request and sends over stdin exactly 1 chunk of data (i.e. bytestring or text) to the Python subprocess; it waits for -- blocking -- exactly 1 chunk of data to be received from the subprocess' stdout, collects the result and returns it as a response.

I've looked around and the closest solution I was able to find where:

  1. Running a Python program from Go and
  2. Persistent python subprocess

Both handle only the part I know how to handle (i.e. calling into a Python subrocess) while not dealing with the details of the Python code run from the subprocess -- hence this question.

The obvious alternative would be to simply create, run and stop a subprocess whenever the Haskell application needs it, but the overhead is unpleasant.

I've tried something whose minimized version looks like:

-- From the Haskell parent process
{-# LANGUAGE OverloadedStrings #-}

import           System.IO
import           System.Process.Typed

configProc :: ProcessConfig Handle Handle ()
configProc =
    setStdin createPipe $
    setStdout createPipe $
    setStderr closed $
    setWorkingDir "/working/directory" $
    shell "python3 my_program.py"

startPyProc :: IO (Process Handle Handle ())
startPyProc = do
    p <- startProcess configProc
    hSetBuffering (getStdin p) NoBuffering
    hSetBuffering (getStdout p) NoBuffering
    pure p

main :: IO ()
main = do
    p <- startPyProc
    let stdin = getStdin p
        stdout = getStdout p
    hSetBuffering stdin NoBuffering
    hSetBuffering stdout NoBuffering
    -- hGetLine won't get anything before I call hClose
    -- making it impossible to stream over both stdin and stout
    hPutStrLn stdin "foo" >> hClose stdin >> hGetLine stdout >>= print 
# From the Python child process
import sys

if '__name__' == '__main__':
    for line in sys.stdin:
        # do some work and finally...
        print(result)

One issue with this code is that I have not been able to send to sdin and receive from stdout without first closing the stdin handle, which makes the implementation unable to do what I want (send 1 chunk to stdin, block, read the result from stout, rinse and repeat). Another potential issue is that the Python code might not adequate at all for the specification I am trying to meet.

12
  • 1
    Why do you think it will deadlock instantly? Commented Jun 11, 2021 at 0:07
  • The fact that your other program is written in Python isn't really relevant. Commented Jun 11, 2021 at 0:09
  • It deadlocks in my tests, but I'd like to focus the question on the approach -- hopefully with examples -- that is recommended to meet the specification I've detailed. Commented Jun 11, 2021 at 0:11
  • 1
    you would have to create minimal working code in Python and Haskell so we could test it. Commented Jun 11, 2021 at 1:46
  • 1
    Don't know any python, but I bet for line in sys.stdin is waiting for a line to be written to stdin. Commented Jun 11, 2021 at 4:27

1 Answer 1

0

Got it fixed by simply replacing print(...) with print(..., flush=True). It appears that in Python stdin/stdout default to block-buffering, which made my call to hGetLine block since it was expecting lines.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.