Stream continuous data from a python script to NodeJS

Question

I have a multi-threaded python script that converts an image to a NumPy array (occupancy grid). The python script then continuously changes the values of the array to simulate a dot moving in a map. This array is converted back to an image using Pillow and then encoded to base64. The NodeJS part has an express server running that also has a SocketIO connection with a mobile app.

What I am trying to do is: send the encoded image to the mobile app from the server, so the option that came to mind was to run the python script from nodeJs, and as the python script transmits an encoded image the server will redirect it to the mobile app.

What I'm looking for is:

stream the python script output to nodeJS without stopping (until the simulation stops)
or use a better-purposed solution to get the data from the python to the mobile phone

Thanks in advance!

python script to run:

import matplotlib.pyplot as plt
import cv2
import threading
import base64
from PIL import Image
temp = 0
global im_bw
pixel = 0

def update_image():
    global im_bw
    im2 = Image.fromarray(im_bw)
    image64 = base64.b64encode(im2.tobytes())
    # print the base64 encoded image to javascript
    print(image64)
    threading.Timer(1, update_image).start()

print("initialization")
im_gray = cv2.imread(r'gridMapBlue.JPG', cv2.IMREAD_GRAYSCALE)
(thresh, im_bw) = cv2.threshold(im_gray, 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
thresh = 127
im_bw = cv2.threshold(im_gray, thresh, 255, cv2.THRESH_BINARY)[1]



# noinspection PyBroadException
try:
    update_image()
except Exception as e:
    print(e.with_traceback())
    pass

# simulate the dot moving by modifying the 
while True:
    if pixel == 200:
        break
    temp = im_bw[150,pixel]
    im_bw[150, pixel] = 127
    pixel += 1

Javascript code:

const express = require("express");
const app = express();
const io = require("./api/services/SocketIoService")
const spawn = require('child_process').spawn;


const socketIO = new io(app);
socketIO.server.listen(6969, () => console.log("server started"))


py = spawn('python',["E:\\QU\\Senior\\robot_positioning\\robot_position_map.py"])

py.stdout.on('data', (data)=>console.log(data));

py.stderr.on("error", err =>console.log(err))

image used

edit: added the code for creating the occupancy grid and the javascript code that I tried (but didn't work)

Tricky, you could/should spawn the python process from node, then you can use the child_process stdio streams. But im not sure if the client detects that the image has changed and re-render it — Marc
– Marc, Commented Oct 18, 2020 at 17:49
I have tried to spawn the python process but I had several issues where the python script runs fine on PyCharm but when running it from the server the python script throws errors — Nayef Radwi
– Nayef Radwi, Commented Oct 18, 2020 at 18:47
Can you add a minimalistic setup for me ? Sounds interesting and i want try to get it working — Marc
– Marc, Commented Oct 18, 2020 at 18:55
@naif-mazan can you please add a complete minimalistic setup, incl. setup scripts/dependences — Marc
– Marc, Commented Oct 19, 2020 at 11:33

Marc · Accepted Answer · 2020-10-19 15:09:57Z

As wished op, not the 100% solution he want, but a working adapted concept.

main file is index.js:

const { spawn } = require("child_process");
const WebSocket = require("ws");

// create websocket server
// open "client.html" in a browser and see how the images flaps between two
const wss = new WebSocket.Server({
    port: 8080
});


// feedback
wss.on("connection", function connection(ws) {
    console.log("Client conneted to websocket");
});


// spawn python child process
const py = spawn("python", ["image.py"]);

console.log("Python image manipulating process has pid:", py.pid)

// listen for the new image
py.stdout.on("data", (data) => {

    // broadcast the new binary image to all clients
    wss.clients.forEach((client) => {
        if (client.readyState === WebSocket.OPEN) {
            client.send(data);
        }
    });

});


py.stderr.on("data", (data) => {
    console.error(data.toString());
});

this communicate between the image manipluating image.py and clients (over WebSockets, should be easy to change that to socket.io if needed):

import time
import sys

toogle = True

while True:

    if(toogle):
        f = open("image1.jpg", "r")
    else:
        f = open("image2.jpg", "r")

    toogle = not toogle

    sys.stdout.write(f.read())
    sys.stdout.flush()
    time.sleep(2)

That file create/manipluate a image and push the image binary over stdout to the parent index.js. For demo purpose we switch between 2 images that get send to the client: "image1.jpg" and "image2.jpg" (use any jpg image you want, if you need other mimetypes/extensions set theme in the html client too).

As client i used a simple html document client.html: When you open file it just shows a white/blank page, because no image was initzial loaded.

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>

<body>

    <canvas id="image">unsupported browser</canvas>

    <script>
        var ws = new WebSocket('ws://localhost:8080');

        ws.onopen = () => {
            console.log("Connected")
        };

        ws.onmessage = (evt) => {

            // re-create(?) blob data from websocket
            var blob = new Blob([evt.data], {
                type: 'image/jpg' // -> MIME TYPES OF IMAGES GOES HERE
            });

            // create blob url
            var url = URL.createObjectURL(blob);

            var image = document.getElementById("image");
            var img = new Image(0, 0);

            img.onload = () => {

                // get canvas context
                let ctx = image.getContext("2d");

                ctx.clearRect(0, 0, image.width, image.height);
                ctx.drawImage(img, 0, 0);

            };

            img.src = url;

        };

    </script>

</body>

</html>

The client renders the received image buffer on the page and refresh/redraw it if the python scripts "says so".

Install the only needed dependencie: npm install ws

To start everything type: node index.js and open the client in a webrowser of your choice (tested with firefox on ubuntu 18.04 LTS)

You should see that the image changes every second. With that working you can start to write the python script that "animate" your moving pixel :)

The script was edited inline here: perhaps its run not on the first hit. I try it by "copy and paste" and fix the found issues (typos, wrong names, etc..)

If you need any help or changes, let me now.

Collectives™ on Stack Overflow

Stream continuous data from a python script to NodeJS

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related