Display stream with FFmpeg, python and opencv

Question

Situation : I have a basler camera connected to a raspberry pi, and I am trying to livestream it's feed with FFmpg to a tcp port in my windows PC in order to monitor whats happening in front of the camera.

Things that work : I manage to set up a python script on the raspberry pi which is responsible for recording the frames, feed them to a pipe and streaming them to a tcp port. From that port, I am able to display the stream using FFplay.

My problem : FFplay is great for testing out quickly and easily if the direction you are heading is correct, but I want to "read" every frame from the stream, do some processing and then displaying the stream with opencv. That, I am not able to do yet.

Minimaly reprsented, that's the code I use on the raspberry pi side of things :

command = ['ffmpeg',
           '-y',
           '-i', '-',
           '-an',
           '-c:v', 'mpeg4',
           '-r', '50',
           '-f', 'rtsp',
           '-rtsp_transport',
           'tcp','rtsp://192.168.1.xxxx:5555/live.sdp']

p = subprocess.Popen(command, stdin=subprocess.PIPE) 

while camera.IsGrabbing():  # send images as stream until Ctrl-C
    grabResult = camera.RetrieveResult(100, pylon.TimeoutHandling_ThrowException)
    
    if grabResult.GrabSucceeded():
        image = grabResult.Array
        image = resize_compress(image)
        p.stdin.write(image)
    grabResult.Release()

On my PC if I use the following FFplay command on a terminal, it works and it displays the stream in real time :

ffplay -rtsp_flags listen rtsp://192.168.1.xxxx:5555/live.sdp?tcp

On my PC if I use the following python script, the stream begins, but it fails in the cv2.imshow function because I am not sure how to decode it:

import subprocess
import cv2

command = ['C:/ffmpeg/bin/ffmpeg.exe',
           '-rtsp_flags', 'listen',
           '-i', 'rtsp://192.168.1.xxxx:5555/live.sdp?tcp?', 
           '-']

p1 = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE)

while True:
    frame = p1.stdout.read()
    cv2.imshow('image', frame)
    cv2.waitKey(1)

Does anyone knows what I need to change in either of those scripts in order to get i to work?

Thank you in advance for any tips.

Rotem · Accepted Answer · 2023-01-19 17:06:20Z

We may read the decoded frames from p1.stdout, convert it to NumPy array, and reshape it.

Change command to get decoded frames in rawvideo format and BGR pixel format:

 command = ['C:/ffmpeg/bin/ffmpeg.exe',
            '-rtsp_flags', 'listen',
            '-i', 'rtsp://192.168.1.xxxx:5555/live.sdp?tcp?',
            '-f', 'rawvideo',      # Get rawvideo output format.
            '-pix_fmt', 'bgr24',   # Set BGR pixel format
            'pipe:']               # Use stdout as output

Read the raw video frame from p1.stdout:

 raw_frame = p1.stdout.read(width*height*3)

Convert the bytes read into a NumPy array, and reshape it to video frame dimensions:

 frame = np.frombuffer(raw_frame, np.uint8)
 frame = frame.reshape((height, width, 3))

Now we can show the frame by calling cv2.imshow('image', frame).

The solution assumes, we know the video frame size (width and height) from advance.

The code sample below, includes a part that reads width and height using cv2.VideoCapture, but I am not sure if it's going to work in your case (due to '-rtsp_flags', 'listen'. (If it does work, you can try capturing using OpenCV instead of FFmpeg).

The following code is a complete "working sample" that uses public RTSP Stream for testing:

import cv2
import numpy as np
import subprocess

# Use public RTSP Stream for testing
in_stream = 'rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4'

if False:
    # Read video width, height and framerate using OpenCV (use it if you don't know the size of the video frames).

    # Use public RTSP Streaming for testing:
    cap = cv2.VideoCapture(in_stream)

    framerate = cap.get(5) #frame rate

    # Get resolution of input video
    width  = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    # Release VideoCapture - it was used just for getting video resolution
    cap.release()
else:
    # Set the size here, if video frame size is known
    width = 240
    height = 160


command = ['C:/ffmpeg/bin/ffmpeg.exe', # Using absolute path for example (in Linux replacing 'C:/ffmpeg/bin/ffmpeg.exe' with 'ffmpeg' supposes to work).
           #'-rtsp_flags', 'listen',   # The "listening" feature is not working (probably because the stream is from the web)
           '-rtsp_transport', 'tcp',   # Force TCP (for testing)
           '-max_delay', '30000000',   # 30 seconds (sometimes needed because the stream is from the web).
           '-i', in_stream,
           '-f', 'rawvideo',           # Video format is raw video
           '-pix_fmt', 'bgr24',        # bgr24 pixel format matches OpenCV default pixels format.
           '-an', 'pipe:']

# Open sub-process that gets in_stream as input and uses stdout as an output PIPE.
ffmpeg_process = subprocess.Popen(command, stdout=subprocess.PIPE)

while True:
    # Read width*height*3 bytes from stdout (1 frame)
    raw_frame = ffmpeg_process.stdout.read(width*height*3)

    if len(raw_frame) != (width*height*3):
        print('Error reading frame!!!')  # Break the loop in case of an error (too few bytes were read).
        break

    # Convert the bytes read into a NumPy array, and reshape it to video frame dimensions
    frame = np.frombuffer(raw_frame, np.uint8).reshape((height, width, 3))

    # Show the video frame
    cv2.imshow('image', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
  

ffmpeg_process.stdout.close()  # Closing stdout terminates FFmpeg sub-process.
ffmpeg_process.wait()  # Wait for FFmpeg sub-process to finish

cv2.destroyAllWindows()

Sample frame (just for fun):

Update:

Reading width and height using FFprobe:

When we don't know the video resolution from advance, we may use FFprobe for getting the information.

Here is a code sample for reading width and height using FFprobe:

import subprocess
import json

# Use public RTSP Stream for testing
in_stream = 'rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4'

probe_command = ['C:/ffmpeg/bin/ffprobe.exe',
                 '-loglevel', 'error',
                 '-rtsp_transport', 'tcp',  # Force TCP (for testing)]
                 '-select_streams', 'v:0',  # Select only video stream 0.
                 '-show_entries', 'stream=width,height', # Select only width and height entries
                 '-of', 'json', # Get output in JSON format
                 in_stream]

# Read video width, height using FFprobe:
p0 = subprocess.Popen(probe_command, stdout=subprocess.PIPE)
probe_str = p0.communicate()[0] # Reading content of p0.stdout (output of FFprobe) as string
p0.wait()
probe_dct = json.loads(probe_str) # Convert string from JSON format to dictonary.

# Get width and height from the dictonary
width = probe_dct['streams'][0]['width']
height = probe_dct['streams'][0]['height']

Thank you very much Rotem, your answer was extremely helpful, and I manage to solve my problem thanks to it, as well as understand the problems I was facing. Just for context and for anyone else using a Balser camera like I do (mine is daA1280-54uc), using ``` cv2.VideoCapture(index) ``` might be a bit of a problem and manualy getting every frame like I do in my first ``` code block ``` could be the sole approach. I actually knew the size of my images since I resize them before the stream, but if someone else doesn't (s)he probably should, just to be sure about the dimensions of the images
I edited the post: Example for reading width and height using FFprobe. Can you please tell me if it's working in your case? I know you don't need it, but it could be helpful for others.
Thank you for the update, unfortunately it doesn't seem to be working in my particular case.

R R · Accepted Answer · 2025-02-23 13:37:17Z

If anyone is trying to use @Rotem's answer for h.265 security camera LAN video and needs low latency, this might help (although it might be better to use Qt6, because apparently ffmpeg has already been incorporated into the multimedia backend):

import cv2
import numpy as np
import subprocess
import json

# Use local RTSP Stream for testing
in_stream = 'rtsp://user:[email protected]:554/live/0/main' + '.h265'

# "ffplay -loglevel error -hide_banner -af \"volume=0.0\" -flags low_delay -vf setpts=0 -tune zerolatency
# -probesize 32 -rtsp_transport tcp rtsp://user:[email protected]:554/live/0/main -left 0 -top 50 -x 400 -y 225"

probe_command = ['ffprobe.exe',
                 '-loglevel', 'error',      # Log level
                 '-rtsp_transport', 'tcp',  # Force TCP (for testing)]
                 '-select_streams', 'v:0',  # Select only video stream 0.
                 '-show_entries', 'stream=width,height', # Select only width and height entries
                 '-of', 'json',             # Get output in JSON format
                 in_stream]

# Read video width, height using FFprobe:
p0 = subprocess.Popen(probe_command, stdout=subprocess.PIPE)
probe_str = p0.communicate()[0] # Reading content of p0.stdout (output of FFprobe) as string
p0.wait()
probe_dct = json.loads(probe_str) # Convert string from JSON format to dictonary.

# Get width and height from the dictonary
width = probe_dct['streams'][0]['width']
height = probe_dct['streams'][0]['height']

# if False:
#     # Read video width, height and framerate using OpenCV (use it if you don't know the size of the video frames).

#     # Use public RTSP Streaming for testing:
#     cap = cv2.VideoCapture(in_stream)

#     framerate = cap.get(5) #frame rate

#     # Get resolution of input video
#     width  = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
#     height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

#     # Release VideoCapture - it was used just for getting video resolution
#     cap.release()
# else:
#     # Set the size here, if video frame size is known
#     width = 240
#     height = 160

command = ['ffmpeg.exe', # Using absolute path for example (in Linux replacing 'C:/ffmpeg/bin/ffmpeg.exe' with 'ffmpeg' supposes to work).
           '-hide_banner',             # Hide banner
           '-rtsp_transport', 'tcp',   # Force TCP (for testing)
           '-flags', 'low_delay',      # Low delay is needed for real-time video streaming.
           '-analyzeduration', '0',    # No analysis is needed for real-time video streaming.
           #'-fflags', 'nobuffer',     # No buffer is needed for real-time video streaming.
           #'-max_delay', '30000000',  # 30 seconds (sometimes needed because the stream is from the web).
           '-i', in_stream,            # '-i', 'input.h265',
           '-f', 'rawvideo',           # Video format is raw video
           '-pix_fmt', 'bgr24',        # bgr24 pixel format matches OpenCV default pixels format.
           '-an', 'pipe:'
           ]               # Drop frames if the consuming process is too slow.

# Open sub-process that gets in_stream as input and uses stdout as an output PIPE.
ffmpeg_process = subprocess.Popen(command, stdout=subprocess.PIPE)

while True:
    # Read width*height*3 bytes from stdout (1 frame)
    raw_frame = ffmpeg_process.stdout.read(width*height*3)

    if len(raw_frame) != (width*height*3):
        print('Error reading frame!!!')  # Break the loop in case of an error (too few bytes were read).
        break

    # Convert the bytes read into a NumPy array, and reshape it to video frame dimensions
    frame = np.frombuffer(raw_frame, np.uint8).reshape((height, width, 3))

    # Show the video frame
    cv2.imshow('image', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

ffmpeg_process.stdout.close()  # Closing stdout terminates FFmpeg sub-process.
ffmpeg_process.wait()  # Wait for FFmpeg sub-process to finish

cv2.destroyAllWindows()

Don't add code-only answers. Explain what your code does, why it works, and how it helps to solve the problem. Answers are meant to be useful for future visitors, even those who may not have the exact same code as the OP, but who may still have a similar issue and benefit from your answer.

Collectives™ on Stack Overflow

Display stream with FFmpeg, python and opencv

2 Answers 2

Update:

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Update:

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related