0

I am facing issues using ffmpeg python to process camera frames. My first approach with process.communicate() worked well, but has latency issues.

process = (
            ffmpeg
            .input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width, height))
            .filter(<filter_params>)
            .output('pipe:', format='rawvideo', pix_fmt='rgb24')
            .run_async(pipe_stdin=True, pipe_stdout=True, pipe_stderr=True)
        )
out, err = process.communicate(input=img.tobytes())
output_image = np.frombuffer(out, np.uint8).reshape((height, width, channels))

To reduce the latency, I'm trying to keep the ffmpeg process open and feed in camera frames for processing. This runs fine for a couple of minutes with acceptable latency values but ends up in deadlock. What is the best way to fix this?

import cv2
import numpy as np
import math

import ffmpeg 

def start_ffmpeg_process_async(width, height):
    return (
        ffmpeg
            .input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width, height))
            .filter('<filter variables>')
            .output('pipe:', format='rawvideo', pix_fmt='rgb24')
            .run_async(pipe_stdin=True, pipe_stdout=True, pipe_stderr=True)
    )

def main():

    cap = cv2.VideoCapture(0)

    if not cap.isOpened():
        print("Camera can't open \nExiting...")
        return -1

    ffmpeg_process_async = start_ffmpeg_process_async(cap.get(cv2.CAP_PROP_FRAME_WIDTH), 
        cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
 
    try:
        while True:

            success, img = cap.read() 
            if success:
                height, width, channels = img.shape
                ffmpeg_process_async.stdin.write(img.tobytes())
                raw_output = ffmpeg_process_async.stdout.read(width * height * channels)
                output_image = np.frombuffer(raw_output, np.uint8).reshape((height, width, channels))
                cv2.imshow('Webcam', output_image)
            else:
                print ("Camera read error")
            if cv2.waitKey(1) == ord('q'):
                print ("Exiting . . .")
                break
    finally:
        print ("Finalizing . . .")
        ffmpeg_process_async.stdin.close()
        ffmpeg_process_async.wait()
        cap.release()
        cv2.destroyAllWindows()


if __name__ == "__main__":
    main()
6
  • you might want to decide which of ffmpeg-python or opencv you care about, and remove the other from your minimal reproducible example. that reduces the search space. you do expect people to run and debug your code, no? Commented Jul 31, 2024 at 23:46
  • I care about both actually. If I directly pass "/dev/video0" to ffmpeg input, it works without deadlock. The problem is that I'm using proprietary camera, so an additional SDK is required to read the camera frames, preprocess it with opencv and pass it to ffmpeg. Commented Aug 2, 2024 at 17:20
  • sure you do, for your ultimate goal, but not in the context of solving this very problem. you'd do yourself a favor if you'd cooperate and present a MRE. the more code you present, the lower the chances that anyone will bother. you are motivated, we aren't. we just look for novel puzzles to solve. "debug this code for me" isn't novel. it's boring. it's "homework". most of the people asking their questions here just want to get out of their own homework (the debugging). we know. us doing your work for you, that is not sustainable. prove that you care. do the homework. Commented Aug 2, 2024 at 18:10
  • specifically: replace the VideoCapture by some static or generated frames. that lifts the requirement that someone has a camera. replace the imshow/waitkey stuff by either nothing or a bit of console output that changes for every frame (value of one specific pixel, assuming you feed in frames filled with randomness or defined changing values). when the problem persists even after removing OpenCV, you can be fairly confident that the opencv part didn't contribute to the problem. if that did solve the problem, you also gained new information. Commented Aug 2, 2024 at 18:15
  • I strongly suspect that "ffmpeg-python" is at fault. it's not actually calling the ffmpeg libraries. it just runs a subprocess. sending data through a pipe is technically not unreasonable, but it's error-prone, and this "library" doesn't check anything. -- if you ask me, this "library/package" is junk. dealing with ffmpeg as a subprocess has limited uses, and is tolerable for individuals to do for their own messing around, but elevating that hack to a "package" is a war crime. besides that, the latest release on PyPI is from 2019 and the github has been dead for several years too. Commented Aug 2, 2024 at 18:22

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.