\$\begingroup\$

I'm currently working on a robotics application where a video feed is being displayed from a Raspberry Pi 3.

I've been working on a way to stream the video directly into JavaFX (the rest of the UI is created in this), however, my knowledge of video streaming is very limited. The goal for the video system is to maintain decent video quality and FPS while reducing latency as much as possible (looking for sub 100 ms). H264 video was chosen as the format for it's speed, but I hear that sending raw video could be faster as there is no compression (could not get raw video to work well at all).

Running my code I am capable of streaming a Pi camera at about 120-130ms of latency and ~48 frames per second. I would like to continue to reduce the latency of this application, and would like to make sure that I'm making decisions for the correct reasons.

The largest issue I have so far is start-up time; it takes about 15-20 seconds for the video to initially launch and catch up to the latest frame.

The following code is an MCVE of the video system. If anyone is interested in reproducing this, you can get it running on a Raspberry Pi (mine is a Raspberry Pi 3) with python-picamera installed. You'll also need a Java Client with JavaCV installed. My version info is org.bytedeco:javacv-platform:1.3.2 .

Python side:

We decided to use a Python library to control the video stream because it provides a nice wrapper around the picamera command-line tool. The output from the video is being sent over a TCP connection and will be received by a Java client. (The way we remotely launch this application has been left out of the review because I just wanted this post to focus on the video aspects)

import picamera import socket import signal import sys with picamera.PiCamera() as camera: camera.resolution = (1296, 720) camera.framerate = 48 soc = socket.socket() soc.connect((sys.argv[1], int(sys.argv[2]))) file = soc.makefile('wb') try: camera.start_recording( file, format='h264', intra_period=0, quality=0, bitrate=25000000) while True: signal.pause() finally: file.close() soc.close()

Why did I choose these values?:

camera.resolution = (1296, 720) , camera.framerate = 48 were the largest images I could output at a frame-rate fast enough to reduce latency.

, were the largest images I could output at a frame-rate fast enough to reduce latency. intra_period=0 Wanted the images to remain small, and by setting the intra_period to zero, no I frames /full frames (apart from the first frame) will be sent; reducing the time between frames

Wanted the images to remain small, and by setting the to zero, no /full frames (apart from the first frame) will be sent; reducing the time between frames quality=0 from the docstring: Quality 0 is special and seems to be a "reasonable quality" default

from the docstring: bitrate=25000000 Wanted to set the bitrate as high as possible to not slow down video transfer when lots of changes in the frames (when P frames /partial frames become large)

Java side:

The Java decoder was written using JavaCV and sends the TCP H264 stream into an FFmpegFrameGrabber . The decoder then converts the Frame into a BufferedImage , and then into a WritableImage for JavaFX.

public class FFmpegFXImageDecoder { private FFmpegFXImageDecoder() { } public static void streamToImageView( final ImageView view, final int port, final int socketBacklog, final String format, final double frameRate, final int bitrate, final String preset, final int numBuffers ) { try (final ServerSocket server = new ServerSocket(port, socketBacklog); final Socket clientSocket = server.accept(); final FrameGrabber grabber = new FFmpegFrameGrabber( clientSocket.getInputStream()); ) { final Java2DFrameConverter converter = new Java2DFrameConverter(); grabber.setFrameRate(frameRate); grabber.setFormat(format); grabber.setVideoBitrate(bitrate); grabber.setVideoOption("preset", preset); grabber.setNumBuffers(numBuffers); grabber.start(); while (!Thread.interrupted()) { final Frame frame = grabber.grab(); if (frame != null) { final BufferedImage bufferedImage = converter.convert(frame); if (bufferedImage != null) { Platform.runLater(() -> view.setImage(SwingFXUtils.toFXImage(bufferedImage, null))); } } } } catch (final IOException e) { e.printStackTrace(); } } }

This can then be placed into a JavaFX view like below:

public class TestApplication extends Application { static final int WIDTH = 1296; static final int HEIGHT = 720; @Override public void start(final Stage primaryStage) throws Exception { final ImageView imageView = new ImageView(); final BorderPane borderPane = new BorderPane(); imageView.fitHeightProperty().bind(borderPane.widthProperty() .divide(WIDTH).multiply(HEIGHT)); imageView.fitWidthProperty().bind(borderPane.widthProperty()); borderPane.setPrefSize(WIDTH, HEIGHT); borderPane.setCenter(imageView); final Scene scene = new Scene(borderPane); primaryStage.setScene(scene); primaryStage.show(); new Thread(() -> FFmpegFXImageDecoder.streamToImageView( imageView, 12345, 100, "h264", 96, 25000000, "ultrafast", 0) ).start(); } }

Why did I choose these values?:

frameRate=96 Wanted the framerate of the Client to be twice the speed of the stream such that I'm not waiting on frames

Wanted the framerate of the Client to be twice the speed of the stream such that I'm not waiting on frames bitrate=25000000 to match the stream

to match the stream VideoOption preset="ultrafast" To try and reduce the startup time for the stream.

Final Questions:

What are some ways I improve the latency of this system?

How can I reduce the start-up time of this stream? It currently takes about 15 seconds to launch and catch up.

Are the parameters chosen for JavaCV and PiCamera logical? Is my understanding of them correct?