Thinking Outside the Bubble

Live-Streaming from a $350 360 camera into Virtual Reality

The Ricoh Theta S has achieved much praise within the VR community as being an easily affordable 360 camera that does all the things. The camera has 14MP resolution with spectacular looking images that are stitched from the camera to produce equirectangular format jpeg sizes: 5376x2688 / 2048x1024

Full HD (1,920 x 1,080 pixel) video time has been increased more than eight-fold to 25 minutes with the frame rate doubling to 30 fps since the last model. This camera has a live streaming and remote shutter options with 8gb of space available. Shutter speed can be set from 1/6400 sec. to 60 seconds allowing for excellent long exposures, excellent for light-painting. I plan to test the 360 light-painting concept with a friend who just acquired the Pixelstick:

Controlling the Camera

Linux command line can be used to control the camera HTTP API. Wireless LAN is used for communication between the client and RICOH THETA. RICOH THETA operates as an HTTP server when the wireless LAN is ON, and API can be used for GET requests and POST requests. You can read more about Open Spherical Camera API. Here's a few examples that use curl:

# Get Information $ curl http: { "manufacturer" : "RICOH" , "model" : "RICOH THETA S" , "serialNumber" : "XXXXXXXX" , "firmwareVersigon" : "01.11" , "supportUrl" : "https://theta360.com/en/support/" , "endpoints" :{ "httpPort" : 80 , "httpUpdatesPort" : 80 }, "gps" : false , "gyro" : false , "uptime" : 583 , "api" :[ "/osc/info" , "/osc/state" , "/osc/checkForUpdates" , "/osc/commands/execute" , "/osc/commands/status" ]} # Initiate Session $ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.startSession"}' {"name":"camera.startSession","state":"done","results":{ "sessionId":"SID_0001","timeout":180}} # Shoot a Still Image $ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.takePicture", "parameters": {"sessionId": "SID_0001"}}' {"name":"camera.takePicture","state":"inProgress","id":"2","progress":{"completion":0.0}} # Acquire Photographic State $ curl -X POST http://192.168.1.1:80/osc/state {"fingerprint":"FIG_0006","state":{"sessionId":"SID_0001","batteryLevel":1.0,"storageChanged":false,"_captureStatus":"idle","_recordedTime":0,"_recordableTime":0,"_latestFileUri":"100RICOH/R0010174.JPG","_batteryState":"charging"}} # Get the File $ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.getImage", "parameters": {"fileUri": "100RICOH/R0010174.JPG"}}' > image.jpg && open image.jpg # Terminate Session $ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.closeSession", "parameters": {"sessionId": "SID_0001"}}' {"name":"camera.closeSession","state":"done"}

There are also a number of scripts to get media from the camera. First you have to set the Ricoh Theta S to WiFi mode and connect to the access point. Here's a python example to test the API:

import urllib import json urllib.urlopen( "http://192.168.1.1/osc/info" ). read () data = json.dumps({ "name" : "camera.startSession" }) res = urllib.urlopen( 'http://192.168.1.1/osc/commands/execute' , data) sessionId = json.loads(res. read ())[ "results" ][ "sessionId" ] data = json.dumps({ "name" : "camera.takePicture" , "parameters" : { "sessionId" : sessionId}}) urllib.urlopen( 'http://192.168.1.1/osc/commands/execute' , data) fileUri = "" while not fileUri: res = urllib.urlopen( 'http://192.168.1.1/osc/state' , urllib.urlencode({})) fileUri = json.loads(res. read ())[ "state" ][ "_latestFileUri" ] data = json.dumps({ "name" : "camera.getImage" , "parameters" : { "fileUri" : fileUri}}) res = urllib.urlopen( 'http://192.168.1.1/osc/commands/execute' , data) with open ( "image.jpg" , "wb" ) as file: file. write (res. read ()) data = json.dumps({ "name" : "camera.closeSession" , "parameters" : { "sessionId" : sessionId}}) urllib.urlopen( 'http://192.168.1.1/osc/commands/execute' , data)

Opening the image can look something like this:

import subprocess subprocess.Popen([ 'open' , 'image.jpg' ])

Here's a more advanced version that uses the better requests module: https://raw.githubusercontent.com/theta360developers/python-download-rossgoderer/master/ThetaS-Download.py

There's also a Node.js client that is awesome and makes taking a picture simple as:

% theta % theta Available options: -h,

By default, the Ricoh Theta will produce still images in the Equirectangular format that can then be wrapped around a sphere for viewing as I have shown how to do in my first project log: https://hackaday.io/project/5077/log/15961-360-photospheres

This type of projection is better for processing with the neural networks [Sublime] but it has proven not be the most optimized format for streaming. According to Facebook, moving from equirectangular layouts to a cube format in 360 video reduced file size by 25 percent against the original and encoding with a pyramid geometry reduces file size by 80 percent. These are significant improvements when dealing with scale. We'll come back to this later.



The other type of image that the Ricoh Theta generates is a Dual-Fisheye. The Ricoh Theta S generates this mp4 format video at sizes 1920x1080 / 1280x720.

My development platform of choice is the Raspberry Pi 2 running a fresh copy of Raspbian Jessie. One method of live-streaming that has worked for me in the past has been mjpeg-streamer. You can refer back to https://hackaday.io/project/5077-metaverse-lab/log/17307-avalon-pt-2 for notes on how to build AVALON on top of the Rpi2 and configure mjpg-streamer to initiate during start-up. The condensed version for installing looks like this: (http://jacobsalmela.com/raspberry-pi-webcam-using-mjpg-streamer-over-internet/)

lsusb sudo apt-get update sudo apt- get upgrade sudo apt- get install libjpeg8-dev imagemagick libv4l-dev sudo ln -s /usr/include/linux/videodev2.h /usr/include/linux/videodev.h sudo apt- get install subversion cd ~ svn co local / bin sudo cp output_http.so input_file.so input_uvc.so /usr/ local /lib/ sudo cp -R www /usr/ local /www sudo vi ~/.bashrc export LD_LIBRARY_PATH=/usr/local/lib/ source ~/.bashrc /usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -r 1280x720" -o "/usr/local/lib/output_http.so -w /usr/local/www -p 8090" #!/ bin /bash /usr/ local / bin /mjpg_streamer -i "/usr/local/lib/input_uvc.so -r 1280x720" -o "/usr/local/lib/output_http.so -w /var/www/stream -c username:password" sudo chmod 755 /etc/init.d/livestream.sh sudo update -rc.d livestream.sh defaults sudo apt-sudo apt-libjpeg8-dev imagemagick libv4l-dev sudo-s /usr/include/linux/videodev2.h /usr/include/linux/videodev.h sudo apt-subversion cd ~ svn co https://svn.code.sf.net/p/mjpg-streamer/code/mjpg-streamer/ mjpg-streamer cd mjpg-streamer make mjpg_streamer input_file.so input_uvc.so output_http.so sudo cp mjpg_streamer /usr/sudo cp output_http.so input_file.so input_uvc.so /usr//lib/ sudo cp -R www /usr//www sudo vi ~/.bashrc export LD_LIBRARY_PATH=/usr/local/lib/ source ~/.bashrc /usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -r 1280x720" -o "/usr/local/lib/output_http.so -w /usr/local/www -p 8090" #!//bash /usr//mjpg_streamer -i-osudo chmod/etc/init.d/livestream.sh sudo-rc.d livestream.sh defaults

This is not ready to be streamed yet because the video remains to be in Dual-Fisheye format. We can see that Ricoh Theta has a desktop program for Windows and Mac which will do stitching from Dual-Fisheye into equirectangular and add the video metadata but this is unavailable for streaming. Linux users running the program through Wine will be disappointed as the it fails to convert video or do real-time stitching for streaming into equirectangular format. The lens geometry for the THETA is based on equidistant projection. RICOH does not make detailed lens parameter information available. This is also known as lens distortion data. Developers often ask for this information to improve stitching. It is proprietary and not available as of December 2015. However, stitching is still possible without this information. GOROman released a shader pack for Unity game engine that UV maps each Fisheye view to a sphere pretty well.

A custom UV mapped is formed for each side then the two images are aligned together as close to being seamless as possible.





Inspired by this, VR developer FireFoxG helped to create a custom UV mapped object that can stitch the two Fisheye images as close as he can. These first attempts were pretty shabby:



I took a picture while holding grid paper to help visualize the alignment between the lenses. This has greatly improved the stitching:

The object file can be found here: Download as .obj

The process for streaming 360 video over the web involved setting up dynamic DNS and port forwarding on the router.



As mentioned earlier, it is found that the equirectangular format is not the best in terms of streaming. Cube maps have been shown to deliver significantly higher quality while being easier to compress. John Carmack was quick to point this early but we have recently seen a new tool from Facebook emerge called Transform, implemented as an ffmpeg video filter that transforms 360 video in equirectangular projection into a cubemap projection. Facebook's primary goal was to tackle the drawback of the standard equirectangular layout for 360 videos, which flattens the sphere around the viewer onto a 2D surface. This layout creates warped images and contains redundant information at the top and bottom of the image. For the size of Facebook, every optimization method will go a long way in scalability.

https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/

When a 360 video is uploaded, we transform it from an equirectangular layout to the pyramid format for each of the 30 viewports, and we create five different resolutions for each stream, for a total of 150 different versions of the same video. We chose to pre-generate the videos and store them on the server rather than encode them in real time for each client request. While this requires more storage and can affect the latency when switching between viewports, it has a simple server-side setup and doesn’t require additional software

The other method of file format compression would be to use H 265 codec for encoding the video. It's extremely efficient for HD and 4K video, using a fraction of predecessor H.264 bandwidth. This provides the highest quality content even at low-bandwidth connections at sometimes half the cost of H.264. Sad to say there's a deal breaker to using this codec:

¯\_(ツ)_/¯

PS. RICOH, please release code for proper stitching or atleast a Linux version of your desktop program.

