This article originally appeared on Qiita in Japanese by mechamogera

I tried to display dualfisheye movie (movie before conversion) in RICOH THETA S with Three.js. While getting help with the calculations, I worked through this myself, so there may be some strange issues. Please remember to take notes as you go along. (The calculations are hard.)

I watched MP4 movies with the above code and could properly display the THETA live streaming image acquired by WebRTC.

Map THETA S’s USB live streaming to a spherical map with a browser.English Translation | Original Japanese

Until distributing the whole spherical image with THETA S (3) I appreciate it because articles such as pasted images were displayed.

Code Explanation

Overview

Referencing Completely Understandable WebGL Programming Even for Beginners, Taking the First Step with Three.js I implemented three.js.

The configuration is as follows.

Three.min.js: Main processing of Three.js Files you have acquired

OrbitControls.js: Files you have acquired for grabbing the mouse with Three.js

Index.html: HTML created to watch Dualfisheye video

Theta-view.js: Dualfisheye JS created to watch videos

M20.mp4: Dualfisheye sample movie to display

I created a sphere, placed the camera inside, pasted and pasted a video texture with the video tag as the source.

Paste the Texture

In theta-view.js the following part sets the UV.

if (i < faceVertexUvs.length / 2) { var correction = (x == 0 && z == 0) ? 1 : (Math.acos(y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI); uvs[ j ].x = x * (404 / 1920) * correction + (447 / 1920); uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080); } else { var correction = ( x == 0 && z == 0) ? 1 : (Math.acos(-y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI); uvs[ j ].x = -1 * x * (404 / 1920) * correction + (1460 / 1920); uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080); }

Magic numbers of 404 and 447 correspond to the following sizes.

However, it is a rough value because it does not measure exactly.

Arrangement of Dualfisheye

The base code refers to this code in the answer to the following question.

Javascript - Mapping image onto a sphere in Three.js - Stack Overflow

faceVertexUvs[ face ][ j ].x = face.vertexNormals[ j ].x * 0.5 + 0.5; faceVertexUvs[ face ][ j ].y = face.vertexNormals[ j ].y * 0.5 + 0.5;

However, if you keep the above code, the image will be distorted.

Therefore, we calculate and correct as follows.

Correction calculation



Sample

index.html

<html> <head> <meta charset="utf-8"> </head> <body> <video id="video" autoplay loop style="display:none"> <source src="m20.MP4" type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'> </video> <script src="three.min.js"></script> <script src="OrbitControls.js"></script> <script src="theta-view.js"></script> <script> window.addEventListener( 'DOMContentLoaded', theta_view('video'), false ); </script> </body> </html>

OrbitControls.js

/** * @author qiao / https://github.com/qiao * @author mrdoob / http://mrdoob.com * @author alteredq / http://alteredqualia.com/ * @author WestLangley / http://github.com/WestLangley * @author erich666 / http://erichaines.com */ /*global THREE, console */ // This set of controls performs orbiting, dollying (zooming), and panning. It maintains // the "up" direction as +Y, unlike the TrackballControls. Touch on tablet and phones is // supported. // // Orbit - left mouse / touch: one finger move // Zoom - middle mouse, or mousewheel / touch: two finger spread or squish // Pan - right mouse, or arrow keys / touch: three finter swipe // // This is a drop-in replacement for (most) TrackballControls used in examples. // That is, include this js file and wherever you see: // controls = new THREE.TrackballControls( camera ); // controls.target.z = 150; // Simple substitute "OrbitControls" and the control should work as-is. THREE.OrbitControls = function ( object, domElement ) { this.object = object; this.domElement = ( domElement !== undefined ) ? domElement : document; // API // Set to false to disable this control this.enabled = true; // "target" sets the location of focus, where the control orbits around // and where it pans with respect to. this.target = new THREE.Vector3(); // center is old, deprecated; use "target" instead this.center = this.target; // This option actually enables dollying in and out; left as "zoom" for // backwards compatibility this.noZoom = false; this.zoomSpeed = 1.0; // Limits to how far you can dolly in and out this.minDistance = 0; this.maxDistance = Infinity; // Set to true to disable this control this.noRotate = false; this.rotateSpeed = 1.0; // Set to true to disable this control this.noPan = false; this.keyPanSpeed = 7.0; // pixels moved per arrow key push // Set to true to automatically rotate around the target this.autoRotate = false; this.autoRotateSpeed = 2.0; // 30 seconds per round when fps is 60 // How far you can orbit vertically, upper and lower limits. // Range is 0 to Math.PI radians. this.minPolarAngle = 0; // radians this.maxPolarAngle = Math.PI; // radians // Set to true to disable use of the keys this.noKeys = false; // The four arrow keys this.keys = { LEFT: 37, UP: 38, RIGHT: 39, BOTTOM: 40 }; //////////// // internals var scope = this; var EPS = 0.000001; var rotateStart = new THREE.Vector2(); var rotateEnd = new THREE.Vector2(); var rotateDelta = new THREE.Vector2(); var panStart = new THREE.Vector2(); var panEnd = new THREE.Vector2(); var panDelta = new THREE.Vector2(); var panOffset = new THREE.Vector3(); var offset = new THREE.Vector3(); var dollyStart = new THREE.Vector2(); var dollyEnd = new THREE.Vector2(); var dollyDelta = new THREE.Vector2(); var phiDelta = 0; var thetaDelta = 0; var scale = 1; var pan = new THREE.Vector3(); var lastPosition = new THREE.Vector3(); var STATE = { NONE : -1, ROTATE : 0, DOLLY : 1, PAN : 2, TOUCH_ROTATE : 3, TOUCH_DOLLY : 4, TOUCH_PAN : 5 }; var state = STATE.NONE; // for reset this.target0 = this.target.clone(); this.position0 = this.object.position.clone(); // so camera.up is the orbit axis var quat = new THREE.Quaternion().setFromUnitVectors( object.up, new THREE.Vector3( 0, 1, 0 ) ); var quatInverse = quat.clone().inverse(); // events var changeEvent = { type: 'change' }; var startEvent = { type: 'start'}; var endEvent = { type: 'end'}; this.rotateLeft = function ( angle ) { if ( angle === undefined ) { angle = getAutoRotationAngle(); } thetaDelta -= angle; }; this.rotateUp = function ( angle ) { if ( angle === undefined ) { angle = getAutoRotationAngle(); } phiDelta -= angle; }; // pass in distance in world space to move left this.panLeft = function ( distance ) { var te = this.object.matrix.elements; // get X column of matrix panOffset.set( te[ 0 ], te[ 1 ], te[ 2 ] ); panOffset.multiplyScalar( - distance ); pan.add( panOffset ); }; // pass in distance in world space to move up this.panUp = function ( distance ) { var te = this.object.matrix.elements; // get Y column of matrix panOffset.set( te[ 4 ], te[ 5 ], te[ 6 ] ); panOffset.multiplyScalar( distance ); pan.add( panOffset ); }; // pass in x,y of change desired in pixel space, // right and down are positive this.pan = function ( deltaX, deltaY ) { var element = scope.domElement === document ? scope.domElement.body : scope.domElement; if ( scope.object.fov !== undefined ) { // perspective var position = scope.object.position; var offset = position.clone().sub( scope.target ); var targetDistance = offset.length(); // half of the fov is center to top of screen targetDistance *= Math.tan( ( scope.object.fov / 2 ) * Math.PI / 180.0 ); // we actually don't use screenWidth, since perspective camera is fixed to screen height scope.panLeft( 2 * deltaX * targetDistance / element.clientHeight ); scope.panUp( 2 * deltaY * targetDistance / element.clientHeight ); } else if ( scope.object.top !== undefined ) { // orthographic scope.panLeft( deltaX * (scope.object.right - scope.object.left) / element.clientWidth ); scope.panUp( deltaY * (scope.object.top - scope.object.bottom) / element.clientHeight ); } else { // camera neither orthographic or perspective console.warn( 'WARNING: OrbitControls.js encountered an unknown camera type - pan disabled.' ); } }; this.dollyIn = function ( dollyScale ) { if ( dollyScale === undefined ) { dollyScale = getZoomScale(); } scale /= dollyScale; }; this.dollyOut = function ( dollyScale ) { if ( dollyScale === undefined ) { dollyScale = getZoomScale(); } scale *= dollyScale; }; this.update = function () { var position = this.object.position; offset.copy( position ).sub( this.target ); // rotate offset to "y-axis-is-up" space offset.applyQuaternion( quat ); // angle from z-axis around y-axis var theta = Math.atan2( offset.x, offset.z ); // angle from y-axis var phi = Math.atan2( Math.sqrt( offset.x * offset.x + offset.z * offset.z ), offset.y ); if ( this.autoRotate ) { this.rotateLeft( getAutoRotationAngle() ); } theta += thetaDelta; phi += phiDelta; // restrict phi to be between desired limits phi = Math.max( this.minPolarAngle, Math.min( this.maxPolarAngle, phi ) ); // restrict phi to be betwee EPS and PI-EPS phi = Math.max( EPS, Math.min( Math.PI - EPS, phi ) ); var radius = offset.length() * scale; // restrict radius to be between desired limits radius = Math.max( this.minDistance, Math.min( this.maxDistance, radius ) ); // move target to panned location this.target.add( pan ); offset.x = radius * Math.sin( phi ) * Math.sin( theta ); offset.y = radius * Math.cos( phi ); offset.z = radius * Math.sin( phi ) * Math.cos( theta ); // rotate offset back to "camera-up-vector-is-up" space offset.applyQuaternion( quatInverse ); position.copy( this.target ).add( offset ); this.object.lookAt( this.target ); thetaDelta = 0; phiDelta = 0; scale = 1; pan.set( 0, 0, 0 ); if ( lastPosition.distanceToSquared( this.object.position ) > EPS ) { this.dispatchEvent( changeEvent ); lastPosition.copy( this.object.position ); } }; this.reset = function () { state = STATE.NONE; this.target.copy( this.target0 ); this.object.position.copy( this.position0 ); this.update(); }; function getAutoRotationAngle() { return 2 * Math.PI / 60 / 60 * scope.autoRotateSpeed; } function getZoomScale() { return Math.pow( 0.95, scope.zoomSpeed ); } function onMouseDown( event ) { if ( scope.enabled === false ) return; event.preventDefault(); if ( event.button === 0 ) { if ( scope.noRotate === true ) return; state = STATE.ROTATE; rotateStart.set( event.clientX, event.clientY ); } else if ( event.button === 1 ) { if ( scope.noZoom === true ) return; state = STATE.DOLLY; dollyStart.set( event.clientX, event.clientY ); } else if ( event.button === 2 ) { if ( scope.noPan === true ) return; state = STATE.PAN; panStart.set( event.clientX, event.clientY ); } scope.domElement.addEventListener( 'mousemove', onMouseMove, false ); scope.domElement.addEventListener( 'mouseup', onMouseUp, false ); scope.dispatchEvent( startEvent ); } function onMouseMove( event ) { if ( scope.enabled === false ) return; event.preventDefault(); var element = scope.domElement === document ? scope.domElement.body : scope.domElement; if ( state === STATE.ROTATE ) { if ( scope.noRotate === true ) return; rotateEnd.set( event.clientX, event.clientY ); rotateDelta.subVectors( rotateEnd, rotateStart ); // rotating across whole screen goes 360 degrees around scope.rotateLeft( 2 * Math.PI * rotateDelta.x / element.clientWidth * scope.rotateSpeed ); // rotating up and down along whole screen attempts to go 360, but limited to 180 scope.rotateUp( 2 * Math.PI * rotateDelta.y / element.clientHeight * scope.rotateSpeed ); rotateStart.copy( rotateEnd ); } else if ( state === STATE.DOLLY ) { if ( scope.noZoom === true ) return; dollyEnd.set( event.clientX, event.clientY ); dollyDelta.subVectors( dollyEnd, dollyStart ); if ( dollyDelta.y > 0 ) { scope.dollyIn(); } else { scope.dollyOut(); } dollyStart.copy( dollyEnd ); } else if ( state === STATE.PAN ) { if ( scope.noPan === true ) return; panEnd.set( event.clientX, event.clientY ); panDelta.subVectors( panEnd, panStart ); scope.pan( panDelta.x, panDelta.y ); panStart.copy( panEnd ); } scope.update(); } function onMouseUp( /* event */ ) { if ( scope.enabled === false ) return; scope.domElement.removeEventListener( 'mousemove', onMouseMove, false ); scope.domElement.removeEventListener( 'mouseup', onMouseUp, false ); scope.dispatchEvent( endEvent ); state = STATE.NONE; } function onMouseWheel( event ) { if ( scope.enabled === false || scope.noZoom === true ) return; event.preventDefault(); event.stopPropagation(); var delta = 0; if ( event.wheelDelta !== undefined ) { // WebKit / Opera / Explorer 9 delta = event.wheelDelta; } else if ( event.detail !== undefined ) { // Firefox delta = - event.detail; } if ( delta > 0 ) { scope.dollyOut(); } else { scope.dollyIn(); } scope.update(); scope.dispatchEvent( startEvent ); scope.dispatchEvent( endEvent ); } function onKeyDown( event ) { if ( scope.enabled === false || scope.noKeys === true || scope.noPan === true ) return; switch ( event.keyCode ) { case scope.keys.UP: scope.pan( 0, scope.keyPanSpeed ); scope.update(); break; case scope.keys.BOTTOM: scope.pan( 0, - scope.keyPanSpeed ); scope.update(); break; case scope.keys.LEFT: scope.pan( scope.keyPanSpeed, 0 ); scope.update(); break; case scope.keys.RIGHT: scope.pan( - scope.keyPanSpeed, 0 ); scope.update(); break; } } function touchstart( event ) { if ( scope.enabled === false ) return; switch ( event.touches.length ) { case 1: // one-fingered touch: rotate if ( scope.noRotate === true ) return; state = STATE.TOUCH_ROTATE; rotateStart.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY ); break; case 2: // two-fingered touch: dolly if ( scope.noZoom === true ) return; state = STATE.TOUCH_DOLLY; var dx = event.touches[ 0 ].pageX - event.touches[ 1 ].pageX; var dy = event.touches[ 0 ].pageY - event.touches[ 1 ].pageY; var distance = Math.sqrt( dx * dx + dy * dy ); dollyStart.set( 0, distance ); break; case 3: // three-fingered touch: pan if ( scope.noPan === true ) return; state = STATE.TOUCH_PAN; panStart.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY ); break; default: state = STATE.NONE; } scope.dispatchEvent( startEvent ); } function touchmove( event ) { if ( scope.enabled === false ) return; event.preventDefault(); event.stopPropagation(); var element = scope.domElement === document ? scope.domElement.body : scope.domElement; switch ( event.touches.length ) { case 1: // one-fingered touch: rotate if ( scope.noRotate === true ) return; if ( state !== STATE.TOUCH_ROTATE ) return; rotateEnd.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY ); rotateDelta.subVectors( rotateEnd, rotateStart ); // rotating across whole screen goes 360 degrees around scope.rotateLeft( 2 * Math.PI * rotateDelta.x / element.clientWidth * scope.rotateSpeed ); // rotating up and down along whole screen attempts to go 360, but limited to 180 scope.rotateUp( 2 * Math.PI * rotateDelta.y / element.clientHeight * scope.rotateSpeed ); rotateStart.copy( rotateEnd ); scope.update(); break; case 2: // two-fingered touch: dolly if ( scope.noZoom === true ) return; if ( state !== STATE.TOUCH_DOLLY ) return; var dx = event.touches[ 0 ].pageX - event.touches[ 1 ].pageX; var dy = event.touches[ 0 ].pageY - event.touches[ 1 ].pageY; var distance = Math.sqrt( dx * dx + dy * dy ); dollyEnd.set( 0, distance ); dollyDelta.subVectors( dollyEnd, dollyStart ); if ( dollyDelta.y > 0 ) { scope.dollyOut(); } else { scope.dollyIn(); } dollyStart.copy( dollyEnd ); scope.update(); break; case 3: // three-fingered touch: pan if ( scope.noPan === true ) return; if ( state !== STATE.TOUCH_PAN ) return; panEnd.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY ); panDelta.subVectors( panEnd, panStart ); scope.pan( panDelta.x, panDelta.y ); panStart.copy( panEnd ); scope.update(); break; default: state = STATE.NONE; } } function touchend( /* event */ ) { if ( scope.enabled === false ) return; scope.dispatchEvent( endEvent ); state = STATE.NONE; } this.domElement.addEventListener( 'contextmenu', function ( event ) { event.preventDefault(); }, false ); this.domElement.addEventListener( 'mousedown', onMouseDown, false ); this.domElement.addEventListener( 'mousewheel', onMouseWheel, false ); this.domElement.addEventListener( 'DOMMouseScroll', onMouseWheel, false ); // firefox this.domElement.addEventListener( 'touchstart', touchstart, false ); this.domElement.addEventListener( 'touchend', touchend, false ); this.domElement.addEventListener( 'touchmove', touchmove, false ); window.addEventListener( 'keydown', onKeyDown, false ); // force an update at start this.update(); }; THREE.OrbitControls.prototype = Object.create( THREE.EventDispatcher.prototype );

theta-view.js

var theta_view = function (video_id) { var scene = new THREE.Scene(); var width = 600; var height = 400; var fov = 60; var aspect = width / height; var near = 1; var far = 1000; var camera = new THREE.PerspectiveCamera( fov, aspect, near, far ); camera.position.set( 0, 0, 0.1 ); var renderer = new THREE.WebGLRenderer(); renderer.setSize( width, height ); var element = renderer.domElement; document.body.appendChild( element ); var directionalLight = new THREE.DirectionalLight( 0xffffff ); directionalLight.position.set( 0, 0.7, 0.7 ); scene.add( directionalLight ); var video = document.getElementById( video_id ); var texture = new THREE.VideoTexture( video ); texture.minFilter = THREE.LinearFilter; texture.magFilter = THREE.LinearFilter; texture.format = THREE.RGBFormat; var geometry = new THREE.SphereGeometry(100, 32, 32, 0); geometry.scale(-1, 1, 1); var faceVertexUvs = geometry.faceVertexUvs[ 0 ]; for ( i = 0; i < faceVertexUvs.length; i ++ ) { var uvs = faceVertexUvs[ i ]; var face = geometry.faces[ i ]; for ( var j = 0; j < 3; j ++ ) { var x = face.vertexNormals[ j ].x; var y = face.vertexNormals[ j ].y; var z = face.vertexNormals[ j ].z; if (i < faceVertexUvs.length / 2) { var correction = (x == 0 && z == 0) ? 1 : (Math.acos(y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI); uvs[ j ].x = x * (404 / 1920) * correction + (447 / 1920); uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080); } else { var correction = ( x == 0 && z == 0) ? 1 : (Math.acos(-y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI); uvs[ j ].x = -1 * x * (404 / 1920) * correction + (1460 / 1920); uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080); } } } geometry.rotateZ(-Math.PI / 2); var material = new THREE.MeshBasicMaterial( { map: texture } ); var mesh = new THREE.Mesh( geometry, material ); scene.add( mesh ); // PCで閲覧時にマウスドラッグで操作 var controls = new THREE.OrbitControls(camera, element); controls.rotateUp(Math.PI / 4); controls.noPan = true; ( function renderLoop () { requestAnimationFrame( renderLoop ); renderer.render( scene, camera ); } )(); };

Editor’s Note:

I believe that the original author’s name of mechamogera is connected to the movie Godzilla vs. SpaceGodzilla, M.O.G.U.E.R.A. (Mobile Operations G-Force Universal Expert Robot-Aero-Type) was a UN-built mecha designed to fight Godzilla following the failure of G-Force’s Mechagodzilla.