Portals with Asymmetric Projection

Portals are often used in immersive experiences: linking together environments, acting as gateways to different worlds, or filters on a current one.

When both the outer world, and the inner portal world share a rendering context, both worlds can be rendered on top of each other, obscuring the portal world outside of the visible gateway via stencil buffer or other techniques. Sometimes the portal world is rendered to a flat quad first before being viewed in an outer world.

An early prototype of model-viewer, a web component to render 3D content, rendered in a webview inside of a Unity VR environment. The webviews' WebGL contexts use an HMD pose from Unity so that it appears to occupy the same space. (July 2018)

Sometimes the outer world is the real world in a head-tracked environment (e.g. head tracking with a Wii remote, or with an iPhone X), or the contexts are completely separate. The camera rendering the portal content will need to use a projection matrix that accounts for its position relative to the viewer.

Johnny Lee's Head-tracking for VR using WiiRemote and a head mounted sensor bar. (2007)

Asymmetric Projection

Render cameras can be defined by near and far planes, and how far the edges of the view are from the focal point along the near plane. Most commonly, projections are symmetrical, meaning the left and right edges of the view are equidistant from the focal point in opposite directions, and same with the top and bottom. In asymmetric projections, the horizontal or vertical extents (or both!) are unbalanced, resulting in the focal point being off-center.

See my notes on 3D projection for more background on perspective projection transformations.

A camera's extents changing every frame, shown as lines, resulting in an asymmetric projection. Notice that the focal point, the rainbow ring, never changes, only what part of the scene is in view.

While traditional 3D graphics like a video game use symmetric projection, platforms leveraging real cameras likeARCore are often slightly asymmetrical. It's also common in VR applications, like Oculus' asymmetric projection features, which maximizes the field of view, as the center of the lens may not be the center of the screen.

Creating the Portal Projection

Robert Kooima's generalized perspective projection is the definitive writing for this problem, which explains how to compose transformation matrices for a system with 12x5 LCD screens that render relative to a viewer's head mounted display (HMD). This can be simplified a bit by abstracting away some steps inside a 3D engine (e.g. view matrix), and exploit the requirement that the portal camera must be oriented to face the portal plane.

A right-handed coordinate system (OpenGL/WebGL) is implied throughout.

Both the outer world and portal world need to use a shared coordinate system, and the portal camera's pose can be calculated by taking the following steps:

Set portal camera position to the viewer camera position.

Orient the portal camera to face perpendicular to the XY plane that the portal lies on.

Align the portal camera's extents to the portal gateway extents.

Figure 1 A simulation of walking through a space with a portal (left), and a debug view of the same scene, visualizing the frustum and extents of the portal camera scaled to the portal plane (middle), and the texture rendered by the portal camera (right).

Astronaut by Google (CC-BY-3.0), Camera by Jason Toff (CC-BY-3.0)

Notice that in Figure 1, the portal camera (middle panel) follows the viewer camera (left panel) in an arc, but is always oriented towards the closest point on the XY plane where the portal lives.

Following the viewer camera is trivial, but finding the closest coplanar point to the portal is a bit more work, further elaborated in Kooima's generalized perspective projection. Luckily, since a camera's forward vector is -Z , and the portal's forward is Z , applying the portal's rotation on the portal camera results in the portal camera facing perpendicularly to the portal's plane.

This trick only works when the camera is in front of the portal (e.g. portal.position.z < 0 in camera space). Otherwise, flipping some negatives or using Kooima's paper is necessary, but this should work for most cases.

The following code examples use three.js, and the portal is represented as an object with a position and orientation, with its scale representing its dimensions.

portalCamera.position.copy(viewerCamera.position); portalCamera.quaternion.copy(portal.quaternion);

Now the portal camera's extents need to extend to include the portal. Extents are measured from the focal point, or where the camera's direction and portal plane intersect. Note that extents may not have opposing signs, e.g. viewing the portal from the right can result in both the right and left extents being negative (see the middle panel in Figure 1).

Rather than storing 3D vectors for the edges of portals, storing as a 4x4 matrix (or position, rotation and scale in these examples) provides a few more shortcuts. Calculating extents in camera space is simplified because the portal camera is oriented perpendicularly to the portal plane. The portal's forward in camera space is (0, 0, 1) and lays on the camera's XY plane. In camera space, the extents can be calculated from the portal's dimensions, and X and Y values.

const portalHalfWidth = portal.scale.x / 2; const portalHalfHeight = portal.scale.y / 2; const portalPosition = new Vector3().copy(portal.position); portalCamera.updateMatrixWorld(); portalCamera.worldToLocal(portalPosition); let left = portalPosition.x - portalHalfWidth; let right = portalPosition.x + portalHalfWidth; let top = portalPosition.y + portalHalfHeight; let bottom = portalPosition.y - portalHalfHeight;

The above extents are calculated from the focal point to the portal edge on the portal's plane, Math.abs(portalPosition.z) units away from the camera. Extents are calculated from the near plane, however, so it is necessary to scale the extents from the portal plane to the portal camera's near plane.

As the camera is looking down the -Z axis, we can get the absolute value of portalPosition 's Z value to find the distance between camera and portal plane, which is also the distance from the camera that the extents were calculated at. Since frustum extents are calculated at the near plane, we can specify a near plane for the perspective projection and scale our extents accordingly.

const near = 0.01; const distance = Math.abs(portalPosition.z); const scale = near / distance; left *= scale; right *= scale; top *= scale; bottom *= scale;

With all of the extents, scaled to the near plane, it's possible to construct a projection matrix.

camera.projectionMatrix.makePerspective(left, right, top, bottom, near, far);

With the correct projection matrix set up, it's now possible to render a portal scene with the portal camera, and use that texture to be displayed elsewhere.

renderer.render(portalScene, portalCamera);

Summary

This technique can be used as a simplification of Kooima's work in modern 3D engines to set up a camera to render a texture that will be viewed from a different angle. These ideas and the following considerations are a valuable tool when creating immersive, spatial experiences across mixed platforms, ultimately using textures as an interface.

Alternatives & Considerations

near == distance

In the above solution, the extents are calculated from the closest point that is coplanar to the portal, d units from the camera, then scaled to some near plane n units from the camera. It's also possible to have the near plane d units from the camera, and not scaling the extents closer to the camera. This has the effect of clipping anything from the portal world that is attempting to render "outside" of the portal.

Figure 2 Similar views as Figure 1, except the near plane lays on the portal's plane, resulting in clipping the portal scene at the plane.

Astronaut by Google (CC-BY-3.0), Camera by Jason Toff (CC-BY-3.0)

Portals vs Windows

The examples shown have a static portal, although the technique also works if the portal was moving. If the portal's coordinate system is static and does not move with the portal, it appears as more of a window shifting along an overlapping world, like a filter. Moving the coordinate system with a portal gives more of a portal effect, such that the gateway is always facing the same direction, coincidentally like the game Portal.

Stereoscopic VR

For rendering a portal in VR, the same technique can be applied to both viewer cameras, one for each eye. Note that the webview-in-Unity videos are only rendering a single webview, resulting inaccurate projections for true stereoscopic experiences (but looks great as a video!).

Non-rectangular Portals

Stencil buffers!

Resources & References