Benchmarking the experience of (current) digital images

Jinsop Lee, a speaker at TED 2013, states that in order to design products that people enjoy, it is important to design for all 5 senses.

According to him, the impact of an experience can be identified by how much an activity affects each of our five human senses: sight, touch, smell, sound and taste.

To determine the richness of an experience, you could ask a group of people to rank their subjective impression of an activity on a scale from zero to ten for each sense. When you place the results into a diagram with the senses represented on the x-axis and the rating on the y-axis, you get a graph which you can compare with experiences of other activities.

Fig. 1 — A perfect experience

For a comparison, Jinsop Lee applies his theory to the experience of riding a motorbike. After going on a ride together with his friend, he rated the impact on each sense. The resulting graph (Figure 2) makes visible and comprehensible why there is such a fascination for motorbikes if compared to the graph of a perfect experience.

Fig. 2 — The experience of Riding A Motorbike

With that theory as a starting point, I thought about how we currently perceive visual content on the internet. After browsing on various online news pages and social networks, I rated ‘sight’ as a six and the remaining senses as zero (Figure 3) because none of the contents, apart from videos, did affect any other senses.

Fig. 3 — The experience of Watching Digital Photographs

I also came to the assumption that the sensory impulse varies depending on how much the seen catches our attention and causes us to linger on it.

But when looking at the resulting graph (Figure 3) it becomes instantly clear that these images are not even close to a perfect experience. Even if I would have risen the value for ‘sight’ to a ten because maybe I saw an absolutely stunning photograph; the graph would have never grown nearly to the same heights.

Conclusion

The exclusion of senses other than ‘sight‘ makes watching digital photographs a comparably poor experience.

The only catch is by making the viewer pause on an image, for example through interactions or its content itself, and by involving other senses, such as the sense of sound perception.

Alternatives to plain digital images

A couple of techniques exist which attempt to make images less passive and more dimensional.

Stereographs and flip images

For example, there are phone apps that achieve an illusion of depth either through stereographic images which require “red and cyan” glasses (known as the complimentary colour anaglyph method) or in form of digital flip images (Figure 4) (making use of the device’s built-in sensors).

Fig. 4— Fyuse App / Staff Pick @misstvmmy

Photogrammetry

A more “serious” approach to create three-dimensional photography is called stereo-photogrammetry. It is a technique where three-dimensional data is derived from multiple photographs which are taken from different positions. Through computer software those flat images are then mapped back as textures onto the digital 3D objects.

The probably best-known showcase is Google Map’s 3D View feature. It is available for lots of famous locations around the globe such as the rock formation El Capitan in the Yosemite National Park, the Pałac Pena in Portugal, the Hoover Dam in Nevada/Arizona or the city Hongkong in China.

Many of those locations are collected on Google Earth VR. It is a platform that belongs to Google VR which is in turn a site where the company promotes its developments for Head-Mounted-Displays (Figure 6) (HDMs) Oculus, HTC Vive, Google Cartboard and Google’s softer fabric headset Daydream View as well as production hardware for filmmakers and developer-tools for programmers.

Fig. 6— Introducing Oculus Go; Photo: Oculus

Virtual Reality

As the name Virtual Reality (VR) suspects, it merges behaviours of the physical world with the digital world. HDMs act as windows to a virtual, spherical surrounding. Combined with spatial audio, it is possible to fully immerse into a new reality.

Such experiences can be created with 3D models which are constructed entirely on computer. This method has the advantage that virtual elements can be three-dimensional and reactive to user inputs. A great example is the “Cardboard Design Lab” (Figure 7). It is a mobile application that takes the best practises as learned in industry and demonstrates them. That way it educates about virtual-reality in virtual-reality.

The alternative production method to Computer Generated Images (Figure 7) (CGI) is the use of optical systems that allow recording or capturing 360-degree images e.g. through an array of cameras (Figure 8).

Fig. 8— Jump used by Anthony Bonello; Photo: GoogleVR

In experiences realised this way, the viewer is limited to looking around spherically from a fixed point of view and is not able to explore a three-dimensional space independently. But it is still a stunning type of storytelling as documentaries e.g. by arte or National Geographic prove.

The viewer can be guided to places he would probably never be able to see, such as the Earth from 20 miles (32 km) above from inside a weather balloon (Figure 9).

The missing link

Ok, so if there exist a couple of solutions already, what’s the purpose of this article?

Well, in regards of virtual reality a lot of development is happening. Especially the large companies Google, Facebook, HTC and Apple are constantly pushing the borders of what is possible. There is just one caveat:

VR that is based on real 360° shoots and recordings is far from being on par with computer-generated environments.

And I believe this should not be like that. Due to their narrower relation to reality, photographs and live-action movies are much more honest compared to their computer-generated counterparts. With CGI possibilities are pretty much endless and the results are often “too perfect” to be credible.

The hurdles

Most problematic is the implementation of three-dimensionality that is captured from more than two perspectives and thus allows the viewers to change their point of view on an object.

Photogrammetry, as a mix of both techniques, comes closest to this desired result. But aesthetics (Figure 5) and the workflow differ a lot from creative photograph- and filmmaking. This means a steep learning curve for people in those professions and a slow implementation to their field of work.