Film cognition can help us understand why higher framerates look so weird, and what filmmakers will have to do to challenge 90 years of convention.

[Editor's note: This story was originally published on Dec 20th, 2012. We're resurfacing it this week as part of our tribute to the great feature work that writer Wes Fenlon has done with Tested, as he embarks on his new career in games journalism.]

Bilbo Baggins waddles down the dimly lit hallway of his cozy hobbit hole, its cramped quarters at once instantly familiar, though they've lain dormant since 2003's Return of the King. Familiar, yet something feels off. The old hobbit moves too quickly, and as he opens a chest, peering fondly at the relics of his adventures collected within, I expect him to pull out a well-worn trinket and suddenly appear on the set of Antiques Roadshow. Or to reminisce himself into a flashback and be carried away into a PBS revolutionary war reenactment, where Bilbo's outfit wouldn't look entirely out of place.

In its opening moments, The Hobbit's 48 frames per second cinematography overwhelmingly reminds me of a public broadcast television program, filmed at a slightly-too-fast 30 frames per second.

Since silent films gave way to talkies in the 1920s, the frame rate of 24 frames per second has become standard in the film industry. 24 fps is not the minimum required for persistence of vision--our brains can spin 16 still images into a continuous motion picture with ease--but the speed struck an easy balance between affordability and quality. For the past century, cinema has trained us to recognize 24 frames per second as a reflection of reality. Or, at least, a readily acceptable unreality.

Doubling that speed to 48 frames per second removes the motion blur and strobing of fast-moving images, argued Peter Jackson in 2011. "3D shows you a window into reality; the higher frame rate takes the glass out of the window," said James Cameron. But to the average viewer, 48 fps looks like an exaggerated version of a television program shot at the common video tape speed of 30 fps.

The stigma around higher frame rates leads to an important, and extremely complicated, question about how we perceive film: Why does 48 frames per second look so weird?

"[Film] is the medium we're exposed most to in our everyday life and it has evolved very rapidly in the last 100 years to permeate all aspects of visual culture. And yet so little is actually known of the psychology of viewers and how we make sense of what's presented on the screen in front of us," says Tim J. Smith, a lecturer in the psychological sciences department at Birkbeck University in London. Smith specializes in film cognition, studying how our brains process images and how perception interacts with the world of film.

In studying film cognition, Smith worked to link the language of film--moviemaking conventions and guidelines like the 180 degree rule--to their cognitive foundations. He came up with the attentional theory.

"The basic idea is that in the first few decades that film was around, at the start of the 20th century, filmmakers went through a rapid phase of self-experimentation," he says. "There were so many things they could do with the camera and with editing that they would try out things and see how they worked on themselves and see how audiences liked it. What they were doing was seeing which techniques were acceptable to their own visual system, which things made it easier for them to see what was happening on the screen and to make sense of the narrative. The things they experimented with that didn't work didn't get picked up by other filmmakers, so they died out very rapidly. You had this very rapid standardization towards how to shoot a scene and actually edit it together."

After about 90 years, that standard of the film language may be rewritten.

Part of that standardization was the frame rate of 24 frames per second. Now, after about 90 years, that part of the film language may be rewritten. The audience will have to learn to read again--and judging by The Hobbit's 48 fps presentation, filmmakers will likewise have to relearn how to write. Smith helped shed some light on the psychology of high frame rate film and why our brains so vehemently reject it. Read on.

How do I hate thee? Let me count the ways

Even though it contradicts our memories and expectations, kids could grow up watching 48 FPS video and find it perfectly palatable.

Smith's attentional theory makes me wonder if the way we watch movies is something we learn subconsciously from the first moments we sit in front of a television. "Certainly there's a familiarity effect," he says, after thinking about it for a moment. "We're aware of what a cinematic image looks like. Now we're aware of what a TV image, even a high def TV image, looks like. So when we see the film image at 48 frames per second, it's queueing all of our memories of seeing something similar in TV. That's why people are calling it the soap opera effect or bad TV movie effect. Because that's what it looks like, what it reminds us of.

"Whether we learned to expect it to be a certain frame rate...I don't think it operates on that level. These low level sensory behaviors are something we don't really have conscious access to, we can't really control it. Our eyes know how to make sense of the real world, and they know how to make sense of a still painting or a movie. We will change what we interpret in the image based on the way it's presented to us. But we can't really see the frame rate directly. We can only see the consequence of it in what we perceive."

This is perhaps the simplest and clearest explanation of why High Frame Rate projection looks so unpleasant: It contradicts our memories and expectations. Even if we haven't "learned" to expect 24 fps playback, by this explanation kids could grow up watching 48 fps video and find it perfectly palatable.

Five minutes into The Hobbit, I would've scoffed at that idea. My brain was, indeed, relating what I was seeing to some cheap television production. Despite Peter Jackson's casual claim that viewers will adjust to HFR in about 10 minutes, it took me at least an hour to really settle in. By the end of the movie, the increased speed was still very visible, but not as terribly jarring as it had been two hours prior. I think it's something we can adjust to, and it's worth remembering that, right now, most of the doom and gloom around 48 fps comes from people who have seen a mere two and a half hours of high frame rate video.

No increase in frame rate will ever exceed the presentation rate of reality.

According to another theory, our discomfort with high frame rates can be explained by studies in quantum consciousness. Filmmaker James Kerwin referenced a theory called orchestrated objective reduction while talking to Movieline about HFR, claiming that the brain essentially processes 40 "moments" in time every second, even if the eye can technically perceive about 66 frames per second. Kerwin claims the brain can easily distinguish 24 fps from reality but can't do the same for 48 fps, which throws us for a loop. Orchestrated objective reduction is far from a proven theory, however.

Smith shoots down the relationship between film and our limited perception of "moments" in time. "There is a little window there where, if you present things in sequence very rapidly, we won't be able to tell whether the first thing actually appeared before the first thing or after it, because our percept sloshes it all together," he explains. But that idea doesn't fly when it comes to movies. He elaborates: "I think it is quite a stretch to go from cortical electrical activity to phenomenological experience of time...the light reflected from a real-visual scene hits our retinas as a continuous stream (except when we blink or move our eyes), not a discrete series of frames as in film so no increase in frame rate will ever exceed the presentation rate of reality."

On a surface level, 48 fps motion simply looks too fast, like a movie playing on fast-forward. But there's something specific about HFR that makes it especially unsettling, which may explain why we find it horrifying in film but don't mind it in video games: the uncanny valley.

Uncanny Reality

"We're incredibly sensitive to the human form and the human face," explains Smith. "The closer you get to a realistic representation of it, the easier it is for us to spot the imperfections." This has long been a problem for CG animation, but most studios avoid the uncanny valley with cartoonish, exaggerated characters. Smith seems to think filmmakers will run into the uncanny valley shooting at high frame rates--but not just with CG.

"What I think we've encountered with the actors in The Hobbit, in the filmed parts of it, is that we can see so much more of the actual information in the scene," he says. "We can see the detail of the props and the detail of the costumes and of the makeup, and we can interrogate it a lot more, whereas previously the lower resolution of the image and the lower presentation rate meant that some of it was slightly blurry, so it was harder for us to interrogate.

"It's the equivalent in switching, in TV production, from standard definition to high definition. All of the TV companies now have to up their game in terms of makeup and costume because everything looks really cheap and tacky in comparison."

Nothing tears us away from immersion like the uncanny valley. In The Hobbit, it rears its head in the casual interactions between characters, the way they move in dialogue scenes. So what, exactly, are high frame rates good for? Action shots, which is where a comparison to video games makes sense.

"I think it's a good parallel because the reason you want...as high a resolution and as high a frame rate as possible in the game is because you need to know when that baddie's creeping up in the corner and you need to respond quickly and shoot him accurately," says Smith. "When you're watching a high speed action sequence in 3D, you need the same thing. You're enlisting the same parts of your brain in order to make sense of sequences panning down a mineshaft really rapidly as you are when you're running through a cavern in Quake or something."

The judder or strobing of 24 fps film stands out in fast-moving scenes, especially on large film screens. The greater the distance an object moves, the more noticeable the displacement as the object moves across your field of view. And for 3D, extra sensory information is even more critical.

"Whatever may have been said by Peter Jackson and James Cameron, the only reason why they're moving to higher frame rates right now is because of 3D," Smith says. We all know that 3D makes for a darker image--65 percent darker, in some cases--and the added dimension can make action scenes far more difficult to follow. "By going to 48 frames a second for each of the left and the right eye, they're giving you more information in the screen, which means you're more likely to see fluid motion in 3D," Smith adds.

48 fps doesn't make up for the darkness of 3D or cure the sensory overload of dense CG battles.

48 fps definitely adds a fluidity to The Hobbit's action scenes, and it transforms slow-motion into a thing of beauty. Still, it doesn't totally make up for the darkness of 3D or magically cure the sensory overload of dense CG battles. But it works far better than it does in the film's quiet moments.

An Unexpected Journey's theatrical release includes no 48 fps 2D showtimes. If the second installment is released in both 2D and 3D HFR in 2013, comparing the two will be interesting.

But even if viewers adjust to the new frame rate, filmmakers must reevaluate how they make movies. As technology evolves, so must style.

Changing filmmaking

While Smith believes that the battle between 2D and 3D will never see a complete changeover--there will always be 2D films, he says--he sees the move towards high frame rates as a turning point. "[High frame rate] could just become the de facto standard," he says. "Objectively, from the vision science side, it makes sense to move to higher frame rates because that's what our visual system needs to see motion. But if it's aesthetically not pleasing, and if filmmakers don't find a way of getting over this 'theatricality' look that seems to have been created in Hobbit, it may not be taken up. I think it's interesting because from the science side we can say directly why you might want it but artistically there's a lot of backlash against it and I think there's some grounds for that."

From a technology perspective, high frame rate video faces some challenges, but the march of progress should quickly render them obsolete. Yes, HFR video requires far more storage space. In a world of digital cameras, that's hardly the obstacle it was in the 1970s, when doubling the length of a film reel was a major expense. And digital projectors actually show images at 144 frames per second, repeating images multiple times to eliminate the flicker between stills. Analog film projectors use a shutter to the same effect at 48 or 72 fps.

Director Joseph Kahn tweeted about Peter Jackson's use of HFR 3D, noting the sensor of the RED camera shows its limits in The Hobbit. He elaborated with "HFR will work better when the sensors improve and latitude gets more powerful. A lot of video feel came from crap highlights" and concluded "I suspect 48fps on HOBBIT pushed the ISO up for more light causing thinner latitude metadata. Never seen RED Epic look that bad."

Cameras will improve, too, and we may well see even higher frame rates (and better image quality) in the next few years. Clarity will be king.

"I think the step up from 48 to 60 may not be noticeable if it had been filmed at the same frame rate," Smith says. "But what you've got if you film something at 60 [fps] is every single frame has less motion blur on it because the shutter is open for less time...I think that's what James Cameron is pushing for, because he wants a highly mobile camera. And if you've got a Steadicam, or a hand-held camera, the image between every frame can slip slightly. If you've got a lower frame rate, that slippage will cause motion blur...you'll see an advantage as you're going faster and faster."

The best-of-both-worlds option may be projecting movies at variable frame rates, showing slow scenes at 24 fps and ramping the frame rate up to 48 fps for action scenes. frame rate could become a stylistic choice, just like lighting or set direction. Douglas Trumbull, who worked on the visual effects of 2001: A Space Odyssey and Blade Runner, developed a 60 fps, 65mm process in the late 1970s. It never caught on, but he's returned to the idea in recent years. Trumbull's website describes a camera system he's designed that could popularize variable frame rate filming:

Shooting at 120 fps makes it possible to digitally merge adjacent frames to recover the blur necessary for 24 fps display.

"I have been shooting at 120 fps, using a 360-degree shutter in the camera. This makes it possible to digitally merge any number of adjacent frames in order to recover the appropriate amount of blur necessary for 24 fps display. Keep in mind that the movie medium we are accustomed to has used a 160-200 degree shutter, resulting in the "texture" that we know as movies (not television). Having shot material at 120 frames with a 360-degree shutter, it is a simple and perfect conversion to digitally merge three frames into one, and delete the next two frames, thus resulting in a 24 frame movie without any artifacts, and while retaining the normal blur. This patented process will provide a compression of visual data that will bring immense improvement to the viewing experience, and also offers the unique opportunity to "embed" 60 fps object motion within a 24 fps overall "look", thus preserving the cinematic texture while enabling unblurred fast action."

As more directors experiment with high frame rates, they'll have to be prepared to change the way they make movies. In the 48 fps Hobbit, Martin Freeman's visual gags fall flat. A single sequence of jump cuts feels oddly quick and jarring. The acting often feels like a stage play, with the actors oddly positioned, the dialogue exaggerated. The Hobbit no doubt has issues, even in 2D--reviews of the movie have been split down the middle--but HFR so dramatically changes the look and feel of a movie that all these basic elements will have to be tweaked to recapture the nebulous "cinematic" quality.

"The fact that you could get away with an imperfect representation of a world, yet it would be perceived as beyond real by the viewer...in some ways that was the magic of cinema," Smith says. "The fact that you could have polystyrene sets and dodgy costumes and the lighting was off and the color balance was all weird, but when you put it up in the image, it had this magic that people would invest in and really believe what was being presented...the solution, given the increased resolution of the image and the increased frame rate, is to try to make things better and more realistic. I do wonder if the downside is that it loses some of the magic of the image."