The third category is temporal. Temporal attributes have to do with motion and time, including things like motion blur, exposure time, frame rate, and sweep speed of a rolling shutter.

Some perceptual attributes don't fit neatly into exactly one of these three categories. For example, film's gate weave (its slight unsteadiness from frame-to-frame) is both temporal and spatial. Or film's grain: that's an attribute that can be addressed equally as temporal and spatial or as temporal and intrapixel.

7. MANIPULATING THE ATTRIBUTES

Moving on from categorizing attributes, let's talk about manipulating them. I'd say there are also three very broad categories of things we need to do with these attributes if we want to be authors of the photographic look.

CATEGORY 1: EMPIRICAL IDENTIFICATION

First, we need to do the work to identify empirically rather than intuitively which of these attributes exist and/or which are perceptually important to us by doing isolated comparisons, so we can actually understand unambiguously their visual and technical impact rather than guessing or assuming. We can't manipulate visual elements creatively if we don't know what they are or how they affect the eye. The previous section merely gave broad categories in which to place the attributes -- it didn't exhaustively identify the possible ones individually or investigate their importance, but doing so is necessary if we want to attain expertise in photographic look.

For example I've often heard the unsubstantiated assertion that film negative can be scanned at ever increasing pixel counts to retrieve ever more resolution information out of it. But film has limited resolution. It's an analog limit instead of a digital one, but it exists. The fact that you don't know how to easily quantify the limit does not mean it's unlimited; it just means that you don't understand the limit. To understand it, you'd have to do a controlled variable test where you scan at various resolutions and see where the advantage breaks down. The same goes for all kinds of attributes which filmmakers too often wrongly assume they know something about just by intuition, by supposed common knowledge, or by doing flawed or biased tests where variables aren't properly isolated.

Grain, contrast, spectral response and many others: we need to study the actual perceptual effects of various aspects rather than making assumptions, because unambiguous and undeniable empirical data often stands in stark contrast to intuition, to common belief, and to anecdotal or cherrypicked evidence.

One of the most prevalent unsubstantiated assumptions is that one single attribute (as opposed to the aggregate of many) is solely responsible for an entire photographic look. For example: the intuitive and unfounded presumption that I've sometimes heard asserted that the photochemical look is comprehensively induced by grain alone with no credit being given to celluloid's other attributes. Or another example: the belief that the so-called "video" look is invoked entirely by edge sharpness alone, when it actually arises from many attributes. Yet another: that counting up the number of photosites on a digital camera's sensor is a comprehensive measure of how "good" the camera is.

CATEGORY 2: DATA COLLECTION

We must push for more rigorous and meaningful evaluation of camera systems.

Today vendors can count on brand allegiance and confirmation bias to guide us and to ensure that our opinions are resistant to scrutiny. They can count on us to psychologically project onto a product any attributes we've been conditioned to believe it has, to cherry pick anecdotal evidence to support preconceptions, and to design biased comparison tests that don't isolate variables. To be authors of our photographic look we must reimagine what a camera test even is to break the cycle of bias.

It's tempting (and common these days) to try to reduce all of the complex qualities and unintuitive attributes of a camera system down to a single slogan that ostensibly encapsulates all of its characteristics. Like: "Kodak's film is magic," "Arri's Alexa is filmy," "Red's Weapon is 8k," or "Sony's F65 is the only true 4k camera." But whether these mantras are more subjective sounding like the first two or more technical sounding like the latter two, these oversimplified slogans (even if true in some sense) don't get us closer to useful understanding, they merely evoke imagination to produce expectation; they reinforce preconceptions and belief bias. They discourage curiosity and instill confidence that we need not investigate any further, while not providing any essential information to be confident about. Simplifying is not the same as clarifying.

We need to test our capture formats in rigorous ways to understand if a camera system is getting enough information and information of the right kind for the look we want. Because as I said earlier, any format can be molded to any aesthetic look, but only if the acquisition format captures enough information to do so. We also need tests that precisely map how a camera packs image data into its media so that the data can be manipulated in a meaningful way further down the image chain.

So, we need camera tests that separate pure data collection from aesthetic look. We also need to separate underlying data from mere out-of-the box display preparation.

Here's a hypothetical illustration of this imperative:

Let's say I'd like to design a traditional photochemical look for a movie and that in preproduction I'm testing two digital cameras as candidate capture devices. Let's imagine that Camera A has manufacturer-provided color mapping that looks to my eye kind of film-like but the camera cannot retain as much highlight data as film negative. Let's also suppose that this camera's rolling shutter sweep time is much slower than a film camera's (yes, it may come as a surprise but film cameras have rolling shutters too. They just have shutters with fast sweep time).

Now let's say that Camera B has manufacturer provided color mapping that looks very garishly electronic and off-putting to my personal taste but the camera has better highlight retention than film does and a shutter sweep that's perceptually indistinguishable from film's.

Well, if I don't understand which attributes of the camera are actual data collection and which are sculptable parts of the look, I may foolishly choose Camera A merely because it looks more filmy to me out of the box or in a test where variables aren't isolated. But Camera A's actual data collection is inferior to the system that I'm trying to emulate (print film). I can never retrieve the lost highlight data from its limited latitude and I can never retrieve the lost temporal data from its slow shutter sweep. Camera B collects more actual data about the scene in front of the lens than Camera A does, so Camera B is the only one of the two candidates that collects enough data to properly emulate a traditional photochemical imaging chain for my taste.

But I need to sculpt the data! I need to replace the manufacturers color mapping that I don't like with a mapping that represents my intent. Recovering data that was never captured is impossible. But what is possible is to remap captured data to our desired look. So Camera B is a better choice.

This has been a simplified example, and the concept applies to many other attributes.

If we rhapsodically intone that a camera has inherent aesthetics and refuse to isolate its technical attributes, all we're doing is limiting creative choice, not expressing it. Reciting the familiar mantra that selecting a camera type is a "creative decision" is really just a way to disavow ourselves of creative control and leave it up to the off-the-shelf bundle provided by our chosen brand.

So how can we sculpt that data to have the desired display attributes if the camera doesn't do it for us automatically out of the box?

CATEGORY 3: TRANSFORMATIONS

In Category 1, we study how aesthetic sensations arise from technical attributes, so that we know what the visual building blocks are empirically, instead of relying on intuition or hearsay. In Category 2, we isolate variables to find out how much information of various types our imaging systems are capturing and how that data is organized within the recoding media. And now, in Category 3, we need to sculpt the camera data for display, giving it all of our desired attributes and authoring the photographic look. We've precisely tracked the data so we know where it's coming from, and we've unambiguously designed a destination for it... but we need a remapping process to bridge the two. So, we need some sort of algorithmic tools for doing these types of complex transformations. This isn’t something you have to do yourself if it is overwhelming, but it’s important as authors to have some understanding of the steps involved and add them to our lexicon.

There are unlimited possible transformations that can be used either to invent a look from scratch or to use data sets to build rigorous mathematical models and then map one system's response onto another's. Let's look summarily at some of the algorithms I used in the Display Prep Demo to conform the three types of attributes. Starting with the intrapixel attributes:

A simple but powerful transformation is tone mapping. This is similar to "custom curves" in a color corrector except it's determined with a combination of empirical experiment and math instead of by subjectivity. In this way, we match one system precisely to another in its tonal response all through the latitude range.

But tone mapping only matches density, luminance, and contrast but not complex color response, so we need more than that. We also need to sculpt data points all through the three dimensional color space in a way that is more complex, nuanced and idiosyncratic than what can be effected with usual color grading tools.

To do this we need both three dimensional geometry and scattered data interpolation. These are two broad categories of processes by which we can either make aesthetic changes or take large data sets about two capture systems and map one onto the other even if they differ in overall shape and/or by many local irregularities.

With 3d geometry, we can sculpt and reshape the whole constellation of 3D color data in various ways that are much more complex than can be done with a conventional color corrector yet are smooth, uniform contours that don't rip the image apart. Or, we can achieve even more complex and non-intuitive transformations with scattered data interpolation, which can re-sculpt the overall shape of our image data in much more irregular and non-uniform ways than even 3d geometry and can also give us very localized idiosyncrasies that don't affect the overall shape.

The animation below shows two data sets regarding color response, acquired in controlled experiment: one for Arri Alexa and the equivalent one for 35mm film (Kodak 5219, scanned with carefully defined settings using the Scanity film scanner). Scattered data interpolation can map one data set precisely onto the other, which is a transformation much richer and more complex than traditional color grading can achieve. In the animation, the data points are graphed in 3D color space (the three axes are Red, Green and Blue), and the view is rotating to help show on a 2D screen what is happening in three dimensions.

Scattered Data Interpolation:

complex and irregular (yet smooth!) re-mapping of color data

Your browser does not support HTML5 video.

That's a summary of intrapixel attributes, but what about spatial and temporal aspects?

Well, for film grain, I've developed my own algorithm which, unlike many grain plug-ins that try to record and then repeat apparent grain geometry that happened to occur on one specific occasion on one specific strand of film, I've taken an empirical data set of real film, studied the probability distribution for various grain amplitudes and emulated that probabilistic distribution of those amplitudes. Thus I use a totally empirical model of real film... But the algorithm is probability-based instead of geometry-based.

I have similarly empirical custom algorithms that I've developed to make mathematical models of film's halation and film’s gate weave. These are attributes whose existence and contribution to the overall look often isn't even recognized, let alone modeled rigorously.

This section has been a mere overview -- an abstract peek into these processes. Getting into functional details is beyond the scope of this document, but I hope some readers will be inspired to dive in for themselves and look at all of this much more closely.

8. NEXT STEPS

Now let's refocus away from finer details and take a broader view: think about what this all says about the current state of affairs for filmmakers: On the one hand this is all possible and within reach today and not just in a sci-fi future. But on the other hand I had to build custom tools from scratch because existing off-the-shelf solutions were not suitable. This says a lot about where we are with these color science studies at this point in history. We are simultaneously woefully mired in legacy problems and yet poised for a big leap forward.

How will we proceed?

If you're a filmmaker who is interested in these possibilities for more meaningful authorship but daunted by the math and computer science, fear not...

Film is a collaborative medium -- being an author instead of a shopper doesn't mean that you have to do everything by yourself. If filmmakers take the first step of merely recognizing what is wrong with our current lexicon, our current preconceptions, and our current dominant narrative, we can begin the long work to resculpt them and to gain momentum for an exciting paradigm shift.

One thing you can do right away without the study of dizzying math is to collaborate with a color scientist at your regular post house. Properly trained color scientists are under-appreciated, and I bet they'd love to collaborate with the type of clients who appreciate their unique expertise. And if your post house doesn’t have a color scientist, push them towards more rigorous methods. If we're going for a film look, we want a comprehensive perceptual model, not empty lip service. Just saying the words "color science" or "film print emulation" doesn't mean you've done the work. As filmmakers, a powerful step forward is merely to recognize that there exists a special skill set unique to a proper color scientist that has little to no overlap with the important but different expertise of a colorist, an on-set digital imaging technician, or a workstation engineer.

Some of these complexities may sound like an overwhelming amount of work, but the exciting part is that doing just a bit of homework can not only open up creative options, but also make both your shoot and post production much simpler, easier, and purely creative. This document isn't a call to filmmakers to do lots of (or any!) color science tinkering while making a movie -- not on set and not in post, but to avoid doing so merely by getting some ducks in a row before shooting. I personally use these techniques just to have everything set up the way I like it in advance, so that by the time I'm shooting I just use a light meter and traditional film lighting ratios -- I don't even use a calibrated monitor, let alone a cumbersome tent full of rack-mounted engineering equipment tethered to the camera. And then in post, color grading is focused and doesn't spiral because much of the intent is already there in the starting point -- in the core transformation -- so it doesn't have to be built from scratch shot by shot. In both production and post, I can be nimble and concentrate on the creative aspects of making a movie rather than on engineering.

I hope this text along with the Demo can be an inspiration to filmmakers: a reminder that we can be authors instead of shoppers. That we can be masters and not slaves of the tools we use. And that it is not beyond our grasp to understand the component attributes that go into the processes rather than letting vendors hand us turn-key systems that we don't understand. Although it does take a little bit of education and shedding of preconceptions, it can be incredibly freeing without being daunting.

I've put a lot of my own color science efforts into the admittedly narrow endeavor of modeling traditionally printed film, but I believe that embracing the importance of display preparation in our artistic voices can do so much more than that in the future. My narrow application is proof of concept of broader applications: that it has the power not only to free us from the tethering of aesthetic looks to camera brands, but to further free us to invent completely new aesthetic looks as well. I haven't as yet pursued the latter path myself, but I hope this will inspire some readers to do so.

It's not the tools that you have faith in - tools are just tools. They work, or they don't work. It's people you have faith in or not. -Steve Jobs

----