Responsive Images Techniques & Beyond Yoav Weiss Velocity EU - London, November 2013 yoavweiss.github.io/velocity-eu-13-presentation here to talk about responsive images, why they're important, what are the use cases and what you can do about it today. Then I'll also give an overview of what the future might look like Need to prepare before: sizersoze test page + results, har before and after preloader

Who????



Picture prototype

srcset implementation Working on resonsive images in my spare time for the last 2 years RICG, Blink, WebKit implemented picture prototype, srcset Went over all the current techniques Here to share that roller-coaster journey

Responsive images problem Load properly dimensioned images that fit the page's design in an efficient manner who knows what the responsive image probem is?

Why bother?

A trip down memory lane

In the begining - mobile only sites Mid 2000s, mobile only sites. Very light weight. iPhone, slightly more content, but still mobile-only

Followed by - iPhone

Then - RWD happened RWD became a thing One code base to rule them all Awesome

Images? Not a problem!

Send the largest image you'd ever need

Let the browser resize it

EAZZZZZZY!!!

Well... That turned out to be... sub-optimal.

This led to BLOAT!!!> Mobile experience became better, but slower "a responsive site" became a synonym of "a slow site"

72% Serve same resources guypo.com Serving the same resources to mobile and desktop hurts performance On mobile the images are much smaller and some of them are not even displayed

Which resources? Images - over 61% httparchive.org

How much can be saved? Up to™ 72% image data savings tkadlec.com A small utility I cooked up and Tim Kadlec wrote about showed 72% data savings

How can we measure? We need to run a perf budget, with no way to measure waste

Sizer Soze Following a post by grigs, hacked together on a saturday morning

What?? Measures the difference between current images & perfectly sized images

How does it work? Download a page using phantomJS

Look at image dimensions

Download the images

Resize to displayed dimensions

SSAAS sizersoze.org Built with help of awesome people from the RICG and the Filament Group. Special thanks to Sven Read for the design and Jeff Lembeck for the frontend work.

Demo time!

Give it a try! sizersoze.org Later... and please be patient! This is the first time this sees any kind of non-test traffic, so I expect it to choke :)

If you want to help out github.com/yoavweiss/Sizer-Soze github.com/ResponsiveImagesCG/Sizer-Soze-frontend Very much in the "pet project" phase, so bug reports, etc will be appreciated

Retina only increases these gaps The difference between low end, small devices and high-end wide devices increases Devs can choose between HQ and LQ for everybody Usually pick HQ

OK, bandwidth is important But It's getting better every year Soon this whole issue will go away

Rendering performance Bandwidth is not the only factor here

Decoding speed impact x2 images take x2 to decode

x6 images take x15-x23 (!!!) to decode tkadlec.com

Resize impact More or less the same time as decoding tkadlec.com

But it's not on the main thread

It still takes CPU time Scrolling lag observed

Battery life may suffer

Memory impact Browsers store encoded, decoded and resized image in memory Browsers can be smart about it and evict some of this in memory cache earlier, but currently, it's a cost

Decoded image memory footprint 4 bytes per pixel - W×H×4

Larger images cost more Dimensions Memory footprint Diff 800x800 2,560,000 400x400 640,000 1,920,000 200x200 160,000 480,000 You can see that the larger the display dimensions of the image, the cost a 2x image incurs is heavier.

Responsive Images Problem 2 major use cases Let's better define what we're talking about when we're talking about the "responsive images problem"

Resolution switching Serving different image dimensions to different devices Image resources only differ in quality. proportions stay the same

DPR switching Sending retina images only to retina screens

viewport switching Sending images adapted to their display dimensions For both res switching, browser should be able to serve lower res resources

Art Direction Getting images to match layout Fit layout and optimal content without wasting bytes

Survey - 40% of devs hack their images to do art direction

japborst.net 23% of responsive sites have art-directed images

Other use cases Monochrome

Print

Image format fallback

Solution constraints Must play well with preloader Since the goal is performance afterall

Use cases doc usecases.responsiveimages.org Released as a W3C WG note Check out the full use cases document if you want to learn more We got upgraded, since there's no CG note

So let's hack around the problem AKA: So, when first encountered this issue, Web devs tries to route around it using the regular methods. But something got in their way

But first...

Who's afraid of the big, bad preloader [Raise your hand if you know what the preloader is]

AKA PreloadScanner

Speculative parser

Look-ahead scanner

"The greatest browser optimization of all times" - Steve Souders

What does it do? Scans the document when the parser is blocked

Figures out resources that need to be downloaded

Adds them to the download queue

How important is that???

SmashingMag - With preloader DCL - 2390; PLT - 3585



Using chromeHAR DCL - 2390; PLT - 3585

SmashingMag - No preloader DCL - 5395; PLT - 7193 DCL - 5395; PLT - 7193

BBC - With preloader DCL - 5022; PLT - 5980



Using chromeHAR DCL - 5022; PLT - 5980

BBC - Without preloader DCL - 7660; PLT - 7957 DCL - 7660; PLT - 7957

Going back to images People came up with some JS based ideas None of them really worked

Turns out You can't modify resources with JS before the preloader

Early days techniques

Cookie based Doesn't work on first load The cookie is not guarantied to be set on the page's images, since they go into the download queue before it is set

Base href modification Terminates in-flight preloaded requests If the base is modified, all realtive URLs the preloader kicked off must be terminated. That's a perf disaster.

Switching src value Double download regardless of preloader... Even without the proloader, the parser triggers an img request as soon as the attribute is parsed.

This led to FRUSTRATION

Working techniques

Cookie based e.g. adaptive-images.com

Example <script> document.cookie = 'width=' + screen.width + ';height='screen.height + ';dpr=' + devicePixelRatio;</script> <img src="cat.jpg">

Advantages No need for markup change

Images download fast

No double download

Image displayed when JS fails

Disadvantages On first page load, user will get wrong image

Cachebility issues

Same domain restriction

Server side solution which may or may not fit your stack

Server-side logic/directory structure to map viewport dimensions to image dimensions

Resizes are not taken into account

LQIP Low Quality Image Placeholder

Example <img src="cat_lq.jpg" data-normal-src="cat.jpg" data-hq-src="cat_hq.jpg"> ... <script> var images = document.getElementsByTagName("img"); var imagesLen = images.length; var hd = screen.width > 800 || devicePixelRatio > 1; for(var i = 0; i < images.length; i++) images[i].src = hd? images[i].dataset.hqSrc: images[i].dataset.normalSrc; </script>

Advantages LQ images download fast. Prog JPEG-like

Can work with no server side logic

LQ image displayed when JS fails

Disadvantages Double download, so some BW waste

Markup contains multiple resources

Final quality image download starts late (near DCL)

Picturefill Based of the <picture> markup pattern

Example <span data-picture> <span data-media="screen and (min-width:1600px)" data-src="wide.jpg"></span> <span data-media="screen and (min-width:800px)" data-src="middle.jpg"></span> <span data-media="screen and (min-width:400px)" data-src="narrow.jpg"></span> <span data-src="tiny.jpg"></span> <noscript> <img src="fallback.jpg"></noscript> </span> ... <script src="picturefill.js"></script>

Example #2 <span data-picture> <span data-media="screen and (min-width:1600px)" data-srcset="wide1x.jpg 1x, wide2x.jpg 2x"> </span> <span data-media="screen and (min-width:800px)" data-srcset="middle1x.jpg 1x, middle2x.jpg 2x"> </span> <span data-media="screen and (min-width:400px)" data-srcset="narrow1x.jpg 1x, narrow2x.jpg 2x"> </span> <span data-srcset="tiny1x.jpg 1x, tiny2x.jpg 2x"></span> <noscript> <img src="fallback.jpg"></noscript> </span> ... <script src="picturefill.js"></script>

What does it do? Meant for art-direction

Can do viewport switching when abused

DPR switching using data-srcset in an experimental branch

Advantages No double download

With scripts disabled, there's a fallback

Using MQs which is a familiar pattern to authors

Disadvantages Markup contains multiple resources

MQs in the markup can get verbose

Image download starts only after DCL

x-picture Web components based solution Web components based <picture> polyfill

polyfill Image download triggered when element is parsed (in supporting browsers)

Depends on polymer Web components based solution

Example <html> <head> <script src="x-picture.js"></script> </head> <body> <x-picture> <source media="screen and (min-width:1600px)" srcset="wide1x.jpg 1x, wide2x.jpg 2x"> <source media="screen and (min-width:800px)" srcset="middle1x.jpg 1x, middle2x.jpg 2x"> <source media="screen and (min-width:400px)" srcset="narrow1x.jpg 1x, narrow2x.jpg 2x"> <source srcset="tiny1x.jpg 1x, tiny2x.jpg 2x"> <noscript> <img src="fallback.jpg"></noscript> </x-picture> </body> </html>

Mobify.js A whole new level of Bat-sh*t-loco-insane™ hack

What? Enables script access to HTML document before preloder or parser see it

How????? Inline script at the doc head injects a <plaintext> tag and downloads mobify.js

Browser treats the rest of the document as text

HTML chunks are read & processed as they're downloaded

Once HTML is done, document.open() and document.write() the new HTML

Page load starts over, preloader and all

Advantages No markup changes needed

No double download

Preloader works (after a delay)

No JS means no optimization, but everything works

Disadvantages Blocking JS at the page's start

Entire HTML is downloaded before parsing start

Compatibility issues

Lots of invasive JS may be hard on legacy devices

JS failure after plaintext injection can result in a blank page

Service Worker

What? Originally, an offline API

Enables URL rewriting in the browser

Implementation underway in FF & Chrome

Let's use that to hack responsive images!!!

Unfortunately Doesn't (yet?) support first load

Can't access window

Browser support will take time

Compressive images No markup, no JS

Take 2x images

Compress them to LQ

Let browser handle resize

Looks good on retina

Not bigger than 1x image

Downsides Memory footprint

Decoding and resizing speed

"Looks good" is subjective

Only DPR switching, up to 2x

Bandwidth testing

Active bandwidth testing e.g. foresight.js Download an object and measure

Download images according to measured BW

Adds a blocking, useless resource e.g. foresight.js

Should I do that???

Passive bandwidth testing Measure HTML download time - NavTiming

Download images according to measured BW

Less bad, but still not a good idea

Related stuff Image resizing services

CMS plugins

Grunt plugins

How to decide? Can I modify the page at all?

Can I modify the markup to contain the image data?

Can I control the server-side serving the images?

Performance take-aways Using src results in a double download

results in a double download Not using src means fallback is not guarantied

means fallback is not guarantied Start downloading images ASAP

Measure perf impact: network, memory and decoding

Proposed standard Solutions srcset attribute

attribute <picture> element

element Client-Hints headers

Src-N

Response Image Container We've covered the problem and the hacks. Now let's move to the standard solutions.

srcset Proposed by Apple, adopted by the WHATWG

Extends <img> to include multiple resources

to include multiple resources Addresses mainly resolution switching The good old img tag, now with extra attributes!!! Mainly res-switch, because browser can override Browser can override means "browser can optimize"

Current implementation DPR switching only <img src="cat.1X.jpg" srcset="cat.1x.jpg 1x, cat.2x.jpg 2x"> Implemented (not yet shipped) in WebKit and Blink Rough consensus around the syntax

srcset's viewport switching syntax <img src="cat.1X.jpg" srcset="cat.1x.jpg 1x 320w, cat.2x.jpg 2x 320w, cat.2x.jpg 1x 640w, cat.4x.jpg 2x 640w, cat 4x.jpg 1x 1280w"> Not many fans for this part of the syntax

<picture> Proposed by Bruce Lawson, adopted by RICG

Adopts <video> and <audio> markup patterns

and markup patterns Targeted at art-direction

Specifies resources using <source> and MQs

and MQs First matching resource downloaded and displayed

Art direction using picture <picture> <source media="screen and (min-width:1600px)" srcset="wide1x.jpg 1x, wide2x.jpg 2x"> <source media="screen and (min-width:800px)" srcset="middle1x.jpg 1x, middle2x.jpg 2x"> <source media="screen and (min-width:400px)" srcset="narrow1x.jpg 1x, narrow2x.jpg 2x"> <source srcset="tiny1x.jpg 1x, tiny2x.jpg 2x"> <img src="fallback.jpg"> </picture>

src-N Proposed by Tab Atkins and John Mellor

Got the RICG's support Tab Atkins' and John Mellor's proposal Pretty soon got the RICG's support behind it Fuses in picture, srcset and viewport-switching done right

src-N's DPR switching x-urls - Basically srcset's DPR syntax

Example <img src-1="pic.png, picHigh.png 2x, picLow.png .5x"> Same as srcset!!!!

src-N's art direction Multiple attributes, each can start with an MQ Picture's source elements squeezed into attributes

Example <img src-1="(max-width: 400px) pic-small.jpg" src-2="(max-width: 1000px) pic-medium.jpg" src="pic-large.jpg" alt="Obama talking to a soldier in hospital scrubs.">

viewport-urls Variable sized images

Composed of Breakpoint definition

Resource & dimension list

Linearly variable image

<img src-1="100%; pic1.png 160, pic2.png 320, pic3.png 640, pic4.png 1280, pic5.png 2560"> A simple "image takes whole width" example

Multiple breakpoint variable image

<img src-1="100% (640px) 50% (960px) 33%; 160.jpg 160, 320.jpg 320, 480.jpg 480, 640.jpg 640, 960.jpg 960, 1280.jpg 1280, 1920.jpg 1920">

Breakpoint definition Officially: size-viewport-list 100% (640px) 50% (960px) 33%;

Resource & dimension list Officially: size-based-urls 160.jpg 160, 320.jpg 320, 480.jpg 480, 640.jpg 640, 960.jpg 960, 1280.jpg 1280, 1920.jpg 1920

Multiple breakpoints with srcset

<img srcset=" 320.jpg .89x 400w, 480.jpg 1.33x 400w, 640.jpg 1.78x 400w, 480.jpg 1.04x 520w, 640.jpg 1.39x 520w, 960.jpg 2.09x 520w, 640.jpg 1.1x 639w, 960.jpg 1.66x 639w, 1280.jpg 2.2x 639w, 320.jpg 0.89x 800w, 480.jpg 1.33x 800w, 640.jpg 1.78x 800w, 480.jpg 1.09x 959w, 640.jpg 1.45x 959w, 960.jpg 2.18x 959w, 320.jpg 0.89x 1200w, 480.jpg 1.33x 1200w, 640.jpg 1.78x 1200w, 480.jpg 1.09x 1440w, 640.jpg 1.45x 1440w, 960.jpg 2.18x 1440w, 480.jpg 0.86x 1920w, 640.jpg 1.14x 1920w, 960.jpg 1.71x 1920w, 1280.jpg 2.29x 1920w, 640.jpg 0.86x, 960.jpg 1.29x, 1280 1.71x, 1920 2.57x ">

Client Hints Proposed by Ilya Grigorik

HTTP based content-negotiation solution

Client sends hints/capabilities

Server side logic decides on resource

Request headers

CH-DPR

CH-RW Resource Width Resource Width Implicitly relies on src-N's viewport-url

Response headers

DPR Server confirmation header Used for intrinsic image sizing

Opt-in only Not sent on initial HTML request

Responsive Image Container File format approach

A "layer" per resolution

Both resolution switching and art-direction

Resolution switching 336x635

Is split into

106x200

211x400

336x635

Art-direction 770x512

Is split into

200x200

400x288

770x512

Advantages Markup left untouched

A single file per image

Better for post-download dimensions changes No markup single file can download diff

disadvantages Touches many layers. Will take time

Decoding performance???

Fetching mechanism network performance??? complicated decoding perf? Fetching mechanism

Questions? Oh... One more thing Progressive image download