We welcome your thoughts, questions, or critique: @inkandswitch or [email protected]

This manuscript documents the project’s design decisions and what we learned from user testing. We encourage you to borrow these ideas — or use them as cautionary tales — in your own tablet app or desktop app development. The Capstone codebase is open-source and can be installed as a Chrome App on any Chrome OS device that includes a stylus.

The prototype is built around a two-step creative process where the user first collects raw material on their desktop computer and later does their thinking on a tablet with stylus. It offers mixed media cards plus freeform inking, spatial navigation, and a shelf which serves as a portal between devices and a reimagined copy-paste. Findings include success with hybrid finger+stylus gestures, the chalk-talk use case, and the unsolved problem of stylus tool-switching.

Capstone is an experimental tool for creative professionals to develop their ideas. It explores questions about digital information curation; how creative people come up with good ideas; and what we at Ink & Switch think the future of power-user computing interfaces could look like.

Motivation and overview

Creative professionals and the two-step process for developing ideas

Leading up to this project, our team interviewed dozens of creative professionals such as professors, authors, film-makers, and web designers. Although the details of how these pros work are different, a pattern emerged in this research: they all live or die by the strength of their ideas.

But ideas aren’t summoned from nowhere: they come from raw material, other ideas or observations about the world. Hence a two-step creative process: collect raw material, then think about it. From this process comes pattern recognition and eventually the insights that form the basis of novel ideas.

Leonardo da Vinci is the quintessential collect-then-think creative professional. His notebooks include observations from the natural world, excerpts from books he has read; then he develops his own ideas in written or sketched form. See Leonardo da Vinci by Walter Issacson. Image from the British Library's digitized manuscripts.

The collection step is gathering raw material in the form of sketches, photos, websites, text excerpts, and manuscripts in a single location. This can happen all at once in a binge-research session or it can be a slow curation over months or years.

Interior architecture firm Novono created this mood board for a client. Mood boards are a classic technique in the collect-then-think process of visual creative professionals including graphic designers and fashion designers.

In the thinking step, the creative pro sifts and sorts their collected raw material. They look at it from different angles, rearrange, try to get new perspectives. They are looking for patterns that can lead to fresh insights.

YouTube video essayist and Mark Brown collected hundreds of dungeon map screenshots in his notes, which developed into the game-design analysis series Boss Keys.

Next, we investigate the tools used by creative professionals in this collect-then-think process.

Paper notebook

The notebook and pen or pencil is a classic and beloved tool for creative people. Notebooks are sketchy and freeform. Turning over a new page in a notebook to capture a fresh thought is fast and easy. And notebooks are portable but also offer a comfortable posture for thinking.

Handwritten lecture notes from a data science course.

Notebooks can be a blank canvas for recording observations and ideas, but they can also be a collection space: for lecture or meeting notes, excerpts from other works such as in a commonplace book, or print media clippings as in a scrap book. There is some evidence that handwritten notes are good for memory. See The Pen Is Mightier Than the Keyboard

Post-its and war rooms

Marcin Wichary uses post-its on a wall for the thinking step of developing his book on the history of keyboards.

Post-it notes on a wall, index cards on a desk or pinboard, and war rooms offer some of the same value as notebooks (sketchy and freeform) while having the additional value of tapping into human spatial memory.

Pinboards used in an art room at Pixar during the creation of the Incredibles 2.

From migratory birds to open-sea navigation by pre-civilization humans, life on earth has a strong innate ability to remember and navigate space. Our brains have a dedicated region in the hippocampus for spatial navigation. It seems to be possible to activate this region in a digital/virtual environment with the right orientation cues.

Pinboards, chalkboards, and war rooms tap into our innate capabilities for spatial reasoning — and have the additional benefit of being a way for small groups to collaborate on collect-then-think.

Note-taking apps

Almost all creative professionals use a smartphone-based note-taking app for quick capture of thoughts on the go. Some examples include Evernote, iOS Notes, Google Keep, Drafts, Bear, iA Writer.

Meeting notes in the Evernote smartphone app.

These tools are great for mobility and sketches in pure text (such as an outline or first draft of an email, paper, or article). But their use is narrow as they are poor at images, web, and other media types.

Virtual pinboards

Virtual pinboards like Milanote, RealtimeBoard, and Jamboard offer a promising “cards on a canvas” model that taps the user’s spatial reasoning and encourage a messy, playful approach.

A team summit planning document in the Milanote collaborative pinboard.

Other pinboard-like digital tools include Pinterest, which is popular for mood boarding; and Prezi, which encourages hierarchical arrangement of information.

Although most of these remain niche products that have not found mainstream usage with creative professionals, we think they show promise as a way to bring the best parts of the analog post-its/war-room approach into the digital realm. Ink & Switch has previous experimented with virtual pinboards via the Pushpin desktop app and the Dossier iPad app.

Filesystems

A venerable approach to colllecting raw material is simply to save it to a hard drive and browse with the operating system-provided file browser.

A collection of research on Leonardo da Vinci stored on a hard drive.

Related and adjascent tools include Dropbox and Network Attached Storage devices. File formats like .txt and Markdown are private, trustworthy, and reliable over time.

But files also tend to be clumsy and old-school, poorly adapted for a multimedia-, web-, and mobile-oriented world.

Digital sketchbooks

Digital sketchbooks and annotation tools with first-class stylus support include Goodnotes, PDFExpert, and the iPad’s built-in markup tools.

Excerpt from the author’s personal annotations in PDFExpert.

Excerpt from Martin Kleppmann’s notes created with Notability on the subject of cryptographic signatures for CRDT change histories.

These have found some traction with the pros we’ve interviewed. The form factor suggests the analog equivalent of notebook and pen, and freeform scribbling is conducive to the creative process. But these apps usually optimize for drawing on a blank canvas or annotating a single image or PDF.

Project vision

We started with the observation that creative professionals develop their ideas in a two-step process. We saw that they get great value from tools that support that process. In surveying those tools, we we were inspired by capabilities like freeform sketching and spatial navigation, but also saw many gaps such as lack of mixed media.

Putting all of these ideas together, we set out to build Capstone: a digital tool for the collect-then-think process of developing ideas.

Capstone’s vision comes in three three main pillars:

A digital-information collection tool. Pros need a place to collect many types of digital media: the raw material for their thinking process. Our focus for this project is collection done on a desktop computer’s web browser. A place to sift, sort, find patterns and insights. Once the raw material is collected, the user should be able to sift and sort through it in a freeform way. All media needs to be represented in the same spatially-organized environment. Use a tablet and stylus to its full potential. Pen-and-paper solutions, especially the A4 notebook, seem to be more conducive to thinking and having better ideas. Our hypothesis is that the thinking step should happen on a tablet with freeform inking via the stylus.

Demo video

Here is a full walkthrough of the resulting application, showing a personal use-case from one of our team members making a life decision about a move to a new city.

In the example, the user wants to record the details of his experience into an archive. That archive will be (1) a place for him to sift, think, and find insights that might guide his decision about whether Austin is a good place for him to live and work longer term, and (2) a visual aid for him to use when explaining to colleagues, friends, or family about his time in the city. Other examples of topics our test users tried with Capstone: planning a trip; researching equipment for a new hobby; dog training; learning about a new technology stack; organizing a team sprint; and preparing writing and speaking topics for upcoming professional conferences.

While the use case as shown in the video is simple, we hope you can extrapolate to what a more mature version of this tool might enable.

What we built

With the project vision established, this next section describes what we built and why.

Briefly, the components are:

Capstone Clipper browser extension to collect text, images, and websites on a desktop computer

The shelf as a portal to transmit collected material to the Capstone tablet

Tablet thinking space with mixed media cards and inking

Spatial navigation with zoomable boards

No titles on items

The shelf as a re-imagined copy-paste buffer

Card duplication with mirror

Command paradigm of hands are safe, stylus edits

Use the edges of the tablet to create and delete cards

The unsolved problem of stylus tool selection

Let’s go deeper on each.

The collection step

Capstone Clipper is a browser extension for the collection step.

Clip images, text, websites

The Capstone clipper can capture media from a desktop computer by dragging in files, pasting from the copy-paste buffer, or capturing directly from the browser.

Capstone users can drag images or paste text on their desktop computer into the shelf.

Right-click context menu offers “Send to Capstone” option for images and text.

Clipped items are saved into the user’s shelf.

Shelf as a portal

The shelf is a freeform space that appears on the user’s desktop and on their tablet. The user can mentally model this as a portal that is always the same between the two devices.

On the desktop, the shelf is a regular window. On the tablet, the shelf is an area in the lower part of the screen which can be summoned by dragging up from the bottom of the screen, and hidden by dragging down.

The thinking step

Now we get to the heart of Capstone: the freeform sift/sort/arrange interface running on the tablet.

Media cards + inking

Capstone’s interface brings together freeform arrangement of media cards (images, text, web clippings, PDFs) and inking (sketches, diagrams, handwritten text, other annotations) in the same space.

Media cards can be arranged freely, like index cards on a table or post-its on a wall. We sought to avoid the precision typically required by digital tools (for example, Illustrator or Keynote) which require the user to work with selections, rotate and resize handles, or front/back arrangement. Instead we wanted immediate interaction moving or resizing of cards in a single gesture.

Inking can happen anywhere. The user can draw and erase on a board and over the top of text, image, and web cards.

Spatial navigation with zoomable boards

In Capstone, the user organizes their clippings and sketches into nested boards which they can zoom in and out of. Movement is continuous and fluid using pinch in and out gestures.

Our hypothesis was that this would tap into spatial memory and provide a sort of digital memory palace. Nested folders on desktop computers are the venerable model for place-based navigation. Despite attempts at innovation such as tags and search-first interfaces, nested folders remain both popular and effective. See The Science of Managing Our Digital Stuff, chapters 4 and 5.

Pinch gestures for zooming in and out are familiar from many touchscreen apps, such as Google Maps. We like the association with real-world navigation, and see it as a good example of innovation from the mobile world that can be brought to bear on creative tools.

No titles

Computer applications typically list items by a title, such as a filename. This abstract representation doesn’t match how the document looks (although sometimes it is accompanied by a preview).

The Windows File Explorer emphasizes filename first, with thumbnail previews for only some file types.

The Files app on iPad gives more prominence to the item title (here, the name of each presentation) than the tiny thumbnail preview. Which is the user more likely to remember, the name of the item or how it looks?

The iOS camera roll offers a more natural way of representing documents: all of the item’s space is given to a small version of the photo. The filename (e.g. “IMG_4879.JPG”) does not appear anywhere.

Our idea for Capstone is that representations should look like the document. Some media types, like images and short texts, can display the whole document. Web clips, multi-page PDFs, and longer text excerpts cannot be shown in full, but we we can show a reasonable summary.

In Capstone, documents look exactly like themselves at all times, with no document titles other than what the user may choose to scribble on the boards. The board pictured above has no explicit title, and the image and text cards have no title or filename.

The shelf as reimagined copy-paste

In desktop operating systems, cut-and-paste is a crucial capability. Pro users have ctrl-C/ctrl-V hotkeys in their muscle memory.

But the traditional copy-paste buffer is due for a refresh. It can only hold one thing at a time, it can’t be edited, and it is invisible. On touch devices, copy-paste is typically hidden behind a set of clumsy submenus. Invisible copy-paste buffers can surprise users, such as accidentally pasting a password into a chat window.

Capstone's solution is called the shelf. Already described is how the shelf is a portal between the desktop computer and tablet, a sort of private inbox. Once within the sifting/sorting/organizing step, it becomes more: a scratch space and a cozy peripheral space that has the same spatial capabilities of boards but always travels with the user.

The user can move or duplicate cards from any board and put them on the shelf, navigate to another board, and then drop them there. Like traditional copy-and-paste, the shelf is a buffer that stays with the user as they navigate around the app; unlike traditional copy-and-paste, it is manifest on the screen, can hold multiple items at the same time, and can be edited.

Item duplication with mirror

Another important function in power tools is the ability to duplicate an item. Desktop productivity tools often have a “duplicate” or “clone” context menu item, and the user can also copy-paste.

Our approach in Capstone is the mirror command. The user can mirror a card by “pinning” down the card with one finger and dragging out with the stylus:

The finger+stylus hybrid action is an example of what can be done if a tablet app seeks to reach beyond the limitations imposed by the one-handed, small-screen assumptions typically used when designing for touch devices. Hybrid gestures between finger and pen are challenging on Chrome OS and other tablet platforms. OS gesture recognizer libraries are not built for this kind of work, with complicating factors like palm rejection.

Using the tablet and stylus

That covers Capstone's features supporting the collection and thinking steps of the creative process. Now we’ll address form factor.

Our user research suggests that the A4 notebook shape, especially when paired with a comfortable reading chair or writing desk, is an ideal posture for thinking and developing ideas. That suggests the digital form factor of tablet and stylus.

Existing tablet apps on mobile platforms don’t tend to be very powerful. We speculate this is because they are built as scaled-up smartphone apps, hence they don’t take full advantage of the expressiveness of two hands and a stylus.

Another design approach to tablet apps is to treat them as a port of desktop applications. Attempts to map mouse and keyboard command vocabularies to a touchscreen environment are often clumsy and confusing.

Capstone’s vision is to avoid both of these problems by treating the tablet as distinct from smartphones or desktop computers. Instead, the tablet is a third class of computing device with its own capabilities and place in the user’s life.

Here are three UI approaches we tried in our efforts to make the tablet a device with its own identity.

Hands are safe, stylus edits

The command model for most tablet and phone apps is onscreen chrome (buttons, toolbars, etc) that the user can tap to activate the app’s functions. The stylus is either undifferentiated (treated identically to a finger) or sometimes offers a limited activity such as a separate markup mode.

For Capstone, we tried a different command model: hands are for navigation, the stylus is for editing.

Users are conditioned by ten years of smartphone apps to be able to drag things onscreen with their finger. But in our tests we found that users quickly acclimated to using the stylus for moving and resizing.

We also found that this is an anatomically correct answer: fingers are coarse / non-precise, and the stylus is fine / precise. The stylus provides a higher degree of spatial freedom by expanding our touch zone with smaller arm moves which causes less fatigue, and higher accuracy with its thin tip.

Edges to create and delete cards

An important operation in any thinking tool is creating a new blank space for work. The user wants the equivalent of turning over a new page in a notebook, or pulling a fresh sheet of paper off of a stack.

A common approach to this is a plus button or menu option to create a new item. These are often paired with decisions the user needs to make — is it a text note or a drawing? Is it landscape or portrait orientation? All this ceremony slows down the user in getting straight to recording their ideas. The Drafts note-taking app deserves special praise for its ease of creating a new blank canvas: each time the user invokes the app, they get a new blank page by default.

Deleting or archiving of items is the symmetrical operation: it must be fast and frictionless to dispose of whatever items the user has decided to cull. Ceremony (such as “Are you sure?” confirmation dialogs) and delay (such as long-press to activate a context menu) are antithetical to a speed-of-thought tool.

For Capstone, our approach is to “pull in from edge” to create a new board and “throw off the edge of the world” to delete.

In our testing we found it natural and even fun to both create and delete items. In creating a board, the motion of the user’s hand can be fluidly continued to put the board where they want it. The two operations (create board, place board) the two flow together into one seamless motion.

It was also fun to “flick” the card away as it approaches the screen edge, including deleting several items in a row (flick, flick, flick).

Stylus tool selection

Tool- or mode-switching seems to be necessary for any sufficiently-powerful interface. To avoid the user needing to constantly reposition their hands, input capabilities will need to serve multiple functions. On the tablet, we have an even more restricted input space, so tool-switching is essential. Desktop programs have a variety of tool-switching approaches, from the tool palettes of Photoshop, to the “dead man’s switch” metakey approach of copy/paste and other shortcut keys, to the highly modal approach of the vim programmer’s editor.

On Capstone, it was one of our design goals to keep as much of the screen space free of administrative debris like toolbars and menus. Despite attempting a wide range of approaches, we failed to solve the tool-selection problem and had to fall back on the unsatisfying solution of an onscreen tool selection palette.

Here are some of the other tool-switching approaches we experimented with:

We don’t like the onscreen tool palette and consider this a significant unsolved design problem. Shortly after the Capstone project wrapped, Apple released their attempt at solving this problem: a stylus which includes a barrel double-tap.

Findings

That concludes the description of Capstone’s major design decisions. If you have a Chrome OS tablet, we encourage you to try it for yourself and tell us what you think.

To bring this manuscript to a close, we’ll share what we observed from testing on our team and with external users.

What worked

Mixed media + sketching

Freely-arrangeable multimedia cards and freeform sketching on the same canvas was a clear winner. Every user immediately understood it and found it enjoyable to use.

Advanced touch UI interactions

The command model of "hands are free, stylus edits" was mostly a success. And hybrid touch-finger interaction like mirror and pulling in cards from the edge of the screen created a fluid and natural environment. Thumb + Pen Interaction on Tablets provides a good look at simultaneous finger and stylus input, including using the non-dominant hand to activate mode-switching.

Zoom navigation and “peek”

Nested boards with pinch zoom in/out worked as expected. But an unexpected benefit that emerged in user tests is the idea of a “peek.”

A user can begin a pinch-out gesture to zoom up a board to almost their full tablet screen and read in detail what is on the board. If it’s not what they’re looking for, they can reverse their pinch direction without lifting fingers from the screen to cancel the operation.

Mobile-style motion design for productivity apps

In desktop productivity apps, results from user actions would snap into place onscreen with no motion or transitions. The addition of mobile-style motion design and transitions to Capstone seemed to make the app come alive.

Our observations suggest that these are more than just visual flare: they help create a fluid, environment where the user can move fast and with confidence, because the app visually reinforces every action they take.

Viability of no titles

We used representations of images, text, websites, and boards which look as much as possible like the original content. Boards in particular give you a feel for the colors and shapes they contain, and the ability to read any handwritten large titles. The “peek” capability reinforced this.

Our observation was that a “no titles” approach is indeed viable if there is enough summary information and it is fast and cheap to quickly look more closely. But given the small data sets used in our test cases, further research would be needed to find out if this will work at scale.

Fractal arrangement of ideas

The boards-within-boards navigation metaphor of Capstone encouraged users to organize their thoughts in a hierarchy. The Pyramid Principle is a managerial technique for presenting ideas in a telescoping arrangement.

For example, one of our test users developing content for a talk about their charitable organization’s mission created a series of nearly-empty boards with titles. Combined with board previews, this creates an informal table of contents for the ideas the user wanted to explore and could start to fill in.

Chalk-talk use case

The cards plus inking plus zoomable boards lent itself naturally to telling a multimedia story, ala chalk talks or Saul Kahn’s videos.

We speculate that a product-quality version of Capstone could be useful for:

distributed teams (screenshare your tablet while talking over video chat)

live or prerecorded educational content as found in MOOCs such as those on Coursera

Unlike slide decks created with software like PowerPoint or Keynote, these “cards and sketching” presentations felt more lively and human.

What didn’t work

Stylus mode-switching

As documented above, our solution to stylus modes (card editing vs inking) was an onscreen tool palette, which users found clumsy and error-prone.

Complex motion and transitions on web stack

The web stack we were building on made complex transitions challenging. Smooth transitions such as moving in and out so that it felt like one continuous space was a significant technical challenge. Our engineers concluded that the web stack (which is optimized for discrete page navigation) is not ideal for this. A better approach would be more like a video game: continuous rendering of an archive as a single “world.” See Appendix A for more.

Left- vs right-handed use

Most of our tablet UI paradigms depend on knowing which hand is dominant for the user. For example, onscreen mode-switching buttons should be located wherever the user’s non-dominant hand is likely to be located (including gripping the tablet). But that is a different place depending on the users’ handedness. “Southpaws” comprise about 10% of humans, but there is some evidence that there is a higher proportion among creative professionals.

A left- vs right-handed toggle would probably be the solution for a more complete prototype.

Problems with lost, malfunctioning, or out-of-battery stylus

Making the stylus a required part of a power-tool interface had many benefits, but it also exposed problems we faced throughout the project:

A stylus that inexplicably ran out of batteries after a week, with no indication as to why it was not being recognized by the tablet

Varying pressure sensitivity, with some users finding they had to press so hard as to distort the screen

Frequent lost styluses (“wait, let me find my stylus…” was a constant refrain)

We speculate that, to be viable, a tablet+stylus platform would need to address these. One solution is that if stylus hardware became cheap enough and worked universally, then workplaces and individuals could keep pen-holders full of styluses around, just as today we keep spare pens around. The Universal Stylus Initiative seeks to standardize stylus-on-touchscreen interfaces to make interchangeable stylus use possible.

Screenshot clipping

While the Capstone clipper allows users to capture entire web pages, images, and fragments of text, often users wanted an “excerpt” of a website or PDF with all formatting intact. The user behavior here was to take a screenshot using the “capture region” screenshot capability of their desktop OS.

Screenshots as a way to capture and archive content has many downsides: it’s not searchable, the text is not selectable, it loses quality when sized up, and most of the metadata (e.g. what website it came from) is lost. See Appendix C for expanded discussion of web page archival.

And yet, screenshots are easy, reliable, and universal on all computing platforms. Users are naturally drawn to take screenshots of text: what could an application or platform do to lean into this tendency, such as automatic OCR? We think this warrants more investigation.

Conclusion

At Ink & Switch we believe in tools that help humans think, create, and grow in our personal and professional lives.

We feel that computers have significant untapped potential in this domain. But as the technology industry puts ever-more effort into consumer products, we wish there was a similar emphasis on productivity and creativity applications.

Capstone is one experiment with innovation in digital thinking tools. It suggests some promising directions such as mixed media cards and inking on a freeform canvas, spatial navigation, and the tablet as a device with a purpose distinct from phones or desktop computers. That is: a quiet space for ruminating on raw material and finding insights to guide career or life decisions. Update August 2019: Many of the ideas from Capstone are making their way into Muse for the iOS platform.

If you’re a developer of iPad or Surface applications, desktop productivity software, or a human-computer interaction researcher we hope you find some inspiration from Capstone. We’d love to hear your thoughts: @inkanswitch or [email protected]