So, what did we learn about developer experience from this study?

1. Provide a lot of examples and present them effectively

After just a few user testing sessions, it became clear that developers expected to learn how to work with a new SDK from examples. The problem, though, wasn’t that Flutter didn’t provide enough examples — it had tons of examples in its Github repository. The problem was that those examples were not organized and presented in a way that were actually helpful to our study participants for two reasons:

First, the code samples in Flutter’s Github repository lacked screenshots. At the time, Flutter’s website provided a link to search all code examples containing a particular widget class in its Github repo, but it was hard for participants to determine which example would produce the desired result. You had to run the example code on a device or a simulator to see the widget’s appearance, which no one bothered to do.

“This is nice, linking to actual code. But it’s very difficult to choose which one you’d like to use unless you see the output.” (P4)

Second, participants expected to have sample code in the API documentation, not in a separate place. Trial and error is a common way to learn an API, and short snippets in API docs enable that method of learning.

“I click on ‘Documentation’, but it’s APIs, not samples.” (P4)

Several engineers on the Flutter team observed study sessions over livestream, and they were struck by the challenges some participants experienced. As a result, the team has been steadily adding more sample code to Flutter’s API docs (e.g., ListView and Card).

In addition, the team started building a curated, visual catalog for larger code samples. There are only a handful of samples for now, but each sample features a screenshot and self-contained code, so developers can quickly determine if a sample is useful to their problem.

2. Accommodate the cognitive abilities of developers

Programming is a cognitively intense activity. In this case, we found that writing a UI layout purely in code was difficult for some developers. In a Flutter app, building a layout involves selecting and nesting widgets in a tree. For example, to create the layout in the cafe information card, you need to organize several Row widgets and several Column widgets correctly. It didn’t look like a hard task, but three participants mixed up Row and Column when they tried to create that layout.

“Can you tell me what you expected the output to be?” (Moderator) [Talking through what he wanted to do] “Oh…I should have probably used a Column not a Row.” (P6)

We turned to Cognitive Psychology for explanations. It turns out that building a layout in code requires the ability to reason about the spatial relationship between objects, and it’s known to cognitive psychologists as spatial visualization ability. It is the same ability that influences how well a person can explain driving directions or rotate a magic cube.

This finding has changed some team members’ view of the need for a visual UI builder. The team was very excited to see community-driven explorations on this front, such as this Web-based UI builder called Flutter Studio.

3. Promote recognition over recall

It is a well known UX principle that user interfaces should avoid forcing users to recall information (e.g., an arcane command or parameter). Instead, the UI should allow users to recognize possible courses of action.

How is this principle relevant to software development? An issue we observed was that it was not intuitive to understand the default layout behavior of Flutter widgets and figure out how to change their behavior. For example, P3 didn’t know why the Card, by default, shrunk to the size of the Text it contained. P3 had trouble figuring out how to make the Card fill the width of the screen.

body: new Card(

child: new Text(

‘1625 Charleston Road, Mountain View, CA 94043’

)

),

“What I wanted is to have it take up the entire width of the screen.” (P3)

Of course, many programmers can figure that out eventually, but they need to recall how to do it the next time they face the same problem. There were no visible clues for developers to recognize a solution in that situation.

The team is exploring a few directions to reduce the burden of recall in building layouts:

Summarizing the layout behaviors of widgets to make them easier to reason about.

Offering layout samples with both code and images to turn some recall tasks into recognition tasks

Providing a Chrome-style inspector to show “computed value” of a widget property

4. Expect developers to be blind to something “right in front of them”

One feature the Flutter team is really proud of is Hot Reload. It allows the developer to apply changes to a running app within 1 second, without losing the app state. Performing a Hot Reload is as simple as clicking a button in the IntelliJ IDE or pressing ‘r’ in the console.

However, in the first few sessions of the study, the team was puzzled by some participants’ expectation of triggering Hot Reload on file save, despite the fact that the Hot Reload button was shown in an animated gif in the Getting Started instructions. How could they not see the Hot Reload button?

It turned out that participants who missed the Hot Reload button and expected to trigger reload on save were users of React Native. They told us that in React Native, Hot Reload was automatically performed on file save.

A developer’s pre-existing mental model can alter their perception and lead to a certain degree of “blindness” to UI elements. The team has added more visual cues to help discover the Hot Reload button. Furthermore, some engineers have been investigating a reliable way to allow reload on save for users who need it.

5. Don’t assume programmers read English words as expected when they appear in code

In Flutter, everything is a widget. A user interface is primarily composed through nesting widgets. Some widgets take only one child, while others take multiple children. This distinction is indicated by the existence of either a “child” property or a “children” property on a widget class. It sounds quite straightforward, right?

We thought so, too. Yet, to some participants, the singular form of the word didn’t successfully signal that only one widget could be nested in the current widget. They doubted that “child” really meant “just one.”

“I’m thinking the ‘child’ can be multiple ones or not. Can I pass in an array or just one is possible?” (P2) “So the ‘child’ will be 4 things, the first item, a separator, and two more items.” (P2)

This mis-interpretation of the semantics of the property name led to erroneous code like this:

And the error message shown in this case, though accurate, wasn’t actionable enough to nudge the participant back on the right path:

It’s easy to dismiss what happened here as a beginner’s mistake. However, the team was not comfortable seeing a professional developer waste several minutes dealing with this simple issue. So a short-term fix was landed a few days after the study findings were reported. It added one of the most useful multi-child widgets, Column, to the app template you get by running the ‘flutter create’ command. The goal is to expose the difference between ‘child’ and ‘children’ to a new developer early enough, so they won’t waste time figuring that out later. In addition, some team members are investigating a longer-term solution to improving the actionability of error messages in situations like this and beyond.

Conclusion

We can learn a lot from observing developers use APIs and apply learnings to improving the User Experience of a developer product. If you write any code or build any tool that is used by other developers, we encourage you to observe how they use it. As a Flutter engineer put it, you always learn something new from observing a user study. As software continues to drive changes in the world, let’s make sure developers are as productive and happy as possible.