[ If you’re already convinced that you want to smoke-test your Flutter samples, you can skip to “Smoke-testing Flutter” below. What follows is a general guideline applicable to any technology. ]

When learning any new technology, sooner or later developers want to look at sample code. Not just a few lines, either. Often, they want to see a whole project and how it’s assembled.

That, by itself, means that any library, framework or SDK needs to maintain a number of sample apps. But there’s more.

Samples are most useful when they’re focused. A classic mistake is to ram everything-there-is-to-know into a single sample app. That’s easier for the author (only one app to create and maintain) but unhelpful for the newcomer. Nobody learns everything at once.

With any sufficiently-complex technology, the developer first wants to see some code sample of the basics (“hello world”), then maybe a small app (TodoMVC, HN app), and then focused samples according to their needs (networking, parallelism, animation, etc.). The idea is that they never need to try and comprehend multiple hard concepts at once.

(Yes, very complex, real-world samples are valuable. But, generally speaking, not for beginners or intermediate developers. They are great for when you already know much of the technology, so you can navigate your way around the complex sample, and identify the parts that are relevant to you at the time.)

For the author of the samples, that means a lot of self-contained little apps. Which in turn means many code repositories. And any of these can (and will) get out of date, sooner or later.

You don’t want sample code to go out of date. For a developer, there are few things more frustrating than getting excited about a new technology, trying to run a sample app, and seeing it fail.

So, to summarize:

A good sample coverage calls for a lot of independent sample apps. For the sake of argument, let’s say we have 100 independent sample apps.

independent sample apps. Every single sample app needs to be kept maintained. If there is any major breakage coming down the line, we should know about this as soon as possible.

What to do? Here are some options:

Do nothing

Errors in samples get caught by the community sooner or later, but by the time you get to know about them, they will have done damage. This does not only lead to frustration of the people who find the breakage, but it will most likely be widely reported sooner or later, leading to damaged reputation of your project.

You will be shocked (shocked!) to learn that this is by far the most widely used option for sample repos.

Test manually

Some poor soul is tasked to manually check the 100 samples every time there’s a new version of the underlying technology.

Many teams do this, and you can probably imagine how that goes, or at least how that keeps the pressure on having significantly less than 100 samples.

Perform static analysis in Continuous Integration (CI)

For strongly typed languages, this is already way better than the previous option. But static analysis doesn’t catch all breakages. An app with no static errors can still fail to build, or crash at runtime.

Full unit testing in CI

Unit testing is indispensable, but not a good tool for this particular job. Most code samples have trivial logic (e.g. “this function always returns 42”) — not really worth testing. Also, few technologies change in a way that breaks your individual unit tests — so it’s not really what you want to test. You want to test that the sample app, when put together, works.

But of course, if you can spare the resources to unit test everything, awesome.

Full functional / integration testing in CI

This is the best option if you have the time and resources. Which—let’s face it—you don’t. Not for 100 different sample repos.

If you can manage, then this is obviously fantastic. You verify that the app not only statically checks and compiles, but also that it runs in a real-world-ish environment. (While we’re in the mode of dreaming: let’s test the sample apps under all supported operating systems and both on real devices and in emulators.)

Put samples directly in the tech’s repository

When you have your samples developed (and tested) in the same repo as the technology they’re “sampling” (like Flutter does with the Flutter Gallery), there is no chance they’ll go out of date. When the underlying technology changes in a way that breaks the sample, the build goes red and the development team is highly motivated to get the example code fixed.

The problem is that you can’t do this with too many samples. That would slow development. Also, more often than not, you depend with your samples on technologies that you don’t own, so you can’t just put the sample in the same repo.