Lessons from the Novena laptop project

LWN.net needs you! Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing

Flock is the annual conference for the Fedora distribution, but, like most free-software events, the program draws on a wide range of projects. At this year's event in Prague, keynote speaker Sean Cross spoke about his work on the Novena laptop project—including some speculation as to why it is succeeding in the demonstrably harsh space of open hardware products. Cross told the audience he hoped to get Fedora running on Novena (which runs Debian only at the moment) over the course of the conference, but he also hoped that the Novena story would be a helpful and informative tale for others undertaking a difficult, large-scale task—such as building a distribution.

Novena, for those unfamiliar, is a do-it-yourself laptop designed with hackers in mind. The specifications are unusual; among other things, the CPU is a quad-core ARM Cortex A9 from Freescale, there is a breakout connector for Arduino- or Raspberry Pi–like aftermarket hardware add-ons, and there is a built-in Xilinx Field-Programmable Gate Array (FPGA) on board. The project raised more than $720,000 in a crowdfunding campaign (against an initial target of $250,000), and is run by just two employees: Cross and Andrew "Bunnie" Huang.

Cross met Huang when both were employees at Chumby, Cross explained. Huang was the hardware lead; Cross worked on kernel development. Both resided in Singapore, which, he explained, was almost a necessity when working on consumer electronics projects, due to its close proximity to China. Chumby was a fun company, he said, and he learned a lot from Huang. When Chumby closed its doors, Huang and Cross collaborated on a series of smaller open-hardware projects under the Kosagi company name. Those projects included the NeTV video overlay device and the Kovan robotics controller.

Both projects involved working with FPGAs during the development process, he explained, and they decided that their next project should be a platform for doing FPGA development. The Novena board itself started off as a design for a Tor router with a built-in FPGA. When they presented it at the 2013 Chaos Communication Congress (CCC), however, people kept telling them "that's a really cool laptop." Eventually, the outside interest persuaded them to attempt the crowdsourcing campaign. That campaign exceeded its target by a considerable margin in May, enabling the team to undertake several "stretch" goals.

The Novena is designed to be as free-software friendly as possible, Cross said. All of the schematics and hardware designs are publicly available, and the software is free of binary blobs. "Well," he added, "there is a small asterisk to that, depending on how you define 'blob.'" The device's WiFi module has free software drivers, he explained, but there are always other places where "blobs" can remain. The Novena board has a smart DMA controller and boot ROM, both of which do not have free firmware available. The Vivante video chip also needs non-free drivers to do 2D and 3D acceleration, at present, although progress is being made on reverse-engineering both cores ("it already plays Quake; the community is close to getting Wayland running," he said). Finally, there is hardware video core that requires a blob to accelerate certain codecs, though these are optional.

Whether these bits disqualify Novena as a "free hardware" device is a matter of contention, apparently; on his blog, Huang commented that the FSF would not certify the Novena in its "Respects Your Freedom" program, even though the laptop can be used with the binary-only units turned off and unloaded. Cross also asked, rhetorically, "what is a blob?" The Xilinx FPGA can only be programmed with Xilinx's IDE, he observed. You program it yourself, but it generates a binary—is that binary considered a "blob" or not, he asked.

Hardware design and development

Although the team was experienced with open-hardware products, Cross said, the laptop form factor quickly proved to be really complicated. For example, he compared the power circuit to that of the NeTV and Kovan board. Novena's power requirements necessitate handling far wider input voltages, employing separate switches to control power in different subsystems, battery control, battery charging logic, as well as over- and under-charge protection. They eventually had to design a custom power board for the device, running its own ARM CPU.

They also had to design custom boards for the laptop front panel, the Hall effect sensor (which detects if the lid is closed), and the cable connecting the mainboard to the embedded DisplayPort screen. He noted (in reference to the latter item) that most people don't think of cables as circuits, but they can be quite complex. Furthermore, the project has had to design every plastic and metal part used to assemble the device, down to the screen bezel and the switches. This can get expensive; the first quote they received for the metal-cutting bill alone was $250,000. By doing the CAD work in-house, though, they managed to find another supplier who would deliver for $100,000. The case is designed to be opened up and tinkered with, he explained, so it does not look like most traditional laptops—a factor that made it "interesting" to take through airport security.

The Novena includes several features not included on commercial laptops, Cross explained. In addition to the FPGA, the laptop includes USB On The Go (OTG), which allows the user to boot another computer off of the Novena's storage, dual Ethernet, and a header connector for electronics tinkering. This header was initially a direct copy of the Raspberry Pi header, he explained, so that users could run Raspberry Pi expansion boards ("but much, much faster"), but it was later reduced in size to save space—and bundled with a breakout cable. This was in the spirit of forking open source projects, Cross said, and it proved beneficial. A company called Myriad noticed that Novena used the same breakout header as its own software-defined radio (SDR) board, and offered to re-spin a version of the board specifically for use with the Novena.

The team has also spent a lot of time sourcing individual components. "If you like electronics," he said, "go to Shenzhen." The city is home to countless electronic supply markets that are "ehh ... I'll say 'gray' markets," he joked. Searching for surplus part lots that are in large quantity compared to what Novena needs, but are leftovers to major manufacturers like Foxconn, saved the project considerable money. Experience with the earlier devices and proximity to China helped a lot.

Software

On the software side, Cross said that the team decided to build Novena on ARM because they already had experience with it, and because an ARM-based machine could be used to compile software for virtually every mobile device on the market today—without requiring a cross-compiler. They chose the Freescale chip they use because it is the best documented: there are 5,827 pages of official documentation available on the web, all without requiring any non-disclosure agreement.

That said, he also reported that working with vendors' official board-support packages (BSPs) was not as smooth as it could be. BSPs tend to come with a specific bundle of "supported" modules (the kernel, Xorg, GStreamer, SurfaceFlinger, etc.). But they are rarely up-to-date and often only work with the other components in the BSP. "They're like off-brand LEGOs: they work okay within the kit, but you can't use them with your other toys." Consequently, the project stopped using vendor BSPs and has been working with upstream, mainline code. It takes longer, but support is much better and "is really long-term."

The development process has meant writing new drivers for several components (such as the onboard temperature sensor), he said, but he has been surprised at how quickly mainline kernel support has caught up. It helps that there are other projects working with the same Freescale parts, including Cubox, the Hummingboard, and the Riotboard.

Sometimes, support work flows in the other direction, too, he said. He pointed out that Freescale subcontracts subsystems like USB and video to other suppliers. In the case of PCIe, the Novena team had to write its own drivers, but discovered subsequently that several Samsung devices use the same PCIe components, so the patch benefits other device makers as well.

At this stage, the Vivante GPU requires a lot of work. As mentioned earlier, only non-free drivers are available for the 2D, 3D, and video cores. Luckily, the Etnaviv project is making progress on a free driver, which he hopes will be usable in the 3.17 or 3.18 time frame. The 3D core support is good, he said, but he wants to be able to use the 2D for as many operations as possible, since it consumes 1% of the 3D core's power.

Debian was chosen as the Linux distribution for the Novena because Debian has strong ties to Tor and the Novena mainboard was initially conceived of as a dedicated Tor platform. But, he added, "I have a spare SSD and a screwdriver with me, so if you want to help me get Fedora running, come talk." In an email, Cross said that he indeed did get Fedora running on the Novena hardware by the end of the event. There were a few stumbling blocks, he said, such as SELinux, but Fedora is now "working as well as Debian." GNOME Shell is not working smoothly yet, due to the 3D driver problems. But he estimates that in six months or so, even that could be solved.

Cross ended his talk with a few bits of stray advice he had gleaned from the work on Novena. First, he said, dogfooding is critical. He and Huang use their prototype Novena hardware as their daily machines; it is the only way to make sure that the solutions you work on really get exercised fully. Second, he said, work upstream. Vendor BSPs just aren't robust enough—the Novena hardware has not run Freescale's kernel for a long time.

Third, he advised the developers in the audience to stop assuming that their code was running on the x86 architecture. There are more and more ARM devices on the market, but there are also MIPS, IBM System z, and OpenPOWER boards coming out. They are all Linux, he said, but none are x86: assumptions about byte sizes and endianness need to go. Next, avoid optimization—or at least make sure you have a fallback. You may see some Intel vector instruction set that sounds great, but making your application depend on it will cause trouble for others. Sticking to a higher-level language instead means that anyone can get started—including those on homemade ARM laptops, presumably: the Kosagi wiki lists a few noteworthy examples.

Finally, Cross advised the audience to have fun. There are loads of options at every level of the free software stack, he said, so find a project that you are enthusiastic about.

The Novena hardware is already in production, according to Cross, although the final assembled units are not expected to ship for a while.

Pessimists might say that this means the Novena is not yet an "available" open-hardware device, but it is clearly much closer to completion than many of the other, canceled projects that the community has seen in the past few years. Having experience with prior devices and with the process of working with suppliers in China is clearly a big part of the equation, but there is probably something to be said for outside-the-box approach that Cross and Huang take. The Novena is almost nothing like a commercial laptop, and that is part of why it has so many fans. Which sounds like a good lesson for free software projects as well.

[The author would like to thank the Fedora project for travel assistance to attend Flock 2014.]

