Newsletter #6 - Ship it!

December 2015

Welcome to the sixth newsletter for Neovim, a project that aims to improve Vim by adding new features and wrap it all in a nice, modern face.

Introduction

Hi, this is @tarruda and I will be addressing the Neovim community directly in this newsletter. Other than that, I will try to keep it structured as @jdavis did previously. Let’s get started!

General News

0.1 release

Neovim now has its first public release!

A few months ago, @justinmk created the 0.1 milestone which greatly helped us focus on more urgent tasks that resulted in the first release. We planned many features not yet available in 0.1, but decided to postpone them for future milestones, which will be more frequent after this newsletter.

This illustrates the path Neovim will take from now on: Instead of preparing big releases that take forever to happen, we’ll focus on smaller, frequent and more stable releases.

The 0.1 release is basically just a tag for users looking to compile Neovim in a version that has a minimum level of stability, but future releases may also contain precompiled binaries and even installers (when Windows is officially supported).

Neovim 0.1.0 and 0.1.1 are already available in the releases page and for Homebrew/Linuxbrew and Arch Linux. Check the installation page on the Neovim Wiki for more possibilities to install Neovim (although, at the time of writing, most of these will install the latest development version of Neovim instead of a 0.1.x release).

For those that prefer(or need) to compile manually from git, the build instructions still work as usual.

Bountysource salt campaign

For those who don’t know yet, Bountysource launched a new platform that allows open source projects to obtain sustainable crowdfunding. This platform is conveniently called “salt”, and Neovim was one of the first projects to use it.

The first campaign was very successful and raised about $35,000, which allowed me to work full-time on Neovim for roughly 6 months. Being very enthusiastic about the project and unable to meet all goals in those months, I continued to dedicate a very significant portion of my time to Neovim, so much that it started hurting my personal and professional life. This continued until February when I saw that I simply couldn’t continue with my old pace. Around that time that @rappo offered me to test the salt platform beta version and I saw it as a way to continue my work on Neovim.

Like it’s predecessor, the salt campaign was very successful and allowed me to continue Neovim contributions (in a healthy way) for the past 8 months, thank you!

Building Neovim from source

Did we ever mention how easy it is to build and install Neovim from source? While it has a good number of dependencies, the build system automatically downloads and builds everything without cluttering your system. Check out the installation page on the Neovim Wiki for the exact steps.

Development News

XDG Support

Neovim now follows the XDG directory specification. This was proposed by @ZyX-I when the project started, but only a couple of months ago we received a PR from @Yamakaky which was superseded by @jck in a later PR and again by @ZyX-I in a final PR that was merged recently.

Since following the XDG directory specification, Neovim now looks for user configuration files such as .nvimrc and those under ~/.nvim in the ~/.config directory, which can be overriden by the $XDG_CONFIG_HOME environment variable. The specification also states that cache files should be stored in a separate directory ( ~/.local/share ), which is where files like viminfo (now ShaDa) or backup/swap can optionally go.

This change makes it simpler for users to backup and manage their configuration since it will be stored with other programs that also follow the specification, not to mention it keeps the home directory cleaner.

Step by step instructions on how to migrate existing configuration can be found at :h nvim-from-vim.

ShaDa (Shared Data)

@ZyX-I major ShaDa PR was merged. It completely replaced the viminfo file for storing user data such as register contents, command history, variables, jump list and so on.

A known problem with viminfo is that two Vim instances running concurrently will override each other’s data. ShaDa is a new storage format created by @ZyX-I that not only fixes the problem but also brings a number of enhancements to Neovim:

File format that: Supports forward (ShaDa files from newer Neovim can be used by older versions) and backward (ShaDa files from older versions can be used by newer ones without problems) compatibility, making ShaDa files future-proof to a great extent. Supports hierarchical data structures, giving Neovim lot of flexibility in serializing any kind of information. Is based on msgpack and explicitly standardized in documentation which allows creation of plugins/tools that perform arbitrary manipulations. Assumes no state stored between data pieces inside the file, which makes such tools simpler and allows such manipulations as “to concatenate two ShaDa files from different Neovim instances simply use cat 1.shada 2.shada > joined.shada , Neovim will handle this properly when reading”. Uses expanded paths(/home instead of ~/).

Embedded timestamps that allow multiple Neovim instances to correctly merge their data when writing or reading ShaDa files.

Forward compatibility includes Neovim core code that preserves additional information found in ShaDa files generated by newer Neovim versions in case they provide any.

While the PR is very big, @ZyX-I has taken care of adding great test coverage. Great work @ZyX-I!

Wrapping the event loop layer

It’s no secret that libuv is the event loop library used by Neovim, and it is what makes it possible for us to implement features that require asynchronous communication (not initiated by the user) with the editor with ease. Unfortunately due to how Neovim code is currently organized, integrating libuv was not a trivial task.

The basic idea is that Neovim receives arbitrary events when it is polling for user input, but these events can’t be processed immediately because Neovim can be in a state that simply can’t handle arbitrary actions. So if Neovim receives an event while checking user input, it will put the event in a queue for later processing.

One example is illustrated in the following scenario: Neovim checks if the user typed ctrl+c while the regexp engine is executing, but it can’t process the event as it is received because it may want to execute vimscript that calls the regexp engine again, and the engine is not reentrant since it relies heavily on global variables. So it has to postpone the event for when it’s safe, and determining when it’s safe to process events is itself another problem.

Another complication of integrating with libuv is that sometimes Neovim must only process events from a certain source. For example, while Neovim is sending a msgpack-rpc call, it should only process events that come from:

the user (eg: ctrl+c to interrupt the call)

the file descriptor that received the msgpack-rpc call (which can be from a child process stdio from a socket)

To allow this kind of selective event processing, Neovim must maintain multiple queues that integrate with each other, and the logic to do this is very repetitive. In one of my latest PRs, some libuv “classes” were wrapped in a way that makes managing these queues much easier.

jemalloc

jemalloc, a high performance general purpose memory allocator, is now used by default. Since Neovim makes heavy use of dynamic queues (see above) in its inner loops, malloc(3) is called a lot more than Vim, so it is important to use a fast implementation that has consistent performance across platforms.

In a recent PR, @fmoralesc modified the jemalloc version used by our build system to target jemalloc 4.0 which brings even more performance enhancements and adds support for more platforms.

Faster travis builds

We now use Travis container-based insfrastructure to run Neovim builds, which makes CI builds to start immediately. This was implemented by @fwalch, which also did many other improvements to our build infrastructure, allowing developers to receive much faster feedback when submitting PRs.

Quickbuild

@jszakmeister is running a quickbuild server in his own infrastructure. This gives us a backup CI that double checks Neovim PRs, also running tests in FreeBSD which is not covered by travis. Thank you for improving Neovim robustness @jszakmeister!

Third-party development

Neomake

Did you know that there is an alternative to syntastic that makes use of Neovim asynchronous capabilities? Neomake is the best plugin for syntatic checking on Neovim: It is extensible like syntastic and the fact that it uses job-control allows it to perform checking in background without blocking the user interface. This is very useful for compiled languages that are slower to check (typescript, java, .NET).

The migration from syntastic is also very trivial, great work @benekastah!

FZF

fzf is a command-line fuzzy finder that thanks to its author (@junegunn, the same developer behind vim-plug), has great Neovim support through a plugin that uses our builtin terminal emulator.

FZF is a great alternative to plugins like ctrlp: It is really fast and has the advantage of running in another process, which can make use of multi-core systems and doesn’t block Neovim user interface. To see how fast and responsive it is, just try running :FZF to search for files on the linux source tree!

Besides fzf, the user is also encouraged to install fzf.vim, a plugin that exposes some very useful commands that are implemented on top of fzf.

Deoplete

@Shougo has created deoplete.nvim, an asynchronous completion engine written as a remote-plugin that makes use of Neovim async capabilities to allow completions to be computed without blocking the user interface. He decided to write a new plugin from the scratch because Neovim doesn’t support the lua interface required for neocomplete. @Shougo is the sith lord of Vim plugins, here’s a list containing some of his previous work:

The community can expect great things from deoplete.nvim!

(Recently @Shougo posted a slide to explain the ideas behind deoplete.)

Neoterm

Neoterm is a plugin for easily running tests in a terminal window. It was written by @kassio and supports the following test libraries:

rspec

cucumber

minitest

go-lang test

nose

Very useful @kassio!