Neovim: vim's rebirth for the 21st century

Refactor vim to enable more advanced features and plugins.

neovim

What you should know about neovim, if nothing else:

It will provide first class support for embedding. It lets you extend the editor in any programming language. It supports more powerful GUIs. Vim plugins will work with it.

Stretch goals

Thank you all for your support! Now that we've hit the initial funding goal, I've come up with the following stretch goals:

$20,000: Reimplement vimscript as a language that compiles to lua. In other words, vimscript will be to lua what coffeescript is to javascript. This will have the following benefits: Vimscript will run much faster, especially if luajit is used. Luajit is probably the fastest scripting runtime out there: https://gist.github.com/spion/3049314 . Instead of reinventing the wheel, vimscript will be backed by one of the best scripting engines out there. Lua is fast, small and very well designed. By removing tons of legacy C code that implement parsing, garbage collection and other runtime features, we will automatically get rid of most vim bugs related to vimscript.

$30,000: Refactor the editor into a library. It will require changing the way vim reads input or emits output. More details here. This will allow programs to embed the editor in the same process for better efficiency(no more marshalling of json/msgpack documents between the GUI and the core).

$40,000: Remove all global variables(about 1k). This will be done by moving all globals into a struct that will contain all the editor state. Most functions will need to be refactored to accept the struct as first argument. Programs embedding the library will be capable of creating multiple editor instances in the same process.

$50,000: Refactor system calls into an abstraction module backed by libuv when compiled to machine code, and backed by a javascript library when compiled to into asm.js through emscripten. Since at this point vimscript will be implemented on top of lua, this project might make the transition easier. In a few words, this will port neovim to run natively in modern web browsers.

Please skip to the FAQ if you've already read through the GitHub repository.

Introduction

vim is a powerful text editor with a big community that is constantly growing. Even though the editor is about two decades old, people still extend and want to improve it, mostly using vimscript or one of the supported scripting languages.

Problem

Over its more than 20 years of life, vim has accumulated about 300k lines of scary C89 code that very few people understand or have the guts to mess with.

Another issue is that as the only person responsible for maintaing vim's big codebase, Bram Moolenaar has to be extra-careful when accepting patches because once merged, the new code will be his responsibility.

These problems make it very difficult to have new features and bug fixes merged into the core. vim just can't keep up with the development speed of its plugin ecosystem.

Solution

neovim is a project that seeks to aggressively refactor vim source code in order to achieve the following goals:

Simplify maintenance to improve the speed that bug fixes and features get merged.

Split the work between multiple developers.

Enable the implementation of new/modern user interfaces without any modifications to the core source.

Improve the extensibility power with a new plugin architecture based on coprocesses. Plugins will be written in any programming language without needing explicit support from the editor.

By achieving these goals, new developers will be more inclined to join the community, consequently improving the editor for all users.

It is important to realize that this is not a project to rewrite vim from scratch or transform it into an IDE (though the new features provided will enable IDE-like distributions of the editor). The changes implemented here should have little impact on vim's editing model or vimscript in general. Most vimscript plugins should continue to work normally.

The following are brief explanations of the major changes that will be performed in the first iteration:

1. Migrate to a cmake-based build

The source tree has dozens (if not hundreds) of files dedicated to building vim on various platforms with different configurations, and many of these files look abandoned or outdated. Most users don't care about selecting individual features and just compile using '--with-features=huge', which still generates an executable that is small enough even for lightweight systems by today's standards.

All those files will be removed and vim will be built using cmake, a modern build system that generates build scripts for the most relevant platforms.

2. Legacy support and compile-time features

vim has a significant amount of code dedicated to supporting legacy systems and compilers. All that code increases the maintenance burden and will be removed.

Most optional features will no longer be optional (see above), with the exception of some broken and useless features (netbeans integration, sun workshop, etc) which will be removed permanently. vim emulation will also be removed (setting 'nocompatible' will be a no-op).

These changes won't affect most users. Those that have a C89 compiler installed or use vim on legacy systems such as Amiga, BeOS, or MSDOS will have two options:

Upgrade their software

Continue using vim

3. Platform-specific code

Most of the platform-specific code will be removed and libuv will be used to handle system differences.

libuv is a modern multi-platform library with functions to perform common system tasks. It supports most unixes and windows, so the vast majority of vim's community will be covered.

4. New plugin architecture

All code supporting embedded scripting language interpreters will be replaced by a new plugin system that will support extensions written in any programming language.

Compatibility layers will be provided for vim plugins written in some of the currently supported scripting languages such as python or ruby. Most plugins should work on neovim with little modifications, if any.

This is how the new plugin system will work:

Plugins are long-running programs/jobs (coprocesses) that communicate with vim through stdin/stdout using msgpack-rpc or json-rpc.

vim will discover and run these programs at startup, keeping two-way communication channels with each plugin through its lifetime.

Plugins will be able to listen to events and send commands to vim asynchronously.

This system will be built on top of a job control mechanism similar to the one implemented by the job control patch.

Here's an idea of how a plugin session might work using json-rpc (jsonrpc version omitted):

plugin -> neovim: {"id": 1, "method": "listenEvent", "params": {"eventName": "keyPressed"}} neovim -> plugin: {"id": 1, "result": true} neovim -> plugin: {"method": "event", "params": {"name": "keyPressed", "eventArgs": {"keys": ["C"]}}} neovim -> plugin: {"method": "event", "params": {"name": "keyPressed", "eventArgs": {"keys": ["Ctrl", "Space"]}}} plugin -> neovim: {"id": 2, "method": "showPopup", "params": {"size": {"width": 10, "height": 2} "position": {"column": 2, "line": 3}, "items": ["Completion1", "Completion2"]}} plugin -> neovim: {"id": 2, "result": true}}

That shows a hypothetical conversation between neovim and completion plugin that displays completions when the user presses Ctrl+Space. The above scheme gives neovim nearly limitless extensibility and also improves stability as plugins will automatically be isolated from the main executable.

This system can also easily emulate the current scripting languages' interfaces to vim. For example, a plugin can emulate the python interface by running python scripts sent by vim in its own context and by exposing a 'vim' module with an API matching the current one. Calls to the API would simply be translated to json-rpc messages sent to vim.

5. New GUI architecture

Another contributing factor to vim's huge codebase is the explicit support for dozens of widget toolkits for GUI interfaces. Just like the legacy code support, gui-specific code will be removed.

neovim will handle GUIs similar to how it will handle plugins:

GUIs are separate programs, possibly written in different programming languages.

neovim will use its own stdin/stdout to receive input and send updates, again using json-rpc or msgpack-rpc.

The difference between plugins and GUIs is that plugins will be started by neovim, where neovim will be started by programs running the GUI. Here's a sample diagram of the process tree:

GUI program | ---> neovim | ---> Plugin 1 | ---> Plugin 2 | ---> Plugin 3

Hypothetical GUI session:

gui -> vim: {"id": 1, "method": "initClient", "params": {"size": {"rows": 20, "columns": 25}}} vim -> gui: {"id": 1, "result": {"clientId": 1}} vim -> gui: {"method": "redraw", "params": {"clientId": 1, "lines": {"5": " Welcome to neovim! "}}} gui -> vim: {"id": 2, "method": "keyPress", "params": {"keys": ["H", "e", "l", "l", "o"]}} vim -> gui: {"method": "redraw", "params": {"clientId": 1, "lines": {"1": "Hello ", "5": " "}}}

This new GUI architecture creates many interesting possibilities:

Modern GUIs written in high-level programming languages that integrate better with the operating system. We can have GUIs written using C#/WPF on Windows or Ruby/Cocoa on Mac, for example.

Plugins will be able emit custom events that may be handled directly by GUIs. This will enable the implementation of advanced features such as sublime's minimap.

A multiplexing daemon could keep neovim instances running in a headless server, while multiple remote GUIs could attach/detach to share editing sessions.

Simplified headless testing.

Embedding the editor into other programs. In fact, a GUI can be seen as a program that embeds neovim.

Here's a diagram that illustrates how a client-server process tree might look like:

Server daemon listening on tcp sockets <------ GUI 1 (attach/detach to running instances using tcp sockets) | | ---> neovim | | GUI 2 (sharing the same session with GUI 1) ---> Plugin 1 | ---> Plugin 2 | ---> Plugin 3

6. Development on Github

Development will happen within the GitHub organization and the code will be split across many repositories, unlike the current vim source tree.

There will be separate repositories for GUIs, plugins, runtime files (official vimscript) and distributions. This will let the editor receive improvements much faster as the patches don't have to all go through a single person for approval.

Travis will be used for continuous integration, so pull requests will be checked automatically.

What's been done so far

Source tree was cleaned up, leaving only files necessary for compilation/testing of the core

Source files were processed with unifdef to remove tons of FEAT_* macros

Files were processed with uncrustify to normalize source code formatting

The autotools build system was replaced by cmake

Currently being worked on: Port all IO to libuv

FAQ

Who are you and why should I believe that you can do this?

Good question. My name is Thiago de Arruda Padilha and I'm a freelance programmer living in Recife, Brazil with my wife. I can be contacted via email, twitter or IRC (tarruda on Freenode).

I'm not a well-known developer but I have made considerable contributions to open-source in the last few years.

I'm the author of vim's event loop patch and the job control patch, which were big motivators for this project.

I'm also responsible for tmux's wait-for command which was merged into tmux 1.8.

In the last few months, I've created pieces of software that might be considered complex by some:

vm.js, a javascript virtual machine implemented in pure javascript.

archdb a NoSQL transactional database, again implemented javascript.

You can see more of my projects and contributions in my GitHub profile.

Why this fundraiser?

I currently work as a freelancer full time and my expenses are increasing with a child on the way. It will be impossible for me to work on this project in my spare time and I really want to make this happen. I'm sure the features implemented by the first iteration would be welcomed by many other members of the vim community.

$10,000 will allow me to dedicate two months of full time work to this project, which will be enough to implement the following:

New, modern multi-platform UI written using qtlua.

New curses UI written using luaposix. It will look exactly like vim's terminal UI, but implemented with a high-level language that will greatly simplify maintenance. Not to mention it will let me remove a great deal of code dedicated to handling terminals in the core source.

New testing UI written in Lua with migration of all tests to this interface. The current test suite is hard to read and takes too long to run. This will let us write vim tests using lua's bdd framework busted which will improve readability and run speed.

New plugin architecture, with a python compatibility layer for using vim plugins written in python.

Full port of the editor IO to libuv.

Cross-platform implementation of job control for vimscript (easy on top of libuv).

Distributions for Windows, Linux and Mac, and a Windows installer.

I don't care about writing plugins or maintaing vim, why should I waste money on something that will benefit mostly vim developers and plugin authors?

Plugin authors will implement more advanced features for your favorite editor, and the new beautiful GUIs will match the eye candy of modern editors such as Sublime Text.