In this blog, I’ll be presenting my Final Report of my experience with Google Summer of Code 2019.

Introduction

During GSoC 2019, I contributed to Neovim. Neovim (nvim) is a project that seeks to aggressively refactor Vim in order to:

Simplify maintenance and encourage contributions

Split the work between multiple developers

Enable advanced UIs without modifications to the core

Maximize extensibility

The main goal of my GSoC project was to provide remote attachment capability to the builtin UI, which is called Terminal UI (TUI). That is, whenever the user starts Nvim as nvim --connect {addr} , this Nvim instance would connect to the Nvim server listening to {addr} , here {addr} can be either a pipe address ( /tmp/pipe ) or a tcp address ( a.b.c.d:port ). The Nvim client instance would send input to the remote Nvim server and reflects the UI of the remote Nvim server. So the Nvim client acts like any other external GUI.

I could complete the proposed objective very early, hence, as an extension to my work I worked on converting Nvim from a multi-threaded process to a multi-process system. This work aimed at running TUI and the nvim server as individual processes which would interact with each other using the msgpack_rpc ( RPC API ) of nvim . I could complete this objective too by the end of GSoC 2019 with very few minor bugs remaining. This implementation also ensures that there is no difference in user experience with the modified nvim when compared with the thread based nvim .

What has been completed

Here I will be discussing the work that has been done during GSoC 2019 in detail along with the technical details of each implementation.

Adding support for remote attachment

nvim already had support for connecting an external GUI client to the running nvim server . Most of the my work here involved making TUI an RPC client. This involved:

Adding support for a new command line argument, namely --connect .

. Sending all user inputs to the nvim server through the RPC channel.

through the channel. Generating redraw event (which are sent by the nvim server ) handlers. Here the generator scripts were written in Lua (code can be found here and here).

event (which are sent by the ) handlers. Here the generator scripts were written in (code can be found here and here). Converting grid_line (the dense array sent by the nvim server which contains the screen state) to raw_line (the array which is used by TUI to represent the screen state) (code can be found here).

(the dense array sent by the which contains the screen state) to (the array which is used by to represent the screen state) (code can be found here). Writing tests to ensure proper functioning of this new feature (code can be found here).

The following video is a short demo of the system working

2. Running TUI and the nvim server as individual processes

nvim was a multi-threaded process, where nvim server and TUI ran in separate threads and communicated with each other using ui_bridge (which was very similar to a proxy ). My work involved making the “builtin” TUI an RPC client and running it as a separate process from that of the nvim server . This also meant complete elimination of ui_bridge .

With the completion of my work, whenever the modified nvim is started, the TUI process is spawned first, which later starts a nvim --embed job. The TUI process then attaches to the embed process as a remote RPC client (this is similar to how TUI works when nvim is started with --connect ). Work here involved:

Starting nvim --embed job from TUI and connecting to it as a client (code can be found here).

job from and connecting to it as a client (code can be found here). Passing through the command line arguments to nvim --embed instance.

instance. Adding nvim_read_stdin() API to support reading from a non-tty stdin . This was done to support features like echo xxx | nvim - and echo xxx | nvim (code can be found here).

API to support reading from a non-tty . This was done to support features like and (code can be found here). Allowing GUI client to let server know when focus is gained. This was done by implementing a new API namely, nvim_ui_set_focus() (code can be found here).

to let know when focus is gained. This was done by implementing a new API namely, (code can be found here). Allowing TUI to set options such as, term_name (to set the terminal name), term_colors (to set t_Co ) and term_background (to set the default background) (code can be found here).

to set options such as, (to set the terminal name), (to set ) and (to set the default background) (code can be found here). Getting the exit code of server at the client .

at the . Supporting suspend - restart workflows (i.e use of ctrl-z to suspend nvim and fg to restart it).

What are the bugs / imperfections seen

Bracketed paste for very large inputs does not work correctly

In the current implementation, the TUI assumes that nvim server consumes all the input keys sent to it, but it is not the case always and this assumption leads to input_buffer overflow. This is the reason for why bracketed paste for large inputs doesn’t function properly. This problem would be resolved once #4448 is completed. “Negative” space doesn’t get cleared upon resizing the client TUI (in --connect mode)

Future work

Work towards proper “multihead” support (i e show different tabpages in two TUI ’s to the same server ).

This would also involve enabling the server to recognize each client, distinguish requests from each client and reply to each client individually. Work to make decoding of msgpack packets more efficient. Currently msgpack bytes are first decoded to the libmsgpack structs, then to the “api” data types (ex: Array , Dictionary , etc), and finally to UI events (in case of TUI client). Shipping the capabilities of TUI as a RPC GUI client in libnvim so that other external GUI clients could just work on the UI of their GUI client and not worry about building a complete event_loop sub-module and a msgpack sub-module.

Links to the project