TL;DR

Stream released vg almost three years ago, and since its inception, Go has improved quite a bit in regards to its dependency management. Since Go has come so far, we’re now advocating that the community use native Go modules rather than vg.

The Gist

Stream provides an API as a service to build highly scalable feeds and white-label chat for any application or messaging service. In the beginning days of Stream, the entire infrastructure utilized Python on the backend; however, the amount of throughput demanded by Stream's customers, in addition to the growing size of the development team, made Python inadequate for continued use. With that, the team decided to switch to a more performant programming language – Go.

We have already written about our decision-making process a couple of years on our blog, there were many things we liked about Go at that time. Unfortunately, dependency management was not one of them – there were no lockfiles, no clear way to version dependencies, etc.

For this reason, we wrote a small library called "virtualgo" that wrapped dep and made it dead simple for developers to work on several Go projects at the same time without dependency clashing.

Note: Everything that motivated us 3 years ago does not apply anymore to dependency management in Go. Modules have solved all these problems and added additional functionality as well such as enforcing semver, immutability and checksum database, etc.

We were very excited about Go modules and tried them as soon as possible (Go 1.11). Sadly we had to postpone using them as many libraries did not support them at all; in July of 2019, all libraries we needed had support for modules, and we started migrating projects over.

Migrating to Go modules is incredibly easy; in our case, because we used vg and dep, we could use go mod itself to migrate the dependency manifests. Surprisingly that was pretty much all we had to do.

A few items to expand:

Cgo isn’t Go and very problematic. It’s almost always a good idea to stay away from it because most of the guarantees of Go, for example, in terms of portability, don’t exist.

We have used tools package pattern to track the exact version of our necessary tools such as stringer, protocol buffers, etc. By this, there is no divergence between developer environments. This package is also guarded by tools build flag, which doesn’t add any overhead into the build speed or output.

Thanks to modules we were able to drop some build optimizations that made our build pretty complex.

We’re using sentry for error reporting, which can generate context snippets whenever an error occurs, we have used vendor mod to generate these snippets for 3rd party dependencies, too.

Summary