.

.

An early peek at Go modules.(20 July 2018) Update:Since the writing of this article, Go1.11 came out with experimental support for Go modules. Some of the commands were also renamed/removed so this blogpost is in some sense outdated. However the main concepts still stand. Intro In this article, we will take go modules(earlier on it had the codename vgo) for a spin. A module is a collection of related Go packages. Modules are the unit of source code interchange and versioning. With modules, you can now work outside of GOPATH and also version your code in such a way that go is aware of. At the time of writing this, we need to be using go compiled from master branch for us to be able to use go modules. So lets do that, We could clone go from master and compile it ourselves, but I won't do that; instead I'll use gimme which is a tool developed by TravisCI peeps to help in installing various go versions. The instructions on how to install gimme can be found here; But since I'm on OSX;

brew install gimme && gimme master

source ~/.gimme/envs/gomaster.env && go version go version devel +d278f09333 Thu Jul 19 05:40:37 2018 +0000 darwin/amd64

git clone git@github.com:komuw/meli.git ~/mystuff/meli && cd ~/mystuff/meli

go mod -init go: creating new go.mod: module github.com/komuW/meli go: copying requirements from Gopkg.lock

module github.com/komuw/meli require ( github.com/Microsoft/go-winio v0.4.8 github.com/docker/distribution v0.0.0-20170720211245-48294d928ced github.com/docker/docker v1.13.1 github.com/docker/docker-credential-helpers v0.6.1 github.com/docker/go-connections v0.3.0 github.com/docker/go-units v0.3.3 github.com/pkg/errors v0.8.0 golang.org/x/net v0.0.0-20180712202826-d0887baf81f4 golang.org/x/sys v0.0.0-20180715085529-ac767d655b30 gopkg.in/yaml.v2 v2.2.1 )

go build -o meli cli/cli.go && ./meli --help Usage of ./meli: -build Rebuild services -d Run containers in the background -f string path to docker-compose.yml file. (default "docker-compose.yml") -up Builds, re/creates, starts, and attaches to containers for a service. -v Show version information. -version Show version information.

cat go.sum github.com/Microsoft/go-winio v0.4.8/go.mod h1:VhR8bwka0BXejwEJY73c50VrPtXAaKcyvVC4A4RozmA= github.com/docker/distribution v0.0.0-20170720211245-48294d928ced h1:/ybq/Enozyi+nBSAkL4j7vd+IBV6brrxB2srIO5VWos= ....

echo "Im a hacker" >> ~/go/src/mod/github.com/pkg/errors@v0.8.0/README.md

go mod -verify github.com/pkg/errors v0.8.0: dir has been modified (~/go/src/mod/github.com/pkg/errors@v0.8.0)

sed -i.bak "s/1NNxqwp/hackedHash/" go.sum && go build -o meli cli/cli.go go: verifying gopkg.in/yaml.v2@v2.2.1/go.mod: checksum mismatch downloaded: h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= go.sum: h1:hI93XBmqTisBFMUTm0b8Fm+jr3DghackedHash+5A1VGuI=

go mod -sync go: finding github.com/stretchr/testify/assert latest go: finding github.com/stevvooe/resumable/sha256 latest ..

go help mod | grep -i indirect -A 2 -B 2 Note that this only describes the go.mod file itself, not other modules referred to indirectly. For the full set of modules available to a build, use 'go list -m -json all'.

go help modules | grep -i indirect -A 2 -B 2 ... Requirements needed only for indirect uses are marked with a "// indirect" comment in the go.mod file. Indirect requirements are automatically removed from the go.mod file once they are implied by other

grep -rsIin testify . ./go.mod:14: github.com/stretchr/testify v1.2.2 // indirect ./go.sum:21:github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= ./go.sum:22:github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=

go mod -graph github.com/komuw/meli github.com/stevvooe/resumable@v0.0.0-20170302213456-2aaf90b2ceea github.com/komuw/meli github.com/stretchr/testify@v1.2.2 github.com/komuw/meli golang.org/x/net@v0.0.0-20180712202826-d0887baf81f4

git clone git@github.com:komuw/moby.git ~/mystuff/moby

func NewEnvClient() (*Client, error) { + fmt.Println("

\t You are using docker version:", api.DefaultVersion) return NewClientWithOpts(FromEnv) }

replace github.com/docker/docker v1.13.1 => ~/mystuff/moby

go mod -verify go: errors parsing go.mod: ~/mystuff/meli/go.mod:20: replacement module without version must be directory path (rooted or starting with ./ or ../)

replace github.com/docker/docker v1.13.1 => ../moby

go mod -verify go: parsing ../moby/go.mod: open ~/mystuff/moby/go.mod: no such file or directory go: error loading module requirements

echo 'module "github.com/docker/docker"' >> ~/mystuff/moby/go.mod && \ go mod -verify all modules verified

go build -o meli cli/cli.go go: finding github.com/gogo/protobuf/proto latest go: finding github.com/gogo/protobuf v1.1.1 go: downloading github.com/gogo/protobuf v1.1.1 go: finding github.com/opencontainers/image-spec/specs-go latest go: finding github.com/opencontainers/image-spec v1.0.1 go: downloading github.com/opencontainers/image-spec v1.0.1 # github.com/komuw/meli ./types.go:77:44: undefined: volume.VolumesCreateBody ./types.go:120:70: undefined: volume.VolumesCreateBody ./volume.go:16:3: undefined: volume.VolumesCreateBody # github.com/docker/docker/client ../moby/client/container_commit.go:17:15: undefined: reference.ParseNormalizedNamed ../moby/client/container_commit.go:25:9: undefined: reference.TagNameOnly ../moby/client/container_commit.go:30:16: undefined: reference.FamiliarName ../moby/client/image_create.go:16:14: undefined: reference.ParseNormalizedNamed ../moby/client/image_create.go:22:25: undefined: reference.FamiliarName ../moby/client/image_import.go:18:16: undefined: reference.ParseNormalizedNamed ../moby/client/image_pull.go:23:14: undefined: reference.ParseNormalizedNamed ../moby/client/image_pull.go:29:25: undefined: reference.FamiliarName ../moby/client/image_pull.go:59:8: undefined: reference.TagNameOnly ../moby/client/image_push.go:19:14: undefined: reference.ParseNormalizedNamed ../moby/client/image_push.go:19:14: too many errors

import "github.com/pkg/errors" func main() { err := errors.New("whoops useless error") fmt.Printf("%v", err) .... }

replace github.com/pkg/errors v0.8.0 => ../errors

echo 'module "github.com/pkg/errors"' >> ~/mystuff/errors/go.mod

func New(message string) error { fmt.Println("

\t hello new called.") return &fundamental{ msg: message, stack: callers(), } }

go build -o meli cli/cli.go && ./meli --help hello new called. whoops useless errorUsage of ./meli: -build Rebuild services

That installs gimme and then uses gimme to install Go from master branch. Lets activate the newly installed go and check versionI have a go package called meli and we are going to convert that to use go modules. meli is a faster, smaller alternative to docker-compose(albeit with less features.) So lets clone meli in a directory that is outside GOPATH. My GOPATH is at ~/go so we'll clone into ~/mystuff instead.run;the -init flag initializes and writes a newto the current directory, in effect creating a new module rooted at the current directory. If you were using another dependency manager before, mod -nit will intialize thefile using that dependency manager's files. I was using dep as my dependency manager so go mod -init used that. From what I understand, go mod "already supports readingdifferent legacy file formats (GLOCKFILE, Godeps/Godeps.json, Gopkg.lock, dependencies.tsv, glide.lock, vendor.conf, vendor.yml, vendor/manifest, vendor/vendor.json)" - see this comment by Russ Cox. It's nice to see that, the Go team has put some thought into that. Let's have a look at thefile it created;All my dependencies that were listed inhave been added towith their correct versions. Notice though that under dep, meli depended on version v2.6.2 However go mod added it with version v0.0.0-20170720211245-48294d928ced That is called a pseudo-version, the second part(20170720211245) is the timestamp in UTC of the commit hash 48294d928ced. The commit 48294d928ced is the commit corresponding to version v2.6.2, see here Note: the pseudo versions are expected behaviour whilst a project is not yet a module (and its versions is >=2) Pretty neat, huh; but does it work? Let's build the damn thing and see if it works(remember we are doing all these outside of GOPATH)It works fine.is not the only file created, afile was also created.contains the expected cryptographic checksums of the content of specific module versions The go command maintains a cache of downloaded packages(in $GOPATH/src/mod) and computes and records the cryptographic checksum of each package at download time. The 'go mod -verify' command checks that the cached copies of module downloads still match both their recorded checksums and the entries inLets check this crypto thing.Then run;If you work in enterprise, this is the point at which you call in your Red team to figure out who is messing up with your packages. Even though, we messed with the cached github.com/pkg/errors package, it doesnt stop us from building our package.still works okay. I do not know if go build should complain if it finds the cached packages have been messed with, or whether it should redownload them afresh or just build the package as if nothing has happened(like it did.) However, if you mess with thefile; go build fails with an error.I'm liking the look of this crypto checksuming thing. go mod has other flags that you can try out, runto see them all. lets try the -sync flag which "synchronizeswith the source code in the module." Synchronization of modules seems like something we might want to do, right?wait, why is it adding new packages? It added new packages towith a comment //indirect Let's see if the documentation can help us discover what is up with these //indirect thing.not useful, lets try go help modules instead. I do not know why the documentation is spread between go help mod and go help modules; but anyway;Okay, so the documentation seems to be saying that, for example in meli's case; although meli does not useone of it's dependencies may be using it. So which dependency of meli is using testify(meli doesnt use testify or any other non-stdlib testing libraries)? Because meli still vendors its dependencies, lets see if we can use grep to find out;grep isn't helping, maybe go mod has a flag to give us this information? go mod has a -graph flag which according to the documentation; "The -graph flag prints the module requirement graph (with replacements applied) in text form." niice, looks like what we need.That's not very helpful, I still do not know which dependency introducedI asked on the #modules slack channel and someone suggested I try go list; butdidn't help either.One of the hardest things I've had before with Go is contributing to other peoples' projects. Usually -in other languages- you would fork the project, make changes, run to make sure everything works okay, then when happy, open a pull request from your fork to the other project. With Go however, I've had problems. I would fork a project and make changes; but when it came to running the thing, things would go haywire. This is because import paths would still be pointing to the old project instead of mine. Note; this is probably my own failing rather than that of Go. If you think I've been doing it wrong, let me know. I recently came across this tweet by Francesc Campoy and it has improved things for me, but it still felt odd. Talking of Campoy, he has a fantastic youtube channel that is all things Go, If you have never checked out, do yourselve a favour. Can go modules help us here? It turns out, they can! I guess. meli depends on. Let's say we wanted to add some feature to the docker client that we would later propose in a pull request to docker. I forked docker over to https://github.com/komuw/moby Now lets clone our fork into a directory that is outside GOPATHThe feature we want to add is; every time you declare a docker client(using), it should log the docker version that you are using. Here's the change I made to thefunctionThe full diff can be seen here We have made changes to the docker client on our local clone at. How do we use that change inbefore even pushing those changes to our fork(before even sending a pull request to docker)? go modules supports dependency replacement. The replacement can point to go code that is anywhere(including in our machine.) Add the following line toThat is telling go to replace the docker dependency with a local dependency at the path ~/mystuff/mobynice error message, let's comply and use relative paths; change the line intoThis time around the error message is not that descriptive. Paul Jolly(who has been doing an amazing job answering go module related questions all over the internet), mentioned that "the new path should be a directory on the local system that contains a module" So lets add a go.mod file to our clone of moby(docker)This is looking good. Lets rebuild meli to use our modified copy of docker.Looks like I picked a hard one. I'm guessing that I'm running into issues that go much deeper than just go modules?? Maybe? My speculation is that the way docker uses import path comments of the form see here , is muddying things. So probably, even though I have added a replace statementthe code instill has import commentsthat makes go try and usewhich is what we wanted to replace in the first place?? I'm just speculating here, there's a good probability that there's something I'm overlooking. I'll try and open an issue with the go repo(or ask on #modules slack channel) sometime later if just to satisfy my curiosty.Even though I was not able to use a local copy of docker, I was able to carry out the same procedure with another package. Lets undo the replace directive we had inWe are going to try and addas a dependency to meli, we will fork it(pkg/errors), clone it outside GOPATH, make changes to it and try to use our local copy. here's the change I made toto usefor error handling;The full diff can be seen here runso that those changes get picked up. When you do that,is added as a direct dependency. Lets fork errors package to https://github.com/komuw/errors and clone it locally to a path outside GOPATH ieThen we add a replacement directive inof course we also add a go.mod file to the local errors packageAs more packages/modules addfiles to their repo's we won't have to do this. I wish go mod would be able to automatically add afile for you, but it seems like per this comment from Russ Cox that it is not possible to do so without unwanted side effects? Let's modify the local errors package. We want to modify thefunction to log something when called. The change I made is;The full diff is here Lets build meli and run it.Look ma, We done made it.is now usingfor error handling, but we are using a local copy of our fork ofPretty neat if you ask me.I like the direction that go modules is taking. Not all t's are tied and i's dotted; At the time of writing this, there are about 40 open issues concerning go modules. I expect that number to rise as more people try out go modules. go modules is expected to ship with Go version 1.11(eta ~August) as an experimental feature. go modules will also make it possible for Go to download packages from a registry of your choice(à la npm, pypi etc). Have a look at https://github.com/gomods/athens , which is an upcoming package registry and proxy server for Go Shout out to Paul Jolly, Jeff Wendling, Bryan C. Mills, Russ Cox among others who have had to do a lot of hand holding all over the internet on go modules related issues/questions. Special shout out to Sam Boyer for his critique(in the literary sense) of go modules(links below.) Related readings: 1. Go & Versioning , by Russ Cox. 2. Taking Go modules for a spin , by Dave Cheney 3. Thoughts on vgo and dep , by Sam Boyer 4. An Analysis of Vgo , by Sam Boyer 5. Project Athens , by Aaron Schlesinger