Stupid Gopher Tricks GolangUK 21 August 2015 Andrew Gerrand

Video A video of this talk was recorded at GolangUK in London. Watch the talk on YouTube 2

This talk This talk is about things you might not know about Go. Some of this stuff you can add to your Go vocabulary. Other things are probably best left for special occasions. 3

Language 4

Type literals (1/2) Here's a familiar type declaration: type Foo struct { i int s string } The latter part of a type declaration is the type literal: struct { i int s string } Other examples of type literals are int and []string ,

which can also be declared as named types: type Bar int type Qux []string 5

Type literals (2/2) While we commonly use int and []string in variable declarations: var i int var s []string It is less common (but equally valid) to do the same with structs: var t struct { i int s string } An unnamed struct literal is often called an anonymous struct. 6

Anonymous structs: template data A common use is providing data to templates: // +build ignore package main import ( "log" "os" "strings" "text/template" ) func main() { tmpl := template.Must(template.New("").Parse(strings.TrimSpace(` Dear {{.Title}} {{.Lastname}}, Congratulations on reaching Level {{.Rank}}! I'm sure your parents would say "Great job, {{.Firstname}}!" Sincerely, Rear Admiral Gopher `))) data := struct { Title string Firstname, Lastname string Rank int }{ "Dr", "Carl", "Sagan", 7, } if err := tmpl.Execute(os.Stdout, data); err != nil { log.Fatal(err) } } You could also use map[string]interface{} ,

but then you sacrifice performance and type safety. 7

Anonymous structs: JSON (1/2) The same technique can be used for encoding JSON objects: // +build ignore package main import ( "encoding/json" "fmt" "log" ) func main() { b, err := json.Marshal(struct { ID int Name string }{42, "The answer"}) if err != nil { log.Fatal(err) } fmt.Printf("%s

", b) } And also decoding: // +build ignore package main import ( "encoding/json" "fmt" "log" ) func main() { var data struct { ID int Name string } err := json.Unmarshal([]byte(`{"ID": 42, "Name": "The answer"}`), &data) if err != nil { log.Fatal(err) } fmt.Println(data.ID, data.Name) } 8

Anonymous structs: JSON (2/2) Structs can be nested to describe more complex JSON objects: // +build ignore package main import ( "encoding/json" "fmt" "log" ) func main() { var data struct { ID int Person struct { Name string Job string } } const s = `{"ID":42,"Person":{"Name":"George Costanza","Job":"Architect"}}` err := json.Unmarshal([]byte(s), &data) if err != nil { log.Fatal(err) } fmt.Println(data.ID, data.Person.Name, data.Person.Job) } 9

Repeated literals and struct names In repeated literals like slices and maps, Go lets you omit the inner type name: type Foo struct { i int s string } var s = []Foo{ {6 * 9, "Question"}, {42, "Answer"}, } var m = map[int]Foo{ 7: {6 * 9, "Question"}, 3: {42, "Answer"}, } 10

Repeated literals and anonymous structs Combined with anonymous structs, this convenience shortens the code dramatically: var s = []struct { i int s string }{ struct { i int s string }{6 * 9, "Question"}, struct { i int s string }{42, "Answer"}, } var t = []struct { i int s string }{ {6 * 9, "Question"}, {42, "Answer"}, } 11

Anonymous structs: test cases (1/2) These properties enable a nice way to express test cases: func TestIndex(t *testing.T) { var tests = []struct { s string sep string out int }{ {"", "", 0}, {"", "a", -1}, {"fo", "foo", -1}, {"foo", "foo", 0}, {"oofofoofooo", "f", 2}, // etc } for _, test := range tests { actual := strings.Index(test.s, test.sep) if actual != test.out { t.Errorf("Index(%q,%q) = %v; want %v", test.s, test.sep, actual, test.out) } } } 12

Anonymous structs: test cases (2/2) You can go a step further and put the composite literal in the range statement itself: func TestIndex(t *testing.T) { for _, test := range []struct { s string sep string out int }{ {"", "", 0}, {"", "a", -1}, {"fo", "foo", -1}, {"foo", "foo", 0}, {"oofofoofooo", "f", 2}, // etc } { actual := strings.Index(test.s, test.sep) if actual != test.out { t.Errorf("Index(%q,%q) = %v; want %v", test.s, test.sep, actual, test.out) } } } But this is harder to read. 13

Embedded fields A struct field that has no name is an embedded field.

The embedded type's methods (and fields, if it is a struct)

are accessible as if they are part of the embedding struct. // +build ignore package main import "fmt" type A struct { s string } func (a A) String() string { return fmt.Sprintf("A's String method called: %v", a.s) } type B struct { A } func main() { b := B{} b.s = "some value" fmt.Println(b) } 14

Anonymous structs: embedded mutex Of course, you can embed fields in an anonymous struct. It's common to protect a global variable with a mutex variable: var ( viewCount int64 viewCountMu sync.Mutex ) By embedding a mutex in an anonymous struct, we can group the related values: var viewCount struct { sync.Mutex n int64 } Users of viewCount access it like this: viewCount.Lock() viewCount.n++ viewCount.Unlock() 15

Anonymous structs: implementing interfaces And you can embed interfaces, too. Here's a real example from Camlistore: The function is expected to return a ReadSeekCloser ,

but the programmer only had a string. Anonymous struct (and its standard library friends) to the rescue! return struct { io.ReadSeeker io.Closer }{ io.NewSectionReader(strings.NewReader(s), 0, int64(len(s))), ioutil.NopCloser(nil), } 16

Anonymous interfaces Interfaces can be anonymous, the most common being interface{} . But the interface needn't be empty: // +build ignore package main import ( "bytes" "fmt" ) func main() { var s interface { String() string } = bytes.NewBufferString("I'm secretly a fmt.Stringer!") fmt.Println(s.String()) } Useful for a sly type assertion (from src/os/exec/exec_test.go ): // Check that we can access methods of the underlying os.File. if _, ok := stdin.(interface { Fd() uintptr }); !ok { t.Error("can't access methods of underlying *os.File") } 17

Method values A "method value" is what you get when you evaluate a method as an expression.

The result is a function value. Evaluating a method from a type yields a function: // +build ignore package main import ( "bytes" "os" ) func main() { var f func(*bytes.Buffer, string) (int, error) var buf bytes.Buffer f = (*bytes.Buffer).WriteString f(&buf, "y u no buf.WriteString?") buf.WriteTo(os.Stdout) } Evaluating a method from a value creates a closure that holds that value: // +build ignore package main import ( "bytes" "os" ) func main() { var f func(string) (int, error) var buf bytes.Buffer f = buf.WriteString f("Hey... ") f("this *is* cute.") buf.WriteTo(os.Stdout) } 18

Method values: sync.Once The sync.Once type is used to perform a task once with concurrency safety. // Once is an object that will perform exactly one action. type Once struct { /* Has unexported fields. */ } func (o *Once) Do(f func()) This LazyPrimes type computes a slice of prime numbers the first time it is used: type LazyPrimes struct { once sync.Once // Guards the primes slice. primes []int } func (p *LazyPrimes) init() { // Populate p.primes with prime numbers. } func (p *LazyPrimes) Primes() []int { p.once.Do(p.init) return p.primes } 19

Method values: HTTP handlers You can use method values to implement multiple HTTP handlers with one type: type Server struct { // Server state. } func (s *Server) index(w http.ResponseWriter, r *http.Request) { /* Implementation. */ } func (s *Server) edit(w http.ResponseWriter, r *http.Request) { /* Implementation. */ } func (s *Server) delete(w http.ResponseWriter, r *http.Request) { /* Implementation. */ } func (s *Server) Register(mux *http.ServeMux) { mux.HandleFunc("/", s.index) mux.HandleFunc("/edit/", s.edit) mux.HandleFunc("/delete/", s.delete) } 20

Method values: another example In package os/exec , the Cmd type implements methods to set up

standard input, output, and error: func (c *Cmd) stdin() (f *os.File, err error) func (c *Cmd) stdout() (f *os.File, err error) func (c *Cmd) stderr() (f *os.File, err error) The caller handles each in the same way,

so it iterates over a slice of method values: type F func(*Cmd) (*os.File, error) for _, setupFd := range []F{(*Cmd).stdin, (*Cmd).stdout, (*Cmd).stderr} { fd, err := setupFd(c) if err != nil { c.closeDescriptors(c.closeAfterStart) c.closeDescriptors(c.closeAfterWait) return err } c.childFiles = append(c.childFiles, fd) } 21

Comparable types The Go spec defines a set of types as "comparable";

they may be compared with == and !=. Bools, ints, floats, complex numbers, strings, pointers,

channels, structs, and interfaces are comparable. // +build ignore package main import "fmt" func main() { var a, b int = 42, 42 fmt.Println(a == b) var i, j interface{} = a, b fmt.Println(i == j) var s, t struct{ i interface{} } s.i, t.i = a, b fmt.Println(s == t) } A struct is comparable only if its fields are comparable: // +build ignore package main import "fmt" func main() { var q, r struct{ s []string } fmt.Println(q == r) } 22

Comparable types and map keys Any comparable type may be used as a map key. a := map[int]bool{} a[42] = true type T struct { i int s string } b := map[*T]bool{} b[&T{}] = true c := map[T]bool{} c[T{37, "hello!"}] = true d := map[interface{}]bool{} d[42] = true d[&T{}] = true d[T{123, "four five six"}] = true d[ioutil.Discard] = true 23

Structs as map keys An example from the Go continuous build infrastructure: type builderRev struct { builder, rev string } var br = builderRev{"linux-amd64", "0cd299"} We track in-flight builds in a map.

The pre-Go 1 way was to flatten the data to a string first: inflight := map[string]bool{} inflight[br.builder + "-" + br.rev] = true But with struct keys, you can avoid the allocation and have cleaner code: inflight := map[builderRev]bool{} inflight[br] = true 24

Interfaces as map keys An example of interface map keys from Docker's broadcastwriter package: type BroadcastWriter struct { sync.Mutex writers map[io.WriteCloser]struct{} } func (w *BroadcastWriter) AddWriter(writer io.WriteCloser) { w.Lock() w.writers[writer] = struct{}{} w.Unlock() } func (w *BroadcastWriter) Write(p []byte) (n int, err error) { w.Lock() for sw := range w.writers { if n, err := sw.Write(p); err != nil || n != len(p) { delete(w.writers, sw) } } w.Unlock() return len(p), nil } 25

Structs and interfaces together as map keys A (very) contrived example: (Don't do this! Ever!) // +build ignore package main import "fmt" type cons struct { car string cdr interface{} } func (c cons) String() string { if c.cdr == nil || c.cdr == (cons{}) { return c.car } return fmt.Sprintf("%v %v", c.car, c.cdr) } func main() { m := map[cons]string{} c := cons{} for _, s := range []string{"life?", "with my", "I doing", "What am"} { c = cons{s, c} } m[c] = "No idea." fmt.Println(c, m[c]) } 26

Libraries 27

sync/atomic For a simple counter, you can use the sync/atomic package's

functions to make atomic updates without the lock. func AddInt64(addr *int64, delta int64) (new int64) func CompareAndSwapInt64(addr *int64, old, new int64) (swapped bool) func LoadInt64(addr *int64) (val int64) func StoreInt64(addr *int64, val int64) func SwapInt64(addr *int64, new int64) (old int64) First, define the global variable (appropriately documented!): // viewCount must be updated atomically. var viewCount int64 Then increment it with AddInt64 : count := atomic.AddInt64(&viewCount, 1) The set are available for Int32 , Uint32 , Int64 , Uint64 , Pointer , and Uintptr . 28

sync/atomic.Value Another option for sharing state is atomic.Value . For instance, to share configuration between many goroutines: type Config struct { Timeout time.Duration } var config atomic.Value To set or update, use the Store method: config.Store(&Config{Timeout: 2*time.Second}) To read, each goroutine calls the Load method: cfg := config.Load().(*Config) Note that storing different types in the same Value will cause a panic. 29

sync/atomic.Value: how it works (1/5) The atomic.Value primitive is the size of a single interface value: package atomic type Value struct { v interface{} } An interface value is represented by the runtime as two pointers:

one for the type, and one for the value. // ifaceWords is interface{} internal representation. type ifaceWords struct { typ unsafe.Pointer data unsafe.Pointer } To load and store an interface value atomically, it operates on the parts of the interface value with atomic.LoadPointer and atomic.StorePointer . 30

sync/atomic.Value: how it works (2/5) The Store method first validates the input: func (v *Value) Store(x interface{}) { if x == nil { panic("sync/atomic: store of nil value into Value") } Then uses unsafe to cast the current and new interface{} values to ifaceWords : // ... vp := (*ifaceWords)(unsafe.Pointer(v)) xp := (*ifaceWords)(unsafe.Pointer(&x)) (This allows us to get at the internals of those interface values.) 31

sync/atomic.Value: how it works (3/5) Spin while loading the type field: // ... for { typ := LoadPointer(&vp.typ) If it's nil the this is the first time the value has been stored.

Put a sentinel value (max uintptr) in the type field to "lock" it while we work with it: // ... if typ == nil { if !CompareAndSwapPointer(&vp.typ, nil, unsafe.Pointer(^uintptr(0))) { continue // Someone beat us to it. Wait. } Store the data field, then the type field, and we're done! // ... StorePointer(&vp.data, xp.data) StorePointer(&vp.typ, xp.typ) return } 32

sync/atomic.Value: how it works (4/5) If this isn't the first store, check whether a store is already happening: // ... if uintptr(typ) == ^uintptr(0) { continue // First store in progress. Wait. } Sanity check whether the type changed: // ... if typ != xp.typ { panic("sync/atomic: store of inconsistently typed value into Value") } If the type field is what we expect, go ahead and atomically store the value: // ... StorePointer(&vp.data, xp.data) return } } 33

sync/atomic.Value: how it works (5/5) The Load method first loads the interface's type field: func (v *Value) Load() (x interface{}) { vp := (*ifaceWords)(unsafe.Pointer(v)) typ := LoadPointer(&vp.typ) Then, check whether a store has happened, or is happening: // ... if typ == nil || uintptr(typ) == ^uintptr(0) { return nil } Otherwise, load the data field and return both type and data as a new interface value: // ... data := LoadPointer(&vp.data) xp := (*ifaceWords)(unsafe.Pointer(&x)) xp.typ = typ xp.data = data return } 34

A note on sync/atomic Usually you don't want or need the stuff in this package. Try channels or the sync package first. You almost certainly shouldn't write code like the atomic.Value implementation.

(And I didn't show all of it; the real code has hooks into the runtime.) 35

Tools 36

Testing 37

Subprocess tests Sometimes you need to test the behavior of a process, not just a function. func Crasher() { fmt.Println("Going down in flames!") os.Exit(1) } To test this code, we invoke the test binary itself as a subprocess: func TestCrasher(t *testing.T) { if os.Getenv("BE_CRASHER") == "1" { Crasher() return } cmd := exec.Command(os.Args[0], "-test.run=TestCrasher") cmd.Env = append(os.Environ(), "BE_CRASHER=1") err := cmd.Run() if e, ok := err.(*exec.ExitError); ok && !e.Success() { return } t.Fatalf("process ran with err %q, want exit status 1", err) } 38

Subprocess benchmarks (1/2) Go's CPU and memory profilers report data for an entire process.

To profile just one side of a concurrent operation, you can use a sub-process. This benchmark from the net/http package spawns a child process to make

requests to a server running in the main process. func BenchmarkServer(b *testing.B) { b.ReportAllocs() // Child process mode; if url := os.Getenv("TEST_BENCH_SERVER_URL"); url != "" { n, err := strconv.Atoi(os.Getenv("TEST_BENCH_CLIENT_N")) if err != nil { panic(err) } for i := 0; i < n; i++ { res, err := Get(url) // ... } os.Exit(0) return } // ... 39

Subprocess benchmarks (2/2) // ... res := []byte("Hello world.

") ts := httptest.NewServer(HandlerFunc(func(rw ResponseWriter, r *Request) { rw.Header().Set("Content-Type", "text/html; charset=utf-8") rw.Write(res) })) defer ts.Close() cmd := exec.Command(os.Args[0], "-test.run=XXXX", "-test.bench=BenchmarkServer$") cmd.Env = append([]string{ fmt.Sprintf("TEST_BENCH_CLIENT_N=%d", b.N), fmt.Sprintf("TEST_BENCH_SERVER_URL=%s", ts.URL), }, os.Environ()...) if out, err := cmd.CombinedOutput(); err != nil { b.Errorf("Test failure: %v, with output: %s", err, out) } } To run: $ go test -run=XX -bench=BenchmarkServer -benchtime=15s -cpuprofile=http.prof $ go tool pprof http.test http.prof 40

go list 41

go list $ go help list usage: go list [-e] [-f format] [-json] [build flags] [packages] List lists the packages named by the import paths, one per line. Show the packages under a path: $ go list golang.org/x/oauth2/... golang.org/x/oauth2 ... golang.org/x/oauth2/vk Show the standard library: $ go list std archive/tar ... unsafe Show all packages: $ go list all 42

go list -json The -json flag tells you everything the go tool knows about a package: $ go list -json bytes { "Dir": "/Users/adg/go/src/bytes", "ImportPath": "bytes", "Name": "bytes", "Doc": "Package bytes implements functions for the manipulation of byte slices.", "Target": "/Users/adg/go/pkg/darwin_amd64/bytes.a", "Goroot": true, "Standard": true, "Stale": true, "Root": "/Users/adg/go", "GoFiles": [ "buffer.go", "bytes.go", "bytes_decl.go", "reader.go" ], ... } It's easy to write programs that consume this data. 43

go list's Package struct (1/3) The go tool's documented Package struct describes all the possible fields: type Package struct { Dir string // directory containing package sources ImportPath string // import path of package in dir ImportComment string // path in import comment on package statement Name string // package name Doc string // package documentation string Target string // install path Shlib string // the shared library that contains this package (only set when -linkshared) Goroot bool // is this package in the Go root? Standard bool // is this package part of the standard Go library? Stale bool // would 'go install' do anything for this package? Root string // Go root or Go path dir containing this package // (more fields on next slide) } 44

go list's Package struct (2/3) type Package struct { // (more fields on previous slide) // Source files GoFiles []string // .go source files (excluding CgoFiles, TestGoFiles, XTestGoFiles) CgoFiles []string // .go sources files that import "C" IgnoredGoFiles []string // .go sources ignored due to build constraints CFiles []string // .c source files CXXFiles []string // .cc, .cxx and .cpp source files MFiles []string // .m source files HFiles []string // .h, .hh, .hpp and .hxx source files SFiles []string // .s source files SwigFiles []string // .swig files SwigCXXFiles []string // .swigcxx files SysoFiles []string // .syso object files to add to archive // (more fields on next slide) } 45

go list's Package struct (3/3) type Package struct { // (more fields on previous slide) // Cgo directives CgoCFLAGS []string // cgo: flags for C compiler CgoCPPFLAGS []string // cgo: flags for C preprocessor CgoCXXFLAGS []string // cgo: flags for C++ compiler CgoLDFLAGS []string // cgo: flags for linker CgoPkgConfig []string // cgo: pkg-config names // Dependency information Imports []string // import paths used by this package Deps []string // all (recursively) imported dependencies // Error information Incomplete bool // this package or a dependency has an error Error *PackageError // error loading package DepsErrors []*PackageError // errors loading dependencies TestGoFiles []string // _test.go files in package TestImports []string // imports from TestGoFiles XTestGoFiles []string // _test.go files outside package XTestImports []string // imports from XTestGoFiles } 46

go list -f (1/2) That's a ton of information, so what can we do with it? The -f flag lets you use Go's text/template package to format the ouput. Show package doc strings: $ go list -f '{{.Doc}}' golang.org/x/oauth2/... Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests. Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0". ... Show the Go files in a package: $ go list -f '{{.GoFiles}}' bytes [buffer.go bytes.go bytes_decl.go reader.go] Make the output cleaner with a join : $ go list -f '{{join .GoFiles " "}}' bytes buffer.go bytes.go bytes_decl.go reader.go 47

go list -f (2/2) With template logic we can test packages for certain conditions. Find standard libraries that lack documentation: $ go list -f '{{if not .Doc}}{{.ImportPath}}{{end}}' std internal/format internal/trace Find packages that depend (directly or indirectly) on a given package: $ go list -f '{{range .Deps}}{{if eq . "golang.org/x/oauth2"}}{{$.ImportPath}}{{end}}{{end}}' all golang.org/x/build/auth golang.org/x/build/buildlet golang.org/x/build/cmd/buildlet ... Find packages that are broken somehow (note -e ): $ go list -e -f '{{with .Error}}{{.}}{{end}}' all package github.com/golang/oauth2: code in directory /Users/adg/src/github.com/golang/oauth2 expects import "golang.org/x/oauth2" ... 48

go list and the shell Things get interesting once we start using the shell. For instance, we can use go list output as input to the go tool. Test all packages except vendored packages: $ go test $(go list ./... | grep -v '/vendor/') Test the dependencies of a specific package: $ go test $(go list -f '{{join .Deps " "}}' golang.org/x/oauth2) The same, but don't test the standard library: $ go test $( go list -f '{{if (not .Goroot)}}{{.ImportPath}}{{end}}' $( go list -f '{{join .Deps " "}}' golang.org/x/oauth2 ) ) 49

go list printing line counts for pkg in $(go list golang.org/x/oauth2/...); do wc -l $(go list -f '{{range .GoFiles}}{{$.Dir}}/{{.}} {{end}}' $pkg) | \ tail -1 | awk '{ print $1 " '$pkg'" }' done | sort -nr The output: 617 golang.org/x/oauth2/google 600 golang.org/x/oauth2 357 golang.org/x/oauth2/internal 160 golang.org/x/oauth2/jws 147 golang.org/x/oauth2/jwt 112 golang.org/x/oauth2/clientcredentials 22 golang.org/x/oauth2/paypal 16 golang.org/x/oauth2/vk 16 golang.org/x/oauth2/odnoklassniki 16 golang.org/x/oauth2/linkedin 16 golang.org/x/oauth2/github 16 golang.org/x/oauth2/facebook 50

go list generating dependency graphs ( echo "digraph G {" go list -f '{{range .Imports}}{{printf "\t%q -> %q;

" $.ImportPath .}}{{end}}' \ $(go list -f '{{join .Deps " "}}' time) time echo "}" ) | dot -Tsvg -o time-deps.svg 51

And there's more! There are many more fun corners of Go.

Can you find them all? :-) Read the docs, explore, and have fun! 52