JSON, interfaces, and go generate Francesc Campoy Developer, Advocate, and Gopher

Your mission Your mission, should you choose to accept it, is to decode this message: { "name": "Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" } into: type Person struct { Name string Born time.Time Size ShirtSize } 2

Your mission (cont.) Where ShirtSize is an enum (1): type ShirtSize byte const ( NA ShirtSize = iota XS S M L XL ) (1): Go doesn't have enums.

In this talk I will refer to constants of integer types as enums. 3

Using a map 4

Using a map Pros: very simple Cons: too simple? we have to write extra code func (p *Person) Parse(s string) error { fields := map[string]string{} dec := json.NewDecoder(strings.NewReader(s)) if err := dec.Decode(&fields); err != nil { return fmt.Errorf("decode person: %v", err) } // Once decoded we can access the fields by name. p.Name = fields["name"] 5

Parsing dates Time format based on a "magic" date: Mon Jan 2 15:04:05 -0700 MST 2006 An example: // +build ignore,OMIT package main import ( "fmt" "time" ) func main() { now := time.Now() fmt.Printf("Standard format: %v

", now) fmt.Printf("American format: %v

", now.Format("Jan 2 2006")) fmt.Printf("European format: %v

", now.Format("02/01/2006")) fmt.Printf("Chinese format: %v

", now.Format("2006/01/02")) } 6

Why that date? Let's reorder: Mon Jan 2 15:04:05 -0700 MST 2006 into: 01/02 03:04:05 PM 2006 -07:00 MST which is: 7

1 2 3 4 5 6 7! 8

Parsing the birth date: Since our input was: { "name": "Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" } Parse the birth date: born, err := time.Parse("2006/01/02", fields["birthdate"]) if err != nil { return fmt.Errorf("invalid date: %v", err) } p.Born = born 9

Parsing the shirt size Many ways of writing this, this is a pretty bad one: func ParseShirtSize(s string) (ShirtSize, error) { sizes := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL} ss, ok := sizes[s] if !ok { return NA, fmt.Errorf("invalid ShirtSize %q", s) } return ss, nil } Use a switch statement, but a map is more compact. 10

Parsing the shirt size Our complete parsing function: func (p *Person) Parse(s string) error { fields := map[string]string{} dec := json.NewDecoder(strings.NewReader(s)) if err := dec.Decode(&fields); err != nil { return fmt.Errorf("decode person: %v", err) } // Once decoded we can access the fields by name. p.Name = fields["name"] born, err := time.Parse("2006/01/02", fields["birthdate"]) if err != nil { return fmt.Errorf("invalid date: %v", err) } p.Born = born p.Size, err = ParseShirtSize(fields["shirt-size"]) return err } 11

Does this work? // +build ignore,OMIT package main import ( "encoding/json" "fmt" "log" "strings" "time" ) const input = ` { "name": "Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" } ` type Person struct { Name string Born time.Time Size ShirtSize } type ShirtSize byte const ( NA ShirtSize = iota XS S M L XL ) func (ss ShirtSize) String() string { sizes := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"} s, ok := sizes[ss] if !ok { return "invalid ShirtSize" } return s } func ParseShirtSize(s string) (ShirtSize, error) { sizes := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL} ss, ok := sizes[s] if !ok { return NA, fmt.Errorf("invalid ShirtSize %q", s) } return ss, nil } func (p *Person) Parse(s string) error { fields := map[string]string{} dec := json.NewDecoder(strings.NewReader(s)) if err := dec.Decode(&fields); err != nil { return fmt.Errorf("decode person: %v", err) } // Once decoded we can access the fields by name. p.Name = fields["name"] born, err := time.Parse("2006/01/02", fields["birthdate"]) if err != nil { return fmt.Errorf("invalid date: %v", err) } p.Born = born p.Size, err = ParseShirtSize(fields["shirt-size"]) return err } func main() { var p Person if err := p.Parse(input); err != nil { log.Fatalf("parse person: %v", err) } fmt.Println(p) } Note: ShirtSize is a fmt.Stringer 12

JSON decoding into structs 13

JSON decoding into structs Use tags to adapt field names: type Person struct { Name string `json:"name"` Born time.Time `json:"birthdate"` Size ShirtSize `json:"shirt-size"` } But this doesn't fit: // +build ignore,OMIT package main import ( "encoding/json" "fmt" "log" "strings" "time" ) const input = ` { "name":"Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" } ` type Person struct { Name string `json:"name"` Born time.Time `json:"birthdate"` Size ShirtSize `json:"shirt-size"` } type ShirtSize byte const ( NA ShirtSize = iota XS S M L XL ) func (ss ShirtSize) String() string { s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss] if !ok { return "invalid ShirtSize" } return s } func main() { var p Person dec := json.NewDecoder(strings.NewReader(input)) if err := dec.Decode(&p); err != nil { log.Fatalf("parse person: %v", err) } fmt.Println(p) } 14

Let's use an auxiliary struct type Use string fields and do any decoding manually afterwards. var aux struct { Name string Born string `json:"birthdate"` Size string `json:"shirt-size"` } Note: the field tag for Name is not needed; the JSON decoder performs a case

insensitive match if the exact form is not found. 15

Let's use an auxiliary struct type (cont.) The rest of the Parse function doesn't change much: func (p *Person) Parse(s string) error { var aux struct { Name string Born string `json:"birthdate"` Size string `json:"shirt-size"` } dec := json.NewDecoder(strings.NewReader(s)) if err := dec.Decode(&aux); err != nil { return fmt.Errorf("decode person: %v", err) } p.Name = aux.Name born, err := time.Parse("2006/01/02", aux.Born) if err != nil { return fmt.Errorf("invalid date: %v", err) } p.Born = born p.Size, err = ParseShirtSize(aux.Size) return err } 16

Can we do better? 17

Current solution Repetition if other types have fields with: date fields with same formatting,

or t-shirt sizes. Let's make the types smarter so json.Decoder will do all the work transparently. Goal: json.Decoder should do all the work for me! 18

Meet Marshaler and Unmarshaler Types satisfying json.Marshaler define how to be encoded into json. type Marshaler interface { MarshalJSON() ([]byte, error) } And json.Unmarshaler for the decoding part. type Unmarshaler interface { UnmarshalJSON([]byte) error } 19

UnmarshalJSON all the things! 20

Let's make Person a json.Unmarshaler Replace: func (p *Person) Parse(s string) error { with: func (p *Person) UnmarshalJSON(data []byte) error { var aux struct { Name string Born string `json:"birthdate"` Size string `json:"shirt-size"` } dec := json.NewDecoder(bytes.NewReader(data)) if err := dec.Decode(&aux); err != nil { return fmt.Errorf("decode person: %v", err) } p.Name = aux.Name // ... rest of function omitted ... 21

Let's make Person a json.Unmarshaler (cont.) And our main function becomes: // +build ignore,OMIT package main import ( "bytes" "encoding/json" "fmt" "log" "strings" "time" ) const input = ` { "name": "Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" } ` type Person struct { Name string Born time.Time Size ShirtSize } type ShirtSize byte const ( NA ShirtSize = iota XS S M L XL ) func (ss ShirtSize) String() string { s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss] if !ok { return "invalid ShirtSize" } return s } func ParseShirtSize(s string) (ShirtSize, error) { ss, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s] if !ok { return NA, fmt.Errorf("invalid ShirtSize %q", s) } return ss, nil } func (p *Person) UnmarshalJSON(data []byte) error { var aux struct { Name string Born string `json:"birthdate"` Size string `json:"shirt-size"` } dec := json.NewDecoder(bytes.NewReader(data)) // HL if err := dec.Decode(&aux); err != nil { return fmt.Errorf("decode person: %v", err) } p.Name = aux.Name // ... rest of function omitted ... born, err := time.Parse("2006/01/02", aux.Born) if err != nil { return fmt.Errorf("invalid date: %v", err) } p.Born = born p.Size, err = ParseShirtSize(aux.Size) return err } func main() { var p Person dec := json.NewDecoder(strings.NewReader(input)) if err := dec.Decode(&p); err != nil { log.Fatalf("parse person: %v", err) } fmt.Println(p) } 22

UnmarshalJSON for enums Substitute ParseShirtSize : func ParseShirtSize(s string) (ShirtSize, error) { with UnmarshalJSON : func (ss *ShirtSize) UnmarshalJSON(data []byte) error { // Extract the string from data. var s string if err := json.Unmarshal(data, &s); err != nil { return fmt.Errorf("shirt-size should be a string, got %s", data) } // The rest is equivalen to ParseShirtSize. got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s] if !ok { return fmt.Errorf("invalid ShirtSize %q", s) } *ss = got return nil } 23

UnmarshalJSON for enums (cont.) Now use ShirtSize in the aux struct: // +build ignore,OMIT package main import ( "bytes" "encoding/json" "fmt" "log" "strings" "time" ) const input = `{ "name":"Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" }` type Person struct { Name string Born time.Time Size ShirtSize } type ShirtSize byte const ( NA ShirtSize = iota XS S M L XL ) func (ss ShirtSize) String() string { s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss] if !ok { return "invalid ShirtSize" } return s } func (ss *ShirtSize) UnmarshalJSON(data []byte) error { // Extract the string from data. var s string if err := json.Unmarshal(data, &s); err != nil { // HL return fmt.Errorf("shirt-size should be a string, got %s", data) } // The rest is equivalen to ParseShirtSize. got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s] if !ok { return fmt.Errorf("invalid ShirtSize %q", s) } *ss = got // HL return nil } func (p *Person) UnmarshalJSON(data []byte) error { var aux struct { Name string Born string `json:"birthdate"` Size ShirtSize `json:"shirt-size"` } dec := json.NewDecoder(bytes.NewReader(data)) if err := dec.Decode(&aux); err != nil { return fmt.Errorf("decode person: %v", err) } p.Name = aux.Name p.Size = aux.Size // ... rest of function omitted ... born, err := time.Parse("2006/01/02", aux.Born) if err != nil { return fmt.Errorf("invalid date: %v", err) } p.Born = born return nil } func main() { var p Person dec := json.NewDecoder(strings.NewReader(input)) if err := dec.Decode(&p); err != nil { log.Fatalf("parse person: %v", err) } fmt.Println(p) } Use the same trick to parse the birthdate. 24

Unmarshaling differently formatted dates Create a new type Date : type Date struct{ time.Time } And make it a json.Unmarshaler : func (d *Date) UnmarshalJSON(data []byte) error { var s string if err := json.Unmarshal(data, &s); err != nil { return fmt.Errorf("birthdate should be a string, got %s", data) } t, err := time.Parse("2006/01/02", s) if err != nil { return fmt.Errorf("invalid date: %v", err) } d.Time = t return nil } 25

Unmarshaling differently formatted dates (cont.) Now use Date in the aux struct: // +build ignore,OMIT package main import ( "bytes" "encoding/json" "fmt" "log" "strings" "time" ) const input = `{ "name":"Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" }` type Person struct { Name string Born Date Size ShirtSize } type ShirtSize byte const ( NA ShirtSize = iota XS S M L XL ) func (ss ShirtSize) String() string { s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss] if !ok { return "invalid ShirtSize" } return s } func (ss *ShirtSize) UnmarshalJSON(data []byte) error { var s string if err := json.Unmarshal(data, &s); err != nil { return fmt.Errorf("shirt-size should be a string, got %s", data) } got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s] if !ok { return fmt.Errorf("invalid ShirtSize %q", s) } *ss = got return nil } type Date struct{ time.Time } func (d Date) String() string { return d.Format("2006/01/02") } func (d *Date) UnmarshalJSON(data []byte) error { var s string if err := json.Unmarshal(data, &s); err != nil { return fmt.Errorf("birthdate should be a string, got %s", data) } t, err := time.Parse("2006/01/02", s) // HL if err != nil { return fmt.Errorf("invalid date: %v", err) } d.Time = t return nil } func (p *Person) UnmarshalJSON(data []byte) error { r := bytes.NewReader(data) var aux struct { Name string Born Date `json:"birthdate"` Size ShirtSize `json:"shirt-size"` } if err := json.NewDecoder(r).Decode(&aux); err != nil { return fmt.Errorf("decode person: %v", err) } p.Name = aux.Name p.Size = aux.Size p.Born = aux.Born return nil } func main() { var p Person dec := json.NewDecoder(strings.NewReader(input)) if err := dec.Decode(&p); err != nil { log.Fatalf("parse person: %v", err) } fmt.Println(p) } Can this code be shorter? 26

Yes! By making the Born field in Person of type Date . Person.UnmarshalJSON is then equivalent to the default behavior! It can be safely removed. // +build ignore,OMIT package main import ( "encoding/json" "fmt" "log" "strings" "time" ) const input = `{ "name":"Gopher", "birthdate": "2009/11/10", "shirt-size": "XS" }` type Person struct { Name string `json:"name"` Born Date `json:"birthdate"` Size ShirtSize `json:"shirt-size"` } type ShirtSize byte const ( NA ShirtSize = iota XS S M L XL ) func (ss ShirtSize) String() string { s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss] if !ok { return "invalid ShirtSize" } return s } func (ss *ShirtSize) UnmarshalJSON(data []byte) error { var s string if err := json.Unmarshal(data, &s); err != nil { return fmt.Errorf("shirt-size should be a string, got %s", data) } got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s] if !ok { return fmt.Errorf("invalid ShirtSize %q", s) } *ss = got return nil } type Date struct{ time.Time } func (d Date) String() string { return d.Format("2006/01/02") } func (d *Date) UnmarshalJSON(data []byte) error { var s string if err := json.Unmarshal(data, &s); err != nil { return fmt.Errorf("birthdate should be a string, got %s", data) } t, err := time.Parse("2006/01/02", s) if err != nil { return fmt.Errorf("invalid date: %v", err) } d.Time = t return nil } func main() { var p Person dec := json.NewDecoder(strings.NewReader(input)) if err := dec.Decode(&p); err != nil { log.Fatalf("parse person: %v", err) } fmt.Println(p) } 27

Was this really better? Code length: 86LoC vs 80LoC Reusability of types Easier to maintain Usage of the standard library 28

Other ideas 29

Roman numerals 30

Roman numerals Because why not? type romanNumeral int And because Roman numerals are classier type Movie struct { Title string Year romanNumeral } 31

Roman numerals (cont.) // +build ignore,OMIT package main import ( "encoding/json" "fmt" "log" "strings" ) type romanNumeral int var numerals = []struct { s string v int }{ {"M", 1000}, {"CM", 900}, {"D", 500}, {"CD", 400}, {"C", 100}, {"XC", 90}, {"L", 50}, {"XL", 40}, {"X", 10}, {"IX", 9}, {"V", 5}, {"IV", 4}, {"I", 1}, } func (n romanNumeral) String() string { res := "" v := int(n) for _, num := range numerals { res += strings.Repeat(num.s, v/num.v) v %= num.v } return res } func parseRomanNumeral(s string) (romanNumeral, error) { res := 0 for _, num := range numerals { for strings.HasPrefix(s, num.s) { res += num.v s = s[len(num.s):] } } return romanNumeral(res), nil } func (n romanNumeral) MarshalJSON() ([]byte, error) { if n <= 0 { return nil, fmt.Errorf("Romans had only natural (=>1) numbers") } return json.Marshal(n.String()) } func (n *romanNumeral) UnmarshalJSON(data []byte) error { var s string if err := json.Unmarshal(data, &s); err != nil { return err } p, err := parseRomanNumeral(s) if err == nil { *n = p } return err } type Movie struct { Title string Year romanNumeral } func main() { // Encoding movies := []Movie{{"E.T.", 1982}, {"The Matrix", 1999}, {"Casablanca", 1942}} res, err := json.MarshalIndent(movies, "", "\t") if err != nil { log.Fatal(err) } fmt.Printf("Movies: %s

", res) // Decoding var m Movie inputText := `{"Title": "Alien", "Year":"MCMLXXIX"}` if err := json.NewDecoder(strings.NewReader(inputText)).Decode(&m); err != nil { log.Fatal(err) } fmt.Printf("%s was released in %d

", m.Title, m.Year) } 32

Secret data 33

Secret data Some data is never to be encoded in clear text. type Person struct { Name string `json:"name"` SSN secret `json:"ssn"` } type secret string Use cryptography to make sure this is safe: func (s secret) MarshalJSON() ([]byte, error) { m, err := rsa.EncryptOAEP(crypto.SHA512.New(), rand.Reader, key.Public().(*rsa.PublicKey), []byte(s), nil) if err != nil { return nil, err } return json.Marshal(base64.StdEncoding.EncodeToString(m)) } Note: This solution is just a toy; don't use it for real systems. 34

Secret data (cont.) And use the same key to decode it when it comes back: func (s *secret) UnmarshalJSON(data []byte) error { var text string if err := json.Unmarshal(data, &text); err != nil { return fmt.Errorf("deocde secret string: %v", err) } cypher, err := base64.StdEncoding.DecodeString(text) if err != nil { return err } raw, err := rsa.DecryptOAEP(crypto.SHA512.New(), rand.Reader, key, cypher, nil) if err == nil { *s = secret(raw) } return err } 35

Secret data (cont.) Let's try it: // +build ignore,OMIT package main import ( "crypto" "crypto/rand" "crypto/rsa" _ "crypto/sha512" "encoding/base64" "encoding/json" "fmt" "log" ) var key *rsa.PrivateKey func init() { k, err := rsa.GenerateKey(rand.Reader, 2048) if err != nil { log.Fatalf("generate key: %v", err) } key = k } type Person struct { Name string `json:"name"` SSN secret `json:"ssn"` } type secret string func (s secret) MarshalJSON() ([]byte, error) { m, err := rsa.EncryptOAEP(crypto.SHA512.New(), rand.Reader, key.Public().(*rsa.PublicKey), []byte(s), nil) if err != nil { return nil, err } return json.Marshal(base64.StdEncoding.EncodeToString(m)) } func (s *secret) UnmarshalJSON(data []byte) error { var text string if err := json.Unmarshal(data, &text); err != nil { // HL return fmt.Errorf("deocde secret string: %v", err) } cypher, err := base64.StdEncoding.DecodeString(text) // HL if err != nil { return err } raw, err := rsa.DecryptOAEP(crypto.SHA512.New(), rand.Reader, key, cypher, nil) // HL if err == nil { *s = secret(raw) } return err } func main() { p := Person{ Name: "Francesc", SSN: "123456789", } b, err := json.MarshalIndent(p, "", "\t") if err != nil { log.Fatalf("Encode person: %v", err) } fmt.Printf("%s

", b) var d Person if err := json.Unmarshal(b, &d); err != nil { log.Fatalf("Decode person: %v", err) } fmt.Println(d) } 36

But most JSON enums are boring 37

go generate to the rescue! go generate : introduced in Go 1.4

a tool for package authors

an extra step before go build You will see it as comments in the code like: //go:generate go tool yacc -o gopher.go -p parser gopher.y More information in the blog post. 38

code generation tools: stringer stringer generates String methods for enum types. package painkiller //go:generate stringer -type=Pill type Pill int const ( Placebo Pill = iota Aspirin Ibuprofen Paracetamol ) Call go generate : $ go generate $GOPATH/src/path_to_painkiller which will create a new file containing the String definition for Pill . 39

jsonenums Around 200 lines of code. Parses and analyses a package using: go/{ast/build/format/parser/token}

golang.org/x/tools/go/exact , golang.org/x/tools/go/types And generates the code using: text/template And it's on github: github.com/campoy/jsonenums 40

Demo 41