Go Serverless From POC to Prod 27 August 2018 Steven Bogacz Software Engineer at SendGrid

Our Goals Have a serverless app in AWS

Idiomatic and approachable Go code

Easy to test and iterate on

Flexible deployment Let's work toward them with a toy app 2

What are we trying to build? We want a lightweight, ephemeral storage API Create blob

Retrieve blob

Delete blob

All blobs can hang around for a day, somewhat flexible on expiration

...and we want it... in the CLOUD! 3

We choose the AWS stack, for reasons APIGW + Lambda to serve the code

Lambda has had Go support since the start of 2018

S3 will be our storage backend

Lifecycle policies can handle the expiration for us 4

Time to POC! Native Go Lambda support with API Gateway isn't quite like using the standard net/http library.

library. Handler interface requires a json.Encode -able object, which precludes using stdlib (unexported fields)

-able object, which precludes using stdlib (unexported fields) aws-lambda-go/events has objects for proxy API Gateway Requests/Responses In our case, that looks like Handler(*events.APIGatewayProxyRequest) (*events.APIGatewayProxyResponse, error) AWS Lambda Handler Documentation 5

POC Handler Code Using the events.APIGatewayProxyRequest type directly: switch req.HTTPMethod { case "POST": key, err = postFile(ctx, req.Body) case "GET": key, err = extractKey(req) if err != nil { return nil, err } data, err = getFile(ctx, key) case "DELETE": key, err = extractKey(req) if err != nil { return nil, err } err = deleteFile(ctx, key) } 6

POC POST Code Some decoupling: func postFile(ctx context.Context, data string) (string, error) { u, err := uuid.NewV4() if err != nil { return "", newInternalServerErr(err, "failed to generate key") } Sub-functions don't know about APIGW-specific types 7

POC tests Using a normal testing approach, we can unit test some helpers func TestHelpers(t *testing.T) { t.Run("extract key should parse the path correctly", func(t *testing.T) { req := &events.APIGatewayProxyRequest{ Path: "blobs/1234", } key, err := extractKey(req) require.NoError(t, err) require.Equal(t, "1234", key) }) // ... } This makes us use dummy input, and doesn't give us great coverage 8

Fully testing the POC We use Terraform (a cloud-agnostic Infrastructure-as-Code tool) Initial deploy terraform apply phase1.plan 33.54s Subsequent plan and apply terraform plan --out=phase1.plan 12.54 terraform apply phase1.plan 19.12 Terraform 9

One approach to lower the dev cycle time Can take a look at: Localstack Supports several locally running versions of AWS Services

- API Gateway

- DynamoDB

- Kinesis

- S3

- etc. 10

One approach to lower the dev cycle time Configuring our deployment isn't exactly trivial. Could use terraform, but there's a current issue open to get the AWS fakes to work: Terraform Provider Open Issue Can have set up scripts to run before we run our tests, e.g. make test spins up, go test s, spins down. Downsides: Lots of overhead to set up (especially without Terraform)

No guarantee APIs will behave exactly as they do in AWS

Lack of IAM, and other such services can make this a partial solution at best 11

A better approach Ask how would I do this as idiomatically as possible? net/http is Go's bread and butter

is Go's bread and butter We know how to test HTTP Servers in Go

We can abstract the store 12

A better approach - Our original layout tree . ├── config.go ├── errors.go ├── errors_test.go ├── main.go ├── main_test.go ├── toy └── toy.zip 0 directories, 7 files 13

A better approach - Our new layout tree . ├── cmd │ ├── http │ │ ├── main.go │ │ └── toy │ └── lambda │ └── main.go ├── internal │ ├── httperrs │ │ ├── errors.go │ │ └── errors_test.go │ ├── s3store │ │ └── s3_store.go │ └── toy │ ├── config.go │ ├── local_store.go │ ├── server.go │ ├── server_test.go │ └── store.go ├── toy └── toy.zip 7 directories, 13 files 14

A better approach - Our new server Now we have a server struct // Server represents all of the config and clients // needed to run our app type Server struct { store Store cfg *Config router *chi.Mux } Where Store is // Store provides an interface to the blobs we // store in the toy app type Store interface { Get(context.Context, string) (string, error) Set(context.Context, string, string) error Del(context.Context, string) error } 15

A better approach - Server Start Starting the server looks familiar // Start starts the server func (s *Server) Start() { s.router.Route("/blobs", func(r chi.Router) { r.Post("/", s.storeBlob) r.Route("/{key}", func(r chi.Router) { r.Get("/", s.getBlob) r.Delete("/", s.deleteBlob) }) }) if err := h.ListenAndServe(); err != nil { if err != http.ErrServerClosed { log.Fatal(err) } } } 16

A better approach - Two Store implementations Local // LocalStore is an in-memory implementation of our // store interface type LocalStore struct { store map[string]string lock *sync.RWMutex } S3 // S3Store is an S3 backed implementation of our Store // interaface type S3Store struct { client *s3.S3 bucket string } 17

A better approach - Now with HTTP tests! With a little setup of our server func TestMain(m *testing.M) { if err := setupServer(); err != nil { log.WithError(err).Fatal("failed to set up server for tests") } go s.Start() time.Sleep(500 * time.Millisecond) status := m.Run() s.Stop() os.Exit(status) } 18

A better approach - Now with HTTP tests! Now we can write more thorough tests in a familiar way testBlob := "this is a test blob" var key string t.Run("create a blob", func(t *testing.T) { resp, err := http.Post(addr, "application/text", strings.NewReader(testBlob)) require.NoError(t, err) require.Equal(t, http.StatusOK, resp.StatusCode) require.NotNil(t, resp) require.NotNil(t, resp.Body) defer resp.Body.Close() b, err := ioutil.ReadAll(resp.Body) require.NoError(t, err) key = string(b) }) 19

A better approach - Lambda It's great that our code and tests look a little more stdlib-ish, but how does that translate to our actual deployment, i.e. Lambda??? We can translate the events.APIGatewayProxyRequest , either by hand, or with: AWS Lambda Go API Proxy You may have noticed two packages under cmd , lambda and http 20

A better approach - Lambda As long as we expose our Router // Router exposes our chi Route externally func (s *Server) Router() *chi.Mux { return s.router } we can change our lambda/main.go to set up the proxy adapter chiLambda = chiadapter.New(s.Router()) and call it in the Handler // Handler satisfies the AWS Lambda Go interface func Handler(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { // If no name is provided in the HTTP request body, throw an error return chiLambda.Proxy(req) } 21

A better approach - Running locally ./toy --local-store Can curl locally, and get faster feedback curl -XPOST -d @key.json http://127.0.0.1:8080/blobs 43823ea6-6f48-49c2-a798-47eb728e50a7 0.02s curl http://127.0.0.1:8080/blobs/43823ea6-6f48-49c2-a798-47eb728e50a7 { "Name": { "S": "banana" }} 0.01s curl -XDELETE -v http://127.0.0.1:8080/blobs/43823ea6-6f48-49c2-a798-47eb728e50a7 * Trying 127.0.0.1... ... < HTTP/1.1 204 No Content 0.02s 22

Can we do even better? What have we gained? More idiomatic code, or at least, less domain specific

Fairly lightweight local tests leverage more of the code path

Can run as either a HTTP server in a traditional deployment or AWS Lambda We're flexible about what we're deployed on now, but what about our backend? 23

Can we do even better? go-cloud go-cloud project The go-cloud project aims to provide common interfaces to common services across cloud providers.

BEAR TRAP ALERT This means that they can only support common operations, wouldn't be helpful for more deployment-specific control BEAR TRAP ALERT Bucket methods cover the functionality we had before func NewBucket(b driver.Bucket) *Bucket func (b *Bucket) Delete(ctx context.Context, key string) error func (b *Bucket) NewReader(ctx context.Context, key string) (*Reader, error) func (b *Bucket) NewWriter(ctx context.Context, key string, opt *WriterOptions) (*Writer, error) We can even delete code! 24

Can we do even better? go-cloud diff --brief -r third fourth Only in third/internal: s3store Only in third/internal/toy: local_store.go Files third/internal/toy/server.go and fourth/internal/toy/server.go differ Only in third/internal/toy: store.go tree -I vendor ├── cmd │ ├── http │ │ ├── main.go │ │ └── toy │ └── lambda │ └── main.go ├── internal │ ├── httperrs │ │ ├── errors.go │ │ └── errors_test.go │ └── toy │ ├── config.go │ ├── server.go │ └── server_test.go 25

Can we do even better? go-cloud We can use the fileblob package for local tests and running locally func setupServer(dir string) error { port, err := freeport.GetFreePort() if err != nil { return errors.Wrap(err, "failed to get free port") } c := &Config{ BucketName: "test-bucket", Port: port, } addr = fmt.Sprintf("http://127.0.0.1:%d/blobs", port) log.Infof("starting server at %s", addr) store, err := fileblob.NewBucket(dir) if err != nil { return errors.Wrap(err, "failed to set up local store") } s = New(c, store) return nil } 26

Can we do even better? go-cloud Our locally running main can now support both backends... and it wouldn't be much work to support a third (e.g. Google Cloud Storage) if localStore { store, cleanup, err = getLocalStore() } else { store, cleanup, err = getS3Store() } if err != nil { return errors.Wrap(err, "failed to initialize store") } defer cleanup() s = toy.New(config, store) go s.Start() 27

Recap Part 1: POC code, tightly coupled to AWS, hard to test

Part 2: Try to use localstack to test, hard work to set up

Part 3: Decouple bulk of logic from target deployment, only worry about incoming messages at the outer layer ( lambda/main.go ) using aws-lambda-go-api-proxy, use an interface for our backend to let us run locally in-memory

) using aws-lambda-go-api-proxy, use an interface for our backend to let us run locally in-memory Part 4: Remove our new store interface and replace it with go-cloud's blob.Bucket , allowing for the same advantages above, and to optionally support more providers in the future 28

Questions? Talk material in GitHub 29