Among the things I dislike doing in web development, I think file upload is in the Top 3. It's not hard and lots of libraries in your favorite language will help you to that. But there are always some small annoyances to store files on servers. Now I deploy my code to Heroku, put my files on S3 and it's done. Far easier! The other day, I wanted to create a small HTTP service to upload files on S3 in Go but there's no packaged solution to do that, so I thought I would write an article about it. Here's how I did it.

Requirements

First we need a package to interact with Amazon Web Services. For this article I chose to use mitchellh/goamz from Mitchell Hashimoto (creator of Vagrant and HashiCorp). It's a fork of goamz/goamz. There is also an "official" package developed by Stripe with Amazon recently making it official, but it's marked as "incredibly experimental" so I prefered to use the more stable one.

For the HTTP part, I decided to use stack, my own framework.

go get -u github.com/mitchellh/goamz
go get -u github.com/nmerouze/stack

Upload Files on S3

I am not going to use a real S3 bucket to write my code, so this article will be written as an example on how to write web services in TDD with Go. First action would be to upload a file on S3. The test will start by initializing a fake S3 server and create the bucket:

package upload_test

import (
  "testing"

  "github.com/mitchellh/goamz/aws"
  "github.com/mitchellh/goamz/s3"
  "github.com/mitchellh/goamz/s3/s3test"
)

func startServer() (*s3test.Server, *s3.Bucket) {
  testServer, _ := s3test.NewServer(new(s3test.Config)) // Fake server
  auth := aws.Auth{"abc", "123", ""} // Fake credentials
  conn := s3.New(auth, aws.Region{Name: "faux-region-1", S3Endpoint: testServer.URL(), S3LocationConstraint: true}) // Needs to be true when testing
  bucket := conn.Bucket("foobar.com") // Fake bucket
  bucket.PutBucket(s3.Private) // Bucket creation
  return testServer, bucket
}

func TestPut(t *testing.T) {
  s, b := startServer()
  defer s.Quit() // Quit the fake server when the test is done
}

As we will have to start the server for each test, we can directly put the code in a function. Then we make a fake request to our web service.

func TestPut(t *testing.T) {
  // ...
  w := httptest.NewRecorder()
  r, _ := http.NewRequest("PUT", "/files/foo.txt", bytes.NewBufferString("foobar")) // We send a file named foo.txt with the content "foobar"
  r.Header.Set("Content-Type", "text/plain")
  upload.Service(b).ServeHTTP(w, r) // "upload" is the name of our package
}

Then we need to see if things go well. The status must be 200 (OK), we want to set the Location header to the URL of the file on S3, we want to check if the file has been created and finally to see if the JSON body is correct.

func TestPut(t *testing.T) {
  // ...
  expCode := 200
  if w.Code != expCode {
    t.Fatalf("Response status expected: %#v, got: %#v", expCode, w.Code)
  }

  expLoc := b.URL("/foo.txt") // We get the full URL of the file stored in the bucket
  if w.Header().Get("Location") != expLoc {
    t.Fatalf("Response Location header expected: %#v, got: %#v", expLoc, w.Header().Get("Location"))
  }

  expBody := fmt.Sprintf(`{"url":"%s"}`, expLoc)
  if w.Body.String() != expBody {
    t.Fatalf("Response body expected: %#v, got: %#v", expBody, w.Body.String())
  }

  data, _ := b.Get("/foo.txt") // We get the file stored in the bucket
  if string(data) != "foobar" {
    t.Fatalf("Stored file content expected: %#v, got: %#v", "foobar", string(data))
  }
}

Now we can see our test fail with go test ./.

# github.com/nmerouze/stack-examples/upload_test
./upload_test.go:32: undefined: upload.Service
FAIL  github.com/nmerouze/stack-examples/upload [build failed]

So now we know what to expect, let's make our first action. First we define the router of our service:

package upload

import (
  "net/http"

  "github.com/mitchellh/goamz/s3"
  "github.com/nmerouze/stack/mux"
)

func Service(bucket *s3.Bucket) http.Handler {
  m := mux.New()
  return m
}

Let's run our test again.

--- FAIL: TestPut (0.00 seconds)
  upload_test.go:36: Response status expected: 200, got: 404
FAIL
FAIL  github.com/nmerouze/stack-examples/upload 0.017s

Great our first expectation is visible, let's add the code for that:

package upload

import (
  "net/http"

  "github.com/mitchellh/goamz/s3"
  "github.com/nmerouze/stack/mux"
)

// All our handlers are methods of this struct so they are able to access the bucket.
type appContext struct {
  bucket *s3.Bucket
}

func (c *appContext) upsertFile(w http.ResponseWriter, r *http.Request) {
}

func Service(bucket *s3.Bucket) http.Handler {
  c := &appContext{bucket}
  m := mux.New()
  m.Put("/files/*path").ThenFunc(c.upsertFile)
  return m
}