Using the great and simple Go http package is more than enough (and what I use) to expose simple HTTP endpoints. But when you need to build a HTTP API with lots of endpoints and user/media types, you have a lot of work ahead.
I found that using goa was a good solutions in those cases, because:
- You design your API using a design language (DSL).
- Lot of things are generated for you (useful code, swagger documentation, a client for the API, etc.).
- You get a Single Source of Trust (SSoT), so for example your swagger documentation is always updated with your controllers.
But I’ll be easier to show it with a quick tutorial, so let’s go
Requirements
- Make sure that you have installed Go and that your GOPATH is correctly set.
$ go version go version go1.8.1 darwin/amd64 $ echo $GOPATH /Users/Redsadic
(I like to use my $HOME
as my GOPATH
)
- Install goa.
$ go get -v github.com/goadesign/goa
- Create your new Go project in the
$GOPATH/src
directory.
$ mkdir -p $GOPATH/src/github.com/julianvilas/dummy-secrets $ cd $GOPATH/src/github.com/julianvilas/dummy-secrets
Now let’s start with the fun
Design
First of all, we need to define the API using the goa DSL. Let’s create a dummy API to share secrets using One-Time links (OTL). It’s not the case of a complex API but it’s easy to show how to work with goa.
We will have 2 endpoints:
- To store a new secret returning the corresponding OTL.
- To retrieve a secret using its OTL.
To define the API create a package called design:
$ mkdir -p $GOPATH/src/github.com/julianvilas/dummy-secrets/design
And edit the file design.go
(can be named as you want, e.g. api.go
or split in different files).
package design import ( . "github.com/goadesign/goa/design" . "github.com/goadesign/goa/design/apidsl" ) var _ = API("dummy-secrets", func() { Title("Dummy secrets") Description("Share your secrets using a dummy API") Version("1.0") BasePath("/v1") Scheme("http") Host("localhost:8080") Consumes("application/json") })
By now we have added some basic information of the API. The BasePath
will affect the URLs of all the resources we are going to add later, and the Scheme
, Host
and Consumes
will affect both the client and the swagger documentation.
Now we can bootstrap the API running:
$ goagen bootstrap -d github.com/julianvilas/dummy-secrets/design app app/contexts.go app/controllers.go app/hrefs.go app/media_types.go app/user_types.go main.go tool/dummy-secrets-cli tool/dummy-secrets-cli/main.go tool/cli tool/cli/commands.go client client/client.go client/user_types.go client/media_types.go swagger swagger/swagger.json swagger/swagger.yaml $ ll total 8 drwxr-xr-x 7 Redsadic staff 238B 1 may 18:14 app/ drwxr-xr-x 5 Redsadic staff 170B 1 may 18:14 client/ drwxr-xr-x 4 Redsadic staff 136B 1 may 18:14 design/ -rw-r--r-- 1 Redsadic staff 555B 1 may 18:14 main.go drwxr-xr-x 4 Redsadic staff 136B 1 may 18:14 swagger/ drwxr-xr-x 4 Redsadic staff 136B 1 may 18:14 tool/
As can be seen, some files have been generated:
- The directory
tool
will contain a cli to interact with the API. - The
client
will contain a SDK, used by the cli, to interact with the API. - The
app
will implement all the needed low-level HTTP functions. - The
swagger
will contain the autogenerated documentation. - In the root directory, the
main.go
will start the HTTP server will all the API controllers (0 by now). That one, with thedesign
package are the only we should modify. The others from above are auto-generated and shouldn’t be edited by hand.
Let’s take a look at the main.go
:
//go:generate goagen bootstrap -d github.com/julianvilas/dummy-secrets/design package main import ( "github.com/goadesign/goa" "github.com/goadesign/goa/middleware" ) func main() { // Create service service := goa.New("dummy-secrets") // Mount middleware service.Use(middleware.RequestID()) service.Use(middleware.LogRequest(true)) service.Use(middleware.ErrorHandler(service, true)) service.Use(middleware.Recover()) // Start service if err := service.ListenAndServe(":8080"); err != nil { service.LogError("startup", "err", err) } }
A new goa service is created, applying some useful middleware to it. And then the HTTP server is run listening at the port 8080 we defined in the design.go
.
Now we can build, install and run it typing:
$ go install ./... $ dummy-secrets dummy-secrets dummy-secrets-cli $ dummy-secrets 2017/05/01 18:33:04 [INFO] listen transport=http addr=:8080
Now we have a really dummy API, because we haven’t defined yet any resource. So let’s add the endpoints we mentioned.
Edit the design/design.go
file and add:
var _ = Resource("Secrets", func() { BasePath("/secrets") Response(Created, func() { Status(201) Headers(func() { Header("Location", String, "Resource location", func() { Pattern("/secrets/[-a-zA-Z0-9]+") }) }) }) Action("create", func() { Routing(POST("/")) Description("Store a new Secret") Payload(SecretPayload) Response(Created) Response(InternalServerError, ErrorMedia) Response(BadRequest) }) Action("show", func() { Routing(GET("/:id")) Params(func() { Param("id", UUID) }) Description("Get a Secret by its ID") Response(OK, SecretMedia) Response(NotFound) Response(InternalServerError, ErrorMedia) Response(BadRequest) }) }) var SecretPayload = Type("SecretPayload", func() { Attribute("secret", String, func() { Description("A secret to be shared with someone.") Example(`I'm a secret, share me with someone safely please.`) MinLength(1) MaxLength(255) Pattern("^[[:print:]]+") }) Required("secret") }) var SecretMedia = MediaType("application/vnd.secret+json", func() { Reference(SecretPayload) Attributes(func() { Attribute("secret") Required("secret") }) View("default", func() { Attribute("secret") }) })
We have created a new Resource
“Secrets”, with path /v1/secrets
. Then we have defined two actions for it:
create
: receives a payload like'{ "secret" : "I am a dummy secret"}'
and returns anHTTP Created
response, with aLocation
header pointing to the secret.show
: retrieves a secret by its ID and returns theSecretMedia
.
As you see we have defined the attributes (with validation requirements), the responses and the data we expect from the users.
Now, let’s bootstrap again (we can delete the main.go
file first to let it be generated again).
$ rm main.go $ goagen bootstrap -d github.com/julianvilas/dummy-secrets/design app app/contexts.go app/controllers.go app/hrefs.go app/media_types.go app/user_types.go app/test app/test/secrets_testing.go main.go secrets.go tool/cli tool/cli/commands.go client client/client.go client/secrets.go client/user_types.go client/media_types.go swagger swagger/swagger.json swagger/swagger.yaml
Now we see that some more stuff has been generated:
- a
secrets.go
file that contains the secrets controller. - a helper under
app
to create tests for the controller.
Tne new main.go
now contains this new content, that is used to mount the controller into the goa service:
// Mount "Secrets" controller c := NewSecretsController(service) app.MountSecretsController(service, c)
And the secrets.go
contains:
package main import ( "github.com/goadesign/goa" "github.com/julianvilas/dummy-secrets/app" ) // SecretsController implements the Secrets resource. type SecretsController struct { *goa.Controller } // NewSecretsController creates a Secrets controller. func NewSecretsController(service *goa.Service) *SecretsController { return &SecretsController{Controller: service.NewController("SecretsController")} } // Create runs the create action. func (c *SecretsController) Create(ctx *app.CreateSecretsContext) error { // SecretsController_Create: start_implement // Put your logic here // SecretsController_Create: end_implement return nil } // Show runs the show action. func (c *SecretsController) Show(ctx *app.ShowSecretsContext) error { // SecretsController_Show: start_implement // Put your logic here // SecretsController_Show: end_implement res := &app.Secret{} return ctx.OK(res) }
Now we have to implement the logic of these two controllers.
Implement
First we need to create a storage where secrets can be stored and retrieved. In order to make things easy to test or replace and well organized, we will define a Persister
interface. Let’s put it in its own package persister
.
$ mkdir persister $ touch persister/persister.go
Edit the persister.go
and add:
package persister import ( uuid "github.com/satori/go.uuid" ) type Persister interface { Store(secret string) uuid.UUID Retrieve(id uuid.UUID) (string, error) }
And add also a implementation for the Persister
interface:
type MemPersister struct { storage map[string]string mux sync.RWMutex } func NewMemPersister() *MemPersister { return &MemPersister{ storage: make(map[string]string), } } func (mp *MemPersister) Store(secret string) uuid.UUID { id := uuid.NewV4() mp.mux.Lock() defer mp.mux.Unlock() mp.storage[id.String()] = secret return id } func (mp *MemPersister) Retrieve(id uuid.UUID) (string, error) { mp.mux.RLock() defer mp.mux.RUnlock() secret, ok := mp.storage[id.String()] if !ok { return "", fmt.Errorf("error: a secret with id %v doesn't exist.", id) } delete(mp.storage, id.String()) return secret, nil }
Now we need to do two things:
- Add a
Persister
to the secrets controller, and implement the logic. - Inject the
MemPersister
in the main.
First we modify the secrets.go
:
package main import ( "github.com/goadesign/goa" "github.com/julianvilas/dummy-secrets/app" "github.com/julianvilas/dummy-secrets/persister" ) // SecretsController implements the Secrets resource. type SecretsController struct { *goa.Controller storage persister.Persister } // NewSecretsController creates a Secrets controller. func NewSecretsController(service *goa.Service, st persister.Persister) *SecretsController { return &SecretsController{ Controller: service.NewController("SecretsController"), storage: st, } } // Create runs the create action. func (c *SecretsController) Create(ctx *app.CreateSecretsContext) error { secret := ctx.Payload.Secret id := c.storage.Store(secret) ctx.ResponseData.Header().Set("Location", app.SecretsHref(id.String())) return ctx.Created() } // Show runs the show action. func (c *SecretsController) Show(ctx *app.ShowSecretsContext) error { id := ctx.ID secret, err := c.storage.Retrieve(id) if err != nil { goa.LogError(ctx, err.Error()) return ctx.NotFound() } res := &app.Secret{secret} if err := res.Validate(); err != nil { goa.LogError(ctx, err.Error()) return ctx.InternalServerError(goa.ErrInternal(err)) } return ctx.OK(res) }
And finally the main.go
:
//go:generate goagen bootstrap -d github.com/julianvilas/dummy-secrets/design package main import ( "github.com/goadesign/goa" "github.com/goadesign/goa/middleware" "github.com/julianvilas/dummy-secrets/app" "github.com/julianvilas/dummy-secrets/persister" ) func main() { // Create service service := goa.New("dummy-secrets") // Mount middleware service.Use(middleware.RequestID()) service.Use(middleware.LogRequest(true)) service.Use(middleware.ErrorHandler(service, true)) service.Use(middleware.Recover()) // Mount "Secrets" controller mp := persister.NewMemPersister() c := NewSecretsController(service, mp) app.MountSecretsController(service, c) // Start service if err := service.ListenAndServe(":8080"); err != nil { service.LogError("startup", "err", err) } }
Run
Let’s test what we did. First of all, build and install again with go install ./...
.
Now use the cli to store a secret and the retrieve it:
$ dummy-secrets-cli CLI client for the dummy-secrets service Usage: dummy-secrets-cli [command] Available Commands: create Store a new Secret help Help about any command show Get a Secret by its ID Flags: --dump Dump HTTP request and response. -H, --host string API hostname (default "localhost:8080") -s, --scheme string Set the requests scheme -t, --timeout duration Set the request timeout (default 20s) Use "dummy-secrets-cli [command] --help" for more information about a command.
Start the service:
$ dummy-secrets 2017/05/01 20:03:21 [INFO] mount ctrl=Secrets action=Create route=POST /v1/secrets 2017/05/01 20:03:21 [INFO] mount ctrl=Secrets action=Show route=GET /v1/secrets/:id 2017/05/01 20:03:21 [INFO] listen transport=http addr=:8080
Store a secret:
$ dummy-secrets-cli create secrets --payload '{ "secret" : "a dummy secret" }' --dump 2017/05/01 20:05:30 [INFO] started id=OIBa7E5P POST=http://localhost:8080/v1/secrets 2017/05/01 20:05:30 [INFO] request headers Content-Type=application/json User-Agent=dummy-secrets-cli/1.0 2017/05/01 20:05:30 [INFO] request body={"secret":"a dummy secret"} 2017/05/01 20:05:30 [INFO] completed id=OIBa7E5P status=201 time=3.515675ms 2017/05/01 20:05:30 [INFO] response headers Date=Mon, 01 May 2017 18:05:30 GMT Content-Length=0 Content-Type=text/plain; charset=utf-8 Location=/v1/secrets/614dcf25-02e2-43dd-9809-e189abb7d8a7
In the last line of the log can be seen the Location
header where the ID of the secret is returned:
Location=/v1/secrets/614dcf25-02e2-43dd-9809-e189abb7d8a7
Now retrieve the secret using the ID:
$ dummy-secrets-cli show secrets --id 614dcf25-02e2-43dd-9809-e189abb7d8a7 --pp 2017/05/01 20:08:28 [INFO] started id=0VBkwhgo GET=http://localhost:8080/v1/secrets/614dcf25-02e2-43dd-9809-e189abb7d8a7 2017/05/01 20:08:28 [INFO] completed id=0VBkwhgo status=200 time=4.074434ms { "secret": "a dummy secret" }
If you try to run it again, you’ll see that the secret is no longer available, as we implemented it as an OTL.
You can test it also with curl:
$ curl -vvv -X POST --data '{ "secret" : "a dummy secret" }' http://localhost:8080/v1/secrets Note: Unnecessary use of -X or --request, POST is already inferred. * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 8080 (#0) > POST /v1/secrets HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.51.0 > Accept: */* > Content-Length: 31 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 31 out of 31 bytes < HTTP/1.1 201 Created < Location: /v1/secrets/e293bf3a-5b0b-487c-841c-a4ad17e4bbe9 < Date: Mon, 01 May 2017 18:13:25 GMT < Content-Length: 0 < Content-Type: text/plain; charset=utf-8 < * Curl_http_done: called premature == 0 * Connection #0 to host localhost left intact $ curl -vvv http://localhost:8080/v1/secrets/e293bf3a-5b0b-487c-841c-a4ad17e4bbe9 * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 8080 (#0) > GET /v1/secrets/e293bf3a-5b0b-487c-841c-a4ad17e4bbe9 HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/vnd.secret+json < Date: Mon, 01 May 2017 18:14:58 GMT < Content-Length: 28 < {"secret":"a dummy secret"} * Curl_http_done: called premature == 0 * Connection #0 to host localhost left intact $ curl -vvv http://localhost:8080/v1/secrets/e293bf3a-5b0b-487c-841c-a4ad17e4bbe9 * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 8080 (#0) > GET /v1/secrets/e293bf3a-5b0b-487c-841c-a4ad17e4bbe9 HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 404 Not Found < Date: Mon, 01 May 2017 18:15:00 GMT < Content-Length: 0 < Content-Type: text/plain; charset=utf-8 < * Curl_http_done: called premature == 0 * Connection #0 to host localhost left intact
And that was all! We got it running
You got the example in github, just:
$ go get -v github.com/julianvilas/dummy-secrets
Also you can see the swagger doc in http://swagger.goa.design/?url=julianvilas/dummy-secrets/design.
Improvements
- Create a
FilePersister
and to store in files instead of memory (or aS3Persister
, it’s easy to replace the Persister as we created it as an interface). - Create tests! You can use the
app/test
helpers goa generated for you. - Create documentation.
References