Go Microservice Architecture

Sample architecture with Go, showing how the different pieces are tied together.

Architecture

TODO: Cleanup the diagram, and add more services

Requirements

Non-functional requirements:

• system should be highly resilient
• system should be available
• system should be scalable
• system should be observable

Abstract

The following architecture aims to solve several problems

• [x] Load balancing between services
• [x] Service discovery and registration
• [x] Resiliency patterns such as circuit breaker, timeout, retries, rate-limiting
• [x] Centralized logging
• [x] Health checks for services
• [x] Telemetry metrics collection and dashboards
• [x] blue/green deployment (traffic splitting), rolling upgrades
• [x] gRPC load balancing and discovery
• [x] open tracing capabilities
• [x] demonstrates the delegation of several capabilities to the infra, rather than repeating it at the code levels
• [ ] add block ip functionality
• [ ] add security pipeline
• [ ] add log analysis to process incoming logs
• [ ] add context logging (request id) that propagates through the system
• [ ] enhance/standardize monitoring with opencensus
• [ ] harden security (container user/group)
• [ ] add policy on resources limit (CPU, memory, number of instance etc)

Start

$ docker-compose up -d

UI

• consul: localhost:8500
• traefik: localhost:8080
• jaeger: localhost:16686
• prometheus: localhost:9090
• grafana: localhost:3000
• linkerd: localhost:9990
• namerd: localhost:9991
• elasticsearch: localhost:9200
• kibana: localhost:9200
• dejavu: localhost:1358

Call the Echo Service

Scale the service:

$ docker-compose up -d --scale node=10

$ repeat 10; curl -H "Host: echo.consul.localhost" localhost:80 && printf "\n";

Output:

{"hostname":"e7d4b1cc317c","text":"hello"}
{"hostname":"0bdb052e096a","text":"hello"}
{"hostname":"62fa842dcf9c","text":"hello"}
{"hostname":"2e7f8dcbbc43","text":"hello"}
{"hostname":"1a59218e8121","text":"hello"}
{"hostname":"bee0e7437024","text":"hello"}
{"hostname":"547af30289ee","text":"hello"}
{"hostname":"fec9b78a7e7c","text":"hello"}
{"hostname":"a27f75db0290","text":"hello"}
{"hostname":"5a9726496329","text":"hello"}

Calling Linkerd

In linkerd.yaml, we set it to listen to consul for changes and register the services there. We expose the port :4040 as the load balancer ingress, and set the identifier to io.l5d.header.token. To call the echo service:

$ curl -H "Host: echo" localhost:4140

Setup Namerd

Creating Egress

$ curl -v -XPUT -d @config/namerd.egress.dtab -H "Content-Type: application/dtab" http://localhost:4180/api/1/dtabs/consul_egress

Output:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4180 (#0)
> PUT /api/1/dtabs/consul_egress HTTP/1.1
> Host: localhost:4180
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/dtab
> Content-Length: 34
>
* upload completely sent off: 34 out of 34 bytes
< HTTP/1.1 204 No Content
<
* Connection #0 to host localhost left intact

Creating Ingress

$ curl -v -XPUT -d @config/namerd.ingress.dtab -H "Content-Type: application/dtab" http://localhost:4180/api/1/dtabs/consul_ingress

Output:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4180 (#0)
> PUT /api/1/dtabs/consul_ingress HTTP/1.1
> Host: localhost:4180
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/dtab
> Content-Length: 35
>
* upload completely sent off: 35 out of 35 bytes
< HTTP/1.1 204 No Content
<
* Connection #0 to host localhost left intact

Make Call to Linkerd

$ curl -H "Host: echo" localhost:4140

To simulate running traffic

# To simulate running traffic, runs for 120s
$ wrk -c1 -d120 -t1  -H "Host: echo" http://localhost:4140

Check Dtabs

$ curl http://localhost:4180/api/1/dtabs/consul_ingress

Fluentd logging

# Find the httpd endpoint and trigger it
$ repeat 10 curl http://localhost:32857/

TODO

• cleanup code
• create kubernetes example

Miscellenaous

# Either
  registrator:
    image: gliderlabs/registrator:vendor
    restart: always
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock
    command: -internal=true -deregister=always -ip=docker.for.mac.localhost -cleanup -tags=registrator consul://consul:8500

# Or
  registrator:
    image: gliderlabs/registrator:vendor
    restart: always
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock
    command: -internal=true -deregister=always -ip=docker.for.mac.localhost -cleanup -tags=registrator consul:8500
    network_mode: host