This is part 2 of 2 part series on A Practical Guide to HashiCorp Consul. The previous part was primarily focused on understanding the problems that Consul solves and how it solves them. This part is focused on a practical application of Consul in a real-life example. Let’s get started.

With most of the theory covered in the previous part, let’s move on to Consul’s practical example.

What are we Building?

We are going to build a Django Web Application that stores its persistent data in MongoDB. We will containerize both of them using Docker. Build and run them using Docker Compose.

To show how our web app would scale in this context, we are going to run two instances of Django app. Also, to make this even more interesting, we will run MongoDB as a Replica Set with one primary node and two secondary nodes.

Given we have two instances of Django app, we will need a way to balance a load among those two instances, so we are going to use Fabio, a Consul aware load-balancer, to reach Django app instances.

This example will roughly help us simulate a real-world practical application.

‍

The complete source code for this application is open-sourced and is available on GitHub – pranavcode/consul-demo.

Note: The architecture we are discussing here is not specifically constraint with any of the technologies used to create app or data layers. This example could very well be built using a combination of Ruby on Rails and Postgres, or Node.js and MongoDB, or Laravel and MySQL.

How Does Consul Come into the Picture?

We are deploying both, the app and the data, layers with Docker containers. They are going to be built as services and will talk to each other over HTTP.

Thus, we will use Consul for Service Discovery. This will allow Django servers to find MongoDB Primary node. We are going to use Consul to resolve services via Consul’s DNS interface for this example.

Consul will also help us with the auto-configuration of Fabio as load-balancer to reach instances of our Django app.

We are also using the health-check feature of Consul to monitor the health of each of our instances in the whole infrastructure.

Consul provides a beautiful user interface, as part of its Web UI, out of the box to show all the services on a single dashboard. We will use it to see how our services are laid out.

Let’s begin.

Setup: MongoDB, Django, Consul, Fabio, and Dockerization

We will keep this as simple and minimal as possible to the extent it fulfills our need for a demonstration.

MongoDB

The MongoDB setup we are targeting is in the form of MongoDB Replica Set. One primary node and two secondary nodes.

The primary node will manage all the write operations and the oplog to maintain the sequence of writes, and replicate the data across secondaries. We are also configuring the secondaries for the read operations. You can learn more about MongoDB Replica Set on their official documentation.

We will call our replication set as ‘consuldemo’.

We will run MongoDB on a standard port 27017 and supply the name of the replica set on the command line using the parameter ‘–replSet’.

As you may read from the documentation MongoDB also allows configuring replica set name via configuration file with the parameter for replication as below:

replication:
    replSetName: "consuldemo"

replication:
    replSetName: "consuldemo"

In our case, the replication set configuration that we will apply on one of the MongoDB nodes, once all the nodes are up and running is as given below:

var config = {
    _id: "consuldemo",
    version: 1,
    members: [{
        _id: 0,
        host: "mongo_1:27017",
    }, {
        _id: 1,
        host: "mongo_2:27017",
    }, {
        _id: 2,
        host: "mongo_3:27017",
    }],
    settings: { 
        chainingAllowed: true 
    }
};
rs.initiate(config, { force: true });
rs.slaveOk();
db.getMongo().setReadPref("nearest");

var config = {
    _id: "consuldemo",
    version: 1,
    members: [{
        _id: 0,
        host: "mongo_1:27017",
    }, {
        _id: 1,
        host: "mongo_2:27017",
    }, {
        _id: 2,
        host: "mongo_3:27017",
    }],
    settings: { 
        chainingAllowed: true 
    }
};
rs.initiate(config, { force: true });
rs.slaveOk();
db.getMongo().setReadPref("nearest");

This configuration will be applied to one of the pre-defined nodes and MongoDB will decide which node will be primary and secondary.

Note: We are not forcing the set creation with any pre-defined designations on who becomes primary and secondary to allow the dynamism in service discovery. Normally, the nodes would be defined for a specific role.

We are allowing slave reads and reads from the nearest node as a Read Preference.

We will start MongoDB on all nodes with the following command:

mongod --bind_ip_all --port 27017 --dbpath /data/db --replSet "consuldemo"

mongod --bind_ip_all --port 27017 --dbpath /data/db --replSet "consuldemo"

This gives us a MongoDB Replica Set with one primary instance and two secondary instances, running and ready to accept connections.

We will discuss containerizing the MongoDB service in the latter part of this article.

Django

We will create a simple Django project that represents Blog application and containerizes it with Docker.

Building the Django app from scratch is beyond the scope of this tutorial, we recommend you to refer to Django’s official documentation to get started with Django project. But, we will still go through some important aspects.

As we need our Django app to talk to MongoDB, we will use a MongoDB connector for Django ORM, Djongo. We will set up our Django settings to use Djongo and connect with our MongoDB. Djongo is pretty straightforward in configuration.

For a local MongoDB installation it would only take two lines of code:

...
DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'db',
    }
}
...

...

DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'db',
    }
}

...

In our case, as we will need to access MongoDB over another container, our config would look like this:

...
DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'db',
        'HOST': 'mongodb://mongo-primary.service.consul',
        'PORT': 27017,
    }
}
...
@velotiotech

...

DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'db',
        'HOST': 'mongodb://mongo-primary.service.consul',
        'PORT': 27017,
    }
}

...
@velotiotech

Details:

• ENGINE: The database connector to use for Django ORM.
• NAME: Name of the database.
• HOST: Host address that has MongoDB running on it.
• PORT: Which port is your MongoDB listening for requests.

Djongo internally talks to PyMongo and uses MongoClient for executing queries on Mongo. We can also use other MongoDB connectors available for Django to achieve this, like, for instance, django-mongodb-engine or pymongo directly, based on our needs.

Note: We are currently reading and writing via Django to a single MongoDB host, the primary one, but we can configure Djongo to also talk to secondary hosts for read-only operations. That is not in the scope of our discussion. You can refer to Djongo’s official documentation to achieve exactly this.

Continuing our Django app building process, we need to define our models. As we are building a blog-like application, our models would look like this:

from djongo import models
class Blog(models.Model):
    name = models.CharField(max_length=100)
    tagline = models.TextField()
    class Meta:
        abstract = True
class Entry(models.Model):
    blog = models.EmbeddedModelField(
        model_container=Blog,
    )
    
    headline = models.CharField(max_length=255)

from djongo import models

class Blog(models.Model):
    name = models.CharField(max_length=100)
    tagline = models.TextField()

    class Meta:
        abstract = True

class Entry(models.Model):
    blog = models.EmbeddedModelField(
        model_container=Blog,
    )
    
    headline = models.CharField(max_length=255)

We can run a local MongoDB instance and create migrations for these models. Also, register these models into our Django Admin, like so:

from django.contrib import admin
from.models import Entry
admin.site.register(Entry)

from django.contrib import admin
from.models import Entry

admin.site.register(Entry)

We can play with the Entry model’s CRUD operations via Django Admin for this example.

Also, to realize the Django-MongoDB connectivity we will create a custom View and Template that displays information about MongoDB setup and currently connected MongoDB host.

Our Django views look like this:

from django.shortcuts import render
from pymongo import MongoClient
def home(request):
    client = MongoClient("mongo-primary.service.consul")
    replica_set = client.admin.command('ismaster')
    return render(request, 'home.html', { 
        'mongo_hosts': replica_set['hosts'],
        'mongo_primary_host': replica_set['primary'],
        'mongo_connected_host': replica_set['me'],
        'mongo_is_primary': replica_set['ismaster'],
        'mongo_is_secondary': replica_set['secondary'],
    })

from django.shortcuts import render
from pymongo import MongoClient

def home(request):
    client = MongoClient("mongo-primary.service.consul")
    replica_set = client.admin.command('ismaster')

    return render(request, 'home.html', { 
        'mongo_hosts': replica_set['hosts'],
        'mongo_primary_host': replica_set['primary'],
        'mongo_connected_host': replica_set['me'],
        'mongo_is_primary': replica_set['ismaster'],
        'mongo_is_secondary': replica_set['secondary'],
    })

Our URLs or routes configuration for the app looks like this:

from django.urls import path
from tweetapp import views
urlpatterns = [
    path('', views.home, name='home'),
]

from django.urls import path
from tweetapp import views

urlpatterns = [
    path('', views.home, name='home'),
]

And for the project – the app URLs are included like so:

from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static
urlpatterns = [
    path('admin/', admin.site.urls),
    path('web', include('tweetapp.urls')),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static

urlpatterns = [
    path('admin/', admin.site.urls),
    path('web', include('tweetapp.urls')),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

Our Django template, ‘templates/home.html’ looks like this:

<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
    <link href="https://fonts.googleapis.com/css?family=Armata" rel="stylesheet">
    <title>Django-Mongo-Consul</title>
</head>
<body class="bg-dark text-white p-5" style="font-family: Armata">
    <div class="p-4 border">
        <div class="m-2">
            <b>Django Database Connection</b>
        </div>
        <table class="table table-dark">
            <thead>
                <tr>
                    <th scope="col">#</th>
                    <th scope="col">Property</th>
                    <th scope="col">Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>1</td>
                    <td>Mongo Hosts</td>
                    <td>{% for host in mongo_hosts %}{{ host }}<br/>{% endfor %}</td>
                </tr>
                <tr>
                    <td>2</td>
                    <td>Mongo Primary Address</td>
                    <td>{{ mongo_primary_host }}</td>
                </tr>
                <tr>
                    <td>3</td>
                    <td>Mongo Connected Address</td>
                    <td>{{ mongo_connected_host }}</td>
                </tr>
                <tr>
                    <td>4</td>
                    <td>Mongo - Is Primary?</td>
                    <td>{{ mongo_is_primary }}</td>
                </tr>
                <tr>
                    <td>5</td>
                    <td>Mongo - Is Secondary?</td>
                    <td>{{ mongo_is_secondary }}</td>
                </tr>
            </tbody>
        </table>
    </div>
    
    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
</body>
</html>

<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
    <link href="https://fonts.googleapis.com/css?family=Armata" rel="stylesheet">

    <title>Django-Mongo-Consul</title>
</head>
<body class="bg-dark text-white p-5" style="font-family: Armata">
    <div class="p-4 border">
        <div class="m-2">
            <b>Django Database Connection</b>
        </div>
        <table class="table table-dark">
            <thead>
                <tr>
                    <th scope="col">#</th>
                    <th scope="col">Property</th>
                    <th scope="col">Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>1</td>
                    <td>Mongo Hosts</td>
                    <td>{% for host in mongo_hosts %}{{ host }}<br/>{% endfor %}</td>
                </tr>
                <tr>
                    <td>2</td>
                    <td>Mongo Primary Address</td>
                    <td>{{ mongo_primary_host }}</td>
                </tr>
                <tr>
                    <td>3</td>
                    <td>Mongo Connected Address</td>
                    <td>{{ mongo_connected_host }}</td>
                </tr>
                <tr>
                    <td>4</td>
                    <td>Mongo - Is Primary?</td>
                    <td>{{ mongo_is_primary }}</td>
                </tr>
                <tr>
                    <td>5</td>
                    <td>Mongo - Is Secondary?</td>
                    <td>{{ mongo_is_secondary }}</td>
                </tr>
            </tbody>
        </table>
    </div>
    
    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
</body>
</html>

To run the app we need to migrate the database first using the command below:

python ./manage.py migrate

python ./manage.py migrate

And also collect all the static assets into static directory:

python ./manage.py collectstatic --noinput

python ./manage.py collectstatic --noinput

Now run the Django app with Gunicorn, a WSGI HTTP server, as given below:

gunicorn --bind 0.0.0.0:8000 --access-logfile - tweeter.wsgi:application

gunicorn --bind 0.0.0.0:8000 --access-logfile - tweeter.wsgi:application

This gives us a basic blog-like Django app that connects to MongoDB backend.

We will discuss containerizing this Django web application in the latter part of this article.

Consul

We place a Consul agent on every service as part of our Consul setup.

The Consul agent is responsible for service discovery by registering the service on the Consul cluster and also monitors the health of every service instance.

Consul on nodes running MongoDB Replica Set

We will discuss Consul setup in the context of MongoDB Replica Set first – as it solves an interesting problem. At any given point of time, one of the MongoDB instances can either be a Primary or a Secondary.

The Consul agent registering and monitoring our MongoDB instance within a Replica Set has a unique mechanism – dynamically registering and deregistering MongoDB service as a Primary instance or a Secondary instance based on what Replica Set has designated it.

We achieve this dynamism by writing and running a shell script after an interval that toggles the Consul service definition for MongoDB Primary and MongoDB Secondary on the instance node’s Consul Agent.

The service definitions for MongoDB services are stored as JSON files on the Consul’s config directory ‘/etc/config.d’.

Service definition for MongoDB Primary instance:

{
    "service": {
        "name": "mongo-primary",
        "port": 27017,
        "tags": [
            "nosql",
            "database"
        ],
        "check": {
            "id": "mongo_primary_status",
            "name": "Mongo Primary Status",
            "args": ["/etc/consul.d/check_scripts/mongo_primary_check.sh"],
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

{
    "service": {
        "name": "mongo-primary",
        "port": 27017,
        "tags": [
            "nosql",
            "database"
        ],
        "check": {
            "id": "mongo_primary_status",
            "name": "Mongo Primary Status",
            "args": ["/etc/consul.d/check_scripts/mongo_primary_check.sh"],
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

If you look closely, the service definition allows us to get a DNS entry specific to MongoDB Primary, rather than a generic MongoDB instance. This allows us to send the database writes to a specific MongoDB instance. In the case of Replica Set, the writes are maintained by MongoDB Primary.

Thus, we are able to achieve both service discovery as well as health monitoring for Primary instance of MongoDB.

Similarly, with a slight change the service definition for MongoDB Secondary instance goes like this:

{
    "service": {
        "name": "mongo-secondary",
        "port": 27017,
        "tags": [
            "nosql",
            "database"
        ],
        "check": {
            "id": "mongo_secondary_status",
            "name": "Mongo Secondary Status",
            "args": ["/etc/consul.d/check_scripts/mongo_secondary_check.sh"],
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

{
    "service": {
        "name": "mongo-secondary",
        "port": 27017,
        "tags": [
            "nosql",
            "database"
        ],
        "check": {
            "id": "mongo_secondary_status",
            "name": "Mongo Secondary Status",
            "args": ["/etc/consul.d/check_scripts/mongo_secondary_check.sh"],
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

Given all this context, can you think of the way we can dynamically switch these service definitions?

We can identify if the given MongoDB instance is primary or not by running command `db.isMaster()` on MongoDB shell.

The check can we drafted as a shell script as:

#!/bin/bash
mongo_primary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .ismaster 2> /dev/null)
if [[ $mongo_primary == false ]]; then
    exit 1
fi
echo "Mongo primary healthy and reachable"

#!/bin/bash

mongo_primary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .ismaster 2> /dev/null)
if [[ $mongo_primary == false ]]; then
    exit 1
fi

echo "Mongo primary healthy and reachable"

Similarly, the non-master or non-primary instances of MongoDB can also be checked against the same command, by checking a `secondary` value:

#!/bin/bash
mongo_secondary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .secondary 2> /dev/null)
if [[ $mongo_secondary == false ]]; then
    exit 1
fi
echo "Mongo secondary healthy and reachable"

#!/bin/bash

mongo_secondary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .secondary 2> /dev/null)
if [[ $mongo_secondary == false ]]; then
    exit 1
fi

echo "Mongo secondary healthy and reachable"

Note: We are using jq – a lightweight and flexible command-line JSON processor – to process the JSON encoded output of MongoDB shell commands.

One way of writing a script that does this dynamic switch looks like this:

#!/bin/bash
# Wait until Mongo starts
while [[ $(ps aux | grep [m]ongod | wc -l) -ne 1 ]]; do
    sleep 5
done
REGISTER_MASTER=0
REGISTER_SECONDARY=0
mongo_primary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .ismaster 2> /dev/null)
mongo_secondary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .secondary 2> /dev/null)  
if [[ $mongo_primary == false && $mongo_secondary == true ]]; then
  # Deregister as Mongo Master
  if [[ -a /etc/consul.d/check_scripts/mongo_primary_check.sh && -a /etc/consul.d/mongo_primary.json ]]; then
    rm -f /etc/consul.d/check_scripts/mongo_primary_check.sh
    rm -f /etc/consul.d/mongo_primary.json
    REGISTER_MASTER=1
  fi
  # Register as Mongo Secondary
  if [[ ! -a /etc/consul.d/check_scripts/mongo_secondary_check.sh && ! -a /etc/consul.d/mongo_secondary.json ]]; then
    cp -u /opt/checks/check_scripts/mongo_secondary_check.sh /etc/consul.d/check_scripts/
    cp -u /opt/checks/mongo_secondary.json /etc/consul.d/
    REGISTER_SECONDARY=1
  fi
else
  # Register as Mongo Master
  if [[ ! -a /etc/consul.d/check_scripts/mongo_primary_check.sh && ! -a /etc/consul.d/mongo_primary.json ]]; then
    cp -u /opt/checks/check_scripts/mongo_primary_check.sh /etc/consul.d/check_scripts/
    cp -u /opt/checks/mongo_primary.json /etc/consul.d/
    REGISTER_MASTER=2
  fi
  # Deregister as Mongo Secondary
  if [[ -a /etc/consul.d/check_scripts/mongo_secondary_check.sh && -a /etc/consul.d/mongo_secondary.json ]]; then
    rm -f /etc/consul.d/check_scripts/mongo_secondary_check.sh
    rm -f /etc/consul.d/mongo_secondary.json
    REGISTER_SECONDARY=2
  fi
fi
if [[ $REGISTER_MASTER -ne 0 && $REGISTER_SECONDARY -ne 0 ]]; then
  consul reload
fi

#!/bin/bash

# Wait until Mongo starts
while [[ $(ps aux | grep [m]ongod | wc -l) -ne 1 ]]; do
    sleep 5
done

REGISTER_MASTER=0
REGISTER_SECONDARY=0

mongo_primary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .ismaster 2> /dev/null)
mongo_secondary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .secondary 2> /dev/null)  

if [[ $mongo_primary == false && $mongo_secondary == true ]]; then

  # Deregister as Mongo Master
  if [[ -a /etc/consul.d/check_scripts/mongo_primary_check.sh && -a /etc/consul.d/mongo_primary.json ]]; then
    rm -f /etc/consul.d/check_scripts/mongo_primary_check.sh
    rm -f /etc/consul.d/mongo_primary.json

    REGISTER_MASTER=1
  fi

  # Register as Mongo Secondary
  if [[ ! -a /etc/consul.d/check_scripts/mongo_secondary_check.sh && ! -a /etc/consul.d/mongo_secondary.json ]]; then
    cp -u /opt/checks/check_scripts/mongo_secondary_check.sh /etc/consul.d/check_scripts/
    cp -u /opt/checks/mongo_secondary.json /etc/consul.d/

    REGISTER_SECONDARY=1
  fi

else

  # Register as Mongo Master
  if [[ ! -a /etc/consul.d/check_scripts/mongo_primary_check.sh && ! -a /etc/consul.d/mongo_primary.json ]]; then
    cp -u /opt/checks/check_scripts/mongo_primary_check.sh /etc/consul.d/check_scripts/
    cp -u /opt/checks/mongo_primary.json /etc/consul.d/

    REGISTER_MASTER=2
  fi

  # Deregister as Mongo Secondary
  if [[ -a /etc/consul.d/check_scripts/mongo_secondary_check.sh && -a /etc/consul.d/mongo_secondary.json ]]; then
    rm -f /etc/consul.d/check_scripts/mongo_secondary_check.sh
    rm -f /etc/consul.d/mongo_secondary.json

    REGISTER_SECONDARY=2
  fi

fi

if [[ $REGISTER_MASTER -ne 0 && $REGISTER_SECONDARY -ne 0 ]]; then
  consul reload
fi

Note: This is an example script, but we can be more creative and optimize the script further.

Once we are done with our service definitions we can run the Consul agent on each MongoDB nodes. To run a agent we will use the following command:

consul agent -bind 33.10.0.3 
    -advertise 33.10.0.3 
    -join consul_server 
    -node mongo_1 
    -dns-port 53 
    -data-dir /data 
    -config-dir /etc/consul.d 
    -enable-local-script-checks

consul agent -bind 33.10.0.3 
    -advertise 33.10.0.3 
    -join consul_server 
    -node mongo_1 
    -dns-port 53 
    -data-dir /data 
    -config-dir /etc/consul.d 
    -enable-local-script-checks

Here,  ‘consul_server’ represents the Consul Server running host. Similarly, we can run such agents on each of the other MongoDB instance nodes.

Note: If we have multiple MongoDB instances running on the same host, the service definition would change to reflect the different ports used by each instance to uniquely identify, discover and monitor individual MongoDB instance.

Consul on nodes running Django App

For the Django application, Consul setup will be very simple. We only need to monitor Django app’s port on which Gunicorn is listening for requests.

The Consul service definition would look like this:

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

Once we have the Consul service definition for Django app in place, we can run the Consul agent sitting on the node Django app is running as a service. To run the Consul agent we would fire the following command:

consul agent -bind 33.10.0.10 
    -advertise 33.10.0.10 
    -join consul_server 
    -node web_1 
    -dns-port 53 
    -data-dir /data 
    -config-dir /etc/consul.d 
    -enable-local-script-checks

consul agent -bind 33.10.0.10 
    -advertise 33.10.0.10 
    -join consul_server 
    -node web_1 
    -dns-port 53 
    -data-dir /data 
    -config-dir /etc/consul.d 
    -enable-local-script-checks

Consul Server

We are running the Consul cluster with a dedicated Consul server node. The Consul server node can easily host, discover and monitor services running on it, exactly the same way as we did in the above sections for MongoDB and Django app.

To run Consul in server mode and allow agents to connect to it, we will fire the following command on the node that we want to run our Consul server:

consul agent -server 
    -bind 33.10.0.2 
    -advertise 33.10.0.2 
    -node consul_server 
    -client 0.0.0.0 
    -dns-port 53 
    -data-dir /data 
    -ui -bootstrap

consul agent -server 
    -bind 33.10.0.2 
    -advertise 33.10.0.2 
    -node consul_server 
    -client 0.0.0.0 
    -dns-port 53 
    -data-dir /data 
    -ui -bootstrap

There are no services on our Consul server node for now, so there are no service definitions associated with this Consul agent configuration.

Fabio

We are using the power of Fabio to be auto-configurable and being Consul-aware.

This makes our task of load-balancing the traffic to our Django app instances very easy.

To allow Fabio to auto-detect the services via Consul, one of the ways is to add a tag or update a tag in the service definition with a prefix and a service identifier `urlprefix-/<service>`. Our Consul’s service definition for Django app would now look like this:</service>

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

In our case, the Django app or service is the only service that will need load-balancing, thus this Consul service definition change completes the requirement on Fabio setup.

Dockerization

Our whole app is going to be deployed as a set of Docker containers. Let’s talk about how we are achieving it in the context of Consul.

Dockerizing MongoDB Replica Set along with Consul Agent

We need to run a Consul agent as described above alongside MongoDB on the same Docker container, so we will need to run a custom ENTRYPOINT on the container to allow running two processes.

Note: This can also be achieved using Docker container level checks in Consul. So, you will be free to run a Consul agent on the host and check across service running in Docker container. Which, will essentially exec into the container to monitor the service.

To achieve this we will use a tool similar to Foreman. It is a lifecycle management tool for physical and virtual servers – including provisioning, monitoring and configuring.

To be precise, we will use the Golang adoption of Foreman, Goreman. It takes the configuration in the form of Heroku’sProcfile to maintain which processes to be kept alive on the host.

In our case, the Procfile looks like this:

# Mongo
mongo: /opt/mongo.sh
# Consul Client Agent
consul: /opt/consul.sh
# Consul Client Health Checks
consul_check: while true; do /opt/checks_toggler.sh && sleep 10; done

# Mongo
mongo: /opt/mongo.sh

# Consul Client Agent
consul: /opt/consul.sh

# Consul Client Health Checks
consul_check: while true; do /opt/checks_toggler.sh && sleep 10; done

The `consul_check` at the end of the Profile maintains the dynamism between both Primary and Secondary MongoDB node checks, based on who is voted for which role within MongoDB Replica Set.

The shell scripts that are executed by the respective keys on the Procfile are as defined previously in this discussion.

Our Dockerfile, with some additional tools for debug and diagnostics, would look like:

FROM ubuntu:18.04
RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping
# Install MongoDB
RUN apt-get install -y mongodb
RUN mkdir -p /data/db
VOLUME data:/data/db
# Setup Consul and Goreman
ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip
ADD https://github.com/mattn/goreman/releases/download/v0.0.10/goreman_linux_amd64.zip /tmp/goreman.zip
RUN cd /bin && unzip /tmp/goreman.zip && chmod +x /bin/goreman && rm /tmp/goreman.zip
RUN mkdir -p /etc/consul.d/check_scripts
ADD ./config/mongod /etc
RUN mkdir -p /etc/checks
ADD ./config/checks /opt/checks
ADD checks_toggler.sh /opt
ADD mongo.sh /opt
ADD consul.sh /opt
ADD Procfile /root/Procfile
EXPOSE 27017
# Launch both MongoDB server and Consul
ENTRYPOINT [ "goreman" ]
CMD [ "-f", "/root/Procfile", "start" ]

FROM ubuntu:18.04

RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping

# Install MongoDB
RUN apt-get install -y mongodb

RUN mkdir -p /data/db
VOLUME data:/data/db

# Setup Consul and Goreman
ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip

ADD https://github.com/mattn/goreman/releases/download/v0.0.10/goreman_linux_amd64.zip /tmp/goreman.zip
RUN cd /bin && unzip /tmp/goreman.zip && chmod +x /bin/goreman && rm /tmp/goreman.zip

RUN mkdir -p /etc/consul.d/check_scripts
ADD ./config/mongod /etc

RUN mkdir -p /etc/checks
ADD ./config/checks /opt/checks

ADD checks_toggler.sh /opt
ADD mongo.sh /opt
ADD consul.sh /opt

ADD Procfile /root/Procfile

EXPOSE 27017

# Launch both MongoDB server and Consul
ENTRYPOINT [ "goreman" ]
CMD [ "-f", "/root/Procfile", "start" ]

Note: We have used bare Ubuntu 18.04 image here for our purposes, but you can use official MongoDB image and adapt it to run Consul alongside MongoDB or even do Consul checks on Docker container level as mentioned in the official documentation.

Dockerizing Django Web Application along with Consul Agent

We also need to run a Consul agent alongside our Django App on the same Docker container as we had with MongoDB container.

# Django
django: /web/tweeter.sh
# Consul Client Agent
consul: /opt/consul.sh

# Django
django: /web/tweeter.sh

# Consul Client Agent
consul: /opt/consul.sh

Similarly, we will have the Dockerfile for Django Web Application as we had for our MongoDB containers.

FROM python:3.7
RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping
# Python Environment Setup
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Setup Consul and Goreman
RUN mkdir -p /data/db /etc/consul.d
ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip
ADD https://github.com/mattn/goreman/releases/download/v0.0.10/goreman_linux_amd64.zip /tmp/goreman.zip
RUN cd /bin && unzip /tmp/goreman.zip && chmod +x /bin/goreman && rm /tmp/goreman.zip
ADD ./consul /etc/consul.d
ADD Procfile /root/Procfile
# Install pipenv
RUN pip3 install --upgrade pip
RUN pip3 install pipenv
# Setting workdir
ADD consul.sh /opt
ADD . /web
WORKDIR /web/tweeter
# Exposing appropriate ports
EXPOSE 8000/tcp
# Install dependencies
RUN pipenv install --system --deploy --ignore-pipfile
# Migrates the database, uploads staticfiles, run API server and background tasks
ENTRYPOINT [ "goreman" ]
CMD [ "-f", "/root/Procfile", "start" ]

FROM python:3.7

RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping

# Python Environment Setup
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Setup Consul and Goreman
RUN mkdir -p /data/db /etc/consul.d

ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip

ADD https://github.com/mattn/goreman/releases/download/v0.0.10/goreman_linux_amd64.zip /tmp/goreman.zip
RUN cd /bin && unzip /tmp/goreman.zip && chmod +x /bin/goreman && rm /tmp/goreman.zip

ADD ./consul /etc/consul.d
ADD Procfile /root/Procfile

# Install pipenv
RUN pip3 install --upgrade pip
RUN pip3 install pipenv

# Setting workdir
ADD consul.sh /opt
ADD . /web
WORKDIR /web/tweeter

# Exposing appropriate ports
EXPOSE 8000/tcp

# Install dependencies
RUN pipenv install --system --deploy --ignore-pipfile

# Migrates the database, uploads staticfiles, run API server and background tasks
ENTRYPOINT [ "goreman" ]
CMD [ "-f", "/root/Procfile", "start" ]

Dockerizing Consul Server

We are maintaining the same flow with Consul server node to run it with custom ENTRYPOINT. It is not a requirement, but we are maintaining a consistent view of different Consul run files.

Also, we are using Ubuntu 18.04 image for the demonstration. You can very well use Consul’s official image for this, that accepts all the custom parameters as are mentioned here.

FROM ubuntu:18.04
RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping
ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip
# Consul ports
EXPOSE 8300 8301 8302 8400 8500
ADD consul_server.sh /opt
RUN mkdir -p /data
VOLUME /data
CMD ["/opt/consul_server.sh"]

FROM ubuntu:18.04

RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping

ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip

# Consul ports
EXPOSE 8300 8301 8302 8400 8500

ADD consul_server.sh /opt
RUN mkdir -p /data
VOLUME /data

CMD ["/opt/consul_server.sh"]

Docker Compose

We are using Compose to run all our Docker containers in a desired, repeatable form.

Our Compose file is written to denote all the aspects that we mentioned above and utilize the power of Docker Compose tool to achieve those in a seamless fashion.

Docker Compose file would look like the one given below:

version: "3.6"
services:
  consul_server:
    build:
      context: consul_server
      dockerfile: Dockerfile
    image: consul_server
    ports:
      - 8300:8300
      - 8301:8301
      - 8302:8302
      - 8400:8400
      - 8500:8500
    environment:
      - NODE=consul_server
      - PRIVATE_IP_ADDRESS=33.10.0.2
    networks:
      consul_network:
        ipv4_address: 33.10.0.2
  load_balancer:
    image: fabiolb/fabio
    ports:
      - 9998:9998
      - 9999:9999
    command: -registry.consul.addr="33.10.0.2:8500"
    networks:
      consul_network:
        ipv4_address: 33.10.0.100
  mongo_1:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_1
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.3
    restart: always
    ports:
      - 27017:27017
      - 28017:28017
    depends_on:
      - consul_server
      - mongo_2
      - mongo_3
    networks:
      consul_network:
        ipv4_address: 33.10.0.3
  mongo_2:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_2
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.4
    restart: always
    ports:
      - 27018:27017
      - 28018:28017
    depends_on:
      - consul_server
    networks:
      consul_network:
        ipv4_address: 33.10.0.4
  mongo_3:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_3
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.5
    restart: always
    ports:
      - 27019:27017
      - 28019:28017
    depends_on:
      - consul_server
    networks:
      consul_network:
        ipv4_address: 33.10.0.5
  web_1:
    build:
      context: django
      dockerfile: Dockerfile
    image: web_consul
    ports:
      - 8080:8000
    environment:
      - NODE=web_1
      - PRIMARY=1
      - LOAD_BALANCER=33.10.0.100
      - PRIVATE_IP_ADDRESS=33.10.0.10
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    depends_on:
      - consul_server
      - mongo_1
    volumes:
      - ./django:/web
    cap_add:
      - NET_ADMIN
    networks:
      consul_network:
        ipv4_address: 33.10.0.10
  web_2:
    build:
      context: django
      dockerfile: Dockerfile
    image: web_consul
    ports:
      - 8081:8000
    environment:
      - NODE=web_2
      - LOAD_BALANCER=33.10.0.100
      - PRIVATE_IP_ADDRESS=33.10.0.11
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    depends_on:
      - consul_server
      - mongo_1
    volumes:
      - ./django:/web
    cap_add:
      - NET_ADMIN
    networks:
      consul_network:
        ipv4_address: 33.10.0.11
networks:
  consul_network:
    driver: bridge
    ipam:
     config:
       - subnet: 33.10.0.0/16

version: "3.6"

services:

  consul_server:
    build:
      context: consul_server
      dockerfile: Dockerfile
    image: consul_server
    ports:
      - 8300:8300
      - 8301:8301
      - 8302:8302
      - 8400:8400
      - 8500:8500
    environment:
      - NODE=consul_server
      - PRIVATE_IP_ADDRESS=33.10.0.2
    networks:
      consul_network:
        ipv4_address: 33.10.0.2

  load_balancer:
    image: fabiolb/fabio
    ports:
      - 9998:9998
      - 9999:9999
    command: -registry.consul.addr="33.10.0.2:8500"
    networks:
      consul_network:
        ipv4_address: 33.10.0.100

  mongo_1:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_1
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.3
    restart: always
    ports:
      - 27017:27017
      - 28017:28017
    depends_on:
      - consul_server
      - mongo_2
      - mongo_3
    networks:
      consul_network:
        ipv4_address: 33.10.0.3

  mongo_2:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_2
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.4
    restart: always
    ports:
      - 27018:27017
      - 28018:28017
    depends_on:
      - consul_server
    networks:
      consul_network:
        ipv4_address: 33.10.0.4

  mongo_3:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_3
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.5
    restart: always
    ports:
      - 27019:27017
      - 28019:28017
    depends_on:
      - consul_server
    networks:
      consul_network:
        ipv4_address: 33.10.0.5

  web_1:
    build:
      context: django
      dockerfile: Dockerfile
    image: web_consul
    ports:
      - 8080:8000
    environment:
      - NODE=web_1
      - PRIMARY=1
      - LOAD_BALANCER=33.10.0.100
      - PRIVATE_IP_ADDRESS=33.10.0.10
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    depends_on:
      - consul_server
      - mongo_1
    volumes:
      - ./django:/web
    cap_add:
      - NET_ADMIN
    networks:
      consul_network:
        ipv4_address: 33.10.0.10

  web_2:
    build:
      context: django
      dockerfile: Dockerfile
    image: web_consul
    ports:
      - 8081:8000
    environment:
      - NODE=web_2
      - LOAD_BALANCER=33.10.0.100
      - PRIVATE_IP_ADDRESS=33.10.0.11
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    depends_on:
      - consul_server
      - mongo_1
    volumes:
      - ./django:/web
    cap_add:
      - NET_ADMIN
    networks:
      consul_network:
        ipv4_address: 33.10.0.11

networks:
  consul_network:
    driver: bridge
    ipam:
     config:
       - subnet: 33.10.0.0/16

That brings us to the end of the whole environment setup. We can now run Docker Compose to build and run the containers.

Service Discovery using Consul

When all the services are up and running the Consul Web UI gives us a nice glance at our overall setup.

‍

The MongoDB service is available for Django app to discover by virtue of Consul’s DNS interface.

root@82857c424b15:/web/tweeter# dig @127.0.0.1 mongo-primary.service.consul
; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 mongo-primary.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8369
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mongo-primary.service.consul.	IN	A
;; ANSWER SECTION:
mongo-primary.service.consul. 0	IN	A	33.10.0.3
;; ADDITIONAL SECTION:
mongo-primary.service.consul. 0	IN	TXT	"consul-network-segment="
;; Query time: 139 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Apr 01 11:50:45 UTC 2019
;; MSG SIZE  rcvd: 109

root@82857c424b15:/web/tweeter# dig @127.0.0.1 mongo-primary.service.consul

; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 mongo-primary.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8369
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mongo-primary.service.consul.	IN	A

;; ANSWER SECTION:
mongo-primary.service.consul. 0	IN	A	33.10.0.3

;; ADDITIONAL SECTION:
mongo-primary.service.consul. 0	IN	TXT	"consul-network-segment="

;; Query time: 139 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Apr 01 11:50:45 UTC 2019
;; MSG SIZE  rcvd: 109

Django App can now connect MongoDB Primary instance and start writing data to it.

We can use Fabio load-balancer to connect to Django App instance by auto-discovering it via Consul registry using specialized service tags and render the page with all the database connection information we are talking about.

Our load-balancer is sitting at ‘33.10.0.100’ and ‘/web’ is configured to be routed to one of our Django application instances running behind the load-balancer.

As you can see from the auto-detection and configuration of Fabio load-balancer from its UI above, it has weighted the Django Web Application end-points equally. This will help balance the request or traffic load on the Django application instances.

When we visit our Fabio URL ‘33.10.0.100:9999’ and use the source route as ‘/web’ we are routed to one of the Django instances. So, visiting ‘33.10.0.100:9999/web’ gives us following output.

We are able to restrict Fabio to only load-balance Django app instances by only adding required tags to Consul’s service definitions of Django app services.

This MongoDB Primary instance discovery helps Django app to do database migration and app deployment.

One can explore Consul Web UI to see all the instances of Django web application services.

‍

Similarly, see how MongoDB Replica Set instances are laid out.

Let’s see how Consul helps with health-checking services and discovering only the alive services.

We will stop the current MongoDB Replica Set Primary (‘mongo_2’) container, to see what happens.

Consul has started failing the health-check for previous MongoDB Primary service. MongoDB Replica Set has also detected that the node is down and the re-election of Primary node needs to be done. Thus, getting us a new MongoDB Primary (‘mongo_3’) automatically.

Our checks toggle has kicked-in and swapped the check on ‘mongo_3’ from MongoDB Secondary check to MongoDB Primary check.

When we take a look at the view from the Django app, we see it is now connected to a new MongoDB Primary service (‘mongo_3’).

Let’s see how this plays out when we bring back the stopped MongoDB instance.

Similarly, if we stop the service instances of Django application, Fabio would now be able to detect only a healthy instance and would only route the traffic to that instance.

This is how one can use Consul’s service discovery capability to discover, monitor and health-check services.

Service Configuration using Consul

Currently, we are configuring Django application instances directly either from environment variables set within the containers by Docker Compose and consuming them in Django project settings or by hard-coding the configuration parameters directly.

We can use Consul’s Key/Value store to share configuration across both the instances of Django app.

We can use Consul’s HTTP interface to store key/value pair and retrieve them within the app using the open-source Python client for Consul, called python-consul. You may also use any other Python library that can interact with Consul’s KV store if you want.

Let’s begin by looking at how we can set a key/value pair in Consul using its HTTP interface.

# Flag to run Django app in debug mode
curl -X PUT -d 'True' consul_server:8500/v1/kv/web/debug

# Dynamic entries into Django app configuration 
# to denote allowed set of hosts
curl -X PUT -d 'localhost, 33.10.0.100' consul_server:8500/v1/kv/web/allowed_hosts

# Dynamic entries into Django app configuration
# to denote installed apps
curl -X PUT -d 'tweetapp' consul_server:8500/v1/kv/web/installed_apps

# Flag to run Django app in debug mode
curl -X PUT -d 'True' consul_server:8500/v1/kv/web/debug

# Dynamic entries into Django app configuration 
# to denote allowed set of hosts
curl -X PUT -d 'localhost, 33.10.0.100' consul_server:8500/v1/kv/web/allowed_hosts

# Dynamic entries into Django app configuration
# to denote installed apps
curl -X PUT -d 'tweetapp' consul_server:8500/v1/kv/web/installed_apps

Once we set the KV store we can consume it on Django app instances to configure it with these values.

Let’s install python-consul and add it as a project dependency.

$ pipenv shell
Launching subshell in virtual environment…
 . /home/pranav/.local/share/virtualenvs/tweeter-PYSn2zRU/bin/activate

$  . /home/pranav/.local/share/virtualenvs/tweeter-PYSn2zRU/bin/activate

(tweeter) $ pipenv install python-consul
Installing python-consul…
Adding python-consul to Pipfile's [packages]…
✔ Installation Succeeded 
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (9590cc)!
Installing dependencies from Pipfile.lock (9590cc)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 14/14 — 00:00:20

$ pipenv shell
Launching subshell in virtual environment…
 . /home/pranav/.local/share/virtualenvs/tweeter-PYSn2zRU/bin/activate

$  . /home/pranav/.local/share/virtualenvs/tweeter-PYSn2zRU/bin/activate

(tweeter) $ pipenv install python-consul
Installing python-consul…
Adding python-consul to Pipfile's [packages]…
✔ Installation Succeeded 
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (9590cc)!
Installing dependencies from Pipfile.lock (9590cc)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 14/14 — 00:00:20

We will need to connect our app to Consul using python-consul.

import consul

consul_client = consul.Consul(
    host='consul_server',
    port=8500,
)

import consul

consul_client = consul.Consul(
    host='consul_server',
    port=8500,
)

We can capture and configure our Django app accordingly using the ‘python-consul’ library.

# Set DEBUG flag using Consul KV store
index, data = consul_client.kv.get('web/debug')
DEBUG = data.get('Value', True)

# Set ALLOWED_HOSTS dynamically using Consul KV store
ALLOWED_HOSTS = []

index, hosts = consul_client.kv.get('web/allowed_hosts')
ALLOWED_HOSTS.append(hosts.get('Value'))

# Set INSTALLED_APPS dynamically using Consul KV store
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

index, apps = consul_client.kv.get('web/installed_apps')
INSTALLED_APPS += (bytes(apps.get('Value')).decode('utf-8'),)

# Set DEBUG flag using Consul KV store
index, data = consul_client.kv.get('web/debug')
DEBUG = data.get('Value', True)

# Set ALLOWED_HOSTS dynamically using Consul KV store
ALLOWED_HOSTS = []

index, hosts = consul_client.kv.get('web/allowed_hosts')
ALLOWED_HOSTS.append(hosts.get('Value'))

# Set INSTALLED_APPS dynamically using Consul KV store
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

index, apps = consul_client.kv.get('web/installed_apps')
INSTALLED_APPS += (bytes(apps.get('Value')).decode('utf-8'),)

These key/value pair from Consul’s KV store can also be viewed and updated from its Web UI.

The code used as part of this guide for Consul’s service configuration section is available on ‘service-configuration’ branch of pranavcode/consul-demo project.

That is how one can use Consul’s KV store and configure individual services in their architecture with ease.

Service Segmentation using Consul

As part of Consul’s Service Segmentation we are going to look at Consul Connect intentions and data center distribution.

Connect provides service-to-service connection authorization and encryption using mutual TLS.

To use Consul you need to enable it in the server configuration. Connect needs to be enabled across the Consul cluster for proper functioning of the cluster.

{
    "connect": {
        "enabled": true
    }
}

{
    "connect": {
        "enabled": true
    }
}

In our context, we can define that the communication is to be TLS identified and secured we will define an upstream sidecar service with a proxy on Django app for its communication with MongoDB Primary instance.

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "connect": {
            "sidecar_service": {
                "proxy": {
                    "upstreams": [{
                        "destination_name": "mongo-primary",
                        "local_bind_port": 5501
                    }]
                }
            }
        },
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "connect": {
            "sidecar_service": {
                "proxy": {
                    "upstreams": [{
                        "destination_name": "mongo-primary",
                        "local_bind_port": 5501
                    }]
                }
            }
        },
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

Along with Connect configuration of sidecar proxy, we will also need to run the Connect proxy for Django app as well. This could be achieved by running the following command.

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "connect": {
            "sidecar_service": {
                "proxy": {
                    "upstreams": [{
                        "destination_name": "mongo-primary",
                        "local_bind_port": 5501
                    }]
                }
            }
        },
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

{
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "connect": {
            "sidecar_service": {
                "proxy": {
                    "upstreams": [{
                        "destination_name": "mongo-primary",
                        "local_bind_port": 5501
                    }]
                }
            }
        },
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

We can add Consul Connect Intentions to create a service graph across all the services and define traffic pattern. We can create intentions as shown below:

$ consul connect proxy -sidecar-for web

$ consul connect proxy -sidecar-for web

Intentions for service graph can also be managed from Consul Web UI.

This defines the service connection restrictions to allow or deny them to talk via Connect.

We have also added ability on Consul agents to denote which datacenters they belong to and be accessible via one or more Consul servers in a given datacenter.

The code used as part of this guide for Consul’s service segmentation section is available on ‘service-segmentation’ branch of velotiotech/consul-demo project.

That is how one can use Consul’s service segmentation feature and configure service level connection access control.

Conclusion

Having an ability to seamlessly control the service mesh that Consul provides makes the life of an operator very easy. We hope you have learnt how Consul can be used for service discovery, configuration, and segmentation with its practical implementation.

As usual, we hope it was an informative ride on the journey of Consul. This was the final piece of this two part series. This part tries to cover most of the aspects of Consul architecture and how it fits into your current project. In case you miss the first part, find it here.

We will continue our endeavors with different technologies and get you the most valuable information that we possibly can in every interaction. Let’s us know what you would like to hear from us more or if you have any questions around the topic, we will be more than happy to answer those.

References

• Consul Demo that complements this guide
• HashiCorp Consul and its repo on GitHub
• HashiCorp Consul Guides and Code

This is part 1 of 2 part series on A Practical Guide to HashiCorp Consul. This part is primarily focused on understanding the problems that Consul solves and how it solves them. The second part is more focused on a practical application of Consul in a real-life example. Let’s get started.

How about setting up discoverable, configurable, and secure service mesh using a single tool?

What if we tell you this tool is platform-agnostic and cloud-ready ?

And comes as a single binary download.

All this is true. The tool we are talking about is HashiCorp Consul.

Consul provides service discovery, health checks, load balancing, service graph, identity enforcement via TLS, and distributed service configuration management.

Let’s learn about Consul in details below and see how it solves these complex challenges and makes the life of a distributed system operator easy.

Introduction

Microservices and other distributed systems can enable faster, simpler software development. But there’s a trade-off resulting in greater operational complexity around inter-service communication, configuration management, and network segmentation.

HashiCorp Consul is an open source tool that solves these new complexities by providing service discovery, health checks, load balancing, a service graph, mutual TLS identity enforcement, and a configuration key-value store. These Consul features make it an ideal control plane for a service mesh.

HashiCorp announced Consul in April 2014 and it has since then got a good community acceptance.

This guide is aimed at discussing some of these crucial problems and exploring the various solutions provided by HashiCorp Consul to tackle these problems.

Let’s rundown through the topics that we are going to cover in this guide. The topics are written to be self-content. You can jump directly to a specific topic if you want to.

Brief Background on Monolithic vs. Service-oriented Architectures (SOA)

Looking at traditional architectures of application delivery, what we find is a classic monolith. When we talk about monolith, we have a single application deployment.

Even if it is a single application, typically it has multiple different sub-components.

One of the examples that HashiCorp’s CTO Armon Dadgar gave during his introductory video for Consul was about – delivering desktop banking application. It has a discrete set of sub-components – for example, authentication (say subsystem A), account management (subsystem B), fund transfer (subsystem C), and foreign exchange (subsystem D).

Now, although these are independent functions – system A authentication vs system C fund transfer – we deploy it as a single, monolith app.

Over the last few years, we have seen a trend away from this kind of architectures. There are several reasons for this shift.

Challenge with monolith is: Suppose there is a bug in one of the subsystems, system A, related to authentication.

We can’t just fix it in system A and update it in production.

We have to update system A and do a redeploy of the whole application, which we need deployment of subsystems B, C, and D as well.

This whole redeployment is not ideal. Instead, we would like to do a deployment of individual services.

The same monolithic app delivered as a set of individual, discrete services.

So, if there is a bug fix in one of our services:

and we fix that bug:

We can do the redeployment of that service without coordinating the deployment with other services. What we are essentially talking about is one form of microservices.

This gives a big boost to our development agility. We don’t need to coordinate our development efforts across different development teams or even systems. We will have freedom of developing and deploying independently. One service on a weekly basis and other on quarterly. This is going to be a big advantage to the development teams.

But, as you know, there is ever such a thing as a “free” lunch.

The development efficiency we have gained introduces its own set of operational challenges. Let’s look at some of those.

Service discovery in a monolith, its challenges in a distributed system, and Consul’s solution

Monolithic applications

Assuming two services in a single application want to talk to one another. One way is to expose a method, make it public and allow other services to call it. In a monolithic application, it is a single app, and the services would expose public functions and it would simply mean function calls across services.

As this is a function call within a process, it has happened in-memory. Thus, it’s fast, and we need not worry about how our data was moved and if it was secure or not.

Distributed Systems

In the distributed world, service A is no longer delivered as the same application as service B. So, how does service A finds service B if it wants to talk to B?

Service A might not even be on the same machine as service B. So, there is a network in play. And it is not as fast and there is a latency that we can measure on the lines of milliseconds, as compared to nanoseconds of a simple function call.

Challenges

As we already know by now, two services on a distributed system have to discover one-another to interact. One of the traditional ways of solving this is by using load balancers.

Load balancers would sit in front of each service with a static IP known to all other services.

This gives an ability to add multiple instances of the same service behind the load balancer and it would direct the traffic accordingly. But this load balancer IP is static and hard-coded within all other services, so services can skip discovery.

The challenge is now to maintain a set of load balancers for each individual services. And we can safely assume, there was originally a load balancer for the whole application as well. The cost and effort for maintaining these load balancers have increased.

With load balancers in front of the services, they are a single point of failures. Even when we have multiple instances of service behind the load balancer if it is down our service is down. No matter how many instances of that service are running.

Load balancers also increase the latency of inter-service communication. If service A wish to talk to service B, request from A will have to first talk to the load balancer of service B and then reach B. The response from B will also have to go through the same drill.

And by nature, load balancers are manually managed in most cases. If we add another instance of service, it will not be readily available. We will need to register that service into the load balancer to make it accessible to the world. This would mean manual effort and time.

Consul’s Solutions

Consul’s solution to service discovery problem in distributed systems is a central service registry.

Consul maintains a central registry which contains the entry for all the upstream services. When a service instance starts, it is registered on the central registry. The registry is populated with all the upstream instances of the service.

When a service A wants to talk to service B, it will discover and communicate with B by querying the registry about the upstream service instances of B. So, instead of talking to a load balancer, the service can directly talk to the desired destination service instance.

Consul also provides health-checks on these service instances. If one of the service instances or service itself is unhealthy or fails its health-check, the registry would then know about this scenario and would avoid returning the service’s address. The work that load-balancer would do is handled by the registry in this case.

Also, if there are multiple instances of the same service, Consul would send the traffic randomly to different instances. Thus, leveling the load among different instances.

Consul has handled our challenges of failure detection and load distribution across multiple instances of services without a necessity of deploying a centralized load balancer.

Traditional problem of slow and manually managed load balancers is taken care of here. Consul programmatically manages registry, which gets updated when any new service registers itself and becomes available for receiving traffic.

This helps with scaling the services with ease.

Configuration Management in a monolith, its challenges in a distributed environment, and Consul’s solution

Monolithic Applications

When we look at the configuration for a monolithic application, they tend to be somewhere along the lines of giant YAML, XML or JSON files. That configuration is supposed to configure the entire application.

Given a single file, all of our subsystems in our monolithic application would now consume the configuration from the same file. Thus creating a consistent view of all our subsystems or services.

If we wish to change the state of the application using configuration update, it would be easily available to all the subsystems. The new configuration is simultaneously consumed by all the components of our application.

Distributed Systems

Unlike monolith, distributed services would not have a common view on configuration. The configuration is now distributed and there every individual service would need to be configured separately.

Challenges in Distributed Systems

• Configuration is to be spread across different services. Maintaining consistency between the configuration on different services after each update is a challenge.
• Moreover, the challenge grows when we expect the configuration to be updated dynamically.

Consul’s Solutions

Consul’s solution for configuration management in distributed environment is the central Key-Value store.

Consul solves this challenge in a unique way. Instead of spreading the configuration across different distributed service as configuration pieces, it pushes the whole configuration to all the services and configures them dynamically on the distributed system.

Let’s take an example of state change in configuration. The changed state is pushed across all the services in real-time. The configuration is consistently present with all the services.

Network segmentation in a monolith, its challenges in distributed systems, and Consul’s solutions

Monolithic Applications

When we look at our classic monolithic architecture, the network is typically divided in three different zones.

The first zone in our network is publicly accessible. The traffic coming to our application via the internet and reaching our load balancers.

The second zone is the traffic from our load balancers to our application. Mostly an internal network zone without direct public access.

The third zone is the closed network zone, primarily designated for data. This is considered to be an isolated zone.

Only the load balancers zone can reach into the application zone and only the application zone can reach into the data zone. It is a straightforward zoning system, simple to implement and manage.

Distributed Systems

The pattern changes drastically for distributed services.

Within our application network zone, there are multiple services which talk to each other within this network, making it a complicated traffic pattern.

Challenges

• The primary challenge is that the traffic is not in any sequential flow. Unlike monolithic architecture, where the flow was defined from load balancers to the application and application to data.
• Depending on the access pattern we want to support, the traffic might come from different endpoints and reaching different services.

• Given multiple services and the ability to support multiple endpoints allows us to deploy multiple service consumers and providers.
• Due to the nature of the system, security is our next challenge. Services should be able to identify that the traffic they are receiving is from a verified and trusted entity in the network.

• Controlling the flow of traffic and segmenting the network into groups or chunks will become a bigger issue. Also, making sure we have strict rules that guide us with partitioning the network based on who should be allowed to talk to whom and vice versa is also vital.

Consul’s Solutions

Consul’s solution to overall network segmentation challenge in distributed systems is by implementing service graph and mutual TLS.

Consul solves the problem of network segmentation by centrally managing the definition around who can talk to whom. Consul has a dedicated feature for this called Consul Connect.

Consul Connect enrolls these policies of inter-service communication that we desire and implements it as part of the service graph. So, a policy might say service A can talk to service B, but B cannot talk to C, for example.

The higher benefit of this is, it is not IP restricted. Rather it’s service level. This makes it scalable. The policy will be enforced on all instances of service and there will be no hard bound firewall rule specific to a service’s IP. Making us independent of the scale of our distributed network.

Consul Connect also handles service identity using popular TLS protocol. It distributes the TLS certificate associated with a service.

These certificates help other services securely identify each other. TLS is also helpful in making the communication between the services secure. This makes for trusted network implementation.

Consul enforces TLS using an agent-based proxy attached to each service instance. This proxy acts as a sidecar. Use of proxy, in this case, prevents us from making any change into the code of original service.

This allows for the higher level benefit of enforcing encryptions on data at rest and data in transit. Moreover, it will assist with fulfilling compliances required by laws around privacy and user identity.

Basic Architecture of Consul

Consul is a distributed and highly available system.  

Consul is shipped as a single binary download for all popular platforms. The executable can run as a client as well as server.

Each node that provides services to Consul runs a Consul agent. Each of these agents talk to one or more Consul servers.

Consul agent is responsible for health-checking the services on the node as the health-check of the node itself. It is not responsible for service discovery or maintaining key/value data.

Consul data and config storage and replication happens via Consul Servers.

Consul can run with single server, but it is recommended by HashiCorp to run a set of 3 to 5 servers to avoid failures. As all the data is stored at Consul server side, with a single server, the failure could cause a data loss.

With multi-servers cluster, they elect a leader among themselves. It is also recommended by HashiCorp to have cluster of servers per datacenter.

During the discovery process, any service in search for other service can query the Consul servers or even Consul agents. The Consul agents forward the queries to Consul servers automatically.

If the query is cross-datacenter, the queries are forwarded by the Consul server to the remote Consul servers. The results from remote Consul servers are returned to the original Consul server.

Getting Started with Consul

This section is dedicated to closely looking at Consul as a tool, with some hands-on experience.

Download and Install

As discussed above, Consul ships as a single binary downloaded from HashiCorps website or from Consul’s GitHub repo releases section.

This binary exhibits the role of both the Consul Server and Consul Client Agent. One can run them in either configuration.

You can download Consul from here – Download Consul page.

We will download Consul on command line using the link from download page

$ wget https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip -O consul.zip

--2019-03-10 00:14:07--  https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.37.183, 2a04:4e42:9::439
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.37.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34777003 (33M) [application/zip]
Saving to: ‘consul.zip’

consul.zip             100%[============================>]  33.17M  4.46MB/s    in 9.2s    

2019-03-10 00:14:17 (3.60 MB/s) - ‘consul.zip’ saved [34777003/34777003]

$ wget https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip -O consul.zip

--2019-03-10 00:14:07--  https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.37.183, 2a04:4e42:9::439
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.37.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34777003 (33M) [application/zip]
Saving to: ‘consul.zip’

consul.zip             100%[============================>]  33.17M  4.46MB/s    in 9.2s    

2019-03-10 00:14:17 (3.60 MB/s) - ‘consul.zip’ saved [34777003/34777003]

Unzip the downloaded zip file.  

$ unzip consul.zip

Archive:  consul.zip
  inflating: consul

$ unzip consul.zip

Archive:  consul.zip
  inflating: consul

Add it to PATH.

$ export PATH="$PATH:/path/to/consul"

$ export PATH="$PATH:/path/to/consul"

Use Consul

Once you unzip the compressed file and put the binary under your PATH, you can run it like this.

$ consul agent -dev

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: 'ef46ebb7-3496-346f-f67a-30117cfec0ad'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:38:01 [DEBUG] agent: Using random ID "ef46ebb7-3496-346f-f67a-30117cfec0ad" as node ID
    2019/03/04 00:38:01 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:ef46ebb7-3496-346f-f67a-30117cfec0ad Address:127.0.0.1:8300}]
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:38:01 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:38:01 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:38:01 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:38:01 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:38:01 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:38:01 [DEBUG] raft: Vote granted from ef46ebb7-3496-346f-f67a-30117cfec0ad in term 2. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:38:01 [INFO] consul: cluster leadership acquired
    2019/03/04 00:38:01 [INFO] consul: New leader elected: devcube
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:38:01 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:38:01 [INFO] agent: started state syncer
    2019/03/04 00:38:01 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:38:01 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:38:01 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [INFO] agent: Synced node info
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync

$ consul agent -dev

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: 'ef46ebb7-3496-346f-f67a-30117cfec0ad'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:38:01 [DEBUG] agent: Using random ID "ef46ebb7-3496-346f-f67a-30117cfec0ad" as node ID
    2019/03/04 00:38:01 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:ef46ebb7-3496-346f-f67a-30117cfec0ad Address:127.0.0.1:8300}]
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:38:01 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:38:01 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:38:01 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:38:01 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:38:01 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:38:01 [DEBUG] raft: Vote granted from ef46ebb7-3496-346f-f67a-30117cfec0ad in term 2. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:38:01 [INFO] consul: cluster leadership acquired
    2019/03/04 00:38:01 [INFO] consul: New leader elected: devcube
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:38:01 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:38:01 [INFO] agent: started state syncer
    2019/03/04 00:38:01 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:38:01 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:38:01 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [INFO] agent: Synced node info
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync

This will start the agent in development mode.

Consul Members

While the above command is running, you can check for all the members in Consul’s network.

$ consul members

Node     Address         Status  Type    Build  Protocol  DC   Segment
devcube  127.0.0.1:8301  alive   server  1.4.0  2         dc1  <all>

$ consul members

Node     Address         Status  Type    Build  Protocol  DC   Segment
devcube  127.0.0.1:8301  alive   server  1.4.0  2         dc1  <all>

Given we only have one node running, it is treated as server by default. You can designate an agent as a server by supplying server as command line parameter or server as configuration parameter to Consul’s config.

The output of the above command is based on the gossip protocol and is eventually consistent.

Consul HTTP API

For strongly consistent view of the Consul’s agent network, we can use HTTP API provided out of the box by Consul.

$ curl localhost:8500/v1/catalog/nodes

[
    {
        "ID": "ef46ebb7-3496-346f-f67a-30117cfec0ad",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "Meta": {
            "consul-network-segment": ""
        },
        "CreateIndex": 9,
        "ModifyIndex": 10
    }
]

$ curl localhost:8500/v1/catalog/nodes

[
    {
        "ID": "ef46ebb7-3496-346f-f67a-30117cfec0ad",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "Meta": {
            "consul-network-segment": ""
        },
        "CreateIndex": 9,
        "ModifyIndex": 10
    }
]

Consul DNS Interface

Consul also provides a DNS interface to query nodes. It serves DNS on 8600 port by default. That port is configurable.

$ dig @127.0.0.1 -p 8600 devcube.node.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 devcube.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42215
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;devcube.node.consul.		IN	A

;; ANSWER SECTION:
devcube.node.consul.	0	IN	A	127.0.0.1

;; ADDITIONAL SECTION:
devcube.node.consul.	0	IN	TXT	"consul-network-segment="

;; Query time: 19 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:45:44 IST 2019
;; MSG SIZE  rcvd: 100

$ dig @127.0.0.1 -p 8600 devcube.node.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 devcube.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42215
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;devcube.node.consul.		IN	A

;; ANSWER SECTION:
devcube.node.consul.	0	IN	A	127.0.0.1

;; ADDITIONAL SECTION:
devcube.node.consul.	0	IN	TXT	"consul-network-segment="

;; Query time: 19 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:45:44 IST 2019
;; MSG SIZE  rcvd: 100

Registering a service on Consul can be achieved either by writing a service definition or by sending a request over an appropriate HTTP API.

Consul Service Definition

Service definition is one of the popular ways of registering a service. Let’s take a look at one of such service definition examples.

To host our service definitions we will add a configuration directory, conventionally names as consul.d – ‘.d’ represents that there are set of configuration files under this directory, instead of single config under name consul.

$ mkdir ./consul.d

$ mkdir ./consul.d

Write the service definition for a fictitious Django web application running on port 80 on localhost.

$ echo '{"service": {"name": "web", "tags": ["django"], "port": 80}}' \
    > ./consul.d/web.json

$ echo '{"service": {"name": "web", "tags": ["django"], "port": 80}}' \
    > ./consul.d/web.json

To make our consul agent aware of this service definition, we can supply the configuration directory to it.

$ consul agent -dev -config-dir=./consul.d

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: '810f4804-dbce-03b1-056a-a81269ca90c1'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:55:28 [DEBUG] agent: Using random ID "810f4804-dbce-03b1-056a-a81269ca90c1" as node ID
    2019/03/04 00:55:28 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:810f4804-dbce-03b1-056a-a81269ca90c1 Address:127.0.0.1:8300}]
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:55:28 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:55:28 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:55:28 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:55:28 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:55:28 [INFO] agent: started state syncer
    2019/03/04 00:55:28 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:55:28 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:55:28 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:55:28 [DEBUG] raft: Vote granted from 810f4804-dbce-03b1-056a-a81269ca90c1 in term 2. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:55:28 [INFO] consul: cluster leadership acquired
    2019/03/04 00:55:28 [INFO] consul: New leader elected: devcube
    2019/03/04 00:55:28 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:55:28 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:55:28 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:55:28 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:28 [INFO] agent: Synced service "web"
    2019/03/04 00:55:28 [DEBUG] agent: Node info in sync
    2019/03/04 00:55:29 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:29 [DEBUG] agent: Service "web" in sync
    2019/03/04 00:55:29 [DEBUG] agent: Node info in sync

$ consul agent -dev -config-dir=./consul.d

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: '810f4804-dbce-03b1-056a-a81269ca90c1'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:55:28 [DEBUG] agent: Using random ID "810f4804-dbce-03b1-056a-a81269ca90c1" as node ID
    2019/03/04 00:55:28 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:810f4804-dbce-03b1-056a-a81269ca90c1 Address:127.0.0.1:8300}]
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:55:28 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:55:28 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:55:28 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:55:28 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:55:28 [INFO] agent: started state syncer
    2019/03/04 00:55:28 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:55:28 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:55:28 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:55:28 [DEBUG] raft: Vote granted from 810f4804-dbce-03b1-056a-a81269ca90c1 in term 2. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:55:28 [INFO] consul: cluster leadership acquired
    2019/03/04 00:55:28 [INFO] consul: New leader elected: devcube
    2019/03/04 00:55:28 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:55:28 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:55:28 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:55:28 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:28 [INFO] agent: Synced service "web"
    2019/03/04 00:55:28 [DEBUG] agent: Node info in sync
    2019/03/04 00:55:29 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:29 [DEBUG] agent: Service "web" in sync
    2019/03/04 00:55:29 [DEBUG] agent: Node info in sync

The relevant information in the log here are the sync statements related to the “web” service. Consul agent as accepted our config and synced it across all nodes. In this case one node.

Consul DNS Service Query

We can query the service with DNS, as we did with node. Like so:

$ dig @127.0.0.1 -p 8600 web.service.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51488
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.		IN	A

;; ANSWER SECTION:
web.service.consul.	0	IN	A	127.0.0.1

;; ADDITIONAL SECTION:
web.service.consul.	0	IN	TXT	"consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:59:32 IST 2019
;; MSG SIZE  rcvd: 99

$ dig @127.0.0.1 -p 8600 web.service.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51488
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.		IN	A

;; ANSWER SECTION:
web.service.consul.	0	IN	A	127.0.0.1

;; ADDITIONAL SECTION:
web.service.consul.	0	IN	TXT	"consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:59:32 IST 2019
;; MSG SIZE  rcvd: 99