My blog

Blog

  • A Practical Guide To HashiCorp Consul – Part 2

    This is part 2 of 2 part series on A Practical Guide to HashiCorp Consul. The previous part was primarily focused on understanding the problems that Consul solves and how it solves them. This part is focused on a practical application of Consul in a real-life example. Let’s get started.

    With most of the theory covered in the previous part, let’s move on to Consul’s practical example.

    What are we Building?

    We are going to build a Django Web Application that stores its persistent data in MongoDB. We will containerize both of them using Docker. Build and run them using Docker Compose.

    To show how our web app would scale in this context, we are going to run two instances of Django app. Also, to make this even more interesting, we will run MongoDB as a Replica Set with one primary node and two secondary nodes.

    Given we have two instances of Django app, we will need a way to balance a load among those two instances, so we are going to use Fabio, a Consul aware load-balancer, to reach Django app instances.

    This example will roughly help us simulate a real-world practical application.

     Example Application nodes and services deployed on them

    ‍

    The complete source code for this application is open-sourced and is available on GitHub – pranavcode/consul-demo.

    Note: The architecture we are discussing here is not specifically constraint with any of the technologies used to create app or data layers. This example could very well be built using a combination of Ruby on Rails and Postgres, or Node.js and MongoDB, or Laravel and MySQL.

    How Does Consul Come into the Picture?

    We are deploying both, the app and the data, layers with Docker containers. They are going to be built as services and will talk to each other over HTTP.

    Thus, we will use Consul for Service Discovery. This will allow Django servers to find MongoDB Primary node. We are going to use Consul to resolve services via Consul’s DNS interface for this example.

    Consul will also help us with the auto-configuration of Fabio as load-balancer to reach instances of our Django app.

    We are also using the health-check feature of Consul to monitor the health of each of our instances in the whole infrastructure.

    Consul provides a beautiful user interface, as part of its Web UI, out of the box to show all the services on a single dashboard. We will use it to see how our services are laid out.

    Let’s begin.

    Setup: MongoDB, Django, Consul, Fabio, and Dockerization

    We will keep this as simple and minimal as possible to the extent it fulfills our need for a demonstration.

    MongoDB

    The MongoDB setup we are targeting is in the form of MongoDB Replica Set. One primary node and two secondary nodes.

    The primary node will manage all the write operations and the oplog to maintain the sequence of writes, and replicate the data across secondaries. We are also configuring the secondaries for the read operations. You can learn more about MongoDB Replica Set on their official documentation.

    We will call our replication set as ‘consuldemo’.

    We will run MongoDB on a standard port 27017 and supply the name of the replica set on the command line using the parameter ‘–replSet’.

    As you may read from the documentation MongoDB also allows configuring replica set name via configuration file with the parameter for replication as below:

    replication:
    replSetName: "consuldemo"

    In our case, the replication set configuration that we will apply on one of the MongoDB nodes, once all the nodes are up and running is as given below:

    var config = {
    _id: "consuldemo",
    version: 1,
    members: [{
        _id: 0,
        host: "mongo_1:27017",
    }, {
        _id: 1,
        host: "mongo_2:27017",
    }, {
        _id: 2,
        host: "mongo_3:27017",
    }],
    settings: { 
        chainingAllowed: true 
    }
};
rs.initiate(config, { force: true });
rs.slaveOk();
db.getMongo().setReadPref("nearest");

    This configuration will be applied to one of the pre-defined nodes and MongoDB will decide which node will be primary and secondary.

    Note: We are not forcing the set creation with any pre-defined designations on who becomes primary and secondary to allow the dynamism in service discovery. Normally, the nodes would be defined for a specific role.

    We are allowing slave reads and reads from the nearest node as a Read Preference.

    We will start MongoDB on all nodes with the following command:

    mongod --bind_ip_all --port 27017 --dbpath /data/db --replSet "consuldemo"

    This gives us a MongoDB Replica Set with one primary instance and two secondary instances, running and ready to accept connections.

    We will discuss containerizing the MongoDB service in the latter part of this article.

    Django

    We will create a simple Django project that represents Blog application and containerizes it with Docker.

    Building the Django app from scratch is beyond the scope of this tutorial, we recommend you to refer to Django’s official documentation to get started with Django project. But, we will still go through some important aspects.

    As we need our Django app to talk to MongoDB, we will use a MongoDB connector for Django ORM, Djongo. We will set up our Django settings to use Djongo and connect with our MongoDB. Djongo is pretty straightforward in configuration.

    For a local MongoDB installation it would only take two lines of code:

    ...

DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'db',
    }
}

...

    In our case, as we will need to access MongoDB over another container, our config would look like this:

    ...

DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'db',
        'HOST': 'mongodb://mongo-primary.service.consul',
        'PORT': 27017,
    }
}

...
@velotiotech

    Details:

    • ENGINE: The database connector to use for Django ORM.
    • NAME: Name of the database.
    • HOST: Host address that has MongoDB running on it.
    • PORT: Which port is your MongoDB listening for requests.

    Djongo internally talks to PyMongo and uses MongoClient for executing queries on Mongo. We can also use other MongoDB connectors available for Django to achieve this, like, for instance, django-mongodb-engine or pymongo directly, based on our needs.

    Note: We are currently reading and writing via Django to a single MongoDB host, the primary one, but we can configure Djongo to also talk to secondary hosts for read-only operations. That is not in the scope of our discussion. You can refer to Djongo’s official documentation to achieve exactly this.

    Continuing our Django app building process, we need to define our models. As we are building a blog-like application, our models would look like this:

    from djongo import models

class Blog(models.Model):
    name = models.CharField(max_length=100)
    tagline = models.TextField()

    class Meta:
        abstract = True

class Entry(models.Model):
    blog = models.EmbeddedModelField(
        model_container=Blog,
    )
    
    headline = models.CharField(max_length=255)

    We can run a local MongoDB instance and create migrations for these models. Also, register these models into our Django Admin, like so:

    from django.contrib import admin
from.models import Entry

admin.site.register(Entry)

    We can play with the Entry model’s CRUD operations via Django Admin for this example.

    Also, to realize the Django-MongoDB connectivity we will create a custom View and Template that displays information about MongoDB setup and currently connected MongoDB host.

    Our Django views look like this:

    from django.shortcuts import render
from pymongo import MongoClient

def home(request):
    client = MongoClient("mongo-primary.service.consul")
    replica_set = client.admin.command('ismaster')

    return render(request, 'home.html', { 
        'mongo_hosts': replica_set['hosts'],
        'mongo_primary_host': replica_set['primary'],
        'mongo_connected_host': replica_set['me'],
        'mongo_is_primary': replica_set['ismaster'],
        'mongo_is_secondary': replica_set['secondary'],
    })

    Our URLs or routes configuration for the app looks like this:

    from django.urls import path
from tweetapp import views

urlpatterns = [
    path('', views.home, name='home'),
]

    And for the project – the app URLs are included like so:

    from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static

urlpatterns = [
    path('admin/', admin.site.urls),
    path('web', include('tweetapp.urls')),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

    Our Django template, ‘templates/home.html’ looks like this:

    <!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
    <link href="https://fonts.googleapis.com/css?family=Armata" rel="stylesheet">

    <title>Django-Mongo-Consul</title>
</head>
<body class="bg-dark text-white p-5" style="font-family: Armata">
    <div class="p-4 border">
        <div class="m-2">
            <b>Django Database Connection</b>
        </div>
        <table class="table table-dark">
            <thead>
                <tr>
                    <th scope="col">#</th>
                    <th scope="col">Property</th>
                    <th scope="col">Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>1</td>
                    <td>Mongo Hosts</td>
                    <td>{% for host in mongo_hosts %}{{ host }}<br/>{% endfor %}</td>
                </tr>
                <tr>
                    <td>2</td>
                    <td>Mongo Primary Address</td>
                    <td>{{ mongo_primary_host }}</td>
                </tr>
                <tr>
                    <td>3</td>
                    <td>Mongo Connected Address</td>
                    <td>{{ mongo_connected_host }}</td>
                </tr>
                <tr>
                    <td>4</td>
                    <td>Mongo - Is Primary?</td>
                    <td>{{ mongo_is_primary }}</td>
                </tr>
                <tr>
                    <td>5</td>
                    <td>Mongo - Is Secondary?</td>
                    <td>{{ mongo_is_secondary }}</td>
                </tr>
            </tbody>
        </table>
    </div>
    
    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
</body>
</html>

    To run the app we need to migrate the database first using the command below:

    python ./manage.py migrate

    And also collect all the static assets into static directory:

    python ./manage.py collectstatic --noinput

    Now run the Django app with Gunicorn, a WSGI HTTP server, as given below:

    gunicorn --bind 0.0.0.0:8000 --access-logfile - tweeter.wsgi:application

    This gives us a basic blog-like Django app that connects to MongoDB backend.

    We will discuss containerizing this Django web application in the latter part of this article.

    Consul

    We place a Consul agent on every service as part of our Consul setup.

    The Consul agent is responsible for service discovery by registering the service on the Consul cluster and also monitors the health of every service instance.

    Consul on nodes running MongoDB Replica Set

    We will discuss Consul setup in the context of MongoDB Replica Set first – as it solves an interesting problem. At any given point of time, one of the MongoDB instances can either be a Primary or a Secondary.

    The Consul agent registering and monitoring our MongoDB instance within a Replica Set has a unique mechanism – dynamically registering and deregistering MongoDB service as a Primary instance or a Secondary instance based on what Replica Set has designated it.

    We achieve this dynamism by writing and running a shell script after an interval that toggles the Consul service definition for MongoDB Primary and MongoDB Secondary on the instance node’s Consul Agent.

    The service definitions for MongoDB services are stored as JSON files on the Consul’s config directory ‘/etc/config.d’.

    Service definition for MongoDB Primary instance:

    {
    "service": {
        "name": "mongo-primary",
        "port": 27017,
        "tags": [
            "nosql",
            "database"
        ],
        "check": {
            "id": "mongo_primary_status",
            "name": "Mongo Primary Status",
            "args": ["/etc/consul.d/check_scripts/mongo_primary_check.sh"],
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

    If you look closely, the service definition allows us to get a DNS entry specific to MongoDB Primary, rather than a generic MongoDB instance. This allows us to send the database writes to a specific MongoDB instance. In the case of Replica Set, the writes are maintained by MongoDB Primary.

    Thus, we are able to achieve both service discovery as well as health monitoring for Primary instance of MongoDB.

    Similarly, with a slight change the service definition for MongoDB Secondary instance goes like this:

    {
    "service": {
        "name": "mongo-secondary",
        "port": 27017,
        "tags": [
            "nosql",
            "database"
        ],
        "check": {
            "id": "mongo_secondary_status",
            "name": "Mongo Secondary Status",
            "args": ["/etc/consul.d/check_scripts/mongo_secondary_check.sh"],
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

    Given all this context, can you think of the way we can dynamically switch these service definitions?

    We can identify if the given MongoDB instance is primary or not by running command `db.isMaster()` on MongoDB shell.

    The check can we drafted as a shell script as:

    #!/bin/bash

mongo_primary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .ismaster 2> /dev/null)
if [[ $mongo_primary == false ]]; then
    exit 1
fi

echo "Mongo primary healthy and reachable"

    Similarly, the non-master or non-primary instances of MongoDB can also be checked against the same command, by checking a `secondary` value:

    #!/bin/bash

mongo_secondary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .secondary 2> /dev/null)
if [[ $mongo_secondary == false ]]; then
    exit 1
fi

echo "Mongo secondary healthy and reachable"

    Note: We are using jq – a lightweight and flexible command-line JSON processor – to process the JSON encoded output of MongoDB shell commands.

    One way of writing a script that does this dynamic switch looks like this:

    #!/bin/bash

# Wait until Mongo starts
while [[ $(ps aux | grep [m]ongod | wc -l) -ne 1 ]]; do
    sleep 5
done

REGISTER_MASTER=0
REGISTER_SECONDARY=0

mongo_primary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .ismaster 2> /dev/null)
mongo_secondary=$(mongo --quiet --eval 'JSON.stringify(db.isMaster())' | jq -r .secondary 2> /dev/null)  

if [[ $mongo_primary == false && $mongo_secondary == true ]]; then

  # Deregister as Mongo Master
  if [[ -a /etc/consul.d/check_scripts/mongo_primary_check.sh && -a /etc/consul.d/mongo_primary.json ]]; then
    rm -f /etc/consul.d/check_scripts/mongo_primary_check.sh
    rm -f /etc/consul.d/mongo_primary.json

    REGISTER_MASTER=1
  fi

  # Register as Mongo Secondary
  if [[ ! -a /etc/consul.d/check_scripts/mongo_secondary_check.sh && ! -a /etc/consul.d/mongo_secondary.json ]]; then
    cp -u /opt/checks/check_scripts/mongo_secondary_check.sh /etc/consul.d/check_scripts/
    cp -u /opt/checks/mongo_secondary.json /etc/consul.d/

    REGISTER_SECONDARY=1
  fi

else

  # Register as Mongo Master
  if [[ ! -a /etc/consul.d/check_scripts/mongo_primary_check.sh && ! -a /etc/consul.d/mongo_primary.json ]]; then
    cp -u /opt/checks/check_scripts/mongo_primary_check.sh /etc/consul.d/check_scripts/
    cp -u /opt/checks/mongo_primary.json /etc/consul.d/

    REGISTER_MASTER=2
  fi

  # Deregister as Mongo Secondary
  if [[ -a /etc/consul.d/check_scripts/mongo_secondary_check.sh && -a /etc/consul.d/mongo_secondary.json ]]; then
    rm -f /etc/consul.d/check_scripts/mongo_secondary_check.sh
    rm -f /etc/consul.d/mongo_secondary.json

    REGISTER_SECONDARY=2
  fi

fi

if [[ $REGISTER_MASTER -ne 0 && $REGISTER_SECONDARY -ne 0 ]]; then
  consul reload
fi

    Note: This is an example script, but we can be more creative and optimize the script further.

    Once we are done with our service definitions we can run the Consul agent on each MongoDB nodes. To run a agent we will use the following command:

    consul agent -bind 33.10.0.3 
    -advertise 33.10.0.3 
    -join consul_server 
    -node mongo_1 
    -dns-port 53 
    -data-dir /data 
    -config-dir /etc/consul.d 
    -enable-local-script-checks

    Here,  ‘consul_server’ represents the Consul Server running host. Similarly, we can run such agents on each of the other MongoDB instance nodes.

    Note: If we have multiple MongoDB instances running on the same host, the service definition would change to reflect the different ports used by each instance to uniquely identify, discover and monitor individual MongoDB instance.

    Consul on nodes running Django App

    For the Django application, Consul setup will be very simple. We only need to monitor Django app’s port on which Gunicorn is listening for requests.

    The Consul service definition would look like this:

    {
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

    Once we have the Consul service definition for Django app in place, we can run the Consul agent sitting on the node Django app is running as a service. To run the Consul agent we would fire the following command:

    consul agent -bind 33.10.0.10 
    -advertise 33.10.0.10 
    -join consul_server 
    -node web_1 
    -dns-port 53 
    -data-dir /data 
    -config-dir /etc/consul.d 
    -enable-local-script-checks

    Consul Server

    We are running the Consul cluster with a dedicated Consul server node. The Consul server node can easily host, discover and monitor services running on it, exactly the same way as we did in the above sections for MongoDB and Django app.

    To run Consul in server mode and allow agents to connect to it, we will fire the following command on the node that we want to run our Consul server:

    consul agent -server 
    -bind 33.10.0.2 
    -advertise 33.10.0.2 
    -node consul_server 
    -client 0.0.0.0 
    -dns-port 53 
    -data-dir /data 
    -ui -bootstrap

    There are no services on our Consul server node for now, so there are no service definitions associated with this Consul agent configuration.

    Fabio

    We are using the power of Fabio to be auto-configurable and being Consul-aware.

    This makes our task of load-balancing the traffic to our Django app instances very easy.

    To allow Fabio to auto-detect the services via Consul, one of the ways is to add a tag or update a tag in the service definition with a prefix and a service identifier `urlprefix-/<service>`. Our Consul’s service definition for Django app would now look like this:</service>

    {
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

    In our case, the Django app or service is the only service that will need load-balancing, thus this Consul service definition change completes the requirement on Fabio setup.

    Dockerization

    Our whole app is going to be deployed as a set of Docker containers. Let’s talk about how we are achieving it in the context of Consul.

    Dockerizing MongoDB Replica Set along with Consul Agent

    We need to run a Consul agent as described above alongside MongoDB on the same Docker container, so we will need to run a custom ENTRYPOINT on the container to allow running two processes.

    Note: This can also be achieved using Docker container level checks in Consul. So, you will be free to run a Consul agent on the host and check across service running in Docker container. Which, will essentially exec into the container to monitor the service.

    To achieve this we will use a tool similar to Foreman. It is a lifecycle management tool for physical and virtual servers – including provisioning, monitoring and configuring.

    To be precise, we will use the Golang adoption of Foreman, Goreman. It takes the configuration in the form of Heroku’sProcfile to maintain which processes to be kept alive on the host.

    In our case, the Procfile looks like this:

    # Mongo
mongo: /opt/mongo.sh

# Consul Client Agent
consul: /opt/consul.sh

# Consul Client Health Checks
consul_check: while true; do /opt/checks_toggler.sh && sleep 10; done

    The `consul_check` at the end of the Profile maintains the dynamism between both Primary and Secondary MongoDB node checks, based on who is voted for which role within MongoDB Replica Set.

    The shell scripts that are executed by the respective keys on the Procfile are as defined previously in this discussion.

    Our Dockerfile, with some additional tools for debug and diagnostics, would look like:

    FROM ubuntu:18.04

RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping

# Install MongoDB
RUN apt-get install -y mongodb

RUN mkdir -p /data/db
VOLUME data:/data/db

# Setup Consul and Goreman
ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip

ADD https://github.com/mattn/goreman/releases/download/v0.0.10/goreman_linux_amd64.zip /tmp/goreman.zip
RUN cd /bin && unzip /tmp/goreman.zip && chmod +x /bin/goreman && rm /tmp/goreman.zip

RUN mkdir -p /etc/consul.d/check_scripts
ADD ./config/mongod /etc

RUN mkdir -p /etc/checks
ADD ./config/checks /opt/checks

ADD checks_toggler.sh /opt
ADD mongo.sh /opt
ADD consul.sh /opt

ADD Procfile /root/Procfile

EXPOSE 27017

# Launch both MongoDB server and Consul
ENTRYPOINT [ "goreman" ]
CMD [ "-f", "/root/Procfile", "start" ]

    Note: We have used bare Ubuntu 18.04 image here for our purposes, but you can use official MongoDB image and adapt it to run Consul alongside MongoDB or even do Consul checks on Docker container level as mentioned in the official documentation.

    Dockerizing Django Web Application along with Consul Agent

    We also need to run a Consul agent alongside our Django App on the same Docker container as we had with MongoDB container.

    # Django
django: /web/tweeter.sh

# Consul Client Agent
consul: /opt/consul.sh

    Similarly, we will have the Dockerfile for Django Web Application as we had for our MongoDB containers.

    FROM python:3.7

RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping

# Python Environment Setup
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Setup Consul and Goreman
RUN mkdir -p /data/db /etc/consul.d

ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip

ADD https://github.com/mattn/goreman/releases/download/v0.0.10/goreman_linux_amd64.zip /tmp/goreman.zip
RUN cd /bin && unzip /tmp/goreman.zip && chmod +x /bin/goreman && rm /tmp/goreman.zip

ADD ./consul /etc/consul.d
ADD Procfile /root/Procfile

# Install pipenv
RUN pip3 install --upgrade pip
RUN pip3 install pipenv

# Setting workdir
ADD consul.sh /opt
ADD . /web
WORKDIR /web/tweeter

# Exposing appropriate ports
EXPOSE 8000/tcp

# Install dependencies
RUN pipenv install --system --deploy --ignore-pipfile

# Migrates the database, uploads staticfiles, run API server and background tasks
ENTRYPOINT [ "goreman" ]
CMD [ "-f", "/root/Procfile", "start" ]

    Dockerizing Consul Server

    We are maintaining the same flow with Consul server node to run it with custom ENTRYPOINT. It is not a requirement, but we are maintaining a consistent view of different Consul run files.

    Also, we are using Ubuntu 18.04 image for the demonstration. You can very well use Consul’s official image for this, that accepts all the custom parameters as are mentioned here.

    FROM ubuntu:18.04

RUN apt-get update && 
    apt-get install -y 
    bash curl nano net-tools zip unzip 
    jq dnsutils iputils-ping

ADD https://releases.hashicorp.com/consul/1.4.4/consul_1.4.4_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && unzip /tmp/consul.zip && chmod +x /bin/consul && rm /tmp/consul.zip

# Consul ports
EXPOSE 8300 8301 8302 8400 8500

ADD consul_server.sh /opt
RUN mkdir -p /data
VOLUME /data

CMD ["/opt/consul_server.sh"]

    Docker Compose

    We are using Compose to run all our Docker containers in a desired, repeatable form.

    Our Compose file is written to denote all the aspects that we mentioned above and utilize the power of Docker Compose tool to achieve those in a seamless fashion.

    Docker Compose file would look like the one given below:

    version: "3.6"

services:

  consul_server:
    build:
      context: consul_server
      dockerfile: Dockerfile
    image: consul_server
    ports:
      - 8300:8300
      - 8301:8301
      - 8302:8302
      - 8400:8400
      - 8500:8500
    environment:
      - NODE=consul_server
      - PRIVATE_IP_ADDRESS=33.10.0.2
    networks:
      consul_network:
        ipv4_address: 33.10.0.2

  load_balancer:
    image: fabiolb/fabio
    ports:
      - 9998:9998
      - 9999:9999
    command: -registry.consul.addr="33.10.0.2:8500"
    networks:
      consul_network:
        ipv4_address: 33.10.0.100

  mongo_1:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_1
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.3
    restart: always
    ports:
      - 27017:27017
      - 28017:28017
    depends_on:
      - consul_server
      - mongo_2
      - mongo_3
    networks:
      consul_network:
        ipv4_address: 33.10.0.3

  mongo_2:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_2
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.4
    restart: always
    ports:
      - 27018:27017
      - 28018:28017
    depends_on:
      - consul_server
    networks:
      consul_network:
        ipv4_address: 33.10.0.4

  mongo_3:
    build:
      context: mongo
      dockerfile: Dockerfile
    image: mongo_consul
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    environment:
      - NODE=mongo_3
      - MONGO_PORT=27017
      - PRIMARY_MONGO=33.10.0.3
      - PRIVATE_IP_ADDRESS=33.10.0.5
    restart: always
    ports:
      - 27019:27017
      - 28019:28017
    depends_on:
      - consul_server
    networks:
      consul_network:
        ipv4_address: 33.10.0.5

  web_1:
    build:
      context: django
      dockerfile: Dockerfile
    image: web_consul
    ports:
      - 8080:8000
    environment:
      - NODE=web_1
      - PRIMARY=1
      - LOAD_BALANCER=33.10.0.100
      - PRIVATE_IP_ADDRESS=33.10.0.10
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    depends_on:
      - consul_server
      - mongo_1
    volumes:
      - ./django:/web
    cap_add:
      - NET_ADMIN
    networks:
      consul_network:
        ipv4_address: 33.10.0.10

  web_2:
    build:
      context: django
      dockerfile: Dockerfile
    image: web_consul
    ports:
      - 8081:8000
    environment:
      - NODE=web_2
      - LOAD_BALANCER=33.10.0.100
      - PRIVATE_IP_ADDRESS=33.10.0.11
    dns:
      - 127.0.0.1
      - 8.8.8.8
      - 8.8.4.4
    depends_on:
      - consul_server
      - mongo_1
    volumes:
      - ./django:/web
    cap_add:
      - NET_ADMIN
    networks:
      consul_network:
        ipv4_address: 33.10.0.11

networks:
  consul_network:
    driver: bridge
    ipam:
     config:
       - subnet: 33.10.0.0/16

    That brings us to the end of the whole environment setup. We can now run Docker Compose to build and run the containers.

    Service Discovery using Consul

    When all the services are up and running the Consul Web UI gives us a nice glance at our overall setup.

     Consul Web UI showing the set of services we are running and their current state

    ‍

    The MongoDB service is available for Django app to discover by virtue of Consul’s DNS interface.

    root@82857c424b15:/web/tweeter# dig @127.0.0.1 mongo-primary.service.consul

; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 mongo-primary.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8369
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mongo-primary.service.consul.	IN	A

;; ANSWER SECTION:
mongo-primary.service.consul. 0	IN	A	33.10.0.3

;; ADDITIONAL SECTION:
mongo-primary.service.consul. 0	IN	TXT	"consul-network-segment="

;; Query time: 139 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Apr 01 11:50:45 UTC 2019
;; MSG SIZE  rcvd: 109

    Django App can now connect MongoDB Primary instance and start writing data to it.

    We can use Fabio load-balancer to connect to Django App instance by auto-discovering it via Consul registry using specialized service tags and render the page with all the database connection information we are talking about.

    Our load-balancer is sitting at ‘33.10.0.100’ and ‘/web’ is configured to be routed to one of our Django application instances running behind the load-balancer.

     Fabio auto-detecting the Django Web Application end-points

    As you can see from the auto-detection and configuration of Fabio load-balancer from its UI above, it has weighted the Django Web Application end-points equally. This will help balance the request or traffic load on the Django application instances.

    When we visit our Fabio URL ‘33.10.0.100:9999’ and use the source route as ‘/web’ we are routed to one of the Django instances. So, visiting ‘33.10.0.100:9999/web’ gives us following output.

    Django Web Application renders the MongoDB connection status on the home page

    We are able to restrict Fabio to only load-balance Django app instances by only adding required tags to Consul’s service definitions of Django app services.

    This MongoDB Primary instance discovery helps Django app to do database migration and app deployment.

    One can explore Consul Web UI to see all the instances of Django web application services.

     Django Web Application services as seen on Consul’s Web UI

    ‍

    Similarly, see how MongoDB Replica Set instances are laid out.

    MongoDB Replica Set Primary service as seen on Consul’s Web UI
     MongoDB Replica Set Secondary services as seen on Consul’s Web UI

    Let’s see how Consul helps with health-checking services and discovering only the alive services.

    We will stop the current MongoDB Replica Set Primary (‘mongo_2’) container, to see what happens.

    MongoDB Primary service being swapped with one of the MongoDB Secondary instances
     MongoDB Secondary instance set is now left with only one service instance

    Consul has started failing the health-check for previous MongoDB Primary service. MongoDB Replica Set has also detected that the node is down and the re-election of Primary node needs to be done. Thus, getting us a new MongoDB Primary (‘mongo_3’) automatically.

    Our checks toggle has kicked-in and swapped the check on ‘mongo_3’ from MongoDB Secondary check to MongoDB Primary check.

    When we take a look at the view from the Django app, we see it is now connected to a new MongoDB Primary service (‘mongo_3’).

    Switching of the MongoDB Primary is also reflected in the Django Web Application

    Let’s see how this plays out when we bring back the stopped MongoDB instance.

     Failing MongoDB Primary service instance is now cleared out from service instances as it is now healthy MongoDB Secondary service instance
     Previously failed MongoDB Primary service instance is now re-adopted as MongoDB Secondary service instance as it has become healthy again

    Similarly, if we stop the service instances of Django application, Fabio would now be able to detect only a healthy instance and would only route the traffic to that instance.

     Fabio is able to auto-configure itself using Consul’s service registry and detecting alive service instances

    This is how one can use Consul’s service discovery capability to discover, monitor and health-check services.

    Service Configuration using Consul

    Currently, we are configuring Django application instances directly either from environment variables set within the containers by Docker Compose and consuming them in Django project settings or by hard-coding the configuration parameters directly.

    We can use Consul’s Key/Value store to share configuration across both the instances of Django app.

    We can use Consul’s HTTP interface to store key/value pair and retrieve them within the app using the open-source Python client for Consul, called python-consul. You may also use any other Python library that can interact with Consul’s KV store if you want.

    Let’s begin by looking at how we can set a key/value pair in Consul using its HTTP interface.

    # Flag to run Django app in debug mode
curl -X PUT -d 'True' consul_server:8500/v1/kv/web/debug

# Dynamic entries into Django app configuration 
# to denote allowed set of hosts
curl -X PUT -d 'localhost, 33.10.0.100' consul_server:8500/v1/kv/web/allowed_hosts

# Dynamic entries into Django app configuration
# to denote installed apps
curl -X PUT -d 'tweetapp' consul_server:8500/v1/kv/web/installed_apps

    Once we set the KV store we can consume it on Django app instances to configure it with these values.

    Let’s install python-consul and add it as a project dependency.

    $ pipenv shell
Launching subshell in virtual environment…
 . /home/pranav/.local/share/virtualenvs/tweeter-PYSn2zRU/bin/activate

$  . /home/pranav/.local/share/virtualenvs/tweeter-PYSn2zRU/bin/activate

(tweeter) $ pipenv install python-consul
Installing python-consul…
Adding python-consul to Pipfile's [packages]…
✔ Installation Succeeded 
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (9590cc)!
Installing dependencies from Pipfile.lock (9590cc)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 14/14 — 00:00:20

    We will need to connect our app to Consul using python-consul.

    import consul

consul_client = consul.Consul(
    host='consul_server',
    port=8500,
)

    We can capture and configure our Django app accordingly using the ‘python-consul’ library.

    # Set DEBUG flag using Consul KV store
index, data = consul_client.kv.get('web/debug')
DEBUG = data.get('Value', True)

# Set ALLOWED_HOSTS dynamically using Consul KV store
ALLOWED_HOSTS = []

index, hosts = consul_client.kv.get('web/allowed_hosts')
ALLOWED_HOSTS.append(hosts.get('Value'))

# Set INSTALLED_APPS dynamically using Consul KV store
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

index, apps = consul_client.kv.get('web/installed_apps')
INSTALLED_APPS += (bytes(apps.get('Value')).decode('utf-8'),)

    These key/value pair from Consul’s KV store can also be viewed and updated from its Web UI.

     Consul KV store as seen on Consul Web UI with Django app configuration parameters

    The code used as part of this guide for Consul’s service configuration section is available on ‘service-configuration’ branch of pranavcode/consul-demo project.

    That is how one can use Consul’s KV store and configure individual services in their architecture with ease.

    Service Segmentation using Consul

    As part of Consul’s Service Segmentation we are going to look at Consul Connect intentions and data center distribution.

    Connect provides service-to-service connection authorization and encryption using mutual TLS.

    To use Consul you need to enable it in the server configuration. Connect needs to be enabled across the Consul cluster for proper functioning of the cluster.

    {
    "connect": {
        "enabled": true
    }
}

    In our context, we can define that the communication is to be TLS identified and secured we will define an upstream sidecar service with a proxy on Django app for its communication with MongoDB Primary instance.

    {
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "connect": {
            "sidecar_service": {
                "proxy": {
                    "upstreams": [{
                        "destination_name": "mongo-primary",
                        "local_bind_port": 5501
                    }]
                }
            }
        },
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

    Along with Connect configuration of sidecar proxy, we will also need to run the Connect proxy for Django app as well. This could be achieved by running the following command.

    {
    "service": {
        "name": "web",
        "port": 8000,
        "tags": [
            "web",
            "application",
            "urlprefix-/web"
        ],
        "connect": {
            "sidecar_service": {
                "proxy": {
                    "upstreams": [{
                        "destination_name": "mongo-primary",
                        "local_bind_port": 5501
                    }]
                }
            }
        },
        "check": {
            "id": "web_app_status",
            "name": "Web Application Status",
            "tcp": "localhost:8000",
            "interval": "30s",
            "timeout": "20s"
        }
    }
}

    We can add Consul Connect Intentions to create a service graph across all the services and define traffic pattern. We can create intentions as shown below:

    $ consul connect proxy -sidecar-for web

    Intentions for service graph can also be managed from Consul Web UI.

     Define access control for services via Connect and service connection restrictions

    This defines the service connection restrictions to allow or deny them to talk via Connect.

    We have also added ability on Consul agents to denote which datacenters they belong to and be accessible via one or more Consul servers in a given datacenter.

    The code used as part of this guide for Consul’s service segmentation section is available on ‘service-segmentation’ branch of velotiotech/consul-demo project.

    That is how one can use Consul’s service segmentation feature and configure service level connection access control.

    Conclusion

    Having an ability to seamlessly control the service mesh that Consul provides makes the life of an operator very easy. We hope you have learnt how Consul can be used for service discovery, configuration, and segmentation with its practical implementation.

    As usual, we hope it was an informative ride on the journey of Consul. This was the final piece of this two part series. This part tries to cover most of the aspects of Consul architecture and how it fits into your current project. In case you miss the first part, find it here.

    We will continue our endeavors with different technologies and get you the most valuable information that we possibly can in every interaction. Let’s us know what you would like to hear from us more or if you have any questions around the topic, we will be more than happy to answer those.

    References

  • A Practical Guide to HashiCorp Consul – Part 1

    This is part 1 of 2 part series on A Practical Guide to HashiCorp Consul. This part is primarily focused on understanding the problems that Consul solves and how it solves them. The second part is more focused on a practical application of Consul in a real-life example. Let’s get started.

    How about setting up discoverable, configurable, and secure service mesh using a single tool?

    What if we tell you this tool is platform-agnostic and cloud-ready ?

    And comes as a single binary download.

    All this is true. The tool we are talking about is HashiCorp Consul.

    Consul provides service discovery, health checks, load balancing, service graph, identity enforcement via TLS, and distributed service configuration management.

    Let’s learn about Consul in details below and see how it solves these complex challenges and makes the life of a distributed system operator easy.

    Introduction

    Microservices and other distributed systems can enable faster, simpler software development. But there’s a trade-off resulting in greater operational complexity around inter-service communication, configuration management, and network segmentation.

    Monolithic Application (representational) – with different subsystems A, B, C and D
    Distributed Application (representational) – with different services A, B, C and D

      

    HashiCorp Consul is an open source tool that solves these new complexities by providing service discovery, health checks, load balancing, a service graph, mutual TLS identity enforcement, and a configuration key-value store. These Consul features make it an ideal control plane for a service mesh.

    HashiCorp Consul supports Service Discovery, Service Configuration, and Service Segmentation

     

    HashiCorp announced Consul in April 2014 and it has since then got a good community acceptance.

    This guide is aimed at discussing some of these crucial problems and exploring the various solutions provided by HashiCorp Consul to tackle these problems.

    Let’s rundown through the topics that we are going to cover in this guide. The topics are written to be self-content. You can jump directly to a specific topic if you want to.

    Brief Background on Monolithic vs. Service-oriented Architectures (SOA)

    Looking at traditional architectures of application delivery, what we find is a classic monolith. When we talk about monolith, we have a single application deployment.

    Even if it is a single application, typically it has multiple different sub-components.

    One of the examples that HashiCorp’s CTO Armon Dadgar gave during his introductory video for Consul was about – delivering desktop banking application. It has a discrete set of sub-components – for example, authentication (say subsystem A), account management (subsystem B), fund transfer (subsystem C), and foreign exchange (subsystem D).

    Now, although these are independent functions – system A authentication vs system C fund transfer – we deploy it as a single, monolith app.

    Over the last few years, we have seen a trend away from this kind of architectures. There are several reasons for this shift.

    Challenge with monolith is: Suppose there is a bug in one of the subsystems, system A, related to authentication.

     Representational bug in Subsystem A in our monolithic application

    We can’t just fix it in system A and update it in production.

     Representational bug fix in Subsystem A in our monolithic application

    We have to update system A and do a redeploy of the whole application, which we need deployment of subsystems B, C, and D as well.

    Bug fix in one subsystem results in redeployment of the whole monolithic application

    This whole redeployment is not ideal. Instead, we would like to do a deployment of individual services.

    The same monolithic app delivered as a set of individual, discrete services.

     Dividing monolithic application into individual services

    So, if there is a bug fix in one of our services:

     Representational bug in one of service, in this case Service A of our SOA application

    and we fix that bug:

    Representational bug fix in Service A of our SOA application

       

    We can do the redeployment of that service without coordinating the deployment with other services. What we are essentially talking about is one form of microservices.

    Bug fix will result into redeployment of only Service A within our whole application

    This gives a big boost to our development agility. We don’t need to coordinate our development efforts across different development teams or even systems. We will have freedom of developing and deploying independently. One service on a weekly basis and other on quarterly. This is going to be a big advantage to the development teams.

    But, as you know, there is ever such a thing as a “free” lunch.

    The development efficiency we have gained introduces its own set of operational challenges. Let’s look at some of those.

    Service discovery in a monolith, its challenges in a distributed system, and Consul’s solution

    Monolithic applications

    Assuming two services in a single application want to talk to one another. One way is to expose a method, make it public and allow other services to call it. In a monolithic application, it is a single app, and the services would expose public functions and it would simply mean function calls across services.

    Subsystems talk to each other via function call within our monolithic application

    As this is a function call within a process, it has happened in-memory. Thus, it’s fast, and we need not worry about how our data was moved and if it was secure or not.

    Distributed Systems

    In the distributed world, service A is no longer delivered as the same application as service B. So, how does service A finds service B if it wants to talk to B?

    Service A tries to find Service B to establish communication

    Service A might not even be on the same machine as service B. So, there is a network in play. And it is not as fast and there is a latency that we can measure on the lines of milliseconds, as compared to nanoseconds of a simple function call.

    Challenges

    As we already know by now, two services on a distributed system have to discover one-another to interact. One of the traditional ways of solving this is by using load balancers.

    A load balancer sits between services to allow them to talk to each other

    Load balancers would sit in front of each service with a static IP known to all other services.

    A load balancer between two services allows two way traffic

    This gives an ability to add multiple instances of the same service behind the load balancer and it would direct the traffic accordingly. But this load balancer IP is static and hard-coded within all other services, so services can skip discovery.

     Load balancers allow communication between multiple instances of same service

    The challenge is now to maintain a set of load balancers for each individual services. And we can safely assume, there was originally a load balancer for the whole application as well. The cost and effort for maintaining these load balancers have increased.

    With load balancers in front of the services, they are a single point of failures. Even when we have multiple instances of service behind the load balancer if it is down our service is down. No matter how many instances of that service are running.

    Load balancers also increase the latency of inter-service communication. If service A wish to talk to service B, request from A will have to first talk to the load balancer of service B and then reach B. The response from B will also have to go through the same drill.

    Maintaining an entry of a service instances on an application-wide load balancer

    And by nature, load balancers are manually managed in most cases. If we add another instance of service, it will not be readily available. We will need to register that service into the load balancer to make it accessible to the world. This would mean manual effort and time.

    Consul’s Solutions

    Consul’s solution to service discovery problem in distributed systems is a central service registry.

    Consul maintains a central registry which contains the entry for all the upstream services. When a service instance starts, it is registered on the central registry. The registry is populated with all the upstream instances of the service.

     Consul’s Service Registry helps Service A find Service B and establish communication

    When a service A wants to talk to service B, it will discover and communicate with B by querying the registry about the upstream service instances of B. So, instead of talking to a load balancer, the service can directly talk to the desired destination service instance.

    Consul also provides health-checks on these service instances. If one of the service instances or service itself is unhealthy or fails its health-check, the registry would then know about this scenario and would avoid returning the service’s address. The work that load-balancer would do is handled by the registry in this case.

    Also, if there are multiple instances of the same service, Consul would send the traffic randomly to different instances. Thus, leveling the load among different instances.

    Consul has handled our challenges of failure detection and load distribution across multiple instances of services without a necessity of deploying a centralized load balancer.

    Traditional problem of slow and manually managed load balancers is taken care of here. Consul programmatically manages registry, which gets updated when any new service registers itself and becomes available for receiving traffic.

    This helps with scaling the services with ease.

    Configuration Management in a monolith, its challenges in a distributed environment, and Consul’s solution

    Monolithic Applications

    When we look at the configuration for a monolithic application, they tend to be somewhere along the lines of giant YAML, XML or JSON files. That configuration is supposed to configure the entire application.

    Single configuration file shared across different parts of our monolithic application

    Given a single file, all of our subsystems in our monolithic application would now consume the configuration from the same file. Thus creating a consistent view of all our subsystems or services.

    If we wish to change the state of the application using configuration update, it would be easily available to all the subsystems. The new configuration is simultaneously consumed by all the components of our application.

    Distributed Systems

    Unlike monolith, distributed services would not have a common view on configuration. The configuration is now distributed and there every individual service would need to be configured separately.

     A copy of application configuration is distributed across different services

    Challenges in Distributed Systems

    • Configuration is to be spread across different services. Maintaining consistency between the configuration on different services after each update is a challenge.
    • Moreover, the challenge grows when we expect the configuration to be updated dynamically.

    Consul’s Solutions

    Consul’s solution for configuration management in distributed environment is the central Key-Value store.

    Consul’s KV store allows seamless configuration mapping on each service

    Consul solves this challenge in a unique way. Instead of spreading the configuration across different distributed service as configuration pieces, it pushes the whole configuration to all the services and configures them dynamically on the distributed system.

    Let’s take an example of state change in configuration. The changed state is pushed across all the services in real-time. The configuration is consistently present with all the services.

    Network segmentation in a monolith, its challenges in distributed systems, and Consul’s solutions

    Monolithic Applications

    When we look at our classic monolithic architecture, the network is typically divided in three different zones.

    The first zone in our network is publicly accessible. The traffic coming to our application via the internet and reaching our load balancers.

    The second zone is the traffic from our load balancers to our application. Mostly an internal network zone without direct public access.

    The third zone is the closed network zone, primarily designated for data. This is considered to be an isolated zone.

    Different network zones in typical application

    Only the load balancers zone can reach into the application zone and only the application zone can reach into the data zone. It is a straightforward zoning system, simple to implement and manage.

    Distributed Systems

    The pattern changes drastically for distributed services.

    Complex pattern of network traffic and routes across different services

       

    Within our application network zone, there are multiple services which talk to each other within this network, making it a complicated traffic pattern.

    Challenges

    • The primary challenge is that the traffic is not in any sequential flow. Unlike monolithic architecture, where the flow was defined from load balancers to the application and application to data.
    • Depending on the access pattern we want to support, the traffic might come from different endpoints and reaching different services.
    Client essentially talks to each service within the application directly or indirectly
     SOA demands control over trusted and untrusted sources of traffic
    • Controlling the flow of traffic and segmenting the network into groups or chunks will become a bigger issue. Also, making sure we have strict rules that guide us with partitioning the network based on who should be allowed to talk to whom and vice versa is also vital.

    Consul’s Solutions

    Consul’s solution to overall network segmentation challenge in distributed systems is by implementing service graph and mutual TLS.

    Service-level policy enforcement to define traffic pattern and segmentation using Consul

    Consul solves the problem of network segmentation by centrally managing the definition around who can talk to whom. Consul has a dedicated feature for this called Consul Connect.

    Consul Connect enrolls these policies of inter-service communication that we desire and implements it as part of the service graph. So, a policy might say service A can talk to service B, but B cannot talk to C, for example.

    The higher benefit of this is, it is not IP restricted. Rather it’s service level. This makes it scalable. The policy will be enforced on all instances of service and there will be no hard bound firewall rule specific to a service’s IP. Making us independent of the scale of our distributed network.

    Consul Connect also handles service identity using popular TLS protocol. It distributes the TLS certificate associated with a service.

    These certificates help other services securely identify each other. TLS is also helpful in making the communication between the services secure. This makes for trusted network implementation.

    Consul enforces TLS using an agent-based proxy attached to each service instance. This proxy acts as a sidecar. Use of proxy, in this case, prevents us from making any change into the code of original service.

    This allows for the higher level benefit of enforcing encryptions on data at rest and data in transit. Moreover, it will assist with fulfilling compliances required by laws around privacy and user identity.

    Basic Architecture of Consul

    Consul is a distributed and highly available system.  

    Consul is shipped as a single binary download for all popular platforms. The executable can run as a client as well as server.

    Each node that provides services to Consul runs a Consul agent. Each of these agents talk to one or more Consul servers.

     Basic Architecture of Consul

    Consul agent is responsible for health-checking the services on the node as the health-check of the node itself. It is not responsible for service discovery or maintaining key/value data.

    Consul data and config storage and replication happens via Consul Servers.

    Consul can run with single server, but it is recommended by HashiCorp to run a set of 3 to 5 servers to avoid failures. As all the data is stored at Consul server side, with a single server, the failure could cause a data loss.

    With multi-servers cluster, they elect a leader among themselves. It is also recommended by HashiCorp to have cluster of servers per datacenter.

    During the discovery process, any service in search for other service can query the Consul servers or even Consul agents. The Consul agents forward the queries to Consul servers automatically.

    Consul Agent sits on a node and talks to other agents on the network synchronizing all service-level information

    If the query is cross-datacenter, the queries are forwarded by the Consul server to the remote Consul servers. The results from remote Consul servers are returned to the original Consul server.

    Getting Started with Consul

    This section is dedicated to closely looking at Consul as a tool, with some hands-on experience.

    Download and Install

    As discussed above, Consul ships as a single binary downloaded from HashiCorps website or from Consul’s GitHub repo releases section.

    This binary exhibits the role of both the Consul Server and Consul Client Agent. One can run them in either configuration.

    You can download Consul from here – Download Consul page.

    Various download options for Consul on different operating systems

    We will download Consul on command line using the link from download page

    $ wget https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip -O consul.zip

--2019-03-10 00:14:07--  https://releases.hashicorp.com/consul/1.4.3/consul_1.4.3_linux_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.37.183, 2a04:4e42:9::439
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.37.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34777003 (33M) [application/zip]
Saving to: ‘consul.zip’

consul.zip             100%[============================>]  33.17M  4.46MB/s    in 9.2s    

2019-03-10 00:14:17 (3.60 MB/s) - ‘consul.zip’ saved [34777003/34777003]

    Unzip the downloaded zip file.  

    $ unzip consul.zip

Archive:  consul.zip
  inflating: consul

    Add it to PATH.

    $ export PATH="$PATH:/path/to/consul"

    Use Consul

    Once you unzip the compressed file and put the binary under your PATH, you can run it like this.

    $ consul agent -dev

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: 'ef46ebb7-3496-346f-f67a-30117cfec0ad'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:38:01 [DEBUG] agent: Using random ID "ef46ebb7-3496-346f-f67a-30117cfec0ad" as node ID
    2019/03/04 00:38:01 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:ef46ebb7-3496-346f-f67a-30117cfec0ad Address:127.0.0.1:8300}]
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:38:01 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:38:01 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:38:01 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:38:01 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:38:01 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:38:01 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:38:01 [DEBUG] raft: Vote granted from ef46ebb7-3496-346f-f67a-30117cfec0ad in term 2. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:38:01 [INFO] consul: cluster leadership acquired
    2019/03/04 00:38:01 [INFO] consul: New leader elected: devcube
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:38:01 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:38:01 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:38:01 [INFO] agent: started state syncer
    2019/03/04 00:38:01 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:38:01 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:38:01 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [INFO] agent: Synced node info
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync
    2019/03/04 00:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:38:01 [DEBUG] agent: Node info in sync

    This will start the agent in development mode.

    Consul Members

    While the above command is running, you can check for all the members in Consul’s network.

    $ consul members

Node     Address         Status  Type    Build  Protocol  DC   Segment
devcube  127.0.0.1:8301  alive   server  1.4.0  2         dc1  <all>

    Given we only have one node running, it is treated as server by default. You can designate an agent as a server by supplying server as command line parameter or server as configuration parameter to Consul’s config.

    The output of the above command is based on the gossip protocol and is eventually consistent.

    Consul HTTP API

    For strongly consistent view of the Consul’s agent network, we can use HTTP API provided out of the box by Consul.

    $ curl localhost:8500/v1/catalog/nodes

[
    {
        "ID": "ef46ebb7-3496-346f-f67a-30117cfec0ad",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "Meta": {
            "consul-network-segment": ""
        },
        "CreateIndex": 9,
        "ModifyIndex": 10
    }
]

    Consul DNS Interface

    Consul also provides a DNS interface to query nodes. It serves DNS on 8600 port by default. That port is configurable.

    $ dig @127.0.0.1 -p 8600 devcube.node.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 devcube.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42215
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;devcube.node.consul.		IN	A

;; ANSWER SECTION:
devcube.node.consul.	0	IN	A	127.0.0.1

;; ADDITIONAL SECTION:
devcube.node.consul.	0	IN	TXT	"consul-network-segment="

;; Query time: 19 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:45:44 IST 2019
;; MSG SIZE  rcvd: 100

    Registering a service on Consul can be achieved either by writing a service definition or by sending a request over an appropriate HTTP API.

    Consul Service Definition

    Service definition is one of the popular ways of registering a service. Let’s take a look at one of such service definition examples.

    To host our service definitions we will add a configuration directory, conventionally names as consul.d – ‘.d’ represents that there are set of configuration files under this directory, instead of single config under name consul.

    $ mkdir ./consul.d

    Write the service definition for a fictitious Django web application running on port 80 on localhost.

    $ echo '{"service": {"name": "web", "tags": ["django"], "port": 80}}' \
    > ./consul.d/web.json

    To make our consul agent aware of this service definition, we can supply the configuration directory to it.

    $ consul agent -dev -config-dir=./consul.d

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.4.2'
           Node ID: '810f4804-dbce-03b1-056a-a81269ca90c1'
         Node name: 'devcube'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2019/03/04 00:55:28 [DEBUG] agent: Using random ID "810f4804-dbce-03b1-056a-a81269ca90c1" as node ID
    2019/03/04 00:55:28 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:810f4804-dbce-03b1-056a-a81269ca90c1 Address:127.0.0.1:8300}]
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.1
    2019/03/04 00:55:28 [INFO] serf: EventMemberJoin: devcube 127.0.0.1
    2019/03/04 00:55:28 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)
    2019/03/04 00:55:28 [DEBUG] agent/proxy: managed Connect proxy manager started
    2019/03/04 00:55:28 [INFO] consul: Handled member-join event for server "devcube.dc1" in area "wan"
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2019/03/04 00:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2019/03/04 00:55:28 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2019/03/04 00:55:28 [INFO] agent: started state syncer
    2019/03/04 00:55:28 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)
    2019/03/04 00:55:28 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
    2019/03/04 00:55:28 [DEBUG] raft: Votes needed: 1
    2019/03/04 00:55:28 [DEBUG] raft: Vote granted from 810f4804-dbce-03b1-056a-a81269ca90c1 in term 2. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Election won. Tally: 1
    2019/03/04 00:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
    2019/03/04 00:55:28 [INFO] consul: cluster leadership acquired
    2019/03/04 00:55:28 [INFO] consul: New leader elected: devcube
    2019/03/04 00:55:28 [INFO] connect: initialized primary datacenter CA with provider "consul"
    2019/03/04 00:55:28 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small
    2019/03/04 00:55:28 [INFO] consul: member 'devcube' joined, marking health alive
    2019/03/04 00:55:28 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:28 [INFO] agent: Synced service "web"
    2019/03/04 00:55:28 [DEBUG] agent: Node info in sync
    2019/03/04 00:55:29 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
    2019/03/04 00:55:29 [DEBUG] agent: Service "web" in sync
    2019/03/04 00:55:29 [DEBUG] agent: Node info in sync

    The relevant information in the log here are the sync statements related to the “web” service. Consul agent as accepted our config and synced it across all nodes. In this case one node.

    Consul DNS Service Query

    We can query the service with DNS, as we did with node. Like so:

    $ dig @127.0.0.1 -p 8600 web.service.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51488
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.		IN	A

;; ANSWER SECTION:
web.service.consul.	0	IN	A	127.0.0.1

;; ADDITIONAL SECTION:
web.service.consul.	0	IN	TXT	"consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:59:32 IST 2019
;; MSG SIZE  rcvd: 99

    We can also query DNS for service records that give us more info into the service specifics like port and node.

    $ dig @127.0.0.1 -p 8600 web.service.consul SRV

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 712
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;web.service.consul.		IN	SRV

;; ANSWER SECTION:
web.service.consul.	0	IN	SRV	1 1 80 devcube.node.dc1.consul.

;; ADDITIONAL SECTION:
devcube.node.dc1.consul. 0	IN	A	127.0.0.1
devcube.node.dc1.consul. 0	IN	TXT	"consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 00:59:43 IST 2019
;; MSG SIZE  rcvd: 142

    You can also use the TAG that we supplied in the service definition to query a specific tag:

    $ dig @127.0.0.1 -p 8600 django.web.service.consul

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1 -p 8600 django.web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12278
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;django.web.service.consul.	IN	A

;; ANSWER SECTION:
django.web.service.consul. 0	IN	A	127.0.0.1

;; ADDITIONAL SECTION:
django.web.service.consul. 0	IN	TXT	"consul-network-segment="

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Mar 04 01:01:17 IST 2019
;; MSG SIZE  rcvd: 106

    Consul Service Catalog Over HTTP API

    Service could similarly be queried using HTTP API:

    $ curl http://localhost:8500/v1/catalog/service/web

[
    {
        "ID": "810f4804-dbce-03b1-056a-a81269ca90c1",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "NodeMeta": {
            "consul-network-segment": ""
        },
        "ServiceKind": "",
        "ServiceID": "web",
        "ServiceName": "web",
        "ServiceTags": [
            "django"
        ],
        "ServiceAddress": "",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {},
        "ServicePort": 80,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 10,
        "ModifyIndex": 10
    }
]

    We can filter the services based on health-checks on HTTP API:

    $ curl http://localhost:8500/v1/catalog/service/web?passing

[
    {
        "ID": "810f4804-dbce-03b1-056a-a81269ca90c1",
        "Node": "devcube",
        "Address": "127.0.0.1",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "127.0.0.1",
            "wan": "127.0.0.1"
        },
        "NodeMeta": {
            "consul-network-segment": ""
        },
        "ServiceKind": "",
        "ServiceID": "web",
        "ServiceName": "web",
        "ServiceTags": [
            "django"
        ],
        "ServiceAddress": "",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {},
        "ServicePort": 80,
        "ServiceEnableTagOverride": false,
        "ServiceProxyDestination": "",
        "ServiceProxy": {},
        "ServiceConnect": {},
        "CreateIndex": 10,
        "ModifyIndex": 10
    }
]

    Update Consul Service Definition

    If you wish to update the service definition on a running Consul agent, it is very simple.

    There are three ways to achieve this. You can send a SIGHUP signal to the process, reload Consul which internally sends SIGHUP on the node or you can call HTTP API dedicated to service definition updates that will internally reload the agent configuration.

    $ ps aux | grep [c]onsul

pranav   21289  2.4  0.3 177012 54924 pts/2    Sl+  00:55   0:22 consul agent -dev -config-dir=./consul.d

    Send SIGHUP to 21289

    $ kill -SIGHUP 21289

    Or reload Consul

    $ consul reload

    Configuration reload triggered

    You should see this in your Consul log.

    ...
    2019/03/04 01:10:46 [INFO] agent: Caught signal:  hangup
    2019/03/04 01:10:46 [INFO] agent: Reloading configuration...
    2019/03/04 01:10:46 [DEBUG] agent: removed service "web"
    2019/03/04 01:10:46 [INFO] agent: Synced service "web"
    2019/03/04 01:10:46 [DEBUG] agent: Node info in sync
...

    Consul Web UI

    Consul provides a beautiful web user interface out-of-the-box. You can access it on port 8500.

    In this case at http://localhost:8500. Let’s look at some of the screens.

    The home page for the Consul UI is services with all the relevant information related to a Consul agent and web service check.

    Exploring defined services on Consul Web UI

    Going into further details on a given service, we get a service dashboard with all the nodes and their health for that service.

     Exploring node-level information for each service on Consul Web U

    On each individual node, we can look at the health-checks, services, and sessions.

    Exploring node-specific health-check information, services information, and sessions information on Consul Web UI

    Overall, Consul Web UI is really impressive and a great companion for the command line tools that Consul provides.

    How is Consul Different From Zookeeper, doozerd, and etcd?

    Consul has a first-class support for service discovery, health-check, key-value storage, multi data centers.

    Zookeeper, doozerd, and etcd are primarily based on key-value store mechanism. To achieve something beyond such key-value, store needs additional tools, libraries, and custom development around them.

    All these tools, including Consul, uses server nodes that require quorum of nodes to operate and are strongly consistent.

    More or less, they all have similar semantics for key/value store management.

    These semantics are attractive for building service discovery systems. Consul has out-of the box support for service discovery, which the other systems lack at.

    A service discovery systems also requires a way to perform health-checks. As it is important to check for service’s health before allowing others to discover it. Some systems use heartbeats with periodic updates and TTL. The work for these health checks grows with scale and requires fixed infra. The failure detection window is as least as long as TTL.

    Unlike Zookeeper, Consul has client agents sitting on each node in the cluster, talking to each other in gossip pool. This allows the clients to be thin, gives better health-checking ability, reduces client-side complexity, and solves debugging challenges.

    Also, Consul provides native support for HTTP or DNS interfaces to perform system-wide, node-wide, or service-wide operations. Other systems need those being developed around the  exposed primitives.

    Consul’s website gives a good commentary on comparisons between Consul and other tools.

    Open Source Tools Around HashiCorp Consul

    HashiCorp and the community has built several tools around Consul

    These tools are built and maintained by the HashiCorp developers:

    Consul Template (3.3k stars) – Generic template rendering and notifications with Consul. Template rendering, notifier, and supervisor for @hashicorp Consul and Vault data. It provides a convenient way to populate values from Consul into the file system using the consul-template daemon.

    Envconsul (1.2k stars) – Read and set environmental variables for processes from Consul. Envconsul provides a convenient way to launch a subprocess with environment variables populated from HashiCorp Consul and Vault.

    Consul Replicate (360 stars) – Consul cross-DC KV replication daemon. This project provides a convenient way to replicate values from one Consul datacenter to another using the consul-replicate daemon.

    Consul Migrate – Data migration tool to handle Consul upgrades to 0.5.1+.

    The community around Consul has also built several tools to help with registering services and managing service configuration, I would like to mention some of the popular and well-maintained ones –

    Confd (5.9k stars) – Manage local application configuration files using templates and data from etcd or consul.

    Fabio (5.4k stars) – Fabio is a fast, modern, zero-conf load balancing HTTP(S) and TCP router for deploying applications managed by consul. Register your services in consul, provide a health check and fabio will start routing traffic to them. No configuration required.

    Registrator (3.9k stars) – Service registry bridge for Docker with pluggable adapters. Registrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online.

    Hashi-UI (871 stars) – A modern user interface for HashiCorp Consul & Nomad.

    Git2consul (594 stars) – Mirrors the contents of a git repository into Consul KVs. git2consul takes one or many git repositories and mirrors them into Consul KVs. The goal is for organizations of any size to use git as the backing store, audit trail, and access control mechanism for configuration changes and Consul as the delivery mechanism.

    Spring-cloud-consul (503 stars) – This project provides Consul integrations for Spring Boot apps through autoconfiguration and binding to the Spring Environment and other Spring programming model idioms. With a few simple annotations, you can quickly enable and configure the common patterns inside your application and build large distributed systems with Consul based components.

    Crypt (453 stars) – Store and retrieve encrypted configs from etcd or consul.

    Mesos-Consul (344 stars) – Mesos to Consul bridge for service discovery. Mesos-consul automatically registers/deregisters services run as Mesos tasks.

    Consul-cli (228 stars) – Command line interface to Consul HTTP API.

    Conclusion

    Distributed systems are not easy to build and setup. Maintaining them and keeping them running is an altogether another piece of work. HashiCorp Consul makes the life of engineers facing such challenges easier.

    As we went through different aspects of Consul, we learnt how straightforward it would become for us to develop and deploy application with distributed or microservices architecture.

    Robust production ready code (most crucial), well written detailed documentation, user friendliness, and solid community, helps in adopting HashiCorp Consul and introducing it in our technology stack straight forward.

    We hope it was an informative ride on the journey of Consul. Our journey has not yet ended, this was just the first half. We will meet you again with the second part of this article that walks us through practical example close to real-life applications. Find the Practical Guide To HashiCorp Consul – Part 2  here.

    Let’s us know what you would like to hear from us more or if you have any questions around the topic, we will be more than happy to answer those.

    Reference

  • Making Your Terminal More Productive With Z-Shell (ZSH)

    When working with servers or command-line-based applications, we spend most of our time on the command line. A good-looking and productive terminal is better in many aspects than a GUI (Graphical User Interface) environment since the command line takes less time for most use cases. Today, we’ll look at some of the features that make a terminal cool and productive.

    You can use the following steps on Ubuntu 20.04. If you are using a different operating system, your commands will likely differ. If you’re using Windows, you can choose between Cygwin, WSL, and Git Bash.

    Prerequisites

    Let’s upgrade the system and install some basic tools needed.

    sudo apt update && sudo apt upgrade
sudo apt install build-essential curl wget git

    Z-Shell (ZSH)

    Zsh is an extended Bourne shell with many improvements, including some features of Bash and other shells.

    Let’s install Z-Shell:

    sudo apt install zsh

    Make it our default shell for our terminal:

    chsh -s $(which zsh)

    Now restart the system and open the terminal again to be welcomed by ZSH. Unlike other shells like Bash, ZSH requires some initial configuration, so it asks for some configuration options the first time we start it and saves them in a file called .zshrc in the home directory (/home/user) where the user is the current system user.

    For now, we’ll skip the manual work and get a head start with the default configuration. Press 2, and ZSH will populate the .zshrc file with some default options. We can change these later.  

    The initial configuration setup can be run again as shown in the below image:

    Oh-My-ZSH

    Oh-My-ZSH is a community-driven, open-source framework to manage your ZSH configuration. It comes with many plugins and helpers. It can be installed with one single command as below.

    Installation

    sh -c "$(wget https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"

    It’d take a backup of our existing .zshrc in a file zshrc.pre-oh-my-zsh, so whenever you uninstall it, the backup would be restored automatically.

    Font

    A good terminal needs some good fonts, we’d use Terminess nerd font to make our terminal look awesome, which can be downloaded here. Once downloaded, extract and move them to ~/.local/share/fonts to make them available for the current user or to /usr/share/fonts to be available for all the users.

    tar -xvf Terminess.zip
mv *.ttf ~/.local/share/fonts

    Once the font is installed, it will look like:

    Among all the things Oh-My-ZSH provides, 2 things are community favorites, plugins, and themes.

    Theme

    My go-to ZSH theme is powerlevel10k because it’s flexible, provides everything out of the box, and is easy to install with one command as shown below:

    git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k

    To set this theme in .zshrc:

    Close the terminal and start it again. Powerlevel10k will welcome you with the initial setup, go through the setup with the options you want. You can run this setup again by executing the below command:

    p10k configure

    Tools and plugins we can’t live without

    Plugins can be added to the plugins array in the .zshrc file. For all the plugins you want to use from the below list, add those to the plugins array in the .zshrc file like so:

    ZSH-Syntax-Highlighting

    This enables the highlighting of commands as you type and helps you catch syntax errors before you execute them:

    As you can see, “ls” is in green but “lss” is in red.

    Execute below command to install it:

    git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting

    ZSH Autosuggestions

    This suggests commands as you type based on your history:

    The below command is how you can install it by cloning the git repo:

    git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

    ZSH Completions

    For some extra Zsh completion scripts, execute below command

    git clone https://github.com/zsh-users/zsh-completions ${ZSH_CUSTOM:=~/.oh-my-zsh/custom}/plugins/zsh-completions

    autojump

    It’s a faster way of navigating the file system; it works by maintaining a database of directories you visit the most. More details can be found here.

    sudo apt install autojump

    You can also use the plugin Z as an alternative if you’re not able to install autojump or for any other reason.

    Internal Plugins

    Some plugins come installed with oh-my-zsh, and they can be included directly in .zshrc file without any installation.

    copyfile

    It copies the content of a file to the clipboard.

    copyfile test.txt

    copypath

    It copies the absolute path of the current directory to the clipboard.

    copybuffer

    This plugin copies the command that is currently typed in the command prompt to the clipboard. It works with the keyboard shortcut CTRL + o.

    sudo

    Sometimes, we forget to prefix a command with sudo, but that can be done in just a second with this plugin. When you hit the ESC key twice, it will prefix the command you’ve typed in the terminal with sudo.

    web-search

    This adds some aliases for searching with Google, Wikipedia, etc. For example, if you want to web-search with Google, you can execute the below command:

    google oh my zsh

    Doing so will open this search in Google:

    More details can be found here.

    Remember, you’d have to add each of these plugins in the .zshrc file as well. So, in the end, this is how the plugins array in .zshrc file should look like:

    plugins=(
        zsh-autosuggestions
        zsh-syntax-highlighting
        zsh-completions
        autojump
        copyfile
        copydir
        copybuffer
        history
        dirhistory
        sudo
        web-search
        git
)

    You can add more plugins, like docker, heroku, kubectl, npm, jsontools, etc., if you’re a developer. There are plugins for system admins as well or for anything else you need. You can explore them here.

    Enhancd

    Enhancd is the next-gen method to navigate file system with cli. It works with a fuzzy finder, we’ll install it fzf for this purpose.

    sudo apt install fzf

    Enhancd can be installed with zplug plugin manager for Zsh, so first we’ll install zplug with the below command:

    $ curl -sL --proto-redir -all,https https://raw.githubusercontent.com/zplug/installer/master/installer.zsh | zsh

    Append the following to .zshrc:

    source ~/.zplug/init.zsh
zplug load

    Now close your terminal, open it again, and use zplug to install enhanced

    zplug "b4b4r07/enhancd", use:init.sh

    Aliases

    As a developer, I need to execute git commands many times a day, typing each command every time is too cumbersome, so we can use aliases for them. Aliases need to be added .zshrc, and here’s how we can add them.

    alias gs='git status'
alias ga='git add .'
alias gf='git fetch'
alias gr='git rebase'
alias gp='git push'
alias gd='git diff'
alias gc='git commit'
alias gh='git checkout'
alias gst='git stash'
alias gl='git log --oneline --graph'

    You can add these anywhere in the .zshrc file.

    Colorls

    Another tool that makes you say wow is Colorls. This tool colorizes the output of the ls command. This is how it looks once you install it:

    It works with Ruby, below is how you can install both Ruby and Colors:

    sudo apt install ruby ruby-dev ruby-colorize
sudo gem install colorls

    Now, restart your terminal and execute the command colors in your terminal to see the magic!

    Bonus – We can add some aliases as well if we want the same output of Colorls when we execute the command ls. Note that we’re adding another alias for ls to make it available as well.

    alias cl='ls'
alias ls='colorls'
alias la='colorls -a'
alias ll='colorls -l'
alias lla='colorls -la'

    These are the tools and plugins I can’t live without now, Let me know if I’ve missed anything.

    Automation

    Do you wanna repeat this process again, if let’s say, you’ve bought a new laptop and want the same setup?

    You can automate all of this if your answer is no, and that’s why I’ve created Project Automator. This project does a lot more than just setting up a terminal: it works with Arch Linux as of now but you can take the parts you need and make it work with almost any *nix system you like.

    Explaining how it works is beyond the scope of this article, so I’ll have to leave you guys here to explore it on your own.

    Conclusion

    We need to perform many tasks on our systems, and using a GUI(Graphical User Interface) tool for a task can consume a lot of your time, especially if you repeat the same task on a daily basis like converting a media stream, setting up tools on a system, etc.

    Using a command-line tool can save you a lot of time and you can automate repetitive tasks with scripting. It can be a great tool for your arsenal.

  • How to Load Unstructured Data into Apache Hive

    In today’s world, a lot of data is being generated daily. To process data that is large and very complex, traditional tools can’t be used. Huge volumes of complex data is simply called Big Data. Converting this raw data into meaningful insights, organizations can make better decisions with their products. We need a dedicated tool to help this raw data to be converted into meaningful data or knowledge. Thankfully, there are certain tools that can help.

    Hadoop is one of the most popular frameworks used to process and store Big Data. Hive, in turn, is a tool that is designed to be used alongside Hadoop. In the blog, we are going to discuss the different ways we can load semi-structured and unstructured data into Hive. We will also be discussing what Hive is and how it works. How does the performance of Hive differ from working with structured vs. semi-structured vs. unstructured data?

    What is Hive?

    Hive is a data warehousing infrastructure tool developed on top of the Hadoop Distributed File System(HDFS). Hive can be used on top of any DFS.) Hive uses Hive query language(HQL), which is very much similar to structured query language(SQL). If you are familiar with SQL, then it is much easier to get started with HQL.

    It is used for data querying and analysis over a large amount of data distributed over the Hadoop Distributed File System(HDFS). Hive supports reading, writing, and managing a large amount of data that is residing in the Hadoop Distributed File System(HDFS). Hive is mostly used for structured data but in this blog, we will see how we can load unstructured data.

    Initially, Hive was developed at Facebook(Meta), and later it became an open-source project of Apache Software Foundation.

    How does Hive work?

    Source – AnalyticsVidya

    Hive was created to allow non-programmers familiar with SQL to work with large datasets, using an HQL interface that is similar to SQL interface. Traditional databases are designed for small or medium datasets and not large ones. But Hive uses a distributed file system and batch processing to process large datasets very efficiently.

    Hive transforms HQL queries into one or more Map-Reduce jobs or Tez jobs, and then these jobs run on Hadoop’s scheduler, YARN. Basically, HQL is an abstraction over Map-Reduce programs. After the execution of the job/query, the resulting data is stored in HDFS.

    What is SerDe in Hive?

    SerDe is short for “Serializer and Deserializer” in Hive. It’s going to be an important topic for this blog. So, you should have a basic understanding of what SerDe is and how it works.

    If not, don’t worry, first of all, we will understand what Serialization and Deserialization is. When an object is converted into a byte stream, it’s into a binary format so that the object can be transmitted over a network or written into persistent storage like HDFS. This process of converting data objects into byte streams is called Serialization.

    Now, we can transmit data objects or write data objects into persistent storage. But how can we receive transmitted data over the network again in a meaningful way because we will not be able to understand binary data properly? So, the process of converting byte stream or binary data back into objects is called Deserialization.

    In Hive, tables are converted into row objects and row objects are written into HDFS using a Built-in Hive Serializer. And these row objects are converted back into tables using a Built-in Hive Deserializer.

    Built-in SerDes:

    • Avro (Hive 0.9.1 and later)

    • ORC (Hive 0.11 and later)

    • RegEx

    • Thrift

    • Parquet (Hive 0.13 and later)

    • CSV (Hive 0.14 and later)

    • JsonSerDe (Hive 0.12 and later in hcatalog-core)

    Now, suppose we have data in a format that Hive’s Built-in SerDe can’t process. In such a scenario, we can write our own custom SerDe. Here, we will discuss how the row is converted into a table and vice versa. But writing your own custom SerDe is a complicated and complex process.

    There is another way: we can use RegexSerDe. RegexSerDe uses a regular expression to serialize and deserialize the data using regex. RegexSerDe extracts groups as columns. Here, group means regular expressions capturing groups. In a regular expression, capturing groups are a way to treat multiple characters as a single unit. Groups can be created using placing parentheses. For example, the regular expression “(velotio)” creates a single group containing the characters “v,” “e,” “l,” “o,” ”t,” “i,” and “o.”

    This is just an overview of SerDe in Hive, but you can deep dive into SerDe. Also, the following image shows the flow of How Hive reads and writes records.

    Source: Dummies.com

    Types of data :

    Big data can be classified in three ways

    Structured data:

    The data that can be organized into a well-defined structure is called Structured Data. Structured data can be easily stored, read, or transferred in the same defined structure. The best example of structured data is the table stored in Relational Databases. Tables have columns and rows that define a well-organized and fixed structure to data. Another example of structured data is an Excel file. An Excel file also has rows and columns that define a proper structure to data.

    Source: O’Reilly

    Semi-structured data:

    The data that can not be organized into a fixed structure like a table but can be represented with properties such as tags, metadata, or other markers that separate data fields are called semi-structured data. Examples of semi-structured data are JSON and XML files. JSON files contain “key” and “values” pairs, where the key is a tag, and the value is actual data to be stored.

    Source: Software Testing Help

    Unstructured data:

    The data which can not be organized into any structure is called unstructured data. The social media messages fall under the unstructured data category as they can not be organized into either a fixed structure like a table or even with tags or markers that will separate data fields. More examples of unstructured data are text files, multimedia content like images and videos.

    Source: Fluxicon

    Performance impact of working with structured vs, semi-structured vs, unstructured data 

    Storage:

    Structured data is always stored in RDBMS. Structured data have a high organization level among all three. Semi-structured data has no schema but has some properties or tags. Structured data have less organization level compared to structured data but higher organization level than unstructured data. While unstructured data has no schema, so it has the lowest organization level.

    Data manipulation:

    Data manipulation includes updating and deleting data. Consider an example where we want to update the name of a student using his roll number. Data manipulation in structured data is easy to perform as we have defined structure and we can manipulate specific records very easily. We can easily update the student’s name using his roll number in structured data. Whereas in unstructured data, there is no schema available, so it is not easy to manipulate data in unstructured data as compared to structured data.

    Searching of data:

    Searching for particular data in structured data is easy compared to searching for data in unstructured data. In unstructured data, we will need to go through all lines, and each word and searching of data in unstructured data will get complex. Searching data in semi-structured data is also easy as we just need to specify the key to get the data.

    Scaling of data:

    Scaling structured data is very hard. It can be scaled vertically by adding an existing machine’s RAM or CPU, but scaling it horizontally is hard to do. However, scaling semi-structured data and unstructured data is easy.

    Data sets we are using:

    1) Video_Games_5.json

    This dataset contains product item reviews and metadata from Amazon, including 142.8 million audits spreading over May 1996 – July 2014. Reviews (ratings, text, helpfulness votes), item metadata (depictions, category information, price, brand, and image features), and links (also viewed/also bought graphs) are all included in this dataset. The dataset represents data in a semi-structured manner.

    The following image shows one record of the entire dataset.

    {
  "reviewerID": "A2HD75EMZR8QLN",
  "asin": "0700099867",
  "reviewerName": "123",
  "helpful": [
    8,
    12
  ],
  "reviewText": "Installing the game was a struggle (because of games for windows live bugs).Some championship races and cars can only be \"unlocked\" by buying them as an addon to the game. I paid nearly 30 dollars when the game was new. I don't like the idea that I have to keep paying to keep playing.I noticed no improvement in the physics or graphics compared to Dirt 2.I tossed it in the garbage and vowed never to buy another codemasters game. I'm really tired of arcade style rally/racing games anyway.I'll continue to get my fix from Richard Burns Rally, and you should to. :)http://www.amazon.com/Richard-Burns-Rally-PC/dp/B000C97156/ref=sr_1_1?ie=UTF8&qid;=1341886844&sr;=8-1&keywords;=richard+burns+rallyThank you for reading my review! If you enjoyed it, be sure to rate it as helpful.",
  "overall": 1,
  "summary": "Pay to unlock content? I don't think so.",
  "unixReviewTime": 1341792000,
  "reviewTime": "07 9, 2012"}

    2) sparkLog.txt

    Apache Spark (https://spark.apache.org) is a unified analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning, and graph processing. Currently, Spark has been widely deployed in the industry.

    The Dataset / Log Set was collected by aggregating logs from the Spark system in a lab at CUHK, which comprises a total of 32 machines. The logs are aggregated at the machine level. Logs are provided as-is without further modification or labeling, which involve both normal and abnormal application runs.

    The dataset represents data in an un-structured manner.

    17/06/09 20:10:40 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
17/06/09 20:10:40 INFO spark.SecurityManager: Changing view acls to: yarn,curi
17/06/09 20:10:40 INFO spark.SecurityManager: Changing modify acls to: yarn,curi
17/06/09 20:10:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, curi); users with modify permissions: Set(yarn, curi)
17/06/09 20:10:41 INFO spark.SecurityManager: Changing view acls to: yarn,curi
17/06/09 20:10:41 INFO spark.SecurityManager: Changing modify acls to: yarn,curi
17/06/09 20:10:41 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, curi); users with modify permissions: Set(yarn, curi)
17/06/09 20:10:41 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/06/09 20:10:41 INFO Remoting: Starting remoting
17/06/09 20:10:41 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@mesos-slave-07:55904]

    Now, we will look into different ways of loading unstructured data into Hive.

    How to load semi-structured data into Hive?

    1) Using Spark

    If you are aware of Spark, loading semi-structured data into the spark is very easy. Spark can read JSON files, XML files, and convert them into Spark DataFrame. In Spark, DataFrame is a distributed collection of data that is organized into columns and rows. It is logically similar to tables in relational databases.

    Now, we have our semi-structured data in an organized way. We can now write this organized DataFrame into Hive as a table from Spark.

    Below is the code to read the JSON file and write it as a table in Hive.

    ‍

    from pyspark.sql import SparkSession

## creating sparkSession to get entrypoint to spark application
sparkSession = SparkSession\
 .builder\
 .appName('Write_table_to_hive')\
 .enableHiveSupport()\
 .getOrCreate()

## reading data from dataset "Video_Games_5.json"
GamesReviewDataFrame = sparkSession.read.format("json") \
          .format("json") \
          .option("path", "/home/velotio/Downloads/UnstructuredData/Video_Games_5.json")\
          .load()

## we can modify data the way we want to represent in table here 
GamesReviewDataFrame.show()

## writing dataframe "GamesReviewDataFrame" as a table in HIVE.
GamesReviewDataFrame.write.saveAsTable("GameReviewTable")

sparkSession.stop()

    Output for above code:

    As you can see in the output, a few records of the DataFrame are displayed in an organized table.

    2) Using built-in SerDe, JSON SerDe

    Hive provides us with a few built-in SerDe. Using this built-in SerDe, we can load data into Hive. In our case, we have used the Video_Games_5.json file as a dataset for semi-structured data, which is a JSON file. So, we will be using built-in JsonSerDe to load Video_Games_5.json data into Hive. This JsonSerDe can be used to read data in JSON format.

    We will need to add JsonSerDe.jar to Hive.

    You can download JsonSerDe here.

    1) Copy dataset Video_Games_5.json from the local file system to the docker container.

    To load data into the Hive table, we need to copy the dataset Video_Games_5.json into HDFS. As we are running HDFS and Hive in the docker container, we will need to copy this dataset from the Local File System to the docker container.

    docker cp /home/velotio/Downloads/UnstructuredData/Video_Games_5.json 0fde53f41006:/aniket

    2) Copy dataset Video_Games_5.json from a docker container to the HDFS file system.

    ls
ls /aniket
hdfs dfs -ls /
hdfs dfs -ls /aniket
hdfs dfs -put /aniket/Video_Games_5.json /aniket
hdfs dfs -ls /aniket

    3) Copy json-serde.jar from the local file system to the docker container

    To use JsonSerDe, add the json-serde.jar file to Hive so that Hive can use it.

    We store this json-serde.jar file to HDFS storage where our dataset is also present. As Hive is running on top of HDFS, we can access the HDFS path from Hive. But to store json-serde.jar on HDFS, the file needs to be present in the docker container. For that, we copy json-serde.jar to the docker container first.

    docker cp /home/velotio/Downloads/json-serde-1.3.7.3.jar 0fde53f41006:/aniket

    4) Copy json-serde.jar from a docker container to the HDFS file system.

    ls /aniket
hdfs dfs -ls /aniket

hdfs dfs -put /aniket/json-serde-1.3.7.3.jar /aniket
hdfs dfs -ls /aniket

    5) Add json-serde.jar file to Hive

    ADD JAR hdfs:///aniket/json-serde-1.3.7.3.jar;

    6) Create Hive table GameReviews

    To load data into Hive and define the structure of our data, we must create a table in Hive before loading the data. The table holds the data in an organized manner.

    While creating the table, we are specifying “row format serde,” which tells Hive to use the provided SerDe for reading and writing Hive data.

    create table GameReviews(
reviewerID string,
asin string,
reviewerName string,
helpful array<int>,
reviewText string,
overall int,
summary string,
unixReviewTime int,
reviewTime string
)
row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
stored as textfile;

    ‍

    7) Load data from the Video_Games_5.json dataset into the table.

    We are loading data from Video_Games_5.json into the Hive table. With the help of SerDe provided while creating a table, Hive will parse this data and load it into the table.

    load data inpath '/aniket/Video_Games_5.json' into table GameReviews;

    8) Check data from the table.

    Just cross-check if the data is loaded properly into the table.

    select reviewerID,asin,reviewerName,overall from GameReviews limit 10;

    How to load unstructured data into Hive ?

    1. Using Regex SerDe

    For unstructured data, the built-in SerDe can’t work with excluded RegxSerDe. To load unstructured data into Hive, we can use RegexSerde. First of all, we will need to figure out what unstructured data is useful. After knowing what data is useful, we can extract data using pattern matching. For that, we can use regular expressions. With regular expressions, we will load unstructured data of the SparkLog.txt dataset into Hive.

    In our case, we are going to use the following regular expression:

    “([0-9]{2}/[0-9]{2}/[0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2}) [a-zA-Z]* ([a-zA-Z0-9.]*): (.*)$”

    “([0-9]{2}/[0-9]{2}/[0-9]{2})”: First group in Regular Expression matches date values.

    “([0-9]{2}:[0-9]{2}:[0-9]{2})”: Second Group in Regular Expression matches timestamp values.

    “[a-zA-Z]*”: This pattern matches any string with multiple occurrences of char a to z and A to Z; this pattern will be ignored in the Hive table as we are not collecting this pattern as a group.

    “([a-zA-Z0-9.]*):”: Third group in regular expression matches with multiple occurrences of char a to z, A to Z, 0 to 9 and “.”

    “(.*)$”: Fourth and last group matches with all characters in the remaining string.

    1) Copy dataset sparkLog.txt from local file system to docker container.

    To load data into Hive table, we need dataset sparkLog.txt into HDFS. As we are running HDFS and Hive in the docker container, we will need to copy this dataset from the Local File System to the docker container first.

    docker cp /home/velotio/Downloads/UnstructuredData/sparkLog.txt 6d94029a1f34:/aniket

    2) Copy dataset sparkLog.txt from a docker container to HDFS file system.

    lsnls /aniketnhdfs dfs -ls /nhdfs dfs -ls /aniketnhdfs dfs -put /aniket/sparkLog.txt /aniketnhdfs dfs -ls /aniket

    3) Create a Hive table sparkLog.

    To load data into Hive and define the structure to our data, we must create a table in Hive before loading the data. The table holds the data in an organized manner.

    While creating the table, we are specifying “row format SerDe,” which tells Hive to use the provided SerDe for reading and writing Hive data. For RegexSerDe, we must specify serdeproperties: “input.regex” and “output.format.string.”

    create table sparkLog 
(
    datedata  string
   ,time string
   ,component string
      ,action string
)
    row format serde 'org.apache.hadoop.hive.serde2.RegexSerDe'
    with serdeproperties 
	(
		"input.regex" = "([0-9]{2}/[0-9]{2}/[0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2}) [a-zA-Z]* ([a-zA-Z0-9.]*): (.*)$",
		"output.format.string" = "%1$s %2$s %3$s %4$s"
	)
STORED AS TEXTFILE;

    4) Load data from the sparkLog.txt dataset into the table.

    We are loading data from sparkLog.txt into the Hive table. With the help of the SerDe provided while creating the table, Hive will parse this data and will load it into the table.

    load data inpath '/aniket/sparkLog.txt’ into table sparkLog;

    5) Check the data from the table.

    Cross-check if the data is loaded properly into the table.

    select * from sparkLog;

    2) Using HQL functions

    For unstructured data, we have already seen how to use RegexSerDe to load unstructured data into Hive. But what if I am not aware of regular expressions or can’t write complex regular expressions to match patterns in a string? There is another way to load unstructured data into Hive using some HQL user-defined functions.

    What we need to do is create a dummy table and load unstructured data as it is into Hive in just one column in the table named “line.” We are loading unstructured data into a dummy Hive table column named as a line. The first record of the “line” column will contain the first line of DataSet, and the second record of the line column will contain the second line of DataSet. Like this, the entire Dataset will be loaded into a dummy table.

    Now, using HQL user-defined functions on the dummy Hive table, we can write specific data to specific columns into the main table using the “insert into” statement. You should be able to extract the data that you want using HQL user-defined functions.

    1) Copy dataset sparkLog.txt from local file system to docker container

    To load data into Hive table, we need dataset sparkLog.txt into HDFS. As we are running HDFS and Hive in the docker container, we will need to copy this dataset from the local file system to the docker container.

    docker cp /home/velotio/Downloads/UnstructuredData/sparkLog.txt 6d94029a1f34:/aniket

    2) Copy the dataset sparkLog.txt from a docker container to HDFS file system.

    ls
ls /aniket
hdfs dfs -ls /
hdfs dfs -ls /aniket
hdfs dfs -put /aniket/sparkLog.txt /aniket
hdfs dfs -ls /aniket

    3) Create Hive table log

    We are creating a dummy Hive table as a log. We are specifying “row format delimited lines terminated by ‘/n’,” which tells Hive to consider default value for fields delimiter and ‘/n’ for line delimiter.

    create table if not exists log 
(
	line string
)
row format delimited
lines terminated by ‘n’
STORED AS TEXTFILE;

    4)   Load the data from sparkLog.txt dataset into the table log.

    We are loading data from sparkLog.txt into Hive table log.

    load data inpath '/aniket/sparkLog.txt’ into table log;

    5) Create Hive table log sparkLog.

    We are creating a Hive table sparkLog to keep our organized data. This organized data will be extracted from a dummy Hive table log.

    create table sparkLog 
(
   datedata  string
   ,time string
   ,component string
      ,action string
)
row format delimited
lines terminated by ‘\n’
STORED AS TEXTFILE;

    6) Parse the data from log table using a case statement and insert records into the sparkLog table.

    We are using HQL user-defined functions to get the specific data and inserting this data into our sparkLog table using insert into statement.

    insert into sparkLog select
split(line, ' ')[0] as datedata,
split(line, ' ')[1] as timedata,
split(split(line, ': ')[0],' ')[3] as component,
split(line, ': ')[1] as action
from log ;

    7) Check data from the table.

    Crosscheck if the data is loaded properly into the table.

    select * from sparkLog;

    Summary

    After going through the above blog, you might have gotten more familiarity with Hive, its architecture. how you can use different serializers and deserializers in Hive. Now, you are able to load not only structured data but also unstructured data into Hive. If you are interested in knowing more about Apache Hive, you can visit the below documentation.

    1. Hive Tutorial
    2. LanguageManual
    3. Hive Wiki Pages

  • Know Everything About Spinnaker & How to Deploy Using Kubernetes Engine

    As marketed, Spinnaker is an open-source, multi-cloud continuous delivery platform that helps you release software changes with high velocity and confidence.

    Open sourced by Netflix and heavily contributed to by Google, it supports all major cloud vendors (AWS, Azure, App Engine, Openstack, etc.) including Kubernetes.

    In this blog I’m going to walk you through all the basic concepts in Spinnaker and help you create a continuous delivery pipeline using Kubernetes Engine, Cloud Source Repositories, Container Builder, Resource Manager, and Spinnaker. After creating a sample application, we will configure these services to automatically build, test, and deploy it. When the application code is modified, the changes trigger the continuous delivery pipeline to automatically rebuild, retest, and redeploy the new version.

    What Spinnaker Provides?

    Application management and Application Deployment are its two core features.

    Application Management

    Spinnaker’s application management features can be used to view and manage your cloud resources.

    Modern tech organizations operate collections of services—sometimes referred to as “applications” or “microservices”. A Spinnaker application models this concept.

    Applications, Clusters, and Server Groups are the key concepts Spinnaker uses to describe services. Load balancers and Firewalls describe how services are exposed to users.

    Application

    • An application in Spinnaker is a collection of clusters, which in turn are collections of server groups. The application also includes firewalls and load balancers. An application represents the service which needs to be deployed using Spinnaker, all configuration for that service, and all the infrastructure on which it will run. Normally, a different application is configured for each service, though Spinnaker does not enforce that.

    Cluster

    • Clusters are logical groupings of Server Groups in Spinnaker.
    • Note: Cluster, here, does not map to a Kubernetes cluster. It’s merely a collection of Server Groups, irrespective of any Kubernetes clusters that might be included in your underlying architecture.

    Server Group

    • The base resource, the Server Group, identifies the deployable artifact (VM image, Docker image, source location) and basic configuration settings such as number of instances, autoscaling policies, metadata, etc. This resource is optionally associated with a Load Balancer and a Firewall. When deployed, a Server Group is a collection of instances of the running software (VM instances, Kubernetes pods).

    Load Balancer

    • A Load Balancer is associated with an ingress protocol and port range. Traffic is balanced among the instances present in Server Groups. Optionally, health checks can be enabled for a load balancer, with flexibility to define health criteria and specify the health check endpoint.

    Firewall

    • A Firewall defines network traffic access. It is effectively a set of firewall rules defined by an IP range (CIDR) along with a communication protocol (e.g., TCP) and port range.

    Application Deployment

    Pipeline

    • The pipeline is the key deployment management construct in Spinnaker. It consists of a sequence of actions, known as stages. Parameters can be passed from one stage to the next one in the pipeline.
    • You can start a pipeline manually, or you can configure it to be automatically triggered by an event, such as a Jenkins job completing, a new Docker image being pushed in your docker registry, a CRON type schedule, or maybe a stage in another pipeline.
    • You can configure the pipeline to emit notifications, by email, SMS or HipChat, to interested parties at various points during pipeline execution (such as on pipeline start/complete/fail).

    Stage

    • A Stage in Spinnaker is an atomic building block for a pipeline, describing an action that the pipeline will perform. You can sequence stages in a Pipeline in any order, though some stage sequences may be more common than others. There are different types of stages in Spinnaker such as Deploy, Manual Judgment, Resize, Disable,  and many more. The full list of stages and read about implementation details for each provider here.

    Deployment Strategies

    • Spinnaker supports all the cloud native deployment strategies including Red/Black (a.k.a Blue/Green), Rolling red/black and Canary deployments, etc.

    What is Spinnaker Made Of?

    Spinnaker is composed of a number of independent microservices:

    • Deck Deck is the custom browser-based GUI.
    • Gate is the API gateway. All the API calls from UI (Deck) and other API callers go to Spinnaker through Gate.
    • Orca is the orchestration engine. It handles all ad-hoc operations and pipelines.
    • Clouddriver is responsible for all mutating calls to the cloud providers and for indexing/caching all deployed resources.
    • Front50 is used to persist the metadata of applications, pipelines, projects and notifications.
    • Rosco is the bakery. It helps to create machine images for various cloud vendors (for example GCE images for GCP, AMIs for AWS, Azure VM images). It currently wraps Packer, but will be expanded to support additional mechanisms for producing images.
    • Igor is used to trigger pipelines via continuous integration jobs in systems like Jenkins and Travis CI, and it allows Jenkins/Travis stages to be used in pipelines.
    • Echo is Spinnaker’s eventing bus. It supports sending notifications (e.g. Slack, email, Hipchat, SMS), and acts on incoming webhooks from services like GitHub.
    • Fiat is Spinnaker’s authorization service. It is used to query a user’s access permissions for accounts, applications and service accounts.
    • Kayenta provides automated canary analysis for Spinnaker.
    • Halyard is Spinnaker’s configuration service. Halyard manages the lifecycle of each of the above services. It only interacts with these services during Spinnaker start-up, updates, and rollbacks.

    By default, Spinnaker binds ports accordingly for all the above mentioned microservices. For us the UI (Deck) will be exposed onto Port 9000.

    What are We Going to Do?

    • Set up your environment by launching Cloud Shell, creating a Kubernetes Engine cluster, and configuring your identity and user management scheme.
    • Download a sample application, create a Git repository, and upload it to a Cloud Source Repository.
    • Deploy Spinnaker to Kubernetes Engine using Helm.
    • Build a Docker image from the source code.
    • Create triggers to create Docker images when the source code for application changes.
    • Configure a Spinnaker pipeline to reliably and continuously deploy your application to Kubernetes Engine.
    • Deploy a code change, triggering the pipeline, and watch it roll out to production.

     Note: This blog post uses various billable components in GCP like GKE, Container Builder etc. 

    Pipeline Architecture

    To continuously deliver application updates to users, companies need an automated process that reliably builds, tests, and updates their software. Code changes should automatically flow through a pipeline that includes artifact creation, unit testing, functional testing, and production rollout. In some cases, they want a code update to apply to only a subset of their users, so that it is exercised realistically before pushing it to entire user base. If one of these canary releases proves unsatisfactory, the automated procedure must be able to quickly roll back the software changes.

    With Kubernetes Engine and Spinnaker, we can create a robust continuous delivery flow that helps us to ensure that software is shipped as quickly as it is developed and validated. Although rapid iteration is the end goal, we must first ensure that each application revision passes through a series of automated validations before becoming a candidate for production rollout. When a given change has been vetted through automation, we can also validate the application manually and conduct further pre-release testing.

    After the team decides the application is ready for production, one of the team members can approve it for production deployment.

    Application Delivery Pipeline

    We are going to build the continuous delivery pipeline shown in the following diagram.

    Prerequisites  

    • Fair bit of experience in GCP services like:  
    • GKE (Google Kubernetes Engine)
    • Google Compute
    • Google APIs
    • Cloud Source Repository
    • Container Builder
    • Cloud Storage
    • Cloud Load Balancing
    • Knowledge in K8s terminology like Services, Deployments, Pods, etc
    • Familiarity with Kubectl and Helm package manager

    Before Starting just enable the APIs needed on GCP

     Set Up a Kubernetes Cluster  

    1. Go to the Console and scroll the left panel down to Compute->Kubernetes Engine->Kubernetes Clusters.
    2. Click Create Cluster.
    3. Choose a name or leave as the default one.
    4. Under Machine Type, click Customize.
    5. Allocate at least 2 vCPU and 10GB of RAM.
    6. Change the cluster size to 2.
    7. Enable Legacy Authorization while customizing the cluster.
    8. Keep the rest of the defaults and click Create.

    In a minute or two the cluster will be created and ready to go.

    Configure identity and access management

    Create a Cloud Identity and Access Management (Cloud IAM) service account to delegate permissions to Spinnaker, allowing it to store data in Cloud Storage. Spinnaker stores its pipeline data in Cloud Storage to ensure reliability and resiliency. If our Spinnaker deployment unexpectedly fails, we can create an identical deployment in minutes with access to the same pipeline data as the original.

    1. Create the service account:

    $ gcloud iam service-accounts create spinnaker-storage-account  --display-name spinnaker-storage-account

    2.  Store the service account email address and our current project ID in environment variables for use in later commands:

    $ export SA_EMAIL=$(gcloud iam service-accounts list  --filter="displayName:spinnaker-storage-account"  --format='value(email)')
$ export PROJECT=$(gcloud info --format='value(config.project)')

    3. Bind the storage.admin role to our service account:  

    $ gcloud projects add-iam-policy-binding  $PROJECT --role roles/storage.admin --member serviceAccount:$SA_EMAI

    4. Download the service account key. We will need this key later while installing Spinnaker and we need to also upload the key to Kubernetes Engine.  

    $ gcloud iam service-accounts keys create spinnaker-sa.json --iam-account $SA_EMAIL

    Deploying Spinnaker using Helm

    In this section, we will deploy Spinnaker onto the K8s cluster via Charts with the help of K8s package manager Helm. Helm has made it very easy to deploy Spinnaker, it can be a very painful act to deploy it manually via Halyard and configure it.

    Install Helm

    1. Download and install the helm binary:

    $ wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.0-linux-amd64.tar.gz

    2. Unzip the file to your local system:

    $ tar zxfv helm-v2.9.0-linux-amd64.tar.gz$ sudo chmod +x linux-amd64/helm && sudo mv linux-amd64/helm /usr/bin/helm

    3. Grant Tiller, the server side of Helm, the cluster-admin role in your cluster:

    $ kubectl create clusterrolebinding user-admin-binding  --clusterrole=cluster-admin --user=$(gcloud config get-value account)
$ kubectl create serviceaccount tiller --namespace kube-system
$ kubectl create clusterrolebinding tiller-admin-binding  --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

    4. Grant Spinnaker the cluster-admin role so it can deploy resources across all namespaces:

    $ kubectl create clusterrolebinding --clusterrole=cluster-admin       --serviceaccount=default:default spinnaker-admin

    5. Initialize Helm to install Tiller in your cluster:

    $ helm init --service-account=tiller --upgrade
$ helm repo update

    6. Ensure that Helm is properly installed by running the following command. If Helm is correctly installed, v2.9.0 appears for both client and server.

    $ helm version

    Configure Spinnaker

    1. Create a bucket for Spinnaker to store its pipeline configuration:

    $ export PROJECT=$(gcloud info --format='value(config.project)')
$ export BUCKET=$PROJECT-spinnaker-configgsutil mb -c regional -l us-central1  gs://$BUCKET

    2. Create the configuration file:

    $ export SA_JSON=$(cat spinnaker-sa.json)
$ export PROJECT=$(gcloud info --format='value(config.project)')
$ export BUCKET=$PROJECT-spinnaker-config
$ cat > spinnaker-config.yaml <

    # Disable minio as the defaultminio:      
enabled: false 

# Configure your Docker registries here accounts:      
name: gcr       
address: https://gcr.io 
username: _json_key 
password: '$SA_JSON'
email: 1234@5678.com EOF

    Deploy the Spinnaker chart

    1. Use the Helm command-line interface to deploy the chart with the configuration set earlier. This command typically takes five to ten minutes to complete, so we will be providing a deploy timeout with ` — timeout`.
    $ helm install -n cd stable/spinnaker -f spinnaker-config.yaml --timeout  600 --version 0.3.1

    After the command completes, run the following command to set up port forwarding to the Spinnaker UI from Cloud Shell:

    $ export DECK_POD=$(kubectl get pods --namespace default -l  "component=deck" -o jsonpath="{.items[0].metadata.name}")
$ kubectl port-forward --namespace default $DECK_POD 8080:9000  >> /dev/null &

    The above command exposes the Spinnaker UI onto the local machine that we’re using to run all the commands. We can use any port of our choosing instead of 8080 in above command. Now the UI can be opened onto the url http://localhost:8080.

    Building the Docker image

    In this section, we will configure Container Builder to detect changes to the application source code, if yes then build a Docker image, and then push it to Container Registry.

    For this step we will use a sample app provided by the Google community  

    Create your source code repository

    1. Download the source code:

    $ wget https://gke-spinnaker.storage.googleapis.com/sample-app.tgz

    2. Unpack the source code:

    $ tar xzfv sample-app.tgz

    3. Change directories to source code:

    $ cd sample-app

    4. Set the username and email address for Git commits in this repository. Replace [EMAIL_ADDRESS] with Git email address, and replace [USERNAME] with Git username.  

    $ git config --global user.email "[EMAIL_ADDRESS]"
$ git config --global user.name "[USERNAME]"

    5. Make the initial commit to source code repository:

    $ git init
$ git add .
$ git commit -m "Initial commit"

    6. Create a repository to host the code:

    $ gcloud source repos create sample-app
$ git config credential.helper gcloud.sh

    7. Add our newly created repository as remote:

    $ export PROJECT=$(gcloud info --format='value(config.project)')
$ git remote add origin  https://source.developers.google.com/p/$PROJECT/r/sample-app

    8. Push the code to the new repository’s master branch:

    $ git push origin master

    9. Check that we can see our source code in the console.

    Configuring the build triggers  

    In this section, we configure Google Container Builder to build and push your Docker images every time we push Git tags to our source repository. Container Builder automatically checks out the source code, builds the Docker image from the Dockerfile in repository, and pushes that image to Container Registry.

    1. In the GCP Console, click Build Triggers in the Container Registry section.
    2. Select Cloud Source Repository and click Continue.
    3. Select your newly created sample-app repository from the list, and click Continue.
    4. Set the following trigger settings:
    5. Name:sample-app-tags
    6. Trigger type: Tag
    7. Tag (regex): v.*
    8. Build configuration: cloudbuild.yaml
    9. cloudbuild.yaml location: /cloudbuild.yaml
    10. Click Create trigger.

    From now on, whenever we push a Git tag prefixed with the letter “v” to source code repository, Container Builder automatically builds and pushes our application as a Docker image to Container Registry.

    Let’s build our first image:

    Push the first image using the following steps:

    1. Go to source code folder in Cloud Shell.

    2. Create a Git tag:

    $ git tag v1.0.0

    3. Push the tag:  

    $ git push --tags

    4. In Container Registry, click Build History to check that the build has been triggered. If not, verify the trigger was configured properly in the previous section.

    Configuring your deployment pipelines

    Now that our images are building automatically, we need to deploy them to the Kubernetes cluster.

    We deploy to a scaled-down environment for integration testing. After the integration tests pass, we must manually approve the changes to deploy the code to production services.

    Create the application

    1. In the Spinnaker UI, click Actions, then click Create Application.

    2. In the New Application dialog, enter the following fields:

    1. Name: sample
    2. Owner Email: [your email address]

    3. Click Create.

    Create service load balancers

    To avoid having to enter the information manually in the UI, use the Kubernetes command-line interface to create load balancers for the services. Alternatively, we can perform this operation in the Spinnaker UI.

    On the local machine where the code resides, run the following command from the sample-app root directory:

    $ kubectl apply -f k8s/services

    Create the deployment pipeline

    Now we create the continuous delivery pipeline. The pipeline is configured to detect when a Docker image with a tag prefixed with “v” has arrived in your Container Registry.

    1. Create a new pipeline named say “Deploy”.

    2. Go to the Config page for the pipeline that we just created and click Pipeline Actions -> Edit as JSON.

    3. Change the directory to the source code directory and update the current pipeline-deploy.json at path spinnaker/pipeline-deploy.json according to our needs.

    $ export PROJECT=$(gcloud info --format='value(config.project)')
$ sed s/PROJECT/$PROJECT/g spinnaker/pipeline-deploy.json > spinnaker/updated-pipeline-deploy.json

    4. Now in the JSON editor just copy the whole file spinnaker/updated-pipeline-deploy.json.

    5. Click on Update Pipeline and we should have an updated pipeline config now.

    6. In the Spinnaker UI, click Pipelines on the top navigation bar.

    7. Click Configure in the Deploy pipeline.

    8. The continuous delivery pipeline configuration appears in the UI:

    Running the pipeline manually

    The configuration we just created contains a trigger to start the pipeline when a new Git tag containing the prefix “v” is pushed. Now we test the pipeline by running it manually.  

    1. Return to the Pipelines page by clicking Pipelines.

    2. Click Start Manual Execution.

    3. Select the v1.0.0 tag from the Tag drop-down list, then click Run.

    4. After the pipeline starts, click Details to see more information about the build’s progress. This section shows the status of the deployment pipeline and its steps. Steps in blue are currently running, green ones have completed successfully, and red ones have failed. Click a stage to see details about it.

    5. After 3 to 5 minutes the integration test phase completes and the pipeline requires manual approval to continue the deployment.

    6. Hover over the yellow “person” icon and click Continue.

    7. Your rollout continues to the production frontend and backend deployments. It completes after a few minutes.

    8. To view the app, click Load Balancers in the top right of the Spinnaker UI.

    9. Scroll down the list of load balancers and click Default, under sample-frontend-prod.  

    10. Scroll down the details pane on the right and copy application’s IP address by clicking the clipboard button on the Ingress IP.

    11. Paste the address into the browser to view the production version of the application.

    12. We have now manually triggered the pipeline to build, test, and deploy your application. 

    Triggering the pipeline automatically via code changes

    Now let’s test the pipeline end to end by making a code change, pushing a Git tag, and watching the pipeline run in response. By pushing a Git tag that starts with “v”, we trigger Container Builder to build a new Docker image and push it to Container Registry. Spinnaker detects that the new image tag begins with “v” and triggers a pipeline to deploy the image to canaries, run tests, and roll out the same image to all pods in the deployment.

    1. Change the colour of the app from orange to blue: 

    $ sed -i 's/orange/blue/g' cmd/gke-info/common-service.go

    view rawcolor.js hosted with ❤ by GitHub

    2. Tag your change and push it to the source code repository:

    $ git commit -a -m "Change colour to blue"git tag v1.0.1git push --tags

    view rawtag_color.js hosted with ❤ by GitHub

    3. See the new build appear in the Container Builder Build History. 

    4. Click Pipelines to watch the pipeline start to deploy the image. 

    5. Observe the canary deployments. When the deployment is paused, waiting to roll out to production, start refreshing the tab that contains our application. Nine of our backends are running the previous version of your application, while only one backend is running the canary. Now we should see the new, blue version of our application appear about every tenth time we refresh. 

    6. After testing completes, return to the Spinnaker tab and approve the deployment. 

    7. When the pipeline completes, application looks like the following screenshot. Note that the colour has changed to blue because of code change, and that the Version field now reads v1.0.1. 

    8. We have now successfully rolled out your application to your entire production environment!!!!!! 

    9. Optionally, we can roll back this change by reverting the previous commit. Rolling back adds a new tag (v1.0.2), and pushes the tag back through the same pipeline we used to deploy v1.0.1: 

    $ git revert v1.0.1
$ git tag v1.0.2
$ git push --tags

    view rawrevert.js hosted with ❤ by GitHub

    Conclusion

    Now that you know how to get Spinnaker up and running in a development environment, start using it already. In this blog, we have done everything from installing a K8s cluster on GCP to deploying an End to End Pipeline just like that in a production environment. Hope you found it helpful.

    References

    https://cloud.google.com/solutions/continuous-delivery-spinnaker-kubernetes-engine

  • Spatial Data Analytics : The What, Why, and How?

    Introduction

    Have you ever wondered how Google Maps, Starlink, Zomato, Arogya Setu, and even methods like population clustering are able to add value to the human world? Well, the common thread between these applications and technologies is the use of spatial data and analysis techniques.

    Both Google Maps and Zomato use spatial techniques to provide navigation and location-based information to their users. While Arogya Setu is a contact tracing app that uses spatial data to track the spread of infectious illnesses, Starlink uses spatial data analysis to provide internet access to remote areas around the world. Population clustering is a technique that can be useful for urban planning, public health, and disaster response. Since the use of spatial data and its analysis techniques has become increasingly critical in the current scenario, let’s understand some fundamentals and explore different aspects of spatial data analytics.

    So, welcome to the world of spatial data analytics, where data meets geography and insights come to life! The use of spatial data analytics has changed the way we understand and interact with the world around us, providing insights and solutions to some of the most pressing challenges facing humanity today. So, let’s cut through the process by taking a quick tour of a spatial journey that you might have never been on before.

    What is spatial data analytics?

    Before we start talking about the process of spatial data analytics, let’s try to understand what is special about the term “spatial data.” Spatial data, also known as “Geospatial data,” refers to data representing features or objects on the Earth’s surface. Whether it’s man-made or natural, if it has to do with a specific location on the surface of the Earth, it’s spatial. Spatial data refers to where things are now, or perhaps, where they were or will be in the future.

    This data can be further classified as:

    Geometric Data:

    Geometric data is a type of spatial data mapped on a two-dimensional flat surface. Google Maps is an application that uses geometric data to provide accurate directions.

    Geographic Data:

    Geographic data is information mapped around the Earth that highlights the latitude and longitude relationships to a specific object or location. A familiar example of geographic data is a Global Positioning System (GPS).

    Spatial data is not limited to structured information; it also comprises imagery from satellites and drones, address data points, and longitudinal and latitudinal data. Primarily, spatial data is classified as vector data and raster data. Vector data consists of coordinate information, while raster data is all about layers of images extracted from camera sensors.

    The real world can be represented as below, where the built environment (roads, buildings) and administrative data (countries, census areas) tend to be represented as vector data. Natural environment (e.g., elevation, temperature, precipitation) is often represented using a raster grid.

    1. Discrete data, stored according to its exact geographic location, is called vector data.
    2. Continuous data is represented by regular grids called raster data.
    3. Attributes
    (Image source:  CVRD)

    Vector Data:

    • Points: A single dot on the layer depicts them. It can be either an x, y, or z coordinate.
    • Lines: This form of vector data is presented using two coordinates, i.e., either the x-y coordinate or the inverse of this, and has a definite length. These are used for rivers, roads, railways, ferry routes, and even major pipeline flows.
    • Polygon: The feature is defined using three or more coordinates. It is used to showcase inland water bodies like lakes, buildings, etc.

    Raster Data:

    • Raster is all about multilayered map images from satellites, drones, and various other camera sensors (ortho-imagery).
    • It is stored in cell-based and color-pixel formats. These pixels are arranged in columns and rows.
    • Analysis can be done better than with vector-based data due to the richness of the data.
    • It can give you more accurate measurements than other types of data.

    Attributes and Properties:

    • Spatial data contains more information than just a location on the surface of the Earth.
    • Any additional information, or non-spatial data, that describes a feature is referred to as an attribute.
    • In addition to locational and attribute information, spatial data inherently contains geometric and topological properties, which help to gain deeper insights.
    • Geometric properties include position and measurements, such as length, direction, area, and volume.
    • Topological properties represent spatial relationships such as connectivity, inclusion, and adjacency.

    As seen above, spatial data includes information such as geographic coordinates, elevation, and demographic information. Hence, it can be used to identify patterns, correlations, and trends that are not readily apparent through other data sources. For instance, geospatial data can be used to map the distribution of air pollution across a city, identify areas at risk for natural disasters like floods or wildfires, or monitor changes in land use over time. Here is where the analytics process takes place to uncover insights that can aid in providing solutions.

    Spatial data analytics involves collecting, processing, and analyzing various types of spatial data with insights to go beyond what occurs to determine not only where and when something occurs but also why it occurs at that specific place and/or time. It can be further viewed as descriptive analytics, which involves summarizing and visualizing spatial data to identify patterns and trends. Predictive analytics uses statistical models to make predictions about future events or trends based on past data. Prescriptive analytics uses optimization techniques to determine the best course of action given a specific set of circumstances.

    Why is spatial data analytics important?

    Spatial data analytics plays an essential role in many industries and fields, providing insights and solutions that can have a significant impact on our daily lives. It aids businesses in gaining a competitive edge through improved decision-making and time and money savings. Urban planning, telecommunications, military, public health, and emergency management are just a few examples of industries that rely heavily on spatial data analytics to make informed decisions.

    (Image source:  OneStopGIS)

    ‍Public Health

    A patient’s location directly influences their health. Whether it’s disease prevention or clinic site selection, considering spatial aspects in healthcare analytics can have a drastic impact.

    Urban Planning

    An urban planner might want to assess the extent of urban fringe growth, quantify the population growth that some suburbs are witnessing, and also understand why these particular suburbs are growing, and others are not.

    Environmental and Natural Resources

    Protecting our world against climate change, promoting biodiversity, exploration, and conservation planning requires spatial storytelling and sophisticated environmental analysis.

    Space and Navigation

    Optimizing transport infrastructure and navigation spatially is key to the future of mobility. The most efficient cities are moving away from traditional methods to analyze new data. 

    Telecommunication

    Since network signal strength fluctuates by location over time, spatial analytics helps telecommunications companies understand where anomalies occur and then resolve them.

    Architecture, Engineering, and Construction

    The leading AEC firms are going beyond traditional workflows to use spatial data science in urban planning and site selection, reducing costs and boosting project profitability. A geological engineer might want to identify the best localities for constructing buildings in an earthquake-prone area by looking at rock formation characteristics.

    Military

    Spatial predictive analytics helps the military optimize the placement of resources while using predictive analytics to assess infrastructure, situational awareness, anticipate maintenance needs, and meet deadlines.

    Weather Forecasting

    Rapid response to extreme weather by visualizing blizzards, wildfires, and hurricanes fast enough for effective evacuation alerts. Spatial data analytics also helps airlines with routing and gives insurance companies a better way to assess property risk.

    How to perform spatial data analytics?

    The process of spatial data analytics involves data gathering, data cleaning, data processing, and visualization, much like any other traditional analytics technique. The specific details of the process will be determined on the basis of the data and the goals of the analysis.

    ‍

    ‍Data Collection: The initial stage in spatial data analytics is to collect the relevant data. This involves gathering data from different sources, such as remote sensing satellites, GPS-enabled devices, social media, or survey instruments. The data may include geographic coordinates, attributes of features, and other pertinent information that can help analyze the data.

    Data Cleaning and Preprocessing: Once the data is collected, it needs to be cleaned and preprocessed to ensure that it is accurate and usable for further processing. This may involve eliminating duplicates, filling in missing values, and standardizing data formats.

    Data Transformation: Spatial data is often obtained from numerous sources and in a variety of forms, so the next step is to transform and combine the data into a single data set. This may involve joining tables or layers based on a shared attribute or location.

    Data Analysis: This part of spatial data analytics involves identifying spatial patterns and relationships in the data. This may involve various techniques such as clustering, interpolation, spatial regression, and spatial autocorrelation analysis. The analysis may also include visualizing the data using maps, charts, and graphs for spatial data exploration.

    Modeling and Prediction: Based on the results of spatial analysis, it may be possible to build models to predict future patterns or trends in the data as a part of predictive analytics. This may involve using machine learning algorithms or other statistical techniques to identify patterns and make predictions.

    Business Intelligence: Finally, the results of spatial data analytics can be used to support decision-making in a variety of contexts, such as urban planning, natural resource management, or emergency response. The decision-making process may involve evaluating trade-offs between different options and considering the potential impact of different decisions on the spatial patterns in the data.

    Tools and Techniques:

    Spatial Data Storage

    Spatial data storage is a specialized form of data storage that takes into account the spatial relationships between various data points, allowing for more efficient and effective analysis and retrieval of information. There are many tools available for spatial data storage, including both open-source and proprietary software. Here are a few instances of such tools.

    (Image source:  Safe Software)

    ‍

    ‍RDBMS (Relational Database Management Systems): RDBMS are among the most used methods for storing geographical data having extensions that enable spatial features. RDBMS examples supporting geographic data include:

    Spatial File Formats: Spatial file formats are widely used for storing and sharing spatial data. Examples of spatial file formats include:

    • Shapefile (.shp)
    • GeoJSON (.geojson)
    • Keyhole Markup Language (KML) (.kml)
    • Geography Markup Language (GML) (.gml)

    NoSQL: NoSQL databases are becoming increasingly popular for spatial data storage due to their ability to handle large and complex datasets, flexible schema, and scalability. Examples of NoSQL databases that support spatial data include:

    Cloud-based Storage Services: Cloud-based storage services like AWS, GCP, Azure are popular options for storing spatial data, which can be termed as DataLakes. Examples of cloud-based storage services that support spatial data include:

    • Amazon S3 with Amazon S3 GeoSpatial Indexing
    • Google Cloud Storage with Google Cloud Storage Geo-Location
    • Microsoft Azure Blob Storage with Azure Spatial Anchors

    Spatial Data Warehouses: Spatial data warehouses are specialized databases designed for spatial data analysis. Examples of spatial data warehouses include:

    It can be noted that tools, such as RDBMS and NoSQL databases, can also be used for spatial data analytics and processing in addition to storage.

    Spatial Data Processing

    Spatial data processing is an important step in spatial data analytics to ensure that the data is properly aligned and in a consistent format before further analysis. This is a must-do step because various applications and data sources use different formats and coordinate systems, which might lead to several difficulties when analyzing the data.

    Below are a few examples of processing methods in a spatial context that ensure that spatial data is compatible and consistent across different applications and data sources.

    Reprojection: Reprojection is the process of converting spatial data from one map projection to another. This is frequently necessary when working with data from multiple sources that use different projections.

    Coordinate System/Datum Transformation: This transformation involves converting spatial data from one coordinate system to another or from one geodetic datum to another. This is important when working with data from different sources that use different coordinate systems and information.

    Resampling: Resampling involves changing the resolution or scale of spatial data. This is often necessary when handling data at different scales coming from different sources.

    Geocoding: Geocoding is the process of converting a street address or other location description into a set of geographic coordinates. This allows the location to be plotted on a map and later analyzed in a spatial context.

    Georeferencing: Georeferencing is the method of aligning geographic data to a specific coordinate system or reference system. This is often required when working with data from several sources, such as aerial photographs or satellite imagery.

    Digitizing: Digitizing is the process of converting analog maps or other spatial data into a digital format. This involves manually tracing features such as roads, buildings, and water bodies using a computer program.

    Several tools are available that can perform such data processing techniques, and a few of these tool instances are given below.

    GIS (Geographic Information Systems): GIS connects data to a map, integrating location data with all types of descriptive information. It helps users understand patterns, relationships, and geographic context. The benefits include improved communication and efficiency, as well as better management and decision-making. Examples of GIS software that supports spatial data processing include:

    • ArcGIS – A proprietary GIS software with a comprehensive set of features and tools
    • QGIS – An open-source GIS software with a wide range of plugins and tools

    Python Libraries: Python is a popular programming language for spatial data processing, and there are several libraries available for this purpose. Examples of Python libraries that support spatial data processing include:

    • GeoPandas: A library for working with geospatial data in Python
    • Shapely: A library for manipulation and analysis of planar geometric objects
    • PySAL: A library for spatial analysis and modeling

    R Packages: Like Python, R is another popular programming language for spatial data processing, and there are several packages available for spatial data operations. Examples of R packages that support spatial data processing include:

    • sf: an R package for working with geospatial data
    • sp: an R package for spatial data analysis
    • raster: an R package for working with raster data

    SQL: SQL can be used for spatial data processing and analysis, especially when working with spatial databases with extensions like PostGIS. Examples of SQL spatial functions include:

    Command-line Tools: There are a handful of command-line tools available for spatial data processing. Examples of command-line tools that support spatial data processing include:

    • GDAL/OGR: a suite of tools for geospatial data processing and conversion
    • GRASS GIS: a command-line tool for geospatial analysis and modeling

    There are a few other tools for data processing worth exploring, such as MATLAB, GeoServer, Global Mapper, and Mapbox. Tools like GIS software and Python libraries can also be used for spatial data storage and analysis in addition to processing.

    (Image source:  Carto)

    ‍Spatial Data Analysis

    Spatial data analysis is the process of examining geographic data to spot trends, correlations, and patterns. It involves the use of statistical, computational, and visualization methods to explore spatial data and extract business insights. There are different categories of spatial data analysis, such as:

    Proximity Analysis: It involves measuring the distance between two or more places in a spatial dataset. It is possible to analyze proximity using methods like Euclidean distance.

    Accessibility Analysis: It is a measure of how easy it is to get to a location from other locations in the dataset. In addition to distance, Accessibility analysis takes into account other factors that affect how easy it is to travel between locations, such as traffic, road conditions, and public transportation.

    Spatial Clustering: Spatial clustering is the process of identifying groups of spatially adjacent objects that have similar characteristics. Hierarchical clustering and k-means clustering are two methods that can be used to accomplish this.

    Spatial Interpolation: Spatial interpolation involves estimating values for locations where data is not available based on nearby data points. This can be done using techniques such as kriging or inverse distance weighting.

    Spatial Exploratory Data Analysis: It involves creating visual representations of spatial data to explore patterns and relationships. Spatial EDA helps to identify patterns and relationships that may not be immediately apparent from the data and can help guide further analysis. This can include techniques such as choropleth maps, heat maps, or scatter plots.

    Spatial Simulation: This involves using simulation models to study the behavior of spatial systems over time. Spatial simulation includes techniques such as cellular automata, agent-based models, and Monte Carlo simulations. Spatial simulation is useful for predicting the future behavior of spatial systems under different scenarios.

    There are other categories, like factor analysis, trajectory analysis, network analysis, etc., that can be used for fine-grained spatial data analysis. Below are a few examples of tools that can be used to devise an analysis of spatial data.

    GIS (Geographic Information Systems): As seen earlier, this software can be used not only for capturing and processing the data but also to analyze and display geographically referenced data. Examples include ArcGIS, QGIS, and GRASS GIS.

    Open Source Libraries and Binaries: It includes various programming languages with many packages for spatial data analysis, such as the sp package for handling spatial data and the rgdal package for reading and writing geospatial data formats in R or packages, such as GeoPandas and Shapely to provide functionality for working with geospatial data in Python. The list can go on with the GDAL framework and its dependencies.

    PostGIS: PostGIS is a spatial database extender for the PostgreSQL database management system. It adds support for geographic objects, allowing you to manipulate and query geospatial data within the database for any kind of analysis purpose.

    Data Visualization Tools: These tools are used to create visual representations of spatial data for exploratory data analysis. Examples include Tableau, ArcGIS Pro, and QGIS.

    Mapbox: Mapbox is a mapping platform that provides APIs and SDKs for building custom maps and applications. It includes tools for data visualization, geocoding, routing, and more.

    ENVI: ENVI is a software package for processing and analyzing remote sensing data. It includes tools for image classification, spectral analysis, and terrain modeling, among others.

    Spatial data analysis plays an essential role in understanding complex spatial patterns and relationships and can help formulate business decisions in a wide range of areas. The choice of tool depends on the type of analysis that needs to be performed, the size of the data set, and the resources available.

    How to solve spatial big data problems?

    Big data refers to datasets that are too large and complex to process and analyze using the traditional methods that we discussed earlier. When dealing with spatial data, the challenges of big data are amplified due to the added dimensions of space and time. Spatial data is being captured at an unusual rate because of the growing numbers of sensors and devices, the networks of GPS satellites and cell towers, and the rise of the Internet of Things.

    Spatial data analytics can leverage strategies and resources, including distributed computing, cloud computing, and parallel processing, to address the above issues. These techniques allow for the processing and analysis of large spatial datasets, enabling real-time decision-making in industries including transportation, agriculture, and public safety. For instance, massive geographic data analytics are used by real-time traffic management systems to optimize traffic flows, ease congestion, and improve safety.

    (Image source:  Utilizing Cloud Computing to Address Big Geospatial Data Challenges Paper)

    ‍Apache Sedona (formerly GeoSpark):

    • Apache Spark with a geospatial extension for geospatial data analytics capabilities.
    • Supports different spatial indexes, such as R-Tree, Quadtree, and K-D Tree, which can improve the performance of spatial queries and operations.
    • Supports various spatial queries, such as range queries, KNN queries, and spatial joins.
    • Designed to work with other components of the Spark ecosystem, such as Spark SQL and Spark Streaming.
    • Provides support for machine learning algorithms on geospatial data, such as clustering and classification.

    SpatialHadoop:

    • Apache Hadoop with a geospatial extension for spatial data analytics capabilities.
    • Can process and analyze large-scale spatial data in a distributed environment using the MapReduce paradigm.
    • Supports different spatial indexes, such as R-Tree and Grid File, which can improve the performance of spatial queries and operations.
    • Supports various spatial queries, such as range queries, KNN queries, and spatial joins.
    • Designed to work with other components of the Hadoop ecosystem, such as HDFS, MapReduce, and Hive.

    BigQuery GIS:

    • Google Cloud Platform that provides geospatial data analytics capabilities.
    • It is a fully-managed service that automatically scales up or down based on the volume of data and the complexity of queries.
    • Supports different spatial indexes, such as R-Tree and Hilbert Curve, which can improve the performance of spatial queries and operations.
    • Supports various spatial queries, such as range queries, KNN queries, and spatial joins.
    • Designed to work with other components of the BigQuery ecosystem, such as BigQuery ML, BigQuery BI Engine, and Bigquery Geo Viz.

    There are a few other tools and extensions, like Esri GIS Tools for Hadoop, SpatialSpark, Google Earth Engine that can be used to gain insights and make informed decisions based on spatial data.

    Case Study

    In the telecommunications industry, spatial analysis can be used to optimize network coverage and capacity, plan new infrastructure, and identify areas of high network congestion.

    Let’s consider a hypothetical telecommunications company that wants to improve its network performance and customer experience by analyzing geospatial data. Specifically, the company wants to analyze call detail records (CDRs) to identify areas of high call volume and network congestion.

    (Image source:  Microsoft Azure Architectures)

    In a given solution by Azure in a published article, the suggested architecture involves:

    • Azure Data Factory, which is used to collect the CDRs from various sources (mainly geospatial databases).
    • Azure Data Factory stores them in Azure Data Lake Storage in formats such as GeoJSON, WKT, and Vector tiles. The bronze container holds raw data, the silver container holds semi-curated data, and the gold container holds fully curated data as the processing proceeds.
    • Azure Databricks and the GeoSpark/Sedona package are being used to convert data formats and efficiently load, process, and analyze large-scale spatial data across machines.
    • GeoPandas exports data in various formats, which are later used by GIS applications such as QGIS and ArcGIS for exploratory analysis.
    • Azure Machine Learning extracts insights from geospatial data, determining, for example, where and when to deploy new wireless access points.
    • Power BI or Azure Maps can be used to visualize the geospatial data and identify areas where network upgrades or infrastructure improvements are needed.
    • A log analytics system is set up to run queries against data in Azure Monitor Logs to implement a robust and fine-grained logging system to analyze events and performance.

    Overall, the Azure-based solution gives an idea about how one can try to perform geospatial analysis in the telecommunications industry and improve network performance and customer experience. You can read more about this solution here.

    Challenges and limitations

    In conclusion, spatial data analytics is an essential component of decision-making across a range of industries. It is important to understand the techniques, infrastructure, and challenges of spatial data analytics to effectively leverage spatial data and make informed decisions. Spatial data collection can be challenging and may contain faults or inconsistencies. The data may not be available for certain geographic areas or for certain time periods. Spatial data analytics can raise privacy concerns if personal data is collected and used without consent. In addition, there may be concerns about the use of spatial data analytics for surveillance or other unethical purposes, which can lead to significant harm.

    Conclusion

    Spatial data analytics is a powerful tool that can help organizations make better-informed decisions and gain a competitive advantage. As the fields of machine learning (AI) and spatial data analysis intertwine, spatial data analytics looks promising and quite useful for real-life problems. The blend of both vector and raster data produces a powerful product that can tackle various economic and earth-related problems. This blog is just a high-level overview of spatial data analytics since you have just scratched the surface, but I can guarantee that this spatial ride will be smoother from here on.

  • The Data Lake Revolution: Unleashing the Power of Delta Lake

    Once upon a time, in the vast and ever-expanding world of data storage and processing, a new hero emerged. Its name? Delta Lake. This unsung champion was about to revolutionize the way organizations handled their data, and its journey was nothing short of remarkable.

    The Need for a Data Savior

    In this world, data was king, and it resided in various formats within the mystical realm of data lakes. Two popular formats, Parquet and Hive, had served their purposes well, but they harbored limitations that often left data warriors frustrated.

    Enterprises faced a conundrum: they needed to make changes, updates, or even deletions to individual records within these data lakes. But it wasn’t as simple as it sounded. Modifying schemas was a perilous endeavor that could potentially disrupt the entire data kingdom.

    Why? Because these traditional table formats lacked a vital attribute: ACID transactions. Without these safeguards, every change was a leap of faith.

    The Rise of Delta Lake

    Amidst this data turmoil, a new contender emerged: Delta Lake. It was more than just a format; it was a game-changer.

    Delta Lake brought with it the power of ACID transactions. Every data operation within the kingdom was now imbued with atomicity, consistency, isolation, and durability. It was as if Delta Lake had handed data warriors an enchanted sword, making them invincible in the face of chaos.

    But that was just the beginning of Delta Lake’s enchantment.

    The Secrets of Delta Lake

    Delta Lake was no ordinary table format; it was a storage layer that transcended the limits of imagination. It integrated seamlessly with Spark APIs, offering features that left data sorcerers in awe.

    • Time Travel: Delta Lake allowed users to peer into the past, accessing previous versions of data. The transaction log became a portal to different eras of data history.
    • Schema Evolution: It had the power to validate and evolve schemas as data changed. A shapeshifter of sorts, it embraced change effortlessly.
    • Change Data Feed: With this feature, it tracked data changes at the granular level. Data sorcerers could now decipher the intricate dance of inserts, updates, and deletions.
    • Data Skipping with Z-ordering: Delta Lake mastered the art of optimizing data retrieval. It skipped irrelevant files, ensuring that data requests were as swift as a summer breeze.
    • DML Operations: It wielded the power of SQL-like data manipulation language (DML) operations. Updates, deletes, and merges were but a wave of its hand.

    Delta Lake’s Allies

    Delta Lake didn’t stand alone; it forged alliances with various data processing tools and platforms. Apache Spark, Apache Flink, Presto, Trino, Hive, DBT, and many others joined its cause. They formed a coalition to champion the cause of efficient data processing.

    In the vast landscape of data management, Delta Lake stands as a beacon of innovation, offering a plethora of features that elevate your data handling capabilities to new heights. In this exhilarating adventure, we’ll explore the key features of Delta Lake and how they triumph over the limitations of traditional file formats, all while embracing the ACID properties.

    ACID Properties: A Solid Foundation

    In the realm of data, ACID isn’t just a chemical term; it’s a set of properties that ensure the reliability and integrity of your data operations. Let’s break down how Delta Lake excels in this regard.

    A for Atomicity: All or Nothing

    Imagine a tightrope walker teetering in the middle of their performance—either they make it to the other side, or they don’t. Atomicity operates on the same principle: either all changes happen, or none at all. In the world of Spark, this principle often takes a tumble. When a write operation fails midway, the old data is removed, and the new data is lost in the abyss. Delta Lake, however, comes to the rescue. It creates a transaction log, recording all changes made along with their versions. In case of a failure, data loss is averted, and your system remains consistent.

    C for Consistency: The Guardians of Validity

    Consistency is the gatekeeper of data validity. It ensures that your data remains rock-solid and valid at all times. Spark sometimes falters here. Picture this: your Spark job fails, leaving your system with invalid data remnants. Consistency crumbles. Delta Lake, on the other hand, is your data’s staunch guardian. With its transaction log, it guarantees that even in the face of job failure, data integrity is preserved.

    I for Isolation: Transactions in Solitude

    Isolation is akin to individual bubbles, where multiple transactions occur in isolation, without interfering with one another. Spark might struggle with this concept. If two Spark jobs manipulate the same dataset concurrently, chaos can ensue. One job overwrites the dataset while the other is still using it—no isolation, no guarantees. Delta Lake, however, introduces order into the chaos. Through its versioning system and transaction log, it ensures that transactions proceed in isolation, mitigating conflicts and ensuring the data’s integrity.

    D for Durability: Unyielding in the Face of Failure

    Durability means that once changes are made, they are etched in stone, impervious to system failures. Spark’s Achilles’ heel lies in its vulnerability to data loss during job failures. Delta Lake, however, boasts a different tale. It secures your data with unwavering determination. Every change is logged, and even in the event of job failure, data remains intact—a testament to true durability.

    Time Travel: Rewriting the Past

    Now, let’s embark on a fascinating journey through time. Delta Lake introduces a feature that can only be described as “time travel.” With this feature, you can revisit previous versions of your data, just like rewinding a movie. All of this magical history is stored in the transaction log, encapsulated within the mystical “_delta_log” folder. When you write data to a Delta table, it’s not just the present that’s captured; the past versions are meticulously preserved, waiting for your beck and call.

    In conclusion, Delta Lake emerges as the hero of the data world, rewriting the rules of traditional file formats and conquering the challenges of the ACID properties. With its robust transaction log, versioning system, and the ability to traverse time, Delta Lake opens up a new dimension in data management. So, if you’re on a quest for data reliability, integrity, and a touch of magic, Delta Lake is your trusted guide through this thrilling journey beyond convention.

    More Features of Delta Lake Are:

    • UPSERT
    • Schema Evolution
    • Change Data Feed
    • Data Skipping with Z-ordering
    • DML Operations

    The Quest for Delta Lake

    Setting up Delta Lake was like embarking on a quest. Data adventurers ventured into the cloud, AWS, GCP, Azure, or even their local domains. They armed themselves with the delta-spark spells and summoned the JARs of delta-core, delta-contrib, and delta-storage, tailored to their Spark versions.

    Requirements:

    • Python
    • Delta-spark
    • Delta jars

    You can configure in a Spark session and define the package name so it will be downloaded at the run time. As I said, I am using Spark version 3.3. We will require these things: delta-core, delta-contribs, delta-storage. You can download them from here: https://github.com/delta-io/delta/releases/ 

    To use and configure various cloud storage options, there are separate .jars you can use: https://docs.delta.io/latest/delta-storage.html. Here, you can find .jars for AWS, GSC, and Azure to configure and use their data storage medium.

    Run this command to install delta-spark first:

    pip  install delta-spark

    (If you are using dataproc or EMR, you can install this while creating cluster as a startup action, and if you are using serverless env like Glue or dataproc batches, you can create docker build or pass the .whl file for this package.)

    You must also do this while downloading the .jar. If it is serverless, download the .jar, store it in cloud storage, like S3 or GS, and use that path while running the job. If it is a cluster like dataproc or EMR, you can download this on the cluster. 

    One can also download these .jars at the run time while creating the Spark session as well.

    Now, create the Spark session, and you are ready to play with Delta tables.

    Environment Setup

    How do you add the Delta Lake dependencies to your environment?

    1. You can directly add them while initializing the Spark session for Delta Lake by passing the specific version, and these packages or dependencies will be downloaded during run time.
    2. You can place the required .jar files in your cluster and provide the reference while initializing the Spark session.
    3. You can download the .jar files and store them in cloud storage, and you can pass them as a run time argument if you don’t want to download the dependencies on your cluster.
    # Initialize Spark Session
import pyspark
from delta import *
from pyspark.sql.types import *
from delta.tables import *
from pyspark.sql.functions import *

builder = pyspark.sql.SparkSession.builder.appName("My App") \
      .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \


         .config("spark.jars.packages", "io.delta:delta-core_2.12:2.2.0") \



# or if jar is already there

builder = pyspark.sql.SparkSession.builder.appName("My App") \
      .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")




spark = builder.getOrCreate()

    You have to add the following properties to use delta in Spark:-

    • Spark.sql.extensions
    • Spark.sql.catalog.spark_catalog

    You can see these values in the above code snippet. If you want to use cloud storage like reading and writing data from S3, GS, or Blob storage, then we have to set some more configs as well in the Spark session. Here, I am providing examples for AWS and GSC only.

    The next thing that will come to your mind: how will you be able to read or write the data into cloud storage?

    For different cloud storages, there are certain .jar files available that are used to connect and to do IO operations on the cloud storage. See the examples below.

    You can use the above approach to make this .jar available for Spark sessions either by downloading at a run time or storing them on the cluster itself.

    AWS 

    spark_jars_packages = “com.amazonaws:aws-java-sdk:1.12.246,org.apache.hadoop:hadoop-aws:3.2.2,io.delta:delta-core_2.12:2.2.0”

    spark = SparkSession.builder.appName(‘delta’)
      .config(“spark.jars.packages”, spark_jars_packages)
      .config(“spark.sql.extensions”, “io.delta.sql.DeltaSparkSessionExtension”)
      .config(“spark.sql.catalog.spark_catalog”, “org.apache.spark.sql.delta.catalog.DeltaCatalog”)
      .config(‘spark.hadoop.fs.s3a.aws.credentials.provider’, ‘org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider’)
      .config(“spark.hadoop.fs.s3.impl”, “org.apache.hadoop.fs.s3a.S3AFileSystem”)
      .config(“spark.hadoop.fs.AbstractFileSystem.s3.impl”, “org.apache.hadoop.fs.s3a.S3AFileSystem”)
      .config(“spark.delta.logStore.class”, “org.apache.spark.sql.delta.storage.S3SingleDriverLogStore”) 

    spark = builder.getOrCreate()

    ‍

    GCS

    spark_session = SparkSession.builder.appName(‘delta’).builder.getOrCreate()

    ‍

    spark_session.conf.set(“fs.gs.impl”, “com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem”)

    spark_session.conf.set(“spark.hadoop.fs.gs.impl”, “com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem”)

    spark_session.conf.set(“fs.gs.auth.service.account.enable”, “true”)

    spark_session.conf.set(“fs.AbstractFileSystem.gs.impl”, “com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS”)

    spark_session.conf.set(“fs.gs.project.id”, project_id)

    spark_session.conf.set(“fs.gs.auth.service.account.email”, credential[“client_email”])

    spark_session.conf.set(“fs.gs.auth.service.account.private.key.id”, credential[“private_key_id”])

    spark_session.conf.set(“fs.gs.auth.service.account.private.key”, credential[“private_key”])‍

    Write into Delta Tables: In the following example, we are using a local system only for reading and writing the data into and from delta lake tables.

    Data Set Used: https://media.githubusercontent.com/media/datablist/sample-csv-files/main/files/organizations/organizations-100.zip

    For reference, I have downloaded this file in my local machine and unzipped the data:

    df = spark.read.option("header", "true").csv("organizations-100.csv")
df.write.mode('overwrite').format("delta").partitionBy(partition_keys).save("./Documents/DE/Delta/test-db/organisatuons")

    There are two modes available in Delta Lake and Spark (Append and Overwrite) while writing the data in the Delta tables from any source.

    For now, we have enabled the Delta catalog to store all metadata-related information. We can also use the hive meta store to store the metadata information and to directly run the SQL queries over the delta tables. You can use the cloud storage path as well.

    Read data from the Delta tables:

    delta_df = spark.read.format("delta").load("./Documents/DE/Delta/test-db/organisatuons")
delta_df.show()

    ‍

    Here, you can see the folder structure, and after writing the data into Delta Tables, it creates one delta log file, which keeps track of metadata, partitions, and files.

    Option 2: Create Delta Table and insert data using Spark SQL.

    spark.sql("CREATE TABLE orgs_data(index String, c_name String, organization_id String, name String, website String, country String, description String, founded String, industry String, num_of_employees String, remarks String) USING DELTA")

    Insert the data:

    df.write.mode('append').format("delta").option("mergeSchema", "true").saveAsTable("orgs_data")
spark.sql("SELECT * FROM orgs_data").show()
spark.sql("DESCRIBE TABLE orgs_data").show()

    This way, we can read the Delta table, and you can use SQL as well if you have enabled the hive.

    Schema Enforcement: Safeguarding Your Data

    In the realm of data management, maintaining the integrity of your dataset is paramount. Delta Lake, with its schema enforcement capabilities, ensures that your data is not just welcomed with open arms but also closely scrutinized for compatibility. Let’s dive into the meticulous checks 

    Delta Lake performs when validating incoming data against the existing schema:

    Column Presence: Delta Lake checks that every column in your DataFrame matches the columns in the target Delta table. If there’s a single mismatch, it won’t let the data in and, instead, will raise a flag in the form of an exception.

    Data Types Harmony: Data types are the secret language of your dataset. Delta Lake insists that the data types in your incoming DataFrame align harmoniously with those in the target Delta table. Any discord in data types will result in a raised exception.

    Name Consistency: In the world of data, names matter. Delta Lake meticulously examines that the column names in your incoming DataFrame are an exact match to those in the target Delta table. No aliases allowed. Any discrepancies will lead to, you guessed it, an exception.

    This meticulous schema validation guarantees that your incoming data seamlessly integrates with the target Delta table. If any aspect of your data doesn’t meet these strict criteria, it won’t find a home in the Delta Lake, and you’ll be greeted by an error message and a raised exception.

    Schema Evolution: Adapting to Changing Data

    In the dynamic landscape of data, change is the only constant. Delta Lake’s schema evolution comes to the rescue when you need to adapt your table’s schema to accommodate incoming data. This powerful feature offers two distinct approaches:

    Overwrite Schema: You can choose to boldly overwrite the existing schema with the schema of your incoming data. This is an excellent option when your data’s structure undergoes significant changes. Just set the “overwriteSchema” option to true, and voila, your table is reborn with the new schema.

    Merge Schema: In some cases, you might want to embrace the new while preserving the old. Delta Lake’s “Merge Schema” property lets you merge the incoming data’s schema with the existing one. This means that if an extra column appears in your data, it elegantly melds into the target table without throwing any schema-related tantrums.

    Should you find the need to tweak column names or data types to better align with the incoming data, Delta Lake’s got you covered. The schema evolution capabilities ensure your dataset stays in tune with the ever-changing data landscape. It’s a smooth transition, no hiccups, and no surprises, just data management at its finest.

    spark.read.table(...) 
  .withColumn("birthDate", col("birthDate").cast("date")) 
  .write 
  .format("delta") 
  .mode("overwrite")
  .option("overwriteSchema", "true") 
  .saveAsTable(...)

    The above code will overwrite the existing delta table with the new schema along with the new data.

    Delta Lake has support for automatic schema evolution. For instance, if you have added two more columns in the Delta Lake tables and still tried to access the existing table, you will be able to read the data without any error.

    There is another way as well. For example, if you have three columns in a Delta table but the incoming table has four columns, you can set up spark.databricks.delta.schema.autoMerge.enabled is true. It can be done for the entire cluster as well.

    spark.sql("DESCRIBE TABLE orgs_data").show()

    Let’s add one more column and try to access the data again:

    spark.sql(“alter table orgs_data add column(exta_col String)”)

    spark.sql(“describe table orgs_data”).show()

    As you can see, that column has been added but has not impacted the data. You can still smoothly and seamlessly read the data. It will set null to a newly created column.

    What happens if we receive the extra column in an incoming CSV that we want to append to the existing delta table? You have to set up one config here for that:

    input_df = spark.read.format('csv').option('header', 'true').load("../Desktop/Data-Engineering/data-samples/input-data/organizations-11111.csv")
input_df.printSchema()
input_df.write.mode('append').format("delta").option("mergeSchema", "true").saveAsTable("orgs_data")
spark.sql("SELECT * FROM orgs_data").show()
spark.sql("DESCRIBE TABLE orgs_data").show()

    You have to add this config mergeSchem=true while appending the data. It will merge the schema of incoming data that is receiving some extra columns.

    The first figure shows the schema of incoming data, and in the previous one, we have already seen the schema of our delta tables.

    Here, we can see that the new column that was coming in the incoming data is merged with the existing schema of the table. Now, the delta table has the latest updated schema.

    Time Travel 

    Basically, Delta Lake keeps track of all the changes in _delta_log by creating a log file. By using this, we can fetch the data of the previous version by specifying the version number.

    df = spark.read.format("delta").option("versionAsOf", 0).load("orgs_data")
df.show()

    Here, we can see the first version of the data, where we have not added any columns. As we know, the Delta table maintains the delta log file, which contains the information of each commit so that we can fetch the data till the particular commit.

    Upsert, Delete, and Merge

    Unlocking the Power of Upsert with Delta Lake

    In the exhilarating realm of data management, upserting shines as a vital operation, allowing you to seamlessly merge new data with your existing dataset. It’s the magic wand that updates, inserts, or even deletes records based on their status in the incoming data. However, for this enchanting process to work its wonders, you need a key—a primary key, to be precise. This key acts as the linchpin for merging data, much like a conductor orchestrating a symphony.

    A Missing Piece: Copy on Write and Merge on Read

    Now, before we delve into the mystical world of upserting with Delta Lake, it’s worth noting that Delta Lake dances to its own tune. Unlike some other table formats like Hudi and Iceberg, Delta Lake doesn’t rely on the concepts of Copy on Write and Merge on Read. These techniques are used elsewhere to speed up data operations.

    Two Paths to Merge: SQL and Spark API

    To harness the power of upserting in Delta Lake, you have two pathways at your disposal: SQL and Spark API. The choice largely depends on your Delta version. In the latest Delta version, 2.2.0, you can seamlessly execute merge operations using Spark API. It’s a breeze. However, if you’re working with an earlier Delta version, say 1.0.0, then Spark SQL is your trusty steed for upserts and merges. Remember, using the right Delta version is crucial, or you might find yourself grappling with the cryptic “Method not found” error, which can turn into a debugging labyrinth.

    In the snippet below, we showcase the elegance of upserting using Spark SQL, a technique that ensures your data management journey is smooth and error-free:

    -- Insert new data and update existing data based on the specified key
MERGE INTO targetTable AS target
USING sourceTable AS source
ON target.id = source.id
WHEN MATCHED THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *
WHEN NOT MATCHED BY SOURCE THEN
  DELETE;

    today_data_df = spark.read.format('csv').option('header', 'true').load("../Desktop/Data-Engineering/data-samples/input-data/organizations-11111.csv")
today_data_df.show()


spark.sql("select * from orgs_data where organization_id = 'FAB0d41d5b5ddd'").show()


# Reading Existing Delta table
deltaTable = DeltaTable.forPath(spark, "orgs_data")

today_data_df.createOrReplaceTempView("incoming_data")

    Here, we are loading the incoming data and showing what is inside. The existing data with the same primary key appears in the Delta table so that we can compare after upserting or merging the data.

    spark.sql(
"""
MERGE INTO orgs_data
USING incoming_data
ON orgs_data.organization_id = incoming_data.organization_id
WHEN MATCHED THEN
  UPDATE SET
    organization_id = incoming_data.organization_id,
    name = incoming_data.name
"""
)

spark.sql("select * from orgs_data where organization_id = 'FAB0d41d5b5ddd'").show()

    ‍

    orgs_data.alias("oldData").merge(
   incoming_data.alias("newData"),
   f"oldData.organization_id = newData.organization_id") 
   .whenMatchedUpdateAll() 
   .whenNotMatchedInsertAll() 
   .execute()

    This is the example of how you can do upsert using Spark APIs. The merge operation creates lots of small files. You can control the number of small files by setting up the following properties in the spark session.

    spark.delta.merge.repartitionBeforeWrite true

    spark.sql.shuffle.partitions 10

    This is how merge operations work. Merge supports one-to-one mapping. What we want to say is that only rows should try to update the one row in the target delta table. If multiple rows try to update the rows in the target Delta table, it will fail. Delta Lake matches the data on the basis of a key in case of an update operation.

    Change Data Feed

    This is also a useful feature of Delta Lake and tracks and maintains the history of all records in the Delta table after upsert or insert at the row level. You can enable these things at the beginning while setting up the Spark session or using Spark SQL by enabling  “change events” for all the data.

    Now, you can see the whole journey of each record in the Delta table, from its insertion to deletion. It introduces one more extra column, _change_type, which contains the type of operations that have been performed on that particular row.

    To enable this, you can set these configurations: 

    spark.sql("set spark.databricks.delta.properties.defaults.enableChangeDataFeed = true;")

    Or you can set this conf while reading the delta table as well. 

    ## Stream Data Generation

data = [{"Category": 'A', "ID": 1, "Value": 121.44, "Truth": True, "Year": 2022},
        {"Category": 'B', "ID": 2, "Value": 300.01, "Truth": False, "Year": 2020},
        {"Category": 'C', "ID": 3, "Value": 10.99, "Truth": None, "Year": 2022},
        {"Category": 'E', "ID": 5, "Value": 33.87, "Truth": True, "Year": 2022}
        ]

df = spark.createDataFrame(data)

df.show()

df.write.mode('overwrite').format("delta").partitionBy("Year").save("silver_table")

    deltaTable = DeltaTable.forPath(spark, "silver_table")
                                 
deltaTable.delete(condition = "ID == 1")

delta_df = spark.read.format("delta").option("readChangeFeed", "true").option("startingVersion", 0).load("silver_table")
delta_df.show()

    Now, after deleting something, you will be able to see the changes, like what is deleted and what is updated. If you are doing upserts on the same Delta table after enabling the change data feed, you will be able to see the update as well, and if you insert anything, you will be able to see what is inserted in your Delta table. 

    If we overwrite the complete Delta table, it will mark all past records as a delete:

    If you want to record each data change, you have to enable this before creating the table so that we can see the data changes for each version. If you’ve already created one table, you won’t be able to see the changes for the previous version once you enable the change data feed, but you will be able to see the changes in all versions that came after this configuration.

    Data Skipping with Z-ordering

    Data skipping is the technique in Delta Lake where, if you have a larger number of records stored in many files, it will read the data from the files that contain required information, but apart from that, other files will get skipped. This makes it faster to read the data from the Delta tables.

    Z-ordering is a technique used to colocate the information in the same dataset files. If you know the column that will be more in use in the select statement and has A cardinality, you can use Z-order by that particular column. It will reduce the large number of files from being read. We can give you multiple columns for Z-order by separating them from commas.

    Let’s suppose you have two tables, a and b, and there is one column that is most frequently used. You can increase the number of files to be skipped, and you can use the columns files running that query. Normal order works linearly, whereas Z-order works in multi- dimensionally.

    OPTIMIZE events
WHERE date >= current_timestamp() - INTERVAL 1 day
ZORDER BY (eventType)

    DML Operations

    Delta Lake has capabilities to run all the DML operations of SQL in the data lake as well as update, delete, and merge operations.

    Integrations and Ecosystem Supported in Delta Lake

    ‍Read Delta Tables

    Unlock the Delta Tables: Tools That Bring Data to Life

    Reading data from Delta tables is like diving into a treasure trove of information, and there’s more than one way to unlock its secrets. Beyond the standard Spark API, we have a squad of powerful allies ready to assist: SQL query engines like Athena and Trino. But they’re not just passive onlookers; they bring their own magic to the table, empowering you to perform data manipulation language (DML) operations that can reshape your data universe.

    Athena: Unleash the SQL Sorcery

    Imagine Athena as the Oracle of data. With SQL as its spellbook, it delves deep into your Delta tables, fetching insights with precision and grace. But here’s the twist: Athena isn’t just for querying; like a skilled blacksmith, it can help you hammer your data into a new shape, creating a masterpiece.

    Trino: The Shape-Shifting Wizard

    Trino, on the other hand, is the shape-shifter of the data realm. It glides through Delta tables, allowing you to perform an array of DML operations that can transform your data into new, dazzling forms. Think of it as a master craftsman who can sculpt your data, creating entirely new narratives and visualizations.

    So, when it comes to Delta tables, these tools are not just readers; they are your co-creators. They enable you to not only glimpse the data’s beauty but also mold it into whatever shape serves your purpose. With Athena and Trino at your side, the possibilities are as boundless as your imagination.

    Read Delta Tables Using Spark APIS

    from delta.tables import *
delta_df =DeltaTable.forPath(spark,"./Documents/DE/Delta/test-db/organisatuons")<br>delta_df.toDf().show()

    Steps to Set Up Delta Lake with S3 on EC2 Or EMR and Access Data through Athena

    Data Set Used – We have generated some dummy data of around 100gb and written that into the Delta tables.

    Step 1- Set up a Spark session along with AWS cloud storage and Delta – Spark. Here, we have used an EC2 instance with Spark 3.3 and Delta version 2.1.1. Here, we are setting up Spark config for Delta and S3.

    AWS_ACCESS_KEY_ID = "XXXXXXXXXXXXXXXXXXXXXX"
AWS_SECRET_ACCESS_KEY = "XXXXXXXXXXXXXXXXXXXXXX+XXXXXXXXXXXXXXXXXXXXXX"

spark_jars_packages = "com.amazonaws:aws-java-sdk:1.12.246,org.apache.hadoop:hadoop-aws:3.2.2,io.delta:delta-core_2.12:2.1.1"

spark = SparkSession.builder.appName('delta') \
   .config("spark.jars.packages", spark_jars_packages) \
   .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
   .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
   .config('spark.hadoop.fs.s3a.aws.credentials.provider', 'org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider') \
   .config("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
   .config("spark.hadoop.fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
   .config("spark.delta.logStore.class", "org.apache.spark.sql.delta.storage.S3SingleDriverLogStore") \
   .config("spark.driver.memory", "20g") \
   .config("spark.memory.offHeap.enabled", "true") \
   .config("spark.memory.offHeap.size", "8g") \
   .getOrCreate()

spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY_ID)
spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.secret.key", AWS_SECRET_ACCESS_KEY)

    Spark Version – You can use any Spark version, but Spark 3.3.1 came along with the pip install. Just make sure whatever version you are using is compatible with the Delta Lake version that you are using; otherwise, most of the features won’t work.

    Step 2 – Here, we are creating a Delta table with an S3 path. We can directly write the data into an S3 bucket as a Delta table, but it is better to create a table first and then write it into S3 to make sure the schema is correct.

    Set the Delta location path if it exists to run the Spark SQL query and create the Delta table along with the S3 path. 

    # If table is already there
delta_path = "s3a://abhishek-test-01012023/delta-lake-sample-data/"
spark.conf.set('table.location', delta_path)

# Creating new delta table on s3 location
spark.sql("CREATE TABLE delta.`s3://abhishek-test-01012023/delta-lake-sample-data/`(id INT, first_name String, "
         "last_name String, address String, pincocde INT, net_income INT, source_of_income String, state String, "
         "email_id String, description String, population INT, population_1 String, population_2 String, "
         "population_3 String, population_4 String, population_5 String, population_6 String, population_7 String, "
         "date String) USING DELTA PARTITIONED BY (date)")

    Step 3 – Here, I have given one link that I have used to generate the dummy data and have written that into the S3 bucket as Delta tables. Feel free to look over this. An example code of writing is given below:

    df.write.format("delta").mode("append").partitionBy("date").save("s3a://abhishek-test-01012023/delta-lake-sample-data/")

    https://github.com/velotio-tech/delta-lake-iceberg-poc/blob/0396cdbf96230609695a907fdbe8c240042fce9e/delta-data-writer.py#L83

    In the above link, you find the code of dummy data generation.

    Step 4 – Here, we are printing the count and selecting some data from the Delta table that we have written in just right away.

    Run the SQL query to check the table data and upsert using S3 bucket data:

    spark.sql("select count(*) from delta.`s3://abhishek-test-01012023/delta-lake-sample-data/` group by id having count("
         "*) > 1").show()

spark.sql("select count(*) from delta.`s3://abhishek-test-01012023/delta-lake-sample-data/`").show()

#############################################################################
# Upsert
#############################################################################

# upserts the starting five records. We will read first five record and will do some changes in some columns and

input_df = spark.read.csv("s3a://abhishek-test-01012023/incoming_data/delta/4e0ae9f5-8c9d-435a-a434-febff1effbc3.csv",inferSchema=True,header=True)
input_df.printSchema()
input_df.createOrReplaceTempView("incoming")
spark.sql("MERGE INTO delta.`s3://abhishek-test-01012023/delta-lake-sample-data/` t USING (SELECT * FROM incoming) s ON t.id = s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *")

    This is the output of select statement:

    This is the schema of the incoming data we are planning to merge into the existing Delta table:

    After upsert, let’s see the data for the particular data partition:

    spark.sql(“select * from delta.`s3://abhishek-test-01012023/delta-lake-sample-data/` where id = 1 and date = 20221206”)

    ‍

    ‍

    Access Delta table using Hive or any other external metastore: 

    For that, we have to create a link between them and to create this link, go to the Spark code and generate the manifest file on the S3 path where we have already written the data.

    spark.sql("GENERATE symlink_format_manifest FOR TABLE 
delta.`s3a://abhishek-test-01012023/delta-lake-sample-data/`")

    This will create the manifest folder not go to Athena and run this query:

    CREATE EXTERNAL TABLE delta_db.delta_table(id INT, first_name String, last_name String, address String, pincocde INT, net_income INT, source_of_income String, state String, email_id String, description String, population INT, population_1 String, population_2 String, population_3 String, population_4 String, population_5 String, population_6 String, population_7 String) 
PARTITIONED BY (date String)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://abhishek-test-01012023/delta-lake-sample-data/_symlink_format_manifest/'

    MSCK REPAIR TABLE delta_db.delta_table

    You will be able to query the data. 

    Conclusion: A New Dawn

    In a world where data continued to grow in volume and complexity, Delta Lake stood as a beacon of hope. It empowered organizations to manage their data lakes with unprecedented efficiency and extract insights with unwavering confidence.

    The adoption of Delta Lake marked a new dawn in the realm of data. Whether dealing with structured or semi-structured data, it was the answer to the prayers of data warriors. As the sun set on traditional formats, Delta Lake emerged as the hero they had been waiting for—a hero who had not only revolutionized data storage and processing but also transformed the way stories were told in the world of data.

    And so, the legend of Delta Lake continued to unfold, inspiring data adventurers across the land to embark on their own quests, armed with the power of ACID transactions, time travel, and the promise of a brighter, data-driven future.

  • How Cross-Functional Data Collaboration Fuels AI Forecasting in Manufacturing

    Factories have always been noisy places. Not just the machines—the data too. Every department speaks a different dialect: engineering in blueprints, quality in percentages, finance in margins. Somewhere in the middle, meaning flickers for a second and disappears again.

    Now and then, though, someone manages to connect those fragments. And suddenly, forecasting stops being merely about predicting the future. It becomes more about teams listening to each other better.

    What the Big Idea Really Is

    Cross-functional data collaboration sounds grand, but in practice, it is a modest habit: people sharing what they know, even when their worlds don’t perfectly align. When that happens often enough, AI forecasting begins to find its footing.

    1. Context changes everything. Data is mute until someone explains it. When quality and operations and finance look at the same trend together, the AI model stops predicting blindly and starts learning the logic behind the numbers.
    2. Shared language breeds trust. The real progress often happens in small arguments—what counts as a defect, what counts as a delay, and so on. Agreement builds a kind of quiet reliability into the system.
    3. Speed follows understanding. Once the translation work is done, decisions move faster without anyone forcing it. Collaboration shrinks the distance between seeing and acting.
    4. Learning runs both ways. AI models learn from data, yes, but teams learn from the models too—what the system notices, what it misses, what it exaggerates.

    What Research and Experience Suggest

    Manufacturing research increasingly confirms what we experience in practice: AI forecasting only becomes truly powerful when data flows freely within the organization—across functions, not just within silos. A literature review in Applied Sciences highlights that machine learning in manufacturing gains its strategic power when production, quality, and system data are treated as interconnected (e.g., the “Four-Know” framework: what, why, when, how).

    Academic work on decentralized manufacturing also shows that sharing insights between units —for example, through knowledge distillation—improves model performance in under-informed parts of the organization. Meanwhile, quality-prediction research demonstrates that explainability techniques can prune irrelevant features from forecasting models, improving accuracy and interpretability. 

    These findings support the idea that cross-functional collaboration is not optional – it’s a pre-condition for success. When AI models are built on data that reflects operations, engineering, procurement, and quality together, forecasting becomes more accurate, more explainable, and more trusted. That kind of collaboration isn’t just technical: it’s deeply human.

    A Small Illustration to Put This Big Idea into Perspective

    Consider a mid-sized electronics manufacturer. Suddenly, the production team notices a subtle decline in yield. The quality department begins reporting higher-than-usual defect rates. Procurement, however, insists that their suppliers are stable—no obvious issues. On the surface, dashboards look fine. But when a cross-functional data task force (operations, quality, procurement, and data engineers) digs deeper, they uncover a misalignment: data from different functions is recorded in inconsistent formats and timestamps, leading to distorted aggregations.

    Together, they re-align the data – standardizing units, recalibrating timestamps, and merging datasets. Over the next few months, their AI forecasting system begins to issue early warning signals. A similar pattern to the previous yield decline appears in the forecast, but this time the team acts before the issue reaches the shop floor. The early alert isn’t magic; it’s the result of shared understanding—of functions working with the same underlying reality.

    Why This Matters for Manufacturers

    • Agility grows quietly. Teams that talk regularly move faster, even without new tools.
    • Transparency replaces suspicion. When everyone helped build the model, no one treats it like a black box.
    • Improvement loops back. Forecasts feed design; design refines process; process refines data.
    • Resilience hides in plain sight. Shop floors that promote cross-functional data sharing tend to respond better, not louder.

    Finding Rhythm in the Rough Edges

    Collaboration can be tiring. Ownership blurs. Tools overlap. Meetings multiply. There’s a temptation to smooth the friction away. But the friction is the signal. It shows that people are actually engaging, not just aligning through slide decks. In data collaboration, as in music, a little dissonance means the system is alive.

    How R Systems Helps Manufacturers Achieve More Reliable Forecasting

    At R Systems, we’ve learned that collaboration cannot be installed; it has to be engineered gently. We help manufacturers connect their data across functions—through clean integration layers, unified data models, and agentic AI systems that learn from how people already work.

    Our AI solutions don’t replace collaboration; they make it easier to sustain. The systems adapt to team rhythms, not the other way around. Forecasting then becomes a shared act between human intuition and machine precision.

    A Closing Thought – Not by Any Means a Final Word

    If perfect data is the goal, collaboration will always feel messy. But if learning is the goal, the mess starts to look like progress.

    We tend to think of AI forecasting here as only about predicting the trends of tomorrow. In fact, it’s not entirely just that. It’s largely about helping manufacturers, see clearly, the facts of today. When cross-functional teams start collaborating and sharing data, the fog around what’s next begins to clear. Talk to our experts today.

  • How AI Knowledge Engines Accelerate Data-to-Decision Cycles in Manufacturing

    Data runs through every corner of manufacturing, from shop-floor sensors to ERP systems, maintenance logs, and supplier feeds. Yet the journey from data to decision often drags, with insights stuck in silos and actions coming too late. That’s where an AI knowledge engine changes the game. It connects raw data with human expertise, reasons across sources, and turns information into timely, actionable insights. For data and AI professionals and business leaders, it’s a way to move from spotting a signal to making a confident decision and fast.

    From Manufacturing Analytics to Decision Acceleration

    Traditional manufacturing analytics has focused on reporting and visualization with dashboards, KPIs, and trend charts that describe what’s happening. But analytics alone doesn’t drive improvement. The real value lies in faster, smarter decisions: when to intervene, how to optimize, and where to allocate resources.

    An AI knowledge engine transforms that process. It absorbs structured and unstructured data from machine sensors, operator logs, quality reports, and supply chain updates and weaves them into a coherent model. Instead of isolated insights, you get connected intelligence.

    Imagine a scenario where one sensor shows a vibration anomaly while another logs a temperature spike in a different module. The engine correlates these signals, links them to past maintenance records, and infers a likely root cause, say a misaligned component creating downstream defects. The system then recommends a precise corrective action before the problem impacts production.

    That’s not just analytics. That’s decision acceleration.

    What Exactly Is an AI Knowledge Engine?

    The term may sound abstract, but the concept is straightforward. An AI knowledge engine combines data integration, semantic understanding, and reasoning to deliver contextual insights. It doesn’t just retrieve data: it learns, connects, and infers.

    Key components include:

    • Data ingestion – capturing both structured data (sensor readings, ERP data) and unstructured data (notes, images, reports).
    • Semantic modeling – using ontologies and knowledge graphs to represent how components, processes, and events relate to each other.
    • Reasoning and inference – applying AI algorithms to detect patterns, diagnose root causes, and recommend actions.
    • Continuous learning – refining models as new data and feedback come in.

    As some sources describe it, knowledge engines encode not just data but also the expertise i.e. the judgment, intuition, and decision-making logic of seasoned engineers into systems that can reason autonomously.

    In manufacturing, that means moving from descriptive (“what happened”) to prescriptive (“what should we do”) and eventually proactive (“what will happen next”).

    Why Manufacturing Needs AI Knowledge Engines

    Manufacturing is inherently complex. It’s a web of machines, materials, people, and suppliers. Decisions ripple across this network. Here’s where AI knowledge engines create tangible value:

    • Speed: They cut decision latency by automating data synthesis and surfacing insights instantly.
    • Integration: They break down silos between production, maintenance, and supply chain systems.
    • Scalability: They capture the expertise of senior engineers and make it reusable across plants and teams.
    • Predictive power: They detect hidden correlations, for instance, between supplier variability and product quality.
    • Resilience: They enable earlier interventions, reducing downtime and waste.

    Each of these outcomes ties directly to measurable KPIs like faster cycle times, higher first-pass yield, lower maintenance costs, and improved operational stability.

    Making AI Knowledge Engine Work

    Deploying an AI knowledge engine isn’t just about technology. It’s about clarity of purpose and disciplined execution.

    1. Start with the decision — Identify which decisions you want to accelerate: maintenance actions, quality interventions, or supplier responses.
    2. Map the data sources — Ensure access to both quantitative data and qualitative context, like technician notes or operator feedback.
    3. Build the semantic layer — Create a knowledge graph that reflects how your processes and assets relate.
    4. Embed into workflows — Integrate the engine’s insights into control systems, dashboards, or maintenance apps so actions follow naturally.
    5. Govern continuously — Monitor model accuracy, data quality, and explainability to build trust and avoid bias.

    When implemented well, the engine doesn’t replace human decision-makers rather, it empowers them. It brings context and foresight into everyday operations.

    A Quick Use Case

    Take a discrete manufacturing plant producing complex assemblies. Sensors monitor vibration, temperature, and throughput. Maintenance logs record interventions. Supplier data tracks material quality.

    An AI knowledge engine connects all this. When a sensor drift appears, it correlates the signal with past maintenance records and supplier deliveries. Within seconds, it flags a likely cause like a worn bearing from a recent batch and suggests a replacement schedule. The maintenance team acts before a breakdown occurs, preventing a 12-hour downtime.

    That’s data turning into decision: seamlessly.

    The Bottom Line

    Manufacturing has long chased the vision of real-time, insight-driven operations. AI knowledge engines make that vision practical. They don’t just analyze data; they understand it in context, reason over it, and translate it into decisions that matter.

    For data and AI professionals, they represent the next step in operational intelligence. For business leaders, they unlock a faster, more resilient enterprise. The gap between knowing and doing is finally closing and it’s powered by knowledge.

    How R Systems Helps Manufacturers Get There

    At R Systems, we help manufacturers bridge the gap between data and decision through AI-driven knowledge systems, analytics, and domain expertise. Our teams combine manufacturing intelligence, data engineering, and applied AI to design solutions that capture context, automate reasoning, and accelerate insights across operations.

    From predictive maintenance and process optimization to digital twins and AI-powered decision systems, we help you move beyond dashboards toward truly intelligent manufacturing.

    Turn your data into knowledge. Turn your knowledge into action. Talk to our experts today.

  • Enabling Data-Driven Demand Planning with AI-Powered Momentum Volume Projection

    • Unified Forecasting Framework – Developed an AI-powered Momentum Volume Projection model to centralize demand forecasting across all product categories and channels.
    • Accuracy & Visibility – Achieved 5–10% forecast error (MAPE) with reliable projections across 1-, 3-, 6-, and 12-month horizons, enhancing business foresight.
    • Operational Agility – Reduced manual forecasting efforts, improved inventory control, and accelerated planning cycles across residential, commercial, and smart access products.
    • Data-Driven Decisioning – Enabled pricing, promotion, and sales teams with real-time dashboards for proactive, evidence-based actions.
    • Scalability & Governance – Deployed a standardized, low-maintenance ML architecture ensuring consistency, model performance, and ease of extension to new product lines.
    • Strategic Outcomes – Strengthened planning precision, improved financial alignment, and established a future-ready AI foundation for sustainable growth.