Category: Software Engineering & Architecture

  • A Beginner’s Guide to Kubernetes Python Client

    A couple of weeks back, we started working with the Kubernetes Python client to carry out basic operations on its components/ resources, and that’s when we realized how few resources there were (guides, docs) on the internet. So, we experimented and decided to share our findings with the community.

    This article is targeted towards an audience that is familiar with Kubernetes, its usage, and its architecture. This is not a simple Kubernetes guide; it’s about Kubernetes using Python, so as we move further, we may shed light on a few things that are required, but a few will be left for self exploration.

    Kubernetes Overview

    Kubernetes is an open-source container orchestration tool, largely used to simplify the process of deployment, maintenance, etc. in application development. Kubernetes is built to offer highly available, scalable, and reliable applications.

    Generally, kubectl commands are used to create, list, and delete the Kubernetes resources, but for this article, we put on a developer’s hat and use the Python way of doing things. In this article, we learn how to create, manage, and interact with Kubernetes resources using the Kubernetes’ Python library.

    But why, you may ask?

    Well, having an option of doing things programmatically creates potential of endless exciting innovations for developers. Using Python, we can:

    • Create and manage Kubernetes resources dynamically
    • Apply algorithms that change the state, amount of resources in our cluster
    • Build a more robust application with solid alerting and monitoring features

    So, let us begin:

    Kubernetes achieves what it does with the help of its resources. These resources are the building blocks for developing a scalable, reliable application.

    Let’s briefly explore these resources to understand what they are and how exactly they work together in Kubernetes:

    • Node: Simple server, a physical/virtual machine.
    • Pod: Smallest unit of Kubernetes, provides abstraction over a container. Creates a running env/layer on top of the container. Usually runs only one application container but can run multiple as well.
    • Service: Static IP address for the pod. Remains the same even after the pod dies. Also doubles as a load-balancer for multiple pods:
      a) External services are used to make the app accessible through external sources.
      b) Internal services are used when accessibility is to be restricted.
    • Ingress: Additional layer of security and address translation for services. All the requests first go to ingress then forwarded to the service.
    • ConfigMap: External configuration of your app like urls of database or other services.
    • Secret: To store secret/sensitive data like db-credentials, etc., encoded in base_64 format.
    • Volumes: Kubernetes does not manage any data persistence on its own. Volumes are used to persist data generated by pods. It attaches a physical storage to the pod that can be both local or remote like cloud or on-premise servers.
    • Deployment: Defines blueprint of the pods and its replication factor. A layer of abstraction over pods makes the configuration convenient.
    • StatefulSet: Applications that are stateful are created using these to avoid data inconsistency. Same as deployment.

    These are a few of the basic Kubernetes resources. If you want to explore the rest of these resources, you can click here.

    Apart from these resources, there are also namespaces in Kubernetes. You will come across them quite a few times in this article, so here are some noteworthy points for Kubernetes namespaces:

    Kubernetes namespaces: Used to group/organize resources in the cluster. It is like a virtual cluster inside a cluster. Namespaces are used to:

    1. Structure resources
    2. Avoid conflicts between teams
    3. Share services between different environments
    4. Access restrictions
    5. Limiting resources.

    Setting up the system

    The Kubernetes library comes to our aid with quite a few modules, the ones featured in this article are client and config modules from the package; we will be using these two heavily. So, let’s install the Kubernetes Python Client:

    To install the Kubernetes Python client, we make use of Python’s standard package installer pip:

    pip install kubernetes

    For installation from the source, we can refer to this guide from the official Python client git repository. 

    Now that we have the python-kubernetes package installed, we can import it as:

    from kubernetes import client, config

    Loading cluster configurations

    To load our cluster configurations, we can use one of the following methods:

    config.load_kube_config()  # for local environment
    # or
    config.load_incluster_config()

    Executing this will load the configurations for your clusters from your local or remote .kube/config file.

    Interacting with Kubernetes Resources

    Now that we have loaded the configurations, we can use the client module to interact with the resources.

    Get Resources: kubectl get commands are used to list all kinds of resources in a cluster for eg:

    – List nodes: To list all the nodes in the cluster, we fire following kubectl command:

    kubectl get nodes  # lists all the nodes

    In Python, we instantiate CoreV1Api class from client module:

    v1 = client.CoreV1Api()
    v1.list_node()  
    # returns a JSON with all the info like spec, metadata etc, for each node

    – List namespaces: To list all the namespaces in your cluster, by-default lists at least four:

    kubectl get namespaces  
    #	NAME          		 STATUS   	AGE
    #	default       		 Active   	94d
    #	kube-public   		 Active   	94d
    #	kube-system   		 Active   	94d

    In the Python client, we can achieve the same by:

    v1.list_namespace()
    """
    returns a JSON with all the info like spec, metadata for each namespace
    For eg:
    {'api_version': 'v1',
     'items': [{'api_version': None,
            	'kind': None,
            	'metadata': {'annotations': None,
                         	'cluster_name': None,
                         	'creation_timestamp': datetime.datetime(2021, 2, 11, 11, 29, 32, tzinfo=tzutc()),
                         	'deletion_grace_period_seconds': None,
                         	'deletion_timestamp': None,
                         	'finalizers': None,
                         	'generate_name': None,
                         	'generation': None,
                         	'labels': None,
                         	'managed_fields': [{'api_version': 'v1',
                                             	'fields_type': 'FieldsV1',
                                             	'fields_v1': {'f:status': {'f:phase': {}}},
                                             	'manager': 'kube-apiserver',
                                             	'operation': 'Update',
                                             	'time': datetime.datetime(2021, 2, 11, 11, 29, 32, tzinfo=tzutc())}],
                         	'name': 'default',
                         	'namespace': None,
                         	'owner_references': None,
                         	'resource_version': '199',
                         	'self_link': None,
                         	'uid': '3a362d64-437d-45b5-af19-4af9ae2c75fc'},
            	'spec': {'finalizers': ['kubernetes']},
            	'status': {'conditions': None, 'phase': 'Active'}}],
    'kind': 'NamespaceList',
     'metadata': {'_continue': None,
              	'remaining_item_count': None,
              	'resource_version': '69139',
              	'self_link': None}}
    """

    Similarly, we can list all the resources or resources in a particular namespace.

    For example, to list pods in all namespaces:

    v1.list_pod_for_all_namespaces()
    v1.list_persistent_volume_claim_for_all_namespaces()

    For all the resources that can be group within a given namespace, we can use:

    # v1.list_namespaced_pod(<namespace>)
    v1.list_namespaced_pod(namespace=’default’)
    
    # v1.list_namespaced_service(<namespace>)
    v1.list_namespaced_service(namespace=’default’)
    and so on.

    Creating Resources: The usual way to create resources in Kubernetes is to use a kubectl create command with required parameters (defaults if not specified) or to use kubectl apply command, which takes a YAML/JSON format configuration file as input. This file contains all the specifications and metadata for the component to be created. For example:

    kubectl create deployment my-nginx-depl --image=nginx
    kubectl apply -f nginx_depl.yaml

    Where the contents of nginx_depl.yaml could be as follows:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      labels:
        app: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.14.2
            ports:
            - containerPort: 80

    To create resources in Python, though, we use create functions from the same instance of CoreV1Api class:

    # v1.create_namespaced_pod(<namespace>, <body>)
    # v1.create_namespaced_persistent_volume_claim(<namespace>, <body>)

    So, basically, we just need two things: a string type namespace in which we want our resource to be in and a body.

    This body is the same as the config.yaml file that we saw earlier. But, how exactly do we create or use that in our code? We utilize the component specific classes that this library offers us for this.

    Let us take an example, to create a pod we use V1Pod class from the Kubernetes.client.

    An instance of this V1Pod contains all the params like kind, metadata, spec, etc., so all we need to pass them and then we are good. And while we are at it, let’s create metadata and spec as well using a couple more classes.

    1. V1ObjectMeta: This takes all the fields that can be part of metadata as parameters, e.g.

    metadata = client.V1ObjectMeta(name='md1')
    
    # We could also set fields by accessing them through instance like:
    metadata.name = 'md2'

    2. V1Container: If you recall the brief definition of Kubernetes pods given earlier, we realize that pods are just layers above containers, which means we will have to provide the container(s) that the pods abstracts over. The V1Container class from Kubernetes client does just what we need.

    These containers run the specified image, with their name taken as a parameter by the object. Containers also have several other parameters like volume_mounts, ports that can also be passed while instantiation or could be set later using object reference.

    We create a container using:

    # container1 = client.V1Container(<name>, <image>) e.g:
    container1 = client.V1Container(‘my_container’, ‘nginx’)

    Kubernetes pods can have multiple containers running inside, hence the V1PodSpec class expects a list of those while we create a pod spec.

    containers = [container1, container2…]

    3. V1PodSpec: Depending on the component we are working on, the class for its spec and params change. For a pod, we can use V1PodSpec as:

    # pod_spec = client.V1PodSpec(<containers_list>)
    pod_spec = client.V1PodSpec(containers=containers)

    Now that we have both metadata and spec, let’s construct the pod’s body:

    pod_body = client.V1Pod(metadata=metadata, spec=pod_spec, kind='Pod', api_version='v1')

    And then, finally we could pass these to create a pod:

    pod = v1.create_namespaced_pod(namespace=my-namespace, body=pod_body)

    And there you have it, that’s how you create a pod.

    Similarly, we can create other resources, although not all resources take the same set of parameters, for example PersistentVolume (PV in short) does not come under namespaces, it is a cluster wide resource, so naturally it won’t be expecting a namespace parameter.

    Fetching Logs:

    When it comes to monitoring and debugging Kubernetes’ resources, logs play a major role. Using the Kubernetes Python client, we can fetch logs for resources. For example, to fetch logs for a pod:

    Using kubectl:

    # kubectl logs pod_name
    kubectl logs my-pod

    Using Python:

    pod_logs = v1.read_namespaced_pod_log(<pod_name>, <namespace>)
    pod_logs = v1.read_namespaced_pod_log(name=’my-app’, namespace=’default’)

    Deleting Resources: For deletion, we will be following the same class that we have been using so far, i.e kubernetes.client.CoreV1Api.

    There are functions that directly deal with deletion of that component, for example:

    #v1.delete_namespaced_pod(<pod_name>, <namespace>)
    v1.delete_namespaced_pod(name=’my-app’, namespace=’default’)

    Pass the required parameters and the deletion will take place as expected.

    Complete Example for creating a Kubernetes Pod:

    from kubernetes import client, config
    	
    config.load_kube_config()
    v1 = client.CoreV1Api()
    	
    namespaces_list = v1.list_namespace()
    namespaces = [item.metadata.name for item in namespaces_list.items]
    	
    pods_list = v1.list_namespaced_pod(namespace=’default’)
    pods = [item.metadata.name for item in pod_list.items]
    
    containers = []
    container1 = client.V1Container(name=’my-nginx-container’, image=’nginx’)
    containers.append(container1)
    	
    pod_spec = client.V1PodSpec(containers=containers)
    pod_metadata = client.V1ObjectMeta(name=’my-pod’, namespace=’default’)
    
    pod_body = client.V1Pod(api_version=’v1’, kind=’Pod’, metadata=pod_metadata, spec=pod_spec)
    	
    v1.create_namespaced_pod(namespace=’default’, body=pod_body)
    	
    pod_logs = v1.read_namespaced_pod_log(name=’my-pod’, namespace='default')
    
    v1.delete_namespaced_pod(namespace=’default’, name=’my-pod’)

    Conclusion

    There are quite a lot of ways this article could have been written, but as we conclude, it’s quite evident that we have barely scratched the surface. There are many more interesting, advanced things that we can do with this library, but those are beyond the scope of this article.

    We can do almost all the operations with the Python client that we usually do with kubectl on Kubernetes resources. We hope that we managed to keep the content both interesting and informative. 

    If you’re looking for a comprehensive guide on Kubernetes or something interesting to do with it, don’t worry, we’ve got you covered. You can refer to a few of our other articles and might find just what you need:

    1. Kubernetes CSI in Action: Explained with Features and Use Cases

    2. Continuous Deployment with Azure Kubernetes Service, Azure Container Registry & Jenkins

    3. Demystifying High Availability in Kubernetes Using Kubeadm

    References

    1. Official Kubernetes documentation: https://kubernetes.io/docs

    2. Kubernetes Resources: https://kubernetes.io/docs/reference/glossary/?core-object=true

    3. Kubernetes Python client: https://github.com/kubernetes-client/python

    4. Kubclt: https://kubernetes.io/docs/reference/kubectl/overview/

  • Scalable Real-time Communication With Pusher

    What and why?

    Pusher is a hosted API service which makes adding real-time data and functionality to web and mobile applications seamless. 

    Pusher works as a real-time communication layer between the server and the client. It maintains persistent connections at the client using WebSockets, as and when new data is added to your server. If a server wants to push new data to clients, they can do it instantly using Pusher. It is highly flexible, scalable, and easy to integrate. Pusher has exposed over 40+ SDKs that support almost all tech stacks.

    In the context of delivering real-time data, there are other hosted and self-hosted services available. It depends on the use case of what exactly one needs, like if you need to broadcast data across all the users or something more complex having specific target groups. In our use case, Pusher was well-suited, as the decision was based on the easy usage, scalability, private and public channels, webhooks, and event-based automation. Other options which we considered were Socket.IO, Firebase & Ably, etc. 

    Pusher is categorically well-suited for communication and collaboration features using WebSockets. The key difference with  Pusher: it’s a hosted service/API.  It takes less work to get started, compared to others, where you need to manage the deployment yourself. Once we do the setup, it comes to scaling, that reduces future efforts/work.

    Some of the most common use cases of Pusher are:

    1. Notification: Pusher can inform users if there is any relevant change.  Notifications can also be thought of as a form of signaling, where there is no representation of the notification in the UI. Still, it triggers a reaction within an application.

    2. Activity streams: Stream of activities which are published when something changes on the server or someone publishes it across all channels.

    3. Live Data Visualizations: Pusher allows you to broadcast continuously changing data when needed.

    4. Chats: You can use Pusher for peer to peer or peer to multichannel communication.

    In this blog, we will be focusing on using Channels, which is an alias for Pub/Sub messaging API for a JavaScript-based application. Pusher also comes with Chatkit and Beams (Push Notification) SDK/APIs.

    • Chatkit is designed to make chat integration to your app as simple as possible. It allows you to add group chat and 1 to 1 chat feature to your app. It also allows you to add file attachments and online indicators.
    • Beams are used for adding Push Notification in your Mobile App. It includes SDKs to seamlessly manage push token and send notifications.

    Step 1: Getting Started

    Setup your account on the Pusher dashboard and get your free API keys.

    Image Source: Pusher

    1. Click on Channels
    2. Create an App. Add details based on the project and the environment
    3. Click on the App Keys tab to get the app keys.
    4. You can also check the getting started page. It will give code snippets to get you started.

    Add Pusher to your project:

    var express = require('express');
    var bodyParser = require('body-parser');
    
    var app = express();
    app.use(bodyParser.json());
    app.use(bodyParser.urlencoded({ extended: false }));
    
    app.post('/pusher/auth', function(req, res) {
      var socketId = req.body.socket_id;
      var channel = req.body.channel_name;
      var auth = pusher.authenticate(socketId, channel);
      res.send(auth);
    });
    
    var port = process.env.PORT || 5000;
    app.listen(port);

    CODE: https://gist.github.com/velotiotech/f09f14363bacd51446d5318e5050d628.js

    or using npm

    npm i pusher

    CODE: https://gist.github.com/velotiotech/423115d0943c1b882c913e437c529d11.js

    Step 2: Subscribing to Channels

    There are three types of channels in Pusher: Public, Private, and Presence.

    • Public channels: These channels are public in nature, so anyone who knows the channel name can subscribe to the channel and start receiving messages from the channel. Public channels are commonly used to broadcast general/public information, which does not contain any secure information or user-specific data.
    • Private channels: These channels have an access control mechanism that allows the server to control who can subscribe to the channel and receive data from the channel. All private channels should have a private- prefixed to the name. They are commonly used when the sever needs to know who can subscribe to the channel and validate the subscribers.
    • Presence channels: It is an extension to the private channel. In addition to the properties which private channels have, it lets the server ‘register’ users information on subscription to the channel. It also enables other members to identify who is online.

    In your application, you can create a subscription and start listening to events on: 

    // Here my-channel is the channel name
    // all the event published to this channel would be available
    // once you subscribe to the channel and start listing to it.
    
    var channel = pusher.subscribe('my-channel');
    
    channel.bind('my-event', function(data) {
      alert('An event was triggered with message: ' + data.message);
    });

    CODE: https://gist.github.com/velotiotech/d8c27960e2fac408a8db57b92f1e846d.js

    Step 3: Creating Channels

    For creating channels, you can use the dashboard or integrate it with your server. For more details on how to integrate Pusher with your server, you can read (Server API). You need to create an app on your Pusher dashboard and can use it to further trigger events to your app.

    or 

    Integrate Pusher with your server. Here is a sample snippet from our node App:

    var Pusher = require('pusher');
    
    var pusher = new Pusher({
      appId: 'APP_ID',
      key: 'APP_KEY',
      secret: 'APP_SECRET',
      cluster: 'APP_CLUSTER'
    });
    
    // Logic which will then trigger events to a channel
    function trigger(){
    ...
    ...
    pusher.trigger('my-channel', 'my-event', {"message": "hello world"});
    ...
    ...
    }

    CODE: https://gist.github.com/velotiotech/6f5b0f6407c0a74a0bce4b398a849410.js

    Step 4: Adding Security

    As a default behavior, anyone who knows your public app key can open a connection to your channels app. This behavior does not add any security risk, as connections can only access data on channels. 

    For more advanced use cases, you need to use the “Authorized Connections” feature. It authorizes every single connection to your channels, and hence, avoids unwanted/unauthorized connection. To enable the authorization, set up an auth endpoint, then modify your client code to look like this.

    const channels = new Pusher(APP_KEY, {
      cluster: APP_CLUSTER,
      authEndpoint: '/your_auth_endpoint'
    });
    
    const channel = channels.subscribe('private-<channel-name>');

    CODE: https://gist.github.com/velotiotech/9369051e5661a95352f08b1fdd8bf9ed.js

    For more details on how to create an auth endpoint for your server, read this. Here is a snippet from Node.js app

    var express = require('express');
    var bodyParser = require('body-parser');
    
    var app = express();
    app.use(bodyParser.json());
    app.use(bodyParser.urlencoded({ extended: false }));
    
    app.post('/pusher/auth', function(req, res) {
      var socketId = req.body.socket_id;
      var channel = req.body.channel_name;
      var auth = pusher.authenticate(socketId, channel);
      res.send(auth);
    });
    
    var port = process.env.PORT || 5000;
    app.listen(port);

    CODE: https://gist.github.com/velotiotech/fb67d5efe3029174abc6991089a910e1.js

    Step 5: Scale as you grow

     

    Pusher comes with a wide range of plans which you can subscribe to based on your usage. You can scale your application as it grows. Here is a snippet from available plans for mode details you can refer this.

    Image Source: Pusher

    Conclusion

    This article has covered a brief description of Pusher, its use cases, and how you can use it to build a scalable real-time application. Using Pusher may vary based on different use cases; it is no real debate on what one can choose. Pusher approach is simple and API based. It enables developers to add real-time functionality to any application in very little time.

    If you want to get hands-on tutorials/blogs, please visit here.

  • Why You Should Prefer Next.js 12 Over Other React Setup

    If you are coming from a robust framework, such as Angular or any other major full-stack framework, you have probably asked yourself why a popular library like React (yes, it’s not a framework, hence this blog) has the worst tooling and developer experience.

    They’ve done the least amount of work possible to build this framework: no routing, no support for SSR, nor a decent design system, or CSS support. While some people might disagree—“The whole idea is to keep it simple so that people can bootstrap their own framework.” –Dan Abramov. However, here’s the catch: Most people don’t want to go through the tedious process of setting up.

    Many just want to install and start building some robust applications, and with the new release of Next.js (12), it’s more production-ready than your own setup can ever be.

    Before we get started discussing what Next.js 12 can do for us, let’s get some facts straight:

    • React is indeed a library that could be used with or without JSX.
    • Next.js is a framework (Not entirely UI ) for building full-stack applications. 
    • Next.js is opinionated, so if your plan is to do whatever you want or how you want, maybe Next isn’t the right thing for you (mind that it’s for production).
    • Although Next is one of  the most updated code bases and has a massive community supporting it, a huge portion of it is handled by Vercel, and like other frameworks backed by a tech giant… be ready for occasional Vendor-lockin (don’t forget React–[Meta] ).
    • This is not a Next.js tutorial; I won’t be going over Next.js. I will be going over the features that are released with V12 that make it go over the inflection point where Next could be considered as the primary framework for React apps.

    ES module support

    ES modules bring a standardized module system to the entire JS ecosystem. They’re supported by all major browsers and node.js, enabling your build to have smaller package sizes. This lets you use any package using a URL—no installation or build step required—use any CDN that serves ES module as well as the design tools of the future (Framer already does it –https://www.framer.com/ ).

    import Card from 'https://framer.com/m/Card-3Yxh.js@gsb1Gjlgc5HwfhuD1VId';
    import Head from 'next/head';
    
    export default class MyDocument extends Document {
      render() {
        return (
          <>
            <Head>
              <title>URL imports for Next 12</title>
            </Head>
            <div>
              <Card variant='R3F' />
            </div>
          </>
        );
      }
    }

    As you can see, we are importing a Card component directly from the framer CDN on the go with all its perks. This would, in turn, be the start of seamless integration with all your developer environments in the not-too-distant future. If you want to learn more about URL imports and how to enable the alpha version, go here.

    New engine for faster DEV run and production build:

    Next.js 12 comes with a new Rust compiler that comes with a native infrastructure. This is built on top of SWC, an open platform for fast tooling systems. It comes with an impressive stat of having 3 times faster local refresh and 5 times faster production builds.

    Contrary to most productions builds with React using webpack, which come with a ton of overheads and don’t really run on the native system, SWC is going to save you a ton of time that you waste during your mundane workloads.

    Source: Nextjs.org

    Next.js Live:

    If you are anything like me, you’ve probably had some changes that you aren’t really sure about and just want to go through them with the designer, but you don’t really wanna push the code to PROD. Taking a call with the designer and sharing your screen isn’t really the best way to do it. If only there were a way to share your workflow on-the-go with your team with some collaboration feature that just wouldn’t take up an entire day to setup. Well, Next.js Live lets you do just that.

    Source: Next.js

    With the help of ES module system and native support for webassembly, Next.js Live runs entirely on the browser, and irrespective of where you host it, the development engine behind it will soon be open source so that more platforms can actually take advantage of this, but for now, it’s all Next.js.

    Go over to V and do a test run.

    Middleware & serverless: 

    These are just repetitive pieces of code that you think could run on their own out of your actual backend. The best part about this is that you don’t really need to place these close to your backend. Before the request gets completed, you can potentially rewrite, redirect, add headers, or even stream HTML., Depending upon how you host your middleware using Vercel edge functions or lambdas with AWS, they can potentially handle

    • Authentication
    • Bot protection
    • Redirects 
    • Browser support
    • Feature flags 
    • A/B tests
    • Server-side analytics 
    • Logging

    And since this is part of the Next build output, you can technically use any hosting providers with an Edge network (No Vendor lock-in)

    For implementing middleware, we can create a file _middleware inside any pages folder that will run before any requests at that particular route (routename)

    pages/routeName/_middleware.ts. 

    import type { NextFetchEvent } from 'next/server';
    import { NextResponse } from 'next/server';
    export function middleware(event: NextFetchEvent) {
      // gram the user's location or use India for default
      const country = event.request.geo.country.toLowerCase() || 'IND';
    
      //rewrite to static, cached page for each local
      return event.respondWith(NextResponse.rewrite(`/routeName/${country}`));
    }

    Since this middleware, each request will be cached, and  rewriting the response change the URL in your client Next.js can make the difference and still provide you the country flag. 

    Server-side streaming:

    React 18 now supports server-side suspense API and SSR streaming. One big drawback of SSR was that it wasn’t restricted to the strict run time of REST fetch standard. So, in theory, any page that needed heavy lifting from the server could give you higher FCP (first contentful paint). Now this will allow you to stream server-rendered pages using HTTP streaming that will solve your problem for higher render time you can take a look at the alpha version by adding. 

    module.exports = {
      experimental: {
        concurrentFeatures: true
      }
    }

    React server components:

    React server components allow us to render almost everything, including the components themselves inside the server. This is fundamentally different from SSR where you are just generating HTML on the server, with server components, there’s zero client-side Javascript needed, making the rendering process much faster (basically no hydration process). This could also be deemed as including the best parts of server rendering with client-side interactivity.

    import Footer from '../components/Footer';
    import Page from '../components/Page';
    import Story from '../components/Story';
    import fetchData from '../lib/api';
    export async function getServerSideProps() {
      const storyIds = await fetchData('storyIds');
      const data = await Promise.all(
        storyIds.slice(0, 30).map(async (id) => await fetchData(`item/${id}`))
      );
    
      return {
        props: {
          data,
        },
      };
    }
    
    export default function News({ data }) {
      return (
        <Page>
          {data?.map((item, i) => (
            <Story key={i} {...item} />
          ))}
          <Footer />
        </Page>
      );
    }

    As you can see in the above SSR example, while we are fetching the stories from the endpoint, our client is actually waiting for a response with a blank page, and depending upon how fast your APIs are, this is a pretty big problem—and the reason we don’t just use SSR blindly everywhere.

    Now, let’s take a look at a server component example:

    Any file ending with .server.js/.ts will be treated as a server component in your Next.js application. 

    export async function NewsWithData() {
      const storyIds = await fetchData('storyIds');
      return (
        <>
          {storyIds.slice(0, 30).map((id) => {
            return (
              <Suspense fallback={<Spinner />}>
                <StoryWithData id={id} />
              </Suspense>
            );
          })}
        </>
      );
    }
    
    export default function News() {
      return (
        <Page>
          <Suspense fallback={<Spinner />}>
            <NewsWithData />
          </Suspense>
          <Footer />
        </Page>
      );
    }

    This implementation will stream your components progressively and eventually show your data as it gets generated in the server component–by-component. The difference is huge; it will be the next level of code-splitting ,and allow you to do data fetching at the component level and you don’t need to worry about making an API call in the browser.

    And functions like getStaticProps and getserverSideProps will be a liability of the past.

    And this also identifies the React Hooks model, going with the de-centralized component model. It also removes the choice we often need to make between static or dynamic, bringing the best of both worlds. In the future, the feature of incremental static regeneration will be based on a per-component level, removing the all or nothing page caching and in terms will allow decisive / intelligent caching based on your needs.

    Next.js is internally working on a data component, which is basically the React suspense API but with surrogate keys, revalidate, and fallback, which will help to realize these things in the future. Defining your caching semantics at the component level

    Conclusion:

    Although all the features mentioned above are still in the development stage, just the inception of these will take the React world and frontend in general into a particular direction, and it’s the reason you should be keeping it as your default go-to production framework. 

  • Building Scalable and Efficient React Applications Using GraphQL and Relay

    Building a React application is not only about creating a user interface. It also has tricky parts like data fetching, re-render performance, and scalability. Many libraries and frameworks try to solve these problems, like Redux, Sagas, etc. But these tools come with their own set of difficulties.

    Redux gives you a single data source, but all the data fetching and rendering logic is handled by developers. Immer gives you immutable data structures, but one needs to handle the re-render performance of applications.

    GraphQL helps developers design and expose APIs on the backend, but no tool on the client side could utilize the full advantage of the single endpoint and data schema provided by GraphQL.

    In this article, we will learn about Relay as a GraphQL client. What are the advantages of using Relay in your application, and what conventions are required to integrate it?  We’ll also cover how following those conventions will give you a better developer experience and a performant app. We will also see how applications built with Relay are modular, scalable, efficient, and, by default, resilient to change.

    About Relay

    Relay is a JavaScript framework to declaratively fetch and manage your GraphQL data inside a React application. Relay uses static queries and ahead-of-time compilation to help you build a high-performance app. 

    But as the great saying goes, “With great power comes great responsibilities.” Relay comes with a set of costs (conventions), which—when compared with the benefits you get—is well worth it. We will explore the trade-offs in this article.

    The Relay framework is built of multiple modules:

    1. The compiler: This is a set of modules designed to extract GraphQL code from across the codebase and do validations and optimizations during build time.

    2. Relay runtime: A high-performance GraphQL runtime that features a normalized cache for objects and highly optimized read/write operations, simplified abstractions over fetching data fields, garbage collection, subscriptions, and more.

    3. React-relay: This provides the high-level APIs to integrate React with the Relay runtime.

    The Relay compiler runs as a separate process, like how webpack works for React. It keeps watching and compiling the GraphQL code, and in case of errors, it simply does not build your code, which prevents bugs from going into higher environments.

    Fragments

    Fragments are at the heart of how Relay blends with GraphQL. A fragment is a selection of fields on a GraphQL type. 

    fragment Avatar_user on User {
      avatarImgUrl
      firstName
      lastName
      userName
    }

    If we look at the sample fragment definition above, the fragment name, Avatar_user, is not just a random name. One of the Relay framework’s important conventions is that fragments have globally unique fragment names and follow a structure of <modulename>_<propertyname>. The example above is a fragment definition for Avatar_user.</propertyname></modulename>

    This fragment can then be reused throughout the queries instead of selecting the fields manually to render the avatar in each view.

    In the below query, we see the author type, and the first two who liked the blog post can use the fragment definition of Avatar_user

    query GetBlogPost($postId: ID!) {
          blogPostById(id: $postId) {
            author {
              firstName
              lastName
              avatarImgUrl
              userName
            }
            likedBy(first: 2) {
              edges {
                node {
                  firstName
                  lastName
                  avatarImgUrl
                  userName
                }
              }
            }
          }
        }

    Now, our new query with fragments looks like this:

    query GetBlogPost($postId: ID!) {
          blogPostById(id: $postId) {
            author {
              ...Avatar_user
            }
            likedBy(first: 2) {
              edges {
                node {
                  ...Avatar_user
                }
              }
            }
          }
        }

    Fragments not only allow us to reuse the definitions but more essentially, they let us add or remove fields needed to render our avatar as we evolve our application.

    Another highly important client-side convention is colocation. This means the data required for a component lives inside the component. This makes maintenance and extending much easier. Just like how React allows us to break our UI elements into components and group/compose different views, fragments in Relay allow us to split the data definitions and colocate the data and the view definitions.

    So, a good practice is to define single or multiple fragments that contain the data component to be rendered. This means that a component depends on some fields from the user type, irrespective of the parent component. In the example above, the <avatar> component will render an avatar using the fields specified in the Avatar_user fragment named.</avatar>

    How Relay leverages the GraphQL Fragment

    Relay wants all components to enlist all the data it needs to render, along with the component itself. Relay uses data and fragments to integrate the component and its data requirement. This convention mandates that every component lists the fields it needs access to. 

    Other advantages of the above are:

    1. Components are not dependent on data they don’t explicitly request.
    2. Components are modular and self-contained.
    3. Reusing and refactoring the components becomes easier.

    Performance

    In Relay, the component re-renders only when its exact fields change, and this feature available is out of the box. The fragment subscribes to updates specifically for data the component selects. This lets Relay enhance how the view is updated, and performance is not affected as codebase scales.

    Now, let’s look at an example of components in a single post of a blog application. Here is a wireframe of a sample post to give an idea of the data and view required.

    Now, let’s write a plain query without Relay, which will fetch all the data in a single query. It will look like this for the above wireframe:

    query GetBlogPost($postId: ID!) {
          blogPostById(id: $postId) {
            author {
              firstName
              lastName
              avatarUrl
              shortBio
            }
            title
            coverImgUrl
            createdAt
            tags {
              slug
              shortName
            }
            body
            likedByMe
            likedBy(first: 2) {
              totalCount
              edges {
                node {
                  firstName
                  lastName
                  avatarUrl
                }
              }
            }
          }
        }

    This one query has all the necessary data. Let’s also write down a sample structure of UI components for the query above:

    <BlogPostContainer>
        <BlogPostHead>
          <BlogPostAuthor>
            <Avatar />
          </BlogPostAuthor>
        </BlogPostHead>
        <BlogPostBody>
          <BlogPostTitle />
          <BlogPostMeta>
            <CreatedAtDisplayer />
            <TagsDisplayer />
          </BlogPostMeta>
          <BlogPostContent />
          <LikeButton>
            <LikedByDisplayer />
          </LikeButton>
        </BlogPostBody>
     </BlogPostContainer>

    In the implementation above, we have a single query that will be managed by the top-level component. It will be the top-level component’s responsibility to fetch the data and pass it down as props. Now, we will look at how we would build this in Relay:

    import * as React from "react";
        import { GetBlogPost } from "./__generated__/GetBlogPost.graphql";
        import { useLazyLoadQuery } from "react-relay/hooks";
        import { BlogPostHead } from "./BlogPostHead";
        import { BlogPostBody } from "./BlogPostBody";
        import { graphql } from "react-relay";
    
    
        interface BlogPostProps {
          postId: string;
        }
    
        export const BlogPost = ({ postId }: BlogPostProps) => {
          const { blogPostById } = useLazyLoadQuery<GetBlogPost>(
            graphql`
              query GetBlogPost($postId: ID!) {
                blogPostById(id: $postId) {
                  ...BlogPostHead_blogPost
                  ...BlogPostBody_blogPost
                }
              }
            `,
            {
              variables: { postId }
            }
          );
    
          if (!blogPostById) {
            return null;
          }
    
          return (
            <div>
              <BlogPostHead blogPost={blogPostById} />
              <BlogPostBody blogPost={blogPostById} />
            </div>
          );
        };

    First, let’s look at the query used inside the component:

    const { blogPostById } = useLazyLoadQuery<GetBlogPost>(
    graphql`
      query GetBlogPost($postId: ID!) {
        blogPostById(id: $postId) {
          ...BlogPostHead_blogPost
          ...BlogPostBody_blogPost
        }
      }
    `,
    {
      variables: { postId }
    }
    );

    The useLazyLoadQuery React hook from Relay will start fetching the GetBlogPost query just as the component renders. 

    NOTE: The useLazyLoadQuery is used here as it follows a common mental model of fetching data after the page is loaded. However, Relay encourages data to be fetched as early as possible using the usePreladedQuery hook. 

    For type safety, we are annotating the useLazyLoadQuery with the type GetBlogPost, which is imported from ./__generated__/GetBlogPost.graphql. This file is auto-generated and synced by the Relay compiler. It contains all the information about the types needed to be queried, along with the return type of data and the input variables for the query.

    The Relay compiler takes all the declared fragments in the codebase and generates the type files, which can then be used to annotate a particular component.

    The GetBlogPost query is defined by composing multiple fragments. Another great aspect of Relay is that there is no need to import the fragments manually. They are automatically included by the Relay compiler. Building the query by composing fragments, just like how we compose our component, is the key here. 

    Another approach can be to define queries per component, which takes full responsibility for its data requirements. But this approach has two problems: 

    1. Multiple queries are sent to the server instead of one.

    2. The loading will be slower as components would have to wait till they render to start fetching the data.

    In the above example, the GetBlogPost only deals with including the fragments for its child components, BlogPostHead and BlogPostBody. It is kept hidden from the actual data fields of the children component.

    When using Relay, components define their data requirement by themselves. These components can then be composed along with other components that have their own separate data. 

    At the same time, no component knows what data the other component needs except from the GraphQL type that has the required component data. Relay makes sure the right data is passed to the respective component, and all input for a query is sent to the server.

    This allows developers to think only about the component and fragments as one while Relay does all the heavy lifting in the background. Relay minimizes the round-trips to the server by placing the fragments from multiple components into optimized and efficient batches. 

    As we said earlier, the two fragments, BlogPostHead_blogPost and BlogPostBody_blogPost, which we referenced in the query, are not imported manually. This is because Relay imposes unique fragment names globally so that the compiler can include the definitions in queries sent to the server. This eliminates the chances of errors and takes away the laborious task of referencing the fragments by hand. 

     if (!blogPostById) {
          return null;
      }
    
      return (
        <div>
          <BlogPostHead blogPost={blogPostById} />
          <BlogPostBody blogPost={blogPostById} />
        </div>
      );

    Now, in the rendering logic above, we render the <BlogPostHead/> and <BlogPostBody/> and pass the blogPostById object as prop. It’s passed because it is the object inside the query that spreads the fragment needed by the two components. This is how Relay transfers fragment data. Because we spread both fragments on this object, it is guaranteed to satisfy both components.

    To put it into simpler terms, we say that to pass the fragment data, we pass the object where the fragment is spread, and the component then uses this object to get the real fragment data. Relay, through its robust type systems, makes sure that the right object is passed with required fragment spread on it.

    The previous component, the BlogPost, was the Parent component, i.e., the component with the root query object. The root query is necessary because it cannot fetch a fragment in isolation. Fragments must be included in the root query in a parent component. The parent can, in turn, be a fragment as long the root query exists in the hierarchy. Now, we will build the BlogPostHead component using fragments:

     import * as React from "react";
        import { useFragment } from "react-relay/hooks";
        import { graphql } from "react-relay";
        import {
          BlogPostHead_blogPost$key, BlogPostHead_blogPost
        } from "./__generated__/BlogPostHead_blogPost.graphql";
        import { BlogPostAuthor } from "./BlogPostAuthor";
        import { BlogPostLikeControls } from "./BlogPostLikeControls";
    
        interface BlogPostHeadProps {
          blogPost: BlogPostHead_blogPost$key;
        }
    
        export const BlogPostHead = ({ blogPost }: BlogPostHeadProps) => {
          const blogPostData = useFragment<BlogPostHead_blogPost>(
            graphql`
              fragment BlogPostHead_blogPost on BlogPost {
                title
                coverImgUrl
                ...BlogPostAuthor_blogPost
                ...BlogPostLikeControls_blogPost
              }
            `,
            blogPost
          );
    
          return (
            <div>
              <img src={blogPostData.coverImgUrl} />
              <h1>{blogPostData.title}</h1>
              <BlogPostAuthor blogPost={blogPostData} />
              <BlogPostLikeControls blogPost={blogPostData} />
            </div>
          );
        };

    NOTE: In our example, the BlogPostHead and BlogPostBody define only one fragment, but in general, a component can have any number of fragments or GraphQL types and even more than one fragments on the same type.

    In the component above, two type definitions, namely BlogPostHead_blogPost$key and BlogPostHead_blogPost, are imported from the file BlogPostHead_blogPost.graphql, generated by the Relay compiler. The compiler extracts the fragment code from this file and generates the types. This process is followed for all the GraphQL code—queries, mutations, fragments, and subscriptions.

    The blogPostHead_blogPost has the fragment type definitions, which is then passed to the useFragment hook to ensure type safety when using the data from the fragment. The other import, blogPostHead_blogPost$key, is used in the interface Props { … }, and this type definition makes sure that we pass the right object to useFragment. Otherwise,  the type system will throw errors during build time. In the above child component, the blogPost object is received as a prop and is passed to useFragment as a second parameter. If the blogPost object did not have the correct fragment, i.e., BlogPostHead_blogPost, spread on it, we would have received a type error. Even if there were another fragment with exact same data selection spread on it, Relay makes sure it’s the right fragment that we use with the useFragement. This allows you to change the update fragment definitions without affecting other components.

    Data masking

    In our example, the fragment BlogPostHead_blogPost explicitly selects two fields for the component:

    1. title
    2. coverImgUrl

    This is because we use/access only these two fields in the view for the <blogposthead></blogposthead> component. So, even if we define another fragment, BlogPostAuthor_blogPost, which selects the title and coverImgUrl, we don’t receive access to them unless we ask for them in the same fragment. This is enforced by Relay’s type system both at compile time and at runtime. This safety feature of Relay makes it impossible for components to depend on data they do not explicitly select. So, developers can refactor the components without risking other components. To reiterate, all components and their data dependencies are self-contained.

    The data for this component, i.e., title and coverImgUrl, will not be accessible on the parent component, BlogPost, even though the props object is sent by the parent. The data becomes available only through the useFragment React hook. This hook can consume the fragment definition. The useFragment takes in the fragment definition and the object where the fragment is spread to get the data listed for the particular fragment.  

    Just like how we spread the fragment for the BlogPostHead component in the BlogPost root query, we an also extend this to the child components of BlogPostHead. We spread the fragments, i.e., BlogPostAuthor_blogPost, BlogPostLikeControls_blogPost, since we are rendering <BlogPostAuthor /> and <BlogPostLikeControls />.

    NOTE: The useFragment hook does not fetch the data. It can be thought of as a selector that grabs only what is needed from the data definitions.

    Performance

    When using a fragment for a component, the component subscribes only to the data it depends on. In our example, the component BlogPostHead will only automatically re-render when the fields “coverImgUrl” or “title” change for a specific blog post the component renders. Since the BlogPostAuthor_blogPost fragment does not select those fields, it will not re-render. Subscription to any updates is made on fragment level. This is an essential feature that works out of the box with Relay for performance.

    Let us now see how general data and components are updated in a different GraphQL framework than Relay. The data that gets rendered on view actually comes from an operation that requests data from the server, i.e., a query or mutation. We write the query that fetches data from the server, and that data is passed down to different components as per their needs as props. The data flows from the root component, i.e., the component with the query, down to the components. 

    Let’s look at a graphical representation of the data flow in other GraphQL frameworks:

    Image source: Dev.to

    NOTE: Here, the framework data store is usually referred to as cache in most frameworks:

    1. The Profile component executes the operation ProfileQuery to a GraphQL server.

    2. The data return is kept in some framework-specific representation of the data store.

    3. The data is passed to the view rendering it.

    4. The view then passes on the data to all the child components who need it. Example: Name, Avatar, and Bio. And finally React renders the view.

    In contrast, the Relay framework takes a different approach:

    Image source: Dev.to

    Let’s breakdown the approach taken by Relay: 

    • For the initial part, we see nothing changes. We still have a query that is sent to the GraphQL server and the data is fetched and stored in the Relay data store.
    • What Relay does after this is different. The components get the data directly from the cache-store(data store). This is because the fragments help Relay integrate deeply with the component data requirements.The component fragments get the data straight from the framework data store and do not rely on data to be passed down as props. Although some information is passed from the query to the fragments used to look up the particular data needed from the data store, the data is fetched by the fragment itself.

    To conclude the above comparison, in other frameworks (like Apollo), the component uses the query as the data source. The implementation details of how the root component executing the query sends data to its descendants is left to us. But Relay takes a different approach of letting the component take care of the data in needs from the data store.

    In an approach used by other GraphQL frameworks, the query is the data source, and updates in the data store forces the component holding the query to re-render. This re-render cascades down to any number of components even if those components do not have to do anything with the updated data other than acting as a layer to pass data from parent to child. In the Relay approach, the components directly subscribe to the updates for the data used. This ensures the best performance as our app scales in size and complexity.

    Developer Experience

    Relay removes the responsibility of developers to route the data down from query to the components that need it. This eliminates the changes of developer error. There is no way for a component to accidentally or deliberately depend on data that it should be just passing down in the component tree if it cannot access it. All the hard work is taken care of by the Relay framework if we follow the conventions discussed.

    Conclusion

    To summarize, we detailed all the work Relay does for us and the effects:

    • The type system of the Relay framework makes sure the right components get the right data they need. Everything in Relay revolves around fragments.
    • In Relay, fragments are coupled and colocated with components, which allows it to mask the data requirements from the outside world. This increases the readability and modularity.
    • By default, Relay takes care of performance as components only re-render when the exact data they use change in the data store.
    • Type generation is a main feature of Relay compiler. Through type generation, interactions with the fragment’s data is typesafe.

    Conventions enforced by Relay’s philosophy and architecture allows it to take advantage of the information available about your component. It knows the exact data dependencies and types. It uses all this information to do a lot of work that developers are required to deal with.

    Related Articles

    1. Enable Real-time Functionality in Your App with GraphQL and Pusher

    2. Build and Deploy a Real-Time React App Using AWS Amplify and GraphQL

  • Building a WebSocket Service with AWS Lambda & DynamoDB

    WebSocket is an effective way for full-duplex, real-time communication between a web server and a client. It is widely used for building real-time web applications along with helper libraries that offer better features. Implementing WebSockets requires a persistent connection between two parties. Serverless functions are known for short execution time and non-persistent behavior. However, with the API Gateway support for WebSocket endpoints, it is possible to implement a Serverless service built on AWS Lambda, API Gateway, and DynamoDB.

    Prerequisites

    A basic understanding of real-time web applications will help with this implementation. Throughout this article, we will be using Serverless Framework for developing and deploying the WebSocket service. Also, Node.js is used to write the business logic. 

    Behind the scenes, Serverless uses Cloudformation to create various required resources, like API Gateway APIs, AWS Lambda functions, IAM roles and policies, etc.

    Why Serverless?

    Serverless Framework abstracts the complex syntax needed for creating the Cloudformation stacks and helps us focus on the business logic of the services. Along with that, there are a variety of plugins available that help developing serverless applications easier.

    Why DynamoDB?

    We need persistent storage for WebSocket connection data, along with AWS Lambda. DynamoDB, a serverless key-value database from AWS, offers low latency, making it a great fit for storing and retrieving WebSocket connection details.

    Overview

    In this application, we’ll be creating an AWS Lambda service that accepts the WebSocket connections coming via API Gateway. The connections and subscriptions to topics are persisted using DynamoDB. We will be using ws for implementing basic WebSocket clients for the demonstration. The implementation has a Lambda consuming WebSocket that receives the connections and handles the communication. 

    Base Setup

    We will be using the default Node.js boilerplate offered by Serverless as a starting point.

    serverless create --template aws-nodejs

    A few of the Serverless plugins are installed and used to speed up the development and deployment of the Serverless stack. We also add the webpack config given here to support the latest JS syntax.

    Adding Lambda role and policies:

    The lambda function requires a role attached to it that has enough permissions to access DynamoDB and Execute API. These are the links for the configuration files:

    Link to dynamoDB.yaml

    Link to lambdaRole.yaml

    Adding custom config for plugins:

    The plugins used for local development must have the custom config added in the yaml file.

    This is how our serverless.yaml file should look like after the base serverless configuration:

    service: websocket-app
    frameworkVersion: '2'
    custom:
     dynamodb:
       stages:
         - dev
       start:
         port: 8000
         inMemory: true
         heapInitial: 200m
         heapMax: 1g
         migrate: true
         convertEmptyValues: true
     webpack:
       keepOutputDirectory: true
       packager: 'npm'
       includeModules:
         forceExclude:
           - aws-sdk
     
    provider:
     name: aws
     runtime: nodejs12.x
     lambdaHashingVersion: 20201221
    plugins:
     - serverless-dynamodb-local
     - serverless-plugin-existing-s3
     - serverless-dotenv-plugin
     - serverless-webpack
     - serverless-offline
    resources:
     - Resources: ${file(./config/dynamoDB.yaml)}
     - Resources: ${file(./config/lambdaRoles.yaml)}
    functions:
     hello:
       handler: handler.hello

    Add WebSocket Lambda:

    We need to create a lambda function that accepts WebSocket events from API Gateway. As you can see, we’ve defined 3 WebSocket events for the lambda function.

    • $connect
    • $disconnect
    • $default

    These 3 events stand for the default routes that come with WebSocket API Gateway offering. $connect and $disconnect are used for initialization and termination of the socket connection, where $default route is for data transfer.

    functions:
     websocket:
       handler: lambda/websocket.handler
       events:
         - websocket:
             route: $connect
         - websocket:
             route: $disconnect
         - websocket:
             route: $default

    We can go ahead and update how data is sent and add custom WebSocket routes to the application.

    The lambda needs to establish a connection with the client and handle the subscriptions. The logic for updating the DynamoDB is written in a utility class client. Whenever a connection is received, we create a record in the topics table.

    console.log(`Received socket connectionId: ${event.requestContext && event.requestContext.connectionId}`);
           if (!(event.requestContext && event.requestContext.connectionId)) {
               throw new Error('Invalid event. Missing `connectionId` parameter.');
           }
           const connectionId = event.requestContext.connectionId;
           const route = event.requestContext.routeKey;
           console.log(`data from ${connectionId} ${event.body}`);
           const connection = new Client(connectionId);
           const response = { statusCode: 200, body: '' };
     
           if (route === '$connect') {
               console.log(`Route ${route} - Socket connectionId connectedconected: ${event.requestContext && event.requestContext.connectionId}`);
               await new Client(connectionId).connect();
               return response;
           } 

    The Client utility class internally creates a record for the given connectionId in the DynamoDB topics table.

    async subscribe({ topic, ttl }) {
       return dynamoDBClient
         .put({ 
            Item: {
             topic,
             connectionId: this.connectionId,
            ttl: typeof ttl === 'number' ? ttl : Math.floor(Date.now() / 1000) + 60 * 60 * 2,
           },
           TableName: process.env.TOPICS_TABLE,
         }).promise();
     }

    Similarly, for the $disconnect route, we remove the INITIAL_CONNECTION topic record when a client disconnects.

    else if (route === '$disconnect') {
     console.log(`Route ${route} - Socket disconnected: ${ event.requestContext.connectionId}`);
               await new Client(connectionId).unsubscribe();
               return response;
           }

    The client.unsubscribe method internally removes the connection entry from the DynamoDB table. Here, the getTopics method fetches all the topics the particular client has subscribed to.

    async unsubscribe() {
       const topics = await this.getTopics();
       if (!topics) {
         throw Error(`Topics got undefined`);
       }
       return this.removeTopics({
         [process.env.TOPICS_TABLE]: topics.map(({ topic, connectionId }) => ({
           DeleteRequest: { Key: { topic, connectionId } },
         })),
       });
     }

    Now comes the default route part of the lambda where we customize message handling. In this implementation, we’re relaying our message handling based on the event.body.type, which indicates what kind of message is received from the client to server. The subscribe type here is used to subscribe to new topics. Similarly, the message type is used to receive the message from one client and then publish it to other clients who have subscribed to the same topic as the sender.

    console.log(`Route ${route} - data from ${connectionId}`);
               if (!event.body) {
                   return response;
               }
               let body = JSON.parse(event.body);
               const topic = body.topic;
               if (body.type === 'subscribe') {
                   connection.subscribe({ topic });
                   console.log(`Client subscribing for topic: ${topic}`);
               }
               if (body.type === 'message') {
                   await new Topic(topic).publishMessage({ data: body.message });
                   console.error(`Published messages to subscribers`);
                   return response;
               }
               return response;

    Similar to $connect, the subscribe type of payload, when received, creates a new subscription for the mentioned topic.

    Publishing the messages

    Here is the interesting part of this lambda. When a client sends a payload with type message, the lambda calls the publishMessage method with the data received. The method gets the subscribers active for the topic and publishes messages using another utility TopicSubscriber.sendMessage

    async publishMessage(data) {
       const subscribers = await this.getSubscribers();
       const promises = subscribers.map(async ({ connectionId, subscriptionId }) => {
         const TopicSubscriber = new Client(connectionId);
           const res = await TopicSubscriber.sendMessage({
             id: subscriptionId,
             payload: { data },
             type: 'data',
           });
           return res;
       });
       return Promise.all(promises);
     }

    The sendMessage executes the API endpoint, which is the API Gateway URL after deployment. As we’re using serverless-offline for the local development, the IS_OFFLINE env variable is automatically set.

    const endpoint =  process.env.IS_OFFLINE ? 'http://localhost:3001' : process.env.PUBLISH_ENDPOINT;
       console.log('publish endpoint', endpoint);
       const gatewayClient = new ApiGatewayManagementApi({
         apiVersion: '2018-11-29',
         credentials: config,
         endpoint,
       });
       return gatewayClient
         .postToConnection({
           ConnectionId: this.connectionId,
           Data: JSON.stringify(message),
         })
         .promise();

    Instead of manually invoking the API endpoint, we can also use DynamoDB streams to trigger a lambda asynchronously and publish messages to topics.

    Implementing the client

    For testing the socket implementation, we will be using a node.js script ws-client.js. This creates two nodejs ws clients: one that sends the data and another that receives it.

    const WebSocket = require('ws');
    const sockedEndpoint = 'http://0.0.0.0:3001';
    const ws1 = new WebSocket(sockedEndpoint, {
     perMessageDeflate: false
    });
    const ws2 = new WebSocket(sockedEndpoint, {
     perMessageDeflate: false
    });

    The first client on connect sends the data at an interval of one second to a topic named general. The count increments each send.

    ws1.on('open', () => {
       console.log('WS1 connected');
       let count = 0;
       setInterval(() => {
         const data = {
           type: 'message',
           message: `count is ${count}`,
           topic: 'general'
         }
         const message  = JSON.stringify(data);
         ws1.send(message, (err) => {
           if(err) {
             console.log(`Error occurred while send data ${err.message}`)
           }
           console.log(`WS1 OUT ${message}`);
         })
         count++;
       }, 15000)
    })

    The second client on connect will first subscribe to the general topic and then attach a handler for receiving data.

    ws2.on('open', () => {
     console.log('WS2 connected');
     const data = {
       type: 'subscribe',
       topic: 'general'
     }
     ws2.send(JSON.stringify(data), (err) => {
       if(err) {
         console.log(`Error occurred while send data ${err.message}`)
       }
     })
    });
    ws2.on('message', ( message) => {
     console.log(`ws2 IN ${message}`);
    });

    Once the service is running, we should be able to see the following output, where the two clients successfully sharing and receiving the messages with our socket server.

    Conclusion

    With API Gateway WebSocket support and DynamoDB, we’re able to implement persistent socket connections using serverless functions. The implementation can be improved and can be as complex as needed.

    WebSocket is an effective way for full-duplex, real-time communication between a web server and a client. It is widely used for building real-time web applications along with helper libraries that offer better features. Implementing WebSockets requires a persistent connection between two parties. Serverless functions are known for short execution time and non-persistent behavior. However, with the API Gateway support for WebSocket endpoints, it is possible to implement a Serverless service built on AWS Lambda, API Gateway, and DynamoDB.

  • Micro Frontends: Reinventing UI In The Microservices World

    It is amazing how the software industry has evolved. Back in the day, a software was a simple program. Some of the first software applications like The Apollo Missions Landing modules and Manchester Baby were basic stored procedures. Software was primarily used for research and mathematical purposes.

    The invention of personal computers and the prominence of the Internet changed the software world. Desktop applications like word processors, spreadsheets, and games grew. Websites gradually emerged. Back then, simple pages were delivered to the client as static documents for viewing. By the mid-1990s, with Netscape introducing client-side scripting language, JavaScript and Macromedia bringing in Flash, the browser became more powerful, allowing websites to become richer and more interactive. In 1999, the Java language introduced Servlets. And thus born the Web Application. Nevertheless, these developments and applications were still simpler. Engineers didn’t emphasize enough on structuring them and mostly built unstructured monolithic applications.

    The advent of disruptive technologies like cloud computing and Big data paved the way for more intricate, convolute web and native mobile applications. From e-commerce and video streaming apps to social media and photo editing, we had applications doing some of the most complicated data processing and storage tasks. The traditional monolithic way now posed several challenges in terms of scalability, team collaboration and integration/deployment, and often led to huge and messy The Ball Of Mud codebases.

    Fig: Monolithic Application Problems – Source

    To untangle this ball of software, came in a number of service-oriented architectures. The most promising of them was Microservices – breaking an application into smaller chunks that can be developed, deployed and tested independently but worked as a single cohesive unit. Its benefits of scalability and ease of deployment by multiple teams proved as a panacea to most of the architectural problems. A few front-end architectures also came up, such as MVC, MVVM, Web Components, to name a few. But none of them were fully able to reap the benefits of Microservices.

    Fig: Microservice Architecture – Source

    ‍Micro Frontends: The Concept‍

    Micro Frontends first came up in ThoughtWorks Technology Radar where they assessed, tried and eventually adopted the technology after noticing significant benefits. It is a Microservice approach to front-end web development where independently deliverable front-end applications are composed as a whole. 

    With Microservices, Micro Frontends breaks the last monolith to create a complete Micro-Architecture design pattern for web applications. It is entirely composed of loosely coupled vertical slices of business functionality rather than in horizontals. We can term these verticals as ‘Microapps’. This concept is not new and has appeared in Scaling with Microservices and Vertical Decomposition. It first presented the idea of every vertical being responsible for a single business domain and having its presentation layer, persistence layer, and a separate database. From the development perspective, every vertical is implemented by exactly one team and no code is shared among different systems.

    Fig: Micro Frontends with Microservices (Micro-architecture)

    Why Micro Frontends?

    A microservice architecture has a whole slew of advantages when compared to monolithic architectures.

    Ease of Upgrades – Micro Frontends build strict bounded contexts in the application. Applications can be updated in a more incremental and isolated fashion without worrying about the risks of breaking up another part of the application.

    Scalability – Horizontal scaling is easy for Micro Frontends. Each Micro Frontend has to be stateless for easier scalability.

    Ease of deployability: Each Micro Frontend has its CI/CD pipeline, that builds, tests and deploys it to production. So it doesn’t matter if another team is working on a feature and has pushed a bug fix or if a cutover or refactoring is taking place. There should be no risks involved in pushing changes done on a Micro Frontend as long as there is only one team working on it.

    Team Collaboration and Ownership: The Scrum Guide says that “Optimal Development Team size is small enough to remain nimble and large enough to complete significant work within a Sprint”. Micro Frontends are perfect for multiple cross-functional teams that can completely own a stack (Micro Frontend) of an application from UX to Database design. In case of an E-commerce site, the Product team and the Payment team can concurrently work on the app without stepping on each other’s toes.

    Micro Frontend Integration Approaches

    There is a multitude of ways to implement Micro Frontends. It is recommended that any approach for this should take a Runtime integration route instead of a Build Time integration, as the former has to re-compile and release on every single Micro Frontend to release any one of the Micro Frontend’s changes.

    We shall learn some of the prominent approaches of Micro Frontends by building a simple Pet Store E-Commerce site. The site has the following aspects (or Microapps, if you will) – Home or Search, Cart, Checkout, Product, and Contact Us. We shall only be working on the Front-end aspect of the site. You can assume that each Microapp has a microservice dedicated to it in the backend. You can view the project demo here and the code repository here. Each way of integration has a branch in the repo code that you can check out to view.

    Single Page Frontends –

    The simplest way (but not the most elegant) to implement Micro Frontends is to treat each Micro Frontend as a single page.

    Fig: Single Page Micro Frontends: Each HTML file is a frontend.
    !DOCTYPE html>
    <html lang="zxx">
    <head>
    	<title>The MicroFrontend - eCommerce Template</title>
    </head>
    <body>
      <header class="header-section header-normal">
        <!-- Header is repeated in each frontend which is difficult to maintain -->
        ....
        ....
      </header
      <main>
      </main>
      <footer
        <!-- Footer is repeated in each frontend which means we have to multiple changes across all frontends-->
      </footer>
      <script>
        <!-- Cross Cutting features like notification, authentication are all replicated in all frontends-->
      </script>
    </body>

    It is one of the purest ways of doing Micro Frontends because no container or stitching element binds the front ends together into an application. Each Micro Frontend is a standalone app with each dependency encapsulated in it and no coupling with the others. The flipside of this approach is that each frontend has a lot of duplication in terms of cross-cutting concerns like headers and footers, which adds redundancy and maintenance burden.

    JavaScript Rendering Components (Or Web Components, Custom Element)-

    As we saw above, single-page Micro Frontend architecture has its share of drawbacks. To overcome these, we should opt for an architecture that has a container element that builds the context of the app and the cross-cutting concerns like authentication, and stitches all the Micro Frontends together to create a cohesive application.

    // A virtual class from which all micro-frontends would extend
    class MicroFrontend {
      
      beforeMount() {
        // do things before the micro front-end mounts
      }
    
      onChange() {
        // do things when the attributes of a micro front-end changes
      }
    
      render() {
        // html of the micro frontend
        return '<div></div>';
      }
    
      onDismount() {
        // do things after the micro front-end dismounts 
      }
    }

    class Cart extends MicroFrontend {
      beforeMount() {
        // get previously saved cart from backend
      }
    
      render() {
        return `<!-- Page -->
        <div class="page-area cart-page spad">
          <div class="container">
            <div class="cart-table">
              <table>
                <thead>
                .....
                
         `
      }
    
      addItemToCart(){
        ...
      }
        
      deleteItemFromCart () {
        ...
      }
    
      applyCouponToCart() {
        ...
      }
        
      onDismount() {
        // save Cart for the user to get back to afterwards
      }
    }

    class Product extends MicroFrontend {
      static get productDetails() {
        return {
          '1': {
            name: 'Cat Table',
            img: 'img/product/cat-table.jpg'
          },
          '2': {
            name: 'Dog House Sofa',
            img: 'img/product/doghousesofa.jpg'
          },
        }
      }
      getProductDetails() {
        var urlParams = new URLSearchParams(window.location.search);
        const productId = urlParams.get('productId');
        return this.constructor.productDetails[productId];
      }
      render() {
        const product = this.getProductDetails();
        return `	<!-- Page -->
        <div class="page-area product-page spad">
          <div class="container">
            <div class="row">
              <div class="col-lg-6">
                <figure>
                  <img class="product-big-img" src="${product.img}" alt="">`
      }
      selectProductColor(color) {}
    
      selectProductSize(size) {}
     
      addToCart() {
        // delegate call to MicroFrontend Cart.addToCart function
      }
      
    }

    <!DOCTYPE html>
    <html lang="zxx">
    <head>
    	<title>PetStore - because Pets love pampering</title>
    	<meta charset="UTF-8
      <link rel="stylesheet" href="css/style.css"/>
    
    </head>
    <body>
    	<!-- Header section -->
    	<header class="header-section">
      ....
      </header>
    	<!-- Header section end -->
    	<main id='microfrontend'>
        <!-- This is where the Micro-frontend gets rendered by utility renderMicroFrontend.js-->
    	</main>
                                    <!-- Header section -->
    	<footer class="header-section">
      ....
      </footer>
    	<!-- Footer section end -->
      	<script src="frontends/MicroFrontend.js"></script>
    	<script src="frontends/Home.js"></script>
    	<script src="frontends/Cart.js"></script>
    	<script src="frontends/Checkout.js"></script>
    	<script src="frontends/Product.js"></script>
    	<script src="frontends/Contact.js"></script>
    	<script src="routes.js"></script>
    	<script src="renderMicroFrontend.js"></script>

    function renderMicroFrontend(pathname) {
      const microFrontend = routes[pathname || window.location.hash];
      const root = document.getElementById('microfrontend');
      root.innerHTML = microFrontend ? new microFrontend().render(): new Home().render();
      $(window).scrollTop(0);
    }
    
    $(window).bind( 'hashchange', function(e) { renderFrontend(window.location.hash); });
    renderFrontend(window.location.hash);
    
    utility routes.js (A map of the hash route to the Microfrontend class)
    const routes = {
      '#': Home,
      '': Home,
      '#home': Home,
      '#cart': Cart,
      '#checkout': Checkout,
      '#product': Product,
      '#contact': Contact,
    };

    As you can see, this approach is pretty neat and encapsulates a separate class for Micro Frontends. All other Micro Frontends extend from this. Notice how all the functionality related to Microapp is encapsulated in the respective Micro Frontend. This makes sure that concurrent work on a Micro Frontend doesn’t mess up some other Micro Frontends.

    Everything will work in a similar paradigm when it comes to Web Components and Custom Elements.

    React

    With the client-side JavaScript frameworks being very popular, it is impossible to leave React from any Front End discussion. React being a component-based JS library, much of the things discussed above will also apply to React. I am going to discuss some of the technicalities and challenges when it comes to Micro Frontends with React.

    Styling

    Since there should be minimum sharing of code between any Micro Frontend, styling the React components can be challenging, considering the global and cascading nature of CSS. We should make sure styles are targeted on a specific Micro Frontend without spilling over to other Micro Frontends. Inline CSS, CSS in JS libraries like Radium,  and CSS Modules, can be used with React.

    Redux

    Using React with Redux is kind of a norm in today’s front-end world. The convention is to use Redux as a single global store for the entire app for cross application communication. A Micro Frontend should be self-contained with no dependencies. Hence each Micro Frontend should have its own Redux store, moving towards a multiple Redux store architecture. 

    Other Noteworthy Integration Approaches  –

    Server-side Rendering – One can use a server to assemble Micro Frontend templates before dispatching it to the browser. SSI techniques can be used too.

    iframes – Each Micro Frontend can be an iframe. They also offer a good degree of isolation in terms of styling, and global variables don’t interfere with each other.

    Summary

    With Microservices, Micro Frontends promise to  bring in a lot of benefits when it comes to structuring a complex application and simplifying its development, deployment and maintenance.

    But there is a wonderful saying that goes “there is no one-size-fits-all approach that anyone can offer you. The same hot water that softens a carrot hardens an egg”. Micro Frontend is no silver bullet for your architectural problems and comes with its own share of downsides. With more repositories, more tools, more build/deploy pipelines, more servers, more domains to maintain, Micro Frontends can increase the complexity of an app. It may render cross-application communication difficult to establish. It can also lead to duplication of dependencies and an increase in application size.

    Your decision to implement this architecture will depend on many factors like the size of your organization and the complexity of your application. Whether it is a new or legacy codebase, it is advisable to apply the technique gradually over time and review its efficacy over time.

  • Introduction to the Modern Server-side Stack – Golang, Protobuf, and gRPC

    There are some new players in town for server programming and this time it’s all about Google. Golang has rapidly been gaining popularity ever since Google started using it for their own production systems. And since the inception of Microservice Architecture, people have been focusing on modern data communication solutions like gRPC along with Protobuf. In this post, I will walk you through each of these briefly.

    Golang

    Golang or Go is an open source, general purpose programming language by Google. It has been gaining popularity recently for all the good reasons. It may come as a surprise to most people that language is almost 10 years old and has been production ready for almost 7 years, according to Google.

    Golang is designed to be simple, modern, easy to understand, and quick to grasp. The creators of the language designed it in such a way that an average programmer can have a working knowledge of the language over a weekend. I can attest to the fact that they definitely succeeded. Speaking of the creators, these are the experts that have been involved in the original draft of the C language so we can be assured that these guys know what they are doing.

    That’s all good but why do we need another language?

    For most of the use cases, we actually don’t. In fact, Go doesn’t solve any new problems that haven’t been solved by some other language/tool before. But it does try to solve a specific set of relevant problems that people generally face in an efficient, elegant, and intuitive manner. Go’s primary focus is the following:

    • First class support for concurrency
    • An elegant, modern language that is very simple to its core
    • Very good performance
    • First hand support for the tools required for modern software development

    I’m going to briefly explain how Go provides all of the above. You can read more about the language and its features in detail from Go’s official website.

    Concurrency

    Concurrency is one of the primary concerns in most of the server applications and it should be the primary concern of the language, considering the modern microprocessors. Go introduces a concept called a ‘goroutine’. A ‘goroutine’ is analogous to a ‘lightweight user-space thread’. It is much more complicated than that in reality as several goroutines multiplex on a single thread but the above expression should give you a general idea. These are light enough that you can actually spin up a million goroutines simultaneously as they start with a very tiny stack. In fact, that’s recommended. Any function/method in Go can be used to spawn a Goroutine. You can just do ‘go myAsyncTask()’ to spawn a goroutine from ‘myAsyncTask’ function. The following is an example:

    // This function performs the given task concurrently by spawing a goroutine
    // for each of those tasks.
    
    func performAsyncTasks(task []Task) {
      for _, task := range tasks {
        // This will spawn a separate goroutine to carry out this task.
        // This call is non-blocking
        go task.Execute()
      }
    }

    Yes, it’s that easy and it is meant to be that way as Go is a simple language and you are expected to spawn a goroutine for every independent async task without caring much. Go’s runtime automatically takes care of running the goroutines in parallel if multiple cores are available. But how do these goroutines communicate? The answer is channels.

    ‘Channel’ is also a language primitive that is meant to be used for communication among goroutines. You can pass anything from a channel to another goroutine (A primitive Go type or a Go struct or even other channels). A channel is essentially a blocking double ended queue (can be single ended too). If you want a goroutine(s) to wait for a certain condition to be met before continuing further you can implement cooperative blocking of goroutines with the help of channels.

    These two primitives give a lot of flexibility and simplicity in writing asynchronous or parallel code. Other helper libraries like a goroutine pool can be easily created from the above primitives. One basic example is:

    package executor
    
    import (
    	"log"
    	"sync/atomic"
    )
    
    // The Executor struct is the main executor for tasks.
    // 'maxWorkers' represents the maximum number of simultaneous goroutines.
    // 'ActiveWorkers' tells the number of active goroutines spawned by the Executor at given time.
    // 'Tasks' is the channel on which the Executor receives the tasks.
    // 'Reports' is channel on which the Executor publishes the every tasks reports.
    // 'signals' is channel that can be used to control the executor. Right now, only the termination
    // signal is supported which is essentially is sending '1' on this channel by the client.
    type Executor struct {
    	maxWorkers    int64
    	ActiveWorkers int64
    
    	Tasks   chan Task
    	Reports chan Report
    	signals chan int
    }
    
    // NewExecutor creates a new Executor.
    // 'maxWorkers' tells the maximum number of simultaneous goroutines.
    // 'signals' channel can be used to control the Executor.
    func NewExecutor(maxWorkers int, signals chan int) *Executor {
    	chanSize := 1000
    
    	if maxWorkers > chanSize {
    		chanSize = maxWorkers
    	}
    
    	executor := Executor{
    		maxWorkers: int64(maxWorkers),
    		Tasks:      make(chan Task, chanSize),
    		Reports:    make(chan Report, chanSize),
    		signals:    signals,
    	}
    
    	go executor.launch()
    
    	return &executor
    }
    
    // launch starts the main loop for polling on the all the relevant channels and handling differents
    // messages.
    func (executor *Executor) launch() int {
    	reports := make(chan Report, executor.maxWorkers)
    
    	for {
    		select {
    		case signal := <-executor.signals:
    			if executor.handleSignals(signal) == 0 {
    				return 0
    			}
    
    		case r := <-reports:
    			executor.addReport(r)
    
    		default:
    			if executor.ActiveWorkers < executor.maxWorkers && len(executor.Tasks) > 0 {
    				task := <-executor.Tasks
    				atomic.AddInt64(&executor.ActiveWorkers, 1)
    				go executor.launchWorker(task, reports)
    			}
    		}
    	}
    }
    
    // handleSignals is called whenever anything is received on the 'signals' channel.
    // It performs the relevant task according to the received signal(request) and then responds either
    // with 0 or 1 indicating whether the request was respected(0) or rejected(1).
    func (executor *Executor) handleSignals(signal int) int {
    	if signal == 1 {
    		log.Println("Received termination request...")
    
    		if executor.Inactive() {
    			log.Println("No active workers, exiting...")
    			executor.signals <- 0
    			return 0
    		}
    
    		executor.signals <- 1
    		log.Println("Some tasks are still active...")
    	}
    
    	return 1
    }
    
    // launchWorker is called whenever a new Task is received and Executor can spawn more workers to spawn
    // a new Worker.
    // Each worker is launched on a new goroutine. It performs the given task and publishes the report on
    // the Executor's internal reports channel.
    func (executor *Executor) launchWorker(task Task, reports chan<- Report) {
    	report := task.Execute()
    
    	if len(reports) < cap(reports) {
    		reports <- report
    	} else {
    		log.Println("Executor's report channel is full...")
    	}
    
    	atomic.AddInt64(&executor.ActiveWorkers, -1)
    }
    
    // AddTask is used to submit a new task to the Executor is a non-blocking way. The Client can submit
    // a new task using the Executor's tasks channel directly but that will block if the tasks channel is
    // full.
    // It should be considered that this method doesn't add the given task if the tasks channel is full
    // and it is up to client to try again later.
    func (executor *Executor) AddTask(task Task) bool {
    	if len(executor.Tasks) == cap(executor.Tasks) {
    		return false
    	}
    
    	executor.Tasks <- task
    	return true
    }
    
    // addReport is used by the Executor to publish the reports in a non-blocking way. It client is not
    // reading the reports channel or is slower that the Executor publishing the reports, the Executor's
    // reports channel is going to get full. In that case this method will not block and that report will
    // not be added.
    func (executor *Executor) addReport(report Report) bool {
    	if len(executor.Reports) == cap(executor.Reports) {
    		return false
    	}
    
    	executor.Reports <- report
    	return true
    }
    
    // Inactive checks if the Executor is idle. This happens when there are no pending tasks, active
    // workers and reports to publish.
    func (executor *Executor) Inactive() bool {
    	return executor.ActiveWorkers == 0 && len(executor.Tasks) == 0 && len(executor.Reports) == 0
    }

    Simple Language

    Unlike a lot of other modern languages, Golang doesn’t have a lot of features. In fact, a compelling case can be made for the language being too restrictive in its feature set and that’s intended. It is not designed around a programming paradigm like Java or designed to support multiple programming paradigms like Python. It’s just bare bones structural programming. Just the essential features thrown into the language and not a single thing more.

    After looking at the language, you may feel that the language doesn’t follow any particular philosophy or direction and it feels like every feature is included in here to solve a specific problem and nothing more than that. For example, it has methods and interfaces but not classes; the compiler produces a statically linked binary but still has a garbage collector; it has strict static typing but doesn’t support generics. The language does have a thin runtime but doesn’t support exceptions.

    The main idea here that the developer should spend the least amount of time expressing his/her idea or algorithm as code without thinking about “What’s the best way to do this in x language?” and it should be easy to understand for others. It’s still not perfect, it does feel limiting from time to time and some of the essential features like Generics and Exceptions are being considered for the ‘Go 2’.

    Performance

    Single threaded execution performance NOT a good metric to judge a language, especially when the language is focused around concurrency and parallelism. But still, Golang sports impressive benchmark numbers only beaten by hardcore system programming languages like C, C++, Rust, etc. and it is still improving. The performance is actually very impressive considering its a Garbage collected language and is good enough for almost every use case.

    (Image Source: Medium)

    Developer Tooling

    The adoption of a new tool/language directly depends on its developer experience. And the adoption of Go does speak for its tooling. Here we can see that same ideas and tooling is very minimal but sufficient. It’s all achieved by the ‘go’ command and its subcommands. It’s all command line.

    There is no package manager for the language like pip, npm. But you can get any community package by just doing

    go get github.com/velotiotech/WebCrawler/blob/master/executor/executor.go

    CODE: https://gist.github.com/velotiotech/3977b7932b96564ac9a041029d760d6d.js

    Yes, it works. You can just pull packages directly from github or anywhere else. They are just source files.

    But what about package.json..? I don’t see any equivalent for `go get`. Because there isn’t. You don’t need to specify all your dependency in a single file. You can directly use:

    import "github.com/xlab/pocketsphinx-go/sphinx"

    In your source file itself and when you do `go build` it will automatically `go get` it for you. You can see the full source file here:

    package main
    
    import (
    	"encoding/binary"
    	"bytes"
    	"log"
    	"os/exec"
    
    	"github.com/xlab/pocketsphinx-go/sphinx"
    	pulse "github.com/mesilliac/pulse-simple" // pulse-simple
    )
    
    var buffSize int
    
    func readInt16(buf []byte) (val int16) {
    	binary.Read(bytes.NewBuffer(buf), binary.LittleEndian, &val)
    	return
    }
    
    func createStream() *pulse.Stream {
    	ss := pulse.SampleSpec{pulse.SAMPLE_S16LE, 16000, 1}
    	buffSize = int(ss.UsecToBytes(1 * 1000000))
    	stream, err := pulse.Capture("pulse-simple test", "capture test", &ss)
    	if err != nil {
    		log.Panicln(err)
    	}
    	return stream
    }
    
    func listen(decoder *sphinx.Decoder) {
    	stream := createStream()
    	defer stream.Free()
    	defer decoder.Destroy()
    	buf := make([]byte, buffSize)
    	var bits []int16
    
    	log.Println("Listening...")
    
    	for {
    		_, err := stream.Read(buf)
    		if err != nil {
    			log.Panicln(err)
    		}
    
    		for i := 0; i < buffSize; i += 2 {
    			bits = append(bits, readInt16(buf[i:i+2]))
    		}
    
    		process(decoder, bits)
    		bits = nil
    	}
    }
    
    func process(dec *sphinx.Decoder, bits []int16) {
    	if !dec.StartUtt() {
    		panic("Decoder failed to start Utt")
    	}
    	
    	dec.ProcessRaw(bits, false, false)
    	dec.EndUtt()
    	hyp, score := dec.Hypothesis()
    	
    	if score > -2500 {
    		log.Println("Predicted:", hyp, score)
    		handleAction(hyp)
    	}
    }
    
    func executeCommand(commands ...string) {
    	cmd := exec.Command(commands[0], commands[1:]...)
    	cmd.Run()
    }
    
    func handleAction(hyp string) {
    	switch hyp {
    		case "SLEEP":
    		executeCommand("loginctl", "lock-session")
    		
    		case "WAKE UP":
    		executeCommand("loginctl", "unlock-session")
    
    		case "POWEROFF":
    		executeCommand("poweroff")
    	}
    }
    
    func main() {
    	cfg := sphinx.NewConfig(
    		sphinx.HMMDirOption("/usr/local/share/pocketsphinx/model/en-us/en-us"),
    		sphinx.DictFileOption("6129.dic"),
    		sphinx.LMFileOption("6129.lm"),
    		sphinx.LogFileOption("commander.log"),
    	)
    	
    	dec, err := sphinx.NewDecoder(cfg)
    	if err != nil {
    		panic(err)
    	}
    
    	listen(dec)
    }

    This binds the dependency declaration with source itself.

    As you can see by now, it’s simple, minimal and yet sufficient and elegant. There is first hand support for both unit tests and benchmarks with flame charts too. Just like the feature set, it also has its downsides. For example, `go get` doesn’t support versions and you are locked to the import URL passed in you source file. It is evolving and other tools have come up for dependency management.

    Golang was originally designed to solve the problems that Google had with their massive code bases and the imperative need to code efficient concurrent apps. It makes coding applications/libraries that utilize the multicore nature of modern microchips very easy. And, it never gets into a developer’s way. It’s a simple modern language and it never tries to become anything more that that.

    Protobuf (Protocol Buffers)

    Protobuf or Protocol Buffers is a binary communication format by Google. It is used to serialize structured data. A communication format? Kind of like JSON? Yes. It’s more than 10 years old and Google has been using it for a while now.

    But don’t we have JSON and it’s so ubiquitous…

    Just like Golang, Protobufs doesn’t really solve anything new. It just solves existing problems more efficiently and in a modern way. Unlike Golang, they are not necessarily more elegant than the existing solutions. Here are the focus points of protobuf:

    • It’s a binary format, unlike JSON and XML, which are text based and hence it’s vastly space efficient.
    • First hand and sophisticated support for schemas.
    • First hand support for generating parsing and consumer code in various languages.

    Binary format and speed

    So are protobuf really that fast? The short answer is, yes. According to the Google Developers they are 3 to 10 times smaller and 20 to 100 times faster than XML. It’s not a surprise as it is a binary format, the serialized data is not human readable.

    (Image Source: Beating JSON performance with Protobuf)

    Protobufs take a more planned approach. You define `.proto` files which are kind of the schema files but are much more powerful. You essentially define how you want your messages to be structured, which fields are optional or required, their data types etc. After that the protobuf compiler will generate the data access classes for you. You can use these classes in your business logic to facilitate communication.

    Looking at a `.proto` file related to a service will also give you a very clear idea of the specifics of the communication and the features that are exposed. A typical .proto file looks like this:

    message Person {
      required string name = 1;
      required int32 id = 2;
      optional string email = 3;
    
      enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }
    
      message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = HOME];
      }
    
      repeated PhoneNumber phone = 4;
    }

    Fun Fact: Jon Skeet, the king of Stack Overflow is one of the main contributors in the project.

    gRPC

    gRPC, as you guessed it, is a modern RPC (Remote Procedure Call) framework. It is a batteries included framework with built in support for load balancing, tracing, health checking, and authentication. It was open sourced by Google in 2015 and it’s been gaining popularity ever since.

    An RPC framework…? What about REST…?

    SOAP with WSDL has been used long time for communication between different systems in a Service Oriented Architecture. At the time, the contracts used to be strictly defined and systems were big and monolithic, exposing a large number of such interfaces.

    Then came the concept of ‘browsing’ where the server and client don’t need to be tightly coupled. A client should be able to browse service offerings even if they were coded independently. If the client demanded the information about a book, the service along with what’s requested may also offer a list of related books so that client can browse. REST paradigm was essential to this as it allows the server and client to communicate freely without strict restriction using some primitive verbs.

    As you can see above, the service is behaving like a monolithic system, which along with what is required is also doing n number of other things to provide the client with the intended `browsing` experience. But this is not always the use case. Is it?

    Enter the Microservices

    There are many reasons to adopt for a Microservice Architecture. The prominent one being the fact that it is very hard to scale a Monolithic system. While designing a big system with Microservices Architecture each business or technical requirement is intended to be carried out as a cooperative composition of several primitive ‘micro’ services.

    These services don’t need to be comprehensive in their responses. They should perform specific duties with expected responses. Ideally, they should behave like pure functions for seamless composability.

    Now using REST as a communication paradigm for such services doesn’t provide us with much of a benefit. However, exposing a REST API for a service does enable a lot of expression capability for that service but again if such expression power is neither required nor intended we can use a paradigm that focuses more on other factors.

    gRPC intends to improve upon the following technical aspects over traditional HTTP requests:

    • HTTP/2 by default with all its goodies.
    • Protobuf as machines are talking.
    • Dedicated support for streaming calls thanks to HTTP/2.
    • Pluggable auth, tracing, load balancing and health checking because you always need these.

    As it’s an RPC framework, we again have concepts like Service Definition and Interface Description Language which may feel alien to the people who were not there before REST but this time it feels a lot less clumsy as gRPC uses Protobuf for both of these.

    Protobuf is designed in such a way that it can be used as a communication format as well as a protocol specification tool without introducing anything new. A typical gRPC service definition looks like this:

    service HelloService {
      rpc SayHello (HelloRequest) returns (HelloResponse);
    }
    
    message HelloRequest {
      string greeting = 1;
    }
    
    message HelloResponse {
      string reply = 1;
    }

    You just write a `.proto` file for your service describing the interface name, what it expects, and what it returns as Protobuf messages. Protobuf compiler will then generate both the client and server side code. Clients can call this directly and server-side can implement these APIs to fill in the business logic.

    Conclusion

    Golang, along with gRPC using Protobuf is an emerging stack for modern server programming. Golang simplifies making concurrent/parallel applications and gRPC with Protobuf enables efficient communication with a pleasing developer experience.

  • Cleaner, Efficient Code with Hooks and Functional Programming

    React Hooks were introduced in 2018 and ever since numerous POCs have been built around the same. Hooks come in at a time when React has become a norm and class components are becoming increasingly complex. With this blog, I will showcase how Hooks can reduce the size of your code up to 90%. Yes, you heard it right. Exciting, isn’t it? 

    Hooks are a powerful upgrade coming with React 16.8 and utilize the functional programming paradigm. React, however, also acknowledges the volume of class components already built, and therefore, comes with backward compatibility. You can practice by refactoring a small chunk of your codebase to use React Hooks, while not impacting the existing functionality. 

    With this article, I tried to show you how Hooks can help you write cleaner, smaller and more efficient code. 90% Remember!

    First, let’s list out the common problems we all face with React Components as they are today:

    1. Huge Components – caused by the distributed logic in lifecycle Hooks

    2. Wrapper Hell – caused by re-using components

    3. Confusing and hard to understand classes

    In my opinion, these are the symptoms of one big problem i.e. React does not provide a stateful primitive simpler, smaller and more lightweight than class component. That is why solving one problem worsens the other. For example, if we put all of the logic in components to fix Wrapper Hell, it leads to Huge Components, that makes it hard to refactor. On the other hand, if we divide the huge components into smaller reusable pieces, it leads to more nests than in the component tree i.e. Wrapper Hell. In either case, there’s always confusion around the classes.

    Let’s approach these problems one by one and solve them in isolation.

    Huge Components –

    We all have used lifecycle Hooks and often with time they contain more and more stateful logic. It is also observed that stateful logic is shared amongst lifecycle Hooks. For example, consider you have a code that adds an event listener in componentDidMount. The componentDidUpdate method might also contain some logic for setting up the event listeners. Now the cleanup code will be written in componentWillUnmount. See how the logic for the same thing is split between these lifecycle Hooks.

    // Class component
    
    import React from "react";
    
    export default class LazyLoader extends React.Component {
      constructor(props) {
        super(props);
    
        this.state = { data: [] };
      }
    
      loadMore = () => {
        // Load More Data
        console.log("loading data");
      };
    
      handleScroll = () => {
        if (!this.props.isLoading && this.props.isCompleted) {
          this.loadMore();
        }
      };
    
      componentDidMount() {
        this.loadMore();
        document.addEventListener("scroll", this.handleScroll, false);
        // more subscribers and event listeners
      }
    
      componentDidUpdate() {
        //
      }
    
      componentWillUnmount() {
        document.removeEventListener("scroll", this.handleScroll, false);
        // unsubscribe and remove listeners
      }
    
      render() {
        return <div>{this.state.data}</div>;
      }
    }

    React Hooks approach this with useEffect.

    import React, { useEffect, useState } from "react";
    
    export const LazyLoader = ({ isLoading, isCompleted }) => {
      const [data, setData] = useState([]);
    
      const loadMore = () => {
        // Load and setData here
      };
    
      const handleScroll = () => {
        if (!isLoading && isCompleted) {
          loadMore();
        }
      };
    
      // cDM and cWU
      useEffect(() => {
        document.addEventListener("scroll", handleScroll, false);
        // more subscribers and event listeners
    
        return () => {
          document.removeEventListener("scroll", handleScroll, false);
          // unsubscribe and remove listeners
        };
      }, []);
    
      // cDU
      useEffect(() => {
        //
      }, [/** dependencies */]);
    
      return data && <div>{data}</div>;
    };

    Now, let’s move the logic to a custom Hook.

    import { useEffect, useState } from "react";
    
    export function useScroll() {
      const [data, setData] = useState([]);
    
      const loadMore = () => {
        // Load and setData here
      };
    
      const handleScroll = () => {
        if (!isLoading && isCompleted) {
          loadMore();
        }
      };
    
      // cDM and cWU
      useEffect(() => {
        document.addEventListener("scroll", handleScroll, false);
        // more subscribers and event listeners
    
        return () => {
          document.removeEventListener("scroll", handleScroll, false);
          // unsubscribe and remove listeners
        };
      }, []);
    
      return data;
    };

    import React, { useEffect } from "react";
    import { useScroll } from "./useScroll";
    
    const LazyLoader = ({ isLoading, isCompleted }) => {
      const data = useScroll();
    
      // cDU
      useEffect(() => {
        //
      }, [/** dependencies */]);
    
      return data && <div>{data}</div>;
    };

    useEffect puts the code that changes together in one place, making the code more readable and easy to understand. You can also write multiple useEffects. The advantage of this is again to separate out the mutually unrelated code.

    Wrapper Hell –

    If you’re well versed with React, you probably know it doesn’t provide a pattern of attaching a reusable code to the component (like “connect” in react-redux). React solves this problem of data sharing by render props and higher-order components patterns. But using this, requires restructuring of your components, that is hard to follow and, at times, cumbersome. This typically leads to a problem called Wrapper Hell. One can check this by looking at the application in React DevTools. There you can see components wrapped by a number of providers, consumers, HOCs and other abstractions. Because of this, React needed a better way of sharing the logic.

    The below code is inspired from React Conf 2018 – 90% cleaner react w/ Hooks.

    import React from "react";
    import Media from "./components/Media";
    
    function App() {
      return (
        <Media query="(max-width: 480px)">
          {small => (
            <Media query="(min-width: 1024px)">
              {large => (
                <div className="media">
                  <h1>Media</h1>
                  <p>{small ? "small screen" : "not a small screen"}</p>
                  <p>{large ? "large screen" : "not a large screen"}</p>
                </div>
              )}
            </Media>
          )}
        </Media>
      );
    }
    
    export default App;

    import React from "react";
    
    export default class Media extends React.Component {
      removeListener = () => null;
    
      constructor(props) {
        super(props);
        this.state = {
          matches: window.matchMedia(this.props.query).matches
        };
      }
    
      componentDidMount() {
        this.init();
      }
    
      init() {
        const media = window.matchMedia(this.props.query);
        if (media.matches !== this.state.matches) {
          this.setState({ matches: media.matches });
        }
    
        const listener = () => this.setState({ matches: media.matches });
        media.addListener(listener);
        this.removeListener = () => media.removeListener(listener);
      }
    
      componentDidUpdate(prevProps) {
        if (prevProps.query !== this.props.query) {
          this.removeListener();
          this.init();
        }
      }
    
      componentWillUnmount() {
        this.removeListener();
      }
    
      render() {
        return this.props.children(this.state.matches);
      }
    }

    We can check the below example to see how Hooks fix this problem.

    import { useState, useEffect } from "react";
    
    export default function(query) {
      let [matches, setMatches] = useState(window.matchMedia(query).matches);
    
      useEffect(() => {
        let media = window.matchMedia(query);
        if (media.matches !== matches) {
          setMatches(media.matches);
        }
        const listener = () => setMatches(media.matches);
        media.addListener(listener);
        return () => media.removeListener(listener);
      }, [query, matches]);
    
      return matches;
    }

    import React from "react";
    import useMedia from "./hooks/useMedia";
    
    function App() {
      let small = useMedia("(max-width: 480px)");
      let large = useMedia("(min-width: 1024px)");
      return (
        <div className="media">
          <h1>Media</h1>
          <p>{small ? "small screen" : "not a small screen"}</p>
          <p>{large ? "large screen" : "not a large screen"}</p>
        </div>
      );
    }
    
    export default App;

    Hooks provide you with a way to extract a reusable stateful logic from a component without affecting the component hierarchy. This enables it to be tested independently.

    Confusing and hard to understand classes

    Classes pose more problems than it solves. We’ve known React for a very long time and there’s no denying that it is hard for humans as well as for machines. It confuses both of them. Here’s why:

    For Humans –

    1. There’s a fair amount of boilerplate when defining a class.

    2. Beginners and even expert developers find it difficult to bind methods and writing class components.

    3. People often couldn’t decide between functional and class components, as with time they might need state.

    For Machines –

    1. In the minified version of a component file, the method names are not minified and the unused methods are not stripped out, as it’s not possible to tell how all the methods fit together.

    2. Classes make it difficult for React to implement hot loading reliably.

    3. Classes encourage patterns that make it difficult for the compiler to optimize.

    Due to the above problems, classes can be a large barrier in learning React. To keep the React relevant, the community has been experimenting with component folding and Prepack, but the classes make optimizations fall back to the slower path. Hence, the community wanted to present an API that makes it more likely for code to stay on the optimizable path.

    React components have always been closer to functions. And since Hooks introduced stateful logic into functional components, it lets you use more of React’s features without classes. Hooks embrace functions without compromising the practical spirit of React. Hooks don’t require you to learn complex functional and reactive programming techniques.

    Conclusion –

    React Hooks got me excited and I am learning new things every day. Hooks are a way to write far less code for the same usecase. Also, Hooks do not ask the developers who are already busy with shipping, to rewrite everything. You can redo small components with Hooks and slowly move to the complex components later.

    The thinking process in Hooks is meant to be gradual. I hope this blog makes you want to get your hands dirty with Hooks. Do share your thoughts and experiences with Hooks. Finally, I would strongly recommend this official documentation which has great content.

    Recommended Reading: React Today and Tomorrow and 90% cleaner React with Hook

  • Creating Faster and High Performing User Interfaces in Web Apps With Web Workers

    The data we render on a UI originates from different sources like databases, APIs, files, and more. In React applications, when the data is received, we first store it in state and then pass it to the other components in multiple ways for rendering.

    But most of the time, the format of the data is inconvenient for the rendering component. So, we have to format data and perform some prior calculations before we give it to the rendering component.

    Sending data directly to the rendering component and processing the data inside that component is not recommended. Not only data processing but also any heavy background jobs that we would have to depend on the backend can now be done on the client-side because React allows the holding of business logic on the front-end.

    A good practice is to create a separate function for processing that data which is isolated from the rendering logic, so that data processing and data representation will be done separately.

    Why? There are two possible reasons:

    – The processed data can be shared/used by other components, too.

    – The main reason to avoid this is: if the data processing is a time-consuming task, you will see some lag on the UI, or in the worst-case scenario, sometimes the page may become unresponsive.

    As JavaScript is a single-threaded environment, it has only one call stack to execute scripts (in a simple way, you cannot run more than one script at the same time).

    For example, suppose you have to do some DOM manipulations and, at the same time, want to do some complex calculations. You can not perform these two operations in parallel. If the JavaScript engine is busy computing the complex computation, then all the other tasks like event listeners and rendering callbacks will get blocked for that amount of time, and the page may become unresponsive.

    ‍How can you solve this problem?

    Though JavaScript is single-threaded, many developers mimic the concurrency with the help of timer functions and event handlers. Like by breaking heavy (time-consuming) tasks into tiny chunks and by using the timers you can split their execution. Let’s take a look at the following example.

    Here, the processDataArray function uses the timer function to split the execution, which internally uses the setTimeout method for processing some items of array, again after a dedicated time passed execute more items, once all the array elements have been processed, send the processed result back by using the finishCallback

    const processDataArray = (dataArray, finishCallback) => {
     // take a new copy of array
     const todo = dataArray.concat();
     // to store each processed data
     let result = [];
     // timer function
     const timedProcessing = () => {
       const start = +new Date();
       do {
         // process each data item and store it's result
         const singleResult = processSingleData(todo.shift());
         result.push(singleResult);
         // check if todo has something to process and the time difference must not be greater than 50
       } while (todo.length > 0 && +new Date() - start < 50);
     
       // check for remaining items to process
       if (todo.length > 0) {
         setTimeout(timedProcessing, 25);
       } else {
         // finished with all the items, initiate finish callback
         finishCallback(result);
       }
     };
     setTimeout(timedProcessing, 25);
    };
    
    
    
    const processSingleData = data => {
     // process data
     return processedData;
    };

    You can find more about how JavaScript timers work internally here.

    The problem is not solved yet, and the main thread is still busy in the computation so you can see the delay in the UI events like button clicks or mouse scroll. This is a bad user experience when you have a big array computation going on and an impatient web user.

    The better and real multithreading way to solve this problem and to run multiple scripts in parallel is by using Web Workers.

    What are Web Workers?‍

    Web Workers provide a mechanism to spawn a separate script in the background. Where you can do any type of calculations without disturbing the UI. Web Workers run outside the context of the HTML document’s script, making it easiest to allow concurrent execution of JavaScript programs. You can experience multithreading behavior while using Web Workers.

    Communication between the page (main thread) and the worker happens using a simple mechanism. They can send messages to each other using the postMessage method, and they can receive the messages using onmessage callback function. Let’s take a look at a simple example:

    In this example, we will delegate the work of multiplying all the numbers in an array to a Web Worker, and the Web Worker returns the result back to the main thread.

    import "./App.css";
    import { useEffect, useState } from "react";
     
    function App() {
     // This will load and execute the worker.js script in the background.
     const [webworker] = useState(new window.Worker("worker.js"));
     const [result, setResult] = useState("Calculating....");
     
     useEffect(() => {
       const message = { multiply: { array: new Array(1000).fill(2) } };
       webworker.postMessage(message);
    
    
    
       webworker.onerror = () => {
         setResult("Error");
       };
     
       webworker.onmessage = (e) => {
         if (e.data) {
           setResult(e.data.result);
         } else {
           setResult("Error");
         }
       };
     }, []);
    
    useEffect(() => {
       return () => {
         webworker.terminate();
       };
     }, []);
    
     return (
       <div className="App">
         <h1>Webworker Example In React</h1>
         <header className="App-header">
           <h1>Multiplication Of large array</h1>
           <h2>Result: {result}</h2>
         </header>
       </div>
     );
    }
     
    export default App;

    onmessage = (e) => {
     const { multiply } = e.data;
     // check data is correctly framed
     if (multiply && multiply.array.length) {
       // intentionally delay the execution
       setTimeout(() => {
         // this post back the result to the page
         postMessage({
           result: multiply.array.reduce(
             (firstItem, secondItem) => firstItem * secondItem
           ),
         });
       }, 2000);
     } else {
       postMessage({ result: 0 });
     }
    };

    If the worker script throws an exception, you can handle it by attaching a callback function to the onerror property of the worker in the App.js script.

    From the main thread, you can terminate the worker immediately if you want by using the worker’s terminate method. Once the worker is terminated, the worker variable becomes undefined. You need to create another instance if needed.

    You can find a working example here.

    Use cases of Web Workers:

    Charting middleware – Suppose you have to design a dashboard that represents the analytics of businesses engagement for a business retention application by means of a pivot table, pie charts, and bar charts. It involves heavy processing of data to convert it to the expected format of a table, pie chart, a bar chart. This may result in the UI failing to update, freezing, or the page becoming unresponsive because of single-threaded behavior. Here, we can use Web Workers and delegate the processing logic to it. So that the main thread is always available to handle other UI events.

    Emulating excel functionality – For example, if you have thousands of rows in the spreadsheet and each one of them needs some calculations (longer), you can write custom functions containing the processing logic and put them in the WebWorker’s script.

    Real-time text analyzer – This is another good example where we can use WebWorker to show the word count, characters count, repeated word count, etc., by analyzing the text typed by the user in real-time. With a traditional implementation, you may experience performance issues as the text size grows, but this can be optimized by using WebWorkers.

    Web Worker limitations:

    Yes, Web Workers are amazing and quite simple to use, but as the WebWorker is a separate thread, it does not have access to the window object, document object, and parent object. And we can not pass functions through postmessage.

    But Web Workers have access to:

    – Navigator object

    – Location object (read-only)

    – XMLHttpRequest

    – setTimeout, setInterval, clearTimeout, clearInterval

    – You can import other scripts in WebWorker using the importScripts() method

    Here are some other types of workers:
    Shared Worker
    Service Worker
    Audio Worklet

    Conclusion:

    Web Workers make our life easier by doing jobs in parallel in the background, but Web Workers are relatively heavy-weight and come with high startup performance cost and a high per-instance memory cost, so as per the WHATWG community, they are not intended to be used in large numbers.

  • Monitoring a Docker Container with Elasticsearch, Kibana, and Metricbeat

    Since you are on this page, you have probably already started using Docker to deploy your applications and are enjoying it compared to virtual machines, because of it being lightweight, easy to deploy and its exceptional security management features.

    And, once the applications are deployed, monitoring your containers and tracking their activities in real time is very essential. Imagine a scenario where you are managing one or many virtual machines. Your pre-configured session will be doing everything, including monitoring. If you face any problems during production, then—with a handful of commands such as top, htop, iotop, and with flags like -o, %CPU, and %MEM—you are good to troubleshoot the issue.

    On the other hand, consider a scenario where you have the same nodes spread across 100-200 containers. You will need to see all activity in one place to query for information about what happened. Here, monitoring comes into the picture. We will be discussing more benefits as we move further.

    This blog will cover Docker monitoring with Elasticsearch, Kibana, and Metricbeat. Basically, Elasticsearch is a platform that allows us to have distributed search and analysis of data in real-time along with visualization. We’ll be discussing how all these work interdependently as we move ahead. Like Elasticsearch, Kibana is also open-source software. Kibana is an interface mainly used to visualize the data sent from Elasticsearch. Metricbeat is a lightweight shipper of collected metrics from your system to the desired target (Elasticsearch in this case). 

    What is Docker Monitoring?

    In simple terms, monitoring containers is how we keep track of the above metrics and analyze them to ensure the performance of applications built on microservices and to keep track of issues so that they can be solved more easily. This monitoring is vital for performance improvement and optimization and to find the RCA of various issues.

    There is a lot of software available for monitoring the Docker container, both open-source as well as proprietary, like Prometheus, AppOptics, Metricbeats, Datadog, Sumologic, etc.

    You can choose any of these based on convenience. 

    Why is Docker Monitoring needed?

    1. Monitoring helps early detection and to fix issues to avoid a breakdown during production
    2. New feature additions/updates implemented safely as the entire application is monitored
    3. Docker monitoring is beneficial for developers, IT pros, and enterprises as well.
    • For developers, Docker monitoring tracks bugs and helps to resolve them quickly along with enhancing security.
    • For IT pros, it helps with flexible integration of existing processes and enterprise systems and satisfies all the requirements.
    • For enterprises, it helps to build the application within a certified container within a secured ecosystem that runs smoothly. 

    Elasticsearch is a platform that allows us to have distributed search and analysis of data in real-time, along with visualization. Elasticsearch is free and open-source software. It goes well with a huge number of technologies, like Metricbeat, Kibana, etc. Let’s move onto the installation of Elasticsearch.

    Installation of Elasticsearch:

    Prerequisite: Elasticsearch is built in Java. So, make sure that your system at least has Java8 to run Elasticsearch.

    For installing Elasticsearch for your OS, please follow the steps at Installing Elasticsearch | Elasticsearch Reference [7.11].

    After installing,  check the status of Elasticsearch by sending an HTTP request on port 9200 on localhost.

    http://localhost:9200/

    This will give you a response as below:

    You can configure Elasticsearch by editing $ES_HOME/config/elasticsearch.yml 

    Learn more about configuring Elasticsearch here.

    Now, we are done with the Elasticsearch setup and are ready to move onto Kibana.

    Kibana:

    Like Elasticsearch, Kibana is also open-source software. Kibana is an interface mainly used to visualize the data from Elasticsearch. Kibana allows you to do anything via query and let’s you generate numerous visuals as per your requirements. Kibana lets you visualize enormous amounts of data in terms of line graphs, gauges, and all other graphs.

    Let’s cover the installation steps of Kibana.

    Installing Kibana

    Prerequisites: 

    • Must have Java1.8+ installed 
    • Elasticsearch v1.4.4+
    • Web browser such as Chrome, Firefox

    For installing Kibana with respect to your OS, please follow the steps at Install Kibana | Kibana Guide [7.11]

    Kibana runs on default port number 5601. Just send an HTTP request to port 5601 on localhost with http://localhost:5601/ 

    You should land on the Kibana dashboard, and it is now ready to use:

    You can configure Kibana by editing $KIBANA_HOME/config. For more about configuring Kibana, visit here.

    Let’s move onto the final part—setting up with Metricbeat.

    Metricbeat

    Metricbeat sends metrics frequently, and we can say it’s a lightweight shipper of collected metrics from your system.

    You can simply install Metricbeat to your system or servers to periodically collect metrics from the OS and the microservices running on services. The collected metrics are shipped to the output you specified, e.g., Elasticsearch, Logstash. 

    Installing Metricbeat

    For installing Metricbeat according to your OS, follow the steps at Install Kibana | Kibana Guide [7.11]

    As soon as we start the Metricbeat service, it sends Docker metrics to the Elasticsearch index, which can be confirmed by curling Elasticsearch indexes with the command:

    curl -XGET 'localhost:9200/_cat/indices?v&pretty'

    How Are They Internally Connected?

    We have now installed all three and they are up and running. As per the period mentioned, docker.yml will hit the Docker API and send the Docker metrics to Elasticsearch. Those metrics are now available in different indexes of Elasticsearch. As mentioned earlier, Kibana queries the data of Elasticsearch and visualizes it in the form of graphs. In this, all three are connected. 

    Please refer to the flow chart for more clarification:

    How to Create Dashboards?

    Now that we are aware of how these three tools work interdependently, let’s create dashboards to monitor our containers and understand those. 

    First of all, open the Dashboards section on Kibana (localhost:5601/) and click the Create dashboard button:

     

    You will be directed to the next page:

    Choose the type of visualization you want from all options:

    For example, let’s go with Lens

    (Learn more about Kibana Lens)

    Here, we will be looking for the number of containers vs. timestamps by selecting the timestamp on X-axis and the unique count of docker.container.created on Y-axis.

    As soon we have selected both parameters, it will generate a graph as shown in the snapshot, and we will be getting the count of created containers (here Count=1). If you create move containers on your system, when that data metric is sent to Kibana, the graph and the counter will be modified. In this way, you can monitor how many containers are created over time. In similar fashion, depending on your monitoring needs, you can choose a parameter from the left panel showing available fields like: 

    activemq.broker.connections.count

    docker.container.status

    Docker.container.tags

    Now, we will show one more example of how to create a bar graph:

    As mentioned above, to create a bar graph just choose “vertical bar” from the above snapshot. Here, I’m trying to get a bar graph for the count of documents vs. metricset names, such as network, file system, cpu, etc. So, as shown in the snapshot on the left, choose the Y-axis parameter as count and X-axis parameter as metricset.name as shown in the right side of the snapshot

    After hitting enter, a graph will be generated: 

    Similarly, you can try it out with multiple parameters with different types of graphs to monitor. Now, we will move onto the most important and widely used monitoring tool to track warnings, errors, etc., which is DISCOVER.

    Discover for Monitoring:

    Basically, Discover provides deep insights into data, showing you where you can apply searches and filters as well. With it, you can show which processes are taking more time and show only those. Filter out errors occurring with the message filter with a value of ERROR. Check the health of the container; check for logged-in users. These kinds of queries can be sent and the desired results can be achieved, leading to good monitoring of containers, same as the SQL queries. 

    [More about Discover here.]

    To apply filters, just click on the “filter by type” from the left panel, and you will see all available filtering options. From there, you can select one as per your requirements, and view those on the central panel. 

    Similar to filter, you can choose fields to be shown on the dashboard from the left panel with “Selected fields” right below the filters. (Here, we have only selected info for Source.)

    Now, if you take a look at the top part of the snapshot, you will find the search bar. This is the most useful part of Discover for monitoring.

    In that bar, you just need to put a query, and according to that query, logs will be filtered. For example, I will be putting a query for error messages equal to No memory stats data available.

    When we hit the update button on the right side, only logs containing that error message will be there and highlighted for differentiation, as shown in the snapshot. All other logs will not be shown. In this way, you can track a particular error and ensure that it does not exist after fixing it.

    In addition to query, it also provides keyword search. So, if you input a word like warning, error, memory, or user, then it will provide logs for that word, like “memory” in the snapshot:

     

    Similar to Kibana, we also receive logs in the terminal. For example, the following highlighted portion is about the state of your cluster. In the terminal, you can put a simple grep command for required logs. 

    With this, you can monitor Docker containers with multiple queries, such as nested queries for the Discover facility. There are many different graphs you can try depending on your requirements to keep your application running smoothly.

    Conclusion

    Monitoring requires a lot of time and effort. What we have seen here is a drop in the ocean. For some next steps, try:

    1. Monitoring network
    2. Aggregating logs from your different applications
    3. Aggregating logs from multiple containers
    4. Alerts setting and monitoring
    5. Nested queries for logs