Almost all the applications that you work on or deal with throughout the day use SMS (short messaging service) as an efficient and effective way to communicate with end users.
Some very common use-cases include:
Receiving an OTP for authenticating your login
Getting deals from the likes of Flipkart and Amazon informing you regarding the latest sale.
Getting reminder notifications for the doctor’s appointment that you have
Getting details for your debit and credit transactions.
The practical use cases for an SMS can be far-reaching.
Even though SMS integration forms an integral part of any application, due to the limitations and complexities involved in automating it via web automation tools like selenium, these are often neglected to be automated.
Teams often opt for verifying these sets of test cases manually, which, even though is important in getting bugs earlier, it does pose some real-time challenges.
Pitfalls with Manual Testing
With these limitations, you obviously do not want your application sending faulty Text Messages after that major Release.
Automation Testing … #theSaviour
To overcome the limitations of manual testing, delegating your task to a machine comes in handy.
Now that we have talked about the WHY, we will look into HOW the feature can be automated. Technically, you shouldn’t / can’t use selenium to read the SMS via mobile. So, we were looking for a third-party library that is
Easy to integrate with the existing code base
Supports a range of languages
Does not involve highly complex codes and focuses on the problem at hand
Supports both incoming and outgoing messages
After a lot of research, we settled with Twilio.
In this article, we will look at an example of working with Twilio APIs to Read SMS and eventually using it to automate SMS flows.
Twilio supports a bunch of different languages. For this article, we stuck with Node.js
Account Setup
Registration
To start working with the service, you need to register.
Once that is done, Twilio will prompt you with a bunch of simple questions to understand why you want to use their service.
Twilio Dashboard
A trial balance of $15.50 is received upon signing up for your usage. This can be used for sending and receiving text messages. A unique Account SID and Auth Token is also generated for your account.
Buy a Number
Navigate to the buy a number link under Phone Numbers > Manage and purchase a number that you would eventually be using in your automation scripts for receiving text messages from the application.
Note – for the free trial, Twilio does not support Indian Number (+91)
Code Setup
Install Twilio in your code base
Code snippet
For simplification, Just pass in the accountSid and authToken that you will receive from the Dashboard Console to the twilio library.This would return you with a client object containing the list of all the messages in your inbox.
List Messages matching filter criteria: If you’d like to have Twilio narrow down this list of messages for you, you can do so by specifying a To number, From the number, and a DateSent.
Get a Message : If you know the message SID (i.e., the message’s unique identifier), then you can retrieve that specific message directly. Using this method, you can send emails without attachments.
The trial version does not support Indian numbers (+91).
The trial version just provides an initial balance of $15.50. This is sufficient enough for your use case that involves only receiving messages on your Twilio number. But if the use case requires sending back the message from the Twilio number, a paid version can solve the purpose.
Messages sent via a short code (557766) are not received on the Twilio number. Only long codes are accepted in the trial version.
You can buy only a single number with the trial version. If purchasing multiple numbers is required, the user may have to switch to a paid version.
Conclusion
In a nutshell, we saw how important it is to thoroughly verify the SMS functionality of our application since it serves as one of the primary ways of communicating with the end users. We also saw what the limitations are with following the traditional manual testing approach and how automating SMS scenarios would help us deliver high-quality products. Finally, we demonstrated a feasible, efficient and easy-to-use way to Automate SMS test scenarios using Twilio APIs.
Hope this was a useful read and that you will now be able to easily automate SMS scenarios. Happy testing… Do like and share …
All modern era programmers can attest that containerization has afforded more flexibility and allows us to build truly cloud-native applications. Containers provide portability – ability to easily move applications across environments. Although complex applications comprise of many (10s or 100s) containers. Managing such applications is a real challenge and that’s where container orchestration and scheduling platforms like Kubernetes, Mesosphere, Docker Swarm, etc. come into the picture. Kubernetes, backed by Google is leading the pack given that Redhat, Microsoft and now Amazon are putting their weight behind it.
Kubernetes can run on any cloud or bare metal infrastructure. Setting up & managing Kubernetes can be a challenge but Google provides an easy way to use Kubernetes through the Google Container Engine(GKE) service.
What is GKE?
Google Container Engine is a Management and orchestration system for Containers. In short, it is a hosted Kubernetes. The goal of GKE is to increase the productivity of DevOps and development teams by hiding the complexity of setting up the Kubernetes cluster, the overlay network, etc.
Why GKE? What are the things that GKE does for the user?
GKE abstracts away the complexity of managing a highly available Kubernetes cluster.
GKE also provides easy integration with the Google storage services.
In this blog, we will see how to create your own Kubernetes cluster in GKE and how to deploy a multi-tier application in it. The blog assumes you have a basic understanding of Kubernetes and have used it before. It also assumes you have created an account with Google Cloud Platform. If you are not familiar with Kubernetes, this guide from Deis is a good place to start.
Google provides a Command-line interface (gcloud) to interact with all Google Cloud Platform products and services. gcloud is a tool that provides the primary command-line interface to Google Cloud Platform. Gcloud tool can be used in the scripts to automate the tasks or directly from the command-line. Follow this guide to install the gcloud tool.
Now let’s begin! The first step is to create the cluster.
Basic Steps to create cluster
In this section, I would like to explain about how to create GKE cluster. We will use a command-line tool to setup the cluster.
Set the zone in which you want to deploy the cluster
Let’s try to understand what each of these parameters mean:
–project: Project Name
–machine-type: Type of the machine like n1-standard-2, n1-standard-4
–image-type: OS image.”COS” i.e. Container Optimized OS from Google: More Info here.
–disk-size: Disk size of each instance.
–num-nodes: Number of nodes in the cluster.
–network: Network that users want to use for the cluster. In this case, we are using default network.
Apart from the above options, you can also use the following to provide specific requirements while creating the cluster:
–scopes: Scopes enable containers to direct access any Google service without needs credentials. You can specify comma separated list of scope APIs. For example:
Compute: Lets you view and manage your Google Compute Engine resources
You can find all the Scopes that Google supports here: .
–additional-zones: Specify additional zones to high availability. Eg. –additional-zones us-east1-b, us-east1-d . Here Kubernetes will create a cluster in 3 zones (1 specified at the beginning and additional 2 here).
–enable-autoscaling : To enable the autoscaling option. If you specify this option then you have to specify the minimum and maximum required nodes as follows; You can read more about how auto-scaling works here. Eg: –enable-autoscaling –min-nodes=15 –max-nodes=50
You can fetch the credentials of the created cluster. This step is to update the credentials in the kubeconfig file, so that kubectl will point to required cluster.
After creating Cluster, now let’s see how to deploy a multi tier application on it. Let’s use simple Python Flask app which will greet the user, store employee data & get employee data.
Application Deployment
I have created simple Python Flask application to deploy on K8S cluster created using GKE. you can go through the source code here. If you check the source code then you will find directory structure as follows:
In this, I have written a Dockerfile for the Python Flask application in order to build our own image to deploy. For MySQL, we won’t build an image of our own. We will use the latest MySQL image from the public docker repository.
Before deploying the application, let’s re-visit some of the important Kubernetes terms:
Pods:
The pod is a Docker container or a group of Docker containers which are deployed together on the host machine. It acts as a single unit of deployment.
Deployments:
Deployment is an entity which manages the ReplicaSets and provides declarative updates to pods. It is recommended to use Deployments instead of directly using ReplicaSets. We can use deployment to create, remove and update ReplicaSets. Deployments have the ability to rollout and rollback the changes.
Services:
Service in K8S is an abstraction which will connect you to one or more pods. You can connect to pod using the pod’s IP Address but since pods come and go, their IP Addresses change. Services get their own IP & DNS and those remain for the entire lifetime of the service.
Each tier in an application is represented by a Deployment. A Deployment is described by the YAML file. We have two YAML files – one for MySQL and one for the Python application.
You will find a ‘kind’ field in each YAML file. It is used to specify whether the given configuration is for deployment, service, pod, etc.
In the Python app service YAML, I am using type = LoadBalancer. In GKE, There are two types of cloud load balancers available to expose the application to outside world.
TCP load balancer: This is a TCP Proxy-based load balancer. We will use this in our example.
HTTP(s) load balancer: It can be created using Ingress. For more information, refer to this post that talks about Ingress in detail.
In the MySQL service, I’ve not specified any type, in that case, type ‘ClusterIP’ will get used, which will make sure that MySQL container is exposed to the cluster and the Python app can access it.
If you check the app.py, you can see that I have used “mysql-service.default” as a hostname. “Mysql-service.default” is a DNS name of the service. The Python application will refer to that DNS name while accessing the MySQL Database.
Now, let’s actually setup the components from the configurations. As mentioned above, we will first create services followed by deployments.
At this stage your application is completely deployed and is externally accessible.
Manual scaling of pods
Scaling your application up or down in Kubernetes is quite straightforward. Let’s scale up the test-app deployment.
$ kubectl scale deployment test-app --replicas=3
Deployment configuration for test-app will get updated and you can see 3 replicas of test-app are running. Verify it using,
kubectl get pods
In the same manner, you can scale down your application by reducing the replica count.
Cleanup :
Un-deploying an application from Kubernetes is also quite straightforward. All we have to do is delete the services and delete the deployments. The only caveat is that the deletion of the load balancer is an asynchronous process. You have to wait until it gets deleted.
$ kubectl delete service mysql-service$ kubectl delete service test-service
The above command will deallocate Load Balancer which was created as a part of test-service. You can check the status of the load balancer with the following command.
$ gcloud compute forwarding-rules list
Once the load balancer is deleted, you can clean-up the deployments as well.
$ kubectl delete deployments test-app$ kubectl delete deployments mysql
In this blog, we saw how easy it is to deploy, scale & terminate applications on Google Container Engine. Google Container Engine abstracts away all the complexity of Kubernetes and gives us a robust platform to run containerised applications. I am super excited about what the future holds for Kubernetes!
Check out some of Velotio’s other blogs on Kubernetes.
Node.js has become the most popular framework for web development surpassing Ruby on Rails and Django in terms of the popularity.The growing popularity of full stack development along with the performance benefits of asynchronous programming has led to the rise of Node’s popularity. ExpressJs is a minimalistic, unopinionated and the most popular web framework built for Node which has become the de-facto framework for many projects. Note — This article is about building a Restful API server with ExpressJs . I won’t be delving into a templating library like handlebars to manage the views.
A quick search on google will lead you a ton of articles agreeing with what I just said which could validate the theory. Your next step would be to go through a couple of videos about ExpressJS on Youtube, try hello world with a boilerplate template, choose few recommended middleware for Express (Helmet, Multer etc), an ORM (mongoose if you are using Mongo or Sequelize if you are using relational DB) and start building the APIs. Wow, that was so fast!
The problem starts to appear after a few weeks when your code gets larger and complex and you realise that there is no standard coding practice followed across the client and the server code, refactoring or updating the code breaks something else, versioning of the APIs becomes difficult, call backs have made your life hell (you are smart if you are using Promises but have you heard of async-await?).
Do you think you your code is not so idiot-proof anymore? Don’t worry! You aren’t the only one who thinks this way after reading this.
Let me break the suspense and list down the technologies and libraries used in our idiot-proof code before you get restless.
Node 8.11.3: This is the latest LTS release from Node. We are using all the ES6 features along with async-await. We have the latest version of ExpressJs (4.16.3).
Typescript: It adds an optional static typing interface to Javascript and also gives us familiar constructs like classes (Es6 also gives provides class as a construct) which makes it easy to maintain a large codebase.
Swagger: It provides a specification to easily design, develop, test and document RESTful interfaces. Swagger also provides many open source tools like codegen and editor that makes it easy to design the app.
TSLint: It performs static code analysis on Typescript for maintainability, readability and functionality errors.
Prettier: It is an opinionated code formatter which maintains a consistent style throughout the project. This only takes care of the styling like the indentation (2 or 4 spaces), should the arguments remain on the same line or go to the next line when the line length exceeds 80 characters etc.
Husky: It allows you to add git hooks (pre-commit, pre-push) which can trigger TSLint, Prettier or Unit tests to automatically format the code and to prevent the push if the lint or the tests fail.
Before you move to the next section I would recommend going through the links to ensure that you have a sound understanding of these tools.
Now I’ll talk about some of the challenges we faced in some of our older projects and how we addressed these issues in the newer projects with the tools/technologies listed above.
Formal API definition
A problem that everyone can relate to is the lack of formal documentation in the project. Swagger addresses a part of this problem with their OpenAPI specification which defines a standard to design REST APIs which can be discovered by both machines and humans. As a practice, we first design the APIs in swagger before writing the code. This has 3 benefits:
It helps us to focus only on the design without having to worry about the code, scaffolder, naming conventions etc. Our API designs are consistent with the implementation because of this focused approach.
We can leverage tools like swagger-express-mw to internally wire the routes in the API doc to the controller, validate request and response object from their definitions etc.
Collaboration between teams becomes very easy, simple and standardised because of the Swagger specification.
Code Consistency
We wanted our code to look consistent across the stack (UI and Backend)and we use ESlint to enforce this consistency. Example – Node traditionally used “require” and the UI based frameworks used “import” based syntax to load the modules. We decided to follow ES6 style across the project and these rules are defined with ESLint.
Note — We have made slight adjustments to the TSlint for the backend and the frontend to make it easy for the developers. For example, we allow upto 120 characters in React as some of our DOM related code gets lengthy very easily.
Code Formatting
This is as important as maintaining the code consistency in the project. It’s easy to read a code which follows a consistent format like indentation, spaces, line breaks etc. Prettier does a great job at this. We have also integrated Prettier with Typescript to highlight the formatting errors along with linting errors. IDE like VS Code also has prettier plugin which supports features like auto-format to make it easy.
Strict Typing
Typescript can be leveraged to the best only if the application follows strict typing. We try to enforce it as much as possible with exceptions made in some cases (mostly when a third party library doesn’t have a type definition). This has the following benefits:
Static code analysis works better when your code is strongly typed. We discover about 80–90% of the issues before compilation itself using the plugins mentioned above.
Refactoring and enhancements becomes very simple with Typescript. We first update the interface or the function definition and then follow the errors thrown by Typescript compiler to refactor the code.
Git Hooks
Husky’s “pre-push” hook runs TSLint to ensure that we don’t push the code with linting issues. If you follow TDD (in the way it’s supposed to be done), then you can also run unit tests before pushing the code. We decided to go with pre-hooks because – Not everyone has CI from the very first day. With a git hook, we at least have some code quality checks from the first day. – Running lint and unit tests on the dev’s system will leave your CI with more resources to run integration and other complex tests which are not possible to do in local environment. – You force the developer to fix the issues at the earliest which results in better code quality, faster code merges and release.
Async-await
We were using promises in our project for all the asynchronous operations. Promises would often lead to a long chain of then-error blocks which is not very comfortable to read and often result in bugs when it got very long (it goes without saying that Promises are much better than the call back function pattern). Async-await provides a very clean syntax to write asynchronous operations which just looks like sequential code. We have seen a drastic improvement in the code quality, fewer bugs and better readability after moving to async-await.
Hope this article gave you some insights into tools and libraries that you can use to build a scalable ExpressJS app.
GitOps is a Continuous Deployment model for cloud-native applications. In GitOps, the Git repositories which contain the declarative descriptions of the infrastructure are considered as the single source of truth for the desired state of the system and we need to have an automated way to ensure that the deployed state of the system always matches the state defined in the Git repository. All the changes (deployment/upgrade/rollback) on the environment are triggered by changes (commits) made on the Git repository
“The artifacts that we run on any environment always have a corresponding code for them on some Git repositories. Can we say the same thing for our infrastructure code?”
Infrastructure as code tools, completely declarative orchestration tools like Kubernetes allow us to represent the entire state of our system in a declarative way. GitOps intends to make use of this ability and make infrastructure-related operations more developer-centric.
Role of Infrastructure as Code (IaC) in GitOps
The ability to represent the infrastructure as code is at the core of GitOps. But just having versioned controlled infrastructure as code doesn’t mean GitOps, we also need to have a mechanism in place to keep (try to keep) our deployed state in sync with the state we define in the Git repository.
“Infrastructure as Code is necessary but not sufficient to achieve GitOps”
GitOps does pull-based deployments
Most of the deployment pipelines we see currently, push the changes in the deployed environment. For example, consider that we need to upgrade our application to a newer version then we will update its docker image tag in some repository which will trigger a deployment pipeline and update the deployed application. Here the changes were pushed on the environment. In GitOps, we just need to update the image tag on the Git repository for that environment and the changes will be pulled to the environment to match the updated state in the Git repository. The magic of keeping the deployed state in sync with state-defined on Git is achieved with the help of operators/agents. The operator is like a control loop which can identify differences between the deployed state and the desired state and make sure they are the same.
Key benefits of GitOps:
All the changes are verifiable and auditable as they make their way into the system through Git repositories.
Easy and consistent replication of the environment as Git repository is the single source of truth. This makes disaster recovery much quicker and simpler.
More developer-centric experience for operating infrastructure. Also a smaller learning curve for deploying dev environments.
Consistent rollback of application as well as infrastructure state.
Introduction to Argo CD
Argo CD is a continuous delivery tool that works on the principles of GitOps and is built specifically for Kubernetes. The product was developed and open-sourced by Intuit and is currently a part of CNCF.
Key components of Argo CD:
API Server: Just like K8s, Argo CD also has an API server that exposes APIs that other systems can interact with. The API server is responsible for managing the application, repository and cluster credentials, enforcing authentication and authorization, etc.
Repository server: The repository server keeps a local cache of the Git repository, which holds the K8s manifest files for the application. This service is called by other services to get the K8s manifests.
Application controller: The application controller continuously watches the deployed state of the application and compares it with the desired state of the application, reports the API server whenever they are not in sync with each other and seldom takes corrective actions as well. It is also responsible for executing user-defined hooks for various lifecycle events of the application.
Key objects/resources in Argo CD:
Application: Argo CD allows us to represent the instance of the application which we want to deploy in an environment by creating Kubernetes objects of a custom resource definition(CRD) named Application. In the specification of Application type objects, we specify the source (repository) of our application’s K8s manifest files, the K8s server where we want to deploy those manifests, namespace, and other information.
AppProject: Just like Application, Argo CD provides another CRD named AppProject. AppProjects are used to logically group related-applications.
Repo Credentials: In the case of private repositories, we need to provide access credentials. For credentials, Argo CD uses the K8s secrets and config map. First, we create objects of secret types and then we update a special-purpose configuration map named argocd-cm with the repository URL and the secret which contains the credentials.
Cluster Credentials: Along with Git repository credentials, we also need to provide the K8s cluster credentials. These credentials are also managed using K8s secret, we are required to add the label argocd.argoproj.io/secret-type: cluster to these secrets.
Demo:
Enough of theory, let’s try out the things we discussed above. For the demo, I have created a simple app named message-app. This app reads a message set in the environment variable named MESSAGE. We will populate the values of this environment variable using a K8s config map. I have kept the K8s manifest files for the app in a separate repository. We have the application and the K8s manifest files ready. Now we are all set to install Argo CD and deploy our application.
Installing Argo CD:
For installing Argo CD, we first need to create a namespace named argocd.
Applying the files from the argo-cd repo directly is fine for demo purposes, but in actual environments, you must copy the file in your repository before applying them.
We can see that this command has created the core components and CRDs we discussed earlier in the blog. There are some additional resources as well but we can ignore them for the time being.
Accessing the Argo CD GUI
We have the Argo CD running in our cluster, Argo CD also provides a GUI which gives us a graphical representation of our k8s objects. It allows us to view events, pod logs, and other configurations.
By default, the GUI service is not exposed outside the cluster. Let us update its service type to LoadBalancer so that we can access it from outside.
After this, we can use the external IP of the argocd-server service and access the GUI.
The initial username is admin and the password is the name of the api-server pod. The password can be obtained by listing the pods in the argocd namespace or directly by this command.
kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/'-f 2
Deploy the app:
Now let’s go ahead and create our application for the staging environment for our message app.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:name: message-app-stagingnamespace: argocdenvironment: stagingfinalizers:- resources-finalizer.argocd.argoproj.iospec:project: default # Source of the application manifestssource:repoURL: https://github.com/akash-gautam/message-app-manifests.gittargetRevision: HEADpath: manifests/staging # Destination cluster and namespacetodeploytheapplicationdestination:server: https://kubernetes.default.svcnamespace: stagingsyncPolicy:automated:prune: falseselfHeal: false
In the application spec, we have specified the repository, where our manifest files are stored and also the path of the files in the repository.
We want to deploy our app in the same k8s cluster where ArgoCD is running so we have specified the local k8s service URL in the destination. We want the resources to be deployed in the staging namespace, so we have set it accordingly.
In the sync policy, we have enabled automated sync. We have kept the project as default.
Adding the resources-finalizer.argocd.argoproj.io ensures that all the resources created for the application are deleted when the Application is deleted. This is fine for our demo setup but might not always be desirable in real-life scenarios.
Our git repos are public so we don’t need to create secrets for git repo credentials.
We are deploying in the same cluster where Argo CD itself is running. As this is a demo setup, we can use the admin user created by Argo CD, so we don’t need to create secrets for cluster credentials either.
Now let’s go ahead and create the application and see the magic happen.
kubectl apply -f message-app-staging.yaml
As soon as the application is created, we can see it on the GUI.
By clicking on the application, we can see all the Kubernetes objects created for it.
It also shows the objects which are indirectly created by the objects we create. In the above image, we can see the replica set and endpoint object which are created as a result of creating the deployment and service respectively.
We can also click on the individual objects and see their configuration. For pods, we can see events and logs as well.
As our app is deployed now, we can grab public IP of message-app service and access it on the browser.
We can see that our app is deployed and accessible.
Updating the app
For updating our application, all we need to do is commit our changes to the GitHub repository. We know the message-app just displays the message we pass to it via. Config map, so let’s update the message and push it to the repository.
apiVersion: v1kind: ConfigMapmetadata:name: message-configmaplabels:app: message-appdata:MESSAGE: "This too shall pass" #Put the message you want to display here.
Once the commit is done, Argo CD will start to sync again.
Once the sync is done, we will restart our message app pod, so that it picks up the latest values in the config map. Then we need to refresh the browser to see updated values.
As we discussed earlier, for making any changes to the environment, we just need to update the repo which is being used as the source for the environment and then the changes will get pulled in the environment.
We can follow an exact similar approach and deploy the application to the production environment as well. We just need to create a new application object and set the manifest path and deployment namespace accordingly.
Conclusion:
It’s still early days for GitOps, but it has already been successfully implemented at scale by many organizations. As the GitOps tools mature along with the ever-growing adoption of Kubernetes, I think many organizations will consider adopting GitOps soon. GitOps is not limited only to Kubernetes, but the completely declarative nature of Kubernetes makes it simpler to achieve GitOps. Argo CD is a deployment tool that’s tailored for Kubernetes and allows us to do deployments in a Kubernetes native way while following the principles of GitOps.I hope this blog helped you in understanding how what and why of GitOps and gave some insights to Argo CD.
Containerized applications and Kubernetes adoption in cloud environments is on the rise. One of the challenges while deploying applications in Kubernetes is exposing these containerized applications to the outside world. This blog explores different options via which applications can be externally accessed with focus on Ingress – a new feature in Kubernetes that provides an external load balancer. This blog also provides a simple hand-on tutorial on Google Cloud Platform (GCP).
Ingress is the new feature (currently in beta) from Kubernetes which aspires to be an Application Load Balancer intending to simplify the ability to expose your applications and services to the outside world. It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting etc. Before we dive into Ingress, let’s look at some of the alternatives currently available that help expose your applications, their complexities/limitations and then try to understand Ingress and how it addresses these problems.
Current ways of exposing applications externally:
There are certain ways using which you can expose your applications externally. Lets look at each of them:
EXPOSE Pod:
You can expose your application directly from your pod by using a port from the node which is running your pod, mapping that port to a port exposed by your container and using the combination of your HOST-IP:HOST-PORT to access your application externally. This is similar to what you would have done when running docker containers directly without using Kubernetes. Using Kubernetes you can use hostPortsetting in service configuration which will do the same thing. Another approach is to set hostNetwork: true in service configuration to use the host’s network interface from your pod.
Limitations:
In both scenarios you should take extra care to avoid port conflicts at the host, and possibly some issues with packet routing and name resolutions.
This would limit running only one replica of the pod per cluster node as the hostport you use is unique and can bind with only one service.
EXPOSE Service:
Kubernetes services primarily work to interconnect different pods which constitute an application. You can scale the pods of your application very easily using services. Services are not primarily intended for external access, but there are some accepted ways to expose services to the external world.
Basically, services provide a routing, balancing and discovery mechanism for the pod’s endpoints. Services target pods using selectors, and can map container ports to service ports. A service exposes one or more ports, although usually, you will find that only one is defined.
A service can be exposed using 3 ServiceType choices:
ClusterIP: Exposes the service on a cluster-internal IP. Choosing this value makes the service only reachable from within the cluster. This is the default ServiceType.
NodePort: Exposes the service on each Node’s IP at a static port (the NodePort). A ClusterIP service, to which the NodePort service will route, is automatically created. You’ll be able to contact the NodePort service, from outside the cluster, by requesting <nodeip>:<nodeport>.Here NodePort remains fixed and NodeIP can be any node IP of your Kubernetes cluster.</nodeport></nodeip>
LoadBalancer: Exposes the service externally using a cloud provider’s load balancer (eg. AWS ELB). NodePort and ClusterIP services, to which the external load balancer will route, are automatically created.
ExternalName: Maps the service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record with its value. No proxying of any kind is set up. This requires version 1.7 or higher of kube-dns
Limitations:
If we choose NodePort to expose our services, kubernetes will generate ports corresponding to the ports of your pods in the range of 30000-32767. You will need to add an external proxy layer that uses DNAT to expose more friendly ports. The external proxy layer will also have to take care of load balancing so that you leverage the power of your pod replicas. Also it would not be easy to add TLS or simple host header routing rules to the external service.
ClusterIP and ExternalName similarly while easy to use have the limitation where we can add any routing or load balancing rules.
Choosing LoadBalancer is probably the easiest of all methods to get your service exposed to the internet. The problem is that there is no standard way of telling a Kubernetes service about the elements that a balancer requires, again TLS and host headers are left out. Another limitation is reliance on an external load balancer (AWS’s ELB, GCP’s Cloud Load Balancer etc.)
Endpoints
Endpoints are usually automatically created by services, unless you are using headless services and adding the endpoints manually. An endpoint is a host:port tuple registered at Kubernetes, and in the service context it is used to route traffic. The service tracks the endpoints as pods, that match the selector are created, deleted and modified. Individually, endpoints are not useful to expose services, since they are to some extent ephemeral objects.
Summary
If you can rely on your cloud provider to correctly implement the LoadBalancer for their API, to keep up-to-date with Kubernetes releases, and you are happy with their management interfaces for DNS and certificates, then setting up your services as type LoadBalancer is quite acceptable.
On the other hand, if you want to manage load balancing systems manually and set up port mappings yourself, NodePort is a low-complexity solution. If you are directly using Endpoints to expose external traffic, perhaps you already know what you are doing (but consider that you might have made a mistake, there could be another option).
Given that none of these elements has been originally designed to expose services to the internet, their functionality may seem limited for this purpose.
Understanding Ingress
Traditionally, you would create a LoadBalancer service for each public application you want to expose. Ingress gives you a way to route requests to services based on the request host or path, centralizing a number of services into a single entrypoint.
Ingress is split up into two main pieces. The first is an Ingress resource, which defines how you want requests routed to the backing services and second is the Ingress Controller which does the routing and also keeps track of the changes on a service level.
Ingress Resources
The Ingress resource is a set of rules that map to Kubernetes services. Ingress resources are defined purely within Kubernetes as an object that other entities can watch and respond to.
Ingress Supports defining following rules in beta stage:
host header: Forward traffic based on domain names.
paths: Looks for a match at the beginning of the path.
TLS: If the ingress adds TLS, HTTPS and a certificate configured through a secret will be used.
When no host header rules are included at an Ingress, requests without a match will use that Ingress and be mapped to the backend service. You will usually do this to send a 404 page to requests for sites/paths which are not sent to the other services. Ingress tries to match requests to rules, and forwards them to backends, which are composed of a service and a port.
Ingress Controllers
Ingress controller is the entity which grants (or remove) access, based on the changes in the services, pods and Ingress resources. Ingress controller gets the state change data by directly calling Kubernetes API.
Ingress controllers are applications that watch Ingresses in the cluster and configure a balancer to apply those rules. You can configure any of the third party balancers like HAProxy, NGINX, Vulcand or Traefik to create your version of the Ingress controller. Ingress controller should track the changes in ingress resources, services and pods and accordingly update configuration of the balancer.
Ingress controllers will usually track and communicate with endpoints behind services instead of using services directly. This way some network plumbing is avoided, and we can also manage the balancing strategy from the balancer. Some of the open source implementations of Ingress Controllers can be found here.
Now, let’s do an exercise of setting up a HTTP Load Balancer using Ingress on Google Cloud Platform (GCP), which has already integrated the ingress feature in it’s Container Engine (GKE) service.
Ingress-based HTTP Load Balancer in Google Cloud Platform
The tutorial assumes that you have your GCP account setup done and a default project created. We will first create a Container cluster, followed by deployment of a nginx server service and an echoserver service. Then we will setup an ingress resource for both the services, which will configure the HTTP Load Balancer provided by GCP
Basic Setup
Get your project ID by going to the “Project info” section in your GCP dashboard. Start the Cloud Shell terminal, set your project id and the compute/zone in which you want to create your cluster.
$ gcloud config set project glassy-chalice-129514$ gcloud config set compute/zone us-east1-d# Create a 3 node cluster with name “loadbalancedcluster”$ gcloud container clusters create loadbalancedcluster
Fetch the cluster credentials for the kubectl tool:
When you create a Service of type NodePort with this command, Container Engine makes your Service available on a randomly-selected high port number (e.g. 30746) on all the nodes in your cluster. Verify the Service was created and a node port was allocated:
$ kubectl get service nginxNAMECLUSTER-IPEXTERNAL-IPPORT(S) AGEnginx 10.47.245.54<nodes>80:30746/TCP 20s$ kubectl get service echoserverNAMECLUSTER-IPEXTERNAL-IPPORT(S) AGEechoserver 10.47.251.9<nodes>8080:32301/TCP 33s
In the output above, the node port for the nginx Service is 30746 and for echoserver service is 32301. Also, note that there is no external IP allocated for this Services. Since the Container Engine nodes are not externally accessible by default, creating this Service does not make your application accessible from the Internet. To make your HTTP(S) web server application publicly accessible, you need to create an Ingress resource.
Step 3: Create an Ingress resource
On Container Engine, Ingress is implemented using Cloud Load Balancing. When you create an Ingress in your cluster, Container Engine creates an HTTP(S) load balancer and configures it to route traffic to your application. Container Engine has internally defined an Ingress Controller, which takes the Ingress resource as input for setting up proxy rules and talk to Kubernetes API to get the service related information.
The following config file defines an Ingress resource that directs traffic to your nginx and echoserver server:
To deploy this Ingress resource run in the cloud shell:
$ kubectl apply -f basic-ingress.yaml
Step 4: Access your application
Find out the external IP address of the load balancer serving your application by running:
$ kubectl get ingress fanout-ingresNAMEHOSTSADDRESSPORTSAGfanout-ingress *130.211.36.16880 36s
Use http://<external-ip-address> </external-ip-address>and http://<external-ip-address>/echo</external-ip-address> to access nginx and the echo-server.
Summary
Ingresses are simple and very easy to deploy, and really fun to play with. However, it’s currently in beta phase and misses some of the features that may restrict it from production use. Stay tuned to get updates in Ingress on Kubernetes page and their Github repo.
Google BigQuery is an enterprise data warehouse built using BigTable and Google Cloud Platform. It’s serverless and completely managed. BigQuery works great with all sizes of data, from a 100 row Excel spreadsheet to several Petabytes of data. Most importantly, it can execute a complex query on those data within a few seconds.
We need to note before we proceed, BigQuery is not a transactional database. It takes around 2 seconds to run a simple query like ‘SELECT * FROM bigquery-public-data.object LIMIT 10’ on a 100 KB table with 500 rows. Hence, it shouldn’t be thought of as OLTP (Online Transaction Processing) database. BigQuery is for Big Data!
BigQuery supports SQL-like query, which makes it user-friendly and beginner friendly. It’s accessible via its web UI, command-line tool, or client library (written in C#, Go, Java, Node.js, PHP, Python, and Ruby). You can also take advantage of its REST APIs and get our job` done by sending a JSON request.
Now, let’s dive deeper to understand it better. Suppose you are a data scientist (or a startup which analyzes data) and you need to analyze terabytes of data. If you choose a tool like MySQL, the first step before even thinking about any query is to have an infrastructure in place, that can store this magnitude of data.
Designing this setup itself will be a difficult task because you have to figure out what will be the RAM size, DCOS or Kubernetes, and other factors. And if you have streaming data coming, you will need to set up and maintain a Kafka cluster. In BigQuery, all you have to do is a bulk upload of your CSV/JSON file, and you are done. BigQuery handles all the backend for you. If you need streaming data ingestion, you can use Fluentd. Another advantage of this is that you can connect Google Analytics with BigQuery seamlessly.
BigQuery is serverless, highly available, and petabyte scalable service which allows you to execute complex SQL queries quickly. It lets you focus on analysis rather than handling infrastructure. The idea of hardware is completely abstracted and not visible to us, not even as virtual machines.
Architecture of Google BigQuery
You don’t need to know too much about the underlying architecture of BigQuery. That’s actually the whole idea of it – you don’t need to worry about architecture and operation.
However, understanding BigQuery Architecture helps us in controlling costs, optimizing query performance, and optimizing storage. BigQuery is built using the Google Dremel paper.
Quoting an Abstract from the Google Dremel Paper –
“Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data and has thousands of users at Google. In this paper, we describe the architecture and implementation of Dremel and explain how it complements MapReduce-based computing. We present a novel columnar storage representation for nested records and discuss experiments on few-thousand node instances of the system.”
Dremel was in production at Google since 2006. Google used it for the following tasks –
Analysis of crawled web documents.
Tracking install data for applications on Android Market.
Crash reporting for Google products.
OCR results from Google Books.
Spam analysis.
Debugging of map tiles on Google Maps.
Tablet migrations in managed Bigtable instances.
Results of tests run on Google’s distributed build system.
Disk I/O statistics for hundreds of thousands of disks.
Resource monitoring for jobs run in Google’s data centers.
Symbols and dependencies in Google’s codebase.
BigQuery is much more than Dremel. Dremel is just a query execution engine, whereas Bigquery is based on interesting technologies like Borg (predecessor of Kubernetes) and Colossus. Colossus is the successor to the Google File System (GFS) as mentioned in Google Spanner Paper.
How BigQuery Stores Data?
BigQuery stores data in a columnar format – Capacitor (which is a successor of ColumnarIO). BigQuery achieves very high compression ratio and scan throughput. Unlike ColumnarIO, now on BigQuery, you can directly operate on compressed data without decompressing it.
Columnar storage has the following advantages:
Traffic minimization – When you submit a query, the required column values on each query are scanned and only those are transferred on query execution. E.g., a query `SELECT title FROM Collection` would access the title column values only.
Higher compression ratio – Columnar storage can achieve a compression ratio of 1:10, whereas ordinary row-based storage can compress at roughly 1:3.
(Image source: Google Dremel Paper)
Columnar storage has the disadvantage of not working efficiently when updating existing records. That is why Dremel doesn’t support any update queries.
How the Query Gets Executed?
BigQuery depends on Borg for data processing. Borg simultaneously instantiates hundreds of Dremel jobs across required clusters made up of thousands of machines. In addition to assigning compute capacity for Dremel jobs, Borg handles fault-tolerance as well.
Now, how do you design/execute a query which can run on thousands of nodes and fetches the result? This challenge was overcome by using the Tree Architecture. This architecture forms a gigantically parallel distributed tree for pushing down a query to the tree and aggregating the results from the leaves at a blazingly fast speed.
(Image source: Google Dremel Paper)
BigQuery vs. MapReduce
The key differences between BigQuery and MapReduce are –
Dremel is designed as an interactive data analysis tool for large datasets
MapReduce is designed as a programming framework to batch process large datasets
Moreover, Dremel finishes most queries within seconds or tens of seconds and can even be used by non-programmers, whereas MapReduce takes much longer (sometimes even hours or days) to process a query.
Following is a comparison on running MapReduce on a row and columnar DB:
(Image source: Google Dremel Paper)
Another important thing to note is that BigQuery is meant to analyze structured data (SQL) but in MapReduce, you can write logic for unstructured data as well.
Comparing BigQuery and Redshift
In Redshift, you need to allocate different instance types and create your own clusters. The benefit of this is that it lets you tune the compute/storage to meet your needs. However, you have to be aware of (virtualized) hardware limits and scale up/out based on that. Note that you are charged by the hour for each instance you spin up.
In BigQuery, you just upload the data and query it. It is a truly managed service. You are charged by storage, streaming inserts, and queries.
There are more similarities in both the data warehouses than the differences.
A smart user will definitely take advantage of the hybrid cloud (GCE+AWS) and leverage different services offered by both the ecosystems. Check out your quintessential guide to AWS Athena here.
Getting Started With Google BigQuery
Following is a quick example to show how you can quickly get started with BigQuery:
There are many public datasets available on bigquery, you are going to play with ‘bigquery-public-data:stackoverflow’ dataset. You can click on the “Add Data” button on the left panel and select datasets.
2. Next, find a language that has the best community, based on the response time. You can write the following query to do that.
WITH question_answers_join AS (SELECT* , GREATEST(1, TIMESTAMP_DIFF(answers.first, creation_date, minute)) minutes_2_answerFROM (SELECT id, creation_date, title , (SELECTASSTRUCTMIN(creation_date) first, COUNT(*) cFROM`bigquery-public-data.stackoverflow.posts_answers`WHERE a.id=parent_id ) answers , SPLIT(tags, '|') tagsFROM`bigquery-public-data.stackoverflow.posts_questions` aWHEREEXTRACT(year FROM creation_date) >2014 ))SELECTCOUNT(*) questions, tag , ROUND(EXP(AVG(LOG(minutes_2_answer))), 2) mean_geo_minutes , APPROX_QUANTILES(minutes_2_answer, 100)[SAFE_OFFSET(50)] medianFROM question_answers_join, UNNEST(tags) tagWHERE tag IN ('javascript', 'python', 'rust', 'java', 'scala', 'ruby', 'go', 'react', 'c', 'c++')AND answers.c >0GROUPBY tagORDERBY mean_geo_minutes
3. Now you can execute the query and get results –
You can see that C has the best community followed by JavaScript!
How to do Machine Learning on BigQuery?
Now that you have a sound understanding of BigQuery. It’s time for some real action.
As discussed above, you can connect Google Analytics with BigQuery by going to the Google Analytics Admin panel, then enable BigQuery by clicking on PROPERTY column, click All Products, then click Link BigQuery. After that, you need to enter BigQuery ID (or project number) and then BigQuery will be linked to Google Analytics. Note – Right now BigQuery integration is only available to Google Analytics 360.
Assuming that you already have uploaded your google analytics data, here is how you can create a logistic regression model. Here, you are predicting whether a website visitor will make a transaction or not.
CREATEMODEL`velotio_tutorial.sample_model`OPTIONS(model_type='logistic_reg') ASSELECTIF(totals.transactions ISNULL, 0, 1) AS label,IFNULL(device.operatingSystem, "") AS os, device.isMobile AS is_mobile,IFNULL(geoNetwork.country, "") AS country,IFNULL(totals.pageviews, 0) AS pageviewsFROM`bigquery-public-data.google_analytics_sample.ga_sessions_*`WHERE _TABLE_SUFFIX BETWEEN'20190401'AND'20180630'
Create a model named ‘velotio_tutorial.sample_model’. Now set the ‘model_type’ as ‘logistic_reg’ because you want to train a logistic regression model. A logistic regression model splits input data into two classes and gives the probability that the data is in one of the classes. Usually, in “spam or not spam” type of problems, you use logistic regression. Here, the problem is similar – a transaction will be made or not.
The above query gets the total number of page views, the country from where the session originated, the operating system of visitors device, the total number of e-commerce transactions within the session, etc.
Now you just press run query to execute the query.
Conclusion
BigQuery is a query service that allows us to run SQL-like queries against multiple terabytes of data in a matter of seconds. If you have structured data, BigQuery is the best option to go for. It can help even a non-programmer to get the analytics right!
A serverless architecture is a way to implement and run applications and services or micro-services without need to manage infrastructure. Your application still runs on servers, but all the servers management is done by AWS. Now we don’t need to provision, scale or maintain servers to run our applications, databases and storage systems. Services which are developed by developers who don’t let developers build application from scratch.
Why Serverless
More focus on development rather than managing servers.
Cost Effective.
Application which scales automatically.
Quick application setup.
Services For ServerLess
For implementing serverless architecture there are multiple services which are provided by cloud partners though we will be exploring most of the services from AWS. Following are the services which we can use depending on the application requirement.
Lambda: It is used to write business logic / schedulers / functions.
S3: It is mostly used for storing objects but it also gives the privilege to host WebApps. You can host a static website on S3.
API Gateway: It is used for creating, publishing, maintaining, monitoring and securing REST and WebSocket APIs at any scale.
Cognito: It provides authentication, authorization & user management for your web and mobile apps. Your users can sign in directly sign in with a username and password or through third parties such as Facebook, Amazon or Google.
DynamoDB: It is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.
Three-tier Serverless Architecture
So, let’s take a use case in which you want to develop a three tier serverless application. The three tier architecture is a popular pattern for user facing applications, The tiers that comprise the architecture include the presentation tier, the logic tier and the data tier. The presentation tier represents the component that users directly interact with web page / mobile app UI. The logic tier contains the code required to translate user action at the presentation tier to the functionality that drives the application’s behaviour. The data tier consists of your storage media (databases, file systems, object stores) that holds the data relevant to the application. Figure shows the simple three-tier application.
Figure: Simple Three-Tier Architectural Pattern
Presentation Tier
The presentation tier of the three tier represents the View part of the application. Here you can use S3 to host static website. On a static website, individual web pages include static content and they also contain client side scripting.
The following is a quick procedure to configure an Amazon S3 bucket for static website hosting in the S3 console.
To configure an S3 bucket for static website hosting
1. Log in to the AWS Management Console and open the S3 console at
2. In the Bucket name list, choose the name of the bucket that you want to enable static website hosting for.
3. Choose Properties.
4. Choose Static Website Hosting
Once you enable your bucket for static website hosting, browsers can access all of your content through the Amazon S3 website endpoint for your bucket.
5. Choose Use this bucket to host.
A. For Index Document, type the name of your index document, which is typically named index.html. When you configure a S3 bucket for website hosting, you must specify an index document, which will be returned by S3 when requests are made to the root domain or any of the subfolders.
B. (Optional) For 4XX errors, you can optionally provide your own custom error document that provides additional guidance for your users. Type the name of the file that contains the custom error document. If an error occurs, S3 returns an error document.
C. (Optional) If you want to give advanced redirection rules, In the edit redirection rule text box, you have to XML to describe the rule. E.g.
7. Add a bucket policy to the website bucket that grants access to the object in the S3 bucket for everyone. You must make the objects that you want to serve publicly readable, when you configure a S3 bucket as a website. To do so, you write a bucket policy that grants everyone S3:GetObject permission. The following bucket policy grants everyone access to the objects in the example-bucket bucket.
Note: If you choose Disable Website Hosting, S3 removes the website configuration from the bucket, so that the bucket no longer accessible from the website endpoint, but the bucket is still available at the REST endpoint.
Logic Tier
The logic tier represents the brains of the application. Here the two core services for serverless will be used i.e. API Gateway and Lambda to form your logic tier can be so revolutionary. The feature of the 2 services allow you to build a serverless production application which is highly scalable, available and secure. Your application could use number of servers, however by leveraging this pattern you do not have to manage a single one. In addition, by using these managed services together you get following benefits:
No operating system to choose, secure or manage.
No servers to right size, monitor.
No risk to your cost by over-provisioning.
No Risk to your performance by under-provisioning.
API Gateway
API Gateway is a fully managed service for defining, deploying and maintaining APIs. Anyone can integrate with the APIs using standard HTTPS requests. However, it has specific features and qualities that result it being an edge for your logic tier.
Integration with Lambda
API Gateway gives your application a simple way to leverage the innovation of AWS lambda directly (HTTPS Requests). API Gateway forms the bridge that connects your presentation tier and the functions you write in Lambda. After defining the client / server relationship using your API, the contents of the client’s HTTPS requests are passed to Lambda function for execution. The content include request metadata, request headers and the request body.
API Performance Across the Globe
Each deployment of API Gateway includes an Amazon CloudFront distribution under the covers. Amazon CloudFront is a content delivery web service that used Amazon’s global network of edge locations as connection points for clients integrating with API. This helps drive down the total response time latency of your API. Through its use of multiple edge locations across the world, Amazon CloudFront also provides you capabilities to combat distributed denial of service (DDoS) attack scenarios.
You can improve the performance of specific API requests by using API Gateway to store responses in an optional in-memory cache. This not only provides performance benefits for repeated API requests, but is also reduces backend executions, which can reduce overall cost.
Let’s dive into each step
1. Create Lambda Function Login to Aws Console and head over to Lambda Service and Click on “Create A Function”
A. Choose first option “Author from scratch” B. Enter Function Name C. Select Runtime e.g. Python 2.7 D. Click on “Create Function”
As your function is ready, you can see your basic function will get generated in language you choose to write. E.g.
import jsondef lambda_handler(event, context): # TODO implement return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }
2. Testing Lambda Function
Click on “Test” button at the top right corner where we need to configure test event. As we are not sending any events, just give event a name, for example, “Hello World” template as it is and “Create” it.
Now, when you hit the “Test” button again, it runs through testing the function we created earlier and returns the configured value.
Create & Configure API Gateway connecting to Lambda
We are done with creating lambda functions but how to invoke function from outside world ? We need endpoint, right ?
Go to API Gateway & click on “Get Started” and agree on creating an Example API but we will not use that API we will create “New API”. Give it a name by keeping “Endpoint Type” regional for now.
Create the API and you will go on the page “resources” page of the created API Gateway. Go through the following steps:
A. Click on the “Actions”, then click on “Create Method”. Select Get method for our function. Then, “Tick Mark” on the right side of “GET” to set it up. B. Choose “Lambda Function” as integration type. C. Choose the region where we created earlier. D. Write the name of Lambda Function we created E. Save the method where it will ask you for confirmation of “Add Permission to Lambda Function”. Agree to that & that is done. F. Now, we can test our setup. Click on “Test” to run API. It should give the response text we had on the lambda test screen.
Now, to get endpoint. We need to deploy the API. On the Actions dropdown, click on Deploy API under API Actions. Fill in the details of deployment and hit Deploy.
After that, we will get our HTTPS endpoint.
On the above screen you can see the things like cache settings, throttling, logging which can be configured. Save the changes and browse the invoke URL from which we will get the response which was earlier getting from Lambda. So, here is our logic tier of serverless application is to be done.
Data Tier
By using Lambda as your logic tier, you have a number of data storage options for your data tier. These options fall into broad categories: Amazon VPC hosted data stores and IAM-enabled data stores. Lambda has the ability to integrate with both securely.
Amazon VPC Hosted Data Stores
Amazon RDS
Amazon ElasticCache
Amazon Redshift
IAM-Enabled Data Stores
Amazon DynamoDB
Amazon S3
Amazon ElasticSearch Service
You can use any of those for storage purpose, But DynamoDB is one of best suited for ServerLess application.
Why DynamoDB ?
It is NoSQL DB, also that is fully managed by AWS.
It provides fast & prectable performance with seamless scalability.
DynamoDB lets you offload the administrative burden of operating and scaling a distributed system.
It offers encryption at rest, which eliminates the operational burden and complexity involved in protecting sensitive data.
You can scale up/down your tables throughput capacity without downtime/performance degradation.
It provides On-Demand backups as well as enable point in time recovery for your DynamoDB tables.
DynamoDB allows you to delete expired items from table automatically to help you reduce storage usage and the cost of storing data that is no longer relevant.
Following is the sample script for DynamoDB with Python which you can use with lambda.
from __future__ import print_function # Python 2/3 compatibilityimport boto3import jsonimport decimalfrom boto3.dynamodb.conditions import Key, Attrfrom botocore.exceptions import ClientError# Helper class to convert a DynamoDB item to JSON.class DecimalEncoder(json.JSONEncoder): def default(self, o): if isinstance(o, decimal.Decimal): if o % 1 > 0: return float(o) else: return int(o) return super(DecimalEncoder, self).default(o)dynamodb = boto3.resource("dynamodb", region_name='us-west-2', endpoint_url="http://localhost:8000")table = dynamodb.Table('Movies')title = "The Big New Movie"year = 2015try: response = table.get_item( Key={ 'year': year, 'title': title } )except ClientError as e: print(e.response['Error']['Message'])else: item = response['Item'] print("GetItem succeeded:") print(json.dumps(item, indent=4, cls=DecimalEncoder))
Note: To run the above script successfully you need to attach policy to your role for lambda. So in this case you need to attach policy for DynamoDB operations to take place & for CloudWatch if required to store your logs. Following is the policy which you can attach to your role for DB executions.
You can implement the following popular architecture patterns using API Gateway & Lambda as your logic tier, Amazon S3 for presentation tier, DynamoDB as your data tier. For each example, we will only use AWS Service that do not require users to manage their own infrastructure.
Mobile Backend
1. Presentation Tier: A mobile application running on each user’s smartphone.
2. Logic Tier: API Gateway & Lambda. The logic tier is globally distributed by the Amazon CloudFront distribution created as part of each API Gateway each API. A set of lambda functions can be specific to user / device identity management and authentication & managed by Amazon Cognito, which provides integration with IAM for temporary user access credentials as well as with popular third party identity providers. Other Lambda functions can define core business logic for your Mobile Back End.
3. Data Tier: The various data storage services can be leveraged as needed; options are given above in data tier.
Amazon S3 Hosted Website
1. Presentation Tier: Static website content hosted on S3, distributed by Amazon CLoudFront. Hosting static website content on S3 is a cost effective alternative to hosting content on server-based infrastructure. However, for a website to contain rich feature, the static content often must integrate with a dynamic back end.
2. Logic Tier: API Gateway & Lambda, static web content hosted in S3 can directly integrate with API Gateway, which can be CORS complaint.
3. Data Tier: The various data storage services can be leveraged based on your requirement.
ServerLess Costing
At the top of the AWS invoice, we can see the total costing of AWS Services. The bill was processed for 2.1 million API request & all of the infrastructure required to support them.
Following is the list of services with their costing.
Note: You can get your costing done from AWS Calculator using following links;
The three-tier architecture pattern encourages the best practice of creating application component that are easy to maintain, develop, decoupled & scalable. Serverless Application services varies based on the requirements over development.
The concept of operators was introduced by CoreOs in the last quarter of 2016 and post the introduction of operator framework last year, operators are rapidly becoming the standard way of managing applications on Kubernetes especially the ones which are stateful in nature. In this blog post, we will learn what an operator is. Why they are needed and what problems do they solve. We will also create a helm based operator as an example.
This is the first part of our Kubernetes Operator Series. In the second part, getting started with Kubernetes operators (Ansible based), and the third part, getting started with Kubernetes operators (Golang based), you can learn how to build Ansible and Golang based operators.
What is an Operator?
Whenever we deploy our application on Kubernetes we leverage multiple Kubernetes objects like deployment, service, role, ingress, config map, etc. As our application gets complex and our requirements become non-generic, managing our application only with the help of native Kubernetes objects becomes difficult and we often need to introduce manual intervention or some other form of automation to make up for it.
Operators solve this problem by making our application first class Kubernetes objects that is we no longer deploy our application as a set of native Kubernetes objects but a custom object/resource of its kind, having a more domain-specific schema and then we bake the “operational intelligence” or the “domain-specific knowledge” into the controller responsible for maintaining the desired state of this object. For example, etcd operator has made etcd-cluster a first class object and for deploying the cluster we create an object of Etcd Cluster kind. With operators, we are able to extend Kubernetes functionalities for custom use cases and manage our applications in a Kubernetes specific way allowing us to leverage Kubernetes APIs and Kubectl tooling.
Operators combine crds and custom controllers and intend to eliminate the requirement for manual intervention (human operator) while performing tasks like an upgrade, handling failure recovery, scaling in case of complex (often stateful) applications and make them more resilient and self-sufficient.
How to Build Operators ?
For building and managing operators we mostly leverage the Operator Framework which is an open source tool kit allowing us to build operators in a highly automated, scalable and effective way. Operator framework comprises of three subcomponents:
Operator SDK: Operator SDK is the most important component of the operator framework. It allows us to bootstrap our operator project in minutes. It exposes higher level APIs and abstraction and saves developers the time to dig deeper into kubernetes APIs and focus more on building the operational logic. It performs common tasks like getting the controller to watch the custom resource (cr) for changes etc as part of the project setup process.
Operator Lifecycle Manager: Operators also run on the same kubernetes clusters in which they manage applications and more often than not we create multiple operators for multiple applications. Operator lifecycle manager (OLM) provides us a declarative way to install, upgrade and manage all the operators and their dependencies in our cluster.
Operator Metering: Operator metering is currently an alpha project. It records historical cluster usage and can generate usage reports showing usage breakdown by pod or namespace over arbitrary time periods.
Types of Operators
Currently there are three different types of operator we can build:
Helm based operators: Helm based operators allow us to use our existing Helm charts and build operators using them. Helm based operators are quite easy to build and are preferred to deploy a stateless application using operator pattern.
Ansible based Operator: Ansible based operator allows us to use our existing ansible playbooks and roles and build operators using them. There are also easy to build and generally preferred for stateless applications.
Go based operators: Go based operators are built to solve the most complex use cases and are generally preferred for stateful applications. In case of an golang based operator, we build the controller logic ourselves providing it with all our custom requirements. This type of operators is also relatively complex to build.
Building a Helm based operator
1. Let’s first install the operator sdk
go get -d github.com/operator-framework/operator-sdkcd $GOPATH/src/github.com/operator-framework/operator-sdkgit checkout mastermake depmake install
Now we will have the operator-sdk binary in the $GOPATH/bin folder.
2. Setup the project
For building a helm based operator we can use an existing Helm chart. We will be using the book-store Helm chart which deploys a simple python app and mongodb instances. This app allows us to perform crud operations via. rest endpoints.
Now we will use the operator-sdk to create our Helm based bookstore-operator project.
operator-sdk new bookstore-operator --api-version=velotio.com/v1alpha1 --kind=BookStore --type=helm --helm-chart=book-store--helm-chart-repo=https://akash-gautam.github.io/helmcharts/
In the above command, the bookstore-operator is the name of our operator/project. –kind is used to specify the kind of objects this operator will watch and –api-verison is used for versioning of this object. The operator sdk takes only this much information and creates the custom resource definition (crd) and also the custom resource (cr) of its type for us (remember we talked about high-level abstraction operator sdk provides). The above command bootstraps a project with below folder structure.
bookstore-operator/||- build/ # Contains the Dockerfile to build the operator image|- deploy/ # Contains the crd,cr and manifest files for deploying operator|- helm-charts/ # Contains the helm chart we used while creating the project|- watches.yaml # Specifies the resource the operator watches (maintains the state of)
We had discussed the operator-sdk automates setting up the operator projects and that is exactly what we can observe here. Under the build folder, we have the Dockerfile to build our operator image. Under deploy folder we have a crd folder containing both the crd and the cr. This folder also has operator.yaml file using which we will run the operator in our cluster, along with this we have manifest files for role, rolebinding and service account file to be used while deploying the operator. We have our book-store helm chart under helm-charts. In the watches.yaml file.
We can see that the bookstore-operator watches events related to BookStore kind objects and executes the helm chart specified.
If we take a look at the cr file under deploy/crds (velotio_v1alpha1_bookstore_cr.yaml) folder then we can see that it looks just like the values.yaml file of our book-store helm chart.
apiVersion: velotio.com/v1alpha1kind: BookStoremetadata:name: example-bookstorespec: # Default values copied from <project_dir>/helm-charts/book-store/values.yaml # Default values for book-store. # This is a YAML-formatted file. # Declare variables to be passed into your templates.replicaCount: 1image:app:repository: akash125/pyapptag: latestpullPolicy: IfNotPresentmongodb:repository: mongotag: latestpullPolicy: IfNotPresentservice:app:type: LoadBalancerport: 80targetPort: 3000mongodb:type: ClusterIPport: 27017targetPort: 27017resources: {} # We usually recommend not to specify default resources and to leave thisasaconscious # choice for the user. This also increases chances charts run on environments with little # resources, such asMinikube. Ifyoudowanttospecifyresources, uncomment the following # lines, adjust them asnecessary, and remove the curly braces after 'resources:'. # limits: # cpu: 100m # memory: 128Mi # requests: # cpu: 100m # memory: 128MinodeSelector: {}tolerations: []affinity: {}
In the case of Helm charts, we use the values.yaml file to pass the parameter to our Helm releases, Helm based operator converts all these configurable parameters into the spec of our custom resource. This allows us to express the values.yaml with a custom resource (CR) which, as a native Kubernetes object, enables the benefits of RBAC applied to it and an audit trail. Now when we want to update out deployed we can simply modify the CR and apply it, and the operator will ensure that the changes we made are reflected in our app.
For each object of `BookStore` kind the bookstore-operator will perform the following actions:
Create the bookstore app deployment if it doesn’t exists.
Create the bookstore app service if it doesn’t exists.
Create the mongodb deployment if it doesn’t exists.
Create the mongodb service if it doesn’t exists.
Ensure deployments and services match their desired configurations like the replica count, image tag, service port etc.
3. Build the Bookstore-operator Image
The Dockerfile for building the operator image is already in our build folder we need to run the below command from the root folder of our operator project to build the image.
As we have our operator image ready we can now go ahead and run it. The deployment file (operator.yaml under deploy folder) for the operator was created as a part of our project setup we just need to set the image for this deployment to the one we built in the previous step.
After updating the image in the operator.yaml we are ready to deploy the operator.
Note: The role created might have more permissions then actually required for the operator so it is always a good idea to review it and trim down the permissions in production setups.
Verify that the operator pod is in running state.
5. Deploy the Bookstore App
Now we have the bookstore-operator running in our cluster we just need to create the custom resource for deploying our bookstore app.
First, we can create bookstore cr we need to register its crd.
Since its early days Kubernetes was believed to be a great tool for managing stateless application but the managing stateful applications on Kubernetes was always considered difficult. Operators are a big leap towards managing stateful applications and other complex distributed, multi (poly) cloud workloads with the same ease that we manage the stateless applications. In this blog post, we learned the basics of Kubernetes operators and build a simple helm based operator. In the next installment of this blog series, we will build an Ansible based Kubernetes operator and then in the last blog we will build a full-fledged Golang based operator for managing stateful workloads.
Autoscaling, a key feature of Kubernetes, lets you improve the resource utilization of your cluster by automatically adjusting the application’s resources or replicas depending on the load at that time.
This blog talks about Pod Autoscaling in Kubernetes and how to set up and configure autoscalers to optimize the resource utilization of your application.
Horizontal Pod Autoscaling
What is the Horizontal Pod Autoscaler?
The Horizontal Pod Autoscaler (HPA) scales the number of pods of a replica-set/ deployment/ statefulset based on per-pod metrics received from resource metrics API (metrics.k8s.io) provided by metrics-server, the custom metrics API (custom.metrics.k8s.io), or the external metrics API (external.metrics.k8s.io).
Fig:- Horizontal Pod Autoscaling
Prerequisite
Verify that the metrics-server is already deployed and running using the command below, or deploy it using instructions here.
kubectl get deployment metrics-server -n kube-system
HPA using Multiple Resource Metrics
HPA fetches per-pod resource metrics (like CPU, memory) from the resource metrics API and calculates the current metric value based on the mean values of all targeted pods. It compares the current metric value with the target metric value specified in the HPA spec and produces a ratio used to scale the number of desired replicas.
A. Setup: Create a Deployment and HPA resource
In this blog post, I have used the config below to create a deployment of 3 replicas, with some memory load defined by “–vm-bytes”, “850M”.
Lets create an HPA resource for this deployment with multiple metric blocks defined. The HPA will consider each metric one-by-one and calculate the desired replica counts based on each of the metrics, and then select the one with the highest replica count.
We have defined the minimum number of replicas HPA can scale down to as 1 and the maximum number that it can scale up to as 10.
Target Average Utilization and Target Average Values implies that the HPA should scale the replicas up/down to keep the Current Metric Value equal or closest to Target Metric Value.
B. Understanding the HPA Algorithm
kubectl describe hpa autoscale-testerName: autoscale-testerNamespace: autoscale-tester...Metrics: ( current / target ) resource memory on pods: 894188202666m / 500Mi resource cpu on pods (asapercentageofrequest): 36% (361m) /50%Min replicas: 1Max replicas: 10Deployment pods: 3 current /6 desiredConditions: Type Status Reason Message----------------------- AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 6 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource ScalingLimited False DesiredWithinRange the desired count is within the acceptable rangeEvents: Type Reason Age From Message------------------------- Normal SuccessfulRescale 7s horizontal-pod-autoscaler New size: 6; reason: memory resource above target
HPA calculates pod utilization as total usage of all containers in the pod divided by total request. It looks at all containers individually and returns if container doesn’t have request.
The calculated Current Metric Value for memory, i,e., 894188202666m, is higher than the Target Average Value of 500Mi, so the replicas need to be scaled up.
The calculated Current Metric Value for CPU i.e., 36%, is lower than the Target Average Utilization of 50, so hence the replicas need to be scaled down.
Replicas are calculated based on both metrics and the highest replica count selected. So, the replicas are scaled up to 6 in this case.
HPA using Custom metrics
We will use the prometheus-adapter resource to expose custom application metrics to custom.metrics.k8s.io/v1beta1, which are retrieved by HPA. By defining our own metrics through the adapter’s configuration, we can let HPA perform scaling based on our custom metrics.
A.Setup: Install Prometheus Adapter
Create prometheus-adapter.yaml with the content below:
kubectl describe hpa autoscale-testerName: autoscale-testerNamespace: autoscale-tester...Metrics: ( current / target )"packets_in" on pods: 18666m /50Min replicas: 1Max replicas: 10Deployment pods: 3 current /3 desiredConditions: Type Status Reason Message----------------------- AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 2 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric packets_in ScalingLimited False DesiredWithinRange the desired count is within the acceptable rangeEvents: Type Reason Age From Message------------------------- Normal SuccessfulRescale 2s horizontal-pod-autoscaler New size: 2; reason: All metrics below target Normal SuccessfulRescale 2m51s horizontal-pod-autoscaler New size: 1; reason: All metrics below target kubectl describe hpa autoscale-testerName: autoscale-testerNamespace: autoscale-tester...Metrics: ( current / target )"packets_in" on pods: 18666m /50Min replicas: 1Max replicas: 10Deployment pods: 3 current /3 desiredConditions: Type Status Reason Message----------------------- AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 2 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric packets_in ScalingLimited False DesiredWithinRange the desired count is within the acceptable rangeEvents: Type Reason Age From Message------------------------- Normal SuccessfulRescale 2s horizontal-pod-autoscaler New size: 2; reason: All metrics below target Normal SuccessfulRescale 2m51s horizontal-pod-autoscaler New size: 1; reason: All metrics below target
Here, the current calculated metric value is 18666m. The m represents milli-units. So, for example, 18666m means 18.666 which is what we expect ((33 + 11 + 10 )/3 = 18.666). Since it’s less than the target average value (i.e., 50), the HPA scales down the replicas to make the Current Metric Value : Target Metric Value ratio closest to 1. Hence, replicas are scaled down to 2 and later to 1.
Fig:- container_network_receive_packets_total
Fig:- Ratio to Target value
Vertical Pod Autoscaling
What is Vertical Pod Autoscaler?
Vertical Pod autoscaling (VPA) ensures that a container’s resources are not under- or over-utilized. It recommends optimized CPU and memory requests/limits values, and can also automatically update them for you so that the cluster resources are efficiently used.
Fig:- Vertical Pod Autoscaling
Architecture
VPA consists of 3 components:
VPA admission controller Once you deploy and enable the Vertical Pod Autoscaler in your cluster, every pod submitted to the cluster goes through this webhook, which checks whether a VPA object is referencing it.
VPA recommender The recommender pulls the current and past resource consumption (CPU and memory) data for each container from metrics-server running in the cluster and provides optimal resource recommendations based on it, so that a container uses only what it needs.
VPA updater The updater checks at regular intervals if a pod is running within the recommended range. Otherwise, it accepts it for update, and the pod is evicted by the VPA updater to apply resource recommendation.
Installation
If you are on Google Cloud Platform, you can simply enable vertical-pod-autoscaling:
Verify that the Vertical Pod Autoscaler pods are up and running:
kubectl get po -n kube-systemNAMEREADYSTATUSRESTARTSAGEvpa-admission-controller-68c748777d-ppspd 1/1 Running 0 7svpa-recommender-6fc8c67d85-gljpl 1/1 Running 0 8svpa-updater-786b96955c-bgp9d 1/1 Running 0 8skubectl get crdverticalpodautoscalers.autoscaling.k8s.io
VPA using Resource Metrics
A. Setup: Create a Deployment and VPA resource
Use the same deployment config to create a new deployment with “–vm-bytes”, “850M”. Then create a VPA resource in Recommendation Mode with updateMode : Off
minAllowed is an optional parameter that specifies the minimum CPU request and memory request allowed for the container.
maxAllowed is an optional parameter that specifies the maximum CPU request and memory request allowed for the container.
B. Check the Pod’s Resource Utilization
Check the resource utilization of the pods. Below, you can see only ~50 Mi memory is being used out of 1000Mi and only ~30m CPU out of 1000m. This clearly indicates that the pod resources are underutilized.
Target: The recommended CPU request and memory request for the container that will be applied to the pod by VPA.
Uncapped Target: The recommended CPU request and memory request for the container if you didn’t configure upper/lower limits in the VPA definition. These values will not be applied to the pod. They’re used only as a status indication.
Lower Bound: The minimum recommended CPU request and memory request for the container. There is a –pod-recommendation-min-memory-mb flag that determines the minimum amount of memory the recommender will set—it defaults to 250MiB.
Upper Bound: The maximum recommended CPU request and memory request for the container. It helps the VPA updater avoid eviction of pods that are close to the recommended target values. Eventually, the Upper Bound is expected to reach close to target recommendation.
Now, if you check the logs of vpa-updater, you can see it’s not processing VPA objects as the Update Mode is set as Off.
kubectl logs -f vpa-updater-675d47464b-k7xbx1 updater.go:135] skipping VPA object autoscale-tester-recommender because its mode is not "Recreate" or "Auto"1 updater.go:151] no VPA objects to process
Let’s change the VPA updateMode to “Auto” to see the processing.
As soon as you do that, you can see vpa-updater has started processing objects, and it’s terminating all 3 pods.
kubectl logs -f vpa-updater-675d47464b-k7xbx1 update_priority_calculator.go:147] pod accepted for update autoscale-tester/autoscale-tester-5d6b48d64f-8zgb9 with priority 11 update_priority_calculator.go:147] pod accepted for update autoscale-tester/autoscale-tester-5d6b48d64f-npts4 with priority 11 update_priority_calculator.go:147] pod accepted for update autoscale-tester/autoscale-tester-5d6b48d64f-vctx5 with priority 11 updater.go:193] evicting pod autoscale-tester-5d6b48d64f-8zgb91 event.go:281] Event(v1.ObjectReference{Kind:"Pod", Namespace:"autoscale-tester", Name:"autoscale-tester-5d6b48d64f-8zgb9", UID:"ed8c54c7-a87a-4c39-a000-0e74245f18c6", APIVersion:"v1", ResourceVersion:"378376", FieldPath:""}): type: 'Normal'reason: 'EvictedByVPA' Pod was evicted by VPA Updater to apply resource recommendation.
You can also check the logs of vpa-admission-controller:
kubectl logs -f vpa-admission-controller-bbf4f4cc7-cb6pbSending patches: [{add /metadata/annotations map[]} {add /spec/containers/0/resources/requests/cpu 500m} {add /spec/containers/0/resources/requests/memory 500Mi} {add /spec/containers/0/resources/limits/cpu 500m} {add /spec/containers/0/resources/limits/memory 500Mi} {add /metadata/annotations/vpaUpdates Pod resources updated by autoscale-tester-recommender: container 0: cpu request, memory request, cpu limit, memory limit} {add /metadata/annotations/vpaObservedContainers autoscale-tester}]
NOTE: Ensure that you have more than 1 running replicas. Otherwise, the pods won’t be restarted, and vpa-updater will give you this warning:
1 pods_eviction_restriction.go:209] too few replicas for ReplicaSet autoscale-tester/autoscale-tester1-7698974f6. Found 1 live pods
Now, describe the new pods created and check that the resources match the Target recommendations:
The Target Recommendation can not go below the minAllowed defined in the VPA spec.
Fig:- Prometheus: Memory Usage Ratio
E. Stress Loading Pods
Let’s recreate the deployment with memory request and limit set to 2000Mi and “–vm-bytes”, “500M”.
Gradually stress load one of these pods to increase its memory utilization. You can login to the pod and run stress –vm 1 –vm-bytes 1400M –timeout 120000s.
Limits v/s Request VPA always works with the requests defined for a container and not the limits. So, the VPA recommendations are also applied to the container requests, and it maintains a limit to request ratio specified for all containers.
For example, if the initial container configuration defines a 100m Memory Request and 300m Memory Limit, then when the VPA target recommendation is 150m Memory, the container Memory Request will be updated to 150m and Memory Limit to 450m.
Selective Container Scaling
If you have a pod with multiple containers and you want to opt-out some of them, you can use the “Off” mode to turn off recommendations for a container.
You can also set containerName: “*” to include all containers.
Both the Horizontal Pod Autoscaler and the Vertical Pod Autoscaler serve different purposes and one can be more useful than the other depending on your application’s requirement.
The HPA can be useful when, for example, your application is serving a large number of lightweight (low resource-consuming) requests. In that case, scaling number of replicas can distribute the workload on each of the pod. The VPA, on the other hand, can be useful when your application serves heavyweight requests, which requires higher resources.
Nightwatch.js is a test automation framework on web applications, developed in Node.js which uses W3C WebDriver API (formerly Selenium WebDriver). It is a complete End-to-End testing solution which aims to simplify writing automated tests and setting up Continuous Integration. Nightwatch works by communicating over a restful HTTP API with a WebDriver server (such as ChromeDriver or Selenium Server). The latest version available in market is 1.0.
Why Use Nightwatch JS Over Any Other Automation Tool?
Selenium is demanded for developing automation framework since it supports various programming languages, provides cross-browser testing and also used in both web application and mobile application testing.
But Nightwatch, built on Node.js, exclusively uses JavaScript as the programming language for end-to-end testing which has the listed advantages –
Lightweight framework
Robust configuration
Integrates with cloud servers like SauceLabs and Browserstack for web and mobile testing with JavaScript, Appium
Allows configuration with Cucumber to build a strong BDD (Behaviour Driven Development) setup
High performance of the automation execution
Improves test structuring
Minimum usage and less Maintenance of code
Installation and Configuration of Nightwatch Framework
For configuring Nightwatch framework, all needed are the following in your system –
Download latest Node.js
Install npm
$ npm install
Package.json file for the test settings and dependencies
$ npm init
Install nightwatch and save as dev dependency
$ npm install nightwatch --save-dev
Install chromedriver/geckodriver and save as dev dependency for running the execution on the required browser
Create a nightwatch.conf.js for webdriver and test settings with nightwatch
constchromedriver=require('chromedriver');module.exports= { src_folders : ["tests"], //tests is a folder in workspace which has the step definitions test_settings: { default: { webdriver: { start_process: true, server_path: chromedriver.path, port: 4444, cli_args: ['--port=4444'] }, desiredCapabilities: { browserName: 'chrome' } } }};
Using Nightwatch – Writing and Running Tests
We create a JavaScript file named demo.js for running a test through nightwatch with the command
$ npm test
//demo.js is a JS file under tests foldermodule.exports= {'step one: navigate to google' : function (browser) { //step one browser .url('https://www.google.com') .waitForElementVisible('body', 1000) .setValue('input[type=text]', 'nightwatch') .waitForElementVisible('input[name=btnK]', 1000) },'step two: click input' : function (browser) { //step two browser .click('input[name=btnK]') .pause(1000) .assert.containsText('#main', 'Night Watch') .end(); //to close the browser session after all the steps }
This command on running picks the value “nightwatch” from “test” key in package.json file which hits the nightwatch api to trigger the URL in chromedriver.
There can be one or more steps in demo.js(step definition js) file as per requirement or test cases.
Also, it is a good practice to maintain a separate .js file for page objects which consists of the locator strategy and selectors of the UI web elements.
Cucumber can be configured in the nightwatch framework to help maintaining the test scenarios in its .feature files. We create a file cucumber.conf.js in the root folder which has the setup of starting, creating and closing webdriver sessions.
Then we create a feature file which has the test scenarios in Given, When, Then format.
Feature: Google SearchScenario: Searching Google Given I open Google's search page Then the title is "Google" And the Google search form exists
For Cucumber to be able to understand and execute the feature file we need to create matching step definitions for every feature step we use in our feature file. Create a step definition file under tests folder called google.js. Step definitions which uses Nightwatch client should return the result of api call as it returns a Promise. For example,
const { client } =require('nightwatch-api');const { Given, Then, When } =require('cucumber');Given(/^I open Google's search page$/, () => {return client.url('http://google.com').waitForElementVisible('body', 1000);});Then(/^the title is "([^"]*)"$/, title=> {return client.assert.title(title);});Then(/^the Google search form exists$/, () => {return client.assert.visible('input[name="q"]');});
$ npm run e2e-test
Executing Individual Feature Files or Scenarios
Single feature file
npm run e2e-test -- features/file1.feature
Multiple feature files
npm run e2e-test -- features/file1.feature features/file2.feature
Scenario by its line number
npm run e2e-test -- features/my_feature.feature:3
Feature directory
npm run e2e-test -- features/dir
Scenario by its name matching a regular expression
npm run e2e-test ----name "topic 1"
Feature and Scenario Tags
Cucumber allows to add tags to features or scenarios and we can selectively run a scenario using those tags. The tags can be used with conditional operators also, depending on the requirement.
Single tag
# google.feature@googleFeature: Google Search@searchScenario: Searching Google Given I open Google's search pageThen the title is "Google"And the Google search form exists
npm run e2e-test ----tags @google
Multiple tags
npm run e2e-test ----tags "@google or @duckduckgo"npm run e2e-test ----tags "(@google or @duckduckgo) and @search"
To skip tags
npm run e2e-test ----tags "not @google"npm run e2e-test ----tags "not(@google or @duckduckgo)"
Custom Reporters in Nightwatch and Cucumber Framework
Reporting is again an advantage provided by Cucumber which generates a report of test results at the end of the execution and it provides an immediate visual clue of a possible problem and will simplify the debugging process. HTML reports are best suited and easy to understand due to its format. To generate the same, we will add cucumber-html-reporter as a dependency in our nightwatch.conf.js file.
Cucumber-html-reporter in node_modules manages the creation of reports and generates in the output location after the test execution. Screenshot feature can enabled by adding the below code snippet in nightwatch.conf.js
The Cucumber configuration file can be extended with the handling of screenshots and attaching them to the report. Now – It also enables generating HTML test report at the end of the execution. It is generated based on Cucumber built-can be configured here in JSON report using different templates. We use a setTimeout() block in our cucumber.conf.js to run the generation after Cucumber finishes with the creation of json report.
In package.json file, we have added the JSON formatter to create a JSON report and it is used by cucumber-html-reporter for the same. We use mkdirp to make sure report folder exists before running the test.
When the test run completes, the HTML report is displayed in a new browser tab in the format given below
Conclusion
Nightwatch-Cucumber is a great module for linking the accessibility of Cucumber.js with the robust testing framework of Nightwatch.js. Together they can not only provide easily readable documentation of test suite, but also highly configurable automated user tests, all while keeping everything in JavaScript.