Tag: continuous delivery

A Guide to End-to-End API Test Automation with Postman and GitHub Actions
Objective
- The blog intends to provide a step-by-step guide on how to automate API testing using Postman. It also demonstrates how we can create a pipeline for periodically running the test suite.
- Further, it explains how the report can be stored in a central S3 bucket, finally sending the status of the execution back to a designated slack Channel, informing stakeholders about the status, and enabling them to obtain detailed information about the quality of the API.
Introduction to Postman
- To speed up the API testing process and improve the accuracy of our APIs, we are going to automate the API functional tests using Postman.
- Postman is a great tool when trying to dissect RESTful APIs.
- It offers a sleek user interface to create our functional tests to validate our API’s functionality.
- Furthermore, the collection of tests will be integrated with GitHub Actions to set up a CI/CD platform that will be used to automate this API testing workflow.
Getting started with Postman

Setting up the environment
- Click on the “New” button on the top left corner.
- Select “Environment” as the building block.
- Give the desired name to the environment file.
Create a collection
- Click on the “New” button on the top left corner.
- Select “Collection” as the building block.
- Give the desired name to the Collection.
‍Adding requests to the collection
- Configure the Requests under test in folders as per requirement.
- Enter the API endpoint in the URL field.
- Set the Auth credentials necessary to run the endpoint.
- Set the header values, if required.
- Enter the request body, if applicable.
- Send the request by clicking on the “Send” button.
- Verify the response status and response body.
Creating TESTS
- Click on the “Tests” tab.
- Write the test scripts in JavaScript using the Postman test API.
- Run the tests by clicking on the “Send” button and validate the execution of the tests written.
- Alternatively, the prebuilt snippets given by Postman can also be used to create the tests.
- In case some test data needs to be created, the “Pre-request Script“ tab can be used.
Running the Collection
- Click on the ellipses beside the collection created.
- Select the environment created in Step 1.
- Click on the “Run Collection” button.
- Alternatively, the collection and the env file can be exported and also run via the Newman command.
Collaboration

The original collection and the environment file can be exported and shared with others by clicking on the “Export” button. These collections and environments can be version controlled using a system such as Git.
- While working in a team, team members raise PRs for their changes against the original collection and env via forking. Create a fork.
- Make necessary changes to the collection and click on Create a Pull request.
- Validate the changes and approve and merge them to the main collection.
‍

Integrating with CI/CD

‍

Creating a pipeline with GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline.

You can create workflows that build and test every pull request to your repository, or deploy merged pull requests to production. To create a pipeline, follow the below steps
- Create a .yml file inside folder .github/workflows at root level.
- The same can also be created via GitHub.
- Configure the necessary actions/steps for the pipeline.
Workflow File
- Add a trigger to run the workflow.
- The schedule in the below code snippet is a GitHub Actions event that triggers the workflow at a specific time interval using a CRON expression.
- The push and pull_request events denote the Actions event that triggers the workflow for each push and pull request on the develop branch.
- The workflow_dispatch tag denotes the ability to run the workflow manually, too, from GitHub Actions.
- Create a job to run the Postman Collection.
- Check out the code from the current repository. Also, create a directory to store the results.
- Install Nodejs.
- Install Newman and necessary dependencies
- Running the collection
- Upload Newman report into the directory
Generating Allure report and hosting the report onto s3.
- Along with the default report that Newman provides, Allure reporting can also be used in order to get a dashboard of the result.
- To generate the Allure report, install the Allure dependencies given in the installation step above.
- Once, that is done, add below code to your .yml file.
- Create a bucket in s3, which you will be using for storing the reports
- Create an iam role for the bucket.
- The below code snipper user aws-actions/configure-aws-credentials@v1 action to configure your AWS.
- Credentials Allure generates 2 separate folders eventually combining them to create a dashboard.
Use the code snippet in the deploy section to upload the contents of the folder onto your s3 bucket.
- Once done, you should be able to see the Allure dashboard hosted on the Static Website URL for your bucket.
Send Slack notification with the Status of the job
- When a job is executed in a CI/CD pipeline, it’s important to keep the team members informed about the status of the job.
- Below GitHub Actions step sends a notification to a Slack channel with the status of the job.
- It uses the “notify-slack-action” GitHub Action, which is defined in the “ravsamhq/notify-slack-action” repository.
- The “if: always()” condition indicates that this step should always be executed, regardless of whether the previous steps in the workflow succeeded or failed.
May 5, 2023

Continuous Integration & Delivery (CI/CD) for Kubernetes Using CircleCI & Helm

Introduction

Kubernetes is getting adopted rapidly across the software industry and is becoming the most preferred option for deploying and managing containerized applications. Once we have a fully functional Kubernetes cluster we need to have an automated process to deploy our applications on it. In this blog post, we will create a fully automated “commit to deploy” pipeline for Kubernetes. We will use CircleCI & helm for it.

What is CircleCI?

CircleCI is a fully managed saas offering which allows us to build, test or deploy our code on every check in. For getting started with circle we need to log into their web console with our GitHub or bitbucket credentials then add a project for the repository we want to build and then add the CircleCI config file to our repository. The CircleCI config file is a yaml file which lists the steps we want to execute on every time code is pushed to that repository.

Some salient features of CircleCI is:

Little or no operational overhead as the infrastructure is managed completely by CircleCI.
User authentication is done via GitHub or bitbucket so user management is quite simple.
It automatically notifies the build status on the github/bitbucket email ids of the users who are following the project on CircleCI.
The UI is quite simple and gives a holistic view of builds.
Can be integrated with Slack, hipchat, jira, etc.

What is Helm?

Helm is chart manager where chart refers to package of Kubernetes resources. Helm allows us to bundle related Kubernetes objects into charts and treat them as a single unit of deployment referred to as release. For example, you have an application app1 which you want to run on Kubernetes. For this app1 you create multiple Kubernetes resources like deployment, service, ingress, horizontal pod scaler, etc. Now while deploying the application you need to create all the Kubernetes resources separately by applying their manifest files. What helm does is it allows us to group all those files into one chart (Helm chart) and then we just need to deploy the chart. This also makes deleting and upgrading the resources quite simple.

Some other benefits of Helm is:

It makes the deployment highly configurable. Thus just by changing the parameters, we can use the same chart for deploying on multiple environments like stag/prod or multiple cloud providers.
We can rollback to a previous release with a single helm command.
It makes managing and sharing Kubernetes specific application much simpler.

Note: Helm is composed of two components one is helm client and the other one is tiller server. Tiller is the component which runs inside the cluster as deployment and serves the requests made by helm client. Tiller has potential security vulnerabilities thus we will use tillerless helm in our pipeline which runs tiller only when we need it.

Building the Pipeline

Overview:

We will create the pipeline for a Golang application. The pipeline will first build the binary, create a docker image from it, push the image to ECR, then deploy it on the Kubernetes cluster using its helm chart.

We will use a simple app which just exposes a `hello` endpoint and returns the hello world message:

package main
import (
	"encoding/json"
	"net/http"
	"log"
	"github.com/gorilla/mux"
)
type Message struct {
	Msg string
}
func helloWorldJSON(w http.ResponseWriter, r *http.Request) {
	m := Message{"Hello World"}
	response, _ := json.Marshal(m)
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	w.Write(response)
}
func main() {
	r := mux.NewRouter()
	r.HandleFunc("/hello", helloWorldJSON).Methods("GET")
	if err := http.ListenAndServe(":8080", r); err != nil {
		log.Fatal(err)
	}
}

package main

import (
	"encoding/json"
	"net/http"
	"log"
	"github.com/gorilla/mux"
)

type Message struct {
	Msg string
}

func helloWorldJSON(w http.ResponseWriter, r *http.Request) {
	m := Message{"Hello World"}
	response, _ := json.Marshal(m)
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	w.Write(response)
}
func main() {
	r := mux.NewRouter()
	r.HandleFunc("/hello", helloWorldJSON).Methods("GET")
	if err := http.ListenAndServe(":8080", r); err != nil {
		log.Fatal(err)
	}
}

We will create a docker image for hello app using the following Dockerfile:

FROM centos/systemd
MAINTAINER "Akash Gautam" <akash.gautam@velotio.com>
COPY hello-app  /
ENTRYPOINT ["/hello-app"]

FROM centos/systemd

MAINTAINER "Akash Gautam" <akash.gautam@velotio.com>

COPY hello-app  /

ENTRYPOINT ["/hello-app"]

Creating Helm Chart:

Now we need to create the helm chart for hello app.

First, we create the Kubernetes manifest files. We will create a deployment and a service file:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: helloapp
spec:
  replicas: 1
  strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 1
  template:
    metadata:
      labels:
        app: helloapp
        env: {{ .Values.labels.env }}
        cluster: {{ .Values.labels.cluster }}
    spec:
      containers:
      - name: helloapp
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        imagePullPolicy: {{ .Values.image.imagePullPolicy }}
        readinessProbe:
          httpGet:
            path: /hello
            port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: helloapp
spec:
  replicas: 1
  strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 1
  template:
    metadata:
      labels:
        app: helloapp
        env: {{ .Values.labels.env }}
        cluster: {{ .Values.labels.cluster }}
    spec:
      containers:
      - name: helloapp
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        imagePullPolicy: {{ .Values.image.imagePullPolicy }}
        readinessProbe:
          httpGet:
            path: /hello
            port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1

apiVersion: v1
kind: Service
metadata:
  name: helloapp
spec:
  type: {{ .Values.service.type }}
  ports:
  - name: helloapp
    port: {{ .Values.service.port }}
    protocol: TCP
    targetPort: {{ .Values.service.targetPort }}
  selector:
    app: helloapp

apiVersion: v1
kind: Service
metadata:
  name: helloapp
spec:
  type: {{ .Values.service.type }}
  ports:
  - name: helloapp
    port: {{ .Values.service.port }}
    protocol: TCP
    targetPort: {{ .Values.service.targetPort }}
  selector:
    app: helloapp

In the above file, you must have noticed that we have used .Values object. All the values that we specify in the values.yaml file in our helm chart can be accessed using the .Values object inside the template.

Let’s create the helm chart now:

helm create helloapp

helm create helloapp

Above command will create a chart helm chart folder structure for us.

helloapp/
|
|- .helmignore # Contains patterns to ignore when packaging Helm charts.
|
|- Chart.yaml # Information about your chart
|
|- values.yaml # The default values for your templates
|
|- charts/ # Charts that this chart depends on
|
|- templates/ # The template files

helloapp/
|
|- .helmignore # Contains patterns to ignore when packaging Helm charts.
|
|- Chart.yaml # Information about your chart
|
|- values.yaml # The default values for your templates
|
|- charts/ # Charts that this chart depends on
|
|- templates/ # The template files

We can remove the charts/ folder inside our helloapp chart as our chart won’t have any sub-charts. Now we need to move our Kubernetes manifest files to the template folder and update our values.yaml and Chart.yaml

Our values.yaml looks like:

image:
  tag: 0.0.1
  repository: 123456789870.dkr.ecr.us-east-1.amazonaws.com/helloapp
  imagePullPolicy: Always
labels:
  env: "staging"
  cluster: "eks-cluster-blog"
service:
  port: 80
  targetPort: 8080
  type: LoadBalancer

image:
  tag: 0.0.1
  repository: 123456789870.dkr.ecr.us-east-1.amazonaws.com/helloapp
  imagePullPolicy: Always

labels:
  env: "staging"
  cluster: "eks-cluster-blog"

service:
  port: 80
  targetPort: 8080
  type: LoadBalancer

This allows us to make our deployment more configurable. For example, here we have set our service type as LoadBalancer in values.yaml but if we want to change it to nodePort we just need to set is as NodePort while installing the chart (–set service.type=NodePort). Similarly, we have set the image pull policy as Always which is fine for development/staging environment but when we deploy to production we may want to set is as ifNotPresent. In our chart, we need to identify the parameters/values which may change from one environment to another and make them configurable. This allows us to be flexible with our deployment and reuse the same chart

Finally, we need to update Chart.yaml file. This file mostly contains metadata about the chart like the name, version, maintainer, etc, where name & version are two mandatory fields for Chart.yaml.

version: 1.0.0
appVersion: 0.0.1
name: helloapp
description: Helm chart for helloapp
source:
  - https://github.com/akash-gautam/helloapp

version: 1.0.0
appVersion: 0.0.1
name: helloapp
description: Helm chart for helloapp
source:
  - https://github.com/akash-gautam/helloapp

Now our Helm chart is ready we can start with the pipeline. We need to create a folder named .circleci in the root folder of our repository and create a file named config.yml in it. In our config.yml we have defined two jobs one is build&pushImage and deploy.

Configure the pipeline:

build&pushImage:
    working_directory: /go/src/hello-app (1)
    docker:
      - image: circleci/golang:1.10 (2)
    steps:
      - checkout (3)
      - run: (4)
          name: build the binary
          command: go build -o hello-app
      - setup_remote_docker: (5)
          docker_layer_caching: true
      - run: (6)
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_BUILD_NUM' >> $BASH_ENV
      - run: (7)
          name: Build the docker image
          command: docker build . -t ${CIRCLE_PROJECT_REPONAME}:$TAG
      - run: (8)
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run: (9)
          name: Login to ECR
          command: $(aws ecr get-login --region $AWS_REGION | sed -e 's/-e none//g')
      - run: (10)
          name: Tag the image with ECR repo name 
          command: docker tag ${CIRCLE_PROJECT_REPONAME}:$TAG ${HELLOAPP_ECR_REPO}:$TAG    
      - run: (11)
          name: Push the image the ECR repo
          command: docker push ${HELLOAPP_ECR_REPO}:$TAG

build&pushImage:
    working_directory: /go/src/hello-app (1)
    docker:
      - image: circleci/golang:1.10 (2)
    steps:
      - checkout (3)
      - run: (4)
          name: build the binary
          command: go build -o hello-app
      - setup_remote_docker: (5)
          docker_layer_caching: true
      - run: (6)
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_BUILD_NUM' >> $BASH_ENV
      - run: (7)
          name: Build the docker image
          command: docker build . -t ${CIRCLE_PROJECT_REPONAME}:$TAG
      - run: (8)
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run: (9)
          name: Login to ECR
          command: $(aws ecr get-login --region $AWS_REGION | sed -e 's/-e none//g')
      - run: (10)
          name: Tag the image with ECR repo name 
          command: docker tag ${CIRCLE_PROJECT_REPONAME}:$TAG ${HELLOAPP_ECR_REPO}:$TAG    
      - run: (11)
          name: Push the image the ECR repo
          command: docker push ${HELLOAPP_ECR_REPO}:$TAG

We set the working directory for our job, we are setting it on the gopath so that we don’t need to do anything additional.
We set the docker image inside which we want the job to run, as our app is built using golang we are using the image which already has golang installed in it.
This step checks out our repository in the working directory
In this step, we build the binary
Here we setup docker with the help of setup_remote_docker key provided by CircleCI.
In this step we create the tag we will be using while building the image, we use the app version available in the VERSION file and append the $CIRCLE_BUILD_NUM value to it, separated by a dash (`-`).
Here we build the image and tag.
Installing AWS CLI to interact with the ECR later.
Here we log into ECR
We tag the image build in step 7 with the ECR repository name.
Finally, we push the image to ECR.

Now we will deploy our helm charts. For this, we have a separate job deploy.

deploy:
    docker: (1)
        - image: circleci/golang:1.10
    steps: (2)
      - checkout
      - run: (3)
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run: (4)
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_PREVIOUS_BUILD_NUM' >> $BASH_ENV
      - run: (5)
          name: Install and confgure kubectl
          command: sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && sudo chmod +x /usr/local/bin/kubectl  
      - run: (6)
          name: Install and confgure kubectl aws-iam-authenticator
          command: curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator && sudo chmod +x ./aws-iam-authenticator && sudo cp ./aws-iam-authenticator /bin/aws-iam-authenticator
       - run: (7)
          name: Install latest awscli version
          command: sudo apt install unzip && curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip &&./awscli-bundle/install -b ~/bin/aws
      - run: (8)
          name: Get the kubeconfig file 
          command: export KUBECONFIG=$HOME/.kube/kubeconfig && /home/circleci/bin/aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
      - run: (9)
          name: Install and configuire helm
          command: sudo curl -L https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz | tar xz && sudo mv linux-amd64/helm /bin/helm && sudo rm -rf linux-amd64
      - run: (10)
          name: Initialize helm
          command:  helm init --client-only --kubeconfig=$HOME/.kube/kubeconfig
      - run: (11)
          name: Install tiller plugin
          command: helm plugin install https://github.com/rimusz/helm-tiller --kubeconfig=$HOME/.kube/kubeconfig        
      - run: (12)
          name: Release helloapp using helm chart 
          command: bash scripts/release-helloapp.sh $TAG

deploy:
    docker: (1)
        - image: circleci/golang:1.10
    steps: (2)
      - checkout
      - run: (3)
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run: (4)
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_PREVIOUS_BUILD_NUM' >> $BASH_ENV
      - run: (5)
          name: Install and confgure kubectl
          command: sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && sudo chmod +x /usr/local/bin/kubectl  
      - run: (6)
          name: Install and confgure kubectl aws-iam-authenticator
          command: curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator && sudo chmod +x ./aws-iam-authenticator && sudo cp ./aws-iam-authenticator /bin/aws-iam-authenticator
       - run: (7)
          name: Install latest awscli version
          command: sudo apt install unzip && curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip &&./awscli-bundle/install -b ~/bin/aws
      - run: (8)
          name: Get the kubeconfig file 
          command: export KUBECONFIG=$HOME/.kube/kubeconfig && /home/circleci/bin/aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
      - run: (9)
          name: Install and configuire helm
          command: sudo curl -L https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz | tar xz && sudo mv linux-amd64/helm /bin/helm && sudo rm -rf linux-amd64
      - run: (10)
          name: Initialize helm
          command:  helm init --client-only --kubeconfig=$HOME/.kube/kubeconfig
      - run: (11)
          name: Install tiller plugin
          command: helm plugin install https://github.com/rimusz/helm-tiller --kubeconfig=$HOME/.kube/kubeconfig        
      - run: (12)
          name: Release helloapp using helm chart 
          command: bash scripts/release-helloapp.sh $TAG

Set the docker image inside which we want to execute the job.
Check out the code using `checkout` key
Install AWS CLI.
Setting the value of tag just like we did in case of build&pushImage job. Note that here we are using CIRCLE_PREVIOUS_BUILD_NUM variable which gives us the build number of build&pushImage job and ensures that the tag values are the same.
Download kubectl and making it executable.
Installing aws-iam-authenticator this is required because my k8s cluster is on EKS.
Here we install the latest version of AWS CLI, EKS is a relatively newer service from AWS and older versions of AWS CLI doesn’t have it.
Here we fetch the kubeconfig file. This step will vary depending upon where the k8s cluster has been set up. As my cluster is on EKS am getting the kubeconfig file via. AWS CLI similarly if your cluster in on GKE then you need to configure gcloud and use the command `gcloud container clusters get-credentials <cluster-name> –zone=<zone-name>`. We can also have the kubeconfig file on some other secure storage system and fetch it from there.</zone-name></cluster-name>
Download Helm and make it executable
Initializing helm, note that we are initializing helm in client only mode so that it doesn’t start the tiller server.
Download the tillerless helm plugin
Execute the release-helloapp.sh shell script and pass it TAG value from step 4.

In the release-helloapp.sh script we first start tiller, after this, we check if the release is already present or not if it is present then we upgrade otherwise we make a new release. Here we override the value of tag for the image present in the chart by setting it to the tag of the newly built image, finally, we stop the tiller server.

#!/bin/bash
TAG=$1
echo "start tiller"
export KUBECONFIG=$HOME/.kube/kubeconfig
helm tiller start-ci
export HELM_HOST=127.0.0.1:44134
result=$(eval helm ls | grep helloapp) 
if [ $? -ne "0" ]; then 
   helm install --timeout 180 --name helloapp --set image.tag=$TAG charts/helloapp
else 
   helm upgrade --timeout 180 helloapp --set image.tag=$TAG charts/helloapp
fi
echo "stop tiller"
helm tiller stop

#!/bin/bash
TAG=$1
echo "start tiller"
export KUBECONFIG=$HOME/.kube/kubeconfig
helm tiller start-ci
export HELM_HOST=127.0.0.1:44134
result=$(eval helm ls | grep helloapp) 
if [ $? -ne "0" ]; then 
   helm install --timeout 180 --name helloapp --set image.tag=$TAG charts/helloapp
else 
   helm upgrade --timeout 180 helloapp --set image.tag=$TAG charts/helloapp
fi
echo "stop tiller"
helm tiller stop

The complete CircleCI config.yml file looks like:

version: 2
jobs:
  build&pushImage:
    working_directory: /go/src/hello-app
    docker:
      - image: circleci/golang:1.10
    steps:
      - checkout
      - run:
          name: build the binary
          command: go build -o hello-app
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_BUILD_NUM' >> $BASH_ENV
      - run:
          name: Build the docker image
          command: docker build . -t ${CIRCLE_PROJECT_REPONAME}:$TAG
      - run:
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run:
          name: Login to ECR
          command: $(aws ecr get-login --region $AWS_REGION | sed -e 's/-e none//g')
      - run: 
          name: Tag the image with ECR repo name 
          command: docker tag ${CIRCLE_PROJECT_REPONAME}:$TAG ${HELLOAPP_ECR_REPO}:$TAG    
      - run: 
          name: Push the image the ECR repo
          command: docker push ${HELLOAPP_ECR_REPO}:$TAG
  deploy:
    docker:
        - image: circleci/golang:1.10
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - checkout
      - run:
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run:
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_PREVIOUS_BUILD_NUM' >> $BASH_ENV
      - run:
          name: Install and confgure kubectl
          command: sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && sudo chmod +x /usr/local/bin/kubectl  
      - run:
          name: Install and confgure kubectl aws-iam-authenticator
          command: curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator && sudo chmod +x ./aws-iam-authenticator && sudo cp ./aws-iam-authenticator /bin/aws-iam-authenticator
       - run:
          name: Install latest awscli version
          command: sudo apt install unzip && curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip &&./awscli-bundle/install -b ~/bin/aws
      - run:
          name: Get the kubeconfig file 
          command: export KUBECONFIG=$HOME/.kube/kubeconfig && /home/circleci/bin/aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
      - run:
          name: Install and configuire helm
          command: sudo curl -L https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz | tar xz && sudo mv linux-amd64/helm /bin/helm && sudo rm -rf linux-amd64
      - run:
          name: Initialize helm
          command:  helm init --client-only --kubeconfig=$HOME/.kube/kubeconfig
      - run:
          name: Install tiller plugin
          command: helm plugin install https://github.com/rimusz/helm-tiller --kubeconfig=$HOME/.kube/kubeconfig        
      - run:
          name: Release helloapp using helm chart 
          command: bash scripts/release-helloapp.sh $TAG
workflows:
  version: 2
  primary:
    jobs:
      - build&pushImage
      - deploy:
          requires:
            - build&pushImage

version: 2

jobs:
  build&pushImage:
    working_directory: /go/src/hello-app
    docker:
      - image: circleci/golang:1.10
    steps:
      - checkout
      - run:
          name: build the binary
          command: go build -o hello-app
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_BUILD_NUM' >> $BASH_ENV
      - run:
          name: Build the docker image
          command: docker build . -t ${CIRCLE_PROJECT_REPONAME}:$TAG
      - run:
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run:
          name: Login to ECR
          command: $(aws ecr get-login --region $AWS_REGION | sed -e 's/-e none//g')
      - run: 
          name: Tag the image with ECR repo name 
          command: docker tag ${CIRCLE_PROJECT_REPONAME}:$TAG ${HELLOAPP_ECR_REPO}:$TAG    
      - run: 
          name: Push the image the ECR repo
          command: docker push ${HELLOAPP_ECR_REPO}:$TAG
  deploy:
    docker:
        - image: circleci/golang:1.10
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - checkout
      - run:
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run:
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_PREVIOUS_BUILD_NUM' >> $BASH_ENV
      - run:
          name: Install and confgure kubectl
          command: sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && sudo chmod +x /usr/local/bin/kubectl  
      - run:
          name: Install and confgure kubectl aws-iam-authenticator
          command: curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator && sudo chmod +x ./aws-iam-authenticator && sudo cp ./aws-iam-authenticator /bin/aws-iam-authenticator
       - run:
          name: Install latest awscli version
          command: sudo apt install unzip && curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip &&./awscli-bundle/install -b ~/bin/aws
      - run:
          name: Get the kubeconfig file 
          command: export KUBECONFIG=$HOME/.kube/kubeconfig && /home/circleci/bin/aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
      - run:
          name: Install and configuire helm
          command: sudo curl -L https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz | tar xz && sudo mv linux-amd64/helm /bin/helm && sudo rm -rf linux-amd64
      - run:
          name: Initialize helm
          command:  helm init --client-only --kubeconfig=$HOME/.kube/kubeconfig
      - run:
          name: Install tiller plugin
          command: helm plugin install https://github.com/rimusz/helm-tiller --kubeconfig=$HOME/.kube/kubeconfig        
      - run:
          name: Release helloapp using helm chart 
          command: bash scripts/release-helloapp.sh $TAG
workflows:
  version: 2
  primary:
    jobs:
      - build&pushImage
      - deploy:
          requires:
            - build&pushImage

At the end of the file, we see the workflows, workflows control the order in which the jobs specified in the file are executed and establishes dependencies and conditions for the job. For example, we may want our deploy job trigger only after my build job is complete so we added a dependency between them. Similarly, we may want to exclude the jobs from running on some particular branch then we can specify those type of conditions as well.

We have used a few environment variables in our pipeline configuration some of them were created by us and some were made available by CircleCI. We created AWS_REGION, HELLOAPP_ECR_REPO, EKS_CLUSTER_NAME, AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY variables. These variables are set via. CircleCI web console by going to the projects settings. Other variables that we have used are made available by CircleCI as a part of its environment setup process. Complete list of environment variables set by CircleCI can be found here.

Verify the working of the pipeline:

Once everything is set up properly then our application will get deployed on the k8s cluster and should be available for access. Get the external IP of the helloapp service and make a curl request to the hello endpoint

$ curl http://a31e25e7553af11e994620aebe144c51-242977608.us-west-2.elb.amazonaws.com/hello && printf "n"
{"Msg":"Hello World"}

$ curl http://a31e25e7553af11e994620aebe144c51-242977608.us-west-2.elb.amazonaws.com/hello && printf "n"

{"Msg":"Hello World"}

Now update the code and change the message “Hello World” to “Hello World Returns” and push your code. It will take a few minutes for the pipeline to complete execution and once it is complete make the curl request again to see the changes getting reflected.

$ curl http://a31e25e7553af11e994620aebe144c51-242977608.us-west-2.elb.amazonaws.com/hello && printf "n"
{"Msg":"Hello World Returns"}

$ curl http://a31e25e7553af11e994620aebe144c51-242977608.us-west-2.elb.amazonaws.com/hello && printf "n"

{"Msg":"Hello World Returns"}

Also, verify that a new tag is also created for the helloapp docker image on ECR.

Conclusion

In this blog post, we explored how we can set up a CI/CD pipeline for kubernetes and got basic exposure to CircleCI and Helm. Although helm is not absolutely necessary for building a pipeline, it has lots of benefits and is widely used across the industry. We can extend the pipeline to consider the cases where we have multiple environments like dev, staging & production and make the pipeline deploy the application to any of them depending upon some conditions. We can also add more jobs like integration tests. All the codes used in the blog post are available here.

Related Reads:

December 12, 2022

How to Avoid Screwing Up CI/CD: Best Practices for DevOps Team
Basic Fundamentals (One-line definition) :

CI/CD is defined as continuous integration, continuous delivery, and/or continuous deployment.

Continuous Integration:

Continuous integration is defined as a practice where a developer’s changes are merged back to the main branch as soon as possible to avoid facing integration challenges.

Continuous Delivery:

Continuous delivery is basically the ability to get all the types of changes deployed to production or delivered to the customer in a safe, quick, and sustainable way.

An oversimplified CI/CD pipeline

Why CI/CD?

‍
- Avoid integration hell
In most modern application development scenarios, multiple developers work on different features simultaneously. However, if all the source code is to be merged on the same day, the result can be a manual, tedious process of resolving conflicts between branches, as well as a lot of rework.

Continuous integration (CI) is the process of merging the code changes frequently (can be daily or multiple times a day also) to a shared branch (aka master or truck branch). The CI process makes it easier and quicker to identify bugs, saving a lot of developer time and effort.
- Faster time to market
Less time is spent on solving integration problems and reworking, allowing faster time to market for products.
- Have a better and more reliable code
The changes are small and thus easier to test. Each change goes through a rigorous cycle of unit tests, integration/regression tests, and performance tests before being pushed to prod, ensuring a better quality code.
- Lower costs
As we have a faster time to market and fewer integration problems, a lot of developer time and development cycles are saved, leading to a lower cost of development.

Enough theory now, let’s dive into “How do I get started ?”

Basic Overview of CI/CD

Decide on your branching strategy

A good branching strategy should have the following characteristics:
- Defines a clear development process from initial commit to production deployment
- Enables parallel development
- Optimizes developer productivity
- Enables faster time to market for products and services
- Facilitates integration with all DevOps practices and tools such as different versions of control systems
Types of branching strategies (please refer to references for more details) :
- Git flow – Ideal when handling multiple versions of the production code and for enterprise customers who have to adhere to release plans and workflows
- Trunk-based development – Ideal for simpler workflows and if automated testing is available, leading to a faster development time
- Other branching strategies that you can read about are Github flow, Gitlab flow, and Forking flow.
Build or compile your code

The next step is to build/compile your code, and if it is interpreted code, go ahead and package it.

Build best practices :
- Build Once – Building the same artifact for multiple env is inadvisable.
- Exact versions of third-party dependencies should be used.
- Libraries used for debugging, etc., should be removed from the product package.
- Have a feedback loop so that the team is made aware of the status of the build step.
- Make sure your builds are versioned correctly using semver 2.0 (https://semver.org/).
- Commit early, commit often.
Select tool for stitching the pipeline together
- You can choose from GitHub actions, Jenkins, circleci, GitLab, etc.
- Tool selection will not affect the quality of your CI/CD pipeline but might increase the maintenance if we go for managed CI/CD services as opposed to services like Jenkins deployed onprem.
Tools and strategy for SAST

Instead of just DevOps, we should think of devsecops. To make the code more secure and reliable, we can introduce a step for SAST (static application security testing).

SAST, or static analysis, is a testing procedure that analyzes source code to find security vulnerabilities. SAST scans the application code before the code is compiled. It’s also known as white-box testing, and it helps shift towards a security-first mindset as the code is scanned right at the start of SDLC.

Problems SAST solves:
- SAST tools give developers real-time feedback as they code, helping them fix issues before they pass the code to the next phase of the SDLC.
- This prevents security-related issues from being considered an afterthought.
Deployment strategies

How will you deploy your code with zero downtime so that the customer has the best experience? Try and implement one of the strategies below automatically via CI/CD. This will help in keeping the blast radius to the minimum in case something goes wrong.
- Ramped (also known as rolling-update or incremental): The new version is slowly rolled out to replace the older version of the product .
- Blue/Green: The new version is released alongside the older version, then the traffic is switched to the newer version.
- Canary: The new version is released to a selected group of users before doing a full rollout. This can be achieved by feature flagging as well. For more information, read about tools like launch darkly(https://launchdarkly.com/) and git unleash (https://github.com/Unleash/unleash).
- A/B testing: The new version is released to a subset of users under specific conditions.
- Shadow: The new version receives real-world traffic alongside the older version and doesn’t impact the response.
Config and Secret Management

According to the 12-factor app, application configs should be exposed to the application with environment variables. However, it does not have restrictions on where these configurations need to be stored and sourced from.

A few things to keep in mind while storing configs.
- Versioning of configs always helps, but storing secrets in VCS is strongly discouraged.
- For an enterprise, it is beneficial to use a cloud-agnostic solution.
Solution:
- Store your configuration secrets outside of the version control system.
- You can use AWS secret manager, Vault, and even S3 for storing your configs, e.g.: S3 with KMS, etc. There are other services available as well, so choose the one which suits your use case the best.
Automate versioning and release notes generation

All the releases should be tagged in the version control system. Versions can be automatically updated by looking at the git commit history and searching for keywords.

There are many modules available for release notes generation. Try and automate these as well as a part of your CI/CD process. If this is done, you can successfully eliminate human intervention from the release process.

Example from GitHub actions workflow :
```
- name: Automated Version Bump
  id: version-bump
  uses: 'phips28/gh-action-bump-version@v9.0.16'
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  with:
    commit-message: 'CI: Bump version to v{{version}}'
```
Have a rollback strategy

In case of regression, performance, or smoke test fails after deployment onto an environment, feedback should be given and the version should be rolled back automatically as a part of the CI/CD process. This makes sure that the environment is up and also reduces the MTTR (mean time to recovery), and MTTD (mean time to detection) in case there is a production outage due to code deployment.

GitOps tools like argocd and flux make it easy to do things like this, but even if you are not using any of the GitOps tools, this can be easily managed using scripts or whatever tool you are using for deployment.

Include db changes as a part of your CI/CD

Databases are often created manually and frequently evolve through manual changes, informal processes, and even testing in production. Manual changes often lack documentation and are harder to review, test, and coordinate with software releases. This makes the system more fragile with a higher risk of failure.

The correct way to do this is to include the database in source control and CI/CD pipeline. This lets the team document each change, follow the code review process, test it thoroughly before release, make rollbacks easier, and coordinate with software releases.

For a more enterprise or structured solution, we could use a tool such as Liquibase, Alembic, or Flyway.

How it should ideally be done:
- We can have a migration-based strategy where, for each DB change, an additional migration script is added and is executed as a part of CI/CD .
- Things to keep in mind are that the CI/CD process should be the same across all the environments. Also, the amount of data on prod and other environments might vary drastically, so batching and limits should be used so that we don’t end up using all the memory of our database server.
- As far as possible, DB migrations should be backward compatible. This makes it easier for rollbacks. This is the reason some companies only allow additive changes as a part of db migration scripts.
Real-world scenarios
- Gated approach
It is not always possible to have a fully automated CI/CD pipeline because the team may have just started the development of a product and might not have automated testing yet.

So, in cases like these, we have manual gates that can be approved by the responsible teams. For example, we will deploy to the development environment and then wait for testers to test the code and approve the manual gate, then the pipeline can go forward.

Most of the tools support these kinds of requests. Make sure that you are not using any kind of resources for this step otherwise you will end up blocking resources for the other pipelines.

Example:

https://www.jenkins.io/doc/pipeline/steps/pipeline-input-step/#input-wait-for-interactive-input
def LABEL_ID = "yourappname-${UUID.randomUUID().toString()}" def BRANCH_NAME = "<Your branch name>" def GIT_URL = "<Your git url>" // Start Agent node(LABEL_ID) { stage('Checkout') { doCheckout(BRANCH_NAME, GIT_URL) } stage('Build') { ... } stage('Tests') { ... } } // Kill Agent // Input Step timeout(time: 15, unit: "MINUTES") { input message: 'Do you want to approve the deploy in production?', ok: 'Yes' } // Start Agent Again node(LABEL_ID) { doCheckout(BRANCH_NAME, GIT_URL) stage('Deploy') { ... } } def doCheckout(branchName, gitUrl){ checkout([$class: 'GitSCM', branches: [[name: branchName]], doGenerateSubmoduleConfigurations: false, extensions:[[$class: 'CloneOption', noTags: true, reference: '', shallow: true]], userRemoteConfigs: [[credentialsId: '<Your credentials id>', url: gitUrl]]]) }
```
def LABEL_ID = "yourappname-${UUID.randomUUID().toString()}"
def BRANCH_NAME = "<Your branch name>"
def GIT_URL = "<Your git url>"
// Start Agent
node(LABEL_ID) {
    stage('Checkout') {
        doCheckout(BRANCH_NAME, GIT_URL)
    }
    stage('Build') {
        ...
    }
    stage('Tests') {
        ...
    }    
}
// Kill Agent
// Input Step
timeout(time: 15, unit: "MINUTES") {
    input message: 'Do you want to approve the deploy in production?', ok: 'Yes'
}
// Start Agent Again
node(LABEL_ID) {
    doCheckout(BRANCH_NAME, GIT_URL) 
    stage('Deploy') {
        ...
    }
}
def doCheckout(branchName, gitUrl){
    checkout([$class: 'GitSCM',
        branches: [[name: branchName]],
        doGenerateSubmoduleConfigurations: false,
        extensions:[[$class: 'CloneOption', noTags: true, reference: '', shallow: true]],
        userRemoteConfigs: [[credentialsId: '<Your credentials id>', url: gitUrl]]])
}
```
Observability of releases

Whenever we are debugging the root cause of issues in production, we might need the information below. As the system gets more complex with multiple upstreams and downstream, it becomes imperative that we have this information, all in one place, for efficient debugging and support by the operations team.
- When was the last deployment? What version was deployed?
- The deployment history as to which version was deployed when along with the code changes that went in.
Below are the 2 ways generally organizations follow to achieve this:
- Have a release workflow that is tracked using a Change request or Service request on Jira or any other tracking tool.
- For GitOps applications using tools like Argo CD and flux, all this information is available as a part of the version control system and can be derived from there.
DORA metrics

DevOps maturity of a team is measured based on mainly four metrics that are defined below, and CI/CD helps in improving all of the below. So, teams and organizations should try and achieve the Elite status for DORA metrics.
- Deployment Frequency— How often an org successfully releases to production
- Lead Time for Changes— The amount of time a commit takes to get into prod
- Change Failure Rate— The percentage of deployments causing a failure in prod
- Time to Restore Service— How long an org takes to recover from a failure in prod
Conclusion

CI/CD forms an integral part of DevOps and SRE practices, and if done correctly, it can impact the team’s and organization’s productivity in a huge way.

So, try and implement the above principles and get one step closer to having a highly productive team and a better product.
December 12, 2022

Tag: continuous delivery

A Guide to End-to-End API Test Automation with Postman and GitHub Actions

Objective

Introduction to Postman

Getting started with Postman

Setting up the environment

Create a collection

‍Adding requests to the collection

Creating TESTS

Running the Collection

Collaboration

Integrating with CI/CD

Creating a pipeline with GitHub Actions

Workflow File

Generating Allure report and hosting the report onto s3.

Send Slack notification with the Status of the job

Continuous Integration & Delivery (CI/CD) for Kubernetes Using CircleCI & Helm

Introduction

What is CircleCI?

What is Helm?

Building the Pipeline

Overview:

Creating Helm Chart:

Configure the pipeline:

Verify the working of the pipeline:

Conclusion

How to Avoid Screwing Up CI/CD: Best Practices for DevOps Team

Continuous Integration:

Continuous Delivery:

Why CI/CD?

Enough theory now, let’s dive into “How do I get started ?”

Decide on your branching strategy

Build or compile your code

Select tool for stitching the pipeline together

Tools and strategy for SAST

Deployment strategies

Config and Secret Management

Automate versioning and release notes generation

Have a rollback strategy

Include db changes as a part of your CI/CD

Real-world scenarios

Observability of releases

DORA metrics

Conclusion