My blog

Tag: helm

  • Continuous Integration & Delivery (CI/CD) for Kubernetes Using CircleCI & Helm

    Introduction

    Kubernetes is getting adopted rapidly across the software industry and is becoming the most preferred option for deploying and managing containerized applications. Once we have a fully functional Kubernetes cluster we need to have an automated process to deploy our applications on it. In this blog post, we will create a fully automated “commit to deploy” pipeline for Kubernetes. We will use CircleCI & helm for it.

    What is CircleCI?

    CircleCI is a fully managed saas offering which allows us to build, test or deploy our code on every check in. For getting started with circle we need to log into their web console with our GitHub or bitbucket credentials then add a project for the repository we want to build and then add the CircleCI config file to our repository. The CircleCI config file is a yaml file which lists the steps we want to execute on every time code is pushed to that repository.

    Some salient features of CircleCI is:

    1. Little or no operational overhead as the infrastructure is managed completely by CircleCI.
    2. User authentication is done via GitHub or bitbucket so user management is quite simple.
    3. It automatically notifies the build status on the github/bitbucket email ids of the users who are following the project on CircleCI.
    4. The UI is quite simple and gives a holistic view of builds.
    5. Can be integrated with Slack, hipchat, jira, etc.

    What is Helm?

    Helm is chart manager where chart refers to package of Kubernetes resources. Helm allows us to bundle related Kubernetes objects into charts and treat them as a single unit of deployment referred to as release.  For example, you have an application app1 which you want to run on Kubernetes. For this app1 you create multiple Kubernetes resources like deployment, service, ingress, horizontal pod scaler, etc. Now while deploying the application you need to create all the Kubernetes resources separately by applying their manifest files. What helm does is it allows us to group all those files into one chart (Helm chart) and then we just need to deploy the chart. This also makes deleting and upgrading the resources quite simple.

    Some other benefits of Helm is:

    1. It makes the deployment highly configurable. Thus just by changing the parameters, we can use the same chart for deploying on multiple environments like stag/prod or multiple cloud providers.
    2. We can rollback to a previous release with a single helm command.
    3. It makes managing and sharing Kubernetes specific application much simpler.

    Note: Helm is composed of two components one is helm client and the other one is tiller server. Tiller is the component which runs inside the cluster as deployment and serves the requests made by helm client. Tiller has potential security vulnerabilities thus we will use tillerless helm in our pipeline which runs tiller only when we need it.

    Building the Pipeline

    Overview:

    We will create the pipeline for a Golang application. The pipeline will first build the binary, create a docker image from it, push the image to ECR, then deploy it on the Kubernetes cluster using its helm chart.

    We will use a simple app which just exposes a `hello` endpoint and returns the hello world message:

    package main

import (
	"encoding/json"
	"net/http"
	"log"
	"github.com/gorilla/mux"
)

type Message struct {
	Msg string
}

func helloWorldJSON(w http.ResponseWriter, r *http.Request) {
	m := Message{"Hello World"}
	response, _ := json.Marshal(m)
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	w.Write(response)
}
func main() {
	r := mux.NewRouter()
	r.HandleFunc("/hello", helloWorldJSON).Methods("GET")
	if err := http.ListenAndServe(":8080", r); err != nil {
		log.Fatal(err)
	}
}

    We will create a docker image for hello app using the following Dockerfile:

    FROM centos/systemd

MAINTAINER "Akash Gautam" <akash.gautam@velotio.com>

COPY hello-app  /

ENTRYPOINT ["/hello-app"]

    Creating Helm Chart:

    Now we need to create the helm chart for hello app.

    First, we create the Kubernetes manifest files. We will create a deployment and a service file:

    apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: helloapp
spec:
  replicas: 1
  strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 1
  template:
    metadata:
      labels:
        app: helloapp
        env: {{ .Values.labels.env }}
        cluster: {{ .Values.labels.cluster }}
    spec:
      containers:
      - name: helloapp
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        imagePullPolicy: {{ .Values.image.imagePullPolicy }}
        readinessProbe:
          httpGet:
            path: /hello
            port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1

    apiVersion: v1
kind: Service
metadata:
  name: helloapp
spec:
  type: {{ .Values.service.type }}
  ports:
  - name: helloapp
    port: {{ .Values.service.port }}
    protocol: TCP
    targetPort: {{ .Values.service.targetPort }}
  selector:
    app: helloapp

    In the above file, you must have noticed that we have used .Values object. All the values that we specify in the values.yaml file in our helm chart can be accessed using the .Values object inside the template.

    Let’s create the helm chart now:

    helm create helloapp

    Above command will create a chart helm chart folder structure for us.

    helloapp/
|
|- .helmignore # Contains patterns to ignore when packaging Helm charts.
|
|- Chart.yaml # Information about your chart
|
|- values.yaml # The default values for your templates
|
|- charts/ # Charts that this chart depends on
|
|- templates/ # The template files

    We can remove the charts/ folder inside our helloapp chart as our chart won’t have any sub-charts. Now we need to move our Kubernetes manifest files to the template folder and update our values.yaml and Chart.yaml

    Our values.yaml looks like:

    image:
  tag: 0.0.1
  repository: 123456789870.dkr.ecr.us-east-1.amazonaws.com/helloapp
  imagePullPolicy: Always

labels:
  env: "staging"
  cluster: "eks-cluster-blog"

service:
  port: 80
  targetPort: 8080
  type: LoadBalancer

    This allows us to make our deployment more configurable. For example, here we have set our service type as LoadBalancer in values.yaml but if we want to change it to nodePort we just need to set is as NodePort while installing the chart (–set service.type=NodePort). Similarly, we have set the image pull policy as Always which is fine for development/staging environment but when we deploy to production we may want to set is as ifNotPresent. In our chart, we need to identify the parameters/values which may change from one environment to another and make them configurable. This allows us to be flexible with our deployment and reuse the same chart

    Finally, we need to update Chart.yaml file. This file mostly contains metadata about the chart like the name, version, maintainer, etc, where name & version are two mandatory fields for Chart.yaml.

    version: 1.0.0
appVersion: 0.0.1
name: helloapp
description: Helm chart for helloapp
source:
  - https://github.com/akash-gautam/helloapp

    Now our Helm chart is ready we can start with the pipeline. We need to create a folder named .circleci in the root folder of our repository and create a file named config.yml in it. In our config.yml we have defined two jobs one is build&pushImage and deploy.

    Configure the pipeline:

    build&pushImage:
    working_directory: /go/src/hello-app (1)
    docker:
      - image: circleci/golang:1.10 (2)
    steps:
      - checkout (3)
      - run: (4)
          name: build the binary
          command: go build -o hello-app
      - setup_remote_docker: (5)
          docker_layer_caching: true
      - run: (6)
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_BUILD_NUM' >> $BASH_ENV
      - run: (7)
          name: Build the docker image
          command: docker build . -t ${CIRCLE_PROJECT_REPONAME}:$TAG
      - run: (8)
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run: (9)
          name: Login to ECR
          command: $(aws ecr get-login --region $AWS_REGION | sed -e 's/-e none//g')
      - run: (10)
          name: Tag the image with ECR repo name 
          command: docker tag ${CIRCLE_PROJECT_REPONAME}:$TAG ${HELLOAPP_ECR_REPO}:$TAG    
      - run: (11)
          name: Push the image the ECR repo
          command: docker push ${HELLOAPP_ECR_REPO}:$TAG

    1. We set the working directory for our job, we are setting it on the gopath so that we don’t need to do anything additional.
    2. We set the docker image inside which we want the job to run, as our app is built using golang we are using the image which already has golang installed in it.
    3. This step checks out our repository in the working directory
    4. In this step, we build the binary
    5. Here we setup docker with the help of  setup_remote_docker  key provided by CircleCI.
    6. In this step we create the tag we will be using while building the image, we use the app version available in the VERSION file and append the $CIRCLE_BUILD_NUM value to it, separated by a dash (`-`).
    7. Here we build the image and tag.
    8. Installing AWS CLI to interact with the ECR later.
    9. Here we log into ECR
    10. We tag the image build in step 7 with the ECR repository name.
    11. Finally, we push the image to ECR.

    Now we will deploy our helm charts. For this, we have a separate job deploy.

    deploy:
    docker: (1)
        - image: circleci/golang:1.10
    steps: (2)
      - checkout
      - run: (3)
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run: (4)
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_PREVIOUS_BUILD_NUM' >> $BASH_ENV
      - run: (5)
          name: Install and confgure kubectl
          command: sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && sudo chmod +x /usr/local/bin/kubectl  
      - run: (6)
          name: Install and confgure kubectl aws-iam-authenticator
          command: curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator && sudo chmod +x ./aws-iam-authenticator && sudo cp ./aws-iam-authenticator /bin/aws-iam-authenticator
       - run: (7)
          name: Install latest awscli version
          command: sudo apt install unzip && curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip &&./awscli-bundle/install -b ~/bin/aws
      - run: (8)
          name: Get the kubeconfig file 
          command: export KUBECONFIG=$HOME/.kube/kubeconfig && /home/circleci/bin/aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
      - run: (9)
          name: Install and configuire helm
          command: sudo curl -L https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz | tar xz && sudo mv linux-amd64/helm /bin/helm && sudo rm -rf linux-amd64
      - run: (10)
          name: Initialize helm
          command:  helm init --client-only --kubeconfig=$HOME/.kube/kubeconfig
      - run: (11)
          name: Install tiller plugin
          command: helm plugin install https://github.com/rimusz/helm-tiller --kubeconfig=$HOME/.kube/kubeconfig        
      - run: (12)
          name: Release helloapp using helm chart 
          command: bash scripts/release-helloapp.sh $TAG

    1. Set the docker image inside which we want to execute the job.
    2. Check out the code using `checkout` key
    3. Install AWS CLI.
    4. Setting the value of tag just like we did in case of build&pushImage job. Note that here we are using CIRCLE_PREVIOUS_BUILD_NUM variable which gives us the build number of build&pushImage job and ensures that the tag values are the same.
    5. Download kubectl and making it executable.
    6. Installing aws-iam-authenticator this is required because my k8s cluster is on EKS.
    7. Here we install the latest version of AWS CLI, EKS is a relatively newer service from AWS and older versions of AWS CLI doesn’t have it.
    8. Here we fetch the kubeconfig file. This step will vary depending upon where the k8s cluster has been set up. As my cluster is on EKS am getting the kubeconfig file via. AWS CLI similarly if your cluster in on GKE then you need to configure gcloud and use the command  `gcloud container clusters get-credentials <cluster-name> –zone=<zone-name>`. We can also have the kubeconfig file on some other secure storage system and fetch it from there.</zone-name></cluster-name>
    9. Download Helm and make it executable
    10. Initializing helm, note that we are initializing helm in client only mode so that it doesn’t start the tiller server.
    11. Download the tillerless helm plugin
    12. Execute the release-helloapp.sh shell script and pass it TAG value from step 4.

    In the release-helloapp.sh script we first start tiller, after this, we check if the release is already present or not if it is present then we upgrade otherwise we make a new release. Here we override the value of tag for the image present in the chart by setting it to the tag of the newly built image, finally, we stop the tiller server.

    #!/bin/bash
TAG=$1
echo "start tiller"
export KUBECONFIG=$HOME/.kube/kubeconfig
helm tiller start-ci
export HELM_HOST=127.0.0.1:44134
result=$(eval helm ls | grep helloapp) 
if [ $? -ne "0" ]; then 
   helm install --timeout 180 --name helloapp --set image.tag=$TAG charts/helloapp
else 
   helm upgrade --timeout 180 helloapp --set image.tag=$TAG charts/helloapp
fi
echo "stop tiller"
helm tiller stop

    The complete CircleCI config.yml file looks like:

    version: 2

jobs:
  build&pushImage:
    working_directory: /go/src/hello-app
    docker:
      - image: circleci/golang:1.10
    steps:
      - checkout
      - run:
          name: build the binary
          command: go build -o hello-app
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_BUILD_NUM' >> $BASH_ENV
      - run:
          name: Build the docker image
          command: docker build . -t ${CIRCLE_PROJECT_REPONAME}:$TAG
      - run:
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run:
          name: Login to ECR
          command: $(aws ecr get-login --region $AWS_REGION | sed -e 's/-e none//g')
      - run: 
          name: Tag the image with ECR repo name 
          command: docker tag ${CIRCLE_PROJECT_REPONAME}:$TAG ${HELLOAPP_ECR_REPO}:$TAG    
      - run: 
          name: Push the image the ECR repo
          command: docker push ${HELLOAPP_ECR_REPO}:$TAG
  deploy:
    docker:
        - image: circleci/golang:1.10
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - checkout
      - run:
          name: Install AWS cli
          command: export TZ=Europe/Minsk && sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > sudo  /etc/timezone && sudo apt-get update && sudo apt-get install -y awscli
      - run:
          name: Set the tag for the image, we will concatenate the app verson and circle build number with a `-` char in between
          command:  echo 'export TAG=$(cat VERSION)-$CIRCLE_PREVIOUS_BUILD_NUM' >> $BASH_ENV
      - run:
          name: Install and confgure kubectl
          command: sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && sudo chmod +x /usr/local/bin/kubectl  
      - run:
          name: Install and confgure kubectl aws-iam-authenticator
          command: curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator && sudo chmod +x ./aws-iam-authenticator && sudo cp ./aws-iam-authenticator /bin/aws-iam-authenticator
       - run:
          name: Install latest awscli version
          command: sudo apt install unzip && curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip &&./awscli-bundle/install -b ~/bin/aws
      - run:
          name: Get the kubeconfig file 
          command: export KUBECONFIG=$HOME/.kube/kubeconfig && /home/circleci/bin/aws eks --region $AWS_REGION update-kubeconfig --name $EKS_CLUSTER_NAME
      - run:
          name: Install and configuire helm
          command: sudo curl -L https://storage.googleapis.com/kubernetes-helm/helm-v2.11.0-linux-amd64.tar.gz | tar xz && sudo mv linux-amd64/helm /bin/helm && sudo rm -rf linux-amd64
      - run:
          name: Initialize helm
          command:  helm init --client-only --kubeconfig=$HOME/.kube/kubeconfig
      - run:
          name: Install tiller plugin
          command: helm plugin install https://github.com/rimusz/helm-tiller --kubeconfig=$HOME/.kube/kubeconfig        
      - run:
          name: Release helloapp using helm chart 
          command: bash scripts/release-helloapp.sh $TAG
workflows:
  version: 2
  primary:
    jobs:
      - build&pushImage
      - deploy:
          requires:
            - build&pushImage

    At the end of the file, we see the workflows, workflows control the order in which the jobs specified in the file are executed and establishes dependencies and conditions for the job. For example, we may want our deploy job trigger only after my build job is complete so we added a dependency between them. Similarly, we may want to exclude the jobs from running on some particular branch then we can specify those type of conditions as well.

    We have used a few environment variables in our pipeline configuration some of them were created by us and some were made available by CircleCI. We created AWS_REGION, HELLOAPP_ECR_REPO, EKS_CLUSTER_NAME, AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY variables. These variables are set via. CircleCI web console by going to the projects settings. Other variables that we have used are made available by CircleCI as a part of its environment setup process. Complete list of environment variables set by CircleCI can be found here.

    Verify the working of the pipeline:

    Once everything is set up properly then our application will get deployed on the k8s cluster and should be available for access. Get the external IP of the helloapp service and make a curl request to the hello endpoint

    $ curl http://a31e25e7553af11e994620aebe144c51-242977608.us-west-2.elb.amazonaws.com/hello && printf "n"

{"Msg":"Hello World"}

    Now update the code and change the message “Hello World” to “Hello World Returns” and push your code. It will take a few minutes for the pipeline to complete execution and once it is complete make the curl request again to see the changes getting reflected.

    $ curl http://a31e25e7553af11e994620aebe144c51-242977608.us-west-2.elb.amazonaws.com/hello && printf "n"

{"Msg":"Hello World Returns"}

    Also, verify that a new tag is also created for the helloapp docker image on ECR.

    Conclusion

    In this blog post, we explored how we can set up a CI/CD pipeline for kubernetes and got basic exposure to CircleCI and Helm. Although helm is not absolutely necessary for building a pipeline, it has lots of benefits and is widely used across the industry. We can extend the pipeline to consider the cases where we have multiple environments like dev, staging & production and make the pipeline deploy the application to any of them depending upon some conditions. We can also add more jobs like integration tests. All the codes used in the blog post are available here.

    Related Reads:

    1. Continuous Deployment with Azure Kubernetes Service, Azure Container Registry & Jenkins
    2. Know Everything About Spinnaker & How to Deploy Using Kubernetes Engine

  • Getting Started With Kubernetes Operators (Helm Based) – Part 1

    Introduction

    The concept of operators was introduced by CoreOs in the last quarter of  2016 and post the introduction of operator framework last year, operators are rapidly becoming the standard way of managing applications on Kubernetes especially the ones which are stateful in nature. In this blog post, we will learn what an operator is. Why they are needed and what problems do they solve. We will also create a helm based operator as an example.

    This is the first part of our Kubernetes Operator Series. In the second part, getting started with Kubernetes operators (Ansible based), and the third part, getting started with Kubernetes operators (Golang based), you can learn how to build Ansible and Golang based operators.

    What is an Operator?

    Whenever we deploy our application on Kubernetes we leverage multiple Kubernetes objects like deployment, service, role, ingress, config map, etc. As our application gets complex and our requirements become non-generic, managing our application only with the help of native Kubernetes objects becomes difficult and we often need to introduce manual intervention or some other form of automation to make up for it.

    Operators solve this problem by making our application first class Kubernetes objects that is we no longer deploy our application as a set of native Kubernetes objects but a custom object/resource of its kind, having a more domain-specific schema and then we bake the “operational intelligence” or the “domain-specific knowledge” into the controller responsible for maintaining the desired state of this object. For example, etcd operator has made etcd-cluster a first class object and for deploying the cluster we create an object of Etcd Cluster kind. With operators, we are able to extend Kubernetes functionalities for custom use cases and manage our applications in a Kubernetes specific way allowing us to leverage Kubernetes APIs and Kubectl tooling.

    Operators combine crds and custom controllers and intend to eliminate the requirement for manual intervention (human operator) while performing tasks like an upgrade, handling failure recovery, scaling in case of complex (often stateful) applications and make them more resilient and self-sufficient.

    How to Build Operators ?

    For building and managing operators we mostly leverage the Operator Framework which is an open source tool kit allowing us to build operators in a highly automated, scalable and effective way.  Operator framework comprises of three subcomponents:

    1. Operator SDK: Operator SDK is the most important component of the operator framework. It allows us to bootstrap our operator project in minutes. It exposes higher level APIs and abstraction and saves developers the time to dig deeper into kubernetes APIs and focus more on building the operational logic. It performs common tasks like getting the controller to watch the custom resource (cr) for changes etc as part of the project setup process.
    2. Operator Lifecycle Manager:  Operators also run on the same kubernetes clusters in which they manage applications and more often than not we create multiple operators for multiple applications. Operator lifecycle manager (OLM) provides us a declarative way to install, upgrade and manage all the operators and their dependencies in our cluster.
    3. Operator Metering:  Operator metering is currently an alpha project. It records historical cluster usage and can generate usage reports showing usage breakdown by pod or namespace over arbitrary time periods.

    Types of Operators

    Currently there are three different types of operator we can build:

    1. Helm based operators: Helm based operators allow us to use our existing Helm charts and build operators using them. Helm based operators are quite easy to build and are preferred to deploy a stateless application using operator pattern.
    2. Ansible based Operator: Ansible based operator allows us to use our existing ansible playbooks and roles and build operators using them. There are also easy to build and generally preferred for stateless applications.
    3. Go based operators: Go based operators are built to solve the most complex use cases and are generally preferred for stateful applications. In case of an golang based operator, we build the controller logic ourselves providing it with all our custom requirements. This type of operators is also relatively complex to build.

    Building a Helm based operator

    1. Let’s first install the operator sdk

    go get -d github.com/operator-framework/operator-sdk
cd $GOPATH/src/github.com/operator-framework/operator-sdk
git checkout master
make dep
make install

    Now we will have the operator-sdk binary in the $GOPATH/bin folder.      

    2.  Setup the project

    For building a helm based operator we can use an existing Helm chart. We will be using the book-store Helm chart which deploys a simple python app and mongodb instances. This app allows us to perform crud operations via. rest endpoints.

    Now we will use the operator-sdk to create our Helm based bookstore-operator project.

    operator-sdk new bookstore-operator --api-version=velotio.com/v1alpha1 --kind=BookStore --type=helm --helm-chart=book-store
  --helm-chart-repo=https://akash-gautam.github.io/helmcharts/

    In the above command, the bookstore-operator is the name of our operator/project. –kind is used to specify the kind of objects this operator will watch and –api-verison is used for versioning of this object. The operator sdk takes only this much information and creates the custom resource definition (crd) and also the custom resource (cr) of its type for us (remember we talked about high-level abstraction operator sdk provides). The above command bootstraps a project with below folder structure.

    bookstore-operator/
|
|- build/ # Contains the Dockerfile to build the operator image
|- deploy/ # Contains the crd,cr and manifest files for deploying operator
|- helm-charts/ # Contains the helm chart we used while creating the project
|- watches.yaml # Specifies the resource the operator watches (maintains the state of)

    We had discussed the operator-sdk automates setting up the operator projects and that is exactly what we can observe here. Under the build folder, we have the Dockerfile to build our operator image. Under deploy folder we have a crd folder containing both the crd and the cr. This folder also has operator.yaml file using which we will run the operator in our cluster, along with this we have manifest files for role, rolebinding and service account file to be used while deploying the operator.  We have our book-store helm chart under helm-charts. In the watches.yaml file.

    ---
- version: v1alpha1
  group: velotio.com
  kind: BookStore
  chart: /opt/helm/helm-charts/book-store

    We can see that the bookstore-operator watches events related to BookStore kind objects and executes the helm chart specified.

    If we take a look at the cr file under deploy/crds (velotio_v1alpha1_bookstore_cr.yaml) folder then we can see that it looks just like the values.yaml file of our book-store helm chart.

    apiVersion: velotio.com/v1alpha1
kind: BookStore
metadata:
  name: example-bookstore
spec:
  # Default values copied from <project_dir>/helm-charts/book-store/values.yaml
  
  # Default values for book-store.
  # This is a YAML-formatted file.
  # Declare variables to be passed into your templates.
  
  replicaCount: 1
  
  image:
    app:
      repository: akash125/pyapp
      tag: latest
      pullPolicy: IfNotPresent
    mongodb:
      repository: mongo
      tag: latest
      pullPolicy: IfNotPresent
      
  service:
    app:
      type: LoadBalancer
      port: 80
      targetPort: 3000
    mongodb:
      type: ClusterIP
      port: 27017
      targetPort: 27017
  
  
  resources: {}
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #  cpu: 100m
    #  memory: 128Mi
    # requests:
    #  cpu: 100m
    #  memory: 128Mi
  
  nodeSelector: {}
  
  tolerations: []
  
  affinity: {}

    In the case of Helm charts, we use the values.yaml file to pass the parameter to our Helm releases, Helm based operator converts all these configurable parameters into the spec of our custom resource. This allows us to express the values.yaml with a custom resource (CR) which, as a native Kubernetes object, enables the benefits of RBAC applied to it and an audit trail. Now when we want to update out deployed we can simply modify the CR and apply it, and the operator will ensure that the changes we made are reflected in our app.

    For each object of  `BookStore` kind  the bookstore-operator will perform the following actions:

    1. Create the bookstore app deployment if it doesn’t exists.
    2. Create the bookstore app service if it doesn’t exists.
    3. Create the mongodb deployment if it doesn’t exists.
    4. Create the mongodb service if it doesn’t exists.
    5. Ensure deployments and services match their desired configurations like the replica count, image tag, service port etc.  

    3. Build the Bookstore-operator Image

    The Dockerfile for building the operator image is already in our build folder we need to run the below command from the root folder of our operator project to build the image.

    operator-sdk build akash125/bookstore-operator:v0.0.1

    4. Run the Bookstore-operator

    As we have our operator image ready we can now go ahead and run it. The deployment file (operator.yaml under deploy folder) for the operator was created as a part of our project setup we just need to set the image for this deployment to the one we built in the previous step.

    After updating the image in the operator.yaml we are ready to deploy the operator.

    kubectl create -f deploy/service_account.yaml
kubectl create -f deploy/role.yaml
kubectl create -f deploy/role_binding.yaml
kubectl create -f deploy/operator.yaml

    Note: The role created might have more permissions then actually required for the operator so it is always a good idea to review it and trim down the permissions in production setups.

    Verify that the operator pod is in running state.

    5. Deploy the Bookstore App

    Now we have the bookstore-operator running in our cluster we just need to create the custom resource for deploying our bookstore app.

    First, we can create bookstore cr we need to register its crd.

    kubectl apply -f deploy/crds/velotio_v1alpha1_bookstore_crd.yaml

    Now we can create the bookstore object.

    kubectl apply -f deploy/crds/velotio_v1alpha1_bookstore_cr.yaml

    Now we can see that our operator has deployed out book-store app.

    Now let’s grab the external IP of the app and make some requests to store details of books.

    Let’s hit the external IP on the browser and see if it lists the books we just stored:

    The bookstore operator build is available here.

    Conclusion

    Since its early days Kubernetes was believed to be a great tool for managing stateless application but the managing stateful applications on Kubernetes was always considered difficult. Operators are a big leap towards managing stateful applications and other complex distributed, multi (poly) cloud workloads with the same ease that we manage the stateless applications. In this blog post, we learned the basics of Kubernetes operators and build a simple helm based operator. In the next installment of this blog series, we will build an Ansible based Kubernetes operator and then in the last blog we will build a full-fledged Golang based operator for managing stateful workloads.

    Related Reads:

  • Helm 3: A More Secured and Simpler Kubernetes Package Manager

    What is Helm?

    Helm helps you manage Kubernetes applications. Helm Charts help developers and operators easily define, install, and upgrade even the most complex Kubernetes application.

    Below are the three big concepts regarding Helm.

    1. Chart – A chart is a Helm package. It contains all resource definitions necessary to run an application, tool or service inside the Kubernetes cluster.

    2. Repository – A repository is a place where charts can be collected and shared.

    3, Release – Release is an instance of a chart running in a Kubernetes cluster. One chart can often be installed many times in the same cluster, and each time it is installed, a new release is created.

    Registry – Helm Registry stores Helm charts in a hierarchy storage structure and provides a function to orchestrate charts from the existing charts. To deploy and configure registry, refer to this.

    Why Helm?

    1. It helps find and use popular software packaged as Kubernetes charts
    2. Shares your own applications as Kubernetes charts
    3. Manages releases of Helm packages
    4. Creates reproducible builds of your Kubernetes applications

    Changes since Helm2

    Helm3 includes following major changes:

    1. Client-only architecture

    Helm 2 is a client-server architecture with the client called as Helm and the server called as Tiller. The client interacts with the Tiller and the chart repository. Tiller interacts with the Kubernetes API server. It renders Helm template files into Kubernetes manifest files, that it uses for operations on the Kubernetes cluster through the Kubernetes API.

    Helm 3 has a client-only architecture with the client still called as Helm. It operates similar to Helm 2 client, but the client interacts directly with the Kubernetes API server. The in-cluster server Tiller is removed in Helm 3.

     

    2. No need to initialize Helm

    Initializing Helm is obsolete in version 3. i.e. Helm init was removed and you don’t need to install Tiller in the cluster and set up a Helm state before using Helm. A Helm state is created automatically, whenever required.

    3. Chart dependency updated

    In Helm 2, chart dependencies are declared in requirements.yaml, as shown in the following example:

    dependencies:

    – name: mysql

      version: “1.3.2”

      repository: “https://example.com/charts/mysql“

    Chart dependencies are consolidated in Helm 3, hence moving the dependency definitions to Chart.yaml.

    4. Chart value validation

    In Helm 3, values passed to a chart during any Helm commands can be validated against a JSON schema. This validation is beneficial to help chart consumers avoid setting incorrect values and help improve chart usability. To enable consumers to avoid setting incorrect values, add a schema file named values.schema.json in the chart folder.

    Following commands call the validation:

    • helm install
    • helm upgrade
    • helm template

    5. Helm test framework updates

    Helm 3 includes following updates to the test framework (helm test):

    • Users can define tests as job resources
    • The test-failure hook was removed
    • The test-success hook was renamed to test, but the alias remains for test-success
    • You can dump logs from test pods with –logs flag

    Helm 3 is more than just removing Tiller. It has a lot of new capabilities. There is little or no difference from CLI or usage point of view in Helm 3 when compared with Helm 2.

    Prerequisites

    1. A running Kubernetes cluster.
    2. The Kubernetes cluster API endpoint should be reachable from the machine you are running Helm commands.

    Installing Helm 

    1. Download binary from here.
    2. Unpack it (tar -zxvf helm-v3.0.0-linux-amd64.tgz)
    3. Find the Helm binary and move it to its desired destination (mv linux-amd64/helm /usr/local/bin/helm)

    From there, you should be able to run the client command: ‘helm help’. 

    Note: We will be using Helm version 3.0.0

    Deploy a sample Helm Chart

    Use below command to create new chart named mysql in a new directory

    $ helm create mysql

    After running above command, Helm creates a directory with the following layout:

    velotiotech:~/work/mysql$ tree
.
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

3 directories, 9 files

    It creates a Chart.yaml file containing global variables for the chart such as version and description.

    velotiotech:~/work/mysql$ cat Chart.yaml 
apiVersion: v2
name: mysql
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: 1.16.0

    Then comes templates directory. There you put all the *.yaml files for Kubernetes. Helm uses Go template markup language to customize *.yaml files. Helm creates three default file types: deployment, service, ingress. All the files in this directory are skeletons that are filled with the variables from the values.yaml when you deploy your Helm chart. File _helpers.tpl contains your custom helper functions for variable calculation.

    By default, Helm creates an nginx deployment. We will customize it to create a Helm Chart to deploy mysql on Kubernetes cluster. Add new deployment to the templates directory.

    velotiotech:~/work/mysql$ cat templates/deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mysql.fullname" . }}
spec:
  selector:
    matchLabels:
      app: {{ include "mysql.name" . }}
  template:
    metadata:
      labels:
        app: {{ include "mysql.name" . }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: {{ .Values.mysql_root_password }}
        ports:
        - containerPort: {{ .Values.service.port }}
          name: mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: {{ .Values.persistentVolumeClaim }}

    Also, let’s create PVC which is used in deployment by just adding below file to the templates directory.

    velotiotech:~/work/mysql$ cat templates/persistentVolumeClaim.yml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ .Values.persistentVolumeClaim }}
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

    Helm runs each file in the templates directory through Go template rendering engine. Let’s create service.yaml for connecting to mysql instance.

    velotiotech:~/work/mysql$ cat templates/service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: {{ include "mysql.fullname" . }}
spec:
  ports:
  - port: {{ .Values.service.port }}
  selector:
    app: {{ include "mysql.name" . }}
  clusterIP: None

    Update values.yaml to populate the above chart’s templates.

    velotiotech:~/work/mysql$ cat values.yaml 
# Default values for mysql.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

image:
  repository: mysql
  tag: 5.6
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

mysql_root_password: password 

service:
  port: 3306

persistentVolumeClaim: mysql-data-disk

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

    After adding above deployment files, directory structure will look like:

    velotiotech:~/work/mysql$ tree
.
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── NOTES.txt
│   ├── persistentVolumeClaim.yml
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

3 directories, 9 files

    To render chart templates locally and display the output to check if everything is correct:

    $ helm template mysql

    Execute the following helm install command to deploy our mysql chart in the Kubernetes cluster.

    $ helm install mysql-release ./mysql

    velotiotech:~/work$ helm install mysql-release ./mysql
NAME: mysql-release
LAST DEPLOYED: Mon Nov 25 14:48:38 2019
NAMESPACE: mysql-chart
STATUS: deployed
REVISION: 1
NOTES:
1. Use below command to connect to mysql:
   kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql-release -ppassword

2. Try creating database in mysql using command:
   create database test;

    Now the chart is installed. Note that installing a Helm chart creates a new release object. The release above is named mysql-release.

    To keep a track of a release’s state, or to re-read configuration information, you can use Helm status:

    $ helm status mysql-release

    Additionally, to create a package, use below command which requires path for chart (which must contain a Chart.yaml file) and then package that directory:

    $ helm package <path_to_Chart.yaml>

    This command creates an archive like mysql-0.1.0.tgz, with which you can share your chart with others. For instance, you can upload this file to the Helm repository.

    You can also delete the sample deployment using delete command. For example,

    $ helm delete mysql-release

    Upgrade a release

    Helm provides a way to perform an install or an upgrade as a single command. Use Helm upgrade with the –install command. This will help Helm to see if the release is already installed. If not, it will run an install. If it is, then the existing release will be upgraded.

    $ helm upgrade --install <release name> --values <values file> <chart directory>