Blog

  • Unveiling the Magic of Golang Interfaces: A Comprehensive Exploration

    Go interfaces are powerful tools for designing flexible and adaptable code. However, their inner workings can often seem hidden behind the simple syntax.

    This blog post aims to peel back the layers and explore the internals of Go interfaces, providing you with a deeper understanding of their power and capabilities.

    1. Interfaces: Not Just Method Signatures

    While interfaces appear as collections of method signatures, they are deeper than that. An interface defines a contract: any type that implements the interface guarantees the ability to perform specific actions through those methods. This contract-based approach promotes loose coupling and enhances code reusability.

    // Interface defining a "printable" behavior
    type Printable interface {
        String() string
    }
    
    // Struct types implementing the Printable interface
    type Book struct {
        Title string
    }
    
    type Article struct {
        Title string
        Content string
    }
    
    // Implement String() method to fulfill the contract
    func (b Book) String() string {
        return b.Title
    }
    
    // Implement String() method to fulfill the contract
    func (a Article) String() string {
        return fmt.Sprintf("%s", a.Title)
    }

    Here, both Book and Article types implement the Printable interface by providing a String() method. This allows us to treat them interchangeably in functions expecting Printable values.

    2. Interface Values and Dynamic Typing

    An interface variable itself cannot hold a value. Instead, it refers to an underlying concrete type that implements the interface. Go uses dynamic typing to determine the actual type at runtime. This allows for flexible operations like:

    func printAll(printables []Printable) {
        for _, p := range printables {
            fmt.Println(p.String()) // Calls the appropriate String() based on concrete type
        }
    }
    
    book := Book{Title: "Go for Beginners"}
    article := Article{Title: "The power of interfaces"}
    
    printables := []Printable{book, article}
    printAll(printables)

    The printAll function takes a slice of Printable and iterates over it. Go dynamically invokes the correct String() method based on the concrete type of each element (Book or Article) within the slice.

    3. Embedded Interfaces and Interface Inheritance

    Go interfaces support embedding existing interfaces to create more complex contracts. This allows for code reuse and hierarchical relationships, further enhancing the flexibility of your code:

    type Writer interface {
        Write(data []byte) (int, error)
    }
    
    type ReadWriter interface {
        Writer
        Read([]byte) (int, error)
    }
    
    type MyFile struct {
        // ... file data and methods
    }
    
    // MyFile implements both Writer and ReadWriter by embedding their interfaces
    func (f *MyFile) Write(data []byte) (int, error) {
        // ... write data to file
    }
    
    func (f *MyFile) Read(data []byte) (int, error) {
        // ... read data from file
    }

    Here, ReadWriter inherits all methods from the embedded Writer interface, effectively creating a more specific “read-write” contract.

    4. The Empty Interface and Its Power

    The special interface{} represents the empty interface, meaning it requires no specific methods. This seemingly simple concept unlocks powerful capabilities:

    // Function accepting any type using the empty interface
    func PrintAnything(value interface{}) {
        fmt.Println(reflect.TypeOf(value), value)
    }
    
    PrintAnything(42)  // Output: int 42
    PrintAnything("Hello") // Output: string Hello
    PrintAnything(MyFile{}) // Output: main.MyFile {}

    This function can accept any type because interface{} has no requirements. Internally, Go uses reflection to extract the actual type and value at runtime, enabling generic operations.

    5. Understanding Interface Equality and Comparisons

    Equality checks on interface values involve both the dynamic type and underlying value:

    book1 := Book{Title: "Go for Beginners"}
    book2 := Book{Title: "Go for Beginners"}
    
    // Same type and value, so equal
    fmt.Println(book1 == book2) // True
    
    differentBook := Book{Title: "Go for Dummies"}
    
    // Same type, different value, so not equal
    fmt.Println(book1 == differentBook) // False
    
    article := Article{Title: "Go for Beginners"}
    
    // This will cause a compilation error
    fmt.Println(book1 == article) // Error: invalid operation: book1 == article (mismatched types Book and Article)

    However, it’s essential to remember that interfaces themselves cannot be directly compared using the == operator unless they both contain exactly the same value of the same type.

    To compare interface values effectively, you can utilize two main approaches:

    1. Type Assertions:
    These allow you to safely access the underlying value and perform comparisons if you’re certain about the actual type:

    func getBookTitleFromPrintable(p Printable) (string, bool) {
        book, ok := p.(Book) // Check if p is a Book
        if ok {
            return book.Title, true
        }
        return "", false // Return empty string and false if not a Book
    }
    
    bookTitle, ok := getBookTitleFromPrintable(article)
    if ok {
        fmt.Println("Extracted book title:", bookTitle)
    } else {
        fmt.Println("Article is not a Book")
    }

    2. Custom Comparison Functions:
    You can also create dedicated functions to compare interface values based on specific criteria:

    func comparePrintablesByTitle(p1, p2 Printable) bool {
        return p1.String() == p2.String()
    }
    
    fmt.Println(comparePrintablesByTitle(book1, article)) // Compares titles regardless of types

    Understanding these limitations and adopting appropriate comparison techniques ensures accurate and meaningful comparisons with Go interfaces.

    6. Interface Methods and Implicit Receivers

    Interface methods implicitly receive a pointer to the underlying value. This enables methods to modify the state of the object they are called on:

    type Counter interface {
        Increment() int
    }
    
    type MyCounter struct {
        count int
    }
    
    func (c *MyCounter) Increment() int {
        c.count++
        return c.count
    }
    
    counter := MyCounter{count: 5}
    fmt.Println(counter.Increment()) // Output: 6

    The Increment method receives a pointer to MyCounter, allowing it to directly modify the count field.

    7. Error Handling and Interfaces

    Go interfaces play a crucial role in error handling. The built-in error interface defines a single method, Error() string, used to represent errors:

    type error interface {
        Error() string
    }
    
    // Custom error type implementing the error interface
    type MyError struct {
        message string
    }
    
    func (e MyError) Error() string {
        return e.message
    }
    
    func myFunction() error {
        // ... some operation
        return MyError{"Something went wrong"}
    }
    
    if err := myFunction(); err != nil {
        fmt.Println("Error:", err.Error()) // Prints "Something went wrong"
    }

    By adhering to the error interface, custom errors can be seamlessly integrated into Go’s error-handling mechanisms.

    8. Interface Values and Nil

    Interface values can be nil, indicating they don’t hold any concrete value. However, attempting to call methods on a nil interface value results in a panic.

    var printable Printable // nil interface value
    fmt.Println(printable.String()) // Panics!

    Always check for nil before calling methods on interface values.

    However, it’s important to understand that an interface{} value doesn’t simply hold a reference to the underlying data. Internally, Go creates a special structure to store both the type information and the actual value. This hidden structure is often referred to as “boxing” the value.

    Imagine a small container holding both a label indicating the type (e.g., int, string) and the actual data inside something like this:

    type iface struct {
         tab   *itab
         data  unsafe.Pointer
    }

    Technically, this structure involves two components:

    • tab: This type descriptor carries details like the interface’s method set, the underlying type, and the methods of the underlying type that implement the interface.
    • data pointer: This pointer directly points to the memory location where the actual value resides.

    When you retrieve a value from an interface{}, Go performs “unboxing.” It reads the type information and data pointer and then creates a new variable of the appropriate type based on this information.

    This internal mechanism might seem complex, but the Go runtime handles it seamlessly. However, understanding this concept can give you deeper insights into how Go interfaces work under the hood.

    9. Conclusion

    This journey through the magic of Go interfaces has hopefully provided you with a deeper understanding of their capabilities and how they work. We’ve explored how they go beyond simple method signatures to define contracts, enable dynamic behavior, and making it way more flexible.

    Remember, interfaces are not just tools for code reuse, but also powerful mechanisms for designing adaptable and maintainable applications.

    Here are some key takeaways to keep in mind:

    • Interfaces define contracts, not just method signatures.
    • Interfaces enable dynamic typing and flexible operations.
    • Embedded interfaces allow for hierarchical relationships and code reuse.
    • The empty interface unlocks powerful generic capabilities.
    • Understand the nuances of interface equality and comparisons.
    • Interfaces play a crucial role in Go’s error-handling mechanisms.
    • Be mindful of nil interface values and potential panics.

    10. References

  • Mastering Prow: A Guide to Developing Your Own Plugin for Kubernetes CI/CD Workflow

    Continuous Integration and Continuous Delivery (CI/CD) pipelines are essential components of modern software development, especially in the world of Kubernetes and containerized applications. To facilitate these pipelines, many organizations use Prow, a CI/CD system built specifically for Kubernetes. While Prow offers a rich set of features out of the box, you may need to develop your own plugins to tailor the system to your organization’s requirements. In this guide, we’ll explore the world of Prow plugin development and show you how to get started.

    Prerequisites

    Before diving into Prow plugin development, ensure you have the following prerequisites:

    • Basic Knowledge of Kubernetes and CI/CD Concepts: Familiarity with Kubernetes concepts such as Pods, Deployments, and Services, as well as understanding CI/CD principles, will be beneficial for understanding Prow plugin development.
    • Access to a Kubernetes Cluster: You’ll need access to a Kubernetes cluster for testing your plugins. If you don’t have one already, you can set up a local cluster using tools like Minikube or use a cloud provider’s managed Kubernetes service.
    • Prow Setup: Install and configure Prow in your Kubernetes cluster. You can visit Velotio Technologies – Getting Started with Prow: A Kubernetes-Native CI/CD Framework
    • Development Environment Setup: Ensure you have Git, Go, and Docker installed on your local machine for developing and testing Prow plugins. You’ll also need to configure your environment to interact with your organization’s Prow setup.

    The Need for Custom Prow Plugins

    While Prow provides a wide range of built-in plugins, your organization’s Kubernetes workflow may have specific requirements that aren’t covered by these defaults. This is where developing custom Prow plugins comes into play. Custom plugins allow you to extend Prow’s functionality to cater to your needs. Whether automating workflows, integrating with other tools, or enforcing custom policies, developing your own Prow plugins gives you the power to tailor your CI/CD pipeline precisely.

    Getting Started with Prow Plugin Development

    Developing a custom Prow plugin may seem daunting, but with the right approach and tools, it can be a rewarding experience. Here’s a step-by-step guide to get you started:

    1. Set Up Your Development Environment

    Before diving into plugin development, you need to set up your development environment. You will need Git, Go, and access to a Kubernetes cluster for testing your plugins. Ensure you have the necessary permissions to make changes to your organization’s Prow setup.

    2. Choose a Plugin Type

    Prow supports various plugin types, including postsubmits, presubmits, triggers, and utilities. Choose the type that best fits your use case.

    • Postsubmits: These plugins are executed after the code is merged and are often used for tasks like publishing artifacts or creating release notes.
    • Presubmits: Presubmit plugins run before code is merged, typically used for running tests and ensuring code quality.
    • Triggers: Trigger plugins allow you to trigger custom jobs based on specific events or criteria.
    • Utilities: Utility plugins offer reusable functions and utilities for other plugins.

    3. Create Your Plugin

    Once you’ve chosen a plugin type, it’s time to create it. Below is an example of a simple Prow plugin written in Go, named comment-plugin.go. It will create a comment on a pull request each time an event is received.

    This code sets up a basic HTTP server that listens for GitHub events and handles them by creating a comment using the GitHub API. Customize this code to fit your specific use case.

    package main
    
    import (
        "encoding/json"
        "flag"
        "net/http"
        "os"
        "strconv"
        "time"
    
        "github.com/sirupsen/logrus"
        "k8s.io/test-infra/pkg/flagutil"
        "k8s.io/test-infra/prow/config"
        "k8s.io/test-infra/prow/config/secret"
        prowflagutil "k8s.io/test-infra/prow/flagutil"
        configflagutil "k8s.io/test-infra/prow/flagutil/config"
        "k8s.io/test-infra/prow/github"
        "k8s.io/test-infra/prow/interrupts"
        "k8s.io/test-infra/prow/logrusutil"
        "k8s.io/test-infra/prow/pjutil"
        "k8s.io/test-infra/prow/pluginhelp"
        "k8s.io/test-infra/prow/pluginhelp/externalplugins"
    )
    
    const pluginName = "comment-plugin"
    
    type options struct {
        port int
    
        config                 configflagutil.ConfigOptions
        dryRun                 bool
        github                 prowflagutil.GitHubOptions
        instrumentationOptions prowflagutil.InstrumentationOptions
    
        webhookSecretFile string
    }
    
    type server struct {
        tokenGenerator func() []byte
        botUser        *github.UserData
        email          string
        ghc            github.Client
        log            *logrus.Entry
        repos          []github.Repo
    }
    
    func helpProvider(_ []config.OrgRepo) (*pluginhelp.PluginHelp, error) {
        pluginHelp := &pluginhelp.PluginHelp{
           Description: `The sample plugin`,
        }
        return pluginHelp, nil
    }
    
    func (o *options) Validate() error {
        return nil
    }
    
    func gatherOptions() options {
        o := options{config: configflagutil.ConfigOptions{ConfigPath: "./config.yaml"}}
        fs := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
        fs.IntVar(&o.port, "port", 8888, "Port to listen on.")
        fs.BoolVar(&o.dryRun, "dry-run", false, "Dry run for testing. Uses API tokens but does not mutate.")
        fs.StringVar(&o.webhookSecretFile, "hmac-secret-file", "/etc/hmac", "Path to the file containing GitHub HMAC secret.")
        for _, group := range []flagutil.OptionGroup{&o.github} {
           group.AddFlags(fs)
        }
        fs.Parse(os.Args[1:])
        return o
    }
    
    func main() {
        o := gatherOptions()
        if err := o.Validate(); err != nil {
           logrus.Fatalf("Invalid options: %v", err)
        }
    
        logrusutil.ComponentInit()
        log := logrus.StandardLogger().WithField("plugin", pluginName)
    
        if err := secret.Add(o.webhookSecretFile); err != nil {
           logrus.WithError(err).Fatal("Error starting secrets agent.")
        }
    
        gitHubClient, err := o.github.GitHubClient(o.dryRun)
        if err != nil {
           logrus.WithError(err).Fatal("Error getting GitHub client.")
        }
    
        email, err := gitHubClient.Email()
        if err != nil {
           log.WithError(err).Fatal("Error getting bot e-mail.")
        }
    
        botUser, err := gitHubClient.BotUser()
        if err != nil {
           logrus.WithError(err).Fatal("Error getting bot name.")
        }
        repos, err := gitHubClient.GetRepos(botUser.Login, true)
        if err != nil {
           log.WithError(err).Fatal("Error listing bot repositories.")
        }
        serv := &server{
           tokenGenerator: secret.GetTokenGenerator(o.webhookSecretFile),
           botUser:        botUser,
           email:          email,
           ghc:            gitHubClient,
           log:            log,
           repos:          repos,
        }
    
        health := pjutil.NewHealthOnPort(o.instrumentationOptions.HealthPort)
        health.ServeReady()
    
        mux := http.NewServeMux()
        mux.Handle("/", serv)
        externalplugins.ServeExternalPluginHelp(mux, log, helpProvider)
        logrus.Info("starting server " + strconv.Itoa(o.port))
        httpServer := &http.Server{Addr: ":" + strconv.Itoa(o.port), Handler: mux}
        defer interrupts.WaitForGracefulShutdown()
        interrupts.ListenAndServe(httpServer, 5*time.Second)
    }
    
    func (s *server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        logrus.Info("inside http server")
        _, _, payload, ok, _ := github.ValidateWebhook(w, r, s.tokenGenerator)
        logrus.Info(string(payload))
        if !ok {
           return
        }
        logrus.Info(w, "Event received. Have a nice day.")
        if err := s.handleEvent(payload); err != nil {
           logrus.WithError(err).Error("Error parsing event.")
        }
    }
    
    func (s *server) handleEvent(payload []byte) error {
        logrus.Info("inside handler")
        var pr github.PullRequestEvent
        if err := json.Unmarshal(payload, &pr); err != nil {
           return err
        }
        logrus.Info(pr.Number)
        if err := s.ghc.CreateComment(pr.PullRequest.Base.Repo.Owner.Login, pr.PullRequest.Base.Repo.Name, pr.Number, "comment from smaple-plugin"); err != nil {
           return err
        }
        return nil
    }

    4. Deploy Your Plugin

    To deploy your custom Prow plugin, you will need to create a Docker image and deploy it into your Prow cluster.

    FROM golang as app-builder
    WORKDIR /app
    RUN apt  update
    RUN apt-get install git
    COPY . .
    RUN CGO_ENABLED=0 go build -o main
    
    FROM alpine:3.9
    RUN apk add ca-certificates git
    COPY --from=app-builder /app/main /app/custom-plugin
    ENTRYPOINT ["/app/custom-plugin"]

    docker build -t jainbhavya65/custom-plugin:v1 .

    docker push jainbhavya65/custom-plugin:v1

    Deploy the Docker image using Kubernetes deployment:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: comment-plugin
    spec:
      progressDeadlineSeconds: 600
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          app: comment-plugin
      strategy:
        rollingUpdate:
          maxSurge: 25%
          maxUnavailable: 25%
        type: RollingUpdate
      template:
        metadata:
          creationTimestamp: null
          labels:
            app: comment-plugin
        spec:
          containers:
          - args:
            - --github-token-path=/etc/github/oauth
            - --hmac-secret-file=/etc/hmac-token/hmac
            - --port=80
            image: <IMAGE>
            imagePullPolicy: Always
            name: comment-plugin
            ports:
            - containerPort: 80
              protocol: TCP
            volumeMounts:
            - mountPath: /etc/github
              name: oauth
              readOnly: true
            - mountPath: /etc/hmac-token
              name: hmac
              readOnly: true
          volumes:
          - name: oauth
            secret:
              defaultMode: 420
              secretName: oauth-token
          - name: hmac
            secret:
              defaultMode: 420
              secretName: hmac-token

    Create a service for deployment:
    apiVersion: v1
    kind: Service
    metadata:
      name: comment-plugin
    spec:
      ports:
      - port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: comment-plugin
      sessionAffinity: None
      type: ClusterIP
    view raw

    After creating the deployment and service, integrate it into your organization’s Prow configuration. This involves updating your Prow plugin.yaml files to include your plugin and specify when it should run.

    external_plugins: 
    - name: comment-plugin
      # No endpoint specified implies "http://{{name}}". // as we deploy plugin into same cluster
      # if plugin is not deployed in same cluster then you can give endpoint
      events:
      # only pull request and issue comment events are send to our plugin
      - pull_request
      - issue_comment

    Conclusion

    Mastering Prow plugin development opens up a world of possibilities for tailoring your Kubernetes CI/CD workflow to meet your organization’s needs. While the initial learning curve may be steep, the benefits of custom plugins in terms of automation, efficiency, and control are well worth the effort.

    Remember that the key to successful Prow plugin development lies in clear documentation, thorough testing, and collaboration with your team to ensure that your custom plugins enhance your CI/CD pipeline’s functionality and reliability. As Kubernetes and containerized applications continue to evolve, Prow will remain a valuable tool for managing your CI/CD processes, and your custom plugins will be the secret sauce that sets your workflow apart from the rest.

  • The Ultimate Guide to Disaster Recovery for Your Kubernetes Clusters

    Kubernetes allows us to run a containerized application at scale without drowning in the details of application load balancing. You can ensure high availability for your applications running on Kubernetes by running multiple replicas (pods) of the application. All the complexity of container orchestrations is hidden away safely so that you can focus on developing application instead of deploying it. Learn more about high availability of Kubernetes Clusters and how you can use Kubedm for high availability in Kubernetes here.

    But using Kubernetes has its own challenges and getting Kubernetes up and running takes some real work. If you are not familiar with getting Kubernetes up and running, you might want to take a look here.

    Kubernetes allows us to have a zero downtime deployment, yet service interrupting events are inevitable and can occur at any time. Your network can go down, your latest application push can introduce a critical bug, or in the rarest case, you might even have to face a natural disaster.

    When you are using Kubernetes, sooner or later, you need to set up a backup. In case your cluster goes into an unrecoverable state, you will need a backup to go back to the previous stable state of the Kubernetes cluster.

    Why Backup and Recovery?

    There are three reasons why you need a backup and recovery mechanism in place for your Kubernetes cluster. These are:

    1. To recover from Disasters: like someone accidentally deleted the namespace where your deployments reside.
    2. Replicate the environment: You want to replicate your production environment to staging environment before any major upgrade.
    3. Migration of Kubernetes Cluster: Let’s say, you want to migrate your Kubernetes cluster from one environment to another.

    What to Backup?

    Now that you know why, let’s see what exactly do you need to backup. The two things you need to backup are:

    1. Your Kubernetes control plane is stored into etcd storage and you need to backup the etcd state to get all the Kubernetes resources.
    2. If you have stateful containers (which you will have in real world), you need a backup of persistent volumes as well.

    How to Backup?

    There have been various tools like Heptio ark and Kube-backup to backup and restore the Kubernetes cluster for cloud providers. But, what if you are not using managed Kubernetes cluster? You might have to get your hands dirty if you are running Kubernetes on Baremetal, just like we are.

    We are running 3 master Kubernetes cluster with 3 etcd members running on each master. If we lose one master, we can still recover the master because etcd quorum is intact. Now if we lose two masters, we need a mechanism to recover from such situations as well for production grade clusters.

    Want to know how to set up multi-master Kubernetes cluster? Keep reading!

    Taking etcd backup:

    There is a different mechanism to take etcd backup depending on how you set up your etcd cluster in Kubernetes environment.

    There are two ways to setup etcd cluster in kubernetes environment:

    1. Internal etcd cluster: It means you’re running your etcd cluster in the form of containers/pods inside the Kubernetes cluster and it is the responsibility of Kubernetes to manage those pods.
    2. External etcd cluster: Etcd cluster you’re running outside of Kubernetes cluster mostly in the form of Linux services and providing its endpoints to Kubernetes cluster to write to.

    Backup Strategy for Internal Etcd Cluster:

    To take a backup from inside a etcd pod, we will be using Kubernetes CronJob functionality which will not require any etcdctl client to be installed on the host.

    Following is the definition of Kubernetes CronJob which will take etcd backup every minute:

    `apiVersion: batch/v1beta1kind: CronJobmetadata: name: backup namespace: kube-systemspec: # activeDeadlineSeconds: 100schedule: "*/1 * * * *"
    jobTemplate:
    spec:
    template:
    spec:
    containers:
    - name: backup
    # Same image as in /etc/kubernetes/manifests/etcd.yaml
    image: k8s.gcr.io/etcd:3.2.24
    env:
    - name: ETCDCTL_API
    value: "3"
    command: ["/bin/sh"]
    args: ["-c", "etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d_%H:%M:%S_%Z).db"]
    volumeMounts:
    - mountPath: /etc/kubernetes/pki/etcd
    name: etcd-certs
    readOnly: true
    - mountPath: /backup
    name: backup
    restartPolicy: OnFailure
    hostNetwork: true
    volumes:
    - name: etcd-certs
    hostPath:
    path: /etc/kubernetes/pki/etcd
    type: DirectoryOrCreate
    - name: backup
    hostPath:
    path: /data/backup
    type: DirectoryOrCreate

    Backup Strategy for External Etcd Cluster:

    If you running etcd cluster on Linux hosts as a service, you should set up a Linux cron job to take backup of your cluster.

    Run the following command to save etcd backup

    ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save /path/for/backup/snapshot.db

    Disaster Recovery

    Now, Let’s say the Kubernetes cluster went completely down and we need to recover the Kubernetes cluster from the etcd snapshot.

    Normally, start the etcd cluster and do the kubeadm init on the master node with etcd endpoints.

    Make sure you put the backup certificates into /etc/kubernetes/pki folder before kubeadm init. It will pick up the same certificates.

    Restore Strategy for Internal Etcd Cluster:

    docker run --rm 
    -v '/data/backup:/backup' 
    -v '/var/lib/etcd:/var/lib/etcd' 
    --env ETCDCTL_API=3 
    k8s.gcr.io/etcd:3.2.24' 
    /bin/sh -c "etcdctl snapshot restore '/backup/etcd-snapshot-2018-12-09_11:12:05_UTC.db' ; mv /default.etcd/member/ /var/lib/etcd/"
    
    kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcd

    Restore Strategy for External Etcd Cluster

    Restore the etcd on 3 nodes using following commands:

    ETCDCTL_API=3 etcdctl snapshot restore snapshot-188.db 
    --name master-0 
    --initial-cluster master-0=http://10.0.1.188:2380,master-01=http://10.0.1.136:2380,master-2=http://10.0.1.155:2380 
    --initial-cluster-token my-etcd-token 
    --initial-advertise-peer-urls http://10.0.1.188:2380
    
    ETCDCTL_API=3 etcdctl snapshot restore snapshot-136.db 
    --name master-1 
    --initial-cluster master-0=http://10.0.1.188:2380,master-1=http://10.0.1.136:2380,master-2=http://10.0.1.155:2380 
    --initial-cluster-token my-etcd-token 
    --initial-advertise-peer-urls http://10.0.1.136:2380
    
    ETCDCTL_API=3 etcdctl snapshot restore snapshot-155.db 
    --name master-2 
    --initial-cluster master-0=http://10.0.1.188:2380,master-1=http://10.0.1.136:2380,master-2=http://10.0.1.155:2380 
    --initial-cluster-token my-etcd-token 
    --initial-advertise-peer-urls http://10.0.1.155:2380

    The above three commands will give you three restored folders on three nodes named master:

    0.etcd, master-1.etcd and master-2.etcd

    Now, Stop all the etcd service on the nodes, replace the restored folder with the restored folders on all nodes and start the etcd service. Now you can see all the nodes, but in some time you will see that only master node is ready and other nodes went into the not ready state. You need to join those two nodes again with the existing ca.crt file (you should have a backup of that).

    Run the following command on master node:

    kubeadm token create --print-join-command

    It will give you kubeadm join command, add one –ignore-preflight-errors and run that command on other two nodes for them to come into the ready state.

    Conclusion

    One way to deal with master failure is to set up multi-master Kubernetes cluster, but even that does not allow you to completely eliminate the Kubernetes etcd backup and restore, and it is still possible that you may accidentally destroy data on the HA environment.

    Need help with disaster recovery for your Kubernetes Cluster? Connect with the experts at Velotio!

    For more insights into Kubernetes Disaster Recovery check out here.

  • How Healthcare Payers Can Leverage Speech Analytics to Generate Value

    Speech Analytics:

    The world has entered into an unprecedented age of information and technology, wherein developing a robust patient experience roadmap has become indispensable. Payers are being incentivized to develop industry-leading skills and strategies that are at par with the changing patient needs and expectations. In order to establish strong footprint in the market, Healthcare organizations must record and monitor patients’ interaction across all the breadth of channels. Equitably, organizations need to ascertain strict adherence to privacy laws to curb fraudulent attempts and practice efficiency. Right now, the focus should be largely emphasized on whether plan enrollees are getting meaningful and swift access to the services they are seeking.

    Key learnings from the whitepaper:
    • Speech Analytics Solution vs. Traditional Call Drivers Evaluating Method
    • Speech Analytics: A Booming Technology
    • How Organizations are Leveraging this Opportunity to Maximize Value
    • A Connotation to ‘WHY’ Makes a Big Difference
    • How R Systems’ Anagram Cuts the Mustard
  • Simplifying MySQL Sharding with ProxySQL: A Step-by-Step Guide

    Introduction:

    ProxySQL is a powerful SQL-aware proxy designed to sit between database servers and client applications, optimizing database traffic with features like load balancing, query routing, and failover. This article focuses on simplifying the setup of ProxySQL, especially for users implementing data-based sharding in a MySQL database.

    What is Sharding?

    Sharding involves partitioning a database into smaller, more manageable pieces called shards based on certain criteria, such as data attributes. ProxySQL supports data-based sharding, allowing users to distribute data across different shards based on specific conditions.

    Understanding the Need for ProxySQL:

    ProxySQL is an intermediary layer that enhances database management, monitoring, and optimization. With features like data-based sharding, ProxySQL is an ideal solution for scenarios where databases need to be distributed based on specific data attributes, such as geographic regions.

    ‍Installation & Setup:‍

    There are two ways to install the proxy, either by installing it using packages or running  ProxySQL in docker. ProxySQL can be installed using two methods: via packages or running it in a Docker container. For this guide, we will focus on the Docker installation.

    1. Install ProxySQL and MySQL Docker Images:

    To start, pull the necessary Docker images for ProxySQL and MySQL using the following commands:

    docker pull mysql:latest
    docker pull proxysql/proxysql

    2. Create Docker Network:

    Create a Docker network for communication between MySQL containers:

    docker network create multi-tenant-network

    Note: ProxySQL setup will need connections to multiple SQL servers. So, we will set up multiple SQL servers on our docker inside a Docker network.

    Containers within the same Docker network can communicate with each other using their container names or IP addresses.

    You can check the list of all the Docker networks currently present by running the following command:

    docker network ls

    3. Set Up MySQL Containers:

    Now, create three MySQL containers within the network:

    Note: We can create any number of MySQL containers.

    docker run -d --name mysql_host_1 --network=multi-tenant-network -p 3307:3306 -e MYSQL_ROOT_PASSWORD=pass123 mysql:latest 
    docker run -d --name mysql_host_2 --network=multi-tenant-network -p 3308:3306 -e MYSQL_ROOT_PASSWORD=pass123 mysql:latest 
    docker run -d --name mysql_host_3 --network=multi-tenant-network -p 3309:3306 -e MYSQL_ROOT_PASSWORD=pass123 mysql:latest

    Note: Adjust port numbers as necessary. 

    The default MySQL protocol port is 3306, but since we cannot access all three of our MySQL containers on the same port, we have set their ports to 3307, 3308, and 3309. Although internally, all MySQL containers will connect using port 3306.

    –network=multi-tenant-network. This specifies that the container should be created under the specified network.

    We have also specified the root password of the MySQL container to log into it, where the username is “root” and the password is “pass123” for all three of them.

    After running the above three commands, three MySQL containers will start running inside the network. You can connect to these three hosts using host = localhost or 127.0.0.1 and port = 3307 / 3308 / 3309.

    To ping the port, use the following command:

    for macOS:

    nc -zv 127.0.0.1 3307

    for Windows: 

    ping 127.0.0.1 3307

    for Linux: 

    telnet 127.0.0.1 3307

    Reference Image

    4. Create Users in MySQL Containers:

    Create “user_shard” and “monitor” users in each MySQL container.

    The “user_shard” user will be used by the proxy to make queries to the DB.

    The “monitor” user will be used by the proxy to monitor the DB.

    Note: To access the MySQL container mysql_host_1, use the command:

    docker exec -it mysql_host_1 mysql -uroot -ppass123

    Use the following commands inside the MySQL container to create the user:

    CREATE USER 'user_shard'@'%' IDENTIFIED BY 'pass123'; 
    GRANT ALL PRIVILEGES ON *.* TO 'user_shard'@'%' WITH GRANT OPTION; 
    FLUSH PRIVILEGES;
    
    CREATE USER monitor@'%' IDENTIFIED BY 'pass123'; 
    GRANT ALL PRIVILEGES ON *.* TO monitor@'%' WITH GRANT OPTION; 
    FLUSH PRIVILEGES;

    Repeat the above steps for mysql_host_2 & mysql_host_3.

    If, at any point, you need to drop the user, you can use the following command:

    DROP USER monitor@’%’;

    5. Prepare ProxySQL Configuration:

    To prepare the configuration, we will need the IP addresses of the MySQL containers. To find those, we can use the following command:

    docker inspect mysql_host_1;
    docker inspect mysql_host_2; 
    docker inspect mysql_host_3;

    By running these commands, you will get all the details of the MySQL Docker container under a field named “IPAddress” inside your network. That is the IP address of that particular MySQL container.

    Example:
    mysql_host_1: 172.19.0.2

    mysql_host_2: 172.19.0.3

    mysql_host_3: 172.19.0.4

    Reference image for IP address of mysql_host_1: 172.19.0.2

    Now, create a ProxySQL configuration file named proxysql.cnf. Include details such as IP addresses of MySQL containers, administrative credentials, and MySQL users.

    Below is the content that needs to be added to the proxysql.cnf file:

    datadir="/var/lib/proxysql"
    
    admin_variables=
    {
        admin_credentials="admin:admin;radmin:radmin"
        mysql_ifaces="0.0.0.0:6032"
        refresh_interval=2000
        hash_passwords=false
    }
    
    mysql_variables=
    {
        threads=4
        max_connections=2048
        default_query_delay=0
        default_query_timeout=36000000
        have_compress=true
        poll_timeout=2000
        interfaces="0.0.0.0:6033;/tmp/proxysql.sock"
        default_schema="information_schema"
        stacksize=1048576
        server_version="5.1.30"
        connect_timeout_server=10000
        monitor_history=60000
        monitor_connect_interval=200000
        monitor_ping_interval=200000
        ping_interval_server_msec=10000
        ping_timeout_server=200
        commands_stats=true
        sessions_sort=true
        monitor_username="monitor"
        monitor_password="pass123"
    }
    
    mysql_servers =
    (
        { address="172.19.0.2" , port=3306 , hostgroup=10, max_connections=100 },
        { address="172.19.0.3" , port=3306 , hostgroup=20, max_connections=100 },
        { address="172.19.0.4" , port=3306 , hostgroup=30, max_connections=100 }
    )
    
    
    mysql_users =
    (
        { username = "user_shard" , password = "pass123" , default_hostgroup = 10 , active = 1 },
        { username = "user_shard" , password = "pass123" , default_hostgroup = 20 , active = 1 },
        { username = "user_shard" , password = "pass123" , default_hostgroup = 30 , active = 1 }
    )

    Most of the settings are default; we won’t go into much detail for each setting. 

    admin_variables: These variables are used for ProxySQL’s administrative interface. It allows you to connect to ProxySQL and perform administrative tasks such as configuring runtime settings, managing servers, and monitoring performance.

    mysql_variables, monitor_username, and monitor_password are used to specify the username that ProxySQL will use when connecting to MySQL servers for monitoring purposes. This monitoring user is used to execute queries and gather statistics about the health and performance of the MySQL servers. This is the user we created during step 4.

    mysql_servers will contain all the MySQL servers we want to be connected with ProxySQL. Each entry will have the IP address of the MySQL container, port, host group, and max_connections. Mysql_users will have all the users we created during step 4.

    7. Run ProxySQL Container:

    Inside the same directory where the proxysql.cnf file is located, run the following command to start ProxySQL:

    docker run -d --rm -p 6032:6032 -p 6033:6033 -p 6080:6080 --name=proxysql --network=multi-tenant-network -v $PWD/proxysql.cnf:/etc/proxysql.cnf proxysql/proxysql

    Here, port 6032 is used for ProxySQL’s administrative interface. It allows you to connect to ProxySQL and perform administrative tasks such as configuring runtime settings, managing servers, and monitoring performance.

    Port 6033 is the default port for ProxySQL’s MySQL protocol interface. It is used for handling MySQL client connections. Our application will use it to access the ProxySQL db and make SQL queries.

    The above command will make ProxySQL run on our Docker with the configuration provided in the proxysql.cnf file.

    Inside ProxySQL Container:

    8. Access ProxySQL Admin Console:

    Now, to access the ProxySQL Docker container, use the following command:

    docker exec -it proxysql bash

    Now, once you’re inside the ProxySQL Docker container, you can access the ProxySQL admin console using the command:

    mysql -u admin -padmin -h 127.0.0.1 -P 6032

    You can run the following queries to get insights into your ProxySQL server:

    i) To get the list of all the connected MySQL servers:

    SELECT * FROM mysql_servers;

    ii) Verify the status of the MySQL backends in the monitor database tables in ProxySQL admin using the following command:

    SHOW TABLES FROM monitor;


    If this returns an empty set, it means that the monitor username and password are not set correctly. You can do so by using the below commands:

    UPDATE global_variables SET variable_value=’monitor’ WHERE variable_name='mysql-monitor_username'; 
    UPDATE global_variables SET variable_value=’pass123’ WHERE variable_name='mysql-monitor_password';
    LOAD MYSQL VARIABLES TO RUNTIME; 
    SAVE MYSQL VARIABLES TO DISK;

    And then restart the proxy Docker container:

    iii) Check the status of DBs connected to ProxySQL using the following command:

    SELECT * FROM monitor.mysql_server_connect_log ORDER BY time_start_us DESC;

    iv) To get a list of all the ProxySQL global variables, use the following command:

    SELECT * FROM global_variables; 

    v) To get all the queries made on ProxySQL, use the following command:

    Select * from stats_mysql_query_digest;

    Note: Whenever we change any row, use the below commands to load them:

    Change in variables:

    LOAD MYSQL VARIABLES TO RUNTIME; 
    SAVE MYSQL VARIABLES TO DISK;
    
    Change in mysql_servers:
    LOAD MYSQL SERVERS TO RUNTIME;
    SAVE MYSQL SERVERS TO DISK;
    
    Change in mysql_query_rules:
    LOAD MYSQL QUERY RULES TO RUNTIME;
    SAVE MYSQL QUERY RULES TO DISK;

    And then restart the proxy docker container.

    IMPORTANT:

    To connect to ProxySQL’s admin console, first get into the Docker container using the following command:

    docker exec -it proxysql bash

    Then, to access the ProxySQL admin console, use the following command:

    mysql -u admin -padmin -h 127.0.0.1 -P6032

    To access the ProxySQL MySQL console, we can directly access it using the following command without going inside the Docker ProxySQL container:

    mysql -u user_shard -ppass123 -h 127.0.0.1 -P6033

    To make queries to the database, we make use of ProxySQL’s 6033 port, where MySQL is being accessed.

    9. Define Query Rules:

    We can add custom query rules inside the mysql_query_rules table to redirect queries to specific databases based on defined patterns. Load the rules to runtime and save to disk.

    12. Sharding Example:

    Now, let’s illustrate how to leverage ProxySQL’s data-based sharding capabilities through a practical example. We’ll create three MySQL containers, each containing data from different continents in the “world” database, specifically within the “countries” table.

    Step 1: Create 3 MySQL containers named mysql_host_1, mysql_host_2 & mysql_host_3.

    Inside all containers, create a database named “world” with a table named “countries”.

    i) Inside mysql_host_1: Insert countries using the following query:

    INSERT INTO `countries` VALUES (1,'India','Asia'),(2,'Japan','Asia'),(3,'China','Asia'),(4,'USA','North America'),(5,'Cuba','North America'),(6,'Honduras','North America');

    ii) Inside mysql_host_2: Insert countries using the following query:

    INSERT INTO `countries` VALUES (1,'Kenya','Africa'),(2,'Ghana','Africa'),(3,'Morocco','Africa'),(4, "Brazil", "South America"), (5, "Chile", "South America"), (6, "Morocco", "South America");

    iii) Inside mysql_host_3: Insert countries using the following query:

    CODE: INSERT INTO `countries` VALUES (1, “Italy”, “Europe”), (2, “Germany”, “Europe”), (3, “France”, “Europe”);

    Now, we have distinct data sets for Asia & North America in mysql_host_1, Africa & South America in mysql_host_2, and Europe in mysql_host_3..js

    Step 2: Define Query Rules for Sharding

    Let’s create custom query rules to redirect queries based on the continent specified in the SQL statement.

    For example, if the query contains the continent “Asia,” we want it to be directed to mysql_host_1.

    — Query Rule for Asia and North America 

    INSERT INTO mysql_query_rules (rule_id, active, username, match_pattern, destination_hostgroup, apply) VALUES (10, 1, 'user_shard', "s*continents*=s*.*?(Asia|North America).*?s*", 10, 0);

    — Query Rule for Africa and South America

    INSERT INTO mysql_query_rules (rule_id, active, username, match_pattern, destination_hostgroup, apply) VALUES (20, 1, 'user_shard', "s*continents*=s*.*?(Africa|South America).*?s*", 20, 0);

    — Query Rule for Europe 

    INSERT INTO mysql_query_rules (rule_id, active, username, match_pattern, destination_hostgroup, apply) VALUES (30, 1, 'user_shard', "s*continents*=s*.*?(Europe).*?s*", 30, 0);

    Step 3: Apply and Save Query Rules

    After adding the query rules, ensure they take effect by running the following commands:

    LOAD MYSQL QUERY RULES TO RUNTIME; 
    SAVE MYSQL QUERY RULES TO DISK;

    Step 4: Test Sharding

    Now, access the MySQL server using the ProxySQL port and execute queries:

    mysql -u user_shard -ppass123 -h 127.0.0.1 -P 6033

    use world;

    — Example Queries:

    Select * from countries where id = 1 and continent = "Asia";

    — This will return id=1, name=India, continent=Asia

    Select * from countries where id = 1 and continent = "Africa";

    — This will return id=1, name=Kenya, continent=Africa.

    Select * from countries where id = 1 and continent = "Africa";

    Based on the defined query rules, the queries will be redirected to the specified MySQL host groups. If no rules match, the default host group that’s specified in mysql_users inside proxysql.cnf will be used.

    Conclusion:

    ProxySQL simplifies access to distributed data through effective sharding strategies. Its flexible query rules, combined with regex patterns and host group definitions, offer significant flexibility with relative simplicity.

    By following this step-by-step guide, users can quickly set up ProxySQL and leverage its capabilities to optimize database performance and achieve efficient data distribution.

    References:

    Download and Install ProxySQL – ProxySQL

    How to configure ProxySQL for the first time – ProxySQL

    Admin Variables – ProxySQL

  • Beginner’s Guide for Writing Unit Test Cases with Jest Framework

    ‍Prerequisite

    Basic JavaScript, TypeScript

    Objective

    To make the reader understand the use/effect of test cases in software development.

    What’s in it for you?‍

    In the world of coding, we’re often in a rush to complete work before a deadline hits. And let’s be honest, writing test cases isn’t usually at the top of our priority list. We get it—they seem tedious, so we’d rather skip this extra step. But here’s the thing: those seemingly boring lines of code have superhero potential. Don’t believe me? You will.

    In this blog, we’re going to break down the mystery around test cases. No jargon, just simple talk. We’ll chat about what they are, explore a handy tool called Jest, and uncover why these little lines are actually the unsung heroes of coding. So, let’s ditch the complications and discover why giving some attention to test cases can level up our coding game. Ready? Let’s dive in!

    What are test cases?

    A test case is a detailed document specifying conditions under which a developer assesses whether a software application aligns with customer requirements. It includes preconditions, the case name, input conditions, and expected results. Derived from test scenarios, test cases cover both positive and negative inputs, providing a roadmap for test execution. This one-time effort aids future regression testing.

    Test cases offer insights into testing strategy, process, preconditions, and expected outputs. Executed during testing, they ensure the software performs its intended tasks. Linking defects to test case IDs facilitates efficient defect reporting. The comprehensive documentation acts as a safeguard, catching any oversights during test case execution and reinforcing the development team’s efforts.

    Different types of test cases exist, including integration, functional, non-functional, and unit.
    For this blog, we will talk about unit test cases.

    What are unit test cases?

    Unit testing is the process of testing the smallest functional unit of code. A functional unit could be a class member or simply a function that does something to your input and provides an output. Test cases around those functional units are called unit test cases.

    Purpose of unit test cases

    • To validate that each unit of the software works as intended and meets the requirements:
      For example, if your requirement is that the function returns an object with specific properties, a unit test will detect whether the code is written accordingly.
    • To check the robustness of code:
      Unit tests are automated and run each time the code is changed to ensure that new code does not break existing functionality.
    • To check the errors and bugs beforehand:
      If a case fails or doesn’t fulfill the requirement, it helps the developer isolate the area and recheck it for bugs before testing on demo/UAT/staging.

    Different frameworks for writing unit test cases

    There are various frameworks for unit test cases, including:

    • Mocha
    • Storybook
    • Cypress
    • Jasmine
    • Puppeteer
    • Jest
    Source: https://raygun.com/blog/javascript-unit-testing-frameworks/

    Why Jest?

    Jest is used and recommended by Facebook and officially supported by the React dev team.

    It has a great community and active support, so if you run into a problem and can’t find a solution in the comprehensive documentation, there are thousands of developers out there who could help you figure it out within hours.

    1. Performance: Ideal for larger projects with continuous deployment needs, Jest delivers enhanced performance.

    2. Compatibility: While Jest is widely used for testing React applications, it seamlessly integrates with other frameworks like Angular, Node, Vue, and Babel-based projects.

    3. Auto Mocking: Jest automatically mocks imported libraries in test files, reducing boilerplate and facilitating smoother testing workflows.

    4. Extended API: Jest comes with a comprehensive API, eliminating the necessity for additional libraries in most cases.

    5. Timer Mocks: Featuring a Time mocking system, Jest accelerates timeout processes, saving valuable testing time.

    6. Active Development & Community: Jest undergoes continuous improvement, boasting the most active community support for rapid issue resolution and updates.

    Components of a test case in Jest‍

    Describe

    • As the name indicates, they are responsible for describing the module we are going to test.
    • It should only describe the module, not the tests, as this describe module is generally not tested by Jest.

    It

    • Here, the actual code is tested and verified with actual or fake (spy, mocks) outputs.
      We can nest various it modules under the describe module.
    • It’s good to describe what the test does or doesn’t do in the description of the it module.

    Matchers

    • Matchers match the output with a real/fake output.
    • A test case without a matcher will always be a true/trivial test case.
    // For each unit test you write,
    // answer these questions:
    
    describe('What component aspect are you testing?', () => {
        it('What should the feature do?', () => {
            const actual = 'What is the actual output?'
            const expected = 'What is the expected output?'
    
            expect(actual).toEqual(expected) // matcher
    
        })
      })

    ‍Mocks and spies in Jest

    Mocks: They are objects or functions that simulate the behavior of real components. They are used to create controlled environments for testing by replacing actual components with simulated ones. Mocks are employed to isolate the code being tested, ensuring that the test focuses solely on the unit or component under examination without interference from external dependencies.

    It is mainly used for mocking a library or function that is most frequently used in the whole file or unit test case.

    Let Code.ts be the file you want to test.

    import { v4 as uuidv4 } from uuid
    
    export const functionToTest = () => {
    
        const id = uuidv4()
        // rest of the code
        return id;
    
    }

    As this is a unit test, we won’t be testing the uuidV4 function, so we will mock the whole uuid module using jest.mock.

    jest.mock('uuid', () => { uuidv4: () => 'random id value' }))  // mocking uuid module which will have uuidV4 as function
    describe('testing code.ts', () => {
        it('i have mocked uuid module', ()=> {
    
        const res = functionToTest()
        expect(res).tobeEqual('random id value')
    })
    
    })

    And that’s it. You have mocked the entire uuid module, so when it is coded during a test, it will return uuidV4 function, and that function, when executed, will give a random id value.

    Spies: They are functions or objects that “spy” on other functions by tracking calls made to them. They allow you to observe and verify the behavior of functions during testing. Spies are useful for checking if certain functions are called, how many times they are called, and with what arguments. They help ensure that functions are interacting as expected.

    This is by far the most used method, as this method works on object values and thus can be used to spy class methods efficiently.

    class DataService {
        fetchData() 
        {
            // code to fetch data
            return { 'real data'}
        }
    }

    describe('DataService Class', () => {
    
        it('should spy on the fetchData method with mockImplementation', () => {
            const dataServiceInstance = new DataService();
            const fetchDataSpy = jest.spyon(DataService.prototype, 'fetchData'); // prototype makes class method to a object
            fetchDataSpy.mockImplementation(() => 'Mocked Data'); // will return mocked data whenever function will be called
    
            const result = dataServiceInstance.fetchData(); // mocked Data
            expect(fetchDataSpy).toHaveBeenCalledTimes(1)
            expect(result).toBe('Mocked Data');
        }
      
      }

    Mocking database call‍

    One of the best uses of Jest is to mock a database call, i.e., mocking create, put, post, and delete calls for a database table.

    We can complete the same action with the help of only Jest spies.

    Let us suppose we have a database called DB, and it has lots of tables in it. Let’s say it has Table Student in it, and we want to mock create a Student database call.

    function async AddStudent(student: Student) 
      {
            await db.Student.create(student) // the call we want to mock
     }

    Now, as we are using the Jest spy method, we know that it will only be applicable to objects, so we will first make the Db. Students table into an object with create as method inside it, which will be jest.fn() (a function which can be used for mocking functions).

    Students an object with create as method inside object which will be jest.fn() (a function which can be used for mocking functions in one line without actually calling that function).

    describe('mocking data base call', () => {
            it('mocking create function', async () => {
                db.Student = {
                    create: jest.fn()
                }
    
                const tempStudent = {
                    name: 'john',
                    age: '12',
                    Rollno: 12
                     }
    
                const mock = jest.spyon(db.Student, 'create').
                    mockResolvedvalue('Student has been created successfully')
    
                await AddStudent(tempStudent)
                expect(mock).tohaveBeenCalledwith(tempStudent);
    
            })
    
        })

    Testing private methods‍

    Sometime, in development, we write private code for classes that can only be used within the class itself. But when writing test cases, we call the function by creating a class instance, and the private functions won’t be accessible to us, so we will not be able to test private functions.

    But in core JavaScript, there is no concept of private and public functions; it is introduced to us as TypeScript. So, we can actually test the private function as a normal public function by using the //@ts-ignore comment just above calling the private function.

     class Test()
      {
    
            private private_fun() {
                console.log("i am in private function");
                return "i am in private function"
            }
    
        }

    describe('Testing test class', () => {
            it('testing private function', () => {
                const test = new Test() 
                
                //calling code with ts-ignore comment
    
                //@ts-ignore
                const res = test.private_fun() //  output ->> "i am in private function "//
                expect(res).toBeEqual("i am in private function")
    
            })
        })

    P.S. One thing to note is that this will only work with TypeScript/JavaScript files.

    The importance of test cases in software development

    Makes code agile:

    In software development, one may have to change the structure or design of your code to add new features. Changing the already-tested code can be risky and costly. When you do the unit test, you just need to test the newly added code instead of the entire program.

    Improves code quality:

    A lot of bugs in software development occur due to unforeseen edge cases. If you forget to predict a single input, you may encounter a major bug in your application. When you write unit tests, think carefully about the edge cases of every function in your application.

    Provides Documentation:

    The unit test gives a basic idea of what the code does, and all the different use cases are covered through the program. It makes documentation easier, increasing the readability and understandability of the code. Anytime other developers can go through the unit test interface, understand the program better, and work on it fast and easily.

    Easy Debugging:

    Unit testing has made debugging a lot easier and quicker. If the test fails at any stage, you only need to debug the latest changes made in the code instead of the entire program. We have also mentioned how unit testing makes debugging easier at the next stage of integration testing as well.

    Conclusion

    So, if you made it to the end, you must have some understanding of the importance of test cases in your code.

    We’ve covered the best framework to choose from and how to write your first test case in Jest. And now, you are more confident in proving bug-free, robust, clean, documented, and tested code in your next MR/PR.

  • A Comprehensive Guide to Unlock Kafka MirrorMaker 2.0

    Overview

    We are covering how Kafka MirrorMaker operates, how to set it up, and how to test mirror data.    

    MirrorMaker 2.0 is the new replication feature of Kafka 2.4, defined as part of the Kafka Improvement Process – KIP 382. Kafka MirrorMaker 2 is designed to replicate or mirror topics from one Kafka cluster to another. It uses the Kafka Connect framework to simplify the configuration and scaling. MirrorMaker dynamically detects changes to source topics and ensures source and target topic properties are synchronized, including topic data, offsets, and partitions. The topic, together with topic data, offsets, and partitions, is replicated in the target cluster when a new topic is created in the source cluster.

    Use Cases

    Disaster Recovery

    Though Kafka is highly distributed and provides a high level of fault tolerance, disasters can still happen, and data can still become temporarily unavailable—or lost altogether. The best way to mitigate the risks is to have a copy of your data in another Kafka cluster in a different data center. MirrorMaker translates and syncs consumer offsets to the target cluster. That way, we can switch clients to it relatively seamlessly, moving to an alternative deployment on the fly with minor or no service interruptions.

    Closer Read / Writes

    Kafka producer clients often prefer to write locally to achieve low latency, but business requirements demand the data be read by different consumers, often deployed in multiple regions. This can easily make deployments complex due to VPC peering. MirrorMaker can handle all complex replication, making it easier to write and read local mechanisms.

    Data Analytics

    Aggregation is also a factor in data pipelines, which might require the consolidation of data from regional Kafka clusters into a single one. That aggregate cluster then broadcasts that data to other clusters and/or data systems for analysis and visualization.

    Supported Topologies

    • Active/Passive or Active/Standby high availability deployments – (ClusterA => ClusterB)
    • Active/Active HA Deployment – (ClusterA => ClusterB and ClusterB => ClusterA)
    • Aggregation (e.g., from many clusters to one): (ClusterA => ClusterK, ClusterB => ClusterK, ClusterC => ClusterK)
    • Fan-out (opposite of Aggregation): (ClusterK => ClusterA, ClusterK => ClusterB, ClusterK => ClusterC)
    • Forwarding: (ClusterA => ClusterB, ClusterB => ClusterC, ClusterC => ClusterD)

    Salient Features of MirrorMaker 2

    • Mirrors Topic and Topic Configuration – Detects and mirrors new topics and config changes automatically, including the number of partitions and replication factors.
    • Mirrors ACLs – Mirrors Topic ACLs as well, though we found issues in replicating WRITE permission. Also, replicated topics often contain source cluster names as a prefix, which means existing ACLs need to be tweaked, or ACL replication may need to be managed externally if the topologies are more complex.
    • Mirrors Consumer Groups and Offsets – Seamlessly translates and syncs Consumer Group Offsets to target clusters to make it easier to switch from one cluster to another in case of disaster.
    • Ability to Update MM2 Config Dynamically – MirrorMaker is backed by Kafka Connect Framework, which provides REST APIs through which MirrorMaker configurations like replicating new topics, stopping replicating certain topics, etc. can be updated without restarting the cluster.
    • Fault-Tolerant and Horizontally Scalable Operations – The number of processes can be scaled horizontally to increase performance.

    How Kafka MirrorMaker 2 Works

    MirrorMaker uses a set of standard Kafka connectors. Each connector has its own role. The listing of connectors and their functions is provided below.

    • MirrorSourceConnector: Replicates topics, topic ACLs, and configs from the source cluster to the target cluster.
    • MirrorCheckpointConnector: Syncs consumer offsets, emits checkpoints, and enables failover.
    • MirrorHeartBeatConnector: Checks connectivity between the source and target clusters.

    MirrorMaker Running Modes

    There are three ways to run MirrorMaker:

    • As a dedicated MirrorMaker cluster (can be distributed with multiple replicas having the same config): In this mode, MirrorMaker does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers.
    • As a standalone Connect worker: In this mode, a single Connect worker runs MirrorSourceConnector. This does not support multi-clusters, but it’s useful for small workloads or for testing.
    • In legacy mode, using existing MirrorMaker scripts: After legacy MirrorMaker is deprecated, the existing ./bin/kafka-mirror-maker.sh scripts will be updated to run MM2 in legacy mode:

    Setting up MirrorMaker 2

    We recommend running MirrorMaker as a dedicated MirrorMaker cluster since it does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers. The cluster can be easily converted to a distributed cluster just by adding multiple replicas of the same configuration. A distributed cluster is required to reduce the load on a single node cluster and also to increase MirrorMaker throughput.

    Prerequisites

    • Docker
    • Docker Compose

    Steps to Set Up MirrorMaker 2

    Set up a single node source, target Kafka cluster, and a MirrorMaker node to run MirrorMaker 2.

    1. Clone repository:

    https://gitlab.com/velotio/kafka-mirror-maker.git

    2. Run the below command to start the Kafka clusters and the MirrorMaker Docker container:

    docker-compose up -d

    3. Login to the mirror-maker docker container: 

    docker exec -it $(docker ps | grep u0022mirror-maker-node-1u0022 | awk '{print $1}') bash

    4. Start MirrorMaker:

    connect-mirror-maker.sh ./mirror-maker-config.properties

    5. Monitor the logs of the MirrorMaker container—it should be something like this: 

    • [2024-02-05 04:07:39,450] INFO [MirrorCheckpointConnector|task-0] sync idle consumer group offset from source to target took 0 ms (org.apache.kafka.connect.mirror.Scheduler:95)
    • [2024-02-05 04:07:49,246] INFO [MirrorCheckpointConnector|worker] refreshing consumer groups took 1 ms (org.apache.kafka.connect.mirror.Scheduler:95)
    • [2024-02-05 04:07:49,337] INFO [MirrorSourceConnector|worker] refreshing topics took 3 ms (org.apache.kafka.connect.mirror.Scheduler:95)
    • [2024-02-05 04:07:49,450] INFO [MirrorCheckpointConnector|task-0] refreshing idle consumers group offsets at target cluster took 2 ms (org.apache.kafka.connect.mirror.Scheduler:95)

    6. Create a topic at the source cluster: 

    kafka-topics.sh --create --bootstrap-server source-kafka:9092 --topic test-topic --partitions 1 --replication-factor 1

    7. List topics and validate the topic: 

    kafka-topics.sh u002du002dlist u002du002dbootstrap-server source-kafka:9092

    8. Produce 100 messages on the topic:

    for x in {1..100}; do echo u0022message $xu0022; done | kafka-console-producer.sh u002du002dbroker-list source-kafka:9092 u002du002dtopic test-topic

    9. Check whether the topic is mirrored in the target cluster.

    Note: The mirrored topic will have a source cluster name prefix to be able to identify which source cluster the topic is mirrored from.

    kafka-topics.sh u002du002dlist u002du002dbootstrap-server target-kafka:9092

    10. Consume 5 messages from the source kafka cluster:

    kafka-console-consumer.sh --bootstrap-server source-kafka:9092  --topic test-topic --max-messages 5 --consumer-property enable.auto.commit=true --consumer-property group.id=test-group —from-beginning

    11. Describe the consumer group at the source and destination to verify that consumer offsets are also mirrored:

    kafka-consumer-groups.sh --bootstrap-server source-kafka:9092 --group test-group --describe

    kafka-consumer-groups.sh --bootstrap-server target-kafka:9092 --group test-group --describe

    12. Consume five messages from the target Kafka cluster. The messages should start from the committed offset in the source cluster. In this case, the message offset will start at 6.

    kafka-console-consumer.sh --bootstrap-server target-kafka:9092  --topic source-kafka.test-topic --max-messages 5 --consumer-property enable.auto.commit=true --consumer-property group.id=test-group —from-beginning

    Conclusion

    We’ve seen how to set up MirrorMaker 2.0 in a dedicated instance. This running mode does not need a running Connect cluster as it leverages a high-level driver that creates a set of Connect workers based on the MirrorMaker properties configuration file.

  • Policy Insights: Chatbots and RAG in Health Insurance Navigation

    Introduction

    Understanding health insurance policies can often be complicated, leaving individuals to tackle lengthy and difficult documents. The complexity introduced by these policies’ language not only adds to the confusion but also leaves policyholders uncertain about the actual extent of their coverage, the best plan for their needs, and how to seek answers to their specific policy-related questions. In response to these ongoing challenges and to facilitate better access to information, a fresh perspective is being explored—an innovative approach to revolutionize how individuals engage with their health insurance policies.

    Challenges in Health Insurance Communication

    Health insurance queries are inherently complex, often involving nuanced details that require precision. Traditional chatbots, lacking the finesse of generative AI (GenAI), struggle to handle the intricacies of healthcare-related questions. The envisioned health insurance chatbot powered by GenAI overcomes these limitations, offering a sophisticated understanding of queries and delivering responses that align with the complexities of the healthcare sphere.

    Retrieval-Augmented Generation

    Retrieval-augmented generation, or RAG, is an architectural approach that can improve the efficacy of large language model (LLM) applications by leveraging custom data. This is done by retrieving relevant data/documents relevant to a question or task and providing them as context for the LLM. RAG has shown success in supporting chatbots and Q&A systems that need to maintain up-to-date information or access domain-specific knowledge.

    To know more about this topic, check here for technical insights and additional information.

    1. https://www.oracle.com/in/artificial-intelligence/generative-ai/retrieval-augmented-generation-rag/

    2. https://research.ibm.com/blog/retrieval-augmented-generation-RAG

    The Dual Phases of RAG: Retrieval and Content Generation

    Retrieval-augmented generation (RAG) smoothly combines two essential steps, carefully blending retrieval and content generation. Initially, algorithms diligently explore external knowledge bases to find relevant data that matches user queries. This gathered information then becomes the foundation for the next phase—content generation. In this step, the large language model uses both the enhanced prompt and its internal training data to create responses that are not only accurate but also contextually appropriate.

    Advantages of Deploying RAG in AI Chatbots

    Scalability is a key advantage of RAG over traditional models. Instead of relying on a monolithic model attempting to memorize vast amounts of information, RAG models can easily scale by updating or expanding the external database. This flexibility enables them to manage and incorporate a broader range of data efficiently.

    Memory efficiency is another strength of RAG in comparison to models like GPT. While traditional models have limitations on the volume of data they can store and recall, RAG efficiently utilizes external databases. This approach allows RAG to fetch fresh, updated, or detailed information as needed, surpassing the memory constraints of conventional models.

    Moreover, RAG offers flexibility in its knowledge sources. By modifying or enlarging the external knowledge base, a RAG model can be adapted to specific domains without the need for retraining the underlying generative model. This adaptability ensures that RAG remains a versatile and efficient solution for various applications.

    The displayed image outlines the application flow. In the development of our health insurance chatbot, we follow a comprehensive training process. Initially, essential PDF documents are loaded to familiarize our model with the intricacies of health insurance. These documents undergo tokenization, breaking them into smaller units for in-depth analysis. Each of these units, referred to as tokens, is then transformed into numerical vectors through a process known as vectorization. These numerical representations are efficiently stored in ChromaDB for quick retrieval.

    When a question is posed by a user, the numerical version of the query is retrieved from ChromaDB by the chatbot. Employing a language model (LLM), the chatbot crafts a nuanced response based on this numerical representation. This method ensures a smooth and efficient conversational experience. Armed with a wealth of health insurance information, the chatbot delivers precise and contextually relevant responses to user inquiries, establishing itself as a valuable resource for navigating the complexities of health insurance queries.

    Role of Vector Embedding

    Traditional search engines mainly focus on finding specific words in your search. For example, if you search “best smartphone,” it looks for pages with exactly those words. On the other hand, semantic search is like a more understanding search engine. It tries to figure out what you really mean by considering the context of your words.

    Imagine you are planning a vacation and want to find a suitable destination, and you input the query “warm places to visit in winter.” In a traditional search, the engine would look for exact matches of these words on web pages. Results might include pages with those specific terms, but the relevance might vary.

    Text, audio, and video can be embedded:

    An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness. Small distances suggest high relatedness, and large distances suggest low relatedness.

    For example:

    bat: [0.6, -0.3, 0.8, …]

    ball: [0.4, -0.2, 0.7, …]

    wicket: [-0.5, 0.6, -0.2, …]

    In this cricket-themed example, each word (bat, ball, wicket) is represented as a vector in a multi-dimensional space, capturing the semantic relationships between cricket-related terms.

    For a deeper understanding, you may explore additional insights in the following articles:

    1. https://www.datastax.com/guides/what-is-a-vector-embedding

    2. https://www.pinecone.io/learn/vector-embeddings/

    3. https://weaviate.io/blog/vector-embeddings-explained/

    A specialized type of database known as a vector database is essential for storing these numerical representations. In a vector database, data is stored as mathematical vectors, providing a unique way to store and retrieve information. This specialized database greatly facilitates machine learning models in retaining and recalling previous inputs, enabling powerful applications in search, recommendations, and text generation.

    Vector retrieval in a database involves finding the nearest neighbors or most similar vectors to a given query vector. These are metrics for finding similar vectors:

    1. The Euclidean distance metric considers both magnitudes and direction, providing a comprehensive measure for assessing the spatial separation between vectors.

    2. Cosine similarity focuses solely on the direction of vectors, offering insights into their alignment within the vector space.

    3. Dot product similarity metric takes into account both magnitudes and direction, offering a versatile approach for evaluating the relationships between vectors.

    ChromaDB, PineCone, and Milvus are a few examples of vector databases.

    For our application, we will be using LangChain, OpenAI embedding and LLM, and ChromaDB.

    1. We need to install Python packages required for this application.
    !pip install -U langchain openai chromadb langchainhub pypdf tiktoken

    A. LangChain is a tool that helps you build intelligent applications using language models. It allows you to develop chatbots, personal assistants, and applications that can summarize, analyze, or respond to questions about documents or data. It’s useful for tasks like coding assistance, working with APIs, and other activities that gain an advantage from AI technology.

    B. OpenAI is a renowned artificial intelligence research lab. Installing the OpenAI package provides access to OpenAI’s language models, including powerful ones like GPT-3. This library is crucial if you plan to integrate OpenAI’s language models into your applications.

    C. As mentioned earlier, ChromaDB is a vector database package designed to handle vector data efficiently, making it suitable for applications that involve similarity searches, clustering, or other operations on vectors.

    D. LangChainHub is a handy tool to make your language tasks easier. It begins with helpful prompts and will soon include even more features like chains and agents.

    E. PyPDF2 is a library for working with PDF files in Python. It allows reading and manipulating PDF documents, making it useful for tasks such as extracting text or merging PDF files.

    F. Tiktoken is a Python library designed for counting the number of tokens in a text string without making an API call. This can be particularly useful for managing token limits when working with language models or APIs that have usage constraints.

    1. Importing Libraries
    from langchain.chat_models import ChatOpenAI 
    from langchain.embeddings import OpenAIEmbeddings 
    from langchain.vectorstores import Chroma 
    from langchain.prompts import ChatPromptTemplate 
    from langchain.prompts import PromptTemplate
    from langchain.schema import StrOutputParser 
    from langchain.schema.runnable import RunnablePassthrough

    1. Initializing OpenAI LLM
    llm = ChatOpenAI(
    api_key=OPENAI_API_KEY”,
    model_name="gpt-4", 
    temperature=0.1
    )

    This line of code initializes a language model (LLM) using OpenAI’s GPT-4 model with 8192 tokens. Temperature parameter influences the randomness of text generated, and increased temperature results in more creative responses, while decreased temperature leads to more focused and deterministic answers.

    1. We will be loading a PDF consisting of material for training the model and also need to divide it into chunks of texts that can be fed to the model.
    from langchain.document_loaders import PyPDFLoader
    loader = PyPDFLoader(
    "HealthInsureBot_GenerativeAI_TrainingGuide.pdf") 
    docs = loader.load_and_split()

    1. We will be loading this chunk of text into the vector Database Chromadb, later used for retrieval and using OpenAI embeddings.
    vectorstore = Chroma.from_documents
    (
    documents=docs,
    embedding=OpenAIEmbeddings(api_key=OPENAI_API_KEY”)
    )

    1. Creating a retrieval object will return the top 3 similar vector matches for the query.
    retriever = vectorstore.as_retriever
    (
    search_type="similarity",
    search_kwargs={"k": 3}
    )

    7. Creating a prompt to pass to the LLM for obtaining specific information involves crafting a well-structured question or instruction that clearly outlines the desired details. RAG chain initiates with the retriever and formatted documents, progresses through the custom prompt template, involves the LLM , and concludes by utilizing a string output parser (StrOutputParser()) to handle the resulting response.

    def format_docs(docs): 
       return "nn".join(doc.page_content for doc in docs)
    
    template = """Use the following pieces of context as a virtual health insurance agent to answer the question and provide relevance score out of 10 for each response. If you don't know the answer, just say that you don't know, don't try to make up an answer. {context} Question: {question} Helpful Answer:""" 
    rag_prompt_custom = PromptTemplate.from_template(template) 
    rag_chain = ( 
    {"context": retriever | format_docs, "question": RunnablePassthrough()} | rag_prompt_custom 
    | llm 
    | StrOutputParser()
    )

    1. Create a function to get a response from the chatbot.
    def chatbot_response(user_query):
        return rag_chain.invoke(user_query)

    We can integrate the Streamlit tool for building a powerful generative app, using this function in Streamlit application to get the AI response.

    import openai 
    import streamlit as st
    from health_insurance_bot import chatbot_response 
    st.title("Health Insurance Chatbot")
    if "messages" not in st.session_state: 
       st.session_state["messages"] = 
       [{"role": "assistant", "content": "How can I help you?"}] 
    
    for msg in st.session_state.messages: 
       st.chat_message(msg["role"]).write(msg["content"]) 
    
    if prompt := st.chat_input(): 
      openai.api_key = st.secrets['openai_api_key']     
      st.session_state.messages.append({"role": "user", "content": prompt}) 
      st.chat_message(name="user").write(prompt) 
      response = chatbot_response(prompt)   
      st.session_state.messages.append({"role": "assistant", "content": response})  
      st.chat_message(name="assistant").write(response)

    Performance Insights

    Conclusion

    In our exploration of developing health insurance chatbots, we’ve dived into the innovative world of retrieval-augmented generation (RAG), where advanced technologies are seamlessly combined to reshape user interactions. The adoption of RAG has proven to be a game-changer, significantly enhancing the chatbot’s abilities to understand, retrieve, and generate contextually relevant responses. However, it’s worth mentioning a couple of limitations, including challenges in accurately calculating premium quotes and occasional inaccuracies in semantic searches.

  • Revolutionizing Android UI with MotionLayout: A Beginner’s Guide

    In the ever-evolving world of Android app development, seamless integration of compelling animations is key to a polished user experience. MotionLayout, a robust tool in the Android toolkit, has an effortless and elegant ability to embed animations directly into the UI. Join us as we navigate through its features and master the skill of effortlessly designing stunning visuals.

    1. Introduction to MotionLayout

    MotionLayout transcends conventional layouts, standing as a specialized tool to seamlessly synchronize a myriad of animations with screen updates in your Android application.

    1.1 Advantages of MotionLayout

    Animation Separation:

    MotionLayout distinguishes itself with the ability to compartmentalize animation logic into a separate XML file. This not only optimizes Java or Kotlin code but also enhances its overall manageability.

    No Dependence on Manager or Controller:

    An exceptional feature of MotionLayout is its user-friendly approach, enabling developers to attach intricate animations to screen changes without requiring a dedicated animation manager or controller.

    Backward Compatibility:

    Of paramount importance, MotionLayout maintains backward compatibility, ensuring its applicability across Android systems starting from API level 14.

    Android Studio Integration:

    Empowering developers further is the seamless integration with Android Studio. The graphical tooling provided by the visual editor facilitates the design and fine-tuning of MotionLayout animations, offering an intuitive workflow.

    Derivation from ConstraintLayout:

    MotionLayout, being a subclass of ConstraintLayout, serves as an extension specifically designed to facilitate the implementation of complex motion and animation design within a ConstraintLayout.

    1.2 Important Tags

    As elucidated earlier, Animation XML is separated into the following important tags and attributes:

    <MotionScene>: The topmost tag in XML, wrapping all subsequent tags.

    <ConstraintSet>: Describes one screen state, with two sets required for animations between states. For example, if we desire an animation where the screen transitions from state A to state B, we necessitate the definition of two ConstraintSets.

    <Transition>: Attaches to two ConstraintSets, triggering animation between them.

    <ViewTransition>: Utilized for changes within a single ConstraintSet.

    As explained before, animation XML is separate following are important tags and attributes that we should know

    1.3 Why It’s Better Than Its Alternatives

    It’s important to note that MotionLayout is not the sole solution for every animation scenario. Similar to the saying that a sword cannot replace a needle, MotionLayout can be a better solution when planning for complex animations. MotionLayout can replace animation created using threads and runnables. Apart from MotionLayout, several common alternatives for creating animations include:

    • Animated Vector Drawable
    • Property animation frameworks
    • LayoutTransition animation
    • Layout Transitions with TransitionManager
    • CoordinatorLayout

    Each alternative has unique advantages and disadvantages compared to MotionLayout. For smaller animations like icon changes, Animated Vector Drawable might be preferred. The choice between alternatives depends on the specific requirements of the animation task at hand.

    MotionLayout is a comprehensive solution, bridging the gap between layout transitions and complex motion handling. It seamlessly integrates features from the property animation framework, TransitionManager, and CoordinatorLayout. Developers can describe transitions between layouts, animate any property, handle touch interactions, and achieve a fully declarative implementation, all through the expressive power of XML.

    2. Configuration

    2.1 System setup

    For optimal development and utilization of the Motion Editor, Android Studio is a prerequisite. Kindly follow this link for the Android Studio installation guide.

    2.2 Project Implementation

    1. Initiate a new Android project and opt for the “Empty View Activity” template.

    1. Since MotionLayout is an extension of ConstraintLayout, it’s essential to include ConstraintLayout in the build.gradle file.

    implementation ‘androidx.constraintlayout:constraintlayout:x.x.x’

    Substitute “x.x.x” with the most recent version of ConstraintLayout.

    1. Replace “ConstraintLayout” with “MotionLayout.” Opting for the right-click method is recommended, as it facilitates automatically creating the necessary animation XML.
    Figure 1

    When converting our existing layout to MotionLayout by right-clicking, a new XML file named “activity_main_scene.xml” is generated in the XML directory. This file is dedicated to storing animation details for MotionLayout.

    1. Execute the following steps:
    1. Click on the “start” ConstraintSet.
    2. Move the Text View by dragging it to a desired position on your screen.
    3. Click on the “end” ConstraintSet.
    4. Move the Text View to another position on your screen.
    5. Click on the arrow above “start” and “end” ConstraintSet.
    6. Click on the “+” symbol in the “Attributes” tab.
    7. Add the attribute “autoTransition” with the value “jumpToEnd.”
    8. Click the play button on the “Transition” tab.

    Preview the animation in real time by running the application. The animation will initiate when called from the associated Java class.

    Note: You can also manually edit the activity_main_scene.xml file to make these changes.

    3. Sample Project and Result

    Until now, we’ve navigated through the complexities of MotionLayout and laid the groundwork for an Android project. Now, let’s transition from theory to practical application by crafting a sample project. In this endeavor, we’ll keep the animation simple and accessible for a clearer understanding.

    3.1 Adding Dependencies

    Include the following lines of code in your gradle.build file (Module: app), and then click on “Sync Now” to ensure synchronization with the project:

    android {
       buildFeatures {
           viewBinding true
       }
    }
    
    dependencies {
       implementation 'androidx.lifecycle:lifecycle-viewmodel-ktx:2.5.1'
       implementation 'androidx.lifecycle:lifecycle-runtime-ktx:2.5.1'
       implementation 'androidx.annotation:annotation:1.5.0'
    }

    3.2 Adding code

    Include the following code snippets in their corresponding classes:

    <?xml version="1.0" encoding="utf-8"?>
    <androidx.constraintlayout.motion.widget.MotionLayout xmlns:android="http://schemas.android.com/apk/res/android"
       xmlns:app="http://schemas.android.com/apk/res-auto"
       xmlns:tools="http://schemas.android.com/tools"
       android:layout_width="match_parent"
       android:layout_height="match_parent"
       android:background="@android:color/darker_gray"
       app:layoutDescription="@xml/activity_main_scene"
       android:id="@+id/layoutMain"
       tools:context=".MainActivity">
    
       <ImageView
           android:id="@+id/image_view_00"
           android:layout_width="80dp"
           android:layout_height="100dp"
           android:layout_margin="32dp"
           android:src="@drawable/card_back"
           app:layout_constraintEnd_toEndOf="parent"
           app:layout_constraintStart_toStartOf="parent"
           app:layout_constraintTop_toTopOf="parent" />
    
       <ImageView
           android:id="@+id/image_view_01"
           android:layout_width="80dp"
           android:layout_height="100dp"
           android:layout_margin="32dp"
           android:src="@drawable/card_back"
           app:layout_constraintEnd_toEndOf="parent"
           app:layout_constraintStart_toStartOf="parent"
           app:layout_constraintTop_toTopOf="parent" />
    
       <ImageView
           android:id="@+id/image_view_02"
           android:layout_width="80dp"
           android:layout_height="100dp"
           android:layout_margin="32dp"
           android:src="@drawable/card_back"
           app:layout_constraintEnd_toEndOf="parent"
           app:layout_constraintStart_toStartOf="parent"
           app:layout_constraintTop_toTopOf="parent" />
    
    </androidx.constraintlayout.motion.widget.MotionLayout>

    <?xml version="1.0" encoding="utf-8"?>
    <MotionScene xmlns:android="http://schemas.android.com/apk/res/android"
       xmlns:motion="http://schemas.android.com/apk/res-auto">
    
       <ConstraintSet android:id="@+id/image_00">
           <Constraint
               android:id="@+id/image_view_00"
               android:layout_width="80dp"
               android:layout_height="100dp"
               android:layout_marginTop="32dp"
               motion:layout_constraintEnd_toEndOf="parent"
               motion:layout_constraintStart_toStartOf="parent"
               motion:layout_constraintTop_toTopOf="parent" />
           <Constraint
               android:id="@+id/image_view_01"
               android:layout_width="80dp"
               android:layout_height="100dp"
               android:layout_marginTop="32dp"
               motion:layout_constraintEnd_toEndOf="parent"
               motion:layout_constraintStart_toStartOf="parent"
               motion:layout_constraintTop_toTopOf="parent" />
           <Constraint
               android:id="@+id/image_view_02"
               android:layout_width="80dp"
               android:layout_height="100dp"
               android:layout_marginTop="32dp"
               motion:layout_constraintEnd_toEndOf="parent"
               motion:layout_constraintStart_toStartOf="parent"
               motion:layout_constraintTop_toTopOf="parent" />
       </ConstraintSet>
       <ConstraintSet android:id="@+id/image_01">
           <Constraint
               android:id="@+id/image_view_00"
               android:layout_width="80dp"
               android:layout_height="100dp"
               android:layout_marginBottom="32dp"
               android:layout_marginEnd="44dp"
               android:src="@drawable/card_back"
               motion:layout_constraintBottom_toBottomOf="parent"
               motion:layout_constraintEnd_toEndOf="@id/image_view_01" />
           <Constraint
               android:id="@+id/image_view_01"
               android:layout_width="80dp"
               android:layout_height="100dp"
               android:layout_marginBottom="32dp"
               android:src="@drawable/card_back"
               motion:layout_constraintBottom_toBottomOf="parent"
               motion:layout_constraintEnd_toEndOf="parent"
               motion:layout_constraintStart_toStartOf="parent" />
           <Constraint
               android:id="@+id/image_view_02"
               android:layout_width="80dp"
               android:layout_height="100dp"
               android:layout_marginBottom="32dp"
               android:layout_marginStart="44dp"
               android:src="@drawable/card_back"
               motion:layout_constraintBottom_toBottomOf="parent"
               motion:layout_constraintStart_toStartOf="@id/image_view_01" />
       </ConstraintSet>
    
       <Transition
           motion:autoTransition="animateToEnd"
           motion:constraintSetEnd="@+id/image_01"
           motion:constraintSetStart="@id/image_00"
           motion:duration="5000" />
    </MotionScene>

    class MainActivity : AppCompatActivity() {
       private lateinit var binding: ActivityMainBinding
       override fun onCreate(savedInstanceState: Bundle?) {
           super.onCreate(savedInstanceState)
           setContentView(R.layout.activity_main)
           binding = ActivityMainBinding.inflate(layoutInflater)
           setContentView(binding.root)
           binding.apply {
               lifecycleScope.launch {
                   delay(1000)
                   layoutMain.transitionToEnd()
               }
           }
       }
    }

    3.3 Result

    For a thorough comprehension of the implementation specifics and complete access to the source code, allowing you to delve into the intricacies of the project and harness its functionalities adeptly, please refer to this repository.

    4. Assignment

    Expanding the animation’s complexity becomes seamless by incorporating additional elements with meticulous handling. Here’s an assignment for you: endeavor to create the specified output below.

    4.1 Assignment 1

    4.2 Assignment 2

    5. Conclusion

    In conclusion, this guide has explored the essentials of using MotionLayout in Android development, highlighting its superiority over other animation methods. While we’ve touched on its basic capabilities here, future installments will explore more advanced features and uses. We hope this piece has ignited your interest in MotionLayout’s potential to enhance your Android apps.

    Thank you for dedicating your time to this informative read!

    6. References

    1. https://developer.android.com/reference/androidx/constraintlayout/motion/widget/MotionLayout
    2. https://developer.android.com/reference/androidx/constraintlayout/widget/ConstraintLayout
    3. https://android-developers.googleblog.com/2021/02/mad-skills-motion-layout-wrap-up.html
    4. https://www.bacancytechnology.com/blog/motionlayout-in-android
    5. https://medium.com/mindful-engineering/getting-started-with-motion-layout-in-android-c52af5d5076c
    6. https://blog.mindorks.com/getting-started-with-motion-layout-android-tutorials/
    7. https://gorillalogic.com/blog/a-motionlayout-tutorial-create-motions-and-animations-for-android
    8. https://taglineinfotech.com/motion-layout-in-android/
    9. https://applover.com/blog/how-to-create-motionlayout-part1/
    10. https://medium.com/simform-engineering/animations-in-android-featuring-motionlayout-from-scratch-3ec5cbd6b616
    11. https://www.nomtek.com/blog/motionlayout
  • The Responsible Use of Artificial Intelligence – Shaping a Safer Tomorrow

    Introduction:

    Artificial intelligence (AI) stands at the forefront of technological innovation, promising transformative changes in our lives. With its continuous advancements, AI has become an integral part of our daily routines, from virtual assistants and personalized recommendations to healthcare diagnostics and autonomous vehicles. However, the rapid integration of AI into our society raises pertinent ethical questions, necessitating a closer examination of its responsible use. In this blog post, we delve into the responsible use of artificial intelligence, exploring the principles that guide its ethical deployment and emphasizing the collaborative efforts required to shape a safer future.

    Understanding Responsible AI:

    Responsible AI signifies more than just technological progress; it embodies the ethical development and deployment of AI systems, emphasizing principles such as fairness, transparency, accountability, and privacy. To ensure that AI benefits society as a whole, it is crucial to address the following key aspects:

    1. Ethical Considerations:
      Ethics serve as the cornerstone of AI development. Collaboration among engineers, data scientists, and policymakers is paramount in establishing ethical guidelines that prevent AI from being used for harmful purposes. It is imperative to avoid deploying AI in situations that could lead to discrimination, manipulation, or privacy erosion. Ethical considerations must permeate every stage of AI development, guiding decisions and actions.

    2. Transparency and Accountability:
      Understanding the functioning of AI algorithms is pivotal for their ethical deployment. Striving for transparency, developers should elucidate, in plain language, how AI-driven decisions are made. Accountability mechanisms must be in place to address errors, biases, and unintended consequences. Regular audits and assessments ensure that AI systems remain ethical and accountable, promoting trust among users.

    3. Bias Mitigation:
      The quality of AI algorithms hinges on the data they are trained on. Identifying and mitigating biases in datasets is imperative to create fair and equitable AI applications. Diverse and representative datasets are essential to reducing biases, ensuring that AI systems work impartially for everyone, irrespective of their background. Bias mitigation is an ongoing process, demanding continuous vigilance throughout AI development.

    4. Privacy Protection:
      Responsible AI use involves safeguarding user privacy. As AI applications require extensive data, concerns arise about how this data is collected, stored, and utilized. Regulations and standards, such as GDPR in Europe, play a pivotal role in protecting user privacy rights and ensuring responsible handling of personal data. Developers and companies must prioritize user privacy in all facets of AI development to foster user trust and confidence.

    5. Continuous Monitoring and Adaptation:
      AI’s landscape is in constant flux, necessitating continuous monitoring and adaptation to evolving ethical standards. Regular updates, feedback loops, and adaptive learning enable AI technologies to remain responsive to societal needs and concerns. Developers and companies must vigilantly monitor AI system performance, ready to make necessary changes to align outcomes with ethical standards.

    6. Public Awareness and Education:
      Raising public awareness about AI and its implications is crucial. Educating the public about the ethical use of AI, potential biases, and privacy concerns empowers individuals to make informed decisions. Workshops, seminars, and accessible online resources can bridge the knowledge gap, ensuring that society comprehends the responsible use of AI and actively participates in shaping its trajectory.

    7. Collaboration Across Sectors:
      Collaboration between governments, private sectors, and non-profit organizations is vital. By working together, diverse perspectives can be integrated, leading to comprehensive policies and guidelines. Initiatives like joint research projects, cross-industry collaborations, and international summits facilitate the exchange of ideas and foster a unified approach to responsible AI deployment.

    Potential Risks of Irresponsible AI Use:

    Irresponsible AI use can have detrimental consequences, impacting individuals and society profoundly. Here are the key risks associated with irresponsible AI deployment:

    1. Bias and Discrimination:
      AI systems can perpetuate existing biases, leading to discriminatory outcomes, particularly in areas such as hiring, lending, and law enforcement.

      – For example, a study revealed that AI algorithms used in criminal justice systems exhibited racial bias, leading to disproportionately harsher sentences for people of color. This instance underscores the critical need for rigorous bias detection and mitigation strategies in AI development.

    2. Privacy Violations:
      Improper AI use can result in unauthorized access to personal data, compromising individuals’ privacy and security. This breach can lead to identity theft, financial fraud, and other cybercrimes, highlighting the urgency of robust data protection measures.

      – For instance, the Cambridge Analytica scandal demonstrated how AI-driven data analysis could lead to unauthorized access to millions of users’ personal information on social media platforms, emphasizing the need for stringent data privacy regulations and ethical data management practices.

    3. Job Displacement:
      AI-driven automation could lead to job displacement, posing economic challenges and social unrest. Industries reliant on routine tasks susceptible to automation are particularly vulnerable, necessitating proactive measures to address potential workforce transitions.

      – An example can be found in the manufacturing sector, where AI-driven robotics have significantly reduced the need for human workers in certain tasks, leading to workforce challenges and economic disparities. Initiatives focusing on retraining and upskilling programs can help mitigate these challenges.

    4. Security Threats:
      AI systems are vulnerable to attacks and manipulation. Malicious actors could exploit these vulnerabilities, causing widespread disruption, manipulating financial markets, or spreading misinformation. Vigilance and robust security measures are paramount to mitigate these threats effectively.

      – For instance, the rise of deepfake technology, enabled by AI, poses significant threats to political, social, and economic stability. Misinformation and manipulated media can influence public opinion and decision-making processes, emphasizing the need for advanced AI-driven detection tools and awareness campaigns to combat this issue.

    5. Loss of Human Control:
      Inadequate regulation could lead to AI systems, especially in military applications and autonomous vehicles, operating beyond human control. This lack of oversight might result in unintended consequences and ethical dilemmas, underscoring the need for stringent regulation and ethical guidelines.

      – A notable example is the debate surrounding autonomous vehicles, where AI-driven decision-making processes raise ethical questions, such as how these vehicles prioritize different lives in emergency situations. Robust ethical frameworks and regulatory guidelines are essential to navigate these complex scenarios responsibly.

    Best AI Practices:

    To ensure responsible AI development and usage, adhering to best practices is imperative:

    1. Research and Development Ethics:
      Organizations should establish research ethics boards to oversee AI projects, ensuring strict adherence to ethical guidelines during the development phase.
    2. Collaboration and Knowledge Sharing:
      Encourage collaboration between industry, academia, and policymakers to facilitate knowledge sharing and establish common ethical standards for AI development and deployment. Collaboration fosters a holistic approach, incorporating diverse perspectives and expertise.
    3. User Education:
      Educating users about AI capabilities and limitations is essential. Raising awareness about the collected data and how it will be used promotes informed decision-making and user empowerment.

      – For instance, companies can provide interactive online resources and workshops explaining how AI algorithms work, demystifying complex concepts for the general public. Transparent communication about data collection and usage practices can enhance user trust and confidence.
    4. Responsible Data Management:
      Implement robust data management practices to ensure data privacy, security, and compliance with regulations. Regularly update privacy policies to reflect the evolving nature of AI technology, demonstrating a commitment to user privacy and data protection.

      – Companies can employ advanced encryption techniques to protect user data and regularly undergo independent audits to assess their data security measures. Transparent reporting of these security practices can build trust with users and regulatory bodies.
    5. Ethical AI Training:
      AI developers and engineers should receive training in ethics, emphasizing the significance of fairness, accountability, and transparency in AI algorithms. Ethical AI training fosters a culture of responsibility, guiding developers to create systems that benefit society.

      Educational institutions and industry organizations can collaborate to offer specialized courses on AI ethics, ensuring that future developers are well-equipped to navigate the ethical challenges of AI technology. Industry-wide certification programs can validate developers’ expertise in ethical AI practices, setting industry standards.

    Examples of Responsible AI Implementation:

    Google’s Ethical Guidelines:

    Google has been a trailblazer in implementing responsible AI practices, demonstrating a commitment to ethical use. The establishment of the Advanced Technology External Advisory Council (ATEAC) underscores Google’s dedication to addressing ethical challenges related to AI. Additionally, their release of guidelines for the ethical use of AI emphasizes fairness, accountability, and transparency. Google’s AI Principles outline their pledge to avoid bias, ensure safety, and provide broad benefits to humanity, setting a commendable precedent for the industry.

    An illustrative example of Google’s commitment to transparency can be seen in their Explainable AI (XAI) research, where efforts are made to make AI algorithms interpretable for users. By providing users with insights into how AI systems make decisions, Google enhances transparency and user understanding, contributing to responsible AI usage.

    Microsoft’s AETHER Committee:

    Microsoft has taken significant strides to ensure the responsible use of AI. The creation of the AI and Ethics in Engineering and Research (AETHER) Committee exemplifies Microsoft’s proactive approach to addressing ethical considerations. Furthermore, their active involvement in AI policy advocacy emphasizes the need for regulation to prevent the misuse of facial recognition technology. Microsoft’s initiatives promote transparency, accountability, and privacy protection, exemplifying a commitment to responsible AI implementation.

    A noteworthy initiative by Microsoft is its collaboration with academic institutions to research bias detection techniques in AI algorithms. By actively engaging in research, Microsoft contributes valuable knowledge to the industry, addressing the challenge of bias mitigation in AI systems and promoting responsible AI development.

    Conclusion:

    Artificial Intelligence holds immense potential to revolutionize our world, but this power must be wielded responsibly. By adhering to ethical guidelines, promoting transparency, and prioritizing privacy and fairness, we can harness the benefits of AI while mitigating its risks. Understanding the responsible use of AI empowers us as users and consumers to demand ethical practices from the companies and developers creating these technologies. Let us collectively work towards ensuring that AI becomes a force for good, shaping a safer and more equitable future for all.