My blog

Category: Industry

  • Building Scalable and Efficient React Applications Using GraphQL and Relay

    Building a React application is not only about creating a user interface. It also has tricky parts like data fetching, re-render performance, and scalability. Many libraries and frameworks try to solve these problems, like Redux, Sagas, etc. But these tools come with their own set of difficulties.

    Redux gives you a single data source, but all the data fetching and rendering logic is handled by developers. Immer gives you immutable data structures, but one needs to handle the re-render performance of applications.

    GraphQL helps developers design and expose APIs on the backend, but no tool on the client side could utilize the full advantage of the single endpoint and data schema provided by GraphQL.

    In this article, we will learn about Relay as a GraphQL client. What are the advantages of using Relay in your application, and what conventions are required to integrate it?  We’ll also cover how following those conventions will give you a better developer experience and a performant app. We will also see how applications built with Relay are modular, scalable, efficient, and, by default, resilient to change.

    About Relay

    Relay is a JavaScript framework to declaratively fetch and manage your GraphQL data inside a React application. Relay uses static queries and ahead-of-time compilation to help you build a high-performance app. 

    But as the great saying goes, “With great power comes great responsibilities.” Relay comes with a set of costs (conventions), which—when compared with the benefits you get—is well worth it. We will explore the trade-offs in this article.

    The Relay framework is built of multiple modules:

    1. The compiler: This is a set of modules designed to extract GraphQL code from across the codebase and do validations and optimizations during build time.

    2. Relay runtime: A high-performance GraphQL runtime that features a normalized cache for objects and highly optimized read/write operations, simplified abstractions over fetching data fields, garbage collection, subscriptions, and more.

    3. React-relay: This provides the high-level APIs to integrate React with the Relay runtime.

    The Relay compiler runs as a separate process, like how webpack works for React. It keeps watching and compiling the GraphQL code, and in case of errors, it simply does not build your code, which prevents bugs from going into higher environments.

    Fragments

    Fragments are at the heart of how Relay blends with GraphQL. A fragment is a selection of fields on a GraphQL type. 

    fragment Avatar_user on User {
  avatarImgUrl
  firstName
  lastName
  userName
}

    If we look at the sample fragment definition above, the fragment name, Avatar_user, is not just a random name. One of the Relay framework’s important conventions is that fragments have globally unique fragment names and follow a structure of <modulename>_<propertyname>. The example above is a fragment definition for Avatar_user.</propertyname></modulename>

    This fragment can then be reused throughout the queries instead of selecting the fields manually to render the avatar in each view.

    In the below query, we see the author type, and the first two who liked the blog post can use the fragment definition of Avatar_user

    query GetBlogPost($postId: ID!) {
      blogPostById(id: $postId) {
        author {
          firstName
          lastName
          avatarImgUrl
          userName
        }
        likedBy(first: 2) {
          edges {
            node {
              firstName
              lastName
              avatarImgUrl
              userName
            }
          }
        }
      }
    }

    Now, our new query with fragments looks like this:

    query GetBlogPost($postId: ID!) {
      blogPostById(id: $postId) {
        author {
          ...Avatar_user
        }
        likedBy(first: 2) {
          edges {
            node {
              ...Avatar_user
            }
          }
        }
      }
    }

    Fragments not only allow us to reuse the definitions but more essentially, they let us add or remove fields needed to render our avatar as we evolve our application.

    Another highly important client-side convention is colocation. This means the data required for a component lives inside the component. This makes maintenance and extending much easier. Just like how React allows us to break our UI elements into components and group/compose different views, fragments in Relay allow us to split the data definitions and colocate the data and the view definitions.

    So, a good practice is to define single or multiple fragments that contain the data component to be rendered. This means that a component depends on some fields from the user type, irrespective of the parent component. In the example above, the <avatar> component will render an avatar using the fields specified in the Avatar_user fragment named.</avatar>

    How Relay leverages the GraphQL Fragment

    Relay wants all components to enlist all the data it needs to render, along with the component itself. Relay uses data and fragments to integrate the component and its data requirement. This convention mandates that every component lists the fields it needs access to. 

    Other advantages of the above are:

    1. Components are not dependent on data they don’t explicitly request.
    2. Components are modular and self-contained.
    3. Reusing and refactoring the components becomes easier.

    Performance

    In Relay, the component re-renders only when its exact fields change, and this feature available is out of the box. The fragment subscribes to updates specifically for data the component selects. This lets Relay enhance how the view is updated, and performance is not affected as codebase scales.

    Now, let’s look at an example of components in a single post of a blog application. Here is a wireframe of a sample post to give an idea of the data and view required.

    Now, let’s write a plain query without Relay, which will fetch all the data in a single query. It will look like this for the above wireframe:

    query GetBlogPost($postId: ID!) {
      blogPostById(id: $postId) {
        author {
          firstName
          lastName
          avatarUrl
          shortBio
        }
        title
        coverImgUrl
        createdAt
        tags {
          slug
          shortName
        }
        body
        likedByMe
        likedBy(first: 2) {
          totalCount
          edges {
            node {
              firstName
              lastName
              avatarUrl
            }
          }
        }
      }
    }

    This one query has all the necessary data. Let’s also write down a sample structure of UI components for the query above:

    <BlogPostContainer>
    <BlogPostHead>
      <BlogPostAuthor>
        <Avatar />
      </BlogPostAuthor>
    </BlogPostHead>
    <BlogPostBody>
      <BlogPostTitle />
      <BlogPostMeta>
        <CreatedAtDisplayer />
        <TagsDisplayer />
      </BlogPostMeta>
      <BlogPostContent />
      <LikeButton>
        <LikedByDisplayer />
      </LikeButton>
    </BlogPostBody>
 </BlogPostContainer>

    In the implementation above, we have a single query that will be managed by the top-level component. It will be the top-level component’s responsibility to fetch the data and pass it down as props. Now, we will look at how we would build this in Relay:

    import * as React from "react";
    import { GetBlogPost } from "./__generated__/GetBlogPost.graphql";
    import { useLazyLoadQuery } from "react-relay/hooks";
    import { BlogPostHead } from "./BlogPostHead";
    import { BlogPostBody } from "./BlogPostBody";
    import { graphql } from "react-relay";


    interface BlogPostProps {
      postId: string;
    }

    export const BlogPost = ({ postId }: BlogPostProps) => {
      const { blogPostById } = useLazyLoadQuery<GetBlogPost>(
        graphql`
          query GetBlogPost($postId: ID!) {
            blogPostById(id: $postId) {
              ...BlogPostHead_blogPost
              ...BlogPostBody_blogPost
            }
          }
        `,
        {
          variables: { postId }
        }
      );

      if (!blogPostById) {
        return null;
      }

      return (
        <div>
          <BlogPostHead blogPost={blogPostById} />
          <BlogPostBody blogPost={blogPostById} />
        </div>
      );
    };

    First, let’s look at the query used inside the component:

    const { blogPostById } = useLazyLoadQuery<GetBlogPost>(
graphql`
  query GetBlogPost($postId: ID!) {
    blogPostById(id: $postId) {
      ...BlogPostHead_blogPost
      ...BlogPostBody_blogPost
    }
  }
`,
{
  variables: { postId }
}
);

    The useLazyLoadQuery React hook from Relay will start fetching the GetBlogPost query just as the component renders. 

    NOTE: The useLazyLoadQuery is used here as it follows a common mental model of fetching data after the page is loaded. However, Relay encourages data to be fetched as early as possible using the usePreladedQuery hook. 

    For type safety, we are annotating the useLazyLoadQuery with the type GetBlogPost, which is imported from ./__generated__/GetBlogPost.graphql. This file is auto-generated and synced by the Relay compiler. It contains all the information about the types needed to be queried, along with the return type of data and the input variables for the query.

    The Relay compiler takes all the declared fragments in the codebase and generates the type files, which can then be used to annotate a particular component.

    The GetBlogPost query is defined by composing multiple fragments. Another great aspect of Relay is that there is no need to import the fragments manually. They are automatically included by the Relay compiler. Building the query by composing fragments, just like how we compose our component, is the key here. 

    Another approach can be to define queries per component, which takes full responsibility for its data requirements. But this approach has two problems: 

    1. Multiple queries are sent to the server instead of one.

    2. The loading will be slower as components would have to wait till they render to start fetching the data.

    In the above example, the GetBlogPost only deals with including the fragments for its child components, BlogPostHead and BlogPostBody. It is kept hidden from the actual data fields of the children component.

    When using Relay, components define their data requirement by themselves. These components can then be composed along with other components that have their own separate data. 

    At the same time, no component knows what data the other component needs except from the GraphQL type that has the required component data. Relay makes sure the right data is passed to the respective component, and all input for a query is sent to the server.

    This allows developers to think only about the component and fragments as one while Relay does all the heavy lifting in the background. Relay minimizes the round-trips to the server by placing the fragments from multiple components into optimized and efficient batches. 

    As we said earlier, the two fragments, BlogPostHead_blogPost and BlogPostBody_blogPost, which we referenced in the query, are not imported manually. This is because Relay imposes unique fragment names globally so that the compiler can include the definitions in queries sent to the server. This eliminates the chances of errors and takes away the laborious task of referencing the fragments by hand. 

     if (!blogPostById) {
      return null;
  }

  return (
    <div>
      <BlogPostHead blogPost={blogPostById} />
      <BlogPostBody blogPost={blogPostById} />
    </div>
  );

    Now, in the rendering logic above, we render the <BlogPostHead/> and <BlogPostBody/> and pass the blogPostById object as prop. It’s passed because it is the object inside the query that spreads the fragment needed by the two components. This is how Relay transfers fragment data. Because we spread both fragments on this object, it is guaranteed to satisfy both components.

    To put it into simpler terms, we say that to pass the fragment data, we pass the object where the fragment is spread, and the component then uses this object to get the real fragment data. Relay, through its robust type systems, makes sure that the right object is passed with required fragment spread on it.

    The previous component, the BlogPost, was the Parent component, i.e., the component with the root query object. The root query is necessary because it cannot fetch a fragment in isolation. Fragments must be included in the root query in a parent component. The parent can, in turn, be a fragment as long the root query exists in the hierarchy. Now, we will build the BlogPostHead component using fragments:

     import * as React from "react";
    import { useFragment } from "react-relay/hooks";
    import { graphql } from "react-relay";
    import {
      BlogPostHead_blogPost$key, BlogPostHead_blogPost
    } from "./__generated__/BlogPostHead_blogPost.graphql";
    import { BlogPostAuthor } from "./BlogPostAuthor";
    import { BlogPostLikeControls } from "./BlogPostLikeControls";

    interface BlogPostHeadProps {
      blogPost: BlogPostHead_blogPost$key;
    }

    export const BlogPostHead = ({ blogPost }: BlogPostHeadProps) => {
      const blogPostData = useFragment<BlogPostHead_blogPost>(
        graphql`
          fragment BlogPostHead_blogPost on BlogPost {
            title
            coverImgUrl
            ...BlogPostAuthor_blogPost
            ...BlogPostLikeControls_blogPost
          }
        `,
        blogPost
      );

      return (
        <div>
          <img src={blogPostData.coverImgUrl} />
          <h1>{blogPostData.title}</h1>
          <BlogPostAuthor blogPost={blogPostData} />
          <BlogPostLikeControls blogPost={blogPostData} />
        </div>
      );
    };

    NOTE: In our example, the BlogPostHead and BlogPostBody define only one fragment, but in general, a component can have any number of fragments or GraphQL types and even more than one fragments on the same type.

    In the component above, two type definitions, namely BlogPostHead_blogPost$key and BlogPostHead_blogPost, are imported from the file BlogPostHead_blogPost.graphql, generated by the Relay compiler. The compiler extracts the fragment code from this file and generates the types. This process is followed for all the GraphQL code—queries, mutations, fragments, and subscriptions.

    The blogPostHead_blogPost has the fragment type definitions, which is then passed to the useFragment hook to ensure type safety when using the data from the fragment. The other import, blogPostHead_blogPost$key, is used in the interface Props { … }, and this type definition makes sure that we pass the right object to useFragment. Otherwise,  the type system will throw errors during build time. In the above child component, the blogPost object is received as a prop and is passed to useFragment as a second parameter. If the blogPost object did not have the correct fragment, i.e., BlogPostHead_blogPost, spread on it, we would have received a type error. Even if there were another fragment with exact same data selection spread on it, Relay makes sure it’s the right fragment that we use with the useFragement. This allows you to change the update fragment definitions without affecting other components.

    Data masking

    In our example, the fragment BlogPostHead_blogPost explicitly selects two fields for the component:

    1. title
    2. coverImgUrl

    This is because we use/access only these two fields in the view for the <blogposthead></blogposthead> component. So, even if we define another fragment, BlogPostAuthor_blogPost, which selects the title and coverImgUrl, we don’t receive access to them unless we ask for them in the same fragment. This is enforced by Relay’s type system both at compile time and at runtime. This safety feature of Relay makes it impossible for components to depend on data they do not explicitly select. So, developers can refactor the components without risking other components. To reiterate, all components and their data dependencies are self-contained.

    The data for this component, i.e., title and coverImgUrl, will not be accessible on the parent component, BlogPost, even though the props object is sent by the parent. The data becomes available only through the useFragment React hook. This hook can consume the fragment definition. The useFragment takes in the fragment definition and the object where the fragment is spread to get the data listed for the particular fragment.  

    Just like how we spread the fragment for the BlogPostHead component in the BlogPost root query, we an also extend this to the child components of BlogPostHead. We spread the fragments, i.e., BlogPostAuthor_blogPost, BlogPostLikeControls_blogPost, since we are rendering <BlogPostAuthor /> and <BlogPostLikeControls />.

    NOTE: The useFragment hook does not fetch the data. It can be thought of as a selector that grabs only what is needed from the data definitions.

    Performance

    When using a fragment for a component, the component subscribes only to the data it depends on. In our example, the component BlogPostHead will only automatically re-render when the fields “coverImgUrl” or “title” change for a specific blog post the component renders. Since the BlogPostAuthor_blogPost fragment does not select those fields, it will not re-render. Subscription to any updates is made on fragment level. This is an essential feature that works out of the box with Relay for performance.

    Let us now see how general data and components are updated in a different GraphQL framework than Relay. The data that gets rendered on view actually comes from an operation that requests data from the server, i.e., a query or mutation. We write the query that fetches data from the server, and that data is passed down to different components as per their needs as props. The data flows from the root component, i.e., the component with the query, down to the components. 

    Let’s look at a graphical representation of the data flow in other GraphQL frameworks:

    Image source: Dev.to

    NOTE: Here, the framework data store is usually referred to as cache in most frameworks:

    1. The Profile component executes the operation ProfileQuery to a GraphQL server.

    2. The data return is kept in some framework-specific representation of the data store.

    3. The data is passed to the view rendering it.

    4. The view then passes on the data to all the child components who need it. Example: Name, Avatar, and Bio. And finally React renders the view.

    In contrast, the Relay framework takes a different approach:

    Image source: Dev.to

    Let’s breakdown the approach taken by Relay: 

    • For the initial part, we see nothing changes. We still have a query that is sent to the GraphQL server and the data is fetched and stored in the Relay data store.
    • What Relay does after this is different. The components get the data directly from the cache-store(data store). This is because the fragments help Relay integrate deeply with the component data requirements.The component fragments get the data straight from the framework data store and do not rely on data to be passed down as props. Although some information is passed from the query to the fragments used to look up the particular data needed from the data store, the data is fetched by the fragment itself.

    To conclude the above comparison, in other frameworks (like Apollo), the component uses the query as the data source. The implementation details of how the root component executing the query sends data to its descendants is left to us. But Relay takes a different approach of letting the component take care of the data in needs from the data store.

    In an approach used by other GraphQL frameworks, the query is the data source, and updates in the data store forces the component holding the query to re-render. This re-render cascades down to any number of components even if those components do not have to do anything with the updated data other than acting as a layer to pass data from parent to child. In the Relay approach, the components directly subscribe to the updates for the data used. This ensures the best performance as our app scales in size and complexity.

    Developer Experience

    Relay removes the responsibility of developers to route the data down from query to the components that need it. This eliminates the changes of developer error. There is no way for a component to accidentally or deliberately depend on data that it should be just passing down in the component tree if it cannot access it. All the hard work is taken care of by the Relay framework if we follow the conventions discussed.

    Conclusion

    To summarize, we detailed all the work Relay does for us and the effects:

    • The type system of the Relay framework makes sure the right components get the right data they need. Everything in Relay revolves around fragments.
    • In Relay, fragments are coupled and colocated with components, which allows it to mask the data requirements from the outside world. This increases the readability and modularity.
    • By default, Relay takes care of performance as components only re-render when the exact data they use change in the data store.
    • Type generation is a main feature of Relay compiler. Through type generation, interactions with the fragment’s data is typesafe.

    Conventions enforced by Relay’s philosophy and architecture allows it to take advantage of the information available about your component. It knows the exact data dependencies and types. It uses all this information to do a lot of work that developers are required to deal with.

    Related Articles

    1. Enable Real-time Functionality in Your App with GraphQL and Pusher

    2. Build and Deploy a Real-Time React App Using AWS Amplify and GraphQL

  • Node.js – Async Your Way out of Callback Hell with Promises, Async & Async/Await

    In this blog, I will compare various methods to avoid the dreaded callback hells that are common in Node.js. What exactly am I talking about? Have a look at this piece of code below. Every child function executes only when the result of its parent function is available. Callbacks are the very essence of the unblocking (and hence performant) nature of Node.js.

    foo(arg, (err, val) => {
     if (err) {
          console.log(err);
     } else {
          val += 1;
          bar(val, (err1, val1) => {
               if (err) {
                    console.log(err1);
               } else {
                    val1 += 2;
                    baz(val1, (err2, result) => {
                         if (err2) {
                              console.log(err2);
                         } else {
                              result += 3;
                              console.log(result); // 6
                         }
                    });
               }
          });
     }
});

    Convinced yet? Even though there is some seemingly unnecessary error handling done here, I assume you get the drift! The problem with such code is more than just indentation. Instead, our programs entire flow is based on side effects – one function only incidentally calling the inner function.

    There are multiple ways in which we can avoid writing such deeply nested code. Let’s have a look at our options:

    Promises

    According to the official specification, promise represents an eventual result of an asynchronous operation. Basically, it represents an operation that has not completed yet but is expected to in the future. The then method is a major component of a promise. It is used to get the return value (fulfilled or rejected) of a promise. Only one of these two values will ever be set. Let’s have a look at a simple file read example without using promises:

    fs.readFile(filePath, (err, result) => {
     if (err) { console.log(err); }
     console.log(data);
});

    Now, if readFile function returned a promise, the same logic could be written like so:

    var fileReadPromise = fs.readFile(filePath);
fileReadPromise.then(console.log, console.error)

    The fileReadPromise can then be passed around multiple times in a code where you need to read a file. This helps in writing robust unit tests for your code since you now only have to write a single test for a promise. And more readable code!

    Chaining using promises

    The then function itself returns a promise which can again be used to do the next operation. Changing the first code snippet to using promises results in this:

    foo(arg, (err, val) => {
     if (err) {
          console.log(err);
     } else {
          val += 1;
          bar(val, (err1, val1) => {
               if (err) {
                    console.log(err1);
               } else {
                    val1 += 2;
                    baz(val1, (err2, result) => {
                         if (err2) {
                              console.log(err2);
                         } else {
                              result += 3;
                              console.log(result); // 6
                         }
                    });
               }
          });
     }
});

    As in evident, it makes the code more composed, readable and easier to maintain. Also, instead of chaining we could have used Promise.all. Promise.all takes an array of promises as input and returns a single promise that resolves when all the promises supplied in the array are resolved. Other useful information on promises can be found here.

    The async utility module

    Async is an utility module which provides a set of over 70 functions that can be used to elegantly solve the problem of callback hells. All these functions follow the Node.js convention of error-first callbacks which means that the first callback argument is assumed to be an error (null in case of success). Let’s try to solve the same foo-bar-baz problem using async module. Here is the code snippet:

    function foo(arg, callback) {
  if (arg < 0) {
    callback('error');
    return;
  }
  callback(null, arg+1);
}

function bar(arg, callback) {
  if (arg < 0) {
    callback('error');
    return;
  }
  callback(null, arg+2);
}

function baz(arg, callback) {
  if (arg < 0) {
    callback('error');
    return;
  }
  callback(null, arg+3);
}

async.waterfall([
  (cb) => {
    foo(0, cb);
  },
  (arg, cb) => {
    bar(arg, cb);
  },
  (arg, cb) => {
    baz(arg, cb);
  }
], (err, result) => {
  if (err) {
    console.log(err);
  } else {
    console.log(result); //6
  }
});

    Here, I have used the async.waterfall function as an example. There are a multiple functions available according to the nature of the problem you are trying to solve like async.each – for parallel execution, async.eachSeries – for serial execution etc.

    Async/Await

    Now, this is one of the most exciting features coming to Javascript in near future. It internally uses promises but handles them in a more intuitive manner. Even though it seems like promises and/or 3rd party modules like async would solve most of the problems, a further simplification is always welcome! For those of you who have worked with C# async/await, this concept is directly cribbed from there and being brought into ES7. 

    Async/await enables us to write asynchronous promise-based code as if it were synchronous, but without blocking the main thread. An async function always returns a promise whether await is used or not. But whenever an await is observed, the function is paused until the promise either resolves or rejects. Following code snippet should make it clearer:

    async function asyncFun() {
  try {
    const result = await promise;
  } catch(error) {
    console.log(error);
  }
}

    Here,  asyncFun is an async function which captures the promised result using await. This has made the code readable and a major convenience for developers who are more comfortable with linearly executed languages, without blocking the main thread. 

    Now, like before, lets solve the foo-bar-baz problem using async/await. Note that foo, bar and baz individually return promises just like before. But instead of chaining, we have written the code linearly.

    async fooBarBaz(arg) {
  try {
  const fooResponse = await foo(arg+1);
  const barResponse = await bar(arg+2);
  const bazResponse = await baz(arg+3);

  return bazResponse;
  } catch (error) {
    return Error(error);
  }
}

    How long should you (a)wait for async to come to fore?

    Well, it’s already here in the Chrome 55 release and the latest update of the V8 engine.  The native support in the language means that we should see a much more widespread use of this feature. The only, catch is that if you would want to use async/await on a codebase which isn’t promise aware and based completely on callbacks, it probably will require a lot of wrapping the existing functions to make them usable.

    To wrap up, async/await definitely make coding numerous async operations an easier job. Although promises and callbacks would do the job for most, async/await looks like the way to make some architectural problems go away and improve code quality. 

  • Hacking Your Way Around AWS IAM Roles

    Identity and Access Management (IAM) offers role-based access control (RBAC) to your AWS account users and resources, and you can granularize the permission set by defining the policy. If you are familiar or even a beginner with AWS cloud, you know how important IAM is.

    “AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources.”

    – AWS IAM User Guide

    ‍

    With the emergence of cloud infrastructure services, the coolest thing you can do is write your infrastructure as code. AWS offers SDKs for various programming/scripting languages, and of course, like any other API call, you need to sign a request with tokens. The AWS IAM console lets you generate access_key and secret_access_key tokens. This token can then be configured with your SDK. 

    Alternatively, you can configure the token with your user profile via aws cli. This also means anyone with access_key and secret_access_key will have permissions configured as per the IAM policy. Thus, keeping credentials on the disk is insecure. You can implement a key rotation policy to keep the environment compliant. To even overcome this, you can use the AWS IAM role for services. 

    Let’s say if you are working on an AWS EC2 instance that needs access to some other AWS service, like S3. You can create an IAM role for EC2 with a policy that has appropriate permission to access the S3 bucket. In this case, your SDK doesn’t need a token (not at least on the disk or hardcoded in code). Let’s take a look at the hierarchy of how the AWS SDK looks for a token for signing requests.

    1. Embedded in your code (very insecure). This is the very first place your SDK looks for. Below is a NodeJS example, where access_key and secret_access_key are part of the code itself.

    const {S3} = require("aws-sdk");
const s3 = new S3({
   accessKeyId : "ABCDEFGHIJKLMNOPQRST",
   secretAccessKey : "7is/HVjA8lm9hRrJyZEPWAs5Bo8KyyvEqjjxIHoO"
  //sessionToken : "options_session_token_if_applicable"
});

    2. AWS environment variables. If the token is not embedded in your code, your SDK looks for AWS environment variables available to process. These environment variables are AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY, and optional AWS_SESSION_TOKEN. Below is an example where AWS credentials are exported and the aws cli command is used to list S3 buckets. Note that once credentials are exported, they are available to all the child processes. Therefore, these credentials are auto looked up by your AWS SDK.

    3. The AWS credentials (default profile) file located at ~/.aws/credentials. This is the third place for the lookup. You can generate this file by running the command aws configure. You may also manually create this file with various profiles. If you happen to have multiple profiles, you can then export an environment variable called AWS_PROFILE. An example credentials file is given below:

    [default] ; default profile
aws_access_key_id = <DEFAULT_ACCESS_KEY_ID>
aws_secret_access_key = <DEFAULT_SECRET_ACCESS_KEY>
  
[personal-account] ; personal account profile
aws_access_key_id = <PERSONAL_ACCESS_KEY_ID>
aws_secret_access_key = <PERSONAL_SECRET_ACCESS_KEY>
  
[work-account] ; work account profile
aws_access_key_id = <WORK_ACCESS_KEY_ID>
aws_secret_access_key = <WORK_SECRET_ACCESS_KEY>

    4. The IAM role attached to your resource. Your resource could be EC2 Instance, Lambda function, AWS glue, ECS Container, RDS, etc. Now, this is a secure way of using credentials. Since your credentials are not stored anywhere on the disk, exported via an environment variable, or hardcoded in the code. You need not worry about key rotation at all.

    TL;DR: IAM roles are a secure way of using credentials. However, they are only applicable to resources within AWS. You can not use them outside of AWS. So, the IAM role can only be attached to resources like EC2, Lambda, ECS, etc.

    The problem statement:

    Let’s say a group of developers needs access to a few S3 buckets and DynamoDB. The organization does not want developers to use access_key and secret_access_key on their local machine (laptop) as access_key and secret_access can be used anywhere or can be stolen. 

    Since IAM roles are more secure, they allocate EC2 with Windows OS and attach the IAM role with appropriate permission to access S3 buckets and DynamoDB and configure IDE and other essential dev tools. Developers then use RDP to connect to EC2 Instance. However, due to license restrictions, only two users can connect with RDP at a given time. So, they add more similar instances. This heavily increases cost. Wouldn’t it be nice, if somehow, IAM roles could be attached to local machines?

    How do IAM roles for resources work?

    Resources like EC2 or Lambda have the link-local address available. The link-local address 169.254.169.254 can be accessed over HTTP port 80 to retrieve instance metadata. For instance, to get the instance-id of an EC2 instance from the host itself, you can query with a GET request to curl -L http://169.254.169.254/latest/meta-data/instance-id/. Similarly, you can retrieve IAM credentials if the IAM role is attached to the EC2 instance. Let’s  assume you have created an IAM role for an EC2 instance with the name “iam-role-for-ec2”. Your SDK will then automatically access credentials via a GET request to curl -L http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-role-for-ec2/

    $ curl -L 169.254.169.254/latest/meta-data/iam/security-credentials/iam-role-for-ec2/
{
 "Code" : "Success",
 "LastUpdated" : "2021-08-03T09:18:49Z",
 "Type" : "AWS-HMAC",
 "AccessKeyId" : "ASIASP26DFHDIOFNJFFX",
 "SecretAccessKey" : "EK1A7x9dntSzF9LlG7BK08C6zpTS/F6MHYTBo/+U",
 "Token" : "IQoJb3JpZ2luX2VjEPr//////////wEaCXVzLXdlc3QtMiJIMEYCIQCOCqHrHjEkYZUFsRtGXwa8gfGjsBmaU+WrL2Z0ihvA3QIhAIsGhJFiPetOod7IUUC++unWZfoUEgjEU0ULYwZUvGwwKvoDCBIQAhoMMTcxNDU5MTYwNTE4IgxFUXJfE/0cdJs2Gigq1wM8Ww8yAS2i2qUqsQ1t+yd4ATkE5fvIMDtHxzPQ2raVQb+cCgC/eJVQpeNET1SP01HnrN5W1QFID+xOPk3vZt6NrCy48OUf6+cCGrd63Jv/7glAsyQGaGM/Jt5ddi6593dgN7VLFHsEBAwqkZ3j/VjAzYbthP3clmRl++6k+vpiUp2j4uwM4zW/6f8faR6awPbPVmJsyh94pXaQXJU+H0w+9Hp0MlUvP6GRqBiuTwv/+EOiRfth1XGRxxOuR5X+fr0Ve4tede2x0ZvSLeUsUENHlOQnUkSGbu1Hiv1BhDEjhzbHi7PXhW1G9N1FZObE+wdF4hGYbe3LUUIrnp2xnIcxKzmume2YQvFE4DvJvBtF22DsdLP4GPmitofhV2FGcVxP1f5Nv76M6SfOQY65vSZQde4LIwcotRIrMgwEWup2Rplq6s56K93IYXp6QmnUWLgdtcMBTMVQsOFhCdj05P+VYqlKe5xRT4/8BucmIHn7+J4indNoL+3BvYvnpiISdcEhlyswNZOPhVQJjwJfKPPdu9NDEKQ+Jep4wpVvOSh+CAtxKtqwGz1wrKzqlRvzqBFaEQrD4WdPdf9YnTvmKIXgPuk74pZRlarVsREL0KmG6G0zzA2lRYow6JOkiAY6pAHIZGH+UH5RL79drKe86tUnWCORcX9omN2uUK7FemTENwyvholib4jLGY6HcjvDF10jqkcu1KEV20xNsPj87BP7irEH7xH//Jz2+rnSaN5PCqLezSsATPYhHFQjg6Oti+0E33F+F5MA25Pn2+u5TDP1VfFgYExwSor79gNtwbOMs76432ssHYFioYjHttPfVwyNXloLCwgphqJBwiNhMDMcKapK6Q==",
 "Expiration" : "2021-08-03T15:47:26Z"
}

    Notice that the response payload is JSON with AccessKeyId, SecretAccessKey, and Token. Additionally, there is an Expiration key, which states the validity of the token. This means the token is autogenerated once they expire.

    Solution:

    Now that you know how IAM roles work and how important link-local address is, you have probably guessed what needs to be done so that you can access IAM role credentials from your local machine. The two solutions that popup in my mind are:

    1. Host a lightweight reverse proxy server like Nginx and then write a wrapper around your SDK so that initial calls are made to EC2 and credentials are retrieved.

    2. Route traffic originating from your system, targeting 169.254.169.254. Traffic should reach the EC2 instance and EC2 itself should take care of forwarding packets to the instance metadata server.

    The second solution may sound pretty techy, but it is the ideal solution, and you don’t need to do additional tweaking in your SDK. The developer is transparent about what is being implemented. This blog will focus on implementing a second solution.

    Implementation:

    1. Launch a Linux (Ubuntu 20.04 LTS prefered) EC2 instance from AWS console and attach the IAM role with appropriate permissions. The instance should be in the public subnet and make sure to attach an Elastic IP address. Whitelist incoming port 1194 UDP (open to world) and port 22 (ssh, open to your IP address only) TCP in your instance security group.

    2. Install OpenVPN and git package. apt update; apt install git openvpn.

    3. Clone easy-rsa repository on your server. cd ~;git clone https://github.com/OpenVPN/easy-rsa.git

    4. Generate certificates for OpenVPN server and client using easy-rsa.

    #switch to easy-rsa directory
cd ~/easy-rsa/easyrsa3
#copy vars.example to vars
cp vars.example vars
#Find below variables in "vars" file and edit them according to your needs
set_var EASYRSA_REQ_COUNTRY    "US"
set_var EASYRSA_REQ_PROVINCE   "California"
set_var EASYRSA_REQ_CITY       "San Francisco"
set_var EASYRSA_REQ_ORG        "Copyleft Certificate Co"
set_var EASYRSA_REQ_EMAIL      "me@example.net"
set_var EASYRSA_REQ_OU         "My Organizational Unit"
#Also edit below two variables if you plan to run easyrsa in non-interactive mode
# EASYRSA_REQ_CN should be set to your ElasticIP Address.
# Note: If your are using openvpn behind a load balancer, or if you plan to map DNS to your server, then this should be set to your DNS name
set_var EASYRSA_REQ_CN         "Your Instance Elastic IP"
set_var EASYRSA_BATCH          "NONEMPTY"
#====================================================
#Generate certificate and keys for server and client
./easyrsa init-pki
./easyrsa build-ca nopass
./easyrsa gen-dh
./easyrsa build-server-full server nopass
./easyrsa build-client-full client nopass
#Copy certificates and keys to server configuration
cp -p ./pki/ca.crt /etc/openvpn/
cp -p ./pki/issued/server.crt /etc/openvpn/
cp -p ./pki/private/server.key /etc/openvpn/
cp -p ./pki/dh.pem /etc/openvpn/dh2049.pem
cd /etc/openvpn
openvpn --genkey --secret myvpn.tlsauth
echo "net.ipv4.ip_forward = 1" >>/etc/sysctl.conf
sysctl -p

    5. Configure OpenVPN server.conf file:

    port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key # This file should be kept secret
dh dh2048.pem
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 1.1.1.1"
push "route 169.254.169.254 255.255.255.255"
keepalive 10 120
tls-auth myvpn.tlsauth 0
cipher AES-256-CBC
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
log-append  /var/log/openvpn.log
verb 4
explicit-exit-notify 1
remote-cert-eku "TLS Web Client Authentication"

    In the above configuration file, make sure line number 9 is not conflicting with your AWS VPC CIDR. Line number 14 (push “route 169.254.169.254 255.255.255.255”) does a trick for us and is the heart of this blog post. This assures that when a client connects via OpenVPN, a route is added to the client machine so that packets targeting 168.254.169.254 are routed via OpenVPN tunnel. (Note: If you do not add this here, you can manually add a route to your client-side once OpenVPN is connected. ip route add 169.254.169.254/32 YOUR_TUNNEL_IP dev tun0)

    6. Generate an OpenVPN client configuration file:

    #These commands are executed on your EC2 (OopenvVpn)
cd ~/easy-rsa/easyrsa3
cat <<EOF >/tmp/client.ovpn
client
dev tun
proto udp
remote YOUR-ELASTIC-IP-ADDRESS 1194
resolv-retry infinite
nobind
persist-key
persist-tun
cipher AES-256-CBC
comp-lzo
verb 3
key-direction 1
EOF
#append ca certificate
echo '<ca>' >>/tmp/client.ovpn
cat ./pki/ca.crt >>/tmp/client.ovpn
echo '</ca>' >>/tmp/client.ovpn
#append client certificate
echo '<cert>' >>/tmp/client.ovpn
sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/{p;/END CERTIFICATE/q}' ./pki/issued/client.crt >>/tmp/client.ovpn
echo '</cert>' >>/tmp/client.ovpn
#append client key
echo '<key>' >>/tmp/client.ovpn
cat ./pki/private/client.key >>/tmp/client.ovpn
echo '</key>' >>/tmp/client.ovpn
#append TLS auth key
echo '<tls-auth>' >>/tmp/client.ovpn
cat /etc/openvpn/myvpn.tlsauth >>/tmp/client.ovpn
echo '</tls-auth>' >>/tmp/client.ovpn

    In the above configuration file, make sure to update line number 9. This could be your EC2 elastic IP address (or domain if mapped and configured).

    7. Finally, download the /tmp/client.ovpn file to your local machine. Install the OpenVPN client software, import the client.ovpn file, and connect. If you are using a Linux machine, you may connect using sudo openvpn –config /path/to/client.ovpn.

    Testing:

    Let us say you have configured the IAM role with permission that lets you list S3 buckets. You should be able to access AWS resources once the OpenVPN client is connected. Your SDK should automatically look for credentials via metadata link-local address. You may install the aws-cli utility and run aws s3 ls to list S3 buckets.

    Conclusion:

    IAM roles are meant to be used with AWS resources like EC2, ECS, Lambda, etc. so that you don’t keep the credentials hardcoded in the code or in the configuration file left unsecured on the disk. Our goal was to use the IAM role directly from the local machine (laptop). We achieved this by using OpenVPN secure SSL tunnel. The VPN assures that we are in a private network, thus keeping the environment compliant. This guide is not meant for how one should set up an OpenVPN server/client. Therefore, you must harden the OpenVPN server. You may put the server behind the network load balancer and may enforce MAC binding features to your clients.

  • Container Security: Let’s Secure Your Enterprise Container Infrastructure!

    Introduction

    Containerized applications are becoming more popular with each passing year. A reason for this rise in popularity could be the pivotal role they play in Continuous Delivery by enabling fast and automated deployment of software services.

    Security still remains a major concern mainly because of the way container images are being used. In the world of VMs, infra/security team used to validate the OS images and installed packages for vulnerabilities. But with the adoption of containers, developers are building their own container images. Images are rarely built from scratch. They are typically built on some base image, which is itself built on top of other base images. When a developer builds a container image, he typically grabs a base image and other layers from public third party sources. These images and libraries may contain obsolete or vulnerable packages, thereby putting your infrastructure at risk. An added complexity is that many existing vulnerability-scanning tools may not work with containers, nor do they support container delivery workflows including registries and CI/CD pipelines. In addition, you can’t simply scan for vulnerabilities – you must scan, manage vulnerability fixes and enforce vulnerability-based policies.

    The Container Security Problem

    The table below shows the number of vulnerabilities found in the images available on dockerhub. Note that (as of April 2016) the worst offending community images contained almost 1,800 vulnerabilities! Official images were much better, but still contained 392 vulnerabilities in the worst case.

    If we look at the distribution of vulnerability severities, we see that pretty much all of them are high severity, for both official and community images. What we’re not told is the underlying distribution of vulnerability severities in the CVE database, so this could simply be a reflection of that distribution.

    Over 80% of the latest versions of official images contained at least one high severity vulnerability!

    • There are so many docker images readily available on dockerhub – are you sure the ones you are using are safe?
    • Do you know where your containers come from?
    • Are your developers downloading container images and libraries from unknown and potentially harmful sources?
    • Do the containers use third party library code that is obsolete or vulnerable?

    In this blog post, I will explain some of the solutions available which can help with these challenges. Solutions like ‘Docker scanning services‘, ‘Twistlock Trust’ and an open-source solution ‘Clair‘ from Coreos.com which can help in scanning and fixing vulnerability problems making your container images secure.

    Clair

    Clair is an open source project for the static analysis of vulnerabilities in application containers. It works as an API that analyzes every container layer to find known vulnerabilities using existing package managers such as Debian (dpkg), Ubuntu (dpkg), CentOS (rpm). It also can be used from the command line. It provides a list of vulnerabilities that threaten a container, and can notify users when new vulnerabilities that affect existing containers become known. In regular intervals, Clair ingests vulnerability metadata from a configured set of sources and stores it in the database. Clients use the Clair API to index their container images; this parses a list of installed source packages and stores them in the database. Clients use the Clair API to query the database; correlating data in real time, rather than a cached result that needs re-scanning.

    Clair identifies security issues that developers introduce in their container images. The vanilla process for using Clair is as follows:

    1. A developer programmatically submits their container image to Clair
    2. Clair analyzes the image, looking for security vulnerabilities
    3. Clair returns a detailed report of security vulnerabilities present in the image
    4. Developer acts based on the report

    How to use Clair

    Docker is required to follow along with this demonstration. Once Docker is installed, use the Dockerfile below to create an Ubuntu image that contains a version of SSL that is susceptible to Heartbleed attacks.

    #Dockerfile
FROM ubuntu:precise-20160303
#Install WGet
RUN apt-get update
RUN apt-get -f install
RUN apt-get install -y wget
#Install an OpenSSL vulnerable to Heartbleed (CVE-2014-0160)
RUN wget --no-check-certificate https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/5436462/+files/openssl_1.0.1-4ubuntu5.11_amd64.deb
RUN dpkg -i openssl_1.0.1-4ubuntu5.11_amd64.deb

    Build the image using below command:

    $ docker build . -t madhurnawandar/heartbeat

    After creating the insecure Docker image, the next step is to download and install Clair from here. The installation choice used for this demonstration was the Docker Compose solution. Once Clair is installed, it can be used via querying its API or through the clairctl command line tool. Submit the insecure Docker image created above to Clair for analysis and it will catch the Heartbleed vulnerability.

    $ clairctl analyze --local madhurnawandar/heartbeat
Image: /madhurnawandar/heartbeat:latest
9 layers found 
➜ Analysis [f3ce93f27451] found 0 vulnerabilities. 
➜ Analysis [738d67d10278] found 0 vulnerabilities. 
➜ Analysis [14dfb8014dea] found 0 vulnerabilities. 
➜ Analysis [2ef560f052c7] found 0 vulnerabilities. 
➜ Analysis [69a7b8948d35] found 0 vulnerabilities. 
➜ Analysis [a246ec1b6259] found 0 vulnerabilities. 
➜ Analysis [fc298ae7d587] found 0 vulnerabilities. 
➜ Analysis [7ebd44baf4ff] found 0 vulnerabilities. 
➜ Analysis [c7aacca5143d] found 52 vulnerabilities.
$ clairctl report --local --format json madhurnawandar/heartbeat
JSON report at reports/json/analysis-madhurnawandar-heartbeat-latest.json

    You can view the detailed report here.

    Docker Security Scanning

    Docker Cloud and Docker Hub can scan images in private repositories to verify that they are free from known security vulnerabilities or exposures, and report the results of the scan for each image tag. Docker Security Scanning is available as an add-on to Docker hosted private repositories on both Docker Cloud and Docker Hub.

    Security scanning is enabled on a per-repository basis and is only available for private repositories. Scans run each time a build pushes a new image to your private repository. They also run when you add a new image or tag. The scan traverses each layer of the image, identifies the software components in each layer, and indexes the SHA of each component.

    The scan compares the SHA of each component against the Common Vulnerabilities and Exposures (CVE®) database. The CVE is a “dictionary” of known information security vulnerabilities. When the CVE database is updated, the service reviews the indexed components for any that match the new vulnerability. If the new vulnerability is detected in an image, the service sends an email alert to the maintainers of the image.

    A single component can contain multiple vulnerabilities or exposures and Docker Security Scanning reports on each one. You can click an individual vulnerability report from the scan results and navigate to the specific CVE report data to learn more about it.

    Twistlock

    Twistlock is a rule-based access control policy system for Docker and Kubernetes containers. Twistlock is able to be fully integrated within Docker, with out-of-the-box security policies that are ready to use.

    Security policies can set the conditions for users to, say, create new containers but not delete them; or, they can launch containers but aren’t allowed to push code to them. Twistlock features the same policy management rules as those on Kubernetes, wherein a user can modify management policies but cannot delete them.

    Twistlock also handles image scanning. Users can scan an entire container image, including any packaged Docker application. Twistlock has done its due-diligence in this area, correlating with Red Hat and Mirantis to ensure no container is left vulnerable while a scan is running.

    Twistlock also deals with image scanning of containers within the registries themselves. In runtime environments, Twistlock features a Docker proxy running on the same server with an application’s other containers. This is essentially traffic filtering, whereupon the application container calling the Docker daemon is then re-routed through Twistlock. This approach enforces access control, allowing for safer configuration where no containers are set to run as root. It’s also able to SSH into an instance, for example. In order to delve into these layers of security, Twistlock enforces the policy at runtime.

    When new code is written in images, it is then integrated into the Twistlock API to push an event, whereupon the new image is deposited into the registry along with its unique IDs. It is then pulled out by Twistlock and scanned to ensure it complies with the set security policies in place. Twistlock deposits the scan result into the CI process so that developers can view the result for debugging purposes.

    Integrating these vulnerability scanning tools into your CI/CD Pipeline:

    These tools becomes more interesting paired with a CI server like Jenkins, TravisCI, etc. Given proper configuration, process becomes:

    1. A developer submits application code to source control
    2. Source control triggers a Jenkins build
    3. Jenkins builds the software containers necessary for the application
    4. Jenkins submits the container images to vulnerability scanning tool
    5. Tool identifies security vulnerabilities in the container
    6. Jenkins receives the security report, identifies a high vulnerability in the report, and stops the build

    Conclusion

    There are many solutions like ‘Docker scanning services’, ‘Twistlock Trust’, ‘Clair‘, etc to secure your containers. It’s critical for organizations to adopt such tools in their CI/CD pipelines. But this itself is not going to make containers secure. There are lot of guidelines available in the CIS Benchmark for containers like tuning kernel parameters, setting proper network configurations for inter-container connectivity, securing access to host level directories and others. I will cover these items in the next set of blogs. Stay tuned!

  • Exploring Upgrade Strategies for Stateful Sets in Kubernetes

    Introduction

    In the age of continuous delivery and agility where the software is being deployed 10s of times per day and sometimes per hour as well using container orchestration platforms, a seamless upgrade mechanism becomes a critical aspect of any technology adoption, Kubernetes being no exception. 

    Kubernetes provides a variety of controllers that define how pods are set up and deployed within the Kubernetes cluster. These controllers can group pods together according to their runtime needs and can be used to define pod replication and pod startup ordering. Kubernetes controllers are nothing but an application pattern. The controller controls the pods(smallest unit in Kubernetes), so, you don’t need to create, manage and delete the pods. There are few types of controllers in Kubernetes like,

    1. Deployment
    2. Statefulset
    3. Daemonset
    4. Job
    5. Replica sets

    Each controller represents an application pattern. For example, Deployment represents the stateless application pattern in which you don’t store the state of your application. Statefulset represents the statefulset application pattern where you store the data, for example, databases, message queues.  We will be focusing on Statefulset controller and its update feature in this blog.

    Statefulset

    The StatefulSet acts as a controller in Kubernetes to deploy applications according to a specified rule set and is aimed towards the use of persistent and stateful applications. It is an ordered and graceful deployment. Statefulset is generally used with a distributed applications that require each node to have a persistent state and the ability to configure an arbitrary number of nodes. StatefulSet pods have a unique identity that is comprised of an ordinal, a stable network identity, and stable storage. The identity sticks to the pod, regardless of which node it’s scheduled on. For more details check here.

    Update Strategies FOR STATEFULSETS

    There are a couple of different strategies available for upgrades – Blue/Green and Rolling updates. Let’s review them in detail:

    Blue-Green Deployment : Blue-green deployment is one of the commonly used update strategies. There are 2 identical environments of your application in this strategy. One is the Blue environment which is running the current deployment and the Green environment is the new deployment to which we want to upgrade. The approach is simple:

    1. Switch the load balancer to route traffic to the Green environment.
    2. Delete the Blue environment once the Green environment is verified. 

    Disadvantages of Blue-Green deployment:

    1. One of the disadvantages of this strategy is that all current transactions and sessions will be lost, due to the physical switch from one machine serving the traffic to another one.
    2. Implementing blue-green deployment become complex with the database, especially if, the database schema changes across version.
    3. In blue-green deployment, you need the extra cloud setup/hardware which increases the overall costing.

    Rolling update strategy

    After Blue-Green deployment, let’s take a look at Rolling updates and how it works.

    1. In short, as the name suggests this strategy replaces currently running instances of the application with new instances, one by one. 
    2. In this strategy, health checks play an important role i.e. old instances of the application are removed only if new version are healthy. Due to this, the existing deployment becomes heterogeneous while moving from the old version of the application to new version. 
    3. The benefit of this strategy is that its incremental approach to roll out the update and verification happens in parallel while increasing traffic to the application.
    4. In rolling update strategy, you don’t need extra hardware/cloud setup and hence it’s cost-effective technique of upgrade.

    Statefulset upgrade strategies

    With the basic understanding of upgrade strategies, let’s explore the update strategies available for Stateful sets in Kubernetes. Statefulsets are used for databases where the state of the application is the crucial part of the deployment. We will take the example of Cassandra to learn about statefulset upgrade feature. We will use the gce-pd storage to store the data. StatefulSets(since Kubernetes 1.7) uses an update strategy to configure and disable automated rolling updates for containers, labels, resource request/limits, and annotations for its pods. The update strategy is configured using the updateStrategy field.

    The updateStrategy field accepts one of the following value 

    1. OnDelete
    2. RollingUpdate

    OnDelete update strategy

    OnDelete prevents the controller from automatically updating its pods. One needs to delete the pod manually for the changes to take effect. It’s more of a manual update process for the Statefulset application and this is the main difference between OnDelete and RollingUpdate strategy. OnDelete update strategy plays an important role where the user needs to perform few action/verification post the update of each pod. For example, after updating a single pod of Cassandra user might need to check if the updated pod joined the Cassandra cluster correctly.

    We will now create a Statefulset deployment first. Let’s take a simple example of Cassandra and deploy it using a Statefulset controller. Persistent storage is the key point in Statefulset controller. You can read more about the storage class here.

    For the purpose of this blog, we will use the Google Kubernetes Engine.

    • First, define the storage class as follows:
    apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

    • Then create the Storage class using kubectl:
    $ kubectl create -f storage_class.yaml

    • Here is the YAML file for the Cassandra service and the Statefulset deployment.
    apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  updateStrategy:
    type: OnDelete
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v12
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
           cpu: "500m"
           memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast"
      resources:
        requests:
          storage: 5Gi

    • Let’s create the Statefulset now.
    $ kubectl create -f cassandra.yaml

    • After creating Cassandra Statefulset, if you check the running pods then you will find something like,
    $ kubectl get podsNAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 2m
cassandra-1 1/1 Running 0 2m
cassandra-2 1/1 Running 0 2m

    • Check if Cassandra cluster is formed correctly using following command:
    $ kubectl exec -it cassandra-0 -- nodetool statusDatacenter: DC1-K8Demo
#ERROR!
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving

Address Load Tokens Owns Host ID Rack
UN 192.168.4.193 101.15 KiB 32 72.0% abd9f52d-85ef-44ee-863c-e1b174cd9412 Rack1-K8Demo
UN 192.168.199.67 187.81 KiB 32 72.8% c40e89e4-44fe-4fc2-9e8a-863b6a74c90c Rack1-K8Demo
UN 192.168.187.196 131.42 KiB 32 55.2% c235505c-eec5-43bc-a4d9-350858814fe5 Rack1-K8Demo

    • Let’s describe the running pod first before updating. Look for the image field in the output of the following command
    $ kubectl describe pod cassandra-0

    • The Image field will show gcr.io/google-samples/cassandra:v12 . Now, let’s patch the Cassandra statefulset with the latest image to which we want to update. The latest image might contain the new Cassandra version or database schema changes. Before upgrading such crucial components, it’s always safe to have the backup of the data,
    $ kubectl patch statefulset cassandra --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"gcr.io/google-samples/cassandra:v13"}]'

    You will see output as `statefulset.apps “cassandra” patched`, but controller won’t update the running pod automatically in this strategy. You need to delete the pods once and wait till pods with new configuration comes up. Let’s try deleting the cassandra-0 pod.

    $ kubectl delete pod cassandra-0

    • Wait till cassandra-0 comes up in running state and then check if the cassandra-0 is running with intended/updated image i.e. gcr.io/google-samples/cassandra:v13 Now, cassandra-0 is running the new image while cassandra-1 and cassandra-2 are still running the old image. You need to delete these pods for the new image to take effect in this strategy.

    Rolling update strategy

    Rolling update is an automated update process. In this, the controller deletes and then recreates each of its pods. Pods get updated one at a time. While updating, the controller makes sure that an updated pod is running and is in ready state before updating its predecessor. The pods in the StatefulSet are updated in reverse ordinal order(same as pod termination order i.e from the largest ordinal to the smallest)

    For the rolling update strategy, we will create the Cassandra statefulset with the .spec.updateStrategy field pointing to RollingUpdate. 

    apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v12
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
           cpu: "500m"
           memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast"
      resources:
        requests:
          storage: 5Gi

    • To try the rolling update feature, we can patch the existing statefulset with the updated image.
    $ kubectl patch statefulset cassandra --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"gcr.io/google-samples/cassandra:v13"}]'

    • Once you execute the above command, monitor the output of the following command,
    $ kubectl get pods -w

    In the case of failure in update process, controller restores any pod that fails during the update to its current version i.e. pods that have already received the update will be restored to the updated version, and pods that have not yet received the update will be restored to the previous version.

    Partitioning a RollingUpdate (Staging an Update)

    The updateStrategy contains one more field for partitioning the RollingUpdate. If a partition is specified, all pods with an ordinal greater than or equal to that of the provided partition will be updated and the pods with an ordinal that is less than the partition will not be updated. If the pods with an ordinal value less than the partition get deleted, then those pods will get recreated with the old definition/version. This partitioning rolling update feature plays important role in the scenario where if you want to stage an update, roll out a canary, or perform a phased rollout.

    RollingUpdate supports partitioning option. You can define the partition parameter in the .spec.updateStrategy

    $ kubectl patch statefulset cassandra -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'

    In the above command, we are giving partition value as 2, which will patch the Cassandra statefulset in such a way that, whenever we try to update the Cassandra statefulset, it will update the cassandra-2 pod only. Let’s try to patch the updated image to existing statefulset.

    $ kubectl patch statefulset cassandra --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"gcr.io/google-samples/cassandra:v14"}]'

    After patching, watch the following command output,

    $ kubectl get pods -w

    You can keep decrementing the partition value and that many pods will keep taking the effect of the applied patch. For example, if you patch the statefulset with partition=0 then all the pods of the Cassandra statefulset will get updated with provided upgrade configuration.

    Verifying if the upgrade was successful

    Verifying the upgrade process of your application is the important step to conclude the upgrade. This step might differ as per the application. Here, in the blog we have taken the Cassandra example, so we will verify if the cluster of the Cassandra nodes is being formed properly.

    Use `nodetool status` command to verify the cluster. After upgrading all the pods, you might want to run some post-processing like migrating schema if your upgrade dictates that etc.

    As per the upgrade strategy, verification of your application can be done by following ways.

    1. In OnDelete update strategy, you can keep updating pod one by one and keep checking the application status to make sure the upgrade working fine.
    2. In RollingUpdate strategy, you can check the application status once all the running pods of your application gets upgraded.

    For Cassandra like application, OnDelete update is more preferred than RollingUpdate. In rolling update, we saw that Cassandra pod gets updated one by one, starting from high to low ordinal index. There might be the case where after updating 2 pods, Cassandra cluster might go in failed state but you can not recover it like the OnDelete strategy. You have to try to recover Cassandra once the complete upgrade is done i.e. once all the pods get upgraded to provided image. If you have to use the rolling update then try partitioning the rolling update.

    Conclusion

    In this blog, we went through the Kubernetes controllers and mainly through statefulsets. We learnt about the differences between blue-green deployment and rolling update strategies then we played with the Cassandra statefulset example and successfully upgraded it with update strategies like OnDelete and RollingUpdate. Do let us know if you have any questions, queries and additional thoughts in the comments section below.

  • A Quick Introduction to Data Analysis With Pandas

    Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages and makes importing and analyzing data much easier.

    Pandas aims to integrate the functionality of NumPy and matplotlib to give you a convenient tool for data analytics and visualization. Besides the integration,  it also makes the usage far more better.

    In this blog, I’ll give you a list of useful pandas snippets that can be reused over and over again. These will definitely save you some time that you may otherwise need to skim through the comprehensive Pandas docs.

    The data structures in Pandas are capable of holding elements of any type: Series, DataFrame.

    Series

    A one-dimensional object that can hold any data type such as integers, floats, and strings

    A Series object can be created of different values. Series can be remembered similar to a Python list.

    In the below example, NaN is NumPy’s nan symbol which tells us that the element is not a number but it can be used as one numerical type pointing out to be not a number. The type of series is an object because the series has mixed contents of strings and numbers.

    >>> import pandas as pd
>>> import numpy as np
>>> series = pd.Series([12,32,54,2, np.nan, "a string", 6])
>>> series
0          12
1          32
2          54
3           2
4         NaN
5    a string
6           6
dtype: object

    Now if we use only numerical values, we get the basic NumPy dtype – float for our series.

    >>> series = pd.Series([1,2,np.nan, 4])
>>> series
0    1.0
1    2.0
2    NaN
3    4.0
dtype: float64

    DataFrame

    A two-dimensional labeled data structure where columns can be of different types.

    Each column in a Pandas DataFrame represents a Series object in memory.

    In order to convert a certain Python object (dictionary, lists, etc) to a DataFrame, it is extremely easy. From the python dictionaries, the keys map to Column names while values correspond to a list of column values.

    >>> d = {
    "stats": pd.Series(np.arange(10,15,1.0)),
    "year": pd.Series(["2012","2007","2012","2003"]),
    "intake": pd.Series(["SUMMER","WINTER","WINTER","SUMMER"]),
}
>>> df = pd.DataFrame(d)
>>> df

    Reading CSV files

    Pandas can work with various file types while reading any file you need to remember.

    pd.read_filetype()

    Now you will have to only replace “filetype” with the actual type of the file, like csv or excel. You will have to give the path of the file inside the parenthesis as the first argument. You can also pass in different arguments that relate to opening the file. (Reading a csv file? See this)

    >>> df = pd.read_csv('companies.csv')
>>> df.head()
view raw

    Accessing Columns and Rows

    DataFrame comprises of three sub-components, the index, columns, and the data (also known as values).

    The index represents a sequence of values. In the DataFrame, it always on the left side. Values in an index are in bold font. Each individual value of the index is called a label. Index is like positions while the labels are values at that particular index. Sometimes the index is also referred to as row labels. In all the examples below, the labels and indexes are the same and are just integers beginning from 0 up to n-1, where n is the number of rows in the table.

    Selecting rows is done using loc and iloc:

    • loc gets rows (or columns) with particular labels from the index. Raises KeyError when the items are not found.
    • iloc gets rows (or columns) at particular positions/index (so it only takes integers). Raises IndexError if a requested indexer is out-of-bounds.
    >>> df.loc[:5]              #similar to df.head()

    Accessing the data using column names

    Pandas takes an extra step and allows us to access data through labels in DataFrames.

    >>> df.loc[:5, ["name","vertical", "url"]]

    In Pandas, selecting data is very easy and similar to accessing an element from a dictionary or a list.

    You can select a column (df[col_name]) and it will return column with label col_name as a Series, because rows and columns are stored as Series in a DataFrame, If you need to access more columns (df[[col_name_1, col_name_2]]) and it returns columns as a new DataFrame.

    Filtering DataFrames with Conditional Logic

    Let’s say we want all the companies with the vertical as B2B, the logic would be:

    >>> df[(df['vertical'] == 'B2B')]

    If we want the companies for the year 2009, we would use:

    >>> df[(df['year'] == 2009)]

    Need to combine them both? Here’s how you would do it:

    >>> df[(df['vertical'] == 'B2B') & (df['year'] == 2009)]

    Get all companies with vertical as B2B for the year 2009

    Sort and Groupby

    Sorting

    Sort values by a certain column in ascending order by using:

    >>> df.sort_values(colname)

    >>> df.sort_values(colname,ascending=False)

    Furthermore, it’s also possible to sort values by multiple columns with different orders. colname_1 is being sorted in ascending order and colname_2 in descending order by using:

    >>> df.sort_values([colname_1,colname_2],ascending=[True,False])

    Grouping

    This operation involves 3 steps; splitting of the data, applying a function on each of the group, and finally combining the results into a data structure. This can be used to group large amounts of data and compute operations on these groups.

    df.groupby(colname) returns a groupby object for values from one column while df.groupby([col1,col2]) returns a groupby object for values from multiple columns.

    Data Cleansing

    Data cleaning is a very important step in data analysis.

    Checking missing values in the data

    Check null values in the DataFrame by using:

    >>> df.isnull()

    This returns a boolean array (an array of true for missing values and false for non-missing values).

    >>> df.isnull().sum()

    Check non null values in the DataFrame using pd.notnull(). It returns a boolean array, exactly converse of df.notnull()

    Removing Empty Values

    Dropping empty values can be done easily by using:

    >>> df.dropna()

    This drops the rows having empty values or df.dropna(axis=1) to drop the columns.

    Also, if you wish to fill the missing values with other values, use df.fillna(x). This fills all the missing values with the value x (here you can put any value that you want) or s.fillna(s.mean()) which replaces null values with the mean (mean can be replaced with any function from the arithmetic section).

    Operations on Complete Rows, Columns, or Even All Data

    >>> df["url_len"] = df["url"].map(len)

    The .map() operation applies a function to each element of a column.

    .apply() applies a function to columns. Use .apply(axis=1) to do it on the rows.

    Iterating over rows

    Very similar to iterating any of the python primitive types such as list, tuples, dictionaries.

    >>> for i, row in df.iterrows():
        print("Index {0}".format(i))
        print("Row {0}".format(row))

    The .iterrows() loops 2 variables together i.e, the index of the row and the row itself, variable i is the index and variable row is the row in the code above.

    Tips & Tricks

    Using ufuncs (also known as Universal Functions). Python has the .apply() which applies a function to columns/rows. Similarly, Ufuncs can be used while preprocessing. What is the difference between ufuncs and .apply()?

    Ufuncs is a numpy library, implemented in C which is highly efficient (ufuncs are around 10 times faster).

    A list of common Ufuncs:

    isinf: Element-wise checks for positive or negative infinity.

    isnan: Element-wise checks for NaN and returns result as a boolean array.

    isnat: Element-wise checks for NaT (not time) and returns result as a boolean array.

    trunc: Return the truncated value of the input, element-wise.

    .dt commands: Element-wise processing for date objects.

    High-Performance Pandas

    Pandas performs various vectorized/broadcasted operations and grouping-type operations. These operations are efficient and effective.

    As of version 0.13, Pandas included tools that allow us to directly access C-speed operations without costly allocation of intermediate arrays. There are two functions, eval() and query().‍

    DataFrame.eval() for efficient operations:

    >>> import pandas as pd
>>> nrows, ncols = 100000, 100
>>> rng = np.random.RandomState(42)
>>> df1, df2, df3, df4 = (pd.DataFrame(rng.rand(nrows, ncols))
                      for i in range(4))

    To compute the sum of df1, df2, df3, and df4 DataFrames using the typical Pandas approach, we can just write the sum:

    >>> %timeit df1 + df2 + df3 + df4

10 loops, best of 3: 103.1 ms per loop

    A better and optimized approach for the same operation can be computed via pd.eval():

    >>> %timeit pd.eval('df1 + df2 + df3 + df4')

10 loops, best of 3: 53.6 ms per loop

    %timeit — Measure execution time of small code snippets.

    The eval() expression is about 50% faster (it also consumes mush less memory).

    And it performs the same result:

    >>> np.allclose(df1 + df2 + df3 + df4,d.eval('df1 + df2 + df3 + df4'))

True

    np.allclose() is a numpy function which returns True if two arrays are element-wise equal within a tolerance.

    Column-Wise & Assignment Operations Using df.eval()

    Normal expression to split the first character of a column and assigning it to the same column can be done by using:

    >>> df['batch'] = df['batch'].str[0]

    By using df.eval(), same expression can be performed much faster:

    >>> df.eval("batch=batch.str[0]")

    DataFrame.query() for efficient operations:

    Similar to performing filtering operations with conditional logic, to filter rows with vertical as B2B and year as 2009, we do it by using:

    >>> %timeit df[(df['vertical'] == 'B2B') & (df['year'] == 2009)]

1.69 ms ± 57 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

    With .query() the same filtering can be performed about 50% faster.

    >>> %timeit df.query("vertical == 'B2B' and year == 2009")

875 µs ± 24.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

    When to use eval() and query()? 

    Two aspects: computation time and memory usage. 

    Memory usage: Every operation which involves NumPy/Pandas DataFrames results into implicit creation of temporary variables. In such cases, if the memory usage of these temporary variables is greater, using eval() and query() is an appropriate choice to reduce the memory usage.

    Computation time: Traditional method of performing NumPy/Pandas operations is faster for smaller arrays! The real benefit of eval()/query() is achieved mainly because of the saved memory, and also because of the cleaner syntax they offer.

    Conclusion

    Pandas is a powerful and fun library for data manipulation/analysis. It comes with easy syntax and fast operations. The blog highlights the most used pandas implementation and optimizations. Best way to master your skills over pandas is to use real datasets, beginning with Kaggle kernels to learning how to use pandas for data analysis. Check out more on real time text classification using Kafka and Scikit-learn and explanatory vs. predictive models in machine learning here.  

  • Monitoring a Docker Container with Elasticsearch, Kibana, and Metricbeat

    Since you are on this page, you have probably already started using Docker to deploy your applications and are enjoying it compared to virtual machines, because of it being lightweight, easy to deploy and its exceptional security management features.

    And, once the applications are deployed, monitoring your containers and tracking their activities in real time is very essential. Imagine a scenario where you are managing one or many virtual machines. Your pre-configured session will be doing everything, including monitoring. If you face any problems during production, then—with a handful of commands such as top, htop, iotop, and with flags like -o, %CPU, and %MEM—you are good to troubleshoot the issue.

    On the other hand, consider a scenario where you have the same nodes spread across 100-200 containers. You will need to see all activity in one place to query for information about what happened. Here, monitoring comes into the picture. We will be discussing more benefits as we move further.

    This blog will cover Docker monitoring with Elasticsearch, Kibana, and Metricbeat. Basically, Elasticsearch is a platform that allows us to have distributed search and analysis of data in real-time along with visualization. We’ll be discussing how all these work interdependently as we move ahead. Like Elasticsearch, Kibana is also open-source software. Kibana is an interface mainly used to visualize the data sent from Elasticsearch. Metricbeat is a lightweight shipper of collected metrics from your system to the desired target (Elasticsearch in this case). 

    What is Docker Monitoring?

    In simple terms, monitoring containers is how we keep track of the above metrics and analyze them to ensure the performance of applications built on microservices and to keep track of issues so that they can be solved more easily. This monitoring is vital for performance improvement and optimization and to find the RCA of various issues.

    There is a lot of software available for monitoring the Docker container, both open-source as well as proprietary, like Prometheus, AppOptics, Metricbeats, Datadog, Sumologic, etc.

    You can choose any of these based on convenience. 

    Why is Docker Monitoring needed?

    1. Monitoring helps early detection and to fix issues to avoid a breakdown during production
      ‍
    2. New feature additions/updates implemented safely as the entire application is monitored
      ‍
    3. Docker monitoring is beneficial for developers, IT pros, and enterprises as well.
    • For developers, Docker monitoring tracks bugs and helps to resolve them quickly along with enhancing security.
    • For IT pros, it helps with flexible integration of existing processes and enterprise systems and satisfies all the requirements.
    • For enterprises, it helps to build the application within a certified container within a secured ecosystem that runs smoothly. 

    Elasticsearch is a platform that allows us to have distributed search and analysis of data in real-time, along with visualization. Elasticsearch is free and open-source software. It goes well with a huge number of technologies, like Metricbeat, Kibana, etc. Let’s move onto the installation of Elasticsearch.

    Installation of Elasticsearch:

    Prerequisite: Elasticsearch is built in Java. So, make sure that your system at least has Java8 to run Elasticsearch.

    For installing Elasticsearch for your OS, please follow the steps at Installing Elasticsearch | Elasticsearch Reference [7.11].

    After installing,  check the status of Elasticsearch by sending an HTTP request on port 9200 on localhost.

    http://localhost:9200/

    This will give you a response as below:

    You can configure Elasticsearch by editing $ES_HOME/config/elasticsearch.yml 

    Learn more about configuring Elasticsearch here.

    Now, we are done with the Elasticsearch setup and are ready to move onto Kibana.

    Kibana:

    Like Elasticsearch, Kibana is also open-source software. Kibana is an interface mainly used to visualize the data from Elasticsearch. Kibana allows you to do anything via query and let’s you generate numerous visuals as per your requirements. Kibana lets you visualize enormous amounts of data in terms of line graphs, gauges, and all other graphs.

    Let’s cover the installation steps of Kibana.

    Installing Kibana

    Prerequisites: 

    • Must have Java1.8+ installed 
    • Elasticsearch v1.4.4+
    • Web browser such as Chrome, Firefox

    For installing Kibana with respect to your OS, please follow the steps at Install Kibana | Kibana Guide [7.11]‍

    Kibana runs on default port number 5601. Just send an HTTP request to port 5601 on localhost with http://localhost:5601/ ‍

    You should land on the Kibana dashboard, and it is now ready to use:

    ‍

    You can configure Kibana by editing $KIBANA_HOME/config. For more about configuring Kibana, visit here.

    Let’s move onto the final part—setting up with Metricbeat.

    Metricbeat

    Metricbeat sends metrics frequently, and we can say it’s a lightweight shipper of collected metrics from your system.

    You can simply install Metricbeat to your system or servers to periodically collect metrics from the OS and the microservices running on services. The collected metrics are shipped to the output you specified, e.g., Elasticsearch, Logstash. 

    Installing Metricbeat

    For installing Metricbeat according to your OS, follow the steps at Install Kibana | Kibana Guide [7.11]

    As soon as we start the Metricbeat service, it sends Docker metrics to the Elasticsearch index, which can be confirmed by curling Elasticsearch indexes with the command:

    curl -XGET 'localhost:9200/_cat/indices?v&pretty'

    How Are They Internally Connected?

    We have now installed all three and they are up and running. As per the period mentioned, docker.yml will hit the Docker API and send the Docker metrics to Elasticsearch. Those metrics are now available in different indexes of Elasticsearch. As mentioned earlier, Kibana queries the data of Elasticsearch and visualizes it in the form of graphs. In this, all three are connected. 

    Please refer to the flow chart for more clarification:

    How to Create Dashboards?

    Now that we are aware of how these three tools work interdependently, let’s create dashboards to monitor our containers and understand those. 

    First of all, open the Dashboards section on Kibana (localhost:5601/) and click the Create dashboard button:

     

    You will be directed to the next page:

    ‍

    Choose the type of visualization you want from all options:

    For example, let’s go with Lens

    (Learn more about Kibana Lens)

    Here, we will be looking for the number of containers vs. timestamps by selecting the timestamp on X-axis and the unique count of docker.container.created on Y-axis.

    ‍

    As soon we have selected both parameters, it will generate a graph as shown in the snapshot, and we will be getting the count of created containers (here Count=1). If you create move containers on your system, when that data metric is sent to Kibana, the graph and the counter will be modified. In this way, you can monitor how many containers are created over time. In similar fashion, depending on your monitoring needs, you can choose a parameter from the left panel showing available fields like: 

    activemq.broker.connections.count

    docker.container.status

    Docker.container.tags

    ‍

    Now, we will show one more example of how to create a bar graph:

    ‍

    As mentioned above, to create a bar graph just choose “vertical bar” from the above snapshot. Here, I’m trying to get a bar graph for the count of documents vs. metricset names, such as network, file system, cpu, etc. So, as shown in the snapshot on the left, choose the Y-axis parameter as count and X-axis parameter as metricset.name as shown in the right side of the snapshot

    After hitting enter, a graph will be generated: 

    Similarly, you can try it out with multiple parameters with different types of graphs to monitor. Now, we will move onto the most important and widely used monitoring tool to track warnings, errors, etc., which is DISCOVER.

    Discover for Monitoring:

    Basically, Discover provides deep insights into data, showing you where you can apply searches and filters as well. With it, you can show which processes are taking more time and show only those. Filter out errors occurring with the message filter with a value of ERROR. Check the health of the container; check for logged-in users. These kinds of queries can be sent and the desired results can be achieved, leading to good monitoring of containers, same as the SQL queries. 

    [More about Discover here.]

    To apply filters, just click on the “filter by type” from the left panel, and you will see all available filtering options. From there, you can select one as per your requirements, and view those on the central panel. 

    ‍

    Similar to filter, you can choose fields to be shown on the dashboard from the left panel with “Selected fields” right below the filters. (Here, we have only selected info for Source.)

    ‍

    Now, if you take a look at the top part of the snapshot, you will find the search bar. This is the most useful part of Discover for monitoring.

    ‍

    In that bar, you just need to put a query, and according to that query, logs will be filtered. For example, I will be putting a query for error messages equal to No memory stats data available.

    When we hit the update button on the right side, only logs containing that error message will be there and highlighted for differentiation, as shown in the snapshot. All other logs will not be shown. In this way, you can track a particular error and ensure that it does not exist after fixing it.

    ‍

    In addition to query, it also provides keyword search. So, if you input a word like warning, error, memory, or user, then it will provide logs for that word, like “memory” in the snapshot:

     

    Similar to Kibana, we also receive logs in the terminal. For example, the following highlighted portion is about the state of your cluster. In the terminal, you can put a simple grep command for required logs. 

    ‍

    With this, you can monitor Docker containers with multiple queries, such as nested queries for the Discover facility. There are many different graphs you can try depending on your requirements to keep your application running smoothly.

    Conclusion

    Monitoring requires a lot of time and effort. What we have seen here is a drop in the ocean. For some next steps, try:

    1. Monitoring network
    2. Aggregating logs from your different applications
    3. Aggregating logs from multiple containers
    4. Alerts setting and monitoring
    5. Nested queries for logs

  • Setting up S3 & CloudFront to Deliver Static Assets Across the Web

    If you have a web application, you probably have static content. Static content might include files like images, videos, and music. One of the simpler approaches to serve your content on the internet is Amazon AWS’s “S3 Bucket.” S3 is very easy to set up and use.

    Problems with only using S3 to serve your resources

    But there are a few limitations of serving content directly using S3. Using S3, you will need:

    • Either keep the bucket public, which is not at all recommended
    • Or, create pre-signed urls to access the private resources. Now, if your application has tons of resources to be loaded, then it will add a lot of latency to pre-sign each and every resource before serving on the UI.

    For these reasons, we will also use AWS’s CloudFront.

    Why use CloudFront with S3?

    Amazon CloudFront (CDN) is designed to work seamlessly with S3 to serve your S3 content in a faster way. Also, using CloudFront to serve s3 content gives you a lot more flexibility and control.

    It has below advantages:

    • Using CloudFront provides authentication, so there’s no need to generate pre-signed urls for each resource.
    • Improved Latency, which results in a better end-user experience.
    • CloudFront provides caching, which can reduce the running costs as content is not always served from S3 when cached.
    • Another case for using CloudFront over S3 is that you can use an SSL certificate to a custom domain in CloudFront.

    Setting up S3 & CloudFront

    Creating an S3 bucket

    1. Navigate to S3 from the AWS console and click on Create Bucket. Enter a unique bucket name and select the AWS Region.

    2. Make sure the Block Public Access settings for this bucket is set to “Block All Public Access,” as it is recommended and we don’t need public access to buckets.

    3. Review other options and create a bucket. Once a bucket is created, you can see it on the S3 dashboard. Open the bucket to view its details, and next, let’s add some assets.

    4. Click on upload and add/drag all the files or folders you want to upload. 

    5. Review the settings and upload. You can see the status on successful upload. Go to bucket details, and, after opening up the uploaded asset, you can see the details of the uploaded asset.

    If you try to copy the object URL and open it in the browser, you will get the access denied error as we have blocked direct public access. 

    We will be using CloudFront to serve the S3 assets in the next step. CloudFront will restrict access to your S3 bucket to CloudFront endpoints rendering your content and application will become more secure and performant.

    Creating a CloudFront

    1. Navigate to CloudFront from AWS console and click on Create Distribution. For the Origin domain, select the bucket from which we want to serve the static assets.

    ‍

    2. Next, we need Use a CloudFront origin access identity (OAI) to access the S3 bucket. This will enable us to access private S3 content via CloudFront. To enable this, under S3 bucket access, select “Yes use OAI.” Select an existing origin access identity or create a new identity.
    You can also choose to update the S3 bucket policy to allow read access to the OAI if it is not already configured previously.

    3. Review all the settings and create distribution. You can see the domain name once it is successfully created.

    ‍

    4. The basic setup is done. If you can try to access the asset we uploaded via the CloudFront domain in your browser, it should serve the asset. You can access assets at {cloudfront domain name}/{s3 asset}
     for e.g.https://d1g71lhh75winl.cloudfront.net/sample.jpeg

    Even though we successfully served the assets via CloudFront. One thing to note is that all the assets are publicly accessible and not secured. In the next section, we will see how you can secure your CloudFront assets.

    Restricting public access

    Previously, while configuring CloudFront, we set Restrict Viewer access to No, which enabled us to access the assets publicly.

    Let’s see how to configure CloudFront to enable signed URLs for assets that should have restricted access. We will be using Trusted key groups, which is the AWS recommended way for restricting access.

    Creating key group

    To create a key pair for a trusted key group, perform the following steps:

    1. Creating the public–private key pair.

    The below commands will generate an RSA key pair and will store the public key & private key in public_key.pem & private_key.pem files respectively.

    openssl genrsa -out private_key.pem 2048
openssl rsa -pubout -in private_key.pem -out public_key.pem

    Note: The above steps use OpenSSL as an example to create a key pair. There are other ways to create an RSA key pair as well.

    2. Uploading the Public Key to CloudFront.

    To upload, in the AWS console, open CloudFront console and navigate to Public Key. Choose Create Public Key. Add name and copy and paste the contents of public_key.pem file under Key. Once done, click Create Public Key.

    3. Adding the public key to a Key Group.

    To do this, navigate to Key Groups. Add name and select the public key we created. Once done, click Create Key Group.

    Adding key group signer to distribution

    1. Navigate to CloudFront and choose the distribution whose files you want to protect with signed URLs or signed cookies.
    2. Navigate to the Behaviors tab. Select the cache behavior, and then choose Edit.
    3. For Restrict Viewer Access (Use Signed URLs or Signed Cookies), choose Yes and choose Trusted Key Groups.
    4. For Trusted Key Groups, select the key group, and then choose Add.
    5. Once done, review and Save Changes.

    Cheers, you have successfully restricted public access to assets. If you try to open any asset urls in the browser, you will see something like this:

    You can either create signed urls or cookies using the private key to access the assets.

    Setting cookies and accessing CloudFront private urls

    You need to create and set cookies on the domain to access your content. Once cookies are set,  they will be sent along with every request by the browser.

    The cookies to be set are:

    • CloudFront-Policy: Your policy statement in JSON format, with white space removed, then base64 encoded.
    • CloudFront-Signature: A hashed, signed using the private key, and base64-encoded version of the JSON policy statement.
    • CloudFront-Key-Pair-Id: The ID for a CloudFront public key, e.g., K4EGX7PEAN4EN. The public key ID tells CloudFront which public key to use to validate the signed URL.

    Please note that the cookie names are case-sensitive. Make sure cookies are http only and secure.

    Set-Cookie: 
CloudFront-Policy=base64 encoded version of the policy statement; 
Domain=optional domain name; 
Path=/optional directory path; 
Secure; 
HttpOnly


Set-Cookie: 
CloudFront-Signature=hashed and signed version of the policy statement; 
Domain=optional domain name; 
Path=/optional directory path; 
Secure; 
HttpOnly

Set-Cookie: 
CloudFront-Key-Pair-Id=public key ID for the CloudFront public key whose corresponding private key you're using to generate the signature; 
Domain=optional domain name; 
Path=/optional directory path; 
Secure; 
HttpOnly

    Cookies can be created in any language you are working on with help of the AWS SDK. For this blog, we will create cookies in python using the botocore module.

    import functools

import rsa
from botocore.signers import CloudFrontSigner

CLOUDFRONT_RESOURCE = # IN format "{protocol}://{domain}/{resource}" for e.g. "https://d1g71lhh75winl.cloudfront.net/*"
CLOUDFRONT_PUBLIC_KEY_ID = # The ID for a CloudFront public key
CLOUDFRONT_PRIVATE_KEY = # contents of the private_key.pem file associated to public key e.g. open('private_key.pem','rb').read()
EXPIRES_AT = # Enter datetime for expiry of cookies e.g.: datetime.datetime.now() + datetime.timedelta(hours=1)

# load the private key
key = rsa.PrivateKey.load_pkcs1(CLOUDFRONT_PRIVATE_KEY)
# create a signer function that can sign message with the private key
rsa_signer = functools.partial(rsa.sign, priv_key=key, hash_method="SHA-1")
# Create a CloudFrontSigner boto3 object
signer = CloudFrontSigner(CLOUDFRONT_PUBLIC_KEY_ID, rsa_signer)

# build the CloudFront Policy
policy = signer.build_policy(CLOUDFRONT_RESOURCE, EXPIRES_AT).encode("utf8")
CLOUDFRONT_POLICY = signer._url_b64encode(policy).decode("utf8")

# create CloudFront Signature
signature = rsa_signer(policy)
CLOUDFRONT_SIGNATURE = signer._url_b64encode(signature).decode("utf8")

# you can set this cookies on response
COOKIES = {
    "CloudFront-Policy": CLOUDFRONT_POLICY,
    "CloudFront-Signature": CLOUDFRONT_SIGNATURE,
    "CloudFront-Key-Pair-Id": CLOUDFRONT_PUBLIC_KEY_ID,
}

    For more details, you can follow AWS official docs.

    Once you set cookies using the above guide, you should be able to access the asset.

    This is how you can effectively use CloudFront along with S3 to securely serve your content.

  • Test Automation in React Native apps using Appium and WebdriverIO

    React Native provides a mobile app development experience without sacrificing user experience or visual performance. And when it comes to mobile app UI testing, Appium is a great way to test indigenous React Native apps out of the box. Creating native apps from the same code and being able to do it using JavaScript has made Appium popular. Apart from this, businesses are attracted by the fact that they can save a lot of money by using this application development framework.

    In this blog, we are going to cover how to add automated tests for React native apps using Appium & WebdriverIO with a Node.js framework. 

    What are React Native Apps

    ‍React Native is an open-source framework for building Android and iOS apps using React and local app capabilities. With React Native, you can use JavaScript to access the APIs on your platform and define the look and behavior of your UI using React components: lots of usable, non-compact code. In the development of Android and iOS apps, “viewing” is the basic building block of a UI: this small rectangular object on the screen can be used to display text, photos, or user input. Even the smallest detail of an app, such as a text line or a button, is a kind of view. Some views may contain other views.

    What is Appium

    ‍Appium is an open-source tool for traditional automation, web, and hybrid apps on iOS, Android, and Windows desktop mobile platforms. Indigenous apps are those written using iOS and Android. Mobile web applications are accessed using a mobile browser (Appium supports Safari for iOS apps and Chrome or the built-in ‘Browser’ for Android apps). Hybrid apps have a wrapper around “web view”—a traditional controller that allows you to interact with web content. Projects like Apache Cordova make it easy to build applications using web technology and integrate it into a traditional wrapper, creating a hybrid application.

    Importantly, Appium is “cross-platform”, allowing you to write tests against multiple platforms (iOS, Android), using the same API. This enables code usage between iOS, Android, and Windows test suites. It runs on iOS and Android applications using the WebDriver protocol.

    Fig:- Appium Architecture

    What is WebDriverIO

    ‍WebdriverIO is a next-gen browser and Node.js automated mobile testing framework. It allows you to customize any application written with modern web frameworks for mobile devices or browsers, such as React, Angular, Polymeror, and Vue.js.

    WebdriverIO is a widely used test automation framework in JavaScript. It has various features like it supports many reports and services, Test Frameworks, and WDIO CLI Test Runners

    The following are examples of supported services:

    • Appium Service
    • Devtools Service
    • Firefox Profile Service
    • Selenium Standalone Service
    • Shared Store Service
    • Static Server Service
    • ChromeDriver Service
    • Report Portal Service
    • Docker Service

    The followings are supported by the test framework:

    • Mocha
    • Jasmine
    • Cucumber 
    Fig:- WebdriverIO Architecture

    Key features of Appium & WebdriverIO

    Appium

    • Does not require application source code or library
    • Provides a strong and active community
    • Has multi-platform support, i.e., it can run the same test cases on multiple platforms
    • Allows the parallel execution of test scripts
    • In Appium, a small change does not require reinstallation of the application
    • Supports various languages like C#, Python, Java, Ruby, PHP, JavaScript with node.js, and many others that have a Selenium client library

    WebdriverIO 

    • Extendable
    • Compatible
    • Feature-rich 
    • Supports modern web and mobile frameworks
    • Runs automation tests both for web applications as well as native mobile apps.
    • Simple and easy syntax
    • Integrates tests to third-party tools such as Appium
    • ‘Wdio setup wizard’ makes the setup simple and easy
    • Integrated test runner

    Installation & Configuration

    $ mkdir Demo_Appium_Project

    • Create a sample Appium Project
    $ npm init
$ package name: (demo_appium_project) demo_appium_test
$ version: (1.0.0) 1.0.0
$ description: demo_appium_practice
$ entry point: (index.js) index.js
$ test command: "./node_modules/.bin/wdio wdio.conf.js"
$ git repository: 
$ keywords: 
$ author: Pushkar
$ license: (ISC) ISC

    This will also create a package.json file for test settings and project dependencies.

    • Install node packages
    $ npm install

    • Install Appium through npm or as a standalone app.
    $ npm install -g appium or npm install --save appium

    $ npm install -g webdriverio or npm install --save-dev webdriverio @wdio/cli
    • Install Chai Assertion library 
    $ npm install -g chai or npm install --save chai

    Make sure you have following versions installed: 

    $ node --version - v.14.17.0
$ npm --version - 7.17.0
$ appium --version - 1.21.0
$ java --version - java 16.0.1
$ allure --version - 2.14.0

    WebdriverIO Configuration 

    The web driver configuration file must be created to apply the configuration during the test Generate command below project:

    $ npx wdio config

    With the following series of questions, install the required dependencies,

    $ Where is your automation backend located? - On my local machine
$ Which framework do you want to use? - mocha	
$ Do you want to use a compiler? No!
$ Where are your test specs located? - ./test/specs/**/*.js
$ Do you want WebdriverIO to autogenerate some test files? - Yes
$ Do you want to use page objects (https://martinfowler.com/bliki/PageObject.html)? - No
$ Which reporter do you want to use? - Allure
$ Do you want to add a service to your test setup? - No
$ What is the base url? - http://localhost

    This is how wdio.conf.js looks:

    exports.config = {
 port: 4724,
 path: '/wd/hub/',
 runner: 'local',
 specs: ['./test/specs/*.js'],
 maxInstances: 1,
 capabilities: [
   {
     platformName: 'Android',
     platformVersion: '11',
     appPackage: 'com.facebook.katana',
     appActivity: 'com.facebook.katana.LoginActivity',
     automationName: 'UiAutomator2'
   }
 ],
 services: [
   [
     'appium',
     {
       args: {
         relaxedSecurity: true
        },
       command: 'appium'
     }
   ]
 ],
 logLevel: 'debug',
 bail: 0,
 baseUrl: 'http://localhost',
 waitforTimeout: 10000,
 connectionRetryTimeout: 90000,
 connectionRetryCount: 3,
 framework: 'mocha',
 reporters: [
   [
     'allure',
     {
       outputDir: 'allure-results',
       disableWebdriverStepsReporting: true,
       disableWebdriverScreenshotsReporting: false
     }
   ]
 ],
 mochaOpts: {
   ui: 'bdd',
   timeout: 60000
 },
 afterTest: function(test, context, { error, result, duration, passed, retries }) {
   if (!passed) {
       browser.takeScreenshot();
   }
 }
}
view raw

    For iOS Automation, just add the following capabilities in wdio.conf.js & the Appium Configuration: 

    {
  "platformName": "IOS",
  "platformVersion": "14.5",
  "app": "/Your_PATH/wdioNativeDemoApp.app",
  "deviceName": "iPhone 12 Pro Max"
}

    Launch the iOS Simulator from Xcode

    Install Appium Doctor for iOS by using following command:

    npm install -g appium-doctor

    Fig:- Appium Doctor Installed

    This is how package.json will look:

    {
 "name": "demo_appium_test",
 "version": "1.0.0",
 "description": "demo_appium_practice",
 "main": "index.js",
 "scripts": {
   "test": "./node_modules/.bin/wdio wdio.conf.js"
 },
 "author": "Pushkar",
 "license": "ISC",
 "dependencies": {
   "@wdio/sync": "^7.7.4",
   "appium": "^1.21.0",
   "chai": "^4.3.4",
   "webdriverio": "^7.7.4"
 },
 "devDependencies": {
   "@wdio/allure-reporter": "^7.7.3",
   "@wdio/appium-service": "^7.7.3",
   "@wdio/cli": "^7.7.4",
   "@wdio/local-runner": "^7.7.4",
   "@wdio/mocha-framework": "^7.7.4",
   "@wdio/selenium-standalone-service": "^7.7.4"
 }
}

    Steps to follow if npm legacy peer deeps problem occurred:

    npm install --save --legacy-peer-deps
npm config set legacy-peer-deps true
npm i --legacy-peer-deps
npm config set legacy-peer-deps true
npm cache clean --force

    This is how the folder structure will look in Appium with the WebDriverIO Framework:

    Fig:- Appium Framework Outline

    Step-by-Step Configuration of Android Emulator using Android Studio

    Fig:- Android Studio Launch

     

    Fig:- Android Studio AVD Manager

     

    Fig:- Create Virtual Device

     

    Fig:- Choose a device Definition

     

    Fig:- Select system image

    ‍

    Fig:- License Agreement

     

    Fig:- Component Installer

     

    Fig:- System Image Download

     

    Fig:- Configuration Verification

    ‍

    Fig:- Virtual Device Listing

    ‍Appium Desktop Configuration

    Fig:- Appium Desktop Launch

    Setup of ANDROID_HOME + ANDROID_SDK_ROOT &  JAVA_HOME

    Follow these steps for setting up ANDROID_HOME: 

    vi ~/.bash_profile
Add following 
export ANDROID_HOME=/Users/pushkar/android-sdk 
export PATH=$PATH:$ANDROID_HOME/platform-tools 
export PATH=$PATH:$ANDROID_HOME/tools 
export PATH=$PATH:$ANDROID_HOME/tools/bin 
export PATH=$PATH:$ANDROID_HOME/emulator
Save ~/.bash_profile 
source ~/.bash_profile 
echo $ANDROID_HOME
/Users/pushkar/Library/Android/sdk

    Follow these steps for setting up ANDROID_SDK_ROOT:

    vi ~/.bash_profile
Add following 
export ANDROID_HOME=/Users/pushkar/Android/sdk
export ANDROID_SDK_ROOT=/Users/pushkar/Android/sdk
export ANDROID_AVD_HOME=/Users/pushkar/.android/avd
Save ~/.bash_profile 
source ~/.bash_profile 
echo $ANDROID_SDK_ROOT
/Users/pushkar/Library/Android/sdk

    Follow these steps for setting up JAVA_HOME:

    java --version
vi ~/.bash_profile
Add following 
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-16.0.1.jdk/Contents/Home.
echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk-16.0.1.jdk/Contents/Home

    Fig:- Environment Variables in Appium

     

    Fig:- Appium Server Starts 

     

    Fig:- Appium Start Inspector Session

    ‍

    Fig:- Inspector Session Configurations

    Note – Make sure you need to install the app from Google Play Store. 

    Fig:- Android Emulator Launch  

     

    Fig: – Android Emulator with Facebook React Native Mobile App

     

    Fig:- Success of Appium with Emulator

     

    Fig:- Locating Elements using Appium Inspector

    How to write E2E React Native Mobile App Tests 

    Fig:- Test Suite Structure of Mocha

    ‍Here is an example of how to write E2E test in Appium:

    Positive Testing Scenario – Validate Login & Nav Bar

    1. Open Facebook React Native App 
    2. Enter valid email and password
    3. Click on Login
    4. Users should be able to login into Facebook 

    Negative Testing Scenario – Invalid Login

    1. Open Facebook React Native App
    2. Enter invalid email and password 
    3. Click on login 
    4. Users should not be able to login after receiving an “Incorrect Password” message popup

    Negative Testing Scenario – Invalid Element

    1. Open Facebook React Native App 
    2. Enter invalid email and  password 
    3. Click on login 
    4. Provide invalid element to capture message

    Make sure test_script should be under test/specs folder 

    var expect = require('chai').expect

beforeEach(() => {
 driver.launchApp()
})

afterEach(() => {
 driver.closeApp()
})

describe('Verify Login Scenarios on Facebook React Native Mobile App', () => {
 it('User should be able to login using valid credentials to Facebook Mobile App', () => {   
   $(`~Username`).waitForDisplayed(20000)
   $(`~Username`).setValue('Valid-Email')
   $(`~Password`).waitForDisplayed(20000)
   $(`~Password`).setValue('Valid-Password')
   $('~Log In').click()
   browser.pause(10000)
 })

 it('User should not be able to login with invalid credentials to Facebook Mobile App', () => {
   $(`~Username`).waitForDisplayed(20000)
   $(`~Username`).setValue('Invalid-Email')
   $(`~Password`).waitForDisplayed(20000)
   $(`~Password`).setValue('Invalid-Password')   
   $('~Log In').click()
   $(
       '//android.widget.TextView[@resource-id="com.facebook.katana:id/(name removed)"]'
     )
     .waitForDisplayed(11000)
   const status = $(
     '//android.widget.TextView[@resource-id="com.facebook.katana:id/(name removed)"]'
   ).getText()
   expect(status).to.equal(
     `You Can't Use This Feature Right Now`     
   )
 })

 it('Test Case should Fail Because of Invalid Element', () => {
   $(`~Username`).waitForDisplayed(20000)
   $(`~Username`).setValue('Invalid-Email')
   $(`~Password`).waitForDisplayed(20000)
   $(`~Password`).setValue('Invalid-Pasword')   
   $('~Log In').click()
   $(
       '//android.widget.TextView[@resource-id="com.facebook.katana:id/(name removed)"'
     )
     .waitForDisplayed(11000)
   const status = $(
     '//android.widget.TextView[@resource-id="com.facebook.katana"'
   ).getText()
   expect(status).to.equal(
     `You Can't Use This Feature Right Now`     
   )
 })

})

    How to Run Mobile Tests Scripts  

    $ npm test 
This will create a Results folder with .xml report

    Reporting

    The following are examples of the supported reporters:

    • Allure Reporter
    • Concise Reporter
    • Dot Reporter
    • JUnit Reporter
    • Spec Reporter
    • Sumologic Reporter
    • Report Portal Reporter
    • Video Reporter
    • HTML Reporter
    • JSON Reporter
    • Mochawesome Reporter
    • Timeline Reporter
    • CucumberJS JSON Reporter

    Here, we are using Allure Reporting. Allure Reporting in WebdriverIO is a plugin to create Allure Test Reports.

    The easiest way is to keep @wdio/allure-reporter as a devDependency in your package.json with

    $ npm install @wdio/allure-reporter --save-dev

    Reporter options can be specified in the wdio.conf.js configuration file 

    reporters: [
   [
     'allure',
     {
       outputDir: 'allure-results',
       disableWebdriverStepsReporting: true,
       disableWebdriverScreenshotsReporting: false
     }
   ]
 ]

    To convert Allure .xml report to .html report, run the following command: 

    $ allure generate && allure open
Allure HTML report should be opened in browser

    This is what Allure Reports look like: 

    Fig:- Allure Report Overview 

     

    Fig:- Allure Categories

     

    Fig:- Allure Suites

     

    Fig: – Allure Graphs

     

    Fig:- Allure Timeline

     

    Fig:- Allure Behaviors

     

    Fig:- Allure Packages

    Limitations with Appium & WebDriverIO

    Appium 

    • Android versions lower than 4.2 are not supported for testing
    • Limited support for hybrid app testing
    • Doesn’t support image comparison.

    WebdriverIO

    • It has a custom implementation
    • It can be used for automating AngularJS apps, but it is not as customized as Protractor.

    Conclusion

    ‍In the QA and developer ecosystem, using Appium to test React native applications is common. Appium makes it easy to record test cases on both Android and iOS platforms while working with React Native. Selenium, a basic web developer, acts as a bridge between Appium and mobile platforms for delivery and testing. Appium is a solid framework for automatic UI testing. This article explains that this framework is capable of conducting test cases quickly and reliably. Most importantly, it can test both Android and iOS apps developed by the React Native framework on the basis of a single code.

    Related Articles –

    References 

    ‍

  • A Leading Telecom Operator increased savings to the tune of $4.2 million annually, by minimizing Custom Churn with R Systems Analytics Gym

    Perform in-depth analysis to help the client discover their process challenges. E.g. Ongoing unresolved technical issues need for competitive pricing and improvements in their overall customer service operations