My blog

Category: Engineering blogs

  • How to Make Your Terminal More Productive with Z-Shell (ZSH)

    When working with servers or command-line-based applications, we spend most of our time on the command line. A good-looking and productive terminal is better in many aspects than a GUI (Graphical User Interface) environment since the command line takes less time for most use cases. Today, we’ll look at some of the features that make a terminal cool and productive.

    You can use the following steps on Ubuntu 20.04. if you are using a different operating system, your commands will likely differ. If you’re using Windows, you can choose between Cygwin, WSL, and Git Bash.

    Prerequisites

    Let’s upgrade the system and install some basic tools needed.

    sudo apt update && sudo apt upgrade
sudo apt install build-essential curl wget git

    Z-Shell (ZSH)

    Zsh is an extended Bourne shell with many improvements, including some features of Bash and other shells.

    Let’s install Z-Shell:

    sudo apt install zsh

    Make it our default shell for our terminal:

    chsh -s $(which zsh)

    Now restart the system and open the terminal again to be welcomed by ZSH. Unlike other shells like Bash, ZSH requires some initial configuration, so it asks for some configuration options the first time we start it and saves them in a file called .zshrc in the home directory (/home/user) where the user is the current system user.

    For now, we’ll skip the manual work and get a head start with the default configuration. Press 2, and ZSH will populate the .zshrc file with some default options. We can change these later.  

    The initial configuration setup can be run again as shown in the below image

    Oh-My-ZSH

    Oh-My-ZSH is a community-driven, open-source framework to manage your ZSH configuration. It comes with many plugins and helpers. It can be installed with one single command as below.

    Installation

    sh -c "$(wget https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"

    It’d take a backup of our existing .zshrc in a file zshrc.pre-oh-my-zsh, so whenever you uninstall it, the backup would be restored automatically.

    Font

    A good terminal needs some good fonts, we’d use Terminess nerd font to make our terminal look awesome, which can be downloaded here. Once downloaded, extract and move them to ~/.local/share/fonts to make them available for the current user or to /usr/share/fonts to be available for all the users.

    tar -xvf Terminess.zip
mv *.ttf ~/.local/share/fonts

    Once the font is installed, it will look like:

    Among all the things Oh-My-ZSH provides, 2 things are community favorites, plugins, and themes.

    Theme

    My go-to ZSH theme is powerlevel10k because it’s flexible, provides everything out of the box, and is easy to install with one command as shown below:

    git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k

    To set this theme in .zshrc:

    Close the terminal and start it again. Powerlevel10k will welcome you with the initial setup, go through the setup with the options you want. You can run this setup again by executing the below command:

    p10k configure

    Tools and plugins we can’t live without

    Plugins can be added to the plugins array in the .zshrc file. For all the plugins you want to use from the below list, add those to the plugins array in the .zshrc file like so:

    ZSH-Syntax-Highlighting

    This enables the highlighting of commands as you type and helps you catch syntax errors before you execute them:

    As you can see, “ls” is in green but “lss” is in red.

    Execute below command to install it:

    git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting

    ZSH Autosuggestions

    This suggests commands as you type based on your history:

    The below command is how you can install it by cloning the git repo:

    git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

    ZSH Completions

    For some extra ZSH completion scripts, execute below command

    git clone https://github.com/zsh-users/zsh-completions ${ZSH_CUSTOM:=~/.oh-my-zsh/custom}/plugins/zsh-completions

    autojump

    It’s a faster way of navigating the file system; it works by maintaining a database of directories you visit the most. More details can be found here.

    sudo apt install autojump

    You can also use the plugin Z as an alternative if you’re not able to install autojump or for any other reason.

    Internal Plugins

    Some plugins come installed with oh-my-zsh, and they can be included directly in .zshrc file without any installation.

    copyfile

    It copies the content of a file to the clipboard.

    copyfile test.txt

    copypath

    It copies the absolute path of the current directory to the clipboard.

    copybuffer

    This plugin copies the command that is currently typed in the command prompt to the clipboard. It works with the keyboard shortcut CTRL + o.

    sudo

    Sometimes, we forget to prefix a command with sudo, but that can be done in just a second with this plugin. When you hit the ESC key twice, it will prefix the command you’ve typed in the terminal with sudo.

    web-search

    This adds some aliases for searching with Google, Wikipedia, etc. For example, if you want to web-search with Google, you can execute the below command:

    google oh my zsh

    Doing so will open this search in Google:

    More details can be found here.

    Remember, you’d have to add each of these plugins in the .zshrc file as well. So, in the end, this is how the plugins array in .zshrc file should look like:

    plugins=(
        zsh-autosuggestions
        zsh-syntax-highlighting
        zsh-completions
        autojump
        copyfile
        copydir
        copybuffer
        history
        dirhistory
        sudo
        web-search
        git
)

    You can add more plugins, like docker, heroku, kubectl, npm, jsontools, etc., if you’re a developer. There are plugins for system admins as well or for anything else you need. You can explore them here.

    Enhancd

    Enhancd is the next-gen method to navigate file system with cli. It works with a fuzzy finder, we’ll install it fzf for this purpose.

    sudo apt install fzf

    Enhancd can be installed with the zplug plugin manager for ZSH, so first we’ll install zplug with the below command:

    $ curl -sL --proto-redir -all,https https://raw.githubusercontent.com/zplug/installer/master/installer.zsh | zsh

    Append the following to .zshrc:

    source ~/.zplug/init.zsh
zplug load

    Now close your terminal, open it again, and use zplug to install enhanced

    zplug "b4b4r07/enhancd", use:init.sh

    Aliases

    As a developer, I need to execute git commands many times a day, typing each command every time is too cumbersome, so we can use aliases for them. Aliases need to be added .zshrc, and here’s how we can add them.

    alias gs='git status'
alias ga='git add .'
alias gf='git fetch'
alias gr='git rebase'
alias gp='git push'
alias gd='git diff'
alias gc='git commit'
alias gh='git checkout'
alias gst='git stash'
alias gl='git log --oneline --graph'

    You can add these anywhere in the .zshrc file.

    Colorls

    Another tool that makes you say wow is Colorls. This tool colorizes the output of the ls command. This is how it looks once you install it:

    It works with ruby, below is how you can install both ruby and colors:

    sudo apt install ruby ruby-dev ruby-colorize
sudo gem install colorls

    Now, restart your terminal and execute the command colors in your terminal to see the magic!

    Bonus – We can add some aliases as well if we want the same output of Colorls when we execute the command ls. Note that we’re adding another alias for ls to make it available as well.

    alias cl='ls'
alias ls='colorls'
alias la='colorls -a'
alias ll='colorls -l'
alias lla='colorls -la'

    These are the tools and plugins I can’t live without now, Let me know if I’ve missed anything.

    Automation

    Do you wanna repeat this process again, if let’s say, you’ve bought a new laptop and want the same setup?

    You can automate all of this if your answer is no, and that’s why I’ve created Project Automator. This project does a lot more than just setting up a terminal: it works with Arch Linux as of now but you can take the parts you need and make it work with almost any *nix system you like.

    Explaining how it works is beyond the scope of this article, so I’ll have to leave you guys here to explore it on your own.

    Conclusion

    We need to perform many tasks on our systems, and using a GUI(Graphical User Interface) tool for a task can consume a lot of your time, especially if you repeat the same task on a daily basis like converting a media stream, setting up tools on a system, etc.

    Using a command-line tool can save you a lot of time and you can automate repetitive tasks with scripting. It can be a great tool for your arsenal.

  • Deploy Serverless, Event-driven Python Applications Using Zappa

    Introduction

    Zappa is a  very powerful open source python project which lets you build, deploy and update your WSGI app hosted on AWS Lambda + API Gateway easily.This blog is a detailed step-by-step focusing on challenges faced while deploying Django application on AWS Lambda using Zappa as a deployment tool.

    Building Your Application

    If you do not have a Django application already you can build one by cloning this GitHub repository.

    $ git clone https://github.com/velotiotech/django-zappa-sample.git    

    Cloning into 'django-zappa-sample'...
remote: Counting objects: 18, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 18 (delta 1), reused 15 (delta 1), pack-reused 0
Unpacking objects: 100% (18/18), done.
Checking connectivity... done.

    Once you have cloned the repository you will need a virtual environment which provides an isolated Python environment for your application. I prefer virtualenvwrapper to create one.

    Command :

    $ mkvirtualenv django_zappa_sample 

    Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/velotio/Envs/django_zappa_sample/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/velotio/Envs/django_zappa_sample/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/velotio/Envs/django_zappa_sample/bin/preactivate
virtualenvwrapper.user_scripts creating /home/velotio/Envs/django_zappa_sample/bin/postactivate
virtualenvwrapper.user_scripts creating /home/velotio/Envs/django_zappa_sample/bin/get_env_details

    Install dependencies from requirements.txt.

    $ pip install -r requirements.txt

    Collecting Django==1.11.11 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/d5/bf/2cd5eb314aa2b89855c01259c94dc48dbd9be6c269370c1f7ae4979e6e2f/Django-1.11.11-py2.py3-none-any.whl (6.9MB)
    100% |████████████████████████████████| 7.0MB 772kB/s 
Collecting zappa==0.45.1 (from -r requirements.txt (line 2))
Collecting pytz (from Django==1.11.11->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl (510kB)
    100% |████████████████████████████████| 512kB 857kB/s 
Collecting future==0.16.0 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting toml>=0.9.3 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting docutils>=0.12 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/50/09/c53398e0005b11f7ffb27b7aa720c617aba53be4fb4f4f3f06b9b5c60f28/docutils-0.14-py2-none-any.whl
Collecting PyYAML==3.12 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting futures==3.1.1 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/a6/1c/72a18c8c7502ee1b38a604a5c5243aa8c2a64f4bba4e6631b1b8972235dd/futures-3.1.1-py2-none-any.whl
Requirement already satisfied: wheel>=0.30.0 in /home/velotio/Envs/django_zappa_sample/lib/python2.7/site-packages (from zappa==0.45.1->-r requirements.txt (line 2)) (0.31.1)
Collecting base58==0.2.4 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting durationpy==0.5 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting kappa==0.6.0 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/ed/cf/a8aa5964557c8a4828da23d210f8827f9ff190318838b382a4fb6f118f5d/kappa-0.6.0-py2-none-any.whl
Collecting Werkzeug==0.12 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/ae/c3/f59f6ade89c811143272161aae8a7898735e7439b9e182d03d141de4804f/Werkzeug-0.12-py2.py3-none-any.whl
Collecting boto3>=1.4.7 (from zappa==0.45.1->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/cd/a3/4d1caf76d8f5aac8ab1ffb4924ecf0a43df1572f6f9a13465a482f94e61c/boto3-1.7.24-py2.py3-none-any.whl (128kB)
    100% |████████████████████████████████| 133kB 1.1MB/s 
Collecting six>=1.11.0 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting tqdm==4.19.1 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/c0/d3/7f930cbfcafae3836be39dd3ed9b77e5bb177bdcf587a80b6cd1c7b85e74/tqdm-4.19.1-py2.py3-none-any.whl
Collecting argcomplete==1.9.2 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/0f/ee/625763d848016115695942dba31a9937679a25622b6f529a2607d51bfbaa/argcomplete-1.9.2-py2.py3-none-any.whl
Collecting hjson==3.0.1 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting troposphere>=1.9.0 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting python-dateutil==2.6.1 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/4b/0d/7ed381ab4fe80b8ebf34411d14f253e1cf3e56e2820ffa1d8844b23859a2/python_dateutil-2.6.1-py2.py3-none-any.whl
Collecting botocore>=1.7.19 (from zappa==0.45.1->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/65/98/12aa979ca3215d69111026405a9812d7bb0c9ae49e2800b00d3bd794705b/botocore-1.10.24-py2.py3-none-any.whl (4.2MB)
    100% |████████████████████████████████| 4.2MB 768kB/s 
Collecting requests>=2.10.0 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl
Collecting jmespath==0.9.3 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/b7/31/05c8d001f7f87f0f07289a5fc0fc3832e9a57f2dbd4d3b0fee70e0d51365/jmespath-0.9.3-py2.py3-none-any.whl
Collecting wsgi-request-logger==0.4.6 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting lambda-packages==0.19.0 (from zappa==0.45.1->-r requirements.txt (line 2))
Collecting python-slugify==1.2.4 (from zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/9f/77/ab7134b731d0e831cf82861c1ab0bb318e80c41155fa9da18958f9d96057/python_slugify-1.2.4-py2.py3-none-any.whl
Collecting placebo>=0.8.1 (from kappa==0.6.0->zappa==0.45.1->-r requirements.txt (line 2))
Collecting click>=5.1 (from kappa==0.6.0->zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl
Collecting s3transfer<0.2.0,>=0.1.10 (from boto3>=1.4.7->zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl
Collecting cfn-flip>=0.2.5 (from troposphere>=1.9.0->zappa==0.45.1->-r requirements.txt (line 2))
Collecting certifi>=2017.4.17 (from requests>=2.10.0->zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.10.0->zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests>=2.10.0->zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests>=2.10.0->zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl
Collecting Unidecode>=0.04.16 (from python-slugify==1.2.4->zappa==0.45.1->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/59/ef/67085e30e8bbcdd76e2f0a4ad8151c13a2c5bce77c85f8cad6e1f16fb141/Unidecode-1.0.22-py2.py3-none-any.whl
Installing collected packages: pytz, Django, future, toml, docutils, PyYAML, futures, base58, durationpy, jmespath, six, python-dateutil, botocore, s3transfer, boto3, placebo, click, kappa, Werkzeug, tqdm, argcomplete, hjson, cfn-flip, troposphere, certifi, chardet, idna, urllib3, requests, wsgi-request-logger, lambda-packages, Unidecode, python-slugify, zappa
Successfully installed Django-1.11.11 PyYAML-3.12 Unidecode-1.0.22 Werkzeug-0.12 argcomplete-1.9.2 base58-0.2.4 boto3-1.7.24 botocore-1.10.24 certifi-2018.4.16 cfn-flip-1.0.3 chardet-3.0.4 click-6.7 docutils-0.14 durationpy-0.5 future-0.16.0 futures-3.1.1 hjson-3.0.1 idna-2.6 jmespath-0.9.3 kappa-0.6.0 lambda-packages-0.19.0 placebo-0.8.1 python-dateutil-2.6.1 python-slugify-1.2.4 pytz-2018.4 requests-2.18.4 s3transfer-0.1.13 six-1.11.0 toml-0.9.4 tqdm-4.19.1 troposphere-2.2.1 urllib3-1.22 wsgi-request-logger-0.4.6 zappa-0.45.1
@velotiotech

    Now if you run the server directly it will log a warning as the database is not set up yet.

    $ python manage.py runserver  

    Performing system checks...

System check identified no issues (0 silenced).

You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

May 20, 2018 - 14:47:32
Django version 1.11.11, using settings 'django_zappa_sample.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

    Also trying to access admin page (http://localhost:8000/admin/) will throw an “OperationalError” exception with below log at server end.

    Internal Server Error: /admin/
Traceback (most recent call last):
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 242, in wrapper
    return self.admin_view(view, cacheable)(*args, **kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 213, in inner
    if not self.has_permission(request):
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 187, in has_permission
    return request.user.is_active and request.user.is_staff
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/utils/functional.py", line 238, in inner
    self._setup()
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/utils/functional.py", line 386, in _setup
    self._wrapped = self._setupfunc()
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 24, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 12, in get_user
    request._cached_user = auth.get_user(request)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 211, in get_user
    user_id = _get_user_session_key(request)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 61, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 57, in __getitem__
    return self._session[key]
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py", line 207, in _get_session
    self._session_cache = self.load()
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py", line 35, in load
    expire_date__gt=timezone.now()
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get
    num = len(clone)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__
    self._fetch_all()
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
OperationalError: no such table: django_session
[20/May/2018 14:59:23] "GET /admin/ HTTP/1.1" 500 153553
Not Found: /favicon.ico

    In order to fix this you need to run the migration into your database so that essential tables like auth_user, sessions, etc are created before any request is made to the server.

    $ python manage.py migrate 

    Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying sessions.0001_initial... OK

    NOTE: Use DATABASES from project settings file to configure your database that you would want your Django application to use once hosted on AWS Lambda. By default, its configured to create a local SQLite database file as backend.

    You can run the server again and it should now load the admin panel of your website.

    Do verify if you have the zappa python package into your virtual environment before moving forward.

    Configuring Zappa Settings

    Deploying with Zappa is simple as it only needs a configuration file to run and rest will be managed by Zappa. To create this configuration file run from your project root directory –

    $ zappa init 

    ███████╗ █████╗ ██████╗ ██████╗  █████╗
╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗
  ███╔╝ ███████║██████╔╝██████╔╝███████║
 ███╔╝  ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║
███████╗██║  ██║██║     ██║     ██║  ██║
╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝     ╚═╝  ╚═╝

Welcome to Zappa!

Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.
This `init` command will help you create and configure your new Zappa deployment.
Let's get started!

Your Zappa configuration can support multiple production stages, like 'dev', 'staging', and 'production'.
What do you want to call this environment (default 'dev'): 

AWS Lambda and API Gateway are only available in certain regions. Let's check to make sure you have a profile set up in one that will work.
We found the following profiles: default, and hdx. Which would you like us to use? (default 'default'): 

Your Zappa deployments will need to be uploaded to a private S3 bucket.
If you don't have a bucket yet, we'll create one for you too.
What do you want call your bucket? (default 'zappa-108wqhyn4'): django-zappa-sample-bucket

It looks like this is a Django application!
What is the module path to your projects's Django settings?
We discovered: django_zappa_sample.settings
Where are your project's settings? (default 'django_zappa_sample.settings'): 

You can optionally deploy to all available regions in order to provide fast global service.
If you are using Zappa for the first time, you probably don't want to do this!
Would you like to deploy this application globally? (default 'n') [y/n/(p)rimary]: n

Okay, here's your zappa_settings.json:

{
    "dev": {
        "aws_region": "us-east-1", 
        "django_settings": "django_zappa_sample.settings", 
        "profile_name": "default", 
        "project_name": "django-zappa-sa", 
        "runtime": "python2.7", 
        "s3_bucket": "django-zappa-sample-bucket"
    }
}

Does this look okay? (default 'y') [y/n]: y

Done! Now you can deploy your Zappa application by executing:

	$ zappa deploy dev

After that, you can update your application code with:

	$ zappa update dev

To learn more, check out our project page on GitHub here: https://github.com/Miserlou/Zappa
and stop by our Slack channel here: https://slack.zappa.io

Enjoy!,
 ~ Team Zappa!

    You can verify zappa_settings.json generated at your project root directory.

    TIP: The virtual environment name should not be the same as the Zappa project name, as this may cause errors.

    Additionally, you could specify other settings in  zappa_settings.json file as per requirement using Advanced Settings.

    Now, you’re ready to deploy!

    IAM Permissions

    In order to deploy the Django Application to Lambda/Gateway, setup an IAM role (eg. ZappaLambdaExecutionRole) with the following permissions:

    {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:AttachRolePolicy",
"iam:CreateRole",
"iam:GetRole",
"iam:PutRolePolicy"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"arn:aws:iam:::role/*-ZappaLambdaExecutionRole"
]
},
{
"Effect": "Allow",
"Action": [
"apigateway:DELETE",
"apigateway:GET",
"apigateway:PATCH",
"apigateway:POST",
"apigateway:PUT",
"events:DeleteRule",
"events:DescribeRule",
"events:ListRules",
"events:ListTargetsByRule",
"events:ListRuleNamesByTarget",
"events:PutRule",
"events:PutTargets",
"events:RemoveTargets",
"lambda:AddPermission",
"lambda:CreateFunction",
"lambda:DeleteFunction",
"lambda:GetFunction",
"lambda:GetPolicy",
"lambda:ListVersionsByFunction",
"lambda:RemovePermission",
"lambda:UpdateFunctionCode",
"lambda:UpdateFunctionConfiguration",
"cloudformation:CreateStack",
"cloudformation:DeleteStack",
"cloudformation:DescribeStackResource",
"cloudformation:DescribeStacks",
"cloudformation:ListStackResources",
"cloudformation:UpdateStack",
"logs:DescribeLogStreams",
"logs:FilterLogEvents",
"route53:ListHostedZones",
"route53:ChangeResourceRecordSets",
"route53:GetHostedZone",
"s3:CreateBucket",
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::"
]
},
{
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObject",
"s3:CreateMultipartUpload",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::/*"
]
}
]
}

    Deploying Django Application

    Before deploying the application, ensure that the IAM role is set in the config JSON as follows:

    {
"dev": {
...
"manage_roles": false, // Disable Zappa client managing roles.
"role_name": "MyLambdaRole", // Name of your Zappa execution role. Optional, default: --ZappaExecutionRole.
"role_arn": "arn:aws:iam::12345:role/app-ZappaLambdaExecutionRole", // ARN of your Zappa execution role. Optional.
...
},
...
}

    Once your settings are configured, you can package and deploy your application to a stage called “dev” with a single command:

    $ zappa deploy dev

    Calling deploy for stage dev..
Downloading and installing dependencies..
Packaging project as zip.
Uploading django-zappa-sa-dev-1526831069.zip (10.9MiB)..
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [01:02<00:00, 75.3KB/s]
Scheduling..
Scheduled django-zappa-sa-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading django-zappa-sa-dev-template-1526831157.json (1.6KiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.60K/1.60K [00:02<00:00, 792B/s]
Waiting for stack django-zappa-sa-dev to create (this can take a bit)..
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:11<00:00,  2.92s/res]
Deploying API Gateway..
Deployment complete!: https://akg59b222b.execute-api.us-east-1.amazonaws.com/dev

    You should see that your Zappa deployment completed successfully with URL to API gateway created for your application.

    Troubleshooting

    1. If you are seeing the following error while deployment, it’s probably because you do not have sufficient privileges to run deployment on AWS Lambda. Ensure your IAM role has all the permissions as described above or set “manage_roles” to true so that Zappa can create and manage the IAM role for you.

    Calling deploy for stage dev..
Creating django-zappa-sa-dev-ZappaLambdaExecutionRole IAM Role..
Error: Failed to manage IAM roles!
You may lack the necessary AWS permissions to automatically manage a Zappa execution role.
To fix this, see here: https://github.com/Miserlou/Zappa#using-custom-aws-iam-roles-and-policies

    2. The below error will be caused as you have not listed “events.amazonaws.com” as Trusted Entity for your IAM Role. You can add the same or set “keep_warm” parameter to false in your Zappa settings file. Your Zappa deployment was partially deployed as it got terminated abnormally.

    Downloading and installing dependencies..
100%|████████████████████████████████████████████| 44/44 [00:05<00:00, 7.92pkg/s]
Packaging project as zip..
Uploading django-zappa-sample-dev-1482817370.zip (8.8MiB)..
100%|█████████████████████████████████████████| 9.22M/9.22M [00:17<00:00, 527KB/s]
Scheduling...
Oh no! An error occurred! :(

==============

Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 2610, in handle
    sys.exit(cli.handle())
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 505, in handle
    self.dispatch_command(self.command, stage)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 539, in dispatch_command
    self.deploy(self.vargs['zip'])
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 800, in deploy
    self.zappa.add_binary_support(api_id=api_id, cors=self.cors)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/core.py", line 1490, in add_binary_support
    restApiId=api_id
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (ValidationError) when calling the PutRole operation: Provided role 'arn:aws:iam:484375727565:role/lambda_basic_execution' cannot be assumed by principal
'events.amazonaws.com'.

==============

Need help? Found a bug? Let us know! :D
File bug reports on GitHub here: https://github.com/Miserlou/Zappa
And join our Slack channel here: https://slack.zappa.io
Love!,
~ Team Zappa!

    3. Adding the parameter and running zappa update will cause above error. As you can see it says “Stack django-zappa-sa-dev does not exists” as the previous deployment was unsuccessful. To fix this, delete the Lambda function from console and rerun the deployment.

    Downloading and installing dependencies..
100%|████████████████████████████████████████████| 44/44 [00:05<00:00, 7.92pkg/s]
Packaging project as zip..
Uploading django-zappa-sample-dev-1482817370.zip (8.8MiB)..
100%|█████████████████████████████████████████| 9.22M/9.22M [00:17<00:00, 527KB/s]
Updating Lambda function code..
Updating Lambda function configuration..
Uploading djangoo-zapppa-sample-dev-template-1482817403.json (1.5KiB)..
100%|████████████████████████████████████████| 1.56K/1.56K [00:00<00:00, 6.56KB/s]
CloudFormation stack missing, re-deploy to enable updates
ERROR:Could not get API ID.
Traceback (most recent call last):
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 2610, in handle
    sys.exit(cli.handle())
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 505, in handle
    self.dispatch_command(self.command, stage)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 539, in dispatch_command
    self.deploy(self.vargs['zip'])
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 800, in deploy
    self.zappa.add_binary_support(api_id=api_id, cors=self.cors)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/core.py", line 1490, in add_binary_support
    restApiId=api_id
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (ValidationError) when calling the DescribeStackResource operation: Stack 'django-zappa-sa-dev' does not exist
Deploying API Gateway..
Oh no! An error occurred! :(

==============

Traceback (most recent call last):
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 1847, in handle
sys.exit(cli.handle())
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 345, in handle
self.dispatch_command(self.command, environment)
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 379, in dispatch_command
self.update()
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 605, in update
endpoint_url = self.deploy_api_gateway(api_id)
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 1816, in deploy_api_gateway
cloudwatch_metrics_enabled=self.zappa_settings[self.api_stage].get('cloudwatch_metrics_enabled', False),
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/zappa.py", line 1014, in deploy_api_gateway
variables=variables or {}
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 513, in _make_api_call
api_params, operation_model, context=request_context)
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 566, in _convert_to_request_dict
api_params, operation_model)
File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/validate.py", line 270, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
ParamValidationError: Parameter validation failed:
Invalid type for parameter restApiId, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

==============

Need help? Found a bug? Let us know! :D
File bug reports on GitHub here: https://github.com/Miserlou/Zappa
And join our Slack channel here: https://slack.zappa.io
Love!,
~ Team Zappa!

    4.  If you run into any distribution error, please try down-grading your pip version to 9.0.1.

    $ pip install pip==9.0.1   

    Calling deploy for stage dev..
Downloading and installing dependencies..
Oh no! An error occurred! :(

==============

Traceback (most recent call last):
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 2610, in handle
    sys.exit(cli.handle())
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 505, in handle
    self.dispatch_command(self.command, stage)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 539, in dispatch_command
    self.deploy(self.vargs['zip'])
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 709, in deploy
    self.create_package()
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 2171, in create_package
    disable_progress=self.disable_progress
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/core.py", line 595, in create_lambda_zip
    installed_packages = self.get_installed_packages(site_packages, site_packages_64)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/core.py", line 751, in get_installed_packages
    pip.get_installed_distributions()
AttributeError: 'module' object has no attribute 'get_installed_distributions'

==============

Need help? Found a bug? Let us know! :D
File bug reports on GitHub here: https://github.com/Miserlou/Zappa
And join our Slack channel here: https://slack.zappa.io
Love!,
 ~ Team Zappa!

    or,

    If you run into NotFoundException(Invalid REST API Identifier issue) please try undeploying the Zappa stage and retry again.

    Calling deploy for stage dev..
Downloading and installing dependencies..
Packaging project as zip.
Uploading django-zappa-sa-dev-1526830532.zip (10.9MiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:42<00:00, 331KB/s]
Scheduling..
Scheduled django-zappa-sa-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading django-zappa-sa-dev-template-1526830690.json (1.6KiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.60K/1.60K [00:01<00:00, 801B/s]
Oh no! An error occurred! :(

==============

Traceback (most recent call last):
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 2610, in handle
    sys.exit(cli.handle())
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 505, in handle
    self.dispatch_command(self.command, stage)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 539, in dispatch_command
    self.deploy(self.vargs['zip'])
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/cli.py", line 800, in deploy
    self.zappa.add_binary_support(api_id=api_id, cors=self.cors)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/zappa/core.py", line 1490, in add_binary_support
    restApiId=api_id
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/velotio/Envs/django_zappa_sample/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
NotFoundException: An error occurred (NotFoundException) when calling the GetRestApi operation: Invalid REST API identifier specified 484375727565:akg59b222b

==============

Need help? Found a bug? Let us know! :D
File bug reports on GitHub here: https://github.com/Miserlou/Zappa
And join our Slack channel here: https://slack.zappa.io
Love!,
 ~ Team Zappa!

    TIP: To understand how your application works on serverless environment please visit this link.

    Post Deployment Setup

    Migrate database

    At this point, you should have an empty database for your Django application to fill up with a schema.

    $ zappa manage.py migrate dev

    Once you run above command the database migrations will be applied on the database as specified in your Django settings.

    Creating Superuser of Django Application

    You also might need to create a new superuser on the database. You could use the following command on your project directory.

    $ zappa invoke --raw dev "from django.contrib.auth.models import User; User.objects.create_superuser('username', 'username@yourdomain.com', 'password')"

    Alternatively,

    $ python manage createsuperuser

    Note that your application must be connected to the same database as this is run as standard Django administration command (not a Zappa command).

    Managing static files

    Your Django application will be having a dependency on static files, Django admin panel uses a combination of JS, CSS and image files.

    NOTE: Zappa is for running your application code, not for serving static web assets. If you plan on serving custom static assets in your web application (CSS/JavaScript/images/etc.), you’ll likely want to use a combination of AWS S3 and AWS CloudFront.

    You will need to add following packages to your virtual environment required for management of files to and from S3 django-storages and boto.

    $ pip install django-storages boto
Add Django-Storage to your INSTALLED_APPS in settings.py
INSTALLED_APPS = (
...,
storages',
)

Configure Django-storage in settings.py as

AWS_STORAGE_BUCKET_NAME = 'django-zappa-sample-bucket'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = "https://%s/" % AWS_S3_CUSTOM_DOMAIN
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

    Once you have setup the Django application to serve your static files from AWS S3, run following command to upload the static file from your project to S3.

    $ python manage.py collectstatic --noinput

    or

    $ zappa update dev
$ zappa manage dev "collectstatic --noinput"

    Check that at least 61 static files are moved to S3 bucket. Admin panel is built over  61 static files.

    NOTE: STATICFILES_DIR must be configured properly to collect your files from the appropriate location.

    Tip: You need to render static files in your templates by loading static path and using the same.  Example, {% static %}

    Setting Up API Gateway

    To connect to your Django application you also need to ensure you have API gateway setup for your AWS Lambda Function.  You need to have GET methods set up for all the URL resources used in your Django application. Alternatively, you can setup a proxy method to allow all subresources to be processed through one API method.

    Go to AWS Lambda function console and add API Gateway from ‘Add triggers’.

    1. Configure API, Deployment Stage, and Security for API Gateway. Click Save once it is done.

    2. Go to API Gateway console and,

    a. Recreate ANY method for / resource.

    i. Check `Use Lambda Proxy integration`

    ii. Set `Lambda Region` and `Lambda Function` and `Save` it.

    a. Recreate ANY method for /{proxy+} resource.

    i. Select `Lambda Function Proxy`

    ii. Set`Lambda Region` and `Lambda Function` and `Save` it.

    3. Click on Action and select Deploy API. Set Deployment Stage and click Deploy

    4. Ensure that GET and POST method for / and Proxy are set as Override for this method

    Setting Up Custom SSL Endpoint

    Optionally, you could also set up your own custom defined SSL endpoint with Zappa and install your certificate with your domain by running certify with Zappa. 

    $ zappa certify dev

...
"certificate_arn": "arn:aws:acm:us-east-1:xxxxxxxxxxxx:certificate/xxxxxxxxxxxx-xxxxxx-xxxx-xxxx-xxxxxxxxxxxxxx",
"domain": "django-zappa-sample.com"

    Now you are ready to launch your Django Application hosted on AWS Lambda.

    Additional Notes:

    •  Once deployed, you must run “zappa update <stage-name>” for updating your already hosted AWS Lambda function.</stage-name>
    • You can check server logs for investigation by running “zappa tail” command.
    • To un-deploy your application, simply run: `zappa undeploy <stage-name>`</stage-name>

    You’ve seen how to deploy Django application on AWS Lambda using Zappa. If you are creating your Django application for first time you might also want to read Edgar Roman’s Django Zappa Guide.

    Start building your Django application and let us know in the comments if you need any help during your application deployment over AWS Lambda.

  • Implementing Federated GraphQL Microservices using Apollo Federation

    Introduction

    GraphQL has revolutionized how a client queries a server. With the thin layer of GraphQL middleware, the client has the ability to query the data more comprehensively than what’s provided by the usual REST APIs.

    One of the key principles of GraphQL involves having a single data graph of the implementing services that will allow the client to have a unified interface to access more data and services through a single query. Having said that, it can be challenging to follow this principle for an enterprise-level application on a single, monolith GraphQL server.

    The Need for Federated Services

    James Baxley III, the Engineering Manager at Apollo, in his talk here, puts forward the rationale behind choosing an independently managed federated set of services very well.

    To summarize his point, let’s consider a very complex enterprise product. This product would essentially have multiple teams responsible for maintaining different modules of the product. Now, if we’re considering implementing a GraphQL layer at the backend, it would only make sense to follow the one graph principle of GraphQL: this says that to maximize the value of GraphQL, we should have a single unified data graph that’s operating at the data layer of this product. With that, it will be easier for a client to query a single graph and get all the data without having to query different graphs for different data portions.

    However, it would be challenging to have all of the huge enterprise data graphs’ layer logic residing on a single codebase. In addition, we want teams to be able to independently implement, maintain, and ship different schemas of the data graph on their own release cycles.

    Though there is only one graph, the implementation of that graph should be federated across multiple teams.

    Now, let’s consider a massive enterprise e-commerce platform as an example. The different schemas of the e-commerce platform look something like:

    Fig:- E-commerce platform set of schemas

    Considering the above example, it would be a chaotic task to maintain the graph implementation logic of all these schemas on a single code base. Another overhead that this would bring is having to scale a huge monolith that’s implementing all these services. 

    Thus, one solution is a federation of services for a single distributed data graph. Each service can be implemented independently by individual teams while maintaining their own release cycles and having their own iterations of their services. Also, a federated set of services would still follow the Onegraph principle of GraphQL, which will allow the client to query a single endpoint for fetching any part of the data graph.

    To further demonstrate the example above, let’s say the client asks for the top-five products, their reviews, and the vendor selling them. In a usual monolith GraphQL server, this query would involve writing a resolver that’s a mesh of the data sources of these individual schemas. It would be a task for teams to collaborate and come up with their individual implementations. Let’s consider a federated approach with separate services implementing products, reviews, and vendors. Each service is responsible for resolving only the part of the data graph that includes the schema and data source. This makes it extremely streamlined to allow different teams managing different schemas to collaborate easily.

    Another advantage would be handling the scaling of individual services rather than maintaining a compute-heavy monolith for a huge data graph. For example, the products service is used the most on the platform, and the vendors service is scarcely used. In case of a monolith approach, the scaling would’ve had to take place on the overall server. This is eliminated with federated services where we can independently maintain and scale individual services like the products service.

    Federated Implementation of GraphQL Services

    A monolith GraphQL server that implements a lot of services for different schemas can be challenging to scale. Instead of implementing the complete data graph on a single codebase, the responsibilities of different parts of the data graph can be split across multiple composable services. Each one will contain the implementation of only the part of the data graph it is responsible for. Apollo Federation allows this division of services and follows a declarative programming model to allow splitting of concerns.

    Architecture Overview

    This article will not cover the basics of GraphQL, such as writing resolvers and schemas. If you’re not acquainted with the basics of GraphQL and setting up a basic GraphQL server using Apollo, I would highly recommend reading about it here. Then, you can come back here to understand the implementation of federated services using Apollo Federation.

    Apollo Federation has two principal parts to it:

    • A collection of services that distinctly define separate GraphQL schemas
    • A gateway that builds the federated data graph and acts as a forefront to distinctly implement queries for different services
    Fig:- Apollo Federation Architecture

    Separation of Concerns

    The usual way of going about implementing federated services would be by splitting an existing monolith based on the existing schemas defined. Although this way seems like a clear approach, it will quickly cause problems when multiple Schemas are involved.

    To illustrate, this is a typical way to split services from a monolith based on the existing defined Schemas:

     

    In the example above, although the tweets field belongs to the User schema, it wouldn’t make sense to populate this field in the User service. The tweets field of a User should be declared and resolved in the Tweet service itself. Similarly, it wouldn’t be right to resolve the creator field inside the Tweet service.

    The reason behind this approach is the separation of concerns. The User service might not even have access to the Tweet datastore to be able to resolve the tweets field of a user. On the other hand, the Tweet service might not have access to the User datastore to resolve the creator field of the Tweet schema.

    Considering the above schemas, each service is responsible for resolving the respective field of each Schema it is responsible for.

    Implementation

    To illustrate an Apollo Federation, we’ll be considering a Nodejs server built with Typescript. The packages used are provided by the Apollo libraries.

    npm i --save apollo-server @apollo/federation @apollo/gateway

    Some additional libraries to help run the services in parallel:

    npm i --save nodemon ts-node concurrently

    Let’s go ahead and write the structure for the gateway service first. Let’s create a file gateway.ts:

    touch gateway.ts

    And add the following code snippet:

    import { ApolloServer } from 'apollo-server';
import { ApolloGateway } from '@apollo/gateway';

const gateway = new ApolloGateway({
  serviceList: [],
});

const server = new ApolloServer({ gateway, subscriptions: false });

server.listen().then(({ url }) => {
  console.log(`Server ready at url: ${url}`);
});

    Note the serviceList is an empty array for now since we’ve yet to implement the individual services. In addition, we pass the subscriptions: false option to the apollo server config because currently, Apollo Federation does not support subscriptions.

    Next, let’s add the User service in a separate file user.ts using:

    touch user.ts

    The code will go in the user service as follows:

    import { buildFederatedSchema } from '@apollo/federation';
import { ApolloServer, gql } from 'apollo-server';
import User from './datasources/models/User';
import mongoStore from './mongoStore';

const typeDefs = gql`
  type User @key(fields: "id") {
    id: ID!
    username: String!
  }
  extend type Query {
    users: [User]
    user(id: ID!): User
  }
  extend type Mutation {
    createUser(userPayload: UserPayload): User
  }
  input UserPayload {
    username: String!
  }
`;

const resolvers = {
  Query: {
    users: async () => {
      const allUsers = await User.find({});
      return allUsers;
    },
    user: async (_, { id }) => {
      const currentUser = await User.findOne({ _id: id });
      return currentUser;
    },
  },
  User: {
    __resolveReference: async (ref) => {
      const currentUser = await User.findOne({ _id: ref.id });
      return currentUser;
    },
  },
  Mutation: {
    createUser: async (_, { userPayload: { username } }) => {
      const user = new User({ username });
      const createdUser = await user.save();
      return createdUser;
    },
  },
};

mongoStore();

const server = new ApolloServer({
  schema: buildFederatedSchema([{ typeDefs, resolvers }]),
});

server.listen({ port: 4001 }).then(({ url }) => {
  console.log(`User service ready at url: ${url}`);
});

    Let’s break down the code that went into the User service.

    Consider the User schema definition:

    type User @key(fields: "id") {
   id: ID!
   username: String!
}

    The @key directive helps other services understand the User schema is, in fact, an entity that can be extended within other individual services. The fields will help other services uniquely identify individual instances of the User schema based on the id.

    The Query and the Mutation types need to be extended by all implementing services according to the Apollo Federation documentation since they are always defined on a gateway level.

    As a side note, the User model imported from datasources/model/User

    import User from ‘./datasources/models/User’; is essentially a Mongoose ORM Model for MongoDB that will help in all the CRUD operations of a User entity in a MongoDB database. In addition, the mongoStore() function is responsible for establishing a connection to the MongoDB database server.

    The User model implementation internally in Mongoose ORM looks something like this:

    export const UserSchema = new Schema({
  username: {
    type: String,
  },
});

export default mongoose.model(
  'User',
  UserSchema
);

    In the Query type, the users and the user(id: ID!) queries fetch a list or the details of individual users.

    In the resolvers, we define a __resolveReference function responsible for returning an instance of the User entity to all other implementing services, which just have a reference id of a User entity and need to return an instance of the User entity. The ref parameter is an object { id: ‘userEntityId’ } that contains the id of an instance of the User entity that may be passed down from other implementing services that need to resolve the reference of a User entity based on the reference id. Internally, we fire a mongoose .findOne query to return an instance of the User from the users database based on the reference id. To illustrate the resolver, 

    User: {
    __resolveReference: async (ref) => {
      const currentUser = User.findOne({ _id: ref.id });
      return currentUser;
    },
  },

    At the end of the file, we make sure the service is running on a unique port number 4001, which we pass as an option while running the apollo server. That concludes the User service.

    Next, let’s add the tweet service by creating a file tweet.ts using:

    touch tweet.ts

    The following code goes as a part of the tweet service:

    import { buildFederatedSchema } from '@apollo/federation';
import { ApolloServer, gql } from 'apollo-server';
import Tweet from './datasources/models/Tweet';
import TweetAPI from './datasources/tweet';
import mongoStore from './mongoStore';

const typeDefs = gql`
  type Tweet {
    text: String
    id: ID!
    creator: User
  }
  extend type User @key(fields: "id") {
    id: ID! @external
    tweets: [Tweet]
  }
  extend type Query {
    tweet(id: ID!): Tweet
    tweets: [Tweet]
  }
  extend type Mutation {
    createTweet(tweetPayload: TweetPayload): Tweet
  }
  input TweetPayload {
    userId: String
    text: String
  }
`;

const resolvers = {
  Query: {
    tweet: async (_, { id }) => {
      const currentTweet = await Tweet.findOne({ _id: id });
      return currentTweet;
    },
    tweets: async () => {
      const tweetsList = await Tweet.find({});
      return tweetsList;
    },
  },
  Tweet: {
    creator: (tweet) => ({ __typename: 'User', id: tweet.userId }),
  },
  User: {
    tweets: async (user) => {
      const tweetsByUser = await Tweet.find({ userId: user.id });
      return tweetsByUser;
    },
  },
  Mutation: {
    createTweet: async (_, { tweetPayload: { text, userId } }) => {
      const newTweet = new Tweet({ text, userId });
      const createdTweet = await newTweet.save();
      return createdTweet;
    },
  },
};

mongoStore();

const server = new ApolloServer({
  schema: buildFederatedSchema([{ typeDefs, resolvers }]),
});

server.listen({ port: 4002 }).then(({ url }) => {
  console.log(`Tweet service ready at url: ${url}`);
});

    Let’s break down the Tweet service as well

    type Tweet {
   text: String
   id: ID!
   creator: User
}

    The Tweet schema has the text field, which is the content of the tweet, a unique id of the tweet,  and a creator field, which is of the User entity type and resolves into the details of the user that created the tweet:

    extend type User @key(fields: "id") {
   id: ID! @external
   tweets: [Tweet]
}

    We extend the User entity schema in this service, which has the id field with an @external directive. This helps the Tweet service understand that based on the given id field of the User entity schema, the instance of the User entity needs to be derived from another service (user service in this case).

    As we discussed previously, the tweets field of the extended User schema for the user entity should be resolved in the Tweet service since all the resolvers and access to the data sources with respect to the Tweets entity resides in this service.

    The Query and Mutation types of the Tweet service are pretty straightforward; we have a tweets and a tweet(id: ID!) queries to resolve a list or resolve an individual instance of the Tweet entity.

    Let’s further break down the resolvers:

    Tweet: {
   creator: (tweet) => ({ __typename: 'User', id: tweet.userId }),
},

    To resolve the creator field of the Tweet entity, the Tweet service needs to tell the gateway that this field will be resolved by the User service. Hence, we pass the id of the User and a __typename for the gateway to be able to call the right service to resolve the User entity instance. In the User service earlier, we wrote a  __resolveReference resolver, which will resolve the reference of a User based on an id.

    User: {
   tweets: async (user) => {
       const tweetsByUser = await Tweet.find({ userId: user.id });
       return tweetsByUser;
   },
},

    Now, we need to resolve the tweets field of the User entity extended in the Tweet service. We need to write a resolver where we get the parent user entity reference in the first argument of the resolver using which we can fire a Mongoose ORM query to return all the tweets created by the user given its id.

    At the end of the file, similar to the User service, we make sure the Tweet service runs on a different port by adding the port: 4002 option to the Apollo server config. That concludes both our implementing services.

    Now that we have our services ready, let’s update our gateway.ts file to reflect the added services:

    import { ApolloServer } from 'apollo-server';
import { ApolloGateway } from '@apollo/gateway';

const gateway = new ApolloGateway({
  serviceList: [
      { name: 'users', url: 'http://localhost:4001' },
      { name: 'tweets', url: 'http://localhost:4002' },
    ],
});

const server = new ApolloServer({ gateway, subscriptions: false });

server.listen().then(({ url }) => {
  console.log(`Server ready at url: ${url}`);
});

    We’ve added two services to the serviceList with a unique name to identify each service followed by the URL they are running on.

    Next, let’s make some small changes to the package.json file to make sure the services and the gateway run in parallel:

    "scripts": {
    "start": "concurrently -k npm:server:*",
    "server:gateway": "nodemon --watch 'src/**/*.ts' --exec 'ts-node' src/gateway.ts",
    "server:user": "nodemon --watch 'src/**/*.ts' --exec 'ts-node' src/user.ts",
    "server:tweet": "nodemon --watch 'src/**/*.ts' --exec 'ts-node' src/tweet.ts"
  },

    The concurrently library helps run 3 separate scripts in parallel. The server:* scripts spin up a dev server using nodemon to watch and reload the server for changes and ts-node to execute Typescript node.

    Let’s spin up our server:

    npm start

    On visiting the http://localhost:4000, you should see the GraphQL query playground running an Apollo server:

    Querying and Mutation from the Client

    Initially, let’s fire some mutations to create two users and some tweets by those users.

    Mutations

    Here we have created a user with the username “@elonmusk” that returns the id of the user. Fire the following mutations in the GraphQL playground:

     

    We will create another user named “@billgates” and take a note of the ID.

    Here is a simple mutation to create a tweet by the user “@elonmusk”. Now that we have two created users, let’s fire some mutations to create tweets by those users:

    Here is another mutation that creates a tweet by the user“@billgates”.

    After adding a couple of those, we are good to fire our queries, which will allow the gateway to compose the data by resolving fields through different services.

    Queries

    Initially, let’s list all the tweets along with their creator, which is of type User. The query will look something like:

    {
 tweets {
   text
   creator {
     username
   }
 }
}

    When the gateway encounters a query asking for tweet data, it forwards that query to the Tweet service since the Tweet service that extends the Query type has a tweet query defined in it. 

    On encountering the creator field of the tweet schema, which is of the type User, the creator resolver within the Tweet service is invoked. This is essentially just passing a __typename and an id, which tells the gateway to resolve this reference from another service.

    In the User service, we have a __resolveReference function, which returns the complete instance of a user given it’s id passed from the Tweet service. It also helps all other implementing services that need the reference of a User entity resolved.

    On firing the query, the response should look something like:

    {
  "data": {
    "tweets": [
      {
        "text": "I own Tesla",
        "creator": {
          "username": "@elonmusk"
        }
      },
      {
        "text": "I own SpaceX",
        "creator": {
          "username": "@elonmusk"
        }
      },
      {
        "text": "I own PayPal",
        "creator": {
          "username": "@elonmusk"
        }
      },
      {
        "text": "I own Microsoft",
        "creator": {
          "username": "@billgates"
        }
      },
      {
        "text": "I own XBOX",
        "creator": {
          "username": "@billgates"
        }
      }
    ]
  }
}

    Now, let’s try it the other way round. Let’s list all users and add the field tweets that will be an array of all the tweets created by that user. The query should look something like:

    {
 users {
   username
   tweets {
     text
   }
 }
}

    When the gateway encounters the query of type users, it passes down that query to the user service. The User service is responsible for resolving the username field of the query.

    On encountering the tweets field of the users query, the gateway checks if any other implementing service has extended the User entity and has a resolver written within the service to resolve any additional fields of the type User.

    The Tweet service has extended the type User and has a resolver for the User type to resolve the tweets field, which will fetch all the tweets created by the user given the id of the user.

    On firing the query, the response should be something like:

    {
  "data": {
    "users": [
      {
        "username": "@elonmusk",
        "tweets": [
          {
            "text": "I own Tesla"
          },
          {
            "text": "I own SpaceX"
          },
          {
            "text": "I own PayPal"
          }
        ]
      },
      {
        "username": "@billgates",
        "tweets": [
          {
            "text": "I own Microsoft"
          },
          {
            "text": "I own XBOX"
          }
        ]
      }
    ]
  }
}

    Conclusion

    To scale an enterprise data graph on a monolith GraphQL service brings along a lot of challenges. Having the ability to distribute our data graph into implementing services that can be individually maintained or scaled using Apollo Federation helps to quell any concerns.

    There are further advantages of federated services. Considering our example above, we could have two different kinds of datastores for the User and the Tweet service. While the User data could reside on a NoSQL database like MongoDB, the Tweet data could be on a SQL database like Postgres or SQL. This would be very easy to implement since each service is only responsible for resolving references only for the type they own.

    Final Thoughts

    One of the key advantages of having different services that can be maintained individually is the ability to deploy each service separately. In addition, this also enables deployment of different services independently to different platforms such as Firebase, Lambdas, etc.

    A single monolith GraphQL server deployed on an instance or a single serverless platform can have some challenges with respect to scaling an instance or handling high concurrency as mentioned above.

    By splitting out the services, we could have a separate serverless function for each implementing service that can be maintained or scaled individually and also a separate function on which the gateway can be deployed.

    One popular usage of GraphQL Federation can be seen in this Netflix Technology blog, where they’ve explained how they solved a bottleneck with the GraphQL APIs in Netflix Studio . What they did was create a federated GraphQL microservices architecture, along with a Schema store using Apollo Federation. This solution helped them create a unified schema but with distributed ownership and implementation.

  • Lessons Learnt While Building an ETL Pipeline for MongoDB & Amazon Redshift Using Apache Airflow

    Recently, I was involved in building an ETL (Extract-Transform-Load) pipeline. It included extracting data from MongoDB collections, perform transformations and then loading it into Redshift tables. Many ETL solutions are available in the market which kind-of solves the issue, but the key part of an ETL process lies in its ability to transform or process raw data before it is pushed to its destination.

    Each ETL pipeline comes with a specific business requirement around processing data which is hard to be achieved using off-the-shelf ETL solutions. This is why a majority of ETL solutions are custom built manually, from scratch. In this blog, I am going to talk about my learning around building a custom ETL solution which involved moving data from MongoDB to Redshift using Apache Airflow.

    Background:

    I began by writing a Python-based command line tool which supported different phases of ETL, like extracting data from MongoDB, processing extracted data locally, uploading the processed data to S3, loading data from S3 to Redshift, post-processing and cleanup. I used the PyMongo library to interact with MongoDB and the Boto library for interacting with Redshift and S3.

    I kept each operation atomic so that multiple instances of each operation can run independently of each other, which will help to achieve parallelism. One of the major challenges was to achieve parallelism while running the ETL tasks. One option was to develop our own framework based on threads or developing a distributed task scheduler tool using a message broker tool like Celery combined with RabbitMQ. After doing some research I settled for Apache Airflow. Airflow is a Python-based scheduler where you can define DAGs (Directed Acyclic Graphs), which would run as per the given schedule and run tasks in parallel in each phase of your ETL. You can define DAG as Python code and it also enables you to handle the state of your DAG run using environment variables. Features like task retries on failure handling are a plus.

    We faced several challenges while getting the above ETL workflow to be near real-time and fault tolerant. We discuss the challenges faced and the solutions below:

    Keeping your ETL code changes in sync with Redshift schema

    While you are building the ETL tool, you may end up fetching a new field from MongoDB, but at the same time, you have to add that column to the corresponding Redshift table. If you fail to do so the ETL pipeline will start failing. In order to tackle this, I created a database migration tool which would become the first step in my ETL workflow.

    The migration tool would:

    • keep the migration status in a Redshift table and
    • would track all migration scripts in a code directory.

    In each ETL run, it would get the most recently ran migrations from Redshift and would search for any new migration script available in the code directory. If found it would run the newly found migration script after which the regular ETL tasks would run. This adds the onus on the developer to add a migration script if he is making any changes like addition or removal of a field that he is fetching from MongoDB.

    Maintaining data consistency

    While extracting data from MongoDB, one needs to ensure all the collections are extracted at a specific point in time else there can be data inconsistency issues. We need to solve this problem at multiple levels:

    • While extracting data from MongoDB define parameters like modified date and extract data from different collections with a filter as records less than or equal to that date. This will ensure you fetch point in time data from MongoDB.
    • While loading data into Redshift tables, don’t load directly to master table, instead load it to some staging table. Once you are done loading data in staging for all related collections, load it to master from staging within a single transaction. This way data is either updated in all related tables or in none of the tables.

    A single bad record can break your ETL

    While moving data across the ETL pipeline into Redshift, one needs to take care of field formats. For example, the Date field in the incoming data can be different than that in the Redshift schema design. Another example can be that the incoming data can exceed the length of the field in the schema. Redshift’s COPY command which is used to load data from files to redshift tables is very vulnerable to such changes in data types. Even a single incorrectly formatted record will lead to all your data getting rejected and effectively breaking the ETL pipeline.

    There are multiple ways in which we can solve this problem. Either handle it in one of the transform jobs in the pipeline. Alternately we put the onus on Redshift to handle these variances. Redshift’s COPY command has many options which can help you solve these problems. Some of the very useful options are

    • ACCEPTANYDATE: Allows any date format, including invalid formats such as 00/00/00 00:00:00, to be loaded without generating an error.
    • ACCEPTINVCHARS: Enables loading of data into VARCHAR columns even if the data contains invalid UTF-8 characters.
    • TRUNCATECOLUMNS: Truncates data in columns to the appropriate number of characters so that it fits the column specification.

    Redshift going out of storage

    Redshift is based on PostgreSQL and one of the common problems is when you delete records from Redshift tables it does not actually free up space. So if your ETL process is deleting and creating new records frequently, then you may run out of Redshift storage space. VACUUM operation for Redshift is the solution to this problem. Instead of making VACUUM operation a part of your main ETL flow, define a different workflow which runs on a different schedule to run VACUUM operation. VACUUM operation reclaims space and resorts rows in either a specified table or all tables in the current database. VACUUM operation can be FULL, SORT ONLY, DELETE ONLY & REINDEX. More information on VACUUM can be found here.

    ETL instance going out of storage

    Your ETL will be generating a lot of files by extracting data from MongoDB onto your ETL instance. It is very important to periodically delete those files otherwise you are very likely to go out of storage on your ETL server. If your data from MongoDB is huge, you might end up creating large files on your ETL server. Again, I would recommend defining a different workflow which runs on a different schedule to run a cleanup operation.

    Making ETL Near Real Time

    Processing only the delta rather than doing a full load in each ETL run

    ETL would be faster if you keep track of the already processed data and process only the new data. If you are doing a full load of data in each ETL run, then the solution would not scale as your data scales. As a solution to this, we made it mandatory for the collection in our MongoDB to have a created and a modified date. Our ETL would check the maximum value of the modified date for the given collection from the Redshift table. It will then generate the filter query to fetch only those records from MongoDB which have modified date greater than that of the maximum value. It may be difficult for you to make changes in your product, but it’s worth the effort!

    Compressing and splitting files while loading

    A good approach is to write files in some compressed format. It saves your storage space on ETL server and also helps when you load data to Redshift. Redshift COPY command suggests that you provide compressed files as input. Also instead of a single huge file, you should split your files into parts and give all files to a single COPY command. This will enable Redshift to use it’s computing resources across the cluster to do the copy in parallel, leading to faster loads.

    Streaming mongo data directly to S3 instead of writing it to ETL server

    One of the major overhead in the ETL process is to write data first to ETL server and then uploading it to S3. In order to reduce disk IO, you should not store data to ETL server. Instead, use MongoDB’s handy stream API. For MongoDB Node driver, both the collection.find() and the collection.aggregate() function return cursors. The stream method also accepts a transform function as a parameter. All your custom transform logic could go into the transform function. AWS S3’s node library’s upload() function, also accepts readable streams. Use the stream from the MongoDB Node stream method, pipe it into zlib to gzip it, then feed the readable stream into AWS S3’s Node library. Simple! You will see a large improvement in your ETL process by this simple but important change.

    Optimizing Redshift Queries

    Optimizing Redshift Queries helps in making the ETL system highly scalable, efficient and also reduce the cost. Lets look at some of the approaches:

    Add a distribution key

    Redshift database is clustered, meaning your data is stored across cluster nodes. When you query for certain set of records, Redshift has to search for those records in each node, leading to slow queries. A distribution key is a single metric, which will decide the data distribution of all data records across your tables. If you have a single metric which is available for all your data, you can specify it as distribution key. When loading data into Redshift, all data for a certain value of distribution key will be placed on a single node of Redshift cluster. So when you query for certain records Redshift knows exactly where to search for your data. This is only useful when you are also using the distribution key to query the data.

    Source: Slideshare

     

    Generating a numeric primary key for string primary key

    In MongoDB, you can have any type of field as your primary key. If your Mongo collections are having a non-numeric primary key and you are using those same keys in Redshift, your joins will end up being on string keys which are slower. Instead, generate numeric keys for your string keys and joining on it which will make queries run much faster. Redshift supports specifying a column with an attribute as IDENTITY which will auto-generate numeric unique value for the column which you can use as your primary key.

    Conclusion:

    In this blog, I have covered the best practices around building ETL pipelines for Redshift  based on my learning. There are many more recommended practices which can be easily found in Redshift and MongoDB documentation. 

  • Prow + Kubernetes – A Perfect Combination To Execute CI/CD At Scale

    Intro

    Kubernetes is currently the hottest and standard way of deploying workloads in the cloud. It’s well-suited for companies and vendors that need self-healing, high availability, cloud-agnostic characteristics, and easy extensibility.

    Now, on another front, a problem has arisen within the CI/CD domain. Since people are using Kubernetes as the underlying orchestrator, they need a robust CI/CD tool that is entirely Kubernetes-native.

    Enter Prow

    Prow compliments the Kubernetes family in the realm of automation and CI/CD.

    In fact, it is the only project that best exemplifies why and how Kubernetes is such a superb platform to execute CI/CD at scale.

    Prow (meaning: portion of a ship’s bow—ship’s front end–that’s above water) is a Kubernetes-native CI/CD system, and it has been used by many companies over the past few years like Kyma, Istio, Kubeflow, Openshift, etc.

    Where did it come from?

    Kubernetes is one of the largest and most successful open-source projects on GitHub. When it comes to Prow’s conception , the Kubernetes community was trying hard to keep its head above water in matters of CI/CD. Their needs included the execution of more than 10k CI/CD jobs/day, spanning over 100+ different repositories in various GitHub organizations—and other automation technology stacks were just not capable of handling everything at this scale.

    So, the Kubernetes Testing SIG created their own tools to compliment Prow. Because Prow is currently residing under Kubernetes test-infra project, one might underestimate its true prowess/capabilities. I personally would like to see Prow receive a dedicated repo coming out from under the umbrella of test-infra.

    What is Prow?

    Prow is not too complex to understand but still vast in a subtle way. It is designed and built on a distributed microservice architecture native to Kubernetes.

    It has many components that integrate with one another (plank, hook, etc.) and a bunch of standalone ones that are more of a plug-n-play nature (trigger, config-updater, etc.).

    For the context of this blog, I will not be covering Prow’s entire architecture, but feel free to dive into it on your own later. 

    Just to name the main building blocks for Prow:

    • Hook – acts as an API gateway to intercept all requests from Github, which then creates a Prow job custom resource that reads the job configuration as well as calls any specific plugin if needed.‍
    • Plank – is the Prow job controller; after Hook creates a Prow job, Plank processes it and creates a Kubernetes pod for it to run the tests.‍
    • Deck – serves as the UI for the history of jobs that ran in the past or are currently running.‍
    • Horologium – is the component that processes periodic jobs only.‍
    • Sinker – responsible for cleaning up old jobs and pods from the cluster.

    More can be found here: Prow Architecture. Note that this link is not the official doc from Kubernetes but from another great open source project that uses Prow extensively day-in-day-out – Kyma.

    This is how Prow can be picturized:


     

     

    Here is a list of things Prow can do and why it was conceived in the first place.

    • GitHub Automation on a wide range
      ‍
      – ChatOps via slash command like “/foo”
      – Fine-tuned policies and permission management in GitHub via OWNERS files
      – tide – PR/merge automation
      – ghProxy – to avoid hitting API limits and to use GitHub API request cache
      – label plugin – labels management 
      – branchprotector – branch protection configuration 
      – releasenote – release notes management
    • Job Execution engine – Plank‍
    • Job status Reporting to CI/CD dashboard – crier‍
    • Dashboards for comprehensive job/PR history, merge status, real-time logs, and other statuses – Deck‍
    • Plug-n-play service to interact with GCS and show job artifacts on dashboard – Spyglass‍
    • Super easy pluggable Prometheus stack for observability – metrics‍
    • Config-as-Code for Prow itself – updateconfig‍
    • And many more, like sinker, branch protector, etc.

    Possible Jobs in Prow

    Here, a job means any “task that is executed over a trigger.” This trigger can be anything from a github commit to a new PR or a periodic cron trigger. Possible jobs in Prow include:  

    • Presubmit – these jobs are triggered when a new github PR is created.‍
    • Postsubmit – triggered when there is a new commit.‍
    • Periodic – triggered on a specific cron time trigger.

    Possible states for a job

    • triggered – a new Prow-job custom resource is created reading the job configs‍
    • pending – a pod is created in response to the Prow-job to run the scripts/tests; Prow-job will be marked pending while the pod is getting created and running ‍
    • success – if a pod succeeds, the Prow-job status will change to success ‍
    • failure – if a pod fails, the Prow-job status will be marked failure‍
    • aborted – when a job is running and the same one is retriggered, then the first pro-job execution will be aborted and its status will change to aborted and the new one is marked pending

    What a job config looks like:

    presubmits:
  kubernetes/community:
  - name: pull-community-verify  # convention: (job type)-(repo name)-(suite name)
    branches:
    - master
    decorate: true
    always_run: true
    spec:
      containers:
      - image: golang:1.12.5
        command:
        - /bin/bash
        args:
        - -c
        - "export PATH=$GOPATH/bin:$PATH && make verify"

    • Here, this job is a “presubmit” type, meaning it will be executed when a PR is created against the “master” branch in repo “kubernetes/community”.
    • As shown in spec, a pod will be created from image “Golang” where this repo will be cloned, and the mentioned command will be executed at the start of the container.
    • The output of that command will decide if the pod has succeeded or failed, which will, in turn, decide if the Prow job has successfully completed.

    More jobs configs used by Kubernetes itself can be found here – Jobs

    Getting a minimalistic Prow cluster up and running on the local system in minutes.

    Pre-reqs:

    • Knowledge of Kubernetes 
    • Knowledge of Google Cloud and IAM

    For the context of this blog, I have created a sample github repo containing all the basic manifest files and config files. For this repo, the basic CI has also been configured. Feel free to clone/fork this and use it as a getting started guide.

    Let’s look at the directory structure for the repo:

    .
├── docker/     # Contains docker image in which all the CI jobs will run
├── hack/       # Contains small hack scripts used in a wide range of jobs 
├── hello.go
├── hello_test.go
├── Dockerfile
├── Makefile
├── prow
│   ├── cluster/       # Install prow on k8s cluster
│   ├── jobs/          # CI jobs config
│   ├── labels.yaml    # Prow label config for managing github labels
│   ├── config.yaml    # Prow config
│   └── plugins.yaml   # Prow plugins config
└── README.md

    1. Create a bot account. For info, look here. Add this bot as a collaborator in your repo. 

    2. Create an OAuth2 token from the GitHub GUI for the bot account.

    $ echo "PUT_TOKEN_HERE" > oauth
$ kubectl create secret generic oauth --from-file=oauth=oauth

    3. Create an OpenSSL token to be used with the Hook.

    $ openssl rand -hex 20 > hmac
$ kubectl create secret generic hmac --from-file=hmac=hmac

    4. Install all the Prow components mentioned in prow-starter.yaml.

    $ make deploy-prow

    5. Update all the jobs and plugins needed for the CI (rules mentioned in the Makefile). Use commands:

    • Updates in plugins.yaml and presubmits.yaml:
    • Change the repo name (velotio-tech/k8s-prow-guide) for the jobs to be configured 
    • Updates in config.yaml:
    • Create a GCS bucket 
    • Update the name of GCS bucket (GCS_BUCKET_NAME) in the config.yaml‍
    • Create a service_account.json with GCS storage permission and download the JSON file 
    • Create a secret from above service_account.json
    $ kubectl create secret generic gcs-sa --from-file=service-account.json=service-account.json

    • Update the secret name (GCS_SERVICE_ACC) in config.yaml
    $ make update-config
$ make update-plugins
$ make update-jobs

    6. For exposing a webhook from GitHub repo and pointing it to the local machine, use Ultrahook. Install Ultrahook. This will give you a publicly accessible endpoint. In my case, the result looked like this: http://github.sanster23.ultrahook.com. 

    $ echo "api_key: <API_KEY_ULTRAHOOK>" > ~/.ultrahook
$ ultrahook github http://<MINIKUBE_IP>:<HOOK_NODE_PORT>/hook

    7. Create a webhook in your repo so that all events can be published to Hook via the public URL above:

    • Set the webhook URL from Step 6
    • Set Content Type as application/json‍
    • Set the value of token the same as hmac token secret, created in Step 2 
    • Check the “Send me everything” box

    8. Create a new PR and see the magic.

    9. Dashboard for Prow will be accessible at http://<minikube_ip>:<deck_node_port></deck_node_port></minikube_ip>

    • MINIKUBE_IP : 192.168.99.100  ( Run “minikube ip”)
    • DECK_NODE_PORT :  32710 ( Run “kubectl get svc deck” )

    I will leave you guys with an official reference of Prow Dashboard:

    What’s Next

    Above is an effort just to give you a taste of what Prow can do with and how easy it is to set up at any scale of infra and for a project of any complexity.

    —

    P.S. – The content surrounding Prow is scarce, making it a bit unexplored in certain ways, but I found this helpful channel on the Kubernetes Slack #prow. Hopefully, this helps you explore the uncharted waters of Kubernetes Native CI/CD. 

  • Set Up A Production-ready REST API Server Using TypeScript, Express And PostgreSQL

    Introduction

    So, you have a brilliant idea for a web application. It’s going to be the next big thing, and you are super-excited about it. Maybe you have already started building the perfect React/Angular UI for your app.

    Eventually, you realize that, like most web apps, your app is going to be data-intensive and will need a lightning-fast web server. You know that Node.js is the de facto standard for web servers for how well it unifies front-end and back-end web development with JavaScript, so you go for it.

    But you want your server to be robust and reliable too. A colleague introduces you to TypeScript, the superset of JavaScript developed by Microsoft, and recommends it for its strict static typing and compilation.

    Now comes storing the data. Naturally, you select PostgreSQL. After all, it is the most advanced Relational Database Management System (RDBMS) in the world, with its object-oriented features and extensibility. But RDBMSs can be slow for frequently used data and caching, so you decide to add Redis, the in-memory cache, to decrease data access latency and ease the load off your relational data store.

    That’s it. You have a perfect server waiting to be built. And while the initial process of getting it up and running can get arduous, you have come to the right place. This blog is going to guide you through the initial setup process.

    Prerequisites

    I am assuming you have a non-root user with sudo privileges running on Ubuntu 16.04. Before we start, please make sure you have the following: 

    1. NPM (~v6.9.0) and Node.js (~v10.16.0) – You can use this How to Install Node.js on Ubuntu 16.04
    2. Redis – How to install Redis on Ubuntu 16.04
    3. PostgreSQL – How to install PostgreSQL on Ubuntu 16.04

    Of course, MacOS or Windows would do fine too for this tutorial, but to use them, please find appropriate installation guides on the Internet before moving forward. 

    If you don’t want to go through the steps below, you can check out my GitHub Repo typescript-express-server and use it as your application skeleton. It has been set up with default configurations, which you can change later. Nevertheless, I strongly recommend going through this guide to further your understanding of the project files and configuration nuances.

    Initializing Server (Express with TypeScript)

    Setting up an Express Application with TypeScript can be done in three steps: 

    Initialize project using NPM

    Create a folder and run:

    npm init

    This will ask you a couple of project-specific questions, like name and version, and will create a package.json file, which may look like this:

    {
 "name": "my-typescript-express-server",
 "version": "0.0.0",
 "scripts": {
   "start": "node ./dist/index.js --env=production",
   "start:dev": "ts-node -r tsconfig-paths/register ./src",
  },
 "dependencies": {
   "cookie-parser": "^1.4.5",
   "dotenv": "^8.2.0"
  },
 "devDependencies": {
   "find": "^0.3.0",
   "fs-extra": "^9.0.1",
 }
}

    This manifest file will contain all the metadata of your project, like module dependencies, configs, and scripts. For more information, check out this very good read about the basics of package.json. 

    Setting up TypeScript Configuration (tsconfig.json)

    This file needs to be created in the root of a TypeScript project. During development, TypeScript provides us with the convenience of running the code directly from the .ts extension files. But during production, since Node.js only understands JS, the entire TS files need to be transpiled to JS. Some of the options are: include – specifies the files to be included, exclude –  the files to exclude, and the compiler options: outFIle and moduleResolution.

    First, we need to install some TypeScript specific modules: 

    npm i typescript ts-node tsconfig-paths

    CODE: https://gist.github.com/velotiotech/95b021f1728a9b8a61d9fca89b0b9e59.js

    This is the tsconfig.json file with some default configurations:

    For a detailed reference, checkout tsconfig.json.

    Setting up ESLint

    It is not mandatory to use this JavaScript linter, but it’s highly recommended for enforcing code standards and keeping code clean. TypeScript projects once used TSLint, but it has been deprecated in favor of ESLint.

    Run this command:

    npm install --save-dev eslint @typescript-eslint/parser @typescript-eslint/eslint-plugin

    Create a .eslintrc file in the project root and use the following starter configuration:

    {
 "root": true,
 "parser": "@typescript-eslint/parser",
 "plugins": [
   "@typescript-eslint"
 ],
 "extends": [
   "eslint:recommended",
   "plugin:@typescript-eslint/eslint-recommended",
   "plugin:@typescript-eslint/recommended"
 ]
}

    Lastly, add a lint script to package.json:

    {
  "name": "my-typescript-express-server",
  "version": "0.0.0",
  "scripts": {
    "start": "node ./dist/index.js --env=production",
    "start:dev": "ts-node -r tsconfig-paths/register ./src",
    "lint": "eslint . --ext .ts",
   },

    Now, you can run the command below to lint your codebase for lint errors:

    npm run lint

    ESLint has ample rules to enforce standards in your code. Please look them up at Eslint with TypeScript.

    Express App

    Finally, we need to install Express, which is as simple as running this command:

    npm install --save express @types/express

    You need a server file (src/Server.ts), which you can create like this:

    import cookieParser from 'cookie-parser';
import express from 'express';
import { BAD_REQUEST } from 'http-status-codes';
import BaseRouter from './routes';
const app = express();
app.use(express.json());
app.use(express.urlencoded({extended: true}));
app.use(cookieParser());
// Add APIs
app.use('/api', BaseRouter);
// Export express instance
export default app;

    You will also need src/index.ts that will be the entry point for your application:

    import app from './Server';
// Start the server
const port = Number(process.env.PORT || 3000);
app.listen(port, () => {
   logger.info('Express server started on port: ' + port);
});

    Error Handling

    Many Express servers are configured to swallow all errors by configuring an Uncaught Exception handler, which in my opinion, is bad news. The best thing to do is to allow the application to crash and restart. Uncaught Exceptions in Node.js is a good read regarding this.

    Nonetheless, we are going to configure an error handler that will print errors and send a BadRequest response when an invalid HTTP request comes your API’s way.

    In the src/Server.ts, add this:

    /// Print API errors
// eslint-disable-next-line @typescript-eslint/no-unused-vars
app.use((err: Error, req: Request, res: Response, next: NextFunction) => {
   logger.error(err.message, err);
   return res.status(BAD_REQUEST).json({
       error: err.message,
   });
});

    Kudos! You have a basic Express server set up. Fire it up by running:

    npm run start:dev

    Connecting with the Database Store using TypeORM

    We have a basic server ready to go, but we need to connect it to our Postgres database using an ORM. TypeORM is a versatile ORM that supports both Active Record and Data Mapper patterns, unlike all other JavaScript ORMs. It can be installed on our server with the following steps:

    npm i --save typeorm pg reflect-metadata

    Create an ormconfig.json file in your project root with the following configuration:

    {
   "synchronize": true,
   "logging": false,
   "entities": [
      "src/entities/**/*.ts"
   ],
   "cli": {
      "entitiesDir": "src/entity",
      "migrationsDir": "src/migration",
      "subscribersDir": "src/subscriber"
    },
   "migrations": [
      "src/migration/**/*.ts"
   ],
   "subscribers": [
      "src/subscriber/**/*.ts"
    ]
}

    Create a src/db.ts file that will initialize the database connection:

    import "reflect-metadata";
import {createConnection} from "typeorm";
import { Tedis } from "tedis";
import logger from '../src/shared/Logger';
export async function intializeDB(): Promise<void> {
  await createConnection();
}

    TypeORM Entities are classes that represent the data models in our application. We are going to build a User Entity (which application doesn’t have a user, duh!) like this in src/entities/User.ts:

    import {Entity, PrimaryGeneratedColumn, Column} from "typeorm";
@Entity()
export class User {
  @PrimaryGeneratedColumn()
  id: number;
  @Column()
  firstName: string;
  @Column()
  lastName: string;
  @Column()
  age: number;
}

    Then, add these lines to src/index.ts:

    import { intializeDB } from './db';
intializeDB();

    You will need the env variables, like TYPEORM_CONNECTION, TYPEORM_HOST, and TYPEORM_USERNAME, with your postgres db’s connection params. Please check TypeORMs documentation for more details. 

    Connecting Redis

    We will use Tedis, the TypeScript wrapper for Redis in our server:

    npm i tedis

    Add these lines to src/db.ts:

    export function initializeCache(port: number | undefined) : unknown {
 const tedis = new Tedis({
   port: port,
   host: "127.0.0.1"
 });
 return tedis;
}

    And these lines to src/index.ts:

    const redisPORT = Number(process.env.REDIS_PORT || 6379)
initializeCache(redisPORT);

    Now, your application code can use the Redis cache using the client created above.

    Configuring Logging

    Logging is pivotal to an application because it gives us a real-time view of the state of our application. For development, we are going to install the Morgan Request Logger, a library that logs HTTP requests params. It comes really handy for debugging. 

    npm i morgan

    And include this in src/Server.ts:

    export function initializeCache(port: number | undefined) : unknown {
 const tedis = new Tedis({
   port: port,
   host: "127.0.0.1"
 });
 return tedis;
}

    Winston can be used as the system-wide universal logger. Install it like this:

    npm i winston

    Then, add a src/shared/Logger.js file:

    import { createLogger, format, transports } from 'winston';
// Import Functions
const { File, Console } = transports;
// Init Logger
const logger = createLogger({
   level: 'info',
});
const errorStackFormat = format((info) => {
   if (info.stack) {
      // tslint:disable-next-line:no-console
      console.log(info.stack);
      return false;
    }
      return info;
   });
   const consoleTransport = new Console({
       format: format.combine(
           format.colorize(),
           format.simple(),
           errorStackFormat(),
       ),
   });
   logger.add(consoleTransport);
}
export default logger;

    Now, you can use this logger from anywhere in the code, be it for error logging in your API methods or for debugging purposes:

    import logger from '@shared/Logger';
export async function intializeDB(): Promise<void> {
 await createConnection()
 logger.info('Database successfully initialized');
}

    Creating your First API Service

    This is the moment you have been waiting for: creating your first API service for your application, the crux of the functionality that will define your web application.

    This API service is a simple GET request handler, which returns all the users in your database. You should have src/Users.ts, which can look like:

    import { Request, Response, Router } from 'express';
import { BAD_REQUEST, CREATED, OK } from 'http-status-codes';
import { ParamsDictionary } from 'express-serve-static-core';
import { getConnection } from "typeorm";
import { User } from "../entities/User";
import { paramMissingError } from '../shared/constants';
const router = Router();
router.get('/all', async (req: Request, res: Response) => {
   const users = await getConnection()
       .getRepository(User)
       .createQueryBuilder("user")
       .getMany();
   return res.status(OK).json({users});
});

    Add src/routes/index.ts

    import { Router } from 'express';
import UserRouter from './Users';
// Init router and path
const router = Router();
// Add sub-routes
router.use('/users', UserRouter);
// Export the base-router
export default router;

    Voila! Your API service is ready. Fire up your server, and then use Postman to make requests to your API and see the magic happen. 

    You can also add other API services for fetching a user by ID, deleting a user, creating a user, and updating a user. I will not discuss them here to keep this blog short. You can find these in the Github repository I mentioned in the beginning.

    Deploying your Server to Production

    What we have been doing has been in the development phase. Now, we need to take this to production. You just need to have a <project-root>/build.js </project-root>script that will create a <project-root>/dist</project-root> folder and transpile all the TypeScript files that you have written. It can look like this: 

    const fsE = require('fs-extra');
const childProcess = require('child_process');
// Remove current build
fsE.removeSync('./dist/');
// Copy front-end files
fsE.copySync('./src/public', './dist/public');
fsE.copySync('./src/views', './dist/views');
// Transpile the typescript files
childProcess.execSync('tsc --build tsconfig.prod.json');

    Then, add this line to your <project-root>/package.json</project-root>:

    "scripts": {
   "build": "node build.js",
   "lint": "eslint . --ext .ts",

    Now, you can use:

    node build.js

    Doing so builds up the <project-root>/dist</project-root> folder and transpiles your code. You can deploy this folder to your deployment environment and run it to start your production server:

    npm start

    Note: You will need to do some additional setting up of your Nginx or AWS Virtual Machine to complete your deployment, which is beyond the scope of this blog.

    Going Forward

    Congratulations. You have made it through this tutorial that guided you through the process of setting up a web server. But this is just the beginning, and there is no end to the improvements and optimizations that you can add to your server to make it better and sturdier. And you will continue to discover them in your journey of developing your web application. Some of the key points that I want to mention are:

    Managing Environments

    Your Web server will be operated in multiple environments, such as development, testing, and production. Some of the vital configurations like AWS credentials and DB passwords are sensitive information, and managing them per environment is key to your development and deployment cycle. I strongly recommend using libraries like Dotenv and keeping your env configurations separate in your codebase. You can look up typescript-express-server for this.

    Configuring Swagger

    Software developers nowadays swear by this tool. It’s proved to be a godsend for API documentation and keeping APIs in confirmation with the OpenAPI standard. On top of that, it also does API requests validation according to your API specifications. I strongly recommend you configure this in your web server.

    Writing Tests

    Writing API tests and unit tests can be a crucial part of web application development as it exposes possible gaps in your systems. You can use Superagent, the lightweight REST API, to test your APIs for all possible requests and response scenarios. Please look up the src/spec in typescript-express-server about how to use it. You can also use Postman for API Testing Automation. For most of the services that you write, you should make sure to add unit tests for each of those using Jest.

    Further Reading

    1. Node.js production checklist
    2. Node.js production best practices
    3. Production best practices: performance and reliability

  • Using DRF Effectively to Build Cleaner and Faster APIs in Django

    Django REST Framework (DRF) is a popular library choice when it comes to creating REST APIs with Django. With minimal effort and time, you can start creating APIs that support authentication, authorization, pagination, sorting, etc. Once we start creating production-level APIs, we must do a lot of customization that are highly supported by DRF.

    In this blog post, I will share some of the features that I have used extensively while working with DRF. We will be covering the following use cases:

    1. Using serializer context to pass data from view to serializer
    2. Handling reverse relationships in serializers
    3. Solving slow queries by eliminating the N+1 query problem
    4. Custom Response Format
    5. SerializerMethodField to add read-only derived data to the response
    6. Using Mixin to enable/disable pagination with Query Param

    This will help you to write cleaner code and improve API performance.

    Prerequisite:

    To understand the things discussed in the blog, the reader should have some prior experience of creating REST APIs using DRF. We will not be covering the basic concepts like serializers, API view/viewsets, generic views, permissions, etc. If you need help in building the basics, here is the list of resources from official documentation.

    Let’s explore Django REST Framework’s (DRF) lesser-known but useful features:

    1. Using Serializer Context to Pass Data from View to Serializer

    Let us consider a case when we need to write some complex validation logic in the serializer. 

    The validation method takes two parameters. One is the self or the serializer object, and the other is the field value received in the request payload. Our validation logic may sometimes need some extra information that must be taken from the database or derived from the view calling the serializer. 

    Next is the role of the serializer’s context data. The serializer takes the context parameter in the form of a python dictionary, and this data is available throughout the serializer methods. The context data can be accessed using self.context in serializer validation methods or any other serializer method. 

    Passing custom context data to the serializer

    To pass the context to the serializer, create a dictionary with the data and pass it in the context parameter when initializing the serializer.

    context_data = {"valid_domains": ValidDomain.objects.all()}
serializer = MySerializer(data=request.data, context=context_data)

    In case of generic view and viewsets, the serializer initialization is handled by the framework and passed the following as default context.

    {
   'request': self.request,
   'format': self.format_kwarg,
   'view': self
}

    Thanks to DRF, we can cleanly and easily customize the context data. 

    # override the get_serializer_context method in the generic viewset
class UserCreateListAPIView(generice.ListCreateAPIView):
    def get_serializer_context(self):
        context = super().get_serializer_context()
        # Update context data to add new data
   	  context.update({"valid_domains": ValidDomain.objects.all()})
   	  return context

    # read the context data in the serializer validation method
class UserSerializer(serializer.Serializer):
    def validate_email(self, val):
        valid_domains = serf.context.get("valid_domains")
        # main validation logic goes here

    2. Handling Reverse Relationships in Serializers 

    To better understand this, take the following example. 

    class User(models.Model):
   name = models.CharField(max_length=60)
   email = models.EmailField()


class Address(models.Model):
   detail = models.CharField(max_length=100)
   city = models.FloatField()
   user = models.ForeignKey(User, related_name="addresses", on_delete=models.CASCADE)

    We have a User model, which contains data about the customer and Address that has the list of addresses added. We need to return the user details along with their address detail, as given below.

    {
   "name": "Velotio",
   "email": "velotio@example.com",
   "addresses": [
   	{
       	"detail": "Akshya Nagar 1st Block 1st Cross, Rammurthy nagar",
       	"city": "Banglore"
   	},
   	{
       	"detail": "50 nd Floor, , Narayan Dhuru Street, Mandvi",
       	"city": "Mumbai"
   	},
   	{
       	"detail": "Ground Floor, 8/5, J K Bldg, H G Marg, Opp Gamdevi Temple, Grant Road",
       	"city": "Banglore"
   	}
   ]
}

    • Forward model relationships are automatically included in the fields returned by the ModelSerializer.
    • The relationship between User and Address is a reverse relationship and needs to be explicitly added in the fields. 
    • We have defined a related_name=addresses for the User Foreign Key in the Address; it can be used in the fields meta option. 
    • If we don’t have the related_name, we can use address_set, which is the default related_name.
    class UserSerializer(serializers.ModelSerializer):
      class Meta:
          model = User
          fields = ("name", "email", "addresses")

    The above code will return the following response:

    {
   "name": "Velotio",
   "email": "velotio@example.com",
   "addresses": [
       10,
       20,
       45
   ]
}

    But this isn’t what we need. We want to return all the information about the address and not just the IDs. DRF gives us the ability to use a serializer as a field to another serializer. 

    The below code shows how to use the nested Serializer to return the address details.

    class AddressSerializer(serializers.ModelSerializer):
   class Meta:
       model = Address
       fields = ("detail", "city") 

class UserSerializer(serializers.ModelSerializer):
   addresses = AddressSerializer(many=True, read_only=True)
   class Meta:
       model = User
       fields = ("name", "email", "addresses")

    • The read_only=True parameter marks the field as a read-only field. 
    • The addresses field will only be used in GET calls and will be ignored in write operations. 
    • Nested Serializers can also be used in write operations, but DRF doesn’t handle the creation/deletion of nested serializers by default.

    3. Solving Slow Queries by Eliminating the N+1 Query Problem

    When using nested serializers, the API needs to run queries over multiple tables and a large number of records. This can often lead to slower APIs. A common and easy mistake to make while using serializer with relationships is the N+1 queries problem. Let’s first understand the problem and ways to solve it.

    Identifying the N+1 Queries Problem 

    Let’s take the following API example and count the number of queries hitting the database on each API call.

    class Author(models.Model):
   name = models.CharField(max_length=20)


class Book(models.Model):
   name = models.CharField(max_length=20)
   author = models.ForeignKey("Author", models.CASCADE, related_name="books")
   created_at = models.DateTimeField(auto_now_add=True)

    class AuthorSerializer(serializers.ModelSerializer):
   class Meta:
   	model = Author
   	fields = "__all__"


class BookSerializer(serializers.ModelSerializer):
   author = AuthorSerializer()
   class Meta:
   	model = Book
   	fields = "__all__"

    class BookListCreateAPIView(generics.ListCreateAPIView):

	serializer_class = BookSerializer
	queryset = Book.objects.all()

    urlpatterns = [
	path('admin/', admin.site.urls),
	path('hello-world/', HelloWorldAPI.as_view()),
	path('books/', BookListCreateAPIView.as_view(), name="book_list")
]

    We are creating a simple API to list the books along with the author’s details. Here is the output:

    {
  "message": "",
  "errors": [],
  "data": [
    {
      "id": 1,
      "author": {
        "id": 3,
        "name": "Meet teacher."
      },
      "name": "Body society.",
      "created_at": "1973-08-03T02:43:22Z"
    },
    {
      "id": 2,
      "author": {
        "id": 49,
        "name": "Cause wait health."
      },
      "name": "Left next pretty.",
      "created_at": "2000-07-07T03:37:10Z"
    },
    {
      "id": 3,
      "author": {
        "id": 7,
        "name": "No figure those."
      },
      "name": "Reflect American.",
      "created_at": "1994-08-14T03:54:38Z"
    },
    {
      "id": 4,
      "author": {
        "id": 35,
        "name": "Garden order table."
      },
      "name": "Throw minute.",
      "created_at": "1993-12-30T20:50:56Z"
    },
    {
      "id": 5,
      "author": {
        "id": 49,
        "name": "Cause wait health."
      },
      "name": "Congress now build.",
      "created_at": "1977-07-21T17:35:42Z"
    },
    {
      "id": 6,
      "author": {
        "id": 39,
        "name": "Involve section."
      },
      "name": "Activity drop fight.",
      "created_at": "2011-04-21T23:09:54Z"
    },
    {
      "id": 7,
      "author": {
        "id": 44,
        "name": "Cost spring our."
      },
      "name": "Because pattern.",
      "created_at": "2010-01-04T08:21:29Z"
    },
    {
      "id": 8,
      "author": {
        "id": 45,
        "name": "Entire we certainly."
      },
      "name": "Program use feel.",
      "created_at": "1972-11-30T15:49:50Z"
    },
    {
      "id": 9,
      "author": {
        "id": 42,
        "name": "Interest drop."
      },
      "name": "Purpose live might.",
      "created_at": "1987-01-31T16:48:54Z"
    },
    {
      "id": 10,
      "author": {
        "id": 12,
        "name": "Sell data contain."
      },
      "name": "Everyone thing seem.",
      "created_at": "2007-10-19T07:16:34Z"
    }
  ],
  "status": "success"
}

    Ideally, we should be able to get data in 1 single SQL query. Now, let’s write a test case and see if our assumption is correct:

    from django.urls import reverse
from django_seed import Seed

from core.models import Author, Book
from rest_framework.test import APITestCase

seeder = Seed.seeder()


class BooksTestCase(APITestCase):
    def test_list_books(self):
        # Add dummy data to the Author and Book Table
        seeder.add_entity(Author, 5)
        seeder.add_entity(Book, 10)
        seeder.execute()
        # we expect the result in 1 query
        with self.assertNumQueries(1):
            response = self.client.get(reverse("book_list"), format="json")

# test output
$ ./manage.py test
.
.
.
AssertionError: 11 != 1 : 11 queries executed, 1 expected
Captured queries were:
1. SELECT "core_book"."id", "core_book"."name", "core_book"."author_id", "core_book"."created_at" FROM "core_book"
2. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 4 LIMIT 21
3. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 1 LIMIT 21
4. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 4 LIMIT 21
5. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 4 LIMIT 21
6. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 5 LIMIT 21
7. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 5 LIMIT 21
8. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 1 LIMIT 21
9. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 3 LIMIT 21
10. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 3 LIMIT 21
11. SELECT "core_author"."id", "core_author"."name" FROM "core_author" WHERE "core_author"."id" = 5 LIMIT 21

----------------------------------------------------------------------
Ran 1 test in 0.027s

FAILED (failures=1)

    As we see, our test case has failed, and it shows that the number of queries running are 11 and not one. In our test case, we added 10 records in the Book model. The number of queries hitting the database is 1(to fetch books list) + the number of records in the Book model (to fetch author details for each book record). The test output shows the SQL queries executed. 

    The side effects of this can easily go unnoticed while working on a test database with a small number of records. But in production, when the data grows to thousands of records, this can seriously degrade the performance of the database and application.

    Let’s Do It the Right Way

    If we think this in terms of a raw SQL query, this can be achieved with a simple Inner Join operation between the Book and the Author table. We need to do something similar in our Django query. 

    Django provides selected_related and prefetch_related to handle query problems around related objects. 

    • select_related works on forward ForeignKey, OneToOne, and backward OneToOne relationships by creating a database JOIN and fetching the related field data in one single query. ‍
    • prefetch_related works on forward ManyToMany and in reverse, ManyToMany, ForeignKey. prefetch_related does a different query for every relationship and plays out the “joining” in Python. 

    Let’s rewrite the above code using select_related and check the number of queries. 

    We only need to change the queryset in the view. 

    class BookListCreateAPIView(generics.ListCreateAPIView):

   serializer_class = BookSerializer

   def get_queryset(self):
       queryset = Book.objects.select_related("author").all()
       return queryset

    Now, we will rerun the test, and this time it should pass:

    $ ./manage.py test	 
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.024s

OK
Destroying test database for alias 'default'...

    If you are interested in knowing the SQL query executed, here it is:

    >> queryset = Book.objects.select_related("author").all()
>> print(queryset.query)

SELECT "core_book"."id",
       "core_book"."name",
       "core_book"."author_id",
       "core_book"."created_at",
       "core_author"."id",
       "core_author"."name"
FROM "core_book"
         INNER JOIN "core_author" ON ("core_book"."author_id" = "core_author"."id")

    4. Custom Response Format

    It’s a good practice to decide the API endpoints and their request/response payload before starting the actual implementation. If you are the developer, by writing the implementation for the API where the response format is already decided, you can not go with the default response returned by DRF. 

    Let’s assume that, below is the decided format for returning the response: 

    {
  "message": "",
  "errors": [],
  "data": [
    {
      "id": 1,
      "author": {
        "id": 3,
        "name": "Meet teacher."
      },
      "name": "Body society.",
      "created_at": "1973-08-03T02:43:22Z"
    },
    {
      "id": 2,
      "author": {
        "id": 49,
        "name": "Cause wait health."
      },
      "name": "Left next pretty.",
      "created_at": "2000-07-07T03:37:10Z"
    }
  ],
  "status": "success"
}

    We can see that the response format has a message, errors, status, and data attributes. Next, we will see how to write a custom renderer to achieve the above response format. Since the format is in JSON , we override the rest_framework.renderers.JSONRenderer.

    from rest_framework.renderers import JSONRenderer
from rest_framework.views import exception_handler


class CustomJSONRenderer(JSONRenderer):
   def render(self, data, accepted_media_type=None, renderer_context=None):
       # reformat the response
       response_data = {"message": "", "errors": [], "data": data, "status": "success"}
       # call super to render the response
       response = super(CustomJSONRenderer, self).render(
           response_data, accepted_media_type, renderer_context
       )

       return response

    To use this new renderer, we need to add it to  DRF settings:

    REST_FRAMEWORK = {
   "DEFAULT_RENDERER_CLASSES": (
       "core.renderer.CustomJSONRenderer",
       "rest_framework.renderers.JSONRenderer",
       "rest_framework.renderers.BrowsableAPIRenderer",
   )
}

    5. Use the SerializerMethodField to add read-only derived data to the response

    The SerializerMethodField can be used when we want to add some derived data to the object. Consider the same Book listing API. If we want to send an additional property display name—which is the book name in uppercase—we can use the serializer method field as below.

    class BookSerializer(serializers.ModelSerializer):
   author = AuthorSerializer()
   book_display_name= serializers.SerializerMethodField(source="get_book_display_name")

   def get_book_display_name(self, book):
       return book.name.upper()

   class Meta:
       model = Book
       fields = "__all__"

    • The SerializerMethodField takes the source parameter, where we can pass the method name that should be called. 
    • The method gets self and the object as the argument.
    • By default, the DRF source parameter uses get_{field_name}, so in the example above, the source parameter can be omitted, and it will still give the same result.
    book_display_name = serializers.SerializerMethodField()

    6. Use Mixin to Enable/disable Pagination with Query Param

    If you are developing APIs for an internal application and want to support APIs with pagination both enabled and disabled, you can make use of the Mixin below. This allows the caller to use the query parameter “pagination” to enable/disable pagination. This Mixin can be used with the generic views.

    class DynamicPaginationMixin(object):
   """
   Controls pagination enable disable option using query param "pagination".
   If pagination=false is passed in query params, data is returned without pagination
   """
   def paginate_queryset(self, queryset):
   	pagination = self.request.query_params.get("pagination", "true")
    	if bool(pagination):
        	return None

   	return super().paginate_queryset(queryset)

    # Remember to use mixin before the generics
class BookListCreateAPIView(DynamicPaginationMixin, generics.ListCreateAPIView):

	serializer_class = BookSerializer

	def get_queryset(self):
    	    queryset = Book.objects.select_related("author").all()
    	    return queryset

    Conclusion

    This was just a small selection of all the awesome features provided by Django and DRF, so keep exploring. I hope you learned something new today. If you are interested in learning more about serverless deployment of Django Applications, you can refer to our comprehensive guide to deploy serverless, event-driven Python applications using Zappa.

    Further Reading

    1. Django Rest framework Documentation
    2. Django Documentation

  • Building High-performance Apps: A Checklist To Get It Right

    An app is only as good as the problem it solves. But your app’s performance can be extremely critical to its success as well. A slow-loading web app can make users quit and try out an alternative in no time. Testing an app’s performance should thus be an integral part of your development process and not an afterthought.

    In this article, we will talk about how you can proactively monitor and boost your app’s performance as well as fix common issues that are slowing down the performance of your app.

    I’ll use the following tools for this blog.

    • Lighthouse – A performance audit tool, developed by Google
    • Webpack – A JavaScript bundler

    You can find similar tools online, both free and paid. So let’s give our Vue a new Angular perspective to make our apps React faster.

    Performance Metrics

    First, we need to understand which metrics play an important role in determining an app’s performance. Lighthouse helps us calculate a score based on a weighted average of the below metrics:

    1. First Contentful Paint (FCP) – 15%
    2. Speed Index (SI) – 15%
    3. Largest Contentful Paint (LCP) – 25%
    4. Time to Interactive (TTI) – 15%
    5. Total Blocking Time (TBT) – 25%
    6. Cumulative Layout Shift (CLS) – 5%

    By taking the above stats into account, Lighthouse gauges your app’s performance as such:

    • 0 to 49 (slow): Red
    • 50 to 89 (moderate): Orange
    • 90 to 100 (fast): Green

    I would recommend going through Lighthouse performance scoring to learn more. Once you understand Lighthouse, you can audit websites of your choosing.

    I gathered audit scores for a few websites, including Walmart, Zomato, Reddit, and British Airways. Almost all of them had a performance score below 30. A few even secured a single-digit. 

    To attract more customers, businesses fill their apps with many attractive features. But they ignore the most important thing: performance, which degrades with the addition of each such feature.

    As I said earlier, it’s all about the user experience. You can read more about why performance matters and how it impacts the overall experience.

    Now, with that being said, I want to challenge you to conduct a performance test on your favorite app. Let me know if it receives a good score. If not, then don’t feel bad.

    Follow along with me. 

    Let’s get your app fixed!

    Source: Giphy

    Exploring Opportunities

    If you’re still reading this blog, I expect that your app received a low score, or maybe, you’re just curious.

    Source: Giphy

    Whatever the reason, let’s get started.

    Below your scores are the possible opportunities suggested by Lighthouse. Fixing these affects the performance metrics above and eventually boosts your app’s performance. So let’s check them out one-by-one.

    Here are all the possible opportunities listed by Lighthouse:

    1. Eliminate render-blocking resources
    2. Properly size images
    3. Defer offscreen images
    4. Minify CSS & JavaScript
    5. Serve images in the next-gen formats
    6. Enable text compression
    7. Preconnect to required origins
    8. Avoid multiple page redirects
    9. Use video formats for animated content

    ‍

    A few other opportunities won’t be covered in this blog, but they are just an extension of the above points. Feel free to read them under the further reading section.

    Eliminate Render-blocking Resources

    Source: Giphy

    This section lists down all the render-blocking resources. The main goal is to reduce their impact by:

    • removing unnecessary resources,
    • deferring non-critical resources, and
    • in-lining critical resources.

    To do that, we need to understand what a render-blocking resource is.

    Render-blocking resource and how to identify

    As the name suggests, it’s a resource that prevents a browser from rendering processed content. Lighthouse identifies the following as render-blocking resources:

    • A <script> </script>tag in <head> </head>that doesn’t have a defer or async attribute
    • A <link rel=””stylesheet””> tag that doesn’t have a media attribute to match a user’s device or a disabled attribute to hint browser to not download if unnecessary
    • A <link rel=””import””> that doesn’t have an async attribute

    To reduce the impact, you need to identify what’s critical and what’s not. You can read how to identify critical resources using the Chrome dev tool.

    Classify Resources

    Classify resources as critical and non-critical based on the following color code:

    • Green (critical): Needed for the first paint.‍
    • Red (non-critical): Not needed for the first paint but will be needed later.

    Solution

    Now, to eliminate render-blocking resources:

    Extract the critical part into an inline resource and add the correct attributes to the non-critical resources. These attributes will indicate to the browser what to download asynchronously. This can be done manually or by using a JS bundler.

    Webpack users can use the libraries below to do it in a few easy steps:

    • For extracting critical CSS, you can use html-critical-webpack-plugin or critters-webpack-plugin. It’ll generate an inline <style></style> tag in the <head></head> with critical CSS stripped out of the main CSS chunk and preloading the main file
    • For extracting CSS depending on media queries, use media-query-splitting-plugin or media-query-plugin
    • The first paint doesn’t need to be dependent on the JavaScript files. Use lazy loading and code splitting techniques to achieve lazy loading resources (downloading only when requested by the browser). The magic comments in lazy loading make it easy
    • And finally, for the main chunk, vendor chunk, or any other external scripts (included in index.html), you can defer them using script-ext-html-webpack-plugin

    There are many more libraries for inlining CSS and deferring external scripts. Feel free to use as per the use case.

    Use Properly Sized Images

    This section lists all the images used in a page that aren’t properly sized, along with the stats on potential savings for each image.

    How Lighthouse Calculate Oversized Images? 

    Lighthouse calculates potential savings by comparing the rendered size of each image on the page with its actual size. The rendered image varies based on the device pixel ratio. If the size difference is at least 25 KB, the image will fail the audit.

    Solution 

    DO NOT serve images that are larger than their rendered versions! The wasted size just hampers the load time. 

    Alternatively,

    • Use responsive images. With this technique, create multiple versions of the images to be used in the application and serve them depending on the media queries, viewport dimensions, etc
    • Use image CDNs to optimize images. These are like a web service API for transforming images
    • Use vector images, like SVG. These are built on simple primitives and can scale without losing data or change in the file size

    You can resize images online or on your system using tools. Learn how to serve responsive images.

    Learn more about replacing complex icons with SVG. For browsers that don’t support SVG format, here’s A Complete Guide to SVG fallbacks.

    Defer Offscreen Images

    An offscreen image is an image located outside of the visible browser viewport. 

    The audit fails if the page has offscreen images. Lighthouse lists all offscreen or hidden images in your page, along with the potential savings. 

    Solution 

    Load offscreen images only when the user focuses on that part of the viewport. To achieve this, lazy-load these images after loading all critical resources.

    There are many libraries available online that will load images depending on the visible viewport. Feel free to use them as per the use case.

    Minify CSS and JavaScript

    Lighthouse identifies all the CSS and JS files that are not minified. It will list all of them along with potential savings.

    Solution 

    Do as the heading says!

    Source: Giphy

    Minifiers can do it for you. Webpack users can use mini-css-extract-plugin and terser-webpack-plugin for minifying CSS and JS, respectively.

    Serve Images in Next-gen Formats

    Following are the next-gen image formats:

    • Webp
    • JPEG 2000
    • JPEG XR

    The image formats we use regularly (i.e., JPEG and PNG) have inferior compression and quality characteristics compared to next-gen formats. Encoding images in these formats can load your website faster and consume less cellular data.

    Lighthouse converts each image of the older format to Webp format and reports those which ones have potential savings of more than 8 KB.

    Solution 

    Convert all, or at least the images Lighthouse recommends, into the above formats. Use your converted images with the fallback technique below to support all browsers.

    <picture>
  <source type="image/jp2" srcset="my-image.jp2">
  <source type="image/jxr" srcset="my-image.jxr">
  <source type="image/webp" srcset="my-image.webp">
  <source type="image/jpeg" srcset="my-image.jpg">
  <img src="my-image.jpg" alt="">
</picture>

    Enable Text Compression

    Source: Giphy

    This technique of compressing the original textual information uses compression algorithms to find repeated sequences and replace them with shorter representations. It’s done to further minimize the total network bytes.

    Lighthouse lists all the text-based resources that are not compressed. 

    It computes the potential savings by identifying text-based resources that do not include a content-encoding header set to br, gzip or deflate and compresses each of them with gzip.

    If the potential compression savings is more than 10% of the original size, then the file fails the audit.

    Solution

    Webpack users can use compression-webpack-plugin for text compression. 

    The best part about this plugin is that it supports Google’s Brotli compression algorithm which is superior to gzip. Alternatively, you can also use brotli-webpack-plugin. All you need to do is configure your server to return Content-Encoding as br.

    Brotli compresses faster than gzip and produces smaller files (up to 20% smaller). As of June 2020, Brotli is supported by all major browsers except Safari on iOS and desktop and Internet Explorer.

    Don’t worry. You can still use gzip as a fallback.

    Preconnect to Required Origins

    This section lists all the key fetch requests that are not yet prioritized using <link rel=””preconnect””>.

    Establishing connections often involves significant time, especially when it comes to secure connections. It encounters DNS lookups, redirects, and several round trips to the final server handling the user’s request.

    Solution

    Establish an early connection to required origins. Doing so will improve the user experience without affecting bandwidth usage. 

    To achieve this connection, use preconnect or dns-prefetch. This informs the browser that the app wants to establish a connection to the third-party origin as soon as possible.

    Use preconnect for most critical connections. For non-critical connections, use dns-prefetch. Check out the browser support for preconnect. You can use dns-prefetch as the fallback.

    Avoid Multiple Page Redirects

    Source: Giphy

    This section focuses on requesting resources that have been redirected multiple times. One must avoid multiple redirects on the final landing pages.

    A browser encounters this response from a server in case of HTTP-redirect:

    HTTP/1.1 301 Moved Permanently
Location: /path/to/new/location

    A typical example of a redirect looks like this:
    ‍
    example.com → www.example.com → m.example.com – very slow mobile experience.

    This eventually makes your page load more slowly.

    Solution

    Don’t leave them hanging!

    Source: Giphy

    Point all your flagged resources to their current location. It’ll help you optimize your pages’ Critical Rendering Path.

    Use Video Formats for Animated Content

    This section lists all the animated GIFs on your page, along with the potential savings. 

    Large GIFs are inefficient when delivering animated content. You can save a significant amount of bandwidth by using videos over GIFs.

    Solution

    Consider using MPEG4 or WebM videos instead of GIFs. Many tools can convert a GIF into a video, such as FFmpeg.

    Use the code below to replicate a GIF’s behavior using MPEG4 and WebM. It’ll be played silent and automatically in an endless loop, just like a GIF. The code ensures that the unsupported format has a fallback.

    <video autoplay loop muted playsinline>  
  <source src="my-funny-animation.webm" type="video/webm">
  <source src="my-funny-animation.mp4" type="video/mp4">
</video>

    Note: Do not use video formats for a small batch of GIF animations. It’s not worth doing it. It comes in handy when your website makes heavy use of animated content.

    Final Thoughts

    I found a great result in my app’s performance after trying out the techniques above.

    Source: Giphy

    While they may not all fit your app, try it and see what works and what doesn’t. I have compiled a list of some resources that will help you enhance performance. Hopefully, they help.

    Do share your starting and final audit scores with me.

    Happy optimized coding!

    Source: Giphy

    Further Reading

    Learn more – web.dev

    Other opportunities to explore:

    1. Remove unused CSS
    2. Efficiently encode images
    3. Reduce server response times (TTFB)
    4. Preload key requests
    5. Reduce the impact of third-party code

  • Building Your First AWS Serverless Application? Here’s Everything You Need to Know

    A serverless architecture is a way to implement and run applications and services or micro-services without need to manage infrastructure. Your application still runs on servers, but all the servers management is done by AWS. Now we don’t need to provision, scale or maintain servers to run our applications, databases and storage systems. Services which are developed by developers who don’t let developers build application from scratch.

    Why Serverless

    1. More focus on development rather than managing servers.
    2. Cost Effective.
    3. Application which scales automatically.
    4. Quick application setup.

    Services For ServerLess

    For implementing serverless architecture there are multiple services which are provided by cloud partners though we will be exploring most of the services from AWS. Following are the services which we can use depending on the application requirement.

    1. Lambda: It is used to write business logic / schedulers / functions.
    2. S3: It is mostly used for storing objects but it also gives the privilege to host WebApps. You can host a static website on S3.
    3. API Gateway: It is used for creating, publishing, maintaining, monitoring and securing REST and WebSocket APIs at any scale.
    4. Cognito: It provides authentication, authorization & user management for your web and mobile apps. Your users can sign in directly sign in with a username and password or through third parties such as Facebook, Amazon or Google.
    5. DynamoDB: It is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.

    Three-tier Serverless Architecture

    So, let’s take a use case in which you want to develop a three tier serverless application. The three tier architecture is a popular pattern for user facing applications, The tiers that comprise the architecture include the presentation tier, the logic tier and the data tier. The presentation tier represents the component that users directly interact with web page / mobile app UI. The logic tier contains the code required to translate user action at the presentation tier to the functionality that drives the application’s behaviour. The data tier consists of your storage media (databases, file systems, object stores) that holds the data relevant to the application. Figure shows the simple three-tier application.

     Figure: Simple Three-Tier Architectural Pattern

    Presentation Tier

    The presentation tier of the three tier represents the View part of the application. Here you can use S3 to host static website. On a static website, individual web pages include static content and they also contain client side scripting.

    The following is a quick procedure to configure an Amazon S3 bucket for static website hosting in the S3 console.

    To configure an S3 bucket for static website hosting

    1. Log in to the AWS Management Console and open the S3 console at

    2. In the Bucket name list, choose the name of the bucket that you want to enable static website hosting for.

    3. Choose Properties.

    4. Choose Static Website Hosting

    Once you enable your bucket for static website hosting, browsers can access all of your content through the Amazon S3 website endpoint for your bucket.

    5. Choose Use this bucket to host.

    A. For Index Document, type the name of your index document, which is typically named index.html. When you configure a S3 bucket for website hosting, you must specify an index document, which will be returned by S3 when requests are made to the root domain or any of the subfolders.

    B. (Optional) For 4XX errors, you can optionally provide your own custom error document that provides additional guidance for your users. Type the name of the file that contains the custom error document. If an error occurs, S3 returns an error document.

    C. (Optional) If you want to give advanced redirection rules, In the edit redirection rule text box, you have to XML to describe the rule.
    E.g.

    <RoutingRules>
    <RoutingRule>
        <Condition>
            <HttpErrorCodeReturnedEquals>403</HttpErrorCodeReturnedEquals>
        </Condition>
        <Redirect>
            <HostName>mywebsite.com</HostName>
            <ReplaceKeyPrefixWith>notfound/</ReplaceKeyPrefixWith>
        </Redirect>
    </RoutingRule>
</RoutingRules>

    6. Choose Save

    7. Add a bucket policy to the website bucket that grants access to the object in the S3 bucket for everyone. You must make the objects that you want to serve publicly readable, when you configure a S3 bucket as a website. To do so, you write a bucket policy that grants everyone S3:GetObject permission. The following bucket policy grants everyone access to the objects in the example-bucket bucket.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::example-bucket/*"
            ]
        }
    ]
}

    Note: If you choose Disable Website Hosting, S3 removes the website configuration from the bucket, so that the bucket no longer accessible from the website endpoint, but the bucket is still available at the REST endpoint.

    Logic Tier

    The logic tier represents the brains of the application. Here the two core services for serverless will be used i.e. API Gateway and Lambda to form your logic tier can be so revolutionary. The feature of the 2 services allow you to build a serverless production application which is highly scalable, available and secure. Your application could use number of servers, however by leveraging this pattern you do not have to manage a single one. In addition, by using these managed services together you get following benefits:

    1. No operating system to choose, secure or manage.
    2. No servers to right size, monitor.
    3. No risk to your cost by over-provisioning.
    4. No Risk to your performance by under-provisioning.

    API Gateway

    API Gateway is a fully managed service for defining, deploying and maintaining APIs. Anyone can integrate with the APIs using standard HTTPS requests. However, it has specific features and qualities that result it being an edge for your logic tier.

    Integration with Lambda

    API Gateway gives your application a simple way to leverage the innovation of AWS lambda directly (HTTPS Requests). API Gateway forms the bridge that connects your presentation tier and the functions you write in Lambda. After defining the client / server relationship using your API, the contents of the client’s HTTPS requests are passed to Lambda function for execution. The content include request metadata, request headers and the request body.

    API Performance Across the Globe

    Each deployment of API Gateway includes an Amazon CloudFront distribution under the covers. Amazon CloudFront is a content delivery web service that used Amazon’s global network of edge locations as connection points for clients integrating with API. This helps drive down the total response time latency of your API. Through its use of multiple edge locations across the world, Amazon CloudFront also provides you capabilities to combat distributed denial of service (DDoS) attack scenarios.

    You can improve the performance of specific API requests by using API Gateway to store responses in an optional in-memory cache. This not only provides performance benefits for repeated API requests, but is also reduces backend executions, which can reduce overall cost.

    Let’s dive into each step

    1. Create Lambda Function
    Login to Aws Console and head over to Lambda Service and Click on “Create A Function”

    A. Choose first option “Author from scratch”
    B. Enter Function Name
    C. Select Runtime e.g. Python 2.7
    D. Click on “Create Function”

    As your function is ready, you can see your basic function will get generated in language you choose to write.
    E.g.

    import json

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

    2. Testing Lambda Function

    Click on “Test” button at the top right corner where we need to configure test event. As we are not sending any events, just give event a name, for example, “Hello World” template as it is and “Create” it.

    Now, when you hit the “Test” button again, it runs through testing the function we created earlier and returns the configured value.

    Create & Configure API Gateway connecting to Lambda

    We are done with creating lambda functions but how to invoke function from outside world ? We need endpoint, right ?

    Go to API Gateway & click on “Get Started” and agree on creating an Example API but we will not use that API we will create “New API”. Give it a name by keeping “Endpoint Type” regional for now.

    Create the API and you will go on the page “resources” page of the created API Gateway. Go through the following steps:

    A. Click on the “Actions”, then click on “Create Method”. Select Get method for our function. Then, “Tick Mark” on the right side of “GET” to set it up.
    B. Choose “Lambda Function” as integration type.
    C. Choose the region where we created earlier.
    D. Write the name of Lambda Function we created
    E. Save the method where it will ask you for confirmation of “Add Permission to Lambda Function”. Agree to that & that is done.
    F. Now, we can test our setup. Click on “Test” to run API. It should give the response text we had on the lambda test screen.

    Now, to get endpoint. We need to deploy the API. On the Actions dropdown, click on Deploy API under API Actions. Fill in the details of deployment and hit Deploy.

    After that, we will get our HTTPS endpoint.

    On the above screen you can see the things like cache settings, throttling, logging which can be configured. Save the changes and browse the invoke URL from which we will get the response which was earlier getting from Lambda. So, here is our logic tier of serverless application is to be done.

    Data Tier

    By using Lambda as your logic tier, you have a number of data storage options for your data tier. These options fall into broad categories: Amazon VPC hosted data stores and IAM-enabled data stores. Lambda has the ability to integrate with both securely.

    Amazon VPC Hosted Data Stores

    1. Amazon RDS
    2. Amazon ElasticCache
    3. Amazon Redshift

    IAM-Enabled Data Stores

    1. Amazon DynamoDB
    2. Amazon S3
    3. Amazon ElasticSearch Service

    You can use any of those for storage purpose, But DynamoDB is one of best suited for ServerLess application.

    Why DynamoDB ?

    1. It is NoSQL DB, also that is fully managed by AWS.
    2. It provides fast & prectable performance with seamless scalability.
    3. DynamoDB lets you offload the administrative burden of operating and scaling a distributed system.
    4. It offers encryption at rest, which eliminates the operational burden and complexity involved in protecting sensitive data.
    5. You can scale up/down your tables throughput capacity without downtime/performance degradation.
    6. It provides On-Demand backups as well as enable point in time recovery for your DynamoDB tables.
    7. DynamoDB allows you to delete expired items from table automatically to help you reduce storage usage and the cost of storing data that is no longer relevant.

    Following is the sample script for DynamoDB with Python which you can use with lambda.

    from __future__ import print_function # Python 2/3 compatibility
import boto3
import json
import decimal
from boto3.dynamodb.conditions import Key, Attr
from botocore.exceptions import ClientError

# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
            else:
                return int(o)
        return super(DecimalEncoder, self).default(o)

dynamodb = boto3.resource("dynamodb", region_name='us-west-2', endpoint_url="http://localhost:8000")

table = dynamodb.Table('Movies')

title = "The Big New Movie"
year = 2015

try:
    response = table.get_item(
        Key={
            'year': year,
            'title': title
        }
    )
except ClientError as e:
    print(e.response['Error']['Message'])
else:
    item = response['Item']
    print("GetItem succeeded:")
    print(json.dumps(item, indent=4, cls=DecimalEncoder))

    Note: To run the above script successfully you need to attach policy to your role for lambda. So in this case you need to attach policy for DynamoDB operations to take place & for CloudWatch if required to store your logs. Following is the policy which you can attach to your role for DB executions.

    {
	"Version": "2012-10-17",
	"Statement": [{
			"Effect": "Allow",
			"Action": [
				"dynamodb:BatchGetItem",
				"dynamodb:GetItem",
				"dynamodb:Query",
				"dynamodb:Scan",
				"dynamodb:BatchWriteItem",
				"dynamodb:PutItem",
				"dynamodb:UpdateItem"
			],
			"Resource": "arn:aws:dynamodb:eu-west-1:123456789012:table/SampleTable"
		},
		{
			"Effect": "Allow",
			"Action": [
				"logs:CreateLogStream",
				"logs:PutLogEvents"
			],
			"Resource": "arn:aws:logs:eu-west-1:123456789012:*"
		},
		{
			"Effect": "Allow",
			"Action": "logs:CreateLogGroup",
			"Resource": "*"
		}
	]
}

    Sample Architecture Patterns

    You can implement the following popular architecture patterns using API Gateway & Lambda as your logic tier, Amazon S3 for presentation tier, DynamoDB as your data tier. For each example, we will only use AWS Service that do not require users to manage their own infrastructure.

    Mobile Backend

    1. Presentation Tier: A mobile application running on each user’s smartphone.

    2. Logic Tier: API Gateway & Lambda. The logic tier is globally distributed by the Amazon CloudFront distribution created as part of each API Gateway each API. A set of lambda functions can be specific to user / device identity management and authentication & managed by Amazon Cognito, which provides integration with IAM for temporary user access credentials as well as with popular third party identity providers. Other Lambda functions can define core business logic for your Mobile Back End.

    3. Data Tier: The various data storage services can be leveraged as needed; options are given above in data tier.

    Amazon S3 Hosted Website

    1. Presentation Tier: Static website content hosted on S3, distributed by Amazon CLoudFront. Hosting static website content on S3 is a cost effective alternative to hosting content on server-based infrastructure. However, for a website to contain rich feature, the static content often must integrate with a dynamic back end.

    2. Logic Tier: API Gateway & Lambda, static web content hosted in S3 can directly integrate with API Gateway, which can be CORS complaint.

    3. Data Tier: The various data storage services can be leveraged based on your requirement.

    ServerLess Costing

    At the top of the AWS invoice, we can see the total costing of AWS Services. The bill was processed for 2.1 million API request & all of the infrastructure required to support them.

    Following is the list of services with their costing.

    Note: You can get your costing done from AWS Calculator using following links;

    1. https://calculator.s3.amazonaws.com/index.html
    2. AWS Pricing Calculator

    Conclusion

    The three-tier architecture pattern encourages the best practice of creating application component that are easy to maintain, develop, decoupled & scalable. Serverless Application services varies based on the requirements over development.

  • Learn How to Quickly Setup Istio Using GKE and its Applications

    In this blog, we will try to understand Istio and its YAML configurations. You will also learn why Istio is great for managing traffic and how to set it up using Google Kubernetes Engine (GKE). I’ve also shed some light on deploying Istio in various environments and applications like intelligent routing, traffic shifting, injecting delays, and testing the resiliency of your application.

    What is Istio?

    The Istio’s website says it is “An open platform to connect, manage, and secure microservices”.

    As a network of microservices known as ‘Service Mesh’ grows in size and complexity, it can become tougher to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring, and often more complex operational requirements such as A/B testing, canary releases, rate limiting, access control, and end-to-end authentication. Istio claims that it provides complete end to end solution to these problems.

    Why Istio?

    • Provides automatic load balancing for various protocols like HTTP, gRPC, WebSocket, and TCP traffic. It means you can cater to the needs of web services and also frameworks like Tensorflow (it uses gRPC).
    • To control the flow of traffic and API calls between services, make calls more reliable, and make the network more robust in the face of adverse conditions.
    • To gain understanding of the dependencies between services and the nature and flow of traffic between them, providing the ability to quickly identify issues etc.

    Let’s explore the architecture of Istio.

    Istio’s service mesh is split logically into two components:

    1. Data plane – set of intelligent proxies (Envoy) deployed as sidecars to the microservice they control communications between microservices.
    2. Control plane – manages and configures proxies to route traffic. It also enforces policies.

    Envoy – Istio uses an extended version of envoy (L7 proxy and communication bus designed for large modern service-oriented architectures) written in C++. It manages inbound and outbound traffic for service mesh.

    Enough of theory, now let us setup Istio to see things in action. A notable point is that Istio is pretty fast. It’s written in Go and adds a very tiny overhead to your system.

    Setup Istio on GKE

    You can either setup Istio via command line or via UI. We have used command line installation for this blog.

    Sample Book Review Application

    Following this link, you can easily

    The Bookinfo application is broken into four separate microservices:

    • productpage. The productpage microservice calls the details and reviews microservices to populate the page.
    • details. The details microservice contains book information.
    • reviews. The reviews microservice contains book reviews. It also calls the ratings microservice.
    • ratings. The ratings microservice contains book ranking information that accompanies a book review.

    There are 3 versions of the reviews microservice:

    • Version v1 doesn’t call the ratings service.
    • Version v2 calls the ratings service and displays each rating as 1 to 5 black stars.
    • Version v3 calls the ratings service and displays each rating as 1 to 5 red stars.

    The end-to-end architecture of the application is shown below.

    If everything goes well, You will have a web app like this (served at http://GATEWAY_URL/productpage)

    Let’s take a case where 50% of traffic is routed to v1 and the remaining 50% to v3.

    This is how the config file looks like (/path/to/istio-0.2.12/samples/bookinfo/kube/route-rule-reviews-50-v3.yaml) 

    apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: reviews-default
spec:
  destination:
    name: reviews
  precedence: 1
  route:
  - labels:
      version: v1
    weight: 50
  - labels:
      version: v3
    weight: 50

    Let’s try to understand the config file above.

    Istio provides a simple Domain-specific language (DSL) to control how API calls and layer-4 traffic flow across various services in the application deployment.

    In the above configuration, we are trying to Add a “Route Rule”. It means we will be routing the traffic coming to destinations. The destination is the name of the service to which the traffic is being routed. The route labels identify the specific service instances that will receive traffic.

    In this Kubernetes deployment of Istio, the route label “version: v1” and “version: v3” indicates that only pods containing the label “version: v1” and “version: v3” will receive 50% traffic each.

    Now multiple route rules could be applied to the same destination. The order of evaluation of rules corresponding to a given destination, when there is more than one, can be specified by setting the precedence field of the rule.

    The precedence field is an optional integer value, 0 by default. Rules with higher precedence values are evaluated first. If there is more than one rule with the same precedence value the order of evaluation is undefined.

    When is precedence useful? Whenever the routing story for a particular service is purely weight based, it can be specified in a single rule.

    Once a rule is found that applies to the incoming request, it will be executed and the rule-evaluation process will terminate. That’s why it’s very important to carefully consider the priorities of each rule when there is more than one.

    In short, it means route label “version: v1” is given preference over route label “version: v3”.

    Intelligent Routing Using Istio

    We will demonstrate an example in which we will be aiming to get more control over routing the traffic coming to our app. Before reading ahead, make sure that you have installed Istio and book review application.

    First, we will set a default version for all microservices.

    > kubectl create -f samples/bookinfo/kube/route-rule-all-v1.yaml

    Then wait a few seconds for the rules to propagate to all pods before attempting to access the application. This will set the default route to v1 version (which doesn’t call rating service). Now we want a specific user, say Velotio, to see v2 version. We write a yaml (test-velotio.yaml) file.

    apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: test-velotio
  namespace: default
  ...
spec:
  destination:
    name: reviews
  match:
    request:
      headers:
        cookie:
          regex: ^(.*?;)?(user=velotio)(;.*)?$
  precedence: 2
  route:
  - labels:
      version: v2

    We then set this rule

    > kubectl create -f path/to/test-velotio.yml

    Now if any other user logs in it won’t see any ratings (it will see v1 version) but when “velotio” user logs in it will see v2 version!

    This is how we can intelligently do content-based routing. We used Istio to send 100% of the traffic to the v1 version of each of the Bookinfo services. You then set a rule to selectively send traffic to version v2 of the reviews service based on a header (i.e., a user cookie) in a request.

    Traffic Shifting

    Now Let’s take a case in which we have to shift traffic from an old service to a new service.

    We can use Istio to gradually transfer traffic from one microservice to another one. For example, we can move 10, 20, 25..100% of traffic. Here for simplicity of the blog, we will move traffic from reviews:v1 to reviews:v3 in two steps 40% to 100%.

    First, we set the default version v1.

    > kubectl create -f samples/bookinfo/kube/route-rule-all-v1.yaml

    We write a yaml file route-rule-reviews-40-v3.yaml

    apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: reviews-default
  namespace: default
spec:
  destination:
    name: reviews
  precedence: 1
  route:
  - labels:
      version: v1
    weight: 60
  - labels:
      version: v3
    weight: 40

    Then we apply a new rule.

    > kubectl create -f path/to/route-rule-reviews-40-v3.yaml

    Now, Refresh the productpage in your browser and you should now see red colored star ratings approximately 40% of the time. Once that is stable, we transfer all the traffic to v3.

    > istioctl replace -f samples/bookinfo/kube/route-rule-reviews-v3.yaml

    Inject Delays and Test the Resiliency of Your Application

    Here we will check fault injection using HTTP delay. To test our Bookinfo application microservices for resiliency, we will inject a 7s delay between the reviews:v2 and ratings microservices, for user “Jason”. Since the reviews:v2 service has a 10s timeout for its calls to the ratings service, we expect the end-to-end flow to continue without any errors.

    > istioctl create -f samples/bookinfo/kube/route-rule-ratings-test-delay.yaml

    Now we check if the rule was applied correctly,

    > istioctl get routerule ratings-test-delay -o yaml

    Now we allow several seconds to account for rule propagation delay to all pods. Log in as user “Jason”. If the application’s front page was set to correctly handle delays, we expect it to load within approximately 7 seconds.

    Conclusion

    In this blog we only explored the routing capabilities of Istio. We found Istio to give us good amount of control over routing, fault injection etc in microservices. Istio has a lot more to offer like load balancing and security. We encourage you guys to toy around with Istio and tell us about your experiences.

    Happy Coding!