These days, we see that most software development is moving towards serverless architecture, and that’s no surprise. Almost all top cloud service providers have serverless services that follow a pay-as-you-go model. This way, consumers don’t have to pay for any unused resources. Also, there’s no need to worry about procuring dedicated servers, network/hardware management, operating system security updates, etc.
Unfortunately, for cloud developers, serverless tools don’t provide auto-deploy services for updating local environments. This is still a headache. The developer must deploy and test changes manually. Web app projects using Node or Django have a watcher on the development environment during app bundling on their respective server runs. Thus, when changes happen in the code directory, the server automatically restarts with these new changes, and the developer can check if the changes are working as expected.
In this blog, we will talk about automating serverless application deployment by changing the local codebase. We are using AWS as a cloud provider and primarily focusing on lambda to demonstrate the functionality.
Prerequisites:
- This article uses AWS, so command and programming access are necessary.
- This article is written with deployment to AWS in mind, so AWS credentials are needed to make changes in the Stack. In the case of other cloud providers, we would require that provider’s command-line access.
- We are using a serverless application framework for deployment. (This example will also work for other tools like Zappa.) So, some serverless context would be required.
Before development, let’s divide the problem statement into sub-tasks and build them one step at a time.
Problem Statement
Create a codebase watcher service that would trigger either a stack update on AWS or run a local test. By doing this, developers would not have to worry about manual deployment on the cloud provider. This service needs to keep an eye on the code and generate events when an update/modify/copy/delete occurs in the given codebase.
Solution
First, to watch the codebase, we need logic that acts as a trigger and notifies when underlining files changes. For this, there are already packages present in different programming languages. In this example, we are using ‘python watchdog.’
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
CODE_PATH = "<codebase path>"
class WatchMyCodebase:
# Set the directory on watch
def __init__(self):
self.observer = Observer()
def run(self):
event_handler = EventHandler()
# recursive flag decides if watcher should collect changes in CODE_PATH directory tree.
self.observer.schedule(event_handler, CODE_PATH, recursive=True)
self.observer.start()
self.observer.join()
class EventHandler(FileSystemEventHandler):
"""Handle events generated by Watchdog Observer"""
@classmethod
def on_any_event(cls, event):
if event.is_directory:
"""Ignore directory level events, like creating new empty directory etc.."""
return None
elif event.event_type == 'modified':
print("file under codebase directory is modified...")
if __name__ == '__main__':
watch = WatchMyCodebase()
watch.run()Here, the on_any_event() class method gets called on any updates in the mentioned directory, and we need to add deployment logic here. However, we can’t just deploy once it receives a notification from the watcher because modern IDEs save files as soon as the user changes them. And if we add logic that deploys on every change, then most of the time, it will deploy half-complete services.
To handle this, we must add some timeout before deploying the service.
Here, the program will wait for some time after the file is changed. And if it finds that, for some time, there have been no new changes in the codebase, it will deploy the service.
import time
import subprocess
import threading
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
valid_events = ['created', 'modified', 'deleted', 'moved']
DEPLOY_AFTER_CHANGE_THRESHOLD = 300
STAGE_NAME = ""
CODE_PATH = "<codebase path>"
def deploy_env():
process = subprocess.Popen(['sls', 'deploy', '--stage', STAGE_NAME, '-v'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
print(stdout, stderr)
def deploy_service_on_change():
while True:
if EventHandler.last_update_time and (int(time.time() - EventHandler.last_update_time) > DEPLOY_AFTER_CHANGE_THRESHOLD):
EventHandler.last_update_time = None
deploy_env()
time.sleep(5)
def start_interval_watcher_thread():
interval_watcher_thread = threading.Thread(target=deploy_service_on_change)
interval_watcher_thread.start()
class WatchMyCodebase:
# Set the directory on watch
def __init__(self):
self.observer = Observer()
def run(self):
event_handler = EventHandler()
self.observer.schedule(event_handler, CODE_PATH, recursive=True)
self.observer.start()
self.observer.join()
class EventHandler(FileSystemEventHandler):
"""Handle events generated by Watchdog Observer"""
last_update_time = None
@classmethod
def on_any_event(cls, event):
if event.is_directory:
"""Ignore directory level events, like creating new empty directory etc.."""
return None
elif event.event_type in valid_events and '.serverless' not in event.src_path:
# Ignore events related to changes in .serverless directory, serverless creates few temp file while deploy
cls.last_update_time = time.time()
if __name__ == '__main__':
start_interval_watcher_thread()
watch = WatchMyCodebase()
watch.run()The specified valid_events acts as a filter to deploy, and we are only considering these events and acting upon them.
Moreover, to add a delay after file changes and ensure that there are no new changes, we added interval_watcher_thread. This checks the difference between current and last directory update time, and if it’s greater than the specified threshold, we deploy serverless resources.
def deploy_service_on_change():
while True:
if EventHandler.last_update_time and (int(time.time() - EventHandler.last_update_time) > DEPLOY_AFTER_CHANGE_SEC):
EventHandler.last_update_time = None
deploy_env()
time.sleep(5)
def start_interval_watcher_thread():
interval_watcher_thread = threading.Thread(target=deploy_service_on_change)
interval_watcher_thread.start()Here, the sleep time in deploy_service_on_change is important. It will prevent the program from consuming more CPU cycles to check whether the condition to deploy serverless is satisfied. Also, too much delay would cause more delay in the deployment than the specified value of DEPLOY_AFTER_CHANGE_THRESHOLD.
Note: With programming languages like Golang, and its features like goroutine and channels, we can build an even more efficient application—or even with Python with the help of thread signals.
Let’s build one lambda function that automatically deploys on a change. Let’s also be a little lazy and develop a basic python lambda that takes a number as an input and returns its factorial value.
import math
def lambda_handler(event, context):
"""
Handler for get factorial
"""
number = event['number']
return math.factorial(number)We are using a serverless application framework, so to deploy this lambda, we need a serverless.yml file that specifies stack details like execution environment, cloud provider, environment variables, etc. More parameters are listed in this guide.
service: get-factorial
provider:
name: aws
runtime: python3.7
functions:
get_factorial:
handler: handler.lambda_handlerWe need to keep both handler.py and serverless.yml in the same folder, or we need to update the function handler in serverless.yml.
We can deploy it manually using this serverless command:
sls deploy --stage production -vNote: Before deploying, export AWS credentials.
The above command deployed a stack using cloud formation:
- –stage is how to specify the environment where the stack should be deployed. Like any other software project, it can have stage names such as production, dev, test, etc.
- -v specifies verbose.
To auto-deploy changes from now on, we can use the watcher.
Start the watcher with this command:
python3 auto_deploy_sls.pyThis will run continuously and keep an eye on the codebase directory, and if any changes are detected, it will deploy them. We can customize this to some extent, like post-deploy, so it can run test cases against a new stack.
If you are worried about network traffic when the stack has lots of dependencies, using an actual cloud provider for testing might increase billing. However, we can easily fix this by using serverless local development.
Here is a serverless blog that specifies local development and testing of a cloudformation stack. It emulates cloud behavior on the local setup, so there’s no need to worry about cloud service billing.
One great upgrade supports complex directory structure.
In the above example, we are assuming that only one single directory is present, so it’s fine to deploy using the command:
sls deploy --stage production -vBut in some projects, one might have multiple stacks present in the codebase at different hierarchies. Consider the below example: We have three different lambdas, so updating in the `check-prime` directory requires updating only that lambda and not the others.
├── check-prime
│ ├── handler.py
│ └── serverless.yml
├── get-factorial
│ ├── handler.py
│ └── serverless.yml
└── get-factors
├── handler.py
└── serverless.ymlThe above can be achieved in on_any_event(). By using the variable event.src_path, we can learn the file path that received the event.
Now, deployment command changes to:
cd <updated_directory> && sls deploy --stage <your-stage> -vThis will deploy only an updated stack.
Conclusion
We learned that even if serverless deployment is a manual task, it can be automated with the help of Watchdog for better developer workflow.
With the help of serverless local development, we can test changes as we are making them without needing an explicit deployment to the cloud environment manually to test all the changes being made.
We hope this helps you improve your serverless development experience and close the loop faster.
Related Articles
1. To Go Serverless Or Not Is The Question
2. Building Your First AWS Serverless Application? Here’s Everything You Need to Know
