My blog

Category: Type

  • Unlocking the Potential of Knowledge Graphs: Exploring Graph Databases

    There is a growing demand for data-driven insights to help businesses make better decisions and stay competitive. To meet this need, organizations are turning to knowledge graphs as a way to access and analyze complex data sets. In this blog post, I will discuss what knowledge graphs are, what graph databases are, how they differ from hierarchical databases, the benefits of graphical representation of data, and more. Lastly, we’ll discuss some of the challenges of graph databases and how they can be overcome.

    What Is a Knowledge Graph?

    A knowledge graph is a visual representation of data or knowledge. In order to make the relationships between various types of facts and data easy to see and understand, facts and data are organized into a graph structure. A knowledge graph typically consists of nodes, which stand in for entities like people or objects, and edges, which stand in for the relationships among these entities.

    Each node in a knowledge graph has characteristics and attributes that describe it. For instance, the node of a person might contain properties like name, age, and occupation. Edges between nodes reveal information about their connections. This makes knowledge graphs a powerful tool for representing and understanding data.

    Benefits of a Knowledge Graph

    There are a number of benefits to using knowledge graphs. 

    • Knowledge graphs(KG) provide a visual representation of data that can be easily understood. This makes it easier to quickly identify patterns, and correlations. 
    • Additionally, knowledge graphs make it simple to locate linkage data by allowing us to quickly access a particular node and obtain all of its child information.
    • These graphs are highly scalable, meaning they can support huge volumes of data. This makes them ideal for applications such as artificial intelligence (AI) and machine learning (ML).
    • Finally, knowledge graphs can be used to connect various types of data, including text, images, and videos, in addition to plain text. This makes them a great tool for data mining and analysis.‍

    ‍What are Graph Databases?

    Graph databases are used to store and manage data in the form of a graph. Unlike traditional databases, they offer a more flexible representation of data using nodes, edges, and properties. Graph databases are designed to support queries that require traversing relationships between different types of data.

    Graph databases are well-suited for applications that require complex data relationships, such as AI and ML. They are also more efficient than traditional databases in queries that involve intricate data relationships, as they can quickly process data without having to make multiple queries.

    Source: Techcrunch

    Comparing Graph Databases to Hierarchical Databases

    It is important to understand the differences between graph databases and hierarchical databases. But first, what is a hierarchical database? Hierarchical databases are structured in a tree-like form, with each record in the database linked to one or more other records. This structure makes hierarchical databases ideal for storing data that is organized in a hierarchical manner, such as an organizational chart. However, hierarchical databases are less efficient at handling complex data relationships. To understand with an example, suppose we have an organization with a CEO at the top, followed by several vice presidents, who are in turn responsible for several managers, who are responsible for teams of employees.

     

    In a hierarchical database, this structure would be represented as a tree, with the CEO at the root, and each level of the organization represented by a different level of the tree. For example:

     

    In a graph database, this same structure would be represented as a graph, with each node representing an entity (e.g., a person), and each edge representing a relationship (e.g., reporting to). For example:

    (Vice President A) — reports_to –> (CEO)

    (Vice President B) — reports_to –> (CEO)

    (Vice President A) — manages –> (Manager A1)

    (Vice President B) — manages –> (Manager B1)

    (Manager A1) — manages –> (Employee A1.1)

    (Manager B1) — manages –> (Employee B1.1)

    (Manager B1) — manages –> (Employee B1.2)‍

     

    As you can see, in a graph database, the relationships between entities are explicit and can be easily queried and traversed. In a hierarchical database, the relationships are implicit and can be more difficult to work with if the hierarchy becomes more complex. Hence the reason graph databases are better suited for complex data relationships is that it gives them the flexibility to easily store and query data.

    Creating a Knowledge Graph from Scratch

    We will now understand how to create a knowledge graph using an example below where we’ll use a simple XML file that contains information about some movies, and we’ll use an XSLT stylesheet to transform the XML data into RDF format along with some python libraries to help us in the overall process.

    Let’s consider an XML file having movie information:

    <movies>
  <movie id="tt0083658">
    <title>Blade Runner</title>
    <year>1982</year>
    <director rid="12341">Ridley Scott</director>
    <genre>Action</genre>
  </movie>

  <movie id="tt0087469">
    <title>Top Gun</title>
    <year>1986</year>
    <director rid="65217">Tony Hank</director>
    <genre>Thriller</genre>
  </movie>
</movies>

    As discussed, to convert this data into a knowledge graph, we will be using an XSL file, now a question may arise that what is an XSL file? Well, XSL files are stylesheet documents that are used to transform XML data. To explore more on XSL, visit here, but don’t worry as we will be starting from scratch. 

    Moving ahead, we also need to know that to convert any data into graph data, we need to use an ontology; there are many ontologies available, like OWL ontology or EBUCore ontology. But what is an ontology? Well, in the context of knowledge graphs, an ontology is a formal specification of the relationships and constraints that exist within a specific domain or knowledge domain. It provides a vocabulary and a set of rules for representing and sharing knowledge, allowing machines to reason about the data they are working with. EBUCore is an ontology developed by the European Broadcasting Union (EBU) to provide a standardized metadata model for the broadcasting industry (OTT platforms, media companies, etc.). Further references on EBUCore can be found here.

    We will be using the below XSL for transforming the above XML with movie info.

    <?xml version="1.0"?>
<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
                xmlns:ebucore="http://www.ebu.ch/metadata/ontologies/ebucore/ebucore#"
                >
    <xsl:template match="movies">
        <rdf:RDF>
            <xsl:apply-templates select="movie"/>
        </rdf:RDF>
    </xsl:template>

    <xsl:template match="movie">
        <ebucore:Feature>

            <xsl:template match="title">
                <ebucore:title>
                    <xsl:value-of select="."/>
                </ebucore:title>
            </xsl:template>

            <xsl:template match="year">
                <ebucore:dateBroadcast>
                    <xsl:value-of select="."/>
                </ebucore:dateBroadcast>
            </xsl:template>

            <xsl:template match="director">
                <ebucore:hasParticipatingAgent>
                    <ebucore:Agent>                                        
                        <ebucore:hasRole>Director</ebucore:hasRole>
                        <ebucore:agentName>
                            <xsl:value-of select="."/>
                        </ebucore:agentName>
                    </ebucore:Agent>
                </ebucore:hasParticipatingAgent>
            </xsl:template>

            <xsl:template match="genre">
                <ebucore:hasGenre>
                    <xsl:value-of select="."/>
                </ebucore:hasGenre>
            </xsl:template>

        </ebucore:Feature>
    </xsl:template>
</xsl:stylesheet>

    To start with XSL, the first line, “<?xml version=”1.0″?>” defines the version of document. The second line opens a stylesheet defining the XSL version we will be using and further having XSL, RDF, EBUCore as their namespaces. These namespaces are required as we will be using elements of those classes to avoid name conflicts in our XML document. The xsl:template match defines which element to match in the XML, as we want to match from the start of the XML. Since movies are the root element of our XML, we will be using xsl:template match=”movies”. 

    After that, we open an RDF tag to start our knowledge graph, this element will contain all the movie details, and hence we are using xsl:apply-templates on “movie” as in our XML we have multiple <movie> elements nested inside <movies> tag. To get further details from <movie> elements, we define a template matching all movie elements, which will help us to fetch all the required details. The tag <ebucore:Feature> defines that all of our contents belong to a feature which is an alternate name for “movie” in EBUCore ontology. We then match details like title, year, genre, etc., from XML and define their corresponding value from EBUCore, like ebucore:title, ebucore:dateBroadcast, and ebucore:hasGenre respectively. 

    Now that we have the XSL ready, we will need to apply this XSL on our XML and get RDF data out of it by following the below Python code:

    import lxml.etree as ET
import xml.dom.minidom as xm

movie_data = """
            <movies>
                <movie id="tt0083658">
                    <title>Blade Runner</title>
                    <year>1982</year>
                    <director rid="12341">Ridley Scott</director>
                    <genre>Action</genre>
                </movie>
                    <movie id="tt0087469">
                    <title>Top Gun</title>
                    <year>1986</year>
                    <director rid="65217">Tony Hank</director>
                    <genre>Thriller</genre>
                </movie>
            </movies>
            """

xslt_file = "transform_movies.xsl"
xslt_root = ET.parse(xslt_file)
transform = ET.XSLT(xslt_root)
rss_root = ET.fromstring(movie_data)
result_tree = transform(rss_root)
result_string = ET.tostring(result_tree)

# Converting bytes to string and pretty formatting
dom_transformed = xm.parseString(result_string)
pretty_xml_as_string = dom_transformed.toprettyxml()

# Saving the output
with open('Path_to_downloads/output_movie.xml', "w") as f:
    f.write(pretty_xml_as_string)
# Print Output
print(pretty_xml_as_string)

    The above code will generate the following output:

    <?xml version="1.0" ?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:ebucore="http://www.ebu.ch/metadata/ontologies/ebucore/ebucore#">

<ebucore:Feature>
   <ebucore:title>Blade Runner</ebucore:title>
   <ebucore:dateBroadcast>1982</ebucore:dateBroadcast>
   <ebucore:hasParticipatingAgent>
       <ebucore:Agent>
           <ebucore:hasRole>Director</ebucore:hasRole>
           <ebucore:agentName>Ridley Scott</ebucore:agentName>
       </ebucore:Agent>
   </ebucore:hasParticipatingAgent>
   <ebucore:hasGenre>Action</ebucore:hasGenre>
</ebucore:Feature>

<ebucore:Feature>
   <ebucore:title>Top Gun</ebucore:title>
   <ebucore:dateBroadcast>1986</ebucore:dateBroadcast>
   <ebucore:hasParticipatingAgent>
       <ebucore:Agent>
           <ebucore:hasRole>Director</ebucore:hasRole>
           <ebucore:agentName>Tony Hank</ebucore:agentName>
       </ebucore:Agent>
   </ebucore:hasParticipatingAgent>
   <ebucore:hasGenre>Thriller</ebucore:hasGenre>
</ebucore:Feature>

</rdf:RDF>

    This output is an RDF XML, which will now be converted to a Graph and we will also visualize it using the following code:

    Note: Install the following library before proceeding.

    pip install graphviz rdflib

    from rdflib import Graph
from rdflib import Graph, Namespace
from rdflib.tools.rdf2dot import rdf2dot
from graphviz import render

# Creating empty graph object
graph = Graph()
graph.parse(result_string, format="xml")
# Saving graph/ttl(Terse RDF Triple Language) in movies.ttl file
graph.serialize(destination=f"Downloads/movies.ttl", format="ttl")

# Steps to Visualize the generated graph
# Define a namespace for the RDF data
ns = Namespace("http://example.com/movies#")
graph.bind("ex", ns)

# Serialize the RDF data to a DOT file
dot_file = open("Downloads/movies.dot", "w")
rdf2dot(graph, dot_file, opts={"label": "Movies Graph", "rankdir": "LR"})
dot_file.close()

# Render the DOT file to a PNG image

render("dot", "png", "Downloads/movies.dot")

    Finally, the above code will yield a movies.dot.png file in the Downloads folder location, which will look something like this:

    This clearly represents the relationship between edges and nodes along with all information in a well-formatted way.

    Examples of Knowledge Graphs

    Now that we have knowledge of how we can create a knowledge graph, let’s explore the big players that are using such graphs for their operations.

    Google Knowledge Graph: This is one of the most well-known examples of a knowledge graph. It is used by Google to enhance its search results with additional information about entities, such as people, places, and things. For example, if you search for “Barack Obama,” the Knowledge Graph will display a panel with information about his birthdate, family members, education, career, and more. All this information is stored in the form of nodes and edges making it easier for the Google search engine to retrieve related information of any topic.

    DBpedia: This is a community-driven project that extracts structured data from Wikipedia and makes it available as a linked data resource. It is primarily used for graph analysis and executing SPARQL queries. It contains information on millions of entities, such as people, places, and things, and their relationships with one another. DBpedia can be used to power applications like question-answering systems, recommendation engines, and more. One of the key advantages of DBpedia is that it is an open and community-driven project, which means that anyone can contribute to it and use it for their own applications. This has led to a wide variety of applications built on top of DBpedia, from academic research to commercial products.

    As we have discussed the examples of knowledge graphs, one should know that they all use SPARQL queries to retrieve data from their huge corpus of graphs. So, let’s write one such query to retrieve data from the knowledge graph created by us for movie data. We will be writing a query to retrieve all movies’ Genre information along with Movie Titles.

    from rdflib.plugins.sparql import prepareQuery

# Define the SPARQL query
query = prepareQuery('''
    PREFIX ebucore: <http://www.ebu.ch/metadata/ontologies/ebucore/ebucore#>
    SELECT ?genre ?title
    WHERE {
      ?movie a ebucore:Feature ;
             ebucore:hasGenre ?genre ;
             ebucore:title ?title .
    }
''', initNs={"ebucore": ns})


# Execute the query and print the results
results = graph.query(query)
for row in results:
    genre, title = row
    print(f"Movie Genre: {genre}, Movie Title: {title}")

    Challenges with Graph Databases:

    Data Complexity: One of the primary challenges with graph databases and knowledge graphs is data complexity. As the size and complexity of the data increase, it can become challenging to manage and query the data efficiently.

    Data Integration: Graph databases and knowledge graphs often need to integrate data from different sources, which can be challenging due to differences in data format, schema, and structure.

    Query Performance: Knowledge graphs are often used for complex queries, which can be slow to execute, especially for large datasets.

    Knowledge Representation: Representing knowledge in a graph database or knowledge graph can be challenging due to the diversity of concepts and relationships that need to be modeled accurately. One should have experience with ontologies, relationships, and business use cases to curate a perfect representation

    Bonus: How to Overcome These Challenges:

    • Use efficient indexing and query optimization techniques to handle data complexity and improve query performance.
    • Use data integration tools and techniques to standardize data formats and structures to improve data integration.
    • Use distributed computing and partitioning techniques to scale the database horizontally.
    • Use caching and precomputing techniques to speed up queries.
    • Use ontology modeling and semantic reasoning techniques to accurately represent knowledge and relationships in the graph database or knowledge graph.

    Conclusion

    In conclusion, graph databases, and knowledge graphs are powerful tools that offer several advantages over traditional relational databases. They enable flexible modeling of complex data and relationships, which can be difficult to achieve using a traditional tabular structure. Moreover, they enhance query performance for complex queries and enable new use cases such as recommendation engines, fraud detection, and knowledge management.

    Despite the aforementioned challenges, graph databases and knowledge graphs are gaining popularity in various industries, ranging from finance to healthcare, and are expected to continue playing a significant role in the future of data management and analysis.

  • Vector Search: The New Frontier in Personalized Recommendations

    Introduction

    Imagine you are a modern-day treasure hunter, not in search of hidden gold, but rather the wealth of knowledge and entertainment hidden within the vast digital ocean of content. In this realm, where every conceivable topic has its own sea of content, discovering what will truly captivate you is like finding a needle in an expansive haystack.

    This challenge leads us to the marvels of recommendation services, acting as your compass in this digital expanse. These services are the unsung heroes behind the scenes of your favorite platforms, from e-commerce sites that suggest enticing products to streaming services that understand your movie preferences better than you might yourself. They sift through immense datasets of user interactions and content features, striving to tailor your online experience to be more personalized, engaging, and enriching.

    But what if I told you that there is a cutting-edge technology that can take personalized recommendations to the next level? Today, I will take you through a journey to build a blog recommendation service that understands the contextual similarities between different pieces of content, transcending beyond basic keyword matching. We’ll harness the power of vector search, a technology that’s revolutionizing personalized recommendations. We’ll explore how recommendation services are traditionally implemented, and then briefly discuss how vector search enhances them.

    Finally, we’ll put this knowledge to work, using OpenAI’s embedding API and Elasticsearch to create a recommendation service that not only finds content but also understands and aligns it with your unique interests.

    Exploring the Landscape: Traditional Recommendation Systems and Their Limits

    Traditionally, these digital compasses, or recommendation systems, employ methods like collaborative and content-based filtering. Imagine sitting in a café where the barista suggests a coffee based on what others with similar tastes enjoyed (collaborative filtering) or based on your past coffee choices (content-based filtering). While these methods have been effective in many scenarios, they come with some limitations. They often stumble when faced with the vast and unstructured wilderness of web data, struggling to make sense of the diverse and ever-expanding content landscape. Additionally, when user preferences are ambiguous or when you want to recommend content by truly understanding it on a semantic level, traditional methods may fall short.

    Enhancing Recommendation with Vector Search and Vector Databases

    Our journey now takes an exciting turn with vector search and vector databases, the modern tools that help us navigate this unstructured data. These technologies transform our café into a futuristic spot where your coffee preference is understood on a deeper, more nuanced level.

    Vector Search: The Art of Finding Similarities

    Vector search operates like a seasoned traveler who understands the essence of every place visited. Text, images, or sounds can be transformed into numerical vectors, like unique coordinates on a map. The magic happens when these vectors are compared, revealing hidden similarities and connections, much like discovering that two seemingly different cities share a similar vibe.

    Vector Databases: Navigating Complex Data Landscapes

    Imagine a vast library of books where each book captures different aspects of a place along with its coordinates. Vector databases are akin to this library, designed to store and navigate these complex data points. They easily handle intricate queries over large datasets, making them perfect for our recommendation service, ensuring no blog worth reading remains undiscovered.

    Embeddings: Semantic Representation

    In our journey, embeddings are akin to a skilled artist who captures not just the visuals but the soul of a landscape. They map items like words or entire documents into real-number vectors, encapsulating their deeper meaning. This helps in understanding and comparing different pieces of content on a semantic level, letting the recommendation service show you things that really match your interests.

    Sample Project: Blog Recommendation Service

    Project Overview

    Now, let’s craft a simple blog recommendation service using OpenAI’s embedding APIs and Elasticsearch as a vector database. The goal is to recommend blogs similar to the current one the user is reading, which can be shown in the read more or recommendation section.

    Our blogs service will be responsible for indexing the blogs, finding similar one,  and interacting with the UI Service.

    Tools and Setup

    We will need the following tools to build our service:

    • OpenAI Account: We will be using OpenAI’s embedding API to generate the embeddings for our blog content. You will need an OpenAI account to use the APIs. Once you have created an account, please create an API key and store it in a secure location.
    • Elasticsearch: A popular database renowned for its full-text search capabilities, which can also be used as a vector database, adept at storing and querying complex embeddings with its dense_vector field type.
    • Docker: A tool that allows developers to package their applications and all the necessary dependencies into containers, ensuring that the application runs smoothly and consistently across different computing environments.
    • Python: A versatile programming language for developers across diverse fields, from web development to data science.

    The APIs will be created using the FastAPI framework, but you can choose any framework.

    Steps

    First, we’ll create a BlogItem class to represent each blog. It has only three fields, which will be enough for this demonstration, but real-world entities would have more details to accommodate a wider range of properties and functionalities.

    class BlogItem:
    blog_id: int
    title: str
    content: str

    def __init__(self, blog_id: int, title: str, content: str):
        self.blog_id = blog_id
        self.title = title
        self.content = content

    Elasticsearch Setup:

    • To store the blog data along with its embedding in Elasticsearch, we need to set up a local Elasticsearch cluster and then create an index for our blogs. You can also use a cloud-based version if you have already procured one for personal use.
    • Install Docker or Docker Desktop on your machine and create Elasticsearch and Kibana docker containers using the below docker compose file. Run the following command to create and start the services in the background:
    • docker compose -f /path/to/your/docker-compose/file up -d. 
    • You can exclude the file path if you are in the same directory as your docker-compose.yml file. The advantage of using docker compose is that it allows you to clean up these resources with just one command.
    • docker compose -f /path/to/your/docker-compose/file down.‍
    version: '3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:<version>
    container_name: elasticsearch
    environment:
      - node.name=docker-cluster
      - discovery.type=single-node
      - cluster.routing.allocation.disk.threshold_enabled=false
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=YourElasticPassword
      - "ELASTICSEARCH_USERNAME=elastic"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - esnet

  kibana:
    image: docker.elastic.co/kibana/kibana:<version>
    container_name: kibana
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      ELASTICSEARCH_USERNAME: elastic
      ELASTICSEARCH_PASSWORD: YourElasticPassword
    ports:
      - "5601:5601"
    networks:
      - esnet
    depends_on:
      - elasticsearch

networks:
  esnet:
    driver: bridge

volumes:
  esdata:
    driver: local

    • Connect to the local ES instance and create an index. Our “blogs” index will have a unique blog ID, blog title, blog content, and an embedding field to store the vector representation of blog content. The text-embedding-ada-002 model we have used here produces vectors with 1536 dimensions; hence, it’s important to use the same in our embeddings field in the blogs index.
    import logging
from elasticsearch import Elasticsearch

# Setting up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(levelname)s] %(filename)s:%(lineno)d - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')
# Elasticsearch client setup
es = Elasticsearch(hosts="http://localhost:9200",
                   basic_auth=("elastic", "YourElasticPassword"))


# Create an index with a mappings for embeddings
def create_index(index_name: str, embedding_dimensions: int):
    try:
        es.indices.create(
            index=index_name,
            body={
                'mappings': {
                    'properties': {
                        'blog_id': {'type': 'long'},
                        'title': {'type': 'text'},
                        'content': {'type': 'text'},
                        'embedding': {'type': 'dense_vector', 'dims': embedding_dimensions}
                    }
                }
            }
        )
    except Exception as e:
        logging.error(f"An error occurred while creating index {index_name} : {e}")


# Sample usage
create_index("blogs", 1536)

    Create Embeddings AND Index Blogs:

    • We use OpenAI’s Embedding API to get a vector representation of our blog title and content. I am using the 002 model here, which is recommended by Open AI for most use cases. The input to the text-embedding-ada-002 should not exceed 8291 tokens (1000 tokens are roughly equal to 750 words) and cannot be empty.
    from openai import OpenAI, OpenAIError

api_key = ‘YourApiKey’
client = OpenAI(api_key=api_key)

def create_embeddings(text: str, model: str = "text-embedding-ada-002") -> list[float]:
    try:
        text = text.replace("n", " ")
        response = client.embeddings.create(input=[text], model=model)
        logging.info(f"Embedding created successfully")
        return response.data[0].embedding
    except OpenAIError as e:
        logging.error(f"An OpenAI error occurred while creating embedding : {e}")
        raise
    except Exception as e:
        logging.exception(f"An unexpected error occurred while creating embedding  : {e}")
        raise

    • When the blogs get created or the content of the blog gets updated, we will call the create_embeddings function to get text embedding and store it in our blogs index.
    # Define the index name as a global constant

ELASTICSEARCH_INDEX = 'blogs'

def index_blog(blog_item: BlogItem):
    try:
        es.index(index=ELASTICSEARCH_INDEX, body={
            'blog_id': blog_item.blog_id,
            'title': blog_item.title,
            'content': blog_item.content,
            'embedding': create_embeddings(blog_item.title + "n" + blog_item.content)
        })
    except Exception as e:
        logging.error(f"Failed to index blog with blog id {blog_item.blog_id} : {e}")
        raise

    • Create a Pydantic model for the request body:
    class BlogItemRequest(BaseModel):
    title: str
    content: str

    • Create an API to save blogs to Elasticsearch. The UI Service would call this API when a new blog post gets created.
    @app.post("/blogs/")
def save_blog(response: Response, blog_item: BlogItemRequest) -> dict[str, str]:
    # Create a BlogItem instance from the request data
    try:
        blog_id = get_blog_id()
        blog_item_obj = BlogItem(
            blog_id=blog_id,
            title=blog_item.title,
            content=blog_item.content,
        )

        # Call the index_blog method to index the blog
        index_blog(blog_item_obj)
        return {"message": "Blog indexed successfully", "blog_id": str(blog_id)}
    except Exception as e:
        logging.error(f"An error occurred while indexing blog with blog_id {blog_id} : {e}")
        response.status_code = status.HTTP_500_INTERNAL_SERVER_ERROR
        return {"error": "Failed to index blog"}

    Finding Relevant Blogs:

    • To find blogs that are similar to the current one, we will compare the current blog’s vector representation with other blogs present in the Elasticsearch index using the cosine similarity function.
    • Cosine similarity is a mathematical measure used to determine the cosine of the angle between two vectors in a multi-dimensional space, often employed to assess the similarity between two documents or data points. 
    • The cosine similarity score ranges from -1 to 1. As the cosine similarity score increases from -1 to 1, it indicates an increasing degree of similarity between the vectors. Higher values represent greater similarity.
    • Create a custom exception to handle a scenario when a blog for a given ID is not present in Elasticsearch.
    class BlogNotFoundException(Exception):
    def __init__(self, message="Blog not found"):
        self.message = message
        super().__init__(self.message)

    • First, we will check if the current blog is present in the blogs index and get its embedding. This is done to prevent unnecessary calls to Open AI APIs as it consumes tokens. Then, we would construct an Elasticsearch dsl query to find the nearest neighbors and return their blog content.
    def get_blog_embedding(blog_id: int) -> Optional[Dict]:
    try:
        response = es.search(index=ELASTICSEARCH_INDEX, body={
            'query': {
                'term': {
                    'blog_id': blog_id
                }
            },
            '_source': ['title', 'content', 'embedding']  # Fetch title, content and embedding
        })

        if response['hits']['hits']:
            logging.info(f"Blog found with blog_id {blog_id}")
            return response['hits']['hits'][0]['_source']
        else:
            logging.info(f"No blog found with blog_id {blog_id}")
            return None
    except Exception as e:
        logging.error(f"Error occurred while searching for blog with blog_id {blog_id}: {e}")
        raise


def find_similar_blog(current_blog_id: int, num_neighbors=2) -> list[dict[str, str]]:
    try:
        blog_data = get_blog_embedding(current_blog_id)
        if not blog_data:
            raise BlogNotFoundException(f"Blog not found for id:{current_blog_id}")
        blog_embedding = blog_data['embedding']
        if not blog_embedding:
            blog_embedding = create_embeddings(blog_data['title'] + 'n' + blog_data['content'])
        # Find similar blogs using the embedding
        response = es.search(index=ELASTICSEARCH_INDEX, body={
            'size': num_neighbors + 1,  # Retrieve extra result as we'll exclude the current blog
            '_source': ['title', 'content', 'blog_id', '_score'],
            'query': {
                'bool': {
                    'must': {
                        'script_score': {
                            'query': {'match_all': {}},
                            'script': {
                                'source': "cosineSimilarity(params.query_vector, 'embedding')",
                                'params': {'query_vector': blog_embedding}
                            }
                        }
                    },
                    'must_not': {
                        'term': {
                            'blog_id': current_blog_id  # Exclude the current blog
                        }
                    }
                }
            }
        })

        # Extract and return the hits
        hits = [
            {
                'title': hit['_source']['title'],
                'content': hit['_source']['content'],
                'blog_id': hit['_source']['blog_id'],
                'score': f"{hit['_score'] * 100:.2f}%"
            }
            for hit in response['hits']['hits']
            if hit['_source']['blog_id'] != current_blog_id
        ]

        return hits
    except Exception as e:
        logging.error(f"An error while finding similar blogs: {e}")
        raise

    • Define a Pydantic model for the response:
    class BlogRecommendation(BaseModel):
    blog_id: int
    title: str
    content: str
    score: str

    • Create an API that would be used by UI Service to find similar blogs as the current one user is reading:
    @app.get("/recommend-blogs/{current_blog_id}")
def recommend_blogs(
        response: Response,
        current_blog_id: int,
        num_neighbors: Optional[int] = 2) -> Union[Dict[str, str], List[BlogRecommendation]]:
    try:
        # Call the find_similar_blog function to get recommended blogs
        recommended_blogs = find_similar_blog(current_blog_id, num_neighbors)
        return recommended_blogs
    except BlogNotFoundException as e:
        response.status_code = status.HTTP_400_BAD_REQUEST
        return {"error": f"Blog not found for id:{current_blog_id}"}
    except Exception as e:
        response.status_code = status.HTTP_500_INTERNAL_SERVER_ERROR
        return {"error": "Unable to process the request"}

    • The below flow diagram summarizes all the steps we have discussed so far:

    Testing the Recommendation Service

    • Ideally, we would be receiving the blog ID from the UI Service and passing the recommendations back, but for illustration purposes, we’ll be calling the recommend blogs API with some test inputs from my test dataset. The blogs in this sample dataset have concise titles and content, which are sufficient for testing purposes, but real-world blogs will be much more detailed and have a significant amount of data. The test dataset has around 1000 blogs on various categories like healthcare, tech, travel, entertainment, and so on.
    • A sample from the test dataset:

     

    • Test Result 1: Medical Research Blog

      Input Blog: Blog_Id: 1, Title: Breakthrough in Heart Disease Treatment, Content: Researchers have developed a new treatment for heart disease that promises to be more effective and less invasive. This breakthrough could save millions of lives every year.


    ‍

    • Test Result 2: Travel Blog

      Input Blog: Blog_Id: 4, Title: Travel Tips for Sustainable Tourism, Content: How to travel responsibly and sustainably.


    I manually tested multiple blogs from the test dataset of 1,000 blogs, representing distinct topics and content, and assessed the quality and relevance of the recommendations. The recommended blogs had scores in the range of 87% to 95%, and upon examination, the blogs often appeared very similar in content and style.

    Based on the test results, it’s evident that utilizing vector search enables us to effectively recommend blogs to users that are semantically similar. This approach ensures that the recommendations are contextually relevant, even when the blogs don’t share identical keywords, enhancing the user’s experience by connecting them with content that aligns more closely with their interests and search intent.

    Limitations

    This approach for finding similar blogs is good enough for our simple recommendation service, but it might have certain limitations in real-world applications.

    • Our similarity search returns the nearest k neighbors as recommendations, but there might be scenarios where no similar blog might exist or the neighbors might have significant score differences. To deal with this, you can set a threshold to filter out recommendations below a certain score. Experiment with different threshold values and observe their impact on recommendation quality. 
    • If your use case involves a small dataset and the relationships between user preferences and item features are straightforward and well-defined, traditional methods like content-based or collaborative filtering might be more efficient and effective than vector search.

    Further Improvements

    • Using LLM for Content Validation: Implement a verification step using large language models (LLMs) to assess the relevance and validity of recommended content. This approach can ensure that the suggestions are not only similar in context but also meaningful and appropriate for your audience.
    • Metadata-based Embeddings: Instead of generating embeddings from the entire blog content, utilize LLMs to extract key metadata such as themes, intent, tone, or key points. Create embeddings based on this extracted metadata, which can lead to more efficient and targeted recommendations, focusing on the core essence of the content rather than its entirety.

    Conclusion

    Our journey concludes here, but yours is just beginning. Armed with the knowledge of vector search, vector databases, and embeddings, you’re now ready to build a recommendation service that doesn’t just guide users to content but connects them to the stories, insights, and experiences they seek. It’s not just about building a service; it’s about enriching the digital exploration experience, one recommendation at a time.

  • Mage: Your New Go-To Tool for Data Orchestration

    In our journey to automate data pipelines, we’ve used tools like Apache Airflow, Dagster, and Prefect to manage complex workflows. However, as data automation continues to change, we’ve added a new tool to our toolkit: Mage AI.

    Mage AI isn’t just another tool; it’s a solution to the evolving demands of data automation. This blog aims to explain how Mage AI is changing the way we automate data pipelines by addressing challenges and introducing innovative features. Let’s explore this evolution, understand the problems we face, and see why we’ve adopted Mage AI.

    What is Mage AI?

    Mage is a user-friendly open-source framework created for transforming and merging data. It’s a valuable tool for developers handling substantial data volumes efficiently. At its heart, Mage relies on “data pipelines,” made up of code blocks. These blocks can run independently or as part of a larger pipeline. Together, these blocks form a structure known as a directed acyclic graph (DAG), which helps manage dependencies. For example, you can use Mage for tasks like loading data, making transformations, or exportation.

    Mage Architecture:

    Before we delve into Mage’s features, let’s take a look at how it works.

    When you use Mage, your request begins its journey in the Mage Server Container, which serves as the central hub for handling requests, processing data, and validation. Here, tasks like data processing and real-time interactions occur. The Scheduler Process ensures tasks are scheduled with precision, while Executor Containers, designed for specific tasks like Python or AWS, carry out the instructions.

    Mage’s scalability is impressive, allowing it to handle growing workloads effectively. It can expand both vertically and horizontally to maintain top-notch performance. Mage efficiently manages data, including code, data, and logs, and takes security seriously when handling databases and sensitive information. This well-coordinated system, combined with Mage’s scalability, guarantees reliable data pipelines, blending technical precision with seamless orchestration.

    Scaling Mage:

    To enhance Mage’s performance and reliability as your workload expands, it’s crucial to scale its architecture effectively. In this concise guide, we’ll concentrate on four key strategies for optimizing Mage’s scalability:

    1. Horizontal Scaling: Ensure responsiveness by running multiple Mage Server and Scheduler instances. This approach keeps the system running smoothly, even during peak usage.
    2. Multiple Executor Containers: Deploy several Executor Containers to handle concurrent task execution. Customize them for specific executors (e.g., Python, PySpark, or AWS) to scale task processing horizontally as needed.
    3. External Load Balancers: Utilize external load balancers to distribute client requests across Mage instances. This not only boosts performance but also ensures high availability by preventing overloading of a single server.
    4. Scaling for Larger Datasets: To efficiently handle larger datasets, consider:

    a. Allocating more resources to executors, empowering them to tackle complex data transformations.

    b. Mage supports direct data warehouse transformation and native Spark integration for massive datasets.

    Features: 

    1) Interactive Coding Experience

    Mage offers an interactive coding experience tailored for data preparation. Each block in the editor is a modular file that can be tested, reused, and chained together to create an executable data pipeline. This means you can build your data pipeline piece by piece, ensuring reliability and efficiency.

    2) UI/IDE for Building and Managing Data Pipelines

    Mage takes data pipeline development to the next level with a user-friendly integrated development environment (IDE). You can build and manage your data pipelines through an intuitive user interface, making the process efficient and accessible to both data scientists and engineers.

    3) Multiple Languages Support

    Mage supports writing pipelines in multiple languages such as Python, SQL, and R. This language versatility means you can work with the languages you’re most comfortable with, making your data preparation process more efficient.

    4) Multiple Types of Pipelines

    Mage caters to diverse data pipeline needs. Whether you require standard batch pipelines, data integration pipelines, streaming pipelines, Spark pipelines, or DBT pipelines, Mage has you covered.

    5) Built-In Engineering Best Practices

    Mage is not just a tool; it’s a promoter of good coding practices. It enables reusable code, data validation in each block, and operationalizes data pipelines with built-in observability, data quality monitoring, and lineage. This ensures that your data pipelines are not only efficient but also maintainable and reliable.

    6) Dynamic Blocks

    Dynamic blocks in Mage allow the output of a block to dynamically create additional blocks. These blocks are spawned at runtime, with the total number of blocks created being equal to the number of items in the output data of the dynamic block multiplied by the number of its downstream blocks.

    7) Triggers

    • Schedule Triggers: These triggers allow you to set specific start dates and intervals for pipeline runs. Choose from daily, weekly, or monthly, or even define custom schedules using Cron syntax. Mage’s Schedule Triggers put you in control of when your pipelines execute.
    • Event Triggers: With Event Triggers, your pipelines respond instantly to specific events, such as the completion of a database query or the creation of a new object in cloud storage services like Amazon S3 or Google Storage. Real-time automation at your fingertips.
    • API Triggers: API Triggers enable your pipelines to run in response to specific API calls. Whether it’s customer requests or external system interactions, these triggers ensure your data workflows stay synchronized with the digital world.

    Different types of Block: 

    Data Loader: Within Mage, Data Loaders are ready-made templates designed to seamlessly link up with a multitude of data sources. These sources span from Postgres, Bigquery, Redshift, and S3 to various others. Additionally, Mage allows for the creation of custom data loaders, enabling connections to APIs. The primary role of Data Loaders is to facilitate the retrieval of data from these designated sources.

    Data Transformer: Much like Data Loaders, Data Transformers provide predefined functions such as handling duplicates, managing missing data, and excluding specific columns. Alternatively, you can craft your own data transformations or merge outputs from multiple data loaders to preprocess and sanitize the data before it advances through the pipeline.

    Data Exporter: Data Exporters within Mage empower you to dispatch data to a diverse array of destinations, including databases, data lakes, data warehouses, or local storage. You can opt for predefined export templates or craft custom exporters tailored to your precise requirements.

    Custom Blocks: Custom blocks in the Mage framework are incredibly flexible and serve various purposes. They can store configuration data and facilitate its transmission across different pipeline stages. Additionally, they prove invaluable for logging purposes, allowing you to categorize and visually distinguish log entries for enhanced organization.

    Sensor: A Sensor, a specialized block within Mage, continuously assesses a condition until it’s met or until a specified time duration has passed. When a block depends on a sensor, it remains inactive until the sensor confirms that its condition has been satisfied. Sensors are especially valuable when there’s a need to wait for external dependencies or handle delayed data before proceeding with downstream tasks.

    Getting Started with Mage

    There are two ways to run mage, either using docker or using pip:
    Docker Command

    Create a new working directory where all the mage files will be stored.

    Then, in that working directory, execute this command:

    Windows CMD: 

    docker run -it -p 6789:6789 -v %cd%:/home/src mageai/mageai /app/run_app.sh mage start [project_name]

    Linux CMD:

    docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]

    Using Pip (Working directory):

    pip install mage-ai

    Mage start [project_name]

    You can browse to http://localhost:6789/overview to get to the Mage UI.

    Let’s build our first pipelineto fetch CSV files from the API for data loading, do some useful transformations, and export that data to our local database.

    Dataset invoices CSV files stored in the current directory of columns:

     (1) First Name; (2) Last Name; (3) E-mail; (4) Product ID; (5) Quantity; (6) Amount; (7) Invoice Date; (8) Address; (9) City; (10) Stock Code

    Create a new pipeline and select a standard batch (we’ll be implementing a batch pipeline) from the dashboard and give it a unique ID.

    Project structure:

    ├── mage_data

    └── [project_name]

        ├── charts

        ├── custom

        ├── data_exporters

        ├── data_loaders

        ├── dbt

        ├── extensions

        ├── pipelines

        │   └── [pipeline_name]

        │       ├── __init__.py

        │       └── metadata.yaml

        ├── scratchpads

        ├── transformers

        ├── utils

        ├── __init__.py

        ├── io_config.yaml

        ├── metadata.yaml

        └── requirements.txt

    This pipeline consists of all the block files, including data loader, transformer, charts, and configuration files for our pipeline io_config.yaml and metadata.yaml files. All block files will contain decorators’ inbuilt function where we will be writing our code.

    1. We begin by loading a CSV file from our local directory, specifically located at /home/src/invoice.csv. To achieve this, we select the “Local File” option from the Templates dropdown and configure the Data Loader block accordingly. Running this configuration will allow us to confirm if the CSV file loads successfully.

    2. In the next step, we introduce a Transformer block using a generic template. On the right side of the user interface, we can observe the directed acyclic graph (DAG) tree. To establish the data flow, we edit the parent of the Transformer block, linking it either directly to the Data Loader block or via the user interface.

    The Transformer block operates on the data frame received from the upstream Data Loader block, which is passed as the first argument to the Transformer function.

    3. Our final step involves exporting the DataFrame to a locally hosted PostgreSQL database. We incorporate a Data Export block and connect it to the Transformer block.

    To establish a connection with the PostgreSQL database, it is imperative to configure the database credentials in the io_config.yaml file. Alternatively, these credentials can be added to environmental variables.

    With these steps completed, we have successfully constructed a foundational batch pipeline. This pipeline efficiently loads, transforms, and exports data, serving as a fundamental building block for more advanced data processing tasks.

    Mage vs Other tools:

    Consistency Across Environments: Some orchestration tools may exhibit inconsistencies between local development and production environments due to varying configurations. Mage tackles this challenge by providing a consistent and reproducible workflow environment through a single configuration file that can be executed uniformly across different environments.

    Reusability: Achieving reusability in workflows can be complex in some tools. Mage simplifies this by allowing tasks and workflows to be defined as reusable components within a Magefile, making it effortless to share them across projects and teams.

    Data Passing: Efficiently passing data between tasks can be a challenge in certain tools, especially when dealing with large datasets. Mage streamlines data passing through straightforward function arguments and returns, enabling seamless data flow and versatile data handling.

    Testing: Some tools lack user-friendly testing utilities, resulting in manual testing and potential coverage gaps. Mage simplifies testing with a robust testing framework that enables the definition of test cases, inputs, and expected outputs directly within the Mage file.

    Debugging: Debugging failed tasks can be time-consuming with certain tools. Mage enhances debugging with detailed logs and error messages, offering clear insights into the causes of failures and expediting issue resolution.

    Conclusion: 

    Mage offers a streamlined and user-friendly approach to data pipeline orchestration, addressing common challenges with simplicity and efficiency. Its single-container deployment, visual interface, and robust features make it a valuable tool for data professionals seeking an intuitive and consistent solution for managing data workflows.

  • Discover App Features with TipKit

    In today’s digital age, the user experience is paramount. Mobile applications need to be intuitive and user-friendly so that users not only enjoy the app’s main functionalities but also easily navigate through its features. There are instances where a little extra guidance can go a long way, whether it’s introducing users to a fresh feature, showing them shortcuts to complete tasks more efficiently, or simply offering tips on getting the most out of an app. Many developers have traditionally crafted custom overlays or tooltips to bridge this gap, often requiring a considerable amount of effort. But the wait for a streamlined solution is over. After much anticipation, Apple has introduced the TipKit framework, a dedicated tool to simplify this endeavor, enhancing user experience with finesse.

    TipKit

    Introduced at WWDC 2023, TipKit emerges as a beacon for app developers aiming to enhance user engagement and experience. This framework is ingeniously crafted to present mini-tutorials, shining a spotlight on new, intriguing, or yet-to-be-discovered features within an application. Its utility isn’t just confined to a single platform—TipKit boasts integration with iCloud to ensure data synchronization across various devices.

    At the heart of TipKit lies its two cornerstone components: the Tip Protocol and the TipView. These components serve as the foundation, enabling developers to craft intuitive and informative tips that resonate with their user base.

    Tip Protocol

    The essence of TipKit lies in its Tip Protocol, which acts as the blueprint for crafting and configuring content-driven tips. To create your tips tailored to your application’s needs, it’s imperative to conform to the Tip Protocol.

    While every Tip demands a title for identification, the protocol offers flexibility by introducing a suite of properties that can be optionally integrated, allowing developers to craft a comprehensive and informative tip.

    1. title(Text): The title of the Tip.
    2. message(Text): A concise description further elaborates the essence of the Tip, providing users with a deeper understanding.
    3. asset(Image): An image to display on the left side of the Tip view.
    4. id(String): A unique identifier to your tip. Default will be the name of the type that conforms to the Tip protocol.
    5. rules(Array of type Tips.Rule): This can be used to add rules to the Tip that can determine when the Tip needs to be displayed.
    6. options(Array of type Tips.Option): Allows to add options for defining the behavior of the Tip.
    7. actions(Array of type Tips.Action): This will provide primary and secondary buttons in the TipView that could help the user learn more about the Tip or execute a custom action when the user interacts with it.

    Creating a Custom Tip

    Let’s create our first Tip. Here, we are going to show a Tip to help the user understand the functionality of the cart button.

    struct CartItemsTip: Tip {
    var title: Text {
        Text("Click the cart button to see what's in your cart")
    }
    var message: Text? {
        Text("You can edit/remove the items from your cart")
    }
    var image: Image? {
        Image(systemName: "cart")
    }
}

    TipView

    As the name suggests, TipView is a user interface that represents the Inline Tip. The initializer of TipView requires an instance of the Tip protocol we discussed above, an Edge parameter, which is optional, for deciding the edge of the tip view that displays the arrow.

    Displaying a Tip

    Following are the two ways the Tip can be displayed.

    • Inline

    You can display the tip along with other views. An object of TipView requires a type conforming Tip protocol used to display the Inline tip. As a developer, handling multiple views on the screen could be a complex and time-consuming task. TipKit framework makes it easy for the developers as it automatically adjusts the layout and the position of the TipView to ensure other views are accessible to the user. 

    struct ProductList: View {
    private let cartTip = CartItemsTip()
    var body: some View {
        \ Other views
        
        TipView(cartTip)
        
        \ Other views
    }
}

    • Popover

    TipKit Frameworks allow you to show a popover Tip for any UI element, e.g., Button, Image, etc. The popover tip appears over the entire screen, thus blocking the other views from user interaction until the tip is dismissed. A popoverTip modifier displays a Popover Tip for any UI element. Consider an example below where a Popover tip is displayed for a cart image.

    private let cartTip = CartItemsTip()
Button {
   cartTip.invalidate(reason: .actionPerformed)
} label: {
   Image(systemName: "cart")
       .popoverTip(cartTip)
}

    Dismissing the Tip

    A TipView can be dismissed in two ways.

    1. The user needs to click on X icon.
    2. Developers can dismiss the Tip programmatically using the invalidate(reason:) method.

    There are 3 options to pass as a reason for dismissing the tip: 

    actionPerformed, userClosedTip, maxDisplayCountExceeded

    private let cartTip = CartItemsTip()
cartTip.invalidate(reason: .actionPerformed)

    Tips Center

    We have discussed essential points to define and display a tip using Tip protocol and TipView, respectively. Still, there is one last and most important step—to configure and load the tip using the configure method as described in the below example. This is mandatory to display the tips within your application. Otherwise, you will not see tips.

    import SwiftUI
import TipKit

@main
struct TipkitDemoApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
                .task {
                    try? Tips.configure([
                        .displayFrequency(.immediate),
                        .datastoreLocation(.applicationDefault)
                    ])
                }
        }
    }
}

    If you see the definition of configure method, it should be something like:

    static func configure(@Tips.ConfigurationBuilder options: @escaping () -> some TipsConfiguration = { defaultConfiguration }) async throws

    If you notice, the configure method accepts a list of types conforming to TipsConfiguration. There are two options available for TipsConfiguration, DisplayFrequency and DataStoreLocation. 

    You can set these values as per your requirement. 

    DisplayFrequency

    DisplayFrequnecy allows you to control the frequency of your tips and has multiple options. 

    • Use the immediate option when you do not want to set any restrictions.
    • Use the hourly, daily, weekly, and monthly values to display no more than one tip hourly, weekly, and so on, respectively. 
    • For some situations, you need to set the custom display frequency as TimeInterval, when all the available options could not serve the purpose. In the below example, we have set a custom display frequency that restricts the tips to be displayed once per two days.
    let customDisplayFrequency: TimeInterval = 2 * 24 * 60 * 60
try? Tips.configure([
     .displayFrequency(customDisplayFrequency),
     .datastoreLocation(.applicationDefault)
])

    DatastoreLocation

    This will be used for persisting tips and associated data. 

    You can use the following initializers to decide how to persist tips and data.

    public init(url: URL, shouldReset: Bool = false)

    url: A specific URL location where you want to persist the data.

    shouldReset: If set to true, it will erase all data from the datastore. Resetting all tips present in the application.

    public init(_ location: DatastoreLocation, shouldReset: Bool = false)

    location: A predefined datastore location. Setting a default value ‘applicationDefault’ would persist the datastore in the app’s support directory. 

    public init(groupIdentifier: String, directoryName: String? = nil, shouldReset: Bool = false) throws

    groupIdentifier: The name of the group whose shared directory is used by the group of your team’s applications. Use the optional directoryName to specify a directory within this group.

    directoryName: The optional directoryName to specify a directory within the group.

    Max Display Count

    As discussed earlier, we can set options to define tip behavior. One such option is MaxDisplayCount. Consider that you want to show CartItemsTip whenever the user is on the Home screen. Showing the tip every time a user comes to the Home screen can be annoying or frustrating. To prevent this, one of the solutions, perhaps the easiest, is using MaxDisplayCount. The other solution could be defining a Rule that determines when the tip needs to be displayed. Below is an example showcasing the use of the MaxDisplayCount option for defining CartItemsTip.

    struct CartItemsTip: Tip {
    var title: Text {
        Text("Click here to see what's in your cart")
    }
    
    var message: Text? {
        Text("You can edit/remove the items from your cart")
    }
    
    var image: Image? {
        Image(systemName: "cart")
    }
    
    var options: [TipOption] {
        [ MaxDisplayCount(2) ]
    }
}

    Rule Based Tips

    Let’s understand how Rules can help you gain more control over displaying your tips. There are two types of Rules: parameter-based rules and event-based rules.

    Parameter Rules

    These are persistent and more useful for State and Boolean comparisons. There are Macros (#Rule, @Parameter) available to define a rule. 

    In the below example, we define a rule that checks if the value stored in static itemsInCart property is greater than or equal to 3. 

    Defining rules ensures displaying tips only when all the conditions are satisfied.

    struct CartTip: Tip {
    
    var title: Text {
        Text("Proceed with buying cart items.")
    }
    
    var message: Text? {
        Text("There are 3 or more items in your cart.")
    }
    
    var image: Image? {
        Image(systemName: "cart")
    }
    
    @Parameter
    static var itemsInCart: Int = 0
    
    var rules: [Rule] {
        #Rule(Self.$itemsInCart) { $0 >= 3 }
    }
}

    Event Rules

    Event-based rules are useful when we want to track occurrences of certain actions in the app. Each event has a unique identifier id of type string, with which we can differentiate between various events. Whenever the action occurs, we need to use the denote() method to increment the counter. 

    Let’s consider the below example where we want to show a Tip to the user when the user selects the iPhone 14 Pro (256 GB) – Purple product more than 2 times.

    The example below creates a didViewProductDetail event with an associated donation value and donates it anytime the ProductDetailsView appears:

    struct ProductDetailsView: View {
    static let didViewProductDetail = Tips.Event<DidViewProduct>(id: "didViewProductDetail")
    var product: ProductModel
    var body: some View {
        VStack(alignment: .leading) {
            HStack(alignment: .top, content: {
                Spacer()
                Image(product.productImage, bundle: Bundle.main)
                Spacer()
            })
            Text(product.productName)
                .font(.title3)
                .lineLimit(nil)
            Text(product.productPrice)
                .font(.title2)
            Text("Get it by Wednesday, 18 October")
                .font(.caption2)
                .lineLimit(nil)
            Spacer()
        }
        .padding()
        .onAppear {
            Self.didViewProductDetail.sendDonation(.init(productID: product.productID, productName: product.productName))
                }
    }
}

struct DidViewProduct: Codable, Sendable {
    let productID: UUID
    let productName: String
}

    The example below creates a display rule for ProductDetailsTip based on the didViewProductDetail event.

    struct ProductDetailsTip: Tip {
    var title: Text {
        Text("Add iPhone 14 Pro (256 GB) - Purple to your cart")
    }
    
    var message: Text? {
        Text("You can edit/remove the items from your cart")
    }
    
    var image: Image? {
        Image(systemName: "cart")
    }
    
    var rules: [Rule] {
        // Tip will only display when the didViewProductDetail event for product name 'iPhone 14 Pro (256 GB) - Purple' has been donated 3 or more times in a day.
        #Rule(ProductDetailsView.didViewProductDetail) {
            $0.donations.donatedWithin(.day).filter( { $0.productName == "iPhone 14 Pro (256 GB) - Purple" }).count >= 3
        }
    }
    
    var actions: [Action] {
        [
            Tip.Action(id: "add-product-to-cart", title: "Add to cart", perform: {
                print("Product added into the cart")
            })
        ]
    }
}

    Customization for Tip

    Customization is the key feature as every app has its own theme throughout the application. Customizing tips to gale along with application themes surely enhances the user experience. Although, as of now, there is not much customization offered by the TipKit framework, but we expect it to get upgraded in the future. Below are the available methods for customization of tips.

    public func tipAssetSize(_ size: CGSize) -> some View

public func tipCornerRadius(_ cornerRadius: Double, 
                            antialiased: Bool = true) -> some View

public func tipBackground(_ style: some ShapeStyle) -> some View

    Testing

    Testing tips is very important as a small issue in the implementation of this framework can ruin your app’s user experience. We can construct UI test cases for various scenarios, and tthe following methods can be helpful to test tips.

    • showAllTips
    • hideAllTips
    • showTips([<instance-of-your-tip>])
    • hideTips([<instance-of-your-tip>])

    Pros

    • Compatibility: TipKit is compatible across all the Apple platforms, including iOS, macOs, watchOs, visionOS.
    • Supports both SwiftUI and UIKit
    • Easy implementation and testing
    • Avoiding dependency on third-party libraries

    Cons 

    • Availability: Only available from iOS 17.0, iPadOS 17.0, macOS 14.0, Mac Catalyst 17.0, tvOS 17.0, watchOS 10.0 and visionOS 1.0 Beta. So no backwards compatibility as of now.
    • It might frustrate the user if the application incorrectly implements this framework

    Conclusion

    The TipKit framework is a great way to introduce new features in our application to the user. It is easy to implement, and it enhances the user experience. Having said that, we should avoid extensive use of it as it may frustrate the user. We should always avoid displaying promotional and error messages in the form of tips.

  • Optimizing iOS Memory Usage with Instruments Xcode Tool

    Introduction

    Developing iOS applications that deliver a smooth user experience requires more than just clean code and engaging features. Efficient memory management helps ensure that your app performs well and avoids common pitfalls like crashes and excessive battery drain. 

    In this blog, we’ll explore how to optimize memory usage in your iOS app using Xcode’s powerful Instruments and other memory management tools.

    Memory Management and Usage

    Before we delve into the other aspects of memory optimization, it’s important to understand why it’s so essential:

    Memory management in iOS refers to the process of allocating and deallocating memory for objects in an iOS application to ensure efficient and reliable operation. Proper memory management prevents issues like memory leaks, crashes, and excessive memory usage, which can degrade an app’s performance and user experience. 

    Memory management in iOS primarily involves the use of Automatic Reference Counting (ARC) and understanding how to manage memory effectively.

    Here are some key concepts and techniques related to memory management in iOS:

    1. Automatic Reference Counting (ARC): ARC is a memory management technique introduced by Apple to automate memory management in Objective-C and Swift. With ARC, the compiler automatically inserts retain, release, and autorelease calls, ensuring that memory is allocated and deallocated as needed. Developers don’t need to manually manage memory by calling “retain,” “release,” or “autorelease`” methods as they did in manual memory management in pre-ARC era.
    2. Strong and Weak References: In ARC, objects have strong, weak, and unowned references. A strong reference keeps an object in memory as long as at least one strong reference to it exists. A weak reference, on the other hand, does not keep an object alive. It’s commonly used to avoid strong reference cycles (retain cycles) and potential memory leaks.
    3. Retain Cycles: A retain cycle occurs when two or more objects hold strong references to each other, creating a situation where they cannot be deallocated, even if they are no longer needed. To prevent retain cycles, you can use weak references, unowned references, or break the cycle manually by setting references to “nil” when appropriate.
    4. Avoiding Strong Reference Cycles: To avoid retain cycles, use weak references (and unowned references when appropriate) in situations where two objects reference each other. Also, consider using closure capture lists to prevent strong reference cycles when using closures.
    5. Resource Management: Memory management also includes managing other resources like files, network connections, and graphics contexts. Ensure you release or close these resources when they are no longer needed.
    6. Memory Profiling: The Memory Report in the Debug Navigator of Xcode is a tool used for monitoring and analyzing the memory usage of your iOS or macOS application during runtime. It provides valuable insights into how your app utilizes memory, helps identify memory-related issues, and allows you to optimize the application’s performance.

    Also, use tools like Instruments to profile your app’s memory usage and identify memory leaks and excessive memory consumption.

    Instruments: Your Ally for Memory Optimization

    In Xcode, “Instruments” refer to a set of performance analysis and debugging tools integrated into the Xcode development environment. These instruments are used by developers to monitor and analyze the performance of their iOS, macOS, watchOS, and tvOS applications during development and testing. Instruments help developers identify and address performance bottlenecks, memory issues, and other problems in their code.

     

    Some of the common instruments available in Xcode include:

    1. Allocations: The Allocations instrument helps you track memory allocations and deallocations in your app. It’s useful for detecting memory leaks and excessive memory usage.
    2. Leaks: The Leaks instrument finds memory leaks in your application. It can identify objects that are not properly deallocated.
    3. Time Profiler: Time Profiler helps you measure and analyze the CPU usage of your application over time. It can identify which functions or methods are consuming the most CPU resources.
    4. Custom Instruments: Xcode also allows you to create custom instruments tailored to your specific needs using the Instruments development framework.

    To use these instruments, you can run your application with profiling enabled, and then choose the instrument that best suits your performance analysis goals. 

    Launching Instruments

    Because Instruments is located inside Xcode’s app bundle, you won’t be able to find it in the Finder. 

    To launch Instruments on macOS, follow these steps:

    1. Open Xcode: Instruments is bundled with Xcode, Apple’s integrated development environment for macOS, iOS, watchOS, and tvOS app development. If you don’t have Xcode installed, you can download it from the Mac App Store or Apple’s developer website.
    2. Open Your Project: Launch Xcode and open the project for which you want to use Instruments. You can do this by selecting “File” > “Open” and then navigating to your project’s folder.
    3. Choose Instruments: Once your project is open, go to the “Xcode” menu at the top-left corner of the screen. From the drop-down menu, select “Open Developer Tool” and choose “Instruments.”
    4. Select a Template: Instruments will open, and you’ll see a window with a list of available performance templates on the left-hand side. These templates correspond to the different types of analysis you can perform. Choose the template that best matches the type of analysis you want to conduct. For example, you can select “Time Profiler” for CPU profiling or “Leaks” for memory analysis.
    5. Configure Settings: Depending on the template you selected, you may need to configure some settings or choose the target process (your app) you want to profile. These settings can typically be adjusted in the template configuration area.
    6. Start Recording: Click the red record button in the top-left corner of the Instruments window to start profiling your application. This will launch your app with the selected template and begin collecting performance data.
    7. Analyze Data: Interact with your application as you normally would to trigger the performance scenarios you want to analyze. Instruments will record data related to CPU usage, memory usage, network activity, and other aspects of your app’s performance.
    8. Stop Recording: When you’re done profiling your app, click the square “Stop” button in Instruments to stop recording data.
    9. Analyze Results: After stopping the recording, Instruments will display a detailed analysis of your app’s performance. You can explore various graphs, timelines, and reports to identify and address performance issues.
    10. Save or Share Results: You can save your Instruments session for future reference or share it with colleagues if needed.

    Using the Allocations Instrument

    The “Allocations” instrument helps you monitor memory allocation and deallocation. Here’s how to use it:

    1. Start the Allocations Instrument: In Instruments, select “Allocations” as your instrument.

    2. Profile Your App: Use your app as you normally would to trigger the scenarios you want to profile.

    3. Examine the Memory Allocation Graph: The graph displays memory usage over time. Look for spikes or steady increases in memory usage.

    4. Inspect Objects: The instrument provides a list of objects that have been allocated and deallocated. You can inspect these objects and their associated memory usage.

    5. Call Tree and Source Code: To pinpoint memory issues, use the Call Tree to identify the functions or methods responsible for memory allocation. You can then inspect the associated source code in the Source View.

    Detecting Memory Leaks with the Leaks Instrument

    Retain Cycle

    A retain cycle in Swift occurs when two or more objects hold strong references to each other in a way that prevents them from being deallocated, causing a memory leak. This situation is also known as a “strong reference cycle.” It’s essential to understand retain cycles because they can lead to increased memory usage and potential app crashes.  

    A common scenario for retain cycles is when two objects reference each other, both using strong references. 

    Here’s an example to illustrate a retain cycle:

    class Person {
    var name: String
    var pet: Pet?

    init(name: String) {
        self.name = name
    }

    deinit {
        print("(name) has been deallocated")
    }
}

class Pet {
    var name: String
    var owner: Person?

    init(name: String) {
        self.name = name
    }

    deinit {
        print("(name) has been deallocated")
    }
}

var rohit: Person? = Person(name: "Rohit")
var jerry: Pet? = Pet(name: "Jerry")

rohit?.pet = jerry
jerry?.owner = rohit

rohit = nil
jerry = nil

    In this example, we have two classes, Person and Pet, representing a person and their pet. Both classes have a property to store a reference to the other class (person.pet and pet.owner).  

    The “Leaks” instrument is designed to detect memory leaks in your app. 

    Here’s how to use it:

    1. Launch Instruments in Xcode: First, open your project in Xcode.  

    2. Commence Profiling: To commence the profiling process, navigate to the “Product” menu and select “Profile.”  

    3. Select the Leaks Instrument: Within the Instruments interface, choose the “Leaks” instrument from the available options.  

    4. Trigger the Memory Leak Scenario: To trigger the scenario where memory is leaked, interact with your application. This interaction, such as creating a retain cycle, will induce the memory leak.

    5. Identify Leaked Objects: The Leaks Instrument will automatically detect and pinpoint the leaked objects, offering information about their origins, including backtraces and the responsible callers.  

    6. Analyze Backtraces and Responsible Callers: To gain insights into the context in which the memory leak occurred, you can inspect the source code in the Source View provided by Instruments.  

    7. Address the Leaks: Armed with this information, you can proceed to fix the memory leaks by making the necessary adjustments in your code to ensure memory is released correctly, preventing future occurrences of memory leaks.

    You should see memory leaks like below in the Instruments.

    The issue in the above code is that both Person and Pet are holding strong references to each other. When you create a Person and a Pet and set their respective references, a retain cycle is established. Even when you set rohit and jerry to nil, the objects are not deallocated, and the deinit methods are not called. This is a memory leak caused by the retain cycle. 

    To break the retain cycle and prevent this memory leak, you can use weak or unowned references. In this case, you can make the owner property in Pet a weak reference because a pet should not own its owner:

    class Pet {
    var name: String
    weak var owner: Person?

    init(name: String) {
        self.name = name
    }

    deinit {
        print("(name) has been deallocated")
    }
}

    By making owner a weak reference, the retain cycle is broken, and when you set rohit and jerry to nil, the objects will be deallocated, and the deinit methods will be called. This ensures proper memory management and avoids memory leaks.

    Best Practices for Memory Optimization

    In addition to using Instruments, consider the following best practices for memory optimization:

    1. Release Memory Properly: Ensure that memory is released when objects are no longer needed.

    2. Use Weak References: Use weak references when appropriate to prevent strong reference cycles.

    3. Using Unowned to break retain cycle: An unowned reference does not increment or decrease an object’s reference count. 

    3. Minimize Singletons and Global Variables: These can lead to retained objects. Use them judiciously.

    4. Implement Lazy Loading: Load resources lazily to reduce initial memory usage.

    Conclusion

    Optimizing memory usage is an essential part of creating high-quality iOS apps. 

    Instruments, integrated into Xcode, is a versatile tool that provides insights into memory allocation, leaks, and CPU-intensive code. By mastering these tools and best practices, you can ensure your app is memory-efficient, stable, and provides a superior user experience. Happy profiling!

  • Choosing Between Scrum and Kanban: Finding the Right Fit for Your Project

    In project management and software development, two popular methodologies have emerged as dominant players: Scrum and Kanban. Both offer effective frameworks for managing work, but they have distinct characteristics and applications. Deciding which one to choose for your project can be a critical decision that significantly impacts its success. In this blog post, we’ll dive deep into Scrum and Kanban, exploring their differences, advantages, and when to use each to help you make an informed decision.

    Understanding Scrum

    Scrum is an agile framework that originated in the early 1990s and has since gained widespread adoption across various industries. The structured approach emphasizes teamwork, collaboration, and iterative development. Scrum divides work into time-bound iterations called “sprints,” typically lasting two to four weeks, during which a cross-functional team works to deliver a potentially shippable product increment.

    Key Principles of Scrum:

    1. Roles: Scrum involves three key roles: the Product Owner, Scrum Master, and Development Team. Each has distinct responsibilities to ensure smooth project progress.

    2. Artifacts: Scrum employs several artifacts, including the Product Backlog, Sprint Backlog, and Increment, to maintain transparency and track progress.

    3. Events: Scrum events, such as Sprint Planning, Daily Standup, Sprint Review, and Sprint Retrospective, provide a structured framework for communication and planning.

    Advantages vs. Disadvantages of Scrum:

    ‍

    Advantages:

    1. Predictable Delivery: Scrum sprints’ time-boxed nature allows for predictable delivery timelines, making it suitable for projects with fixed deadlines.

    Example: Consider a software development project with a hard release date. By using Scrum, the team can plan sprints to ensure that key features are completed in time for the release. This predictability is crucial when dealing with external commitments.

    2. Continuous Improvement: Regular Sprint Retrospectives encourage teams to reflect on their work and identify areas for improvement, fostering a culture of continuous learning and adaptation.

    3. Clear Prioritization: The Product Backlog prioritizes the most important features and tasks, helping teams focus on delivering value.

    4. Enhanced Collaboration: Cross-functional teams collaborate closely throughout the project, leading to better communication and a shared understanding of project goals.

    Disadvantages:

    1. Rigidity: Scrum’s structured nature can be considered too rigid for some projects. It may not adapt to highly variable workloads or projects with frequently changing requirements.

    2. Role Dependencies: The success of Scrum heavily relies on having skilled and dedicated Product Owners, Scrum Masters, and Development Team members. Without these roles, the methodology’s effectiveness can be compromised.

    3. Learning Curve: Implementing Scrum effectively requires a thorough understanding of its principles and practices. Teams new to Scrum may face a learning curve that impacts productivity initially.

    Understanding Kanban

    Kanban, which means “visual signal” in Japanese, originated at Toyota in the 1940s as a manufacturing system. Kanban is a visual and flow-based method for managing work in project management. Unlike Scrum, Kanban does not prescribe specific roles, time-boxed iterations, or ceremonies. Instead, it provides a flexible framework for visualizing work, setting work-in-progress (WIP) limits, and optimizing workflow.

    Key Principles of Kanban

    1. Visualize Work: Kanban boards display work items in columns representing various workflow stages, clearly representing tasks and their status.

    2. Limit Work in Progress (WIP): Kanban sets WIP limits for each column to prevent overloading team members and maintain a steady workflow.

    3. Manage Flow: Kanban focuses on optimizing workflow, ensuring that tasks move smoothly from one stage to the next.

    Advantages vs. Disadvantages of Kanban:

    Advantages:

    1. Flexibility: Kanban adapts well to changing project requirements and workloads, making it ideal for highly variable projects.

    Example: Imagine a customer support team using Kanban. They may receive a varying number of support requests daily. Kanban allows them to adjust their work capacity based on demand, ensuring they can address urgent issues without overloading their resources.

    2. Reduced Waste: By limiting WIP, Kanban minimizes multitasking and reduces wasted effort, improving efficiency.

    3. Continuous Delivery: Teams using Kanban can continuously release features or products, allowing faster response to customer needs.

    4. Simplicity: Kanban is easy to understand and implement, making it accessible to teams with varying experience levels.

    Disadvantages:

    1. Lack of Structure: While flexibility is a strength, Kanban’s lack of defined roles and ceremonies can be a drawback for teams that thrive in more structured environments. It may lead to a lack of clarity in responsibilities and expectations.

    2. Difficulty in Predictability: Kanban’s focus on continuous flow can make it challenging to predict when specific features or tasks will be completed. This can be problematic for projects with strict deadlines.

    3. Dependency on Team Discipline: Kanban relies heavily on the discipline of team members to follow WIP limits and maintain flow. Without a disciplined team, the system can break down.

    Choosing Between Scrum and Kanban

    The decision between Scrum and Kanban should be based on your project’s specific characteristics, requirements, and organizational culture. To help you make an informed choice, consider the following factors:

    Hybrid Approaches

    In some cases, project teams may opt for hybrid approaches that combine elements of both Scrum and Kanban to suit their unique needs. For example, a team might use Scrum for high-priority feature development while employing Kanban for bug fixes and support requests.

    When to Use Scrum and Kanban Together

    In certain situations, it may be advantageous to use both Scrum and Kanban in conjunction to harness the strengths of both methodologies:

    Conclusion

    Choosing between Scrum and Kanban ultimately depends on your project’s specific needs and characteristics, your team’s experience and culture, and the level of flexibility required. Both methodologies offer valuable tools and principles for managing work effectively, but they do so in different ways.

    Scrum provides structure and predictability through time-boxed sprints, making it suitable for projects with defined requirements and fixed deadlines. On the other hand, Kanban offers flexibility and adaptability, making it an excellent choice for projects with variable workloads and evolving requirements.

    Remember that these are not rigid choices; you can always tailor the methodology to fit your project’s unique circumstances. The key is to stay true to the underlying principles of agility, collaboration, and continuous improvement, regardless of whether you choose Scrum, Kanban, or a hybrid approach. Doing so will increase your chances of project success and deliver value to your stakeholders.

  • Unlocking Legal Insights: Effortless Document Summarization with OpenAI’s LLM and LangChain

    ‍

    The Rising Demand for Legal Document Summarization:

    • In a world where data, information, and legal complexities is prevalent, the volume of legal documents is growing rapidly. Law firms, legal professionals, and businesses are dealing with an ever-increasing number of legal texts, including contracts, court rulings, statutes, and regulations. 
    • These documents contain important insights, but understanding them can be overwhelming. This is where the demand for legal document summarization comes in. 
    • In this blog, we’ll discuss the increasing need for summarizing legal documents and how modern technology is changing the way we analyze legal information, making it more efficient and accessible.

    Overview OpenAI and LangChain

    • We’ll use the LangChain framework to build our application with LLMs. These models, powered by deep learning, have been extensively trained on large text datasets. They excel in various language tasks like translation, sentiment analysis, chatbots, and more. 
    • LLMs can understand complex text, identify entities, establish connections, and generate coherent content. We can use meta LLaMA LLMs, OpenAI LLMs and others as well. For this case, we will be using OpenAI’s LLM.

    ‍

    • OpenAI is a leader in the field of artificial intelligence and machine learning. They have developed powerful Large Language Models (LLMs) that are capable of understanding and generating human-like text.
    •  These models have been trained on vast amounts of textual data and can perform a wide range of natural language processing tasks.

    LangChain is an innovative framework designed to simplify and enhance the development of applications and systems that involve natural language processing (NLP) and large language models (LLMs). 

    It provides a structured and efficient approach for working with LLMs like OpenAI’s GPT-3 and GPT-4 to tackle various NLP tasks. Here’s an overview of LangChain’s key features and capabilities:

    • Modular NLP Workflow: Build flexible NLP pipelines using modular blocks. 
    • Chain-Based Processing: Define processing flows using chain-based structures. 
    • Easy Integration: Seamlessly integrate LangChain with other tools and libraries.
    • Scalability: Scale NLP workflows to handle large datasets and complex tasks. 
    • Extensive Language Support: Work with multiple languages and models. 
    • Data Visualization: Visualize NLP pipeline results for better insights.
    • Version Control: Track changes and manage NLP workflows efficiently. 
    • Collaboration: Enable collaborative NLP development and experimentation.

    Setting Up Environment

    Setting Up Google Colab

    Google Colab provides a powerful and convenient platform for running Python code with the added benefit of free GPU support. To get started, follow these steps:

    1. Visit Google Colab: Open your web browser and navigate to Google Colab.
    2. Sign In or Create a Google Account: You’ll need to sign in with your Google account to use Google Colab. If you don’t have one, you can create an account for free.
    3. Create a New Notebook: Once signed in, click on “New Notebook” to create a new Colab notebook.
    4. Choose Python Version: In the notebook, click on “Runtime” in the menu and select “Change runtime type.” Choose your preferred Python version (usually Python 3) and set the hardware accelerator to “GPU.” Also, make sure to turn on the “Internet” toggle.

    OpenAI API Key Generation:-

    1. Visit the OpenAI Website Go to the OpenAI website.
    2.  Sign In or Create an Account Sign in or create a new OpenAI account. 
    3. Generate a New API Key Access the API section and generate a new API key. 
    4. Name Your API Key Give your API key a name that reflects its purpose. 
    5. Copy the API Key Copy the generated API key to your clipboard. 
    6. Store the API Key Safely Securely store the API key and do not share it publicly.

    Understanding Legal Document Summarization Workflow

    1. Map Step:

    • At the heart of our legal document summarization process is the Map-Reduce paradigm.
    • In the Map step, we treat each legal document individually. Think of it as dissecting a large puzzle into smaller, manageable pieces.
    • For each document, we employ a sophisticated Language Model (LLM). This LLM acts as our expert, breaking down complex legal language and extracting meaningful content.
    • The LLM generates concise summaries for each document section, essentially translating legalese into understandable insights.
    • These individual summaries become our building blocks, our pieces of the puzzle.

    2. Reduce Step:

    • Now, let’s shift our focus to the Reduce step.
    • Here’s where we bring everything together. We’ve generated summaries for all the document sections, and it’s time to assemble them into a cohesive whole.
    • Imagine the Reduce step as the puzzle solver. It takes all those individual pieces (summaries) and arranges them to form the big picture.
    • The goal is to produce a single, comprehensive summary that encapsulates the essence of the entire legal document.

    3. Compression – Ensuring a Smooth Fit:

    • One challenge we encounter is the potential length of these individual summaries. Some legal documents can produce quite lengthy summaries.
    • To ensure a smooth flow within our summarization process, we’ve introduced a compression step.

    4. Recursive Compression:

    • In some cases, even the compressed summaries might need further adjustment.
    • That’s where the concept of recursive compression comes into play.
    • If necessary, we’ll apply compression multiple times, refining and optimizing the summaries until they seamlessly fit into our summarization pipeline.

    Let’s Get Started

    Step 1: Installing python libraries

    Create a new notebook in Google Colab and install the required Python libraries.

    !pip install openai langchain tiktoken

    OpenAI: Installed to access OpenAI’s powerful language models for legal document summarization.

    LangChain: Essential for implementing document mapping, reduction, and combining workflows efficiently.

    Tiktoken: Helps manage token counts within text data, ensuring efficient usage of language models and avoiding token limit issues.

    Step 2: Adding OpenAI API key to Colab

    Integrate your openapi key in Google Colab Secrets.

    from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()
API_KEY= user_secrets.get_secret("YOUR_SECRET_KEY_NAME")

    Step 3: Initializing OpenAI LLM

    Here, we import the OpenAI module from LangChain and initialize it with the provided API key to utilize advanced language models for document summarization.

    from langchain.llms import OpenAI
llm = OpenAI(openai_api_key=API_KEY)

    Step 4: Splitting text by Character

    The Text Splitter, in this case, overcomes the token limit by breaking down the text into smaller chunks that are each within the token limit. This ensures that the text can be processed effectively by the language model without exceeding its token capacity. 

    The “chunk_overlap” parameter allows for some overlap between chunks to ensure that no information is lost during the splitting process.

    from langchain.text_splitter import CharacterTextSplitter
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=1000, chunk_overlap=120
)

    Step 5 : Loading PDF documents

    from langchain.document_loaders import PyPDFLoader
def chunks(pdf_file_path):
    loader = PyPDFLoader(pdf_file_path)
    docs = loader.load_and_split()
    return docs

    It initializes a PyPDFLoader object named “loader” using the provided PDF file path. This loader is responsible for loading and processing the contents of the PDF file. 

    It then uses the “loader” to load and split the PDF document into smaller “docs” or document chunks. These document chunks likely represent different sections or pages of the PDF file. 

    Finally, it returns the list of document chunks, making them available for further processing or analysis.

    Step 6: Map Reduce Prompt Templates

    Import libraries required for the implementation of LangChain MapReduce.

    from langchain.chains.mapreduce import MapReduceChain
from langchain.text_splitter import CharacterTextSplitter
from langchain.chains import ReduceDocumentsChain, MapReduceDocumentsChain
from langchain import PromptTemplate
from langchain.chains import LLMChain
from langchain.chains.combine_documents.stuff import StuffDocumentsChain

    map_template = """The following is a set of documents
{docs}
Based on this list of docs, summarised into meaningful
Helpful Answer:"""

map_prompt = PromptTemplate.from_template(map_template)
map_chain = LLMChain(llm=llm, prompt=map_prompt)

reduce_template = """The following is set of summaries:
{doc_summaries}
Take these and distil it into a final consolidated summary with title(mandatory) in bold with important key points . 
Helpful Answer:"""

reduce_prompt = PromptTemplate.from_template(reduce_template)
reduce_chain = LLMChain(llm=llm, prompt=reduce_prompt)

    Template Definition: 

    The code defines two templates, map_template and reduce_template, which serve as structured prompts for instructing a language model on how to process and summarise sets of documents. 

    LLMChains for Mapping and Reduction: 

    Two LLMChains, map_chain, and reduce_chain, are configured with these templates to execute the mapping and reduction steps in the document summarization process, making it more structured and manageable.

    Step 7 : Map and Reduce LLM Chains

    combine_documents_chain = StuffDocumentsChain(
    llm_chain=reduce_chain, document_variable_name="doc_summaries"
)
reduce_documents_chain = ReduceDocumentsChain(
    combine_documents_chain=combine_documents_chain,
    collapse_documents_chain=combine_documents_chain,
    token_max=5000,
)

    map_reduce_chain = MapReduceDocumentsChain(
    llm_chain=map_chain,
    reduce_documents_chain=reduce_documents_chain,
    document_variable_name="docs",
    return_intermediate_steps=False,
)

    Combining Documents Chain (combine_documents_chain): 

    • This chain plays a crucial role in the document summarization process. It takes the individual legal document summaries, generated in the “Map” step, and combines them into a single, cohesive text string. 
    • By consolidating the summaries, it prepares the data for further processing in the “Reduce” step. The resulting combined document string is assigned the variable name “doc_summaries.” 

    Reduce Documents Chain (reduce_documents_chain): 

    • This chain represents the final phase of the summarization process. Its primary function is to take the combined document string from the combine_documents_chain and perform in-depth reduction and summarization. 
    • To address potential issues related to token limits (where documents may exceed a certain token count), this chain offers a clever solution. It can recursively collapse or compress lengthy documents into smaller, more manageable chunks. 
    • This ensures that the summarization process remains efficient and avoids token limit constraints. The maximum token limit for each chunk is set at 5,000 tokens, helping control the size of the summarization output. 

    Map-Reduce Documents Chain (map_reduce_chain): 

    • This chain follows the well-known MapReduce paradigm, a framework often used in distributed computing for processing and generating large datasets. In the “Map” step, it employs the map_chain to process each individual legal document. 
    • This results in initial document summaries. In the subsequent “Reduce” step, the chain uses the reduce_documents_chain to consolidate these initial summaries into a final, comprehensive document summary. 
    • The summarization result, representing the distilled insights from the legal documents, is stored in the variable named “docs” within the LLM chain. 

    Step 8: Summarization Function

    def summarize_pdf(file_path):
    split_docs = text_splitter.split_documents(chunks(file_path))
    return map_reduce_chain.run(split_docs)

result_sumary=summarize_pdf(file_path)
print(result_summary)

    Our summarization process centers around the ‘summarize_pdf’ function. This function takes a PDF file path as input and follows a two-step approach. 

    First, it splits the PDF into manageable sections using the ‘text_splitter’ module. Then, it runs the ‘map_reduce_chain,’ which handles the summarization process. 

    By providing the PDF file path as input, you can easily generate a concise summary of the legal document within the Google Colab environment, thanks to LangChain and LLM.

    Output

    1. Original Document – https://www.safetyforward.com/docs/legal.pdf

    This document is about not using mobile phones while driving a motor vehicle and prohibits disabling its motion restriction features.

    Summarization –

    2. Original Document – https://static.abhibus.com/ks/pdf/Loan-Agreement.pdf

    India and the International Bank for Reconstruction and Development have formed an agreement for the Sustainable Urban Transport Project, focusing on sustainable transportation while adhering to anti-corruption guidelines.

    Summarization –

    Limitations :

    Complex Legal Terminology: 

    LLMs may struggle with accurately summarizing documents containing intricate legal terminology, which requires domain-specific knowledge to interpret correctly. 

    Loss of Context: 

    Summarization processes, especially in lengthy legal documents, may result in the loss of important contextual details, potentially affecting the comprehensiveness of the summaries. 

    Inherent Bias: 

    LLMs can inadvertently introduce bias into summaries based on the biases present in their training data. This is a critical concern when dealing with legal documents that require impartiality. 

    Document Structure: 

    Summarization models might not always understand the hierarchical or structural elements of legal documents, making it challenging to generate summaries that reflect the intended structure.

    Limited Abstraction: 

    LLMs excel at generating detailed summaries, but they may struggle with abstracting complex legal arguments, which is essential for high-level understanding.

    Conclusion : 

    • In a nutshell, this project uses LangChain and OpenAI’s LLM to bring in a fresh way of summarizing legal documents. This collaboration makes legal document management more accurate and efficient.
    • However, we faced some big challenges, like handling lots of legal documents and dealing with AI bias. As we move forward, we need to find new ways to make our automated summarization even better and meet the demands of the legal profession.
    • In the future, we’re committed to improving our approach. We’ll focus on fine-tuning algorithms for more accuracy and exploring new techniques, like combining different methods, to keep enhancing legal document summarization. Our aim is to meet the ever-growing needs of the legal profession.

  • Unlocking Key Insights in NATS Development: My Journey from Novice to Expert – Part 1

    By examining my personal journey from a NATS novice to mastering its intricacies, this long-form article aims to showcase the importance and applicability of NATS in the software development landscape. Through comprehensive exploration of various topics, readers will gain a solid foundation, advanced techniques, and best practices for leveraging NATS effectively in their projects.

    Introduction

    Our topic for today is about how you can get started with NATS. We are assuming that you are aware of why you need NATS and want to know the concepts of NATS along with a walkthrough for how to deploy those concepts/components in your organization.

    The first part would include the basic concept and Installation Guide, and setup, admin-related CRUD operations, shell scripts which might not be needed immediately but would be good to have in your arsenal. Whereas the second part would be more developer-focused – applying NATS in application, etc. Let’s Begin

    Understanding NATS

    In this section, we will delve into the fundamentals of NATS and its key components.

    A. Definition and Overview

    Nats, which stands for “Naturally Adaptable and Transparent System,” is a lightweight, high-performance messaging system known for its simplicity and scalability. It enables the exchange of messages between applications in a distributed architecture, allowing for seamless communication and increased efficiency.

    B. Architecture Diagram

    To better understand the inner workings of NATS, let’s take a closer look at its architecture. The diagram below illustrates the key components involved in a typical Nats deployment:

    C. Key Features

    NATS offers several key features that make it a powerful messaging system. These include:

    • Publish-Subscribe Model: NATS follows a publish-subscribe model where publishers send messages to subjects and subscribers receive those messages based on their interest in specific subjects.
    • This model allows for flexible and decoupled communication between different parts of an application or across multiple applications.‍
    • Scalability: With support for horizontal scaling, NATS can handle high loads of message traffic, making it suitable for large-scale systems.‍
    • Performance: NATS is built for speed, providing low-latency message delivery and high throughput.‍
    • Reliability: NATS ensures that messages are reliably delivered to subscribers, even in the presence of network interruptions or failures.‍
    • Security: NATS supports secure communication through various authentication and encryption mechanisms, protecting sensitive data.

    D. Use Cases and Applications

    NATS’ simplicity and versatility make it suitable for a wide range of use cases and applications. Some common use cases include:

    • Real-time data streaming and processing
    • Event-driven architectures
    • Microservices communication
    • IoT (Internet of Things) systems
    • Distributed systems and cloud-native applications

    E. Concepts

    To better grasp the various components and terminologies associated with NATS, let’s explore some key concepts:

    1. NATS server: The NATS server acts as the central messaging infrastructure, responsible for routing messages between publishers and subscribers.
    2. NATS CLI: The NATS command-line interface (CLI) is a tool that provides developers with a command-line interface to interact with the NATS server and perform various administrative tasks.
    3. NATS clients: NATS CLI and clients both are different. The NATS client is an API/code-based approach to access the NATS server. Clients are not as powerful as CLI but are mainly used along with source code to achieve a specified goal. We won’t be covering this as it is not part of the scope.
    4. Routes: Routes allow NATS clusters to bridge and share messages with other nodes within and outside clusters, enabling communication across geographically distributed systems.
    5. Accounts: Accounts in NATS provide isolation and access control mechanisms, ensuring that messages are exchanged securely and only between authorized parties.
    6. Gateway: Gateways list all the servers in different clusters that you want to connect in order to create a supercluster.
    7. SuperCluster: SuperCluster is a powerful feature that allows scaling NATS horizontally across multiple clusters, providing enhanced performance and fault tolerance.

    F. System Requirements

    Before diving into NATS, it’s important to ensure that our system meets the necessary requirements. The system requirements for NATS will vary depending on the specific deployment scenario and use case. However, in general, the minimum requirements include:

    Hardware:

    Network:

    • All the VMs should be part of the same cluster.
    • 4222, 8222, 4248, and 7222 ports should be open for inter-server and client connection.
    • Whitelisting of GitHub EMU account on prod servers (Phase 2).
    • Get AVI VIP for all the clusters from the network team.

    Logs:

    By default, logs will be disabled, but the configuration file will have placeholders for logs enablement. Some of the important changes include:

    • debug: It will show system logs in verbose.
    • trace: It will record every message processed on NATS.
    • logtime, logfile_size_limit, log_file: As the name represents, it will show the time when recording the logs, individual file limit for log files (once filled, auto rotation is done by NATS), and the name of the file, respectively.

    TLS:

    I will be showing the configuration of how to use the certs. Do remember, this setup is being done for the development environment to allow more flexibility towards explaining things and executing it.

    Getting Started with NATS

    In this section, we will guide you through the installation and setup process for NATS.

    Building the Foundation

    First, we will focus on building a strong foundation in NATS by understanding its core concepts and implementing basic messaging patterns.

    A. Understanding NATS Subjects

    In NATS, subjects serve as identifiers that help publishers and subscribers establish communication channels. They are represented as hierarchical strings, allowing for flexibility in message routing and subscription matching.

    B. Exploring Messages, Publishers, and Subscribers

    Messages are the units of data exchanged between applications through NATS. Publishers create and send messages, while subscribers receive and process them based on their subscribed subjects of interest.

    C. Implementing Basic Pub/Sub Pattern

    The publish-subscribe pattern is a fundamental messaging pattern in NATS. It allows publishers to distribute messages to multiple subscribers interested in specific subjects, enabling decoupled and efficient communication between different parts of the system.

    D. JetStream

    JetStream is an advanced addition to NATS that provides durable, persistent message storage and retention policies. It is designed to handle high-throughput streaming scenarios while ensuring data integrity and fault tolerance.

    E. Single Cluster vs. SuperCluster

    NATS supports both single clusters and superclusters. Single clusters are ideal for smaller deployments, whereas superclusters provide the ability to horizontally scale NATS across multiple clusters, enhancing performance and fault tolerance.

    Implementation

    As this blog is about deploying NATS from an admin perspective. We will be using only shell script for this purpose.

    Let’s start with the implementation process:

    Prerequisite

    These commands are required to be run on all the servers hosting Nats-server. In this blog, we will cover a 3-node cluster that will be working at Jetstream.

    Installing NatsCLI and Nats-server:

    mkdir -p rpm

    # NATSCLI

    curl -o rpm/nats-0.0.35.rpm  -L https://github.com/nats-io/natscli/releases/download/v0.0.35/nats-0.0.35-amd64.rpm

    sudo yum install -y rpm/nats-0.0.35.rpm

    # NATS-server

    curl -o rpm/nats-server-2.9.20.rpm  -L https://github.com/nats-io/nats-server/releases/download/v2.9.20/nats-server-v2.9.20-amd64.rpm

    sudo yum install -y rpm/nats-server-2.9.20.rpm

    Local Machine Setup for JetStream:

    # Create User

    sudo useradd –system –home /nats –shell /bin/false nats

    # Jetstream Storage

    sudo mkdir -p /nats/storage

    # Certs

    sudo mkdir -p /nats/certs

    # Logs

    sudo mkdir -p /nats/logs

    # Setting Right Permission

    sudo chown –recursive nats:nats /nats

    sudo chmod 777 /nats

    sudo chmod 777 /nats/storage

    Next, we will create the service file in the servers at /etc/systemd/system/nats.service

    sudo bash -c ‘cat <<EOF > /etc/systemd/system/nats.service

    [Unit]

    Description=NATS Streaming Daemon

    Requires=network-online.target

    After=network-online.target

    ConditionFileNotEmpty=/nats/nats.conf

    [Service]

    #Type=notify

    User=nats

    Group=nats

    ExecStart=/usr/local/bin/nats-server -config=/nats/nats.conf

    #KillMode=process

    Restart=always

    RestartSec=10

    StandardOutput=syslog

    StandardError=syslog

    #TimeoutSec=900

    #LimitNOFILE=65536

    #LimitMEMLOCK=infinity

    [Install]

    WantedBy=multi-user.target

    EOF’

    Full File will look like:

    #!/bin/bash

mkdir -p rpm

# NATSCLI
curl -o rpm/nats-0.0.35.rpm  -L https://github.com/nats-io/natscli/releases/download/v0.0.35/nats-0.0.35-amd64.rpm
sudo yum install -y rpm/nats-0.0.35.rpm

# NATS-server
curl -o rpm/nats-server-2.9.20.rpm  -L https://github.com/nats-io/nats-server/releases/download/v2.9.20/nats-server-v2.9.20-amd64.rpm
sudo yum install -y rpm/nats-server-2.9.20.rpm

# Create User
sudo useradd --system --home /nats --shell /bin/false nats

# Jetstream Storage
sudo mkdir -p /nats/storage

# Certs
sudo mkdir -p /nats/certs

# Logs
sudo mkdir -p /nats/logs

# Setting Right Permission
sudo chown --recursive nats:nats /nats
sudo chmod 777 /nats
sudo chmod 777 /nats/storage
sudo bash -c 'cat <<EOF > /etc/systemd/system/nats.service
[Unit]
Description=NATS Streaming Daemon
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/nats/nats.conf
[Service]
#Type=notify
User=nats
Group=nats
ExecStart=/usr/local/bin/nats-server -config=/nats/nats.conf
#KillMode=process
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
#TimeoutSec=900
#LimitNOFILE=65536
#LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target
EOF'

    Creating conf file at all the servers at /nats directory

    Server setup

    server_name=nts0

    listen: <IP/DNS-First>:4222 # For other servers edit the IP/DNS remaining in the cluster

    https: <DNS-First>:8222

    #http: <IP/DNS-First>:8222 # Uncommnet this if you are running without tls certs 

    JetStream Configuration

    jetstream {

      store_dir=/nats/storage

      max_mem_store: 6GB

      max_file_store: 90GB

    }

    Intra Cluster Setup

    cluster {

      name: dev-nats # Super Cluster should have unique Cluster names

      host: <IP/DNS-First>

      port: 4248

      routes = [

        nats-route://<IP/DNS-First>:4248

        nats-route://<IP/DNS-Second>:4248

        nats-route://<IP/DNS-Third>:4248

      ]

    }

    Account Setup

    accounts: {

      $SYS: {

        users: [

          { user: admin, password: password }

        ]

      },

      B: {

        users: [

          {user: b, password: b}

        ],

        jetstream: enabled,

        imports: [

        # {stream: {account: “$G”}}

        ]

      },

      C: {

        users: [

          {user: c, password: c}

        ],

        jetstream: enabled,

        imports: [

        ]

      },

      E: {

        users: [

          {user: e, password: e}

        ],

        jetstream: enabled,

        imports: [

        ]

      }

    }

    no_auth_user: e # Change this on every server to have a user in the system which does not need password, allowing local account in supercluster

    We can use “Accounts” to help us provide local and global stream separation, the configuration is identical except for the changes in the no_auth_user which must be unique for each cluster, making the stream only accessible from the given cluster without the need of providing credentials exclusively.

    Gateway Setup: 

    Intra Cluster/Route Setup and Account Setup remain similar and need to be present in another cluster with the cluster having the name “new-dev-nats.”

    gateway {

      name: dev-nats

      listen: <IP/DNS-First>:7222

      gateways: [

        {name: dev-nats, urls: [nats://<IP/DNS-First>:7222, nats://<IP/DNS-Second>:7222, nats://<IP/DNS-Third>:7222]},

        {name: new-dev-nats, urls: [nats://<NEW-IP/DNS-First>:7222, nats://<NEW-IP/DNS-Second>:7222, nats://<NEW-IP/DNS-Third>:7222]}

      ]

    }

    TLS setup

    tls: {

      cert_file: “/nats/certs/natsio.crt”

      key_file: “/nats/certs/natsio.key”

      ca_file: “/nats/certs/natsio_rootCA.pem”

    }

    NOTE: no_auth_user: b is a special directive within NATS. If you choose to keep it seperate accross all the nodes in the cluster, you can have a “local account” setup in supercluster. This is beneficial when you want to publish data which should not be accessible by any other server.

    Complete conf file on <IP/DNS-First> machine would look like this:

    # `server_name`: Unique name for your node; attaching a number with increment value is recommended
# listen: DNS name for the current node:4222
# https: DNS name for the current node:8222
# cluster.name: This is the name of your cluster. It is compulsory for them to be the same across all nodes.
# cluster.host: DNS name for the current node
# cluster.routes: List of all the DNS entries which will be part of the cluster in separate lines:4248
# account.user: Make sure to use proper names here and also keep the same across all the nodes which will be involved as a super cluster
# no_auth_user: To be unique for individual cluster
# gateway.name: Should be for the current cluster the node is part of. (Best to match with cluster.name mentioned above)
# gateway.listen: The same logic mentioned for listen is applicable here with port 7222
# gateways:Mention all the nodes in all the cluster here with nodes separated logically by the cluster they are part of via name
# tls: Make sure to have the certs ready to place at /nats/certs

server_name=nts0
listen: <IP/DNS-First>:4222 # For other servers edit the IP/DNS remaining in the cluster
https: <DNS-First>:8222
#http: <IP/DNS-First>:8222 # Uncommnet this if you are running without tls certs 

jetstream {
  store_dir=/nats/storage
  max_mem_store: 6GB
  max_file_store: 90GB
}

cluster {
  name: dev-nats # Super Cluster should have unique Cluster names
  host: <IP/DNS-First>
  port: 4248
  routes = [
    nats-route://<IP/DNS-First>:4248
    nats-route://<IP/DNS-Second>:4248
    nats-route://<IP/DNS-Third>:4248
  ]
}

accounts: {
  $SYS: {
    users: [
      { user: admin, password: password }
    ]
  },
  B: {
    users: [
      {user: b, password: b}
    ],
    jetstream: enabled,
    imports: [
    # {stream: {account: "$G"}}
    ]
  },
  C: {
    users: [
      {user: c, password: c}
    ],
    jetstream: enabled,
    imports: [
    ]
  },
  E: {
    users: [
      {user: e, password: e}
    ],
    jetstream: enabled,
    imports: [
    ]
  }
}

no_auth_user: e # Change this on every server to have a user in the system which does not need password, allowing local account in supercluster

gateway {
  name: dev-nats
  listen: <IP/DNS-First>:7222
  gateways: [
    {name: dev-nats, urls: [nats://<IP/DNS-First>:7222, nats://<IP/DNS-Second>:7222, nats://<IP/DNS-Third>:7222]},
    {name: new-dev-nats, urls: [nats://<NEW-IP/DNS-First>:7222, nats://<NEW-IP/DNS-Second>:7222, nats://<NEW-IP/DNS-Third>:7222]}
  ]
}

tls: {
  cert_file: "/nats/certs/natsio.crt"
  key_file: "/nats/certs/natsio.key"
  ca_file: "/nats/certs/natsio_rootCA.pem"
}

    Recap on Conf File Changes

    The configuration file in all the nodes for all the environments will need to be updated, to support “gateway” and “accounts.”

    • Individual changes on all the conf files need to be done.
    • Changes for the gateway will be almost similar except for the change in the name, which will be specific to the local cluster of which the given node is part of.
    • Changes for an “account” will be almost similar except for the “no_auth_user” parameter, which will be specific to the local cluster of which the given node is part of.
    • The “nats-server –signal reload” command should be able to pick up the changes.

    Starting Service

    After adding the certs, re-own the files:

    sudo chown –recursive nats:nats /nats

    Creating firewall rules:

    sudo firewall-cmd –permanent –add-port=4222/tcp

    sudo firewall-cmd –permanent –add-port=8222/tcp

    sudo firewall-cmd –permanent –add-port=4248/tcp

    sudo firewall-cmd –permanent –add-port=7222/tcp

    sudo firewall-cmd –reload

    Start the service:

    sudo systemctl start nats.service

    sudo systemctl enable nats.service

    Check status:

    sudo systemctl status nats.service -l

    Note: Remember to check logs of status commands in node2 and node3; it should show a connection with node1 and also confirm that node1 has been made the leader.

    Setting up the context:

    Setting up context will help us in managing our cluster better with NATSCLI.

    # pass the –tlsca flag in dev because we do not have the DNS registered. In staging and Production the `tlsca` flag will not be needed because certs will be registered.

    nats context add nats –server <IP/DNS-First>:4222,<IP/DNS-Second>:4222,<IP/DNS-Third>:4222 –description “Awesome Nats Servers List” –tlsca /nats/certs/natsio_rootCA.pem –select

    nats context ls

    nats account info

    Complete file for starting service would like this:

    #!/bin/bash

# Own the files
sudo chown --recursive nats:nats /nats

# Create Firewall Rules
sudo firewall-cmd --permanent --add-port=4222/tcp
sudo firewall-cmd --permanent --add-port=8222/tcp
sudo firewall-cmd --permanent --add-port=4248/tcp
sudo firewall-cmd --permanent --add-port=7222/tcp
sudo firewall-cmd --reload

# Start Service
sudo systemctl start nats.service
sudo systemctl enable nats.service
sudo systemctl status nats.service -l

# Setup Context
# pass the --tlsca flag in dev because we do not have the DNS registered. In staging and Production the `tlsca` flag will not be needed because certs will be registered.
nats context add nats --server <IP/DNS-First>:4222,<IP/DNS-Second>:4222,<IP/DNS-Third>:4222 --description "Awesome Nats Servers List" --tlsca /nats/certs/natsio_rootCA.pem --select

nats context ls
nats account info

    Validation: 

    The account info command should shuffle among the servers in the Connected URL string.

    Stream Listing:

    Streams that will be available across the regions would require the credentials. The creds should be common across all clusters:

    The same info can be obtained from the different clusters when the same command is fired:

    To fetch local streams that are present under the no_auth_user:

    And from the different clusters using the same command (without credentials), we should get a different stream:

    Advanced Messaging Patterns with NATS

    In this section, we will explore advanced messaging patterns that leverage the capabilities of NATS for more complex communication scenarios.

    A. Request-Reply Pattern

    The request-reply pattern allows applications to send requests and receive corresponding responses through NATS. It enables synchronous communication, making it suitable for scenarios where immediate responses are required.

    B. Publish-Subscribe Pattern with Wildcards

    Nats introduces the concept of wildcards to the publish-subscribe pattern, allowing subscribers to receive messages based on pattern matching. This enables greater flexibility in subscription matching and expands the possibilities of message distribution.

    C. Queue Groups for Load Balancing and Fault Tolerance

    Queue groups provide load balancing and fault tolerance capabilities in NATS. By grouping subscribers together, NATS ensures that messages are distributed evenly across the subscribers within the group, preventing any single subscriber from being overwhelmed.

    Overcoming Real-World Challenges

    In this section, we will discuss real-world challenges that developers may encounter when working with NATS and explore strategies to overcome them.

    A. Scalability and High Availability in NATS

    As applications grow and message traffic increases, scalability, and high availability become crucial considerations. NATS offers various techniques and features to address these challenges, including clustering, load balancing, and fault tolerance mechanisms.

    B. Securing NATS Communication

    Security is paramount in any messaging system, and NATS provides several mechanisms to secure communication. These include authentication, encryption, access control, and secure network configurations.

    C. Monitoring and Debugging Techniques

    Efficiently monitoring and troubleshooting a NATS deployment is essential for maintaining system health. NATS provides tools and techniques to monitor message traffic, track performance metrics, and identify and resolve potential issues in real time.

    Recovery Scenarios in NATS 

    This section is intended to help in scenarios when NATS services are not usable. Scenarios such as Node failure, Not reachable, Service down, or region down are some examples of such a situation.

    Summary

    In this article, we have embarked on a journey from being a NATS novice to mastering its intricacies. We have explored the importance and applicability of NATS in the software development landscape. Through a comprehensive exploration of NATS’ definition, architecture, key features, and use cases, we have built a strong foundation in NATS. We have also examined advanced messaging patterns and discussed strategies to overcome real-world challenges in scalability, security, and monitoring. Furthermore, we have delved into the Recovery scenarios, which might come in handy when things don’t behave as expected. Armed with this knowledge, developers can confidently utilize NATS to unlock its full potential in their projects.

  • JNIgen: Simplify Native Integration in Flutter

    Prepare to embark on a groundbreaking journey through the realms of Flutter as we uncover the remarkable new feature—JNIgen. In this blog, we pull back the curtain to reveal JNIgen’s transformative power, from simplifying intricate tasks to amplifying scalability; this blog serves as a guiding light along the path to a seamlessly integrated Flutter ecosystem.

    As Flutter continues to mesmerize developers with its constant evolution, each release unveiling a treasure trove of thrilling new features, the highly anticipated Google I/O 2023 was an extraordinary milestone. Amidst the excitement, a groundbreaking technique was unveiled: JNIgen, offering effortless access to native code like never before.

    Let this blog guide you towards a future where your Flutter projects transcend limitations and manifest into awe-inspiring creations.

    1. What is JNIgen?

    JNIgen, which stands for Java native interface generator,  is an innovative tool that automates the process of generating Dart bindings for Android APIs accessible through Java or Kotlin code. By utilizing these generated bindings, developers can invoke Android APIs with a syntax that closely resembles native code.

    With JNIgen, developers can seamlessly bridge the gap between Dart and the rich ecosystem of Android APIs. This empowers them to leverage the full spectrum of Android’s functionality, ranging from system-level operations to platform-specific features. By effortlessly integrating with Android APIs through JNIgen-generated bindings, developers can harness the power of native code and build robust applications with ease.

    1.1. Default approach: 

    In the current Flutter framework, we rely on Platform channels to establish a seamless communication channel between Dart code and native code. These channels serve as a bridge for exchanging messages and data.

    Typically, we have a Flutter app acting as the client, while the native code contains the desired methods to be executed. The Flutter app sends a message containing the method name to the native code, which then executes the requested method and sends the response back to the Flutter app.

    However, this approach requires the manual implementation of handlers on both the Dart and native code sides. It entails writing code to handle method calls and manage the exchange of responses. Additionally, developers need to carefully manage method names and channel names on both sides to ensure proper communication.

    1.2. Working principle of JNIgen: 

    Figure 1

     

    In JNIgen, our native code path is passed to the JNIgen generator, which initiates the generation of an intermediate layer of C code. This C code is followed by the necessary boilerplate in Dart, facilitating access to the C methods. All data binding and C files are automatically generated in the directory specified in the .yaml file, which we will explore shortly.

    Consequently, as a Flutter application, our interaction is solely focused on interfacing with the newly generated Dart code, eliminating the need for direct utilization of native code.

    1.3. Similar tools: 

    During the Google I/O 2023 event, JNIgen was introduced as a tool for native code integration. However, it is important to note that not all external libraries available on www.pub.dev are developed exclusively using channels. Another tool, FFIgen, was introduced earlier at Google I/O 2021 and serves a similar purpose. Both FFIgen and JNIgen function similarly, converting native code into intermediate C code with corresponding Dart dependencies to establish the necessary connections.

    While JNIgen primarily facilitates communication between Android native code and Dart code, FFIgen has become the preferred choice for establishing communication between iOS native code and Dart code. Both tools are specifically designed to convert native code into intermediate code, enabling seamless interoperability within their respective platforms.

    2. Configuration

    Prior to proceeding with the code implementation, it is essential to set up and install the necessary tools.

    2.1. System setup: 

    2.1.1 Install MVN

    Windows

    • Download the Maven archive for Windows from the link here [download Binary zip archive]
    • After Extracting the zip file, you will get a folder with name “apache-maven-x.x.x”
    • Create a new folder with the name “ApacheMaven” in “C:Program Files” and paste the above folder in it. [Your current path will be “C:Program FilesApacheMavenapache-maven-x.x.x”]
    • Add the following entry in “Environment Variable” →  “User Variables”
      M2 ⇒ “C:Program FilesApacheMavenapache-maven-x.x.xbin”
      M2_HOME ⇒ “C:Program FilesApacheMavenapache-maven-x.x.x”
    • Add a new entry “%M2_HOME%bin” in “path” variable

    Mac

    • Download Maven archive for mac from the link here [download Binary tar.gz archive]
    • Run the following command where you have downloaded the *.tar.gz file
    tar -xvf apache-maven-3.8.7.bin.tar.gz

    • Add the following entry in .zshrc or .bash_profile to set Maven path “export PATH=”$PATH:/Users/username/Downloads/apache-maven-x.x.x/bin”

    Or

    • You can use brew to install llvm 
    brew install llvm

    • Brew will give you instruction like this for further setup
    ==> llvm
To use the bundled libc++ please add the following LDFLAGS:
LDFLAGS="-L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++"

llvm is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have llvm first in your PATH, run:
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc

For compilers to find llvm you may need to set:
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"

    2.1.1 Install Clang-Format

    Windows

    • Download the latest version of LLVM for windows from the link here

    Mac

    • Run the following brew command: 
    brew install clang-format

    2.2. Flutter setup: 

    2.2.1 Get Dependencies

    Run the following commands with Flutter:

    flutter pub add jni

    flutter pub add jnigen

    2.2.2 Setup configuration file

    Figure 01 provides a visual representation of the .yaml file, which holds crucial configurations utilized by JNIgen. These configurations serve the purpose of identifying paths for native classes, as well as specifying the locations where JNIgen generates the resulting C and Dart files. Furthermore, the .yaml file allows for specifying Maven configurations, enabling the selection of specific third-party libraries that need to be downloaded to facilitate code generation.

    By leveraging the power of the .yaml file, developers gain control over the path identification process and ensure that the generated code is placed in the desired locations. Additionally, the ability to define Maven configurations grants flexibility in managing dependencies, allowing the seamless integration of required third-party libraries into the generated code. This comprehensive approach enables precise control and customization over the code generation process, enhancing the overall efficiency and effectiveness of the development workflow.

    Let’s explore the properties that we have utilized within the .yaml file (Please refer “3.2.2. code implementation” section’s example for better understanding):

    • android_sdk_config: 

    When the value of a specific property is set to “true,” it triggers the execution of a Gradle stub during the invocation of JNIgen. Additionally, it includes the Android compile classpath in the classpath of JNIgen. However, to ensure that all dependencies are cached appropriately, it is necessary to have previously performed a release build.

    • output 

    As the name implies, the “output” section defines the configuration related to the generation of intermediate code. This section plays a crucial role in determining how the intermediate code will be generated and organized.

    •  c >> library_name &&  c >> path:
      Here we are setting details for c_based binding code.

    •  dart >> path &&  dart >> structure:

    Here we are defining configuration for dart_based binding code.

    •  source_path:

    These are specific directories that are scanned during the process of locating the relevant source files.

    •  classes:

    By providing a comprehensive list of classes or packages, developers can effectively control the scope of the code generation process. This ensures that the binding code is generated only for the desired components, minimizing unnecessary code generation

    By utilizing these properties within the .yaml file, developers can effectively control various aspects of the code generation process, including path identification, code organization, and dependency management. To get more in-depth information, please check out the official documentation here.

    2.3. Generate bindings files:

    Once this setup is complete, the final step for JNIgen is to obtain the jar file that will be scanned to generate the required bindings. To initiate the process of generating the Android APK, you can execute the following command:

    flutter build apk

    Run the following command in your terminal to generate code:

    dart run jnigen --config jnigen.yaml

    2.3. Android setup: 

    Add the address of CMakeLists.txt file in your android >> app >> build.gradle file’s buildTypes section:

    buildTypes {
        externalNativeBuild {
            cmake {
                path <address of CMakeLists.txt>
            }
        }
    }

    With this configuration, we are specifying the path for the CMake file that will been generated by JNIgen.This path declaration is crucial for identifying the location of the generated CMake file within the project structure.

    With the completion of the aforementioned steps, you are now ready to run your application and leverage all the native functions that have been integrated.

    3. Sample Project

    To gain hands-on experience and better understand the JNIgen, let’s create a small project together. Follow the steps below to get started. 

    Let’s start with:

    3.1. Packages & directories:

    3.1.1 Create a project using the following command:

    flutter create jnigen_integration_project

    3.1.2 Add these under dependencies of pubspec.yaml (and run command flutter pub get):

    jni: ^0.5.0
jnigen: ^0.5.0

    3.1.3. Got to android >> app >> src >> main directory.

    3.1.4. Create directories inside the main as show below:

    Figure 02 

    ‍

    3.2. Code Implementation:

    3.2.1 We will start with Android code. Create 2 files HardwareUtils.java & HardwareUtilsKotlin.kt inside the utils directory.

     HardwareUtilsKotlin.kt

    package com.hardware.utils

import android.os.Build

class HardwareUtilsKotlin {

   fun getHardwareDetails(): Map<String, String>? {
       val hardwareDetails: MutableMap<String, String> = HashMap()
       hardwareDetails["Language"] = "Kotlin"
       hardwareDetails["Manufacture"] = Build.MANUFACTURER
       hardwareDetails["Model No."] = Build.MODEL
       hardwareDetails["Type"] = Build.TYPE
       hardwareDetails["User"] = Build.USER
       hardwareDetails["SDK"] = Build.VERSION.SDK
       hardwareDetails["Board"] = Build.BOARD
       hardwareDetails["Version Code"] = Build.VERSION.RELEASE
       return hardwareDetails
   }
}

     HardwareUtils.java 

    package com.hardware.utils;


import android.os.Build;


import java.util.HashMap;
import java.util.Map;


public class HardwareUtils {


   public Map<String, String> getHardwareDetails() {
       Map<String, String> hardwareDetails = new HashMap<String, String>();
       hardwareDetails.put("Language", "JAVA");
       hardwareDetails.put("Manufacture", Build.MANUFACTURER);
       hardwareDetails.put("Model No.", Build.MODEL);
       hardwareDetails.put("Type", Build.TYPE);
       hardwareDetails.put("User", Build.USER);
       hardwareDetails.put("SDK", Build.VERSION.SDK);
       hardwareDetails.put("Board", Build.BOARD);
       hardwareDetails.put("Version Code", Build.VERSION.RELEASE);
       return hardwareDetails;
   }


   public Map<String, String> getHardwareDetailsKotlin() {
       return new HardwareUtilsKotlin().getHardwareDetails();
   }


}

    3.2.2 To provide the necessary configurations to JNIGen for code generation, we will create a .yaml file named JNIgen.yaml in the root of the project.

       jnigen.yaml 

    android_sdk_config:
 add_gradle_deps: true


output:
 c:
   library_name: hardware_utils
   path: src/
 dart:
   path: lib/hardware_utils.dart
   structure: single_file


source_path:
 - 'android/app/src/main/java'


classes:
 - 'com.hardware.utils'

    3.2.3 Let’s generate C & Dart code.

    Execute the following command to create APK:

    flutter build apk

    After the successful execution of the above command, execute the following command:

    dart run jnigen --config jnigen.yaml

    3.2.4 Add the address of CMakeLists.txt in your android >> app >> build.gradle file’s buildTypes section as shown below :

    buildTypes {
        externalNativeBuild {
            cmake {
                path "../../src/CMakeLists.txt"
            }
        }
  }

    3.2.5. Final step is to call the methods from Dart code, which was generated by JNIgen.

    To do this, replace the MyHomePage class code with the below code from main.dart file.

    class MyHomePage extends StatefulWidget {
 const MyHomePage({super.key, required this.title});

 final String title;

 @override
 State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
 String _hardwareDetails = '';
 String _hardwareDetailsKotlin = '';
 JObject activity = JObject.fromRef(Jni.getCurrentActivity());

 @override
 void initState() {
   JMap<JString, JString> deviceHardwareDetails =
       HardwareUtils().getHardwareDetails();
   _hardwareDetails = 'This device details from Java class:n';
   deviceHardwareDetails.forEach((key, value) {
     _hardwareDetails =
         '$_hardwareDetailsn${key.toDartString()} is ${value.toDartString()}';
   });

   JMap<JString, JString> deviceHardwareDetailsKotlin =
       HardwareUtils().getHardwareDetailsKotlin();
   _hardwareDetailsKotlin = 'This device details from Kotlin class:n';
   deviceHardwareDetailsKotlin.forEach((key, value) {
     _hardwareDetailsKotlin =
         '$_hardwareDetailsKotlinn${key.toDartString()} is ${value.toDartString()}';
   });

   setState(() {
     _hardwareDetails;
     _hardwareDetailsKotlin;
   });
   super.initState();
 }

 @override
 Widget build(BuildContext context) {
   return Scaffold(
     appBar: AppBar(
       title: Text(widget.title),
     ),
     body: Center(
       child: Column(
         mainAxisAlignment: MainAxisAlignment.center,
         children: <Widget>[
           Text(
             _hardwareDetails,
             textAlign: TextAlign.center,
           ),
           SizedBox(height: 20,),
           Text(
             _hardwareDetailsKotlin,
             textAlign: TextAlign.center,
           ),
         ],
       ),
     ),
   );
 }
}

    After all of this, when we launch our app, we will see information about our Android device.

    4. Result

    For your convenience, the complete code for the project can be found here. Feel free to refer to this code repository for a comprehensive overview of the implementation details and to access the entirety of the source code.

    5. Conclusion

    In conclusion, we explored the limitations of the traditional approach to native API access in Flutter for mid to large-scale projects. Through our insightful exploration of JNIgen’s working principles, we uncovered its remarkable potential for simplifying the native integration process.

    By gaining a deep understanding of JNIgen’s inner workings, we successfully developed a sample project and provided detailed guidance on the essential setup requirements. Armed with this knowledge, developers can embrace JNIgen’s capabilities to streamline their native integration process effectively.

    We can say that JNIgen is a valuable tool for Flutter developers seeking to combine the power of Flutter’s cross-platform capabilities with the flexibility and performance benefits offered by native code. It empowers developers to build high-quality apps that seamlessly integrate platform-specific features and existing native code libraries, ultimately enhancing the overall user experience. 

    Hopefully, this blog post has inspired you to explore the immense potential of JNIgen in your Flutter applications. By harnessing the JNIgen, we can open doors to new possibilities.

    Thank you for taking the time to read through this blog!

    6. Reference

    1. https://docs.flutter.dev/
    2. https://pub.dev/packages/jnigen
    3. https://pub.dev/packages/jni
    4. https://github.com/dart-lang/jnigen
    5. https://github.com/dart-lang/jnigen#readme
    6. https://github.com/dart-lang/jnigen/wiki/Architecture-&-Design-Notes
    7. https://medium.com/simform-engineering/jnigen-an-easy-way-to-access-platform-apis-cb1fd3101e33
    8. https://medium.com/@marcoedomingos/the-ultimate-showdown-methodchannel-vs-d83135f2392d

  • Serverpod: The Ultimate Backend for Flutter

    Join us on this exhilarating journey, where we bridge the gap between frontend and backend development with the seamless integration of Serverpod and Flutter.

    Gone are the days of relying on different programming languages for frontend and backend development. With Flutter’s versatile framework, you can effortlessly create stunning user interfaces for a myriad of platforms. However, the missing piece has always been the ability to build the backend in Dart as well—until now.

    Introducing Serverpod, the missing link that completes the Flutter ecosystem. Now, with Serverpod, you can develop your entire application, from frontend to backend, all within the familiar and elegant Dart language. This synergy enables a seamless exchange of data and functions between the client and the server, reducing development complexities and boosting productivity.

    1. What is Serverpod?

    As a developer or tech enthusiast, we recognize the critical role backend services play in the success of any application. Whether you’re building a web, mobile, or desktop project, a robust backend infrastructure is the backbone that ensures seamless functionality and scalability.

    That’s where “Serverpod” comes into the picture—an innovative backend solution developed entirely in Dart, just like your Flutter projects. With Serverpod at your disposal, you can harness the full power of Dart on both the frontend and backend, creating a harmonious development environment that streamlines your workflow.

    The biggest advantage of using Serverpod is that it automates protocol and client-side code generation by analyzing your server, making remote endpoint calls as simple as local method calls.

    1.1. Current market status

    The top 10 programming languages for backend development in 2023 are as follows: 

    [Note: The results presented here are not absolute and are based on a combination of surveys conducted in 2023, including ‘Stack Overflow Developer Survey – 2023,’ ‘State of the Developer Ecosystem Survey,’ ‘New Stack Developer Survey,’ and more.]

    • Node.js – ~32%
    • Python (Django, Flask) – ~28%
    • Java (Spring Boot, Java EE) – ~18%
    • Ruby (Ruby on Rails) – ~7%
    • PHP (Laravel, Symfony) – ~6%
    • Go (Golang) – ~3%
    • .NET (C#) – ~2%
    • Rust – Approximately 1%
    • Kotlin (Spring Boot with Kotlin) – ~1%
    • Express.js (for Node.js) – ~1%
    Figure 01

    Figure 01 provides a comprehensive overview of the current usage of backend development technologies, showcasing a plethora of options with diverse features and capabilities. However, the landscape takes a different turn when it comes to frontend development. While the backend technologies offer a wealth of choices, most of these languages lack native multiplatform support for frontend applications.

    As a result, developers find themselves in a situation where they must choose between two sets of languages or technologies for backend and frontend business logic development.

    1.2. New solution

    As the demand for multiplatform applications continues to grow, developers are actively exploring new frameworks and languages that bridge the gap between backend and frontend development. Recently, a groundbreaking solution has emerged in the form of Serverpod. With Serverpod, developers can now accomplish server development in Dart, filling the crucial gap that was previously missing in the Flutter ecosystem.

    Flutter has already demonstrated its remarkable support for a wide range of platforms. The absence of server development capabilities was a notable limitation that has now been triumphantly addressed with the introduction of Serverpod. This remarkable achievement enables developers to harness the power of Dart to build both frontend and backend components, creating unified applications with a shared codebase.

    2. Configurations 

    Prior to proceeding with the code implementation, it is essential to set up and install the necessary tools.

    [Note: Given Serverpod’s initial stage, encountering errors without readily available online solutions is plausible. In such instances, seeking assistance from the Flutter community forum is highly recommended. Drawing from my experience, I suggest running the application on Flutter web first, particularly for Serverpod version 1.1.1, to ensure a smoother development process and gain insights into potential challenges.]

    2.1. Initial setup

    2.1.1 Install Docker

    Docker serves a crucial role in Serverpod, facilitating:

    • Containerization: Applications are packaged and shipped as containers, enabling seamless deployment and execution across diverse infrastructures.
    • Isolation: Applications are isolated from one another, enhancing both security and performance aspects, safeguarding against potential vulnerabilities, and optimizing system efficiency.

    Download & Install Docker from here.

    2.1.2 Install Serverpod CLI 

    • Run the following command:
    dart pub global activate serverpod_cli

    • Now test the installation by running:
    serverpod

    With proper configuration, the Serverpod command displays help information.

    2.2. Project creation

    To initiate serverpod commands, the Docker application must be launched first. Ensuring an active Docker instance in the backend environment is imperative to execute Serverpod commands successfully.

    • Create a new project with the command:
    serverpod create <your_project_name>

    Upon execution, a new directory will be generated with the specified project name, comprising three Dart packages:

    <your_project_name>_server: This package is designated for server-side code, encompassing essential components such as business logic, API endpoints, DB connections, and more.
    <your_project_name>_client: Within this package, the code responsible for server communication is auto-generated. Manual editing of files in this package is typically avoided.
    <your_project_name>_flutter: Representing the Flutter app, it comes pre-configured to seamlessly connect with your local server, ensuring seamless communication between frontend and backend elements.

    2.3. Project execution

    Step 1: Navigate to the server package with the following command:

    cd <your_project_name>/<your_project_name>_server

    Step 2: (Optional) Open the project in the VS Code IDE using the command:

    (Note: You can use any IDE you prefer, but for our purposes, we’ll use VS Code, which also simplifies DB connection later.)

    code .

    Step 3: Once the project is open in the IDE, stop any existing Docker containers with this command:

    .setup-tables.cmd

    Step 4: Before starting the server, initiate new Docker containers with the following command:

    docker-compose up --build --detach

    Step 5: The command above will start PostgreSQL and Redis containers, and you should receive the output:

    ~> docker-compose up --build --detach
	[+] Running 2/2
 	✔ Container <your_project_name>_server-redis-1     Started                                                                                                
 	✔ Container <your_project_name>_server-postgres-1  Started

    (Note: If the output doesn’t match, refer to this Stack Overflow link for missing commands in the official documentation.)

    Step 6: Proceed to start the server with this command:

    dart bin/main.dart

    Step 7: Upon successful execution, you will receive the following output, where the “Server Default listening on port” value is crucial. Please take note of this value.

    ~> dart bin/main.dart
 	SERVERPOD version: 1.1.1, dart: 3.0.5 (stable) (Mon Jun 12 18:31:49 2023 +0000) on "windows_x64", time: 2023-07-19 15:24:27.704037Z
 	mode: development, role: monolith, logging: normal, serverId: default
 	Insights listening on port 8081
 	Server default listening on port 8080
 	Webserver listening on port 8082
 	CPU and memory usage metrics are not supported on this platform.

    Step 8: Use the “Server Default listening on port” value after “localhost,” i.e., “127.0.0.1,” and load this URL in your browser. Accessing “localhost:8080” will display the desired output, indicating that your server is running and ready to process requests.

    Figure 02

    Step 9: Now, as the containers reach the “Started” state, you can establish a connection with the database. We have opted for PostgreSQL as our DB choice, and the rationale behind this selection lies in the “docker-compose.yaml” file at the server project’s root. In the “service” section, PostgreSQL is already added, making it an ideal choice as the required setup is readily available. 

    Figure 03

    For the database setup, we need key information, such as Host, Port, Username, and Password. You can find all this vital information in the “config” directory’s “development.yaml” and “passwords.yaml” files. If you encounter difficulties locating these details, please refer to Figure 04.

    Figure 04

    ‍Step 10: To establish the connection, you can install an application similar to Postico or, alternatively, I recommend using the MySQL extension, which can be installed in your VS Code with just one click.

    Figure 05

    ‍Step 11: Follow these next steps:

    1. Select the “Database” option.
    2. Click on “Create Connection.”
    3. Choose the “PostgreSQL” option.
    4. Add a name for your Connection.
    5. Fill in the information collected in the last step.
    6. Finally, select the “Connect” option.
    Figure 06
    1. Upon success, you will receive a “Connect Success!” message, and the new connection will be added to the Explorer Tab.
    Figure 07

    Step 12: Now, we shift our focus to the Flutter project (Frontend):

    Thus far, we have been working on the server project. Let us open a new VS Code instance for a separate Flutter project while keeping the current VS Code instance active, serving as the server.‍

    Step 13: Execute the following command to run the Flutter project on Chrome:

    flutter run -d chrome

    With this, the default project will generate the following output:

    Step 14: When you are finished, you can shut down Serverpod with “Ctrl-C.”

    Step 15: Then stop Postgres and Redis.

    docker compose stop

    Figure 08

    3. Sample Project

    So far, we have successfully created and executed the project, identifying three distinct components. The server project caters to server/backend development, while the Flutter project handles application/frontend development. The client project, automatically generated, serves as the vital intermediary, bridging the gap between the frontend and backend.

    However, merely acknowledging the projects’ existence is insufficient. To maximize our proficiency, it is crucial to grasp the code and file structure comprehensively. To achieve this, we will embark on a practical journey, constructing a small project to gain hands-on experience and unlock deeper insights into all three components. This approach empowers us with a well-rounded understanding, further enhancing our capabilities in building remarkable applications.

    3.1. What are we building?

    In this blog, we will construct a sample project with basic Login and SignUp functionality. The SignUp process will collect user information such as Email, Password, Username, and age. Users can subsequently log in using their email and password, leading to the display of user details on the dashboard screen. With the initial system setup complete and the newly created project up and running, it’s time to commence coding. 

    3.1.1 Create custom models for API endpoints

    Step1: Create a new file in the “lib >> src >> protocol” directory named “users.yaml”:

    class: Users
table: users
fields:
  username: String
  email: String
  password: String
  age: int

    Step 2: Save the file and run the following command to generate essential data classes and table creation queries:

    serverpod generate

    (Note: Add “–watch” after the command for continuous code generation). 

    Successful execution of the above command will generate a new file named “users.dart” in the “lib >> src >> generated” folder. Additionally, the “tables.pgsql” file now contains SQL queries for creating the “users” table. The same command updates the auto-generated code in the client project. 

    3.1.2 Create Tables in DB for the generated model 

    Step 1: Copy the queries written in the “generated >> tables.pgsql” file.

    In the MySQL Extension’s Database section, select the created database >> [project_name] >> public >> Tables >> + (Create New Table).

    Figure 09

    Step 2: Paste the queries into the newly created .sql file and click “Execute” above both queries.

    Figure 10

    Step 3: After execution, you will obtain an empty table with the “id” as the Primary key.

    Figure 11

    If you found multiple tables already present in your database like shown in the next figure, you can ignore those. These tables are created by the system with queries present in the “generated >> tables-serverpod.pgsql” file.

    Figure 12

    3.1.3 Create an API endpoint

    Step 1: Generate a new file in the “lib >> src >> endpoints” directory named “session_endpoints.dart”:

    class SessionEndpoint extends Endpoint {
  Future<Users?> login(Session session, String email, String password) async {
    List<Users> userList = await Users.find(session,
        where: (p0) =>
            (p0.email.equals(email)) & (p0.password.equals(password)));
    return userList.isEmpty ? null : userList[0];
  }


  Future<bool> signUp(Session session, Users newUser) async {
    try {
      await Users.insert(session, newUser);
      return true;
    } catch (e) {
      print(e.toString());
      return false;
    }
  }
}

    If “serverpod generate –watch” is already running, you can ignore this step 2.

    Step 2: Run the command:

    serverpod generate

    Step 3: Start the server.
    [For help, check out Step 1 – Step 6 mentioned in Project Execution part.]

    3.1.3 Create three screens

    Login Screen:

    Figure 13

    SignUp Screen:

    Figure 14

    Dashboard Screen:

    Figure 15

    3.1.4 Setup Flutter code

    Step 1: Add the code provided to the SignUp button in the SignUp screen to handle user signups.

    try {
        final result = await client.session.signUp(
          Users(
            email: _emailEditingController.text.trim(),
            username: _usernameEditingController.text.trim(),
            password: _passwordEditingController.text.trim(),
            age: int.parse(_ageEditingController.text.trim()),
          ),
        );
        if (result) {
          Navigator.pop(context);
        } else {
          _errorText = 'Something went wrong, Try again.';
        }
      } catch (e) {
        debugPrint(e.toString());
        _errorText = e.toString();
      }

    Step 2: Add the code provided to the Login button in the Login screen to handle user logins.

    try {
        final result = await client.session.login(
          _emailEditingController.text.trim(),
          _passwordEditingController.text.trim(),
        );
        if (result != null) {
          _emailEditingController.text = '';
          _passwordEditingController.text = '';
          Navigator.push(
            context,
            MaterialPageRoute(
              builder: (context) => DashboardPage(user: result),
            ),
          );
        } else {
          _errorText = 'Something went wrong, Try again.';
        }
      } catch (e) {
        debugPrint(e.toString());
        _errorText = e.toString();
      }

    Step 3: Implement logic to display user data on the dashboard screen.

    With these steps completed, our Flutter app becomes a fully functional project, showcasing the power of this new technology. Armed with Dart knowledge, every Flutter developer can transform into a proficient full-stack developer.

    4. Result

    Figure 16

    To facilitate your exploration, the entire project code is conveniently available in this code repository. Feel free to refer to this repository for an in-depth understanding of the implementation details and access to the complete source code, enabling you to delve deeper into the project’s intricacies and leverage its functionalities effectively.

    5. Conclusion

    In conclusion, we have provided a comprehensive walkthrough of the step-by-step setup process for running Serverpod seamlessly. We explored creating data models, integrating the database with our server project, defining tables, executing data operations, and establishing accessible API endpoints for Flutter applications.

    Hopefully, this blog post has kindled your curiosity to delve deeper into Serverpod’s immense potential for elevating your Flutter applications. Embracing Serverpod unlocks a world of boundless possibilities, empowering you to achieve remarkable feats in your development endeavors.

    Thank you for investing your time in reading this informative blog!

    6. References

    1. https://docs.flutter.dev/
    2. https://pub.dev/packages/serverpod/
    3. https://serverpod.dev/
    4. https://docs.docker.com/get-docker/
    5. https://medium.com/serverpod/introducing-serverpod-a-complete-backend-for-flutter-written-in-dart-f348de228e19
    6. https://medium.com/serverpod/serverpod-our-vision-for-a-seamless-scalable-backend-for-the-flutter-community-24ba311b306b
    7. https://stackoverflow.com/questions/76180598/serverpod-sql-error-when-starting-a-clean-project
    8. https://www.youtube.com/watch?v=3Q2vKGacfh0
    9. https://www.youtube.com/watch?v=8sCxWBWhm2Y

    ‍