Tag: backend development

  • Beginner’s Guide for Writing Unit Test Cases with Jest Framework

    ‍Prerequisite

    Basic JavaScript, TypeScript

    Objective

    To make the reader understand the use/effect of test cases in software development.

    What’s in it for you?‍

    In the world of coding, we’re often in a rush to complete work before a deadline hits. And let’s be honest, writing test cases isn’t usually at the top of our priority list. We get it—they seem tedious, so we’d rather skip this extra step. But here’s the thing: those seemingly boring lines of code have superhero potential. Don’t believe me? You will.

    In this blog, we’re going to break down the mystery around test cases. No jargon, just simple talk. We’ll chat about what they are, explore a handy tool called Jest, and uncover why these little lines are actually the unsung heroes of coding. So, let’s ditch the complications and discover why giving some attention to test cases can level up our coding game. Ready? Let’s dive in!

    What are test cases?

    A test case is a detailed document specifying conditions under which a developer assesses whether a software application aligns with customer requirements. It includes preconditions, the case name, input conditions, and expected results. Derived from test scenarios, test cases cover both positive and negative inputs, providing a roadmap for test execution. This one-time effort aids future regression testing.

    Test cases offer insights into testing strategy, process, preconditions, and expected outputs. Executed during testing, they ensure the software performs its intended tasks. Linking defects to test case IDs facilitates efficient defect reporting. The comprehensive documentation acts as a safeguard, catching any oversights during test case execution and reinforcing the development team’s efforts.

    Different types of test cases exist, including integration, functional, non-functional, and unit.
    For this blog, we will talk about unit test cases.

    What are unit test cases?

    Unit testing is the process of testing the smallest functional unit of code. A functional unit could be a class member or simply a function that does something to your input and provides an output. Test cases around those functional units are called unit test cases.

    Purpose of unit test cases

    • To validate that each unit of the software works as intended and meets the requirements:
      For example, if your requirement is that the function returns an object with specific properties, a unit test will detect whether the code is written accordingly.
    • To check the robustness of code:
      Unit tests are automated and run each time the code is changed to ensure that new code does not break existing functionality.
    • To check the errors and bugs beforehand:
      If a case fails or doesn’t fulfill the requirement, it helps the developer isolate the area and recheck it for bugs before testing on demo/UAT/staging.

    Different frameworks for writing unit test cases

    There are various frameworks for unit test cases, including:

    • Mocha
    • Storybook
    • Cypress
    • Jasmine
    • Puppeteer
    • Jest
    Source: https://raygun.com/blog/javascript-unit-testing-frameworks/

    Why Jest?

    Jest is used and recommended by Facebook and officially supported by the React dev team.

    It has a great community and active support, so if you run into a problem and can’t find a solution in the comprehensive documentation, there are thousands of developers out there who could help you figure it out within hours.

    1. Performance: Ideal for larger projects with continuous deployment needs, Jest delivers enhanced performance.

    2. Compatibility: While Jest is widely used for testing React applications, it seamlessly integrates with other frameworks like Angular, Node, Vue, and Babel-based projects.

    3. Auto Mocking: Jest automatically mocks imported libraries in test files, reducing boilerplate and facilitating smoother testing workflows.

    4. Extended API: Jest comes with a comprehensive API, eliminating the necessity for additional libraries in most cases.

    5. Timer Mocks: Featuring a Time mocking system, Jest accelerates timeout processes, saving valuable testing time.

    6. Active Development & Community: Jest undergoes continuous improvement, boasting the most active community support for rapid issue resolution and updates.

    Components of a test case in Jest‍

    Describe

    • As the name indicates, they are responsible for describing the module we are going to test.
    • It should only describe the module, not the tests, as this describe module is generally not tested by Jest.

    It

    • Here, the actual code is tested and verified with actual or fake (spy, mocks) outputs.
      We can nest various it modules under the describe module.
    • It’s good to describe what the test does or doesn’t do in the description of the it module.

    Matchers

    • Matchers match the output with a real/fake output.
    • A test case without a matcher will always be a true/trivial test case.
    // For each unit test you write,
    // answer these questions:
    
    describe('What component aspect are you testing?', () => {
        it('What should the feature do?', () => {
            const actual = 'What is the actual output?'
            const expected = 'What is the expected output?'
    
            expect(actual).toEqual(expected) // matcher
    
        })
      })

    ‍Mocks and spies in Jest

    Mocks: They are objects or functions that simulate the behavior of real components. They are used to create controlled environments for testing by replacing actual components with simulated ones. Mocks are employed to isolate the code being tested, ensuring that the test focuses solely on the unit or component under examination without interference from external dependencies.

    It is mainly used for mocking a library or function that is most frequently used in the whole file or unit test case.

    Let Code.ts be the file you want to test.

    import { v4 as uuidv4 } from uuid
    
    export const functionToTest = () => {
    
        const id = uuidv4()
        // rest of the code
        return id;
    
    }

    As this is a unit test, we won’t be testing the uuidV4 function, so we will mock the whole uuid module using jest.mock.

    jest.mock('uuid', () => { uuidv4: () => 'random id value' }))  // mocking uuid module which will have uuidV4 as function
    describe('testing code.ts', () => {
        it('i have mocked uuid module', ()=> {
    
        const res = functionToTest()
        expect(res).tobeEqual('random id value')
    })
    
    })

    And that’s it. You have mocked the entire uuid module, so when it is coded during a test, it will return uuidV4 function, and that function, when executed, will give a random id value.

    Spies: They are functions or objects that “spy” on other functions by tracking calls made to them. They allow you to observe and verify the behavior of functions during testing. Spies are useful for checking if certain functions are called, how many times they are called, and with what arguments. They help ensure that functions are interacting as expected.

    This is by far the most used method, as this method works on object values and thus can be used to spy class methods efficiently.

    class DataService {
        fetchData() 
        {
            // code to fetch data
            return { 'real data'}
        }
    }

    describe('DataService Class', () => {
    
        it('should spy on the fetchData method with mockImplementation', () => {
            const dataServiceInstance = new DataService();
            const fetchDataSpy = jest.spyon(DataService.prototype, 'fetchData'); // prototype makes class method to a object
            fetchDataSpy.mockImplementation(() => 'Mocked Data'); // will return mocked data whenever function will be called
    
            const result = dataServiceInstance.fetchData(); // mocked Data
            expect(fetchDataSpy).toHaveBeenCalledTimes(1)
            expect(result).toBe('Mocked Data');
        }
      
      }

    Mocking database call‍

    One of the best uses of Jest is to mock a database call, i.e., mocking create, put, post, and delete calls for a database table.

    We can complete the same action with the help of only Jest spies.

    Let us suppose we have a database called DB, and it has lots of tables in it. Let’s say it has Table Student in it, and we want to mock create a Student database call.

    function async AddStudent(student: Student) 
      {
            await db.Student.create(student) // the call we want to mock
     }

    Now, as we are using the Jest spy method, we know that it will only be applicable to objects, so we will first make the Db. Students table into an object with create as method inside it, which will be jest.fn() (a function which can be used for mocking functions).

    Students an object with create as method inside object which will be jest.fn() (a function which can be used for mocking functions in one line without actually calling that function).

    describe('mocking data base call', () => {
            it('mocking create function', async () => {
                db.Student = {
                    create: jest.fn()
                }
    
                const tempStudent = {
                    name: 'john',
                    age: '12',
                    Rollno: 12
                     }
    
                const mock = jest.spyon(db.Student, 'create').
                    mockResolvedvalue('Student has been created successfully')
    
                await AddStudent(tempStudent)
                expect(mock).tohaveBeenCalledwith(tempStudent);
    
            })
    
        })

    Testing private methods‍

    Sometime, in development, we write private code for classes that can only be used within the class itself. But when writing test cases, we call the function by creating a class instance, and the private functions won’t be accessible to us, so we will not be able to test private functions.

    But in core JavaScript, there is no concept of private and public functions; it is introduced to us as TypeScript. So, we can actually test the private function as a normal public function by using the //@ts-ignore comment just above calling the private function.

     class Test()
      {
    
            private private_fun() {
                console.log("i am in private function");
                return "i am in private function"
            }
    
        }

    describe('Testing test class', () => {
            it('testing private function', () => {
                const test = new Test() 
                
                //calling code with ts-ignore comment
    
                //@ts-ignore
                const res = test.private_fun() //  output ->> "i am in private function "//
                expect(res).toBeEqual("i am in private function")
    
            })
        })

    P.S. One thing to note is that this will only work with TypeScript/JavaScript files.

    The importance of test cases in software development

    Makes code agile:

    In software development, one may have to change the structure or design of your code to add new features. Changing the already-tested code can be risky and costly. When you do the unit test, you just need to test the newly added code instead of the entire program.

    Improves code quality:

    A lot of bugs in software development occur due to unforeseen edge cases. If you forget to predict a single input, you may encounter a major bug in your application. When you write unit tests, think carefully about the edge cases of every function in your application.

    Provides Documentation:

    The unit test gives a basic idea of what the code does, and all the different use cases are covered through the program. It makes documentation easier, increasing the readability and understandability of the code. Anytime other developers can go through the unit test interface, understand the program better, and work on it fast and easily.

    Easy Debugging:

    Unit testing has made debugging a lot easier and quicker. If the test fails at any stage, you only need to debug the latest changes made in the code instead of the entire program. We have also mentioned how unit testing makes debugging easier at the next stage of integration testing as well.

    Conclusion

    So, if you made it to the end, you must have some understanding of the importance of test cases in your code.

    We’ve covered the best framework to choose from and how to write your first test case in Jest. And now, you are more confident in proving bug-free, robust, clean, documented, and tested code in your next MR/PR.

  • Serverpod: The Ultimate Backend for Flutter

    Join us on this exhilarating journey, where we bridge the gap between frontend and backend development with the seamless integration of Serverpod and Flutter.

    Gone are the days of relying on different programming languages for frontend and backend development. With Flutter’s versatile framework, you can effortlessly create stunning user interfaces for a myriad of platforms. However, the missing piece has always been the ability to build the backend in Dart as well—until now.

    Introducing Serverpod, the missing link that completes the Flutter ecosystem. Now, with Serverpod, you can develop your entire application, from frontend to backend, all within the familiar and elegant Dart language. This synergy enables a seamless exchange of data and functions between the client and the server, reducing development complexities and boosting productivity.

    1. What is Serverpod?

    As a developer or tech enthusiast, we recognize the critical role backend services play in the success of any application. Whether you’re building a web, mobile, or desktop project, a robust backend infrastructure is the backbone that ensures seamless functionality and scalability.

    That’s where “Serverpod” comes into the picture—an innovative backend solution developed entirely in Dart, just like your Flutter projects. With Serverpod at your disposal, you can harness the full power of Dart on both the frontend and backend, creating a harmonious development environment that streamlines your workflow.

    The biggest advantage of using Serverpod is that it automates protocol and client-side code generation by analyzing your server, making remote endpoint calls as simple as local method calls.

    1.1. Current market status

    The top 10 programming languages for backend development in 2023 are as follows: 

    [Note: The results presented here are not absolute and are based on a combination of surveys conducted in 2023, including ‘Stack Overflow Developer Survey – 2023,’ ‘State of the Developer Ecosystem Survey,’ ‘New Stack Developer Survey,’ and more.]

    • Node.js – ~32%
    • Python (Django, Flask) – ~28%
    • Java (Spring Boot, Java EE) – ~18%
    • Ruby (Ruby on Rails) – ~7%
    • PHP (Laravel, Symfony) – ~6%
    • Go (Golang) – ~3%
    • .NET (C#) – ~2%
    • Rust – Approximately 1%
    • Kotlin (Spring Boot with Kotlin) – ~1%
    • Express.js (for Node.js) – ~1%
    Figure 01

    Figure 01 provides a comprehensive overview of the current usage of backend development technologies, showcasing a plethora of options with diverse features and capabilities. However, the landscape takes a different turn when it comes to frontend development. While the backend technologies offer a wealth of choices, most of these languages lack native multiplatform support for frontend applications.

    As a result, developers find themselves in a situation where they must choose between two sets of languages or technologies for backend and frontend business logic development.

    1.2. New solution

    As the demand for multiplatform applications continues to grow, developers are actively exploring new frameworks and languages that bridge the gap between backend and frontend development. Recently, a groundbreaking solution has emerged in the form of Serverpod. With Serverpod, developers can now accomplish server development in Dart, filling the crucial gap that was previously missing in the Flutter ecosystem.

    Flutter has already demonstrated its remarkable support for a wide range of platforms. The absence of server development capabilities was a notable limitation that has now been triumphantly addressed with the introduction of Serverpod. This remarkable achievement enables developers to harness the power of Dart to build both frontend and backend components, creating unified applications with a shared codebase.

    2. Configurations 

    Prior to proceeding with the code implementation, it is essential to set up and install the necessary tools.

    [Note: Given Serverpod’s initial stage, encountering errors without readily available online solutions is plausible. In such instances, seeking assistance from the Flutter community forum is highly recommended. Drawing from my experience, I suggest running the application on Flutter web first, particularly for Serverpod version 1.1.1, to ensure a smoother development process and gain insights into potential challenges.]

    2.1. Initial setup

    2.1.1 Install Docker

    Docker serves a crucial role in Serverpod, facilitating:

    • Containerization: Applications are packaged and shipped as containers, enabling seamless deployment and execution across diverse infrastructures.
    • Isolation: Applications are isolated from one another, enhancing both security and performance aspects, safeguarding against potential vulnerabilities, and optimizing system efficiency.

    Download & Install Docker from here.

    2.1.2 Install Serverpod CLI 

    • Run the following command:
    dart pub global activate serverpod_cli

    • Now test the installation by running:
    serverpod

    With proper configuration, the Serverpod command displays help information.

    2.2. Project creation

    To initiate serverpod commands, the Docker application must be launched first. Ensuring an active Docker instance in the backend environment is imperative to execute Serverpod commands successfully.

    • Create a new project with the command:
    serverpod create <your_project_name>

    Upon execution, a new directory will be generated with the specified project name, comprising three Dart packages:

    <your_project_name>_server: This package is designated for server-side code, encompassing essential components such as business logic, API endpoints, DB connections, and more.
    <your_project_name>_client: Within this package, the code responsible for server communication is auto-generated. Manual editing of files in this package is typically avoided.
    <your_project_name>_flutter: Representing the Flutter app, it comes pre-configured to seamlessly connect with your local server, ensuring seamless communication between frontend and backend elements.

    2.3. Project execution

    Step 1: Navigate to the server package with the following command:

    cd <your_project_name>/<your_project_name>_server

    Step 2: (Optional) Open the project in the VS Code IDE using the command:

    (Note: You can use any IDE you prefer, but for our purposes, we’ll use VS Code, which also simplifies DB connection later.)

    code .

    Step 3: Once the project is open in the IDE, stop any existing Docker containers with this command:

    .setup-tables.cmd

    Step 4: Before starting the server, initiate new Docker containers with the following command:

    docker-compose up --build --detach

    Step 5: The command above will start PostgreSQL and Redis containers, and you should receive the output:

    ~> docker-compose up --build --detach
    	[+] Running 2/2
     	✔ Container <your_project_name>_server-redis-1     Started                                                                                                
     	✔ Container <your_project_name>_server-postgres-1  Started

    (Note: If the output doesn’t match, refer to this Stack Overflow link for missing commands in the official documentation.)

    Step 6: Proceed to start the server with this command:

    dart bin/main.dart

    Step 7: Upon successful execution, you will receive the following output, where the “Server Default listening on port” value is crucial. Please take note of this value.

    ~> dart bin/main.dart
     	SERVERPOD version: 1.1.1, dart: 3.0.5 (stable) (Mon Jun 12 18:31:49 2023 +0000) on "windows_x64", time: 2023-07-19 15:24:27.704037Z
     	mode: development, role: monolith, logging: normal, serverId: default
     	Insights listening on port 8081
     	Server default listening on port 8080
     	Webserver listening on port 8082
     	CPU and memory usage metrics are not supported on this platform.

    Step 8: Use the “Server Default listening on port” value after “localhost,” i.e., “127.0.0.1,” and load this URL in your browser. Accessing “localhost:8080” will display the desired output, indicating that your server is running and ready to process requests.

    Figure 02

    Step 9: Now, as the containers reach the “Started” state, you can establish a connection with the database. We have opted for PostgreSQL as our DB choice, and the rationale behind this selection lies in the “docker-compose.yaml” file at the server project’s root. In the “service” section, PostgreSQL is already added, making it an ideal choice as the required setup is readily available. 

    Figure 03

    For the database setup, we need key information, such as Host, Port, Username, and Password. You can find all this vital information in the “config” directory’s “development.yaml” and “passwords.yaml” files. If you encounter difficulties locating these details, please refer to Figure 04.

    Figure 04

    Step 10: To establish the connection, you can install an application similar to Postico or, alternatively, I recommend using the MySQL extension, which can be installed in your VS Code with just one click.

    Figure 05

    Step 11: Follow these next steps:

    1. Select the “Database” option.
    2. Click on “Create Connection.”
    3. Choose the “PostgreSQL” option.
    4. Add a name for your Connection.
    5. Fill in the information collected in the last step.
    6. Finally, select the “Connect” option.
    Figure 06
    1. Upon success, you will receive a “Connect Success!” message, and the new connection will be added to the Explorer Tab.
    Figure 07

    Step 12: Now, we shift our focus to the Flutter project (Frontend):

    Thus far, we have been working on the server project. Let us open a new VS Code instance for a separate Flutter project while keeping the current VS Code instance active, serving as the server.

    Step 13: Execute the following command to run the Flutter project on Chrome:

    flutter run -d chrome

    With this, the default project will generate the following output:

    Step 14: When you are finished, you can shut down Serverpod with “Ctrl-C.”

    Step 15: Then stop Postgres and Redis.

    docker compose stop

    Figure 08

    3. Sample Project

    So far, we have successfully created and executed the project, identifying three distinct components. The server project caters to server/backend development, while the Flutter project handles application/frontend development. The client project, automatically generated, serves as the vital intermediary, bridging the gap between the frontend and backend.

    However, merely acknowledging the projects’ existence is insufficient. To maximize our proficiency, it is crucial to grasp the code and file structure comprehensively. To achieve this, we will embark on a practical journey, constructing a small project to gain hands-on experience and unlock deeper insights into all three components. This approach empowers us with a well-rounded understanding, further enhancing our capabilities in building remarkable applications.

    3.1. What are we building?

    In this blog, we will construct a sample project with basic Login and SignUp functionality. The SignUp process will collect user information such as Email, Password, Username, and age. Users can subsequently log in using their email and password, leading to the display of user details on the dashboard screen. With the initial system setup complete and the newly created project up and running, it’s time to commence coding. 

    3.1.1 Create custom models for API endpoints

    Step1: Create a new file in the “lib >> src >> protocol” directory named “users.yaml”:

    class: Users
    table: users
    fields:
      username: String
      email: String
      password: String
      age: int

    Step 2: Save the file and run the following command to generate essential data classes and table creation queries:

    serverpod generate

    (Note: Add “–watch” after the command for continuous code generation). 

    Successful execution of the above command will generate a new file named “users.dart” in the “lib >> src >> generated” folder. Additionally, the “tables.pgsql” file now contains SQL queries for creating the “users” table. The same command updates the auto-generated code in the client project. 

    3.1.2 Create Tables in DB for the generated model 

    Step 1: Copy the queries written in the “generated >> tables.pgsql” file.

    In the MySQL Extension’s Database section, select the created database >> [project_name] >> public >> Tables >> + (Create New Table).

    Figure 09

    Step 2: Paste the queries into the newly created .sql file and click “Execute” above both queries.

    Figure 10

    Step 3: After execution, you will obtain an empty table with the “id” as the Primary key.

    Figure 11

    If you found multiple tables already present in your database like shown in the next figure, you can ignore those. These tables are created by the system with queries present in the “generated >> tables-serverpod.pgsql” file.

    Figure 12

    3.1.3 Create an API endpoint

    Step 1: Generate a new file in the “lib >> src >> endpoints” directory named “session_endpoints.dart”:

    class SessionEndpoint extends Endpoint {
      Future<Users?> login(Session session, String email, String password) async {
        List<Users> userList = await Users.find(session,
            where: (p0) =>
                (p0.email.equals(email)) & (p0.password.equals(password)));
        return userList.isEmpty ? null : userList[0];
      }
    
    
      Future<bool> signUp(Session session, Users newUser) async {
        try {
          await Users.insert(session, newUser);
          return true;
        } catch (e) {
          print(e.toString());
          return false;
        }
      }
    }

    If “serverpod generate –watch” is already running, you can ignore this step 2.

    Step 2: Run the command:

    serverpod generate

    Step 3: Start the server.
    [For help, check out Step 1 Step 6 mentioned in Project Execution part.]

    3.1.3 Create three screens

    Login Screen:

    Figure 13

    SignUp Screen:

    Figure 14

    Dashboard Screen:

    Figure 15

    3.1.4 Setup Flutter code

    Step 1: Add the code provided to the SignUp button in the SignUp screen to handle user signups.

    try {
            final result = await client.session.signUp(
              Users(
                email: _emailEditingController.text.trim(),
                username: _usernameEditingController.text.trim(),
                password: _passwordEditingController.text.trim(),
                age: int.parse(_ageEditingController.text.trim()),
              ),
            );
            if (result) {
              Navigator.pop(context);
            } else {
              _errorText = 'Something went wrong, Try again.';
            }
          } catch (e) {
            debugPrint(e.toString());
            _errorText = e.toString();
          }

    Step 2: Add the code provided to the Login button in the Login screen to handle user logins.

    try {
            final result = await client.session.login(
              _emailEditingController.text.trim(),
              _passwordEditingController.text.trim(),
            );
            if (result != null) {
              _emailEditingController.text = '';
              _passwordEditingController.text = '';
              Navigator.push(
                context,
                MaterialPageRoute(
                  builder: (context) => DashboardPage(user: result),
                ),
              );
            } else {
              _errorText = 'Something went wrong, Try again.';
            }
          } catch (e) {
            debugPrint(e.toString());
            _errorText = e.toString();
          }

    Step 3: Implement logic to display user data on the dashboard screen.

    With these steps completed, our Flutter app becomes a fully functional project, showcasing the power of this new technology. Armed with Dart knowledge, every Flutter developer can transform into a proficient full-stack developer.

    4. Result

    Figure 16

    To facilitate your exploration, the entire project code is conveniently available in this code repository. Feel free to refer to this repository for an in-depth understanding of the implementation details and access to the complete source code, enabling you to delve deeper into the project’s intricacies and leverage its functionalities effectively.

    5. Conclusion

    In conclusion, we have provided a comprehensive walkthrough of the step-by-step setup process for running Serverpod seamlessly. We explored creating data models, integrating the database with our server project, defining tables, executing data operations, and establishing accessible API endpoints for Flutter applications.

    Hopefully, this blog post has kindled your curiosity to delve deeper into Serverpod’s immense potential for elevating your Flutter applications. Embracing Serverpod unlocks a world of boundless possibilities, empowering you to achieve remarkable feats in your development endeavors.

    Thank you for investing your time in reading this informative blog!

    6. References

    1. https://docs.flutter.dev/
    2. https://pub.dev/packages/serverpod/
    3. https://serverpod.dev/
    4. https://docs.docker.com/get-docker/
    5. https://medium.com/serverpod/introducing-serverpod-a-complete-backend-for-flutter-written-in-dart-f348de228e19
    6. https://medium.com/serverpod/serverpod-our-vision-for-a-seamless-scalable-backend-for-the-flutter-community-24ba311b306b
    7. https://stackoverflow.com/questions/76180598/serverpod-sql-error-when-starting-a-clean-project
    8. https://www.youtube.com/watch?v=3Q2vKGacfh0
    9. https://www.youtube.com/watch?v=8sCxWBWhm2Y

  • Cube – An Innovative Framework to Build Embedded Analytics

    Historically, embedded analytics was thought of as an integral part of a comprehensive business intelligence (BI) system. However, when we considered our particular needs, we soon realized something more innovative was necessary. That is when we came across Cube (formerly CubeJS), a powerful platform that could revolutionize how we think about embedded analytics solutions.

    This new way of modularizing analytics solutions means businesses can access the exact services and features they require at any given time without purchasing a comprehensive suite of analytics services, which can often be more expensive and complex than necessary.

    Furthermore, Cube makes it very easy to link up data sources and start to get to grips with analytics, which provides clear and tangible benefits for businesses. This new tool has the potential to be a real game changer in the world of embedded analytics, and we are very excited to explore its potential.

    Understanding Embedded Analytics

    When you read a word like “embedded analytics” or something similar, you probably think of an HTML embed tag or an iFrame tag. This is because analytics was considered a separate application and not part of the SaaS application, so the market had tools specifically for analytics.

    “Embedded analytics is a digital workplace capability where data analysis occurs within a user’s natural workflow, without the need to toggle to another application. Moreover, embedded analytics tends to be narrowly deployed around specific processes such as marketing campaign optimization, sales lead conversions, inventory demand planning, and financial budgeting.” – Gartner

    Embedded Analytics is not just about importing data into an iFrame—it’s all about creating an optimal user experience where the analytics feel like they are an integral part of the native application. To ensure that the user experience is as seamless as possible, great attention must be paid to how the analytics are integrated into the application. This can be done with careful thought to design and by anticipating user needs and ensuring that the analytics are intuitive and easy to use. This way, users can get the most out of their analytics experience.

    Existing Solutions

    With the rising need for SaaS applications and the number of SaaS applications being built daily, analytics must be part of the SaaS application.

    We have identified three different categories of exciting solutions available in the market.

    Traditional BI Platforms

    Many tools, such as GoodData, Tableau, Metabase, Looker, and Power BI, are part of the big and traditional BI platforms. Despite their wide range of features and capabilities, these platforms need more support with their Big Monolith Architecture, limited customization, and less-than-intuitive user interfaces, making them difficult and time-consuming.

    Here are a few reasons these are not suitable for us:

    • They lack customization, and their UI is not intuitive, so they won’t be able to match our UX needs.
    • They charge a hefty amount, which is unsuitable for startups or small-scale companies.
    • They have a big monolith architecture, making integrating with other solutions difficult.

    New Generation Tools

    The next experiment taking place in the market is the introduction of tools such as Hex, Observable, Streamlit, etc. These tools are better suited for embedded needs and customization, but they are designed for developers and data scientists. Although the go-to-market time is shorter, all these tools cannot integrate into SaaS applications.

    Here are a few reasons why these are not suitable for us:

    • They are not suitable for non-technical people and cannot integrate with Software-as-a-Service (SaaS) applications.
    • Since they are mainly built for developers and data scientists, they don’t provide a good user experience.
    • They are not capable of handling multiple data sources simultaneously.
    • They do not provide pre-aggregation and caching solutions.

    In House Tools

    Building everything in-house, instead of paying other platforms to build everything from scratch, is possible using API servers and GraphQL. However, there is a catch: the requirements for analytics are not straightforward, which will require a lot of expertise to build, causing a big hurdle in adaptation and resulting in a longer time-to-market.

    Here are a few reasons why these are not suitable for us:

    • Building everything in-house requires a lot of expertise and time, thus resulting in a longer time to market.
    • It requires developing a secure authentication and authorization system, which adds to the complexity.
    • It requires the development of a caching system to improve the performance of analytics.
    • It requires the development of a real-time system for dynamic dashboards.
    • It requires the development of complex SQL queries to query multiple data sources.

    Typical Analytics Features

    If you want to build analytics features, the typical requirements look like this:

    Multi-Tenancy

    When developing software-as-a-service (SaaS) applications, it is often necessary to incorporate multi-tenancy into the architecture. This means multiple users will be accessing the same software application, but with a unique and individualized experience. To guarantee that this experience is not compromised, it is essential to ensure that the same multi-tenancy principles are carried over into the analytics solution that you are integrating into your SaaS application. It is important to remember that this will require additional configuration and setup on your part to ensure that all of your users have access to the same level of tools and insights.

    Intuitive Charts

    If you look at some of the available analytics tools, they may have good charting features, but they may not be able to meet your specific UX needs. In today’s world, many advanced UI libraries and designs are available, which are often far more effective than the charting features of analytics tools. Integrating these solutions could help you create a more user-friendly experience tailored specifically to your business requirements.

    Security

    You want to have authentication and authorization for your analytics so that managers can get an overview of the analytics for their entire team, while individual users can only see their own analytics. Furthermore, you may want to grant users with certain roles access to certain analytics charts and other data to better understand how their team is performing. To ensure that your analytics are secure and that only the right people have access to the right information, it is vital to set up an authentication and authorization system.

    Caching

    Caching is an incredibly powerful tool for improving the performance and economics of serving your analytics. By implementing a good caching solution, you can see drastic improvements in the speed and efficiency of your analytics, while also providing an improved user experience. Additionally, the cost savings associated with this approach can be quite significant, providing you with a greater return on investment. Caching can be implemented in various ways, but the most effective approaches are tailored to the specific needs of your analytics. By leveraging the right caching solutions, you can maximize the benefits of your analytics and ensure that your users have an optimized experience.

    Real-time

    Nowadays, every successful SaaS company understands the importance of having dynamic and real-time dashboards; these dashboards provide users with the ability to access the latest data without requiring them to refresh the tab each and every time. By having real-time dashboards, companies can ensure their customers have access to the latest information, which can help them make more informed decisions. This is why it is becoming increasingly important for SaaS organizations to invest in robust, low-latency dashboard solutions that can deliver accurate, up-to-date data to their customers.

    Drilldowns

    Drilldown is an incredibly powerful analytics capability that enables users to rapidly transition from an aggregated, top-level overview of their data to a more granular, in-depth view. This can be achieved simply by clicking on a metric within a dashboard or report. With drill-down, users can gain a greater understanding of the data by uncovering deeper insights, allowing them to more effectively evaluate the data and gain a more accurate understanding of their data trends.

    Data Sources

    With the prevalence of software as a service (SaaS) applications, there could be a range of different data sources used, including PostgreSQL, DynamoDB, and other types of databases. As such, it is important for analytics solutions to be capable of accommodating multiple data sources at once to provide the most comprehensive insights. By leveraging the various sources of information, in conjunction with advanced analytics, businesses can gain a thorough understanding of their customers, as well as trends and behaviors. Additionally, accessing and combining data from multiple sources can allow for more precise predictions and recommendations, thereby optimizing the customer experience and improving overall performance.

    Budget

    Pricing is one of the most vital aspects to consider when selecting an analytics tool. There are various pricing models, such as AWS Quick-sight, which can be quite complex, or per-user basis costs, which can be very expensive for larger organizations. Additionally, there is custom pricing, which requires you to contact customer care to get the right pricing; this can be quite a difficult process and may cause a big barrier to adoption. Ultimately, it is important to understand the different pricing models available and how they may affect your budget before selecting an analytics tool.

    After examining all the requirements, we came across a solution like Cube, which is an innovative solution with the following features:

    • Open Source: Since it is open source, you can easily do a proof-of-concept (POC) and get good support, as any vulnerabilities will be fixed quickly.
    • Modular Architecture: It can provide good customizations, such as using Cube to use any custom charting library you prefer in your current framework.
    • Embedded Analytics-as-a-Code: You can easily replicate your analytics and version control it, as Cube is analytics in the form of code.
    • Cloud Deployments: It is a new-age tool, so it comes with good support with Docker or Kubernetes (K8s). Therefore, you can easily deploy it on the cloud.

    Cube Architecture

    Let’s look at the Cube architecture to understand why Cube is an innovative solution.

    • Cube supports multiple data sources simultaneously; your data may be stored in Postgres, Snowflake, and Redshift, and you can connect to all of them simultaneously. Additionally, they have a long list of data sources they can support.
    • Cube provides analytics over a REST API; very few analytics solutions provide chart data or metrics over REST APIs.
    • The security you might be using for your application can easily be mirrored for Cube. This helps simplify the security aspects, as you don’t need to maintain multiple tokens for the app and analytics tool.
    • Cube provides a unique way to model your data in JSON format; it’s more similar to an ORM. You don’t need to write complex SQL queries; once you model your data, Cube will generate the SQL to query the data source.
    • Cube has very good pre-aggregation and caching solutions.

    Cube Deep Dive

    Let’s look into different concepts that we just saw briefly in the architecture diagram.

    Data Modeling

    Cube

    A cube represents a table of data and is conceptually similar to a view in SQL. It’s like an ORM where you can define schema, extend it, or define abstract cubes to make use of code reusable. For example, if you have a Customer table, you need to write a Cube for it. Using Cubes, you can build analytical queries.

    Each cube contains definitions of measures, dimensions, segments, and joins between cubes. Cube bifurcates columns into measures and dimensions. Similar to tables, every cube can be referenced in another cube. Even though a cube is a table representation, you can choose which columns you want to expose for analytics. You can only add columns you want to expose to analytics; this will translate into SQL for the dimensions and measures you use in the SQL query (Push Down Mechanism).

    cube('Orders', {
      sql: `SELECT * FROM orders`,
    });

    Dimensions

    You can think about a dimension as an attribute related to a measure, for example, the measure userCount. This measure can have different dimensions, such as country, age, occupation, etc.

    Dimensions allow us to further subdivide and analyze the measure, providing a more detailed and comprehensive picture of the data.

    cube('Orders', {
    
      ...,
    
      dimensions: {
        status: {
          sql: `status`,
          type: `string`},
      },
    });

    Measures

    These parameters/SQL columns allow you to define the aggregations for numeric or quantitative data. Measures can be used to perform calculations such as sum, minimum, maximum, average, and count on any set of data.

    Measures also help you define filters if you want to add some conditions for a metric calculation. For example, you can set thresholds to filter out any data that is not within the range of values you are looking for.

    Additionally, measures can be used to create additional metrics, such as the ratio between two different measures or the percentage of a measure. With these powerful tools, you can effectively analyze and interpret your data to gain valuable insights.

    cube('Orders', {
    
      ...,
    
      measures: {
        count: {
          type: `count`,
        },
      },
    });

    Joins

    Joins define the relationships between cubes, which then allows accessing and comparing properties from two or more cubes at the same time. In Cube, all joins are LEFT JOINs. This also allows you to represent one-to-one, many-to-one relationships easily.

    cube('Orders', {
    
      ...,
    
      joins: {
        LineItems: {
          relationship: `belongsTo`,
          // Here we use the `CUBE` global to refer to the current cube,
          // so the following is equivalent to `Orders.id = LineItems.order_id`
          sql: `${CUBE}.id = ${LineItems}.order_id`,
        },
      },
    });

    There are three kinds of join relationships:

    • belongsTo
    • hasOne
    • hasMany

    Segments

    Segments are filters predefined in the schema instead of a Cube query. Segments help pre-build complex filtering logic, simplifying Cube queries and making it easy to re-use common filters across a variety of queries.

    To add a segment that limits results to completed orders, we can do the following:

    cube('Orders', {
      ...,
      segments: {
        onlyCompleted: {
          sql: `${CUBE}.status = 'completed'`},
      },
    });

    Pre-Aggregations

    Pre-aggregations are a powerful way of caching frequently-used, expensive queries and keeping the cache up-to-date periodically. The most popular roll-up pre-aggregation is summarized data of the original cube grouped by any selected dimensions of interest. It works on “measure types” like count, sum, min, max, etc.

    Cube analyzes queries against a defined set of pre-aggregation rules to choose the optimal one that will be used to create pre-aggregation table. When there is a smaller dataset that queries execute over, the application works well and delivers responses within acceptable thresholds. However, as the size of the dataset grows, the time-to-response from a user’s perspective can often suffer quite heavily. It specifies attributes from the source, which Cube uses to condense (or crunch) the data. This simple yet powerful optimization can reduce the size of the dataset by several orders of magnitude, and ensures subsequent queries can be served by the same condensed dataset if any matching attributes are found.

    Even granularity can be specified, which defines the granularity of data within the pre-aggregation. If set to week, for example, then Cube will pre-aggregate the data by week and persist it to Cube Store.

    Cube can also take care of keeping pre-aggregations up-to-date with the refreshKey property. By default, it is set to every: ‘1 hour’.

    cube('Orders', {
    
      ...,
    
      preAggregations: {
        main: {
          measures: [CUBE.count],
          dimensions: [CUBE.status],
          timeDimension: CUBE.createdAt,
          granularity: 'day',
        },
      },
    });

    Additional Cube Concepts

    Let’s look into some of the additional concepts that Cube provides that make it a very unique solution.

    Caching

    Cube provides a two-level caching system. The first level is in-memory cache, which is active by default. Cube in-memory cache acts as a buffer for your database when there is a burst of requests hitting the same data from multiple concurrent users, while pre-aggregations are designed to provide the right balance between time to insight and querying performance.

    The second level of caching is called pre-aggregations, and requires explicit configuration to activate.

    Drilldowns

    Drilldowns are a powerful feature to facilitate data exploration. It allows building an interface to let users dive deeper into visualizations and data tables. See ResultSet.drillDown() on how to use this feature on the client side.

    A drilldown is defined on the measure level in your data schema. It is defined as a list of dimensions called drill members. Once defined, these drill members will always be used to show underlying data when drilling into that measure.

    Subquery

    You can use subqueries within dimensions to reference measures from other cubes inside a dimension. Under the hood, it behaves as a correlated subquery, but is implemented via joins for optimal performance and portability.

    For example, the following SQL can be written using a subquery in cubes as:

    SELECT
      id,
      (SELECT SUM(amount)FROM dealsWHERE deals.sales_manager_id = sales_managers.id)AS deals_amount
    FROM sales_managers
    GROUPBY 1

    Cube Representation

    cube(`Deals`, {
      sql: `SELECT * FROM deals`,
      measures: {
        amount: {
          sql: `amount`,
          type: `sum`,
        },
      },
    });
    
    cube(`SalesManagers`, {
      sql: `SELECT * FROM sales_managers`,
    
      joins: {
        Deals: {
          relationship: `hasMany`,
          sql: `${SalesManagers}.id = ${Deals}.sales_manager_id`,
        },
      },
    
      measures: {
        averageDealAmount: {
          sql: `${dealsAmount}`,
          type: `avg`,
        },
      },
    
      dimensions: {
        dealsAmount: {
          sql: `${Deals.amount}`,
          type: `number`,
          subQuery: true,
        },
      },
    });

    Apart from these, Cube also provides advanced concepts such as Export and Import, Extending Cubes, Data Blending, Dynamic Schema Creation, and Polymorphic Cubes. You can read more about them in the Cube documentation.

    Getting Started with Cube

    Getting started with Cube is very easy. All you need to do is follow the instructions on the Cube documentation page.

    To get started you can use Docker to get started quickly. With Docker, you can install Cube in a few easy steps:

    1. In a new folder for your project, run the following command:

    docker run -p 4000:4000 -p 3000:3000 
      -v ${PWD}:/cube/conf 
      -e CUBEJS_DEV_MODE=true 
      cubejs/cube

    2. Head to http://localhost:4000 to open Developer Playground.

    The Developer Playground has a database connection wizard that loads when Cube is first started up and no .env file is found. After database credentials have been set up, an .env file will automatically be created and populated with the same credentials.

    Click on the type of database to connect to, and you’ll be able to enter credentials:

    After clicking Apply, you should see available tables from the configured database. Select one to generate a data schema. Once the schema is generated, you can execute queries on the Build tab.****

    Conclusion

    Cube is a revolutionary, open-source framework for building embedded analytics applications. It offers a unified API for connecting to any data source, comprehensive visualization libraries, and a data-driven user experience that makes it easy for developers to build interactive applications quickly. With Cube, developers can focus on the application logic and let the framework take care of the data, making it an ideal platform for creating data-driven applications that can be deployed on the web, mobile, and desktop. It is an invaluable tool for any developer interested in building sophisticated analytics applications quickly and easily.