Cloud costs rarely spiral out of control overnight. More often, they drift quietly and steadily until finance teams are left explaining overruns and engineering teams are asked to “optimize” after the fact.
This reactive approach to FinOps is becoming harder to sustain. Cloud environments today are far more dynamic than the tools and processes designed to manage them. Monthly reviews, static rules, and backward-looking reports simply cannot keep up.
This is where AI-driven FinOps steps in. Not as another dashboard, but as the next evolution of FinOps itself but one that helps teams predict what’s coming, prevent waste before it happens, and continuously improve performance.
From Cost Visibility to Cost Intelligence
Traditional FinOps gives you visibility. You can see where money is being spent, which teams own which resources, and how costs trend over time. That foundation still matters.
But visibility alone doesn’t answer the questions that really matter now:
Where is spend likely to increase next?
Which workloads are behaving differently than expected?
What should teams act on today, not at the end of the month?
AI adds intelligence to FinOps by connecting historical patterns with real-time data. Instead of just reporting on spend, AI helps teams understand why costs are changing and what to do about it.
Predict: Forecasting That Keeps Up with Change
Forecasting cloud spend has always been difficult. Usage shifts with new releases, customer demand, and infrastructure changes, often making static forecasts outdated almost as soon as they’re created.
AI-driven FinOps improves this by:
Continuously forecasting spend using live usage data
Learning from patterns like seasonality and growth trends
Adjusting predictions as workloads and architectures evolve
The result is forecasting that feels less like guesswork and more like guidance. Finance teams gain clearer budget visibility, while engineering teams better understand how their decisions shape future costs.
Prevent: Catching Anomalies Before They Become Problems
In many organizations, cost anomalies are discovered only after the bill arrives. By then, teams are already behind.
AI changes that dynamic. By learning what “normal” looks like for each workload, AI-powered FinOps tools can spot unusual behavior as it happens whether it’s a sudden traffic spike, a misconfigured autoscaling rule, or resources running idle longer than expected.
Even more important, these alerts are contextual. They don’t just flag a spike; they explain where it’s coming from and why it matters. That clarity helps teams respond faster, with less finger-pointing and fewer manual investigations.
Perform: Continuous Optimization, Not Periodic Cleanup
FinOps works best when finance and engineering operate as partners, not gatekeepers and enforcers. AI makes that collaboration easier by translating complex cost data into insights each team can act on.
With predictive insights in place:
Finance teams can focus on planning and accountability, not policing
Engineering teams can design with cost in mind, without slowing delivery
Optimization becomes ongoing, not something squeezed into quarterly reviews
Savings are identified earlier, responses are faster, and performance goals stay intact, all without adding operational overhead.
Case Study: Optimizing Petabyte-Scale Workloads for Cost and Continuity
The value of AI-driven FinOps becomes clear at scale.
A content-intelligence platform processing petabytes of data every day needed to control cloud costs without compromising performance or availability. Manual reviews and static optimization rules were no longer enough.
By introducing predictive planning and real-time anomaly detection, the organization gained early visibility into cost deviations and the ability to act before issues escalated.
The R Systems Approach: AI-Powered FinOps, Built for Continuous Optimization
AI is powerful, but it delivers real value only when embedded into everyday cloud operations.
R Systems brings together AI-driven forecasting and anomaly detection with continuous optimization practices that align finance, engineering, and operations. The focus is not on one-time savings, but on building a FinOps capability that evolves alongside the cloud environment.
The outcome is a FinOps model that is proactive, collaborative, and resilient, designed to keep pace with both growth and change.
As cloud environments grow more complex, the cost of reacting late keeps rising. AI-driven FinOps offers a practical alternative: predict earlier, prevent waste, and perform with confidence.
For organizations that see cloud efficiency as a long-term discipline and not a quarterly exercise, there AI is no longer optional. It is foundational.
Agentic AI has moved quickly from experimentation to expectation. Most enterprises today have pilots in motion, proofs of concept delivering early promise, and leadership teams asking a sharper question: How do we scale this safely, reliably, and with real business impact?
That question is often followed by fatigue. Too many pilots stall. Too many promising demos fail to survive real-world complexity. And too often, the issue isn’t the technology itself.
The uncomfortable truth is this: most agentic AI failures are not technology failures. They are partner failures.
As enterprises move from pilots to production especially within Global Capability Centers (GCCs), partner selection has become a strategic decision, not a procurement one. The difference between experimentation and enterprise value increasingly comes down to who you build with.
Why Partner Choice Matters More Than Ever
Agentic AI is fundamentally different from earlier waves of automation. It introduces autonomy into business workflows, systems that can sense, decide, and act with limited human intervention.
That kind of capability doesn’t scale through tools alone.
Scaling agentic AI requires deep enterprise context, operating-model alignment, strong governance, and ownership of outcomes. Yet many organizations still choose partners based on narrow criteria: a compelling demo, a preferred toolset, or short-term cost efficiency.
Those choices may work for pilots. They rarely work for production.
As organizations mature, a clear realization is emerging: the partner matters as much as the platform or often more.
Innovation Readiness Is Not Optional
Agentic AI is advancing faster than most enterprise operating models can comfortably absorb. New orchestration patterns, reasoning techniques, safety mechanisms, and runtime optimizations are emerging at a pace that outstrips traditional delivery and governance cycles.
In such an environment, partner capability cannot remain static. Enterprises need partners with a sustained capacity for innovation not merely the ability to implement what is already familiar.
The most effective agentic AI partners operate through a mature AI Center of Excellence: one that systematically experiments, evaluates new tools and approaches, and converts what proves viable into production-ready practices before they enter core enterprise systems.
Without this discipline, organizations risk committing too early to architectural choices that do not age well, making choices that introduce technical debt, constrain future evolution, and limit the scope of autonomy over time.
Innovation readiness in agentic AI, then, is not a matter of chasing what is new. It is the ability to distinguish signal from noise, to decide deliberately what belongs in production, and to industrialize proven approaches with consistency, safety, and repeatability.
The Common Partner Pitfalls
Most enterprises don’t choose the wrong partners intentionally. They choose partners that are right for a different stage of maturity.
Some common pitfalls we see:
Tool-first vendors who excel at showcasing AI capabilities but lack experience running mission-critical enterprise systems.
Traditional system integrators with scale and delivery muscle, but limited depth in agentic AI design and orchestration.
Niche AI firms that can build impressive pilots but struggle with integration, governance, and long-term operations.
Delivery partners focused on execution, not accountability leaving enterprises to own risk, outcomes, and scale alone.
Partners who lack domain or functional depth, resulting in agents that understand tools but not the business context, decision logic, or real operational constraints.
None of these partners are inherently flawed. But agentic AI demands a broader, more integrated capability set.
The Agentic AI Partner Readiness Checklist
Before trusting a partner to take agentic AI into production, leaders should ask a simpler, more direct question:
Can this partner scale autonomy responsibly inside my enterprise?
Here is a practical checklist to help answer that question.
1. Enterprise & GCC Readiness
Has this partner run large-scale, production systems and not just pilots?
Do they understand GCC operating models, governance structures, and decision rights?
Can they embed AI ownership into teams, not just deliver projects?
2. Agentic AI Depth
Do they go beyond chatbots and copilots?
Have they designed and deployed multi-agent systems in real environments?
Do they build in human-in-the-loop controls by default?
3. Scalability & Reusability
Do they think in platforms, not one-off agents?
Can their solutions be reused across functions and workflows?
Is observability and lifecycle management part of the design and not just an afterthought?
4. Data & Integration Maturity
Can they work with messy, legacy, enterprise data?
Do they integrate cleanly with core business systems?
Is data governance built into the solution from day one?
5. Security, Risk & Governance
Are guardrails designed in, not bolted on?
Can decisions be explained, audited, and governed?
Are solutions built for regulated, compliance-heavy environments?
6. Outcome Ownership
Are success metrics tied to business outcomes not activity?
Will the partner co-own KPIs, risk, and accountability?
Do they stay invested beyond go-live?
This checklist shifts the conversation from capabilities to credibility.
Why This Checklist Changes the Conversation
Used well, this framework changes how enterprises approach agentic AI adoption.
It shifts the focus from vendors to partners, from pilots to platforms, and from experiments to operating models.
It also makes one thing clear: scaling agentic AI is not a one-time implementation. It is a capability that must be built, governed, and evolved over time.
Organizations that succeed tend to work with partners who understand enterprise realities, operate comfortably inside GCC environments, and engineer autonomy with accountability at the core.
That is where agentic AI becomes sustainable.
The Partner as a Force Multiplier
Agentic AI is not a shortcut. It is a long-term capability play.
The right partner accelerates scale, reduces risk, and protects ROI by ensuring that autonomy is introduced not with disruption but with discipline.
The wrong partner adds complexity, creates fragility, and leaves enterprises managing outcomes they never fully owned.
As leaders move from pilots to production, the question is no longer whether agentic AI can deliver value.
It is whether you have the right partner to deliver it at scale, in the real world, and over time.
Why Domain & Functional Context Make or Break Agentic AI
Agentic AI systems do not simply automate tasks, they make decisions inside business workflows. That makes domain and functional context non-negotiable.
An agent operating in finance, supply chain, customer service, or engineering must understand far more than APIs and prompts. It must respect process boundaries, exception handling, regulatory constraints, and the implicit rules humans apply every day.
Partners without functional or industry depth often build agents that technically work but fail operationally, producing decisions that are correct in isolation yet wrong in context.
The most effective partners combine agentic AI engineering with deep functional understanding, enabling agents to operate with judgment, not just intelligence.
Every few months, someone in the telecom space claims that the self-healing network is just around the corner. This has been happening for years. Yet, many regional operators are still handling incidents manually, with their engineers triaging alarms and switching between legacy dashboards and SNMP traps.
And the problem isn’t that operators lack ambition, or the drive for change – it’s that they don’t trust automation enough. That’s because they’ve learned, often the hard way, that even the smallest glitch can take a stable network down in seconds. This brings us to the real barrier to AI adoption in network operations, not technology, but trust. And honestly, that’s a rational response.
AI’s first job is to earn engineers’ trust, not to replace them
Most automation stories start from an ideal scenario: clean data, cloud-native infrastructure, and teams fluent in DevOps and data science. However, that’s not the reality for most Tier-2 operators. These are lean teams running multi-vendor environments, juggling with limited budgets and decades-old systems.
After over 20 years in telecom, at R Systems we’ve worked with operators who’ve run anomaly detection pilots that technically worked but stayed in read-only mode for months because no one in the Network Operations Center (NOC) trusted the system enough to act on its recommendations. That’s rather a failure of design philosophy, than AI. The automation model might be perfect, but if the trust is low, it won’t go live.
That’s why your first automation should first build trust and then trigger growth and digital transformation. It doesn’t need to be “zero-touch” solution. It needs to be safe andreversible, because engineers trust what they can override.
Start where failure costs are low and wins are visible
From what I’ve seen in most Tier-2 operators, about half the workload of their NOC comes from low-impact, repetitive incidents, like interface flaps, link degradations, or simple routing resets.
These are the perfect starting points for AI. They happen often enough for models to learn quickly, and even if something goes wrong, the impact is minimal. Automating such tasks can cut alert fatigue dramatically, without touching high-risk infrastructure. The goal isn’t to replace engineer teams, but to help them focus on innovation and growth, while allowing AI to handle high-frequency, low-risk tasks.
Reversible automation builds confidence, one task at a time
Every successful small automation builds political capital for bigger steps. Operators gain confidence when they see an AI system take on simple, reversible tasks and get them right.
Features like explain-why outputs, detailed logs, and one-click rollbacks allow engineers to stay in control. This “supervised automation” mindset is how AI earns its place in runbooks and not the other way around. Because when the NOC team feels that AI is a partner, not a blocker, adoption accelerates naturally.
AI in the NOC: how your first 90 days will look like
If you’re wondering where to start, here’s what’s worked in practice:
Step 1: Identify your top 10 high-frequency, low-risk runbooks.
Work with your NOC managers and subject matter experts to pinpoint repetitive incident types that drain the most time.
Step 2: Roll out AI in read-only mode.
Have the Ops / DevOps teams use it for auto-diagnosis and ticket enrichment. This builds trust with zero risk.
Step 3: Move to supervised automation with rollback options.
Let the AI recommend and occasionally execute known-safe actions, with human oversight, to reduce MTTS and false-positive rates.
If you follow this sequence, you can realistically target a 20–30% reduction in incident triage time within 12 weeks, without ever touching core routing policies.
What success looks like
A regional fiber ISP ran a small pilot with AI-based anomaly detection on its edge routers. Before the pilot, the six-person NOC was logging 15+ manual tickets every night.
After the AI grouped and labeled similar alarms automatically, that number dropped to just four incidents requiring human confirmation. The mean time to resolution (MTTR) went down by 28%.
That’s not science fiction, it’s what happens when trust comes before automation.
“Start Small” isn’t playing small
Some leaders worry that starting with small, reversible AI automations means they’ll fall behind the big players. Actually, it’s the other way around. Tier-1s often spend years (and millions) chasing “autonomous” dreams, but you can deliver measurable value in 90 days with a laptop, good logs, and the right mindset.
The key is to think of AI not as a leap of faith, but as a series of safe, reversible steps that gradually earn your confidence and your engineers’.
Because the truth is, AI doesn’t need to replace the human operator to transform the NOC. It just needs to make their 2 a.m. shift a little quieter, a little smarter, and a lot more human.
We have seen the same pattern repeat across insurance clients more times than we can count: a significant investment in a “strategic” BI platform, followed by growing frustration just a few years later. The dashboards still run, but the platform starts to feel heavy. Costs increase. New data sources take longer to onboard. Regulatory requirements evolve faster than the analytics stack can adapt.
For data and BI leaders in insurance, this is not a hypothetical scenario — it’s a familiar one.
The reality is simple: BI tools age faster than most organizations anticipate. Data volumes grow exponentially, operating models change, and regulatory goalposts continue to shift. In our experience at R Systems, the challenge is rarely the BI tool itself; it’s how tightly business logic, governance, and skills are coupled to that tool.
The Reality of Today’s Insurance BI Landscape
There is no such thing as a perfect BI tool — only the right tool for a given context. And in insurance, that context is constantly evolving.
Over the last decade, our teams have worked across a wide spectrum of analytics environments, from mainframe-driven reporting to cloud-native, AI-enabled platforms. Insurance organizations bring unique complexity to this journey: legacy core systems, fragmented actuarial and claims data, strict compliance requirements, and constant pressure to deliver more insight with fewer resources.
Most insurers still rely on a familiar set of BI platforms:
MicroStrategy
Tableau
Qlik
Oracle BI
And increasingly, Power BI
What we see most often is not a clean replacement of one tool with another, but a multi-tool landscape where new platforms are introduced alongside existing ones. This coexistence phase is where long-term success — or failure — is determined.
The biggest mistake organizations make is assuming that today’s “strategic BI choice” will remain optimal as business priorities, data platforms, and regulatory expectations evolve.
A Candid View of the Major BI Platforms in Insurance
MicroStrategy We’ve seen MicroStrategy perform extremely well in large insurance environments that demand strong governance, complex security models, and predictable enterprise reporting. It scales reliably and meets regulatory expectations. At the same time, it can feel restrictive for agile analytics or rapid experimentation, especially when business users seek faster self-service capabilities.
Tableau Tableau consistently drives high adoption due to its intuitive visual experience. Actuaries, underwriters, and analysts value the ability to explore data quickly and independently. Where insurers often struggle is governance at scale — particularly as data sources proliferate and business logic fragments across workbooks. Without strong discipline, performance and lineage challenges emerge.
Qlik Qlik is often underestimated in insurance contexts. Its associative model excels in ad hoc exploration, especially for claims analysis, fraud detection, and investigative use cases. Challenges tend to arise in deeply governed enterprise scenarios or where long-term extensibility and integration with modern data platforms are priorities.
Oracle BI Oracle BI remains a common choice for insurers heavily invested in Oracle ecosystems. It offers robust security and strong integration. However, innovation cycles can be slower, and business-user agility is often limited. Many teams rely on it out of necessity rather than preference.
Power BI and Its Growing Role Power BI has become a significant part of the insurance analytics conversation. Its integration with modern data platforms such as Databricks and Snowflake, improving enterprise governance, and rapidly evolving AI capabilities have made it a strategic option for many insurers.
In practice, we frequently see Power BI introduced alongside existing BI platforms — supporting executive reporting, self-service analytics, embedded use cases, or AI-driven insights — rather than as an immediate replacement. This coexistence reinforces the need for a flexible, decoupled architecture.
The Hidden Risk: Where Business Logic Lives
Across migrations and modernization programs, one risk appears repeatedly: deeply embedded business logic inside BI semantic layers.
When regulatory calculations, actuarial formulas, and financial metrics are hard-coded into a specific BI tool:
Migrations become slow and expensive
Parallel runs are difficult to validate
Flexibility disappears during mergers, acquisitions, or platform shifts
At that point, the BI tool stops being a presentation layer and becomes a structural constraint.
Five Questions We Use to Future-Proof Insurance BI Decisions
Based on our delivery experience, we encourage insurance BI leaders to ask five critical questions before making — or renewing — a BI investment:
How easily can BI tools be swapped or augmented as strategies and vendors change? Rigid architectures increase risk during integrations and modernization efforts.
Can governance models evolve with regulatory and data privacy demands? Many BI failures stem from brittle access controls and manual processes.
How well does the BI layer integrate with modern data platforms and AI services? Cloud-native and AI-enabled analytics are no longer optional.
How is the balance managed between self-service and enterprise control? Too much freedom leads to chaos; too much control drives shadow IT.
Are investments being made in skills and architecture, not just licenses? Tools change, but strong teams and sound design principles endure.
Lessons Learned From Real Programs
In one engagement, we supported an insurer migrating from Oracle BI to Jasper to improve operations. While the target state made sense, a significant amount of critical logic was embedded in Oracle’s semantic layer. Rebuilding these calculations extended the program timeline by nearly 40%.
In contrast, we’ve worked with insurers who deliberately decoupled their transformation and metric layers from the BI tool. When licensing or strategic priorities shifted, they were able to introduce Power BI with minimal disruption. That architectural choice saved months of effort and reduced long-term risk.
Trends Insurance BI Teams Can No Longer Ignore
Across recent insurance RFPs and transformation programs, several patterns are now consistent:
Cloud-native data platforms (Databricks, Snowflake, BigQuery)
Power BI and embedded analytics for agents, partners, and customers
AI-driven insights and natural language querying
Data mesh and data fabric operating models
These are no longer emerging trends — they are current expectations.
This is part 1 of 2 part series on A Practical Guide to HashiCorp Consul. This part is primarily focused on understanding the problems that Consul solves and how it solves them. The second part is more focused on a practical application of Consul in a real-life example. Let’s get started.
How about setting up discoverable, configurable, and secure service mesh using a single tool?
What if we tell you this tool is platform-agnostic and cloud-ready ?
And comes as a single binary download.
All this is true. The tool we are talking about is HashiCorp Consul.
Let’s learn about Consul in details below and see how it solves these complex challenges and makes the life of a distributed system operator easy.
Introduction
Microservices and other distributed systems can enable faster, simpler software development. But there’s a trade-off resulting in greater operational complexity around inter-service communication, configuration management, and network segmentation.
Monolithic Application (representational) – with different subsystems A, B, C and DDistributed Application (representational) – with different services A, B, C and D
HashiCorp Consul is an open source tool that solves these new complexities by providing service discovery, health checks, load balancing, a service graph, mutual TLS identity enforcement, and a configuration key-value store. These Consul features make it an ideal control plane for a service mesh.
HashiCorp Consul supports Service Discovery, Service Configuration, and Service Segmentation
This guide is aimed at discussing some of these crucial problems and exploring the various solutions provided by HashiCorp Consul to tackle these problems.
Let’s rundown through the topics that we are going to cover in this guide. The topics are written to be self-content. You can jump directly to a specific topic if you want to.
Brief Background on Monolithic vs. Service-oriented Architectures (SOA)
Even if it is a single application, typically it has multiple different sub-components.
One of the examples that HashiCorp’s CTO Armon Dadgar gave during his introductory video for Consul was about – delivering desktop banking application. It has a discrete set of sub-components – for example, authentication (say subsystem A), account management (subsystem B), fund transfer (subsystem C), and foreign exchange (subsystem D).
Now, although these are independent functions – system A authentication vs system C fund transfer – we deploy it as a single, monolith app.
Challenge with monolith is: Suppose there is a bug in one of the subsystems, system A, related to authentication.
Representational bug in Subsystem A in our monolithic application
We can’t just fix it in system A and update it in production.
Representational bug fix in Subsystem A in our monolithic application
We have to update system A and do a redeploy of the whole application, which we need deployment of subsystems B, C, and D as well.
Bug fix in one subsystem results in redeployment of the whole monolithic application
This whole redeployment is not ideal. Instead, we would like to do a deployment of individual services.
The same monolithic app delivered as a set of individual, discrete services.
Dividing monolithic application into individual services
So, if there is a bug fix in one of our services:
Representational bug in one of service, in this case Service A of our SOA application
and we fix that bug:
Representational bug fix in Service A of our SOA application
We can do the redeployment of that service without coordinating the deployment with other services. What we are essentially talking about is one form of microservices.
Bug fix will result into redeployment of only Service A within our whole application
This gives a big boost to our development agility. We don’t need to coordinate our development efforts across different development teams or even systems. We will have freedom of developing and deploying independently. One service on a weekly basis and other on quarterly. This is going to be a big advantage to the development teams.
But, as you know, there is ever such a thing as a “free” lunch.
The development efficiency we have gained introduces its own set of operational challenges. Let’s look at some of those.
Service discovery in a monolith, its challenges in a distributed system, and Consul’s solution
Monolithic applications
Assuming two services in a single application want to talk to one another. One way is to expose a method, make it public and allow other services to call it. In a monolithic application, it is a single app, and the services would expose public functions and it would simply mean function calls across services.
Subsystems talk to each other via function call within our monolithic application
As this is a function call within a process, it has happened in-memory. Thus, it’s fast, and we need not worry about how our data was moved and if it was secure or not.
Distributed Systems
In the distributed world, service A is no longer delivered as the same application as service B. So, how does service A finds service B if it wants to talk to B?
Service A tries to find Service B to establish communication
Service A might not even be on the same machine as service B. So, there is a network in play. And it is not as fast and there is a latency that we can measure on the lines of milliseconds, as compared to nanoseconds of a simple function call.
Challenges
As we already know by now, two services on a distributed system have to discover one-another to interact. One of the traditional ways of solving this is by using load balancers.
A load balancer sits between services to allow them to talk to each other
Load balancers would sit in front of each service with a static IP known to all other services.
A load balancer between two services allows two way traffic
This gives an ability to add multiple instances of the same service behind the load balancer and it would direct the traffic accordingly. But this load balancer IP is static and hard-coded within all other services, so services can skip discovery.
Load balancers allow communication between multiple instances of same service
The challenge is now to maintain a set of load balancers for each individual services. And we can safely assume, there was originally a load balancer for the whole application as well. The cost and effort for maintaining these load balancers have increased.
With load balancers in front of the services, they are a single point of failures. Even when we have multiple instances of service behind the load balancer if it is down our service is down. No matter how many instances of that service are running.
Load balancers also increase the latency of inter-service communication. If service A wish to talk to service B, request from A will have to first talk to the load balancer of service B and then reach B. The response from B will also have to go through the same drill.
Maintaining an entry of a service instances on an application-wide load balancer
And by nature, load balancers are manually managed in most cases. If we add another instance of service, it will not be readily available. We will need to register that service into the load balancer to make it accessible to the world. This would mean manual effort and time.
Consul maintains a central registry which contains the entry for all the upstream services. When a service instance starts, it is registered on the central registry. The registry is populated with all the upstream instances of the service.
Consul’s Service Registry helps Service A find Service B and establish communication
When a service A wants to talk to service B, it will discover and communicate with B by querying the registry about the upstream service instances of B. So, instead of talking to a load balancer, the service can directly talk to the desired destination service instance.
Consul also provides health-checks on these service instances. If one of the service instances or service itself is unhealthy or fails its health-check, the registry would then know about this scenario and would avoid returning the service’s address. The work that load-balancer would do is handled by the registry in this case.
Also, if there are multiple instances of the same service, Consul would send the traffic randomly to different instances. Thus, leveling the load among different instances.
Consul has handled our challenges of failure detection and load distribution across multiple instances of services without a necessity of deploying a centralized load balancer.
Traditional problem of slow and manually managed load balancers is taken care of here. Consul programmatically manages registry, which gets updated when any new service registers itself and becomes available for receiving traffic.
This helps with scaling the services with ease.
Configuration Management in a monolith, its challenges in a distributed environment, and Consul’s solution
Monolithic Applications
When we look at the configuration for a monolithic application, they tend to be somewhere along the lines of giant YAML, XML or JSON files. That configuration is supposed to configure the entire application.
Single configuration file shared across different parts of our monolithic application
Given a single file, all of our subsystems in our monolithic application would now consume the configuration from the same file. Thus creating a consistent view of all our subsystems or services.
If we wish to change the state of the application using configuration update, it would be easily available to all the subsystems. The new configuration is simultaneously consumed by all the components of our application.
Distributed Systems
Unlike monolith, distributed services would not have a common view on configuration. The configuration is now distributed and there every individual service would need to be configured separately.
A copy of application configuration is distributed across different services
Challenges in Distributed Systems
Configuration is to be spread across different services. Maintaining consistency between the configuration on different services after each update is a challenge.
Moreover, the challenge grows when we expect the configuration to be updated dynamically.
Consul’s Solutions
Consul’s solution for configuration management in distributed environment is the central Key-Value store.
Consul’s KV store allows seamless configuration mapping on each service
Consul solves this challenge in a unique way. Instead of spreading the configuration across different distributed service as configuration pieces, it pushes the whole configuration to all the services and configures them dynamically on the distributed system.
Let’s take an example of state change in configuration. The changed state is pushed across all the services in real-time. The configuration is consistently present with all the services.
Network segmentation in a monolith, its challenges in distributed systems, and Consul’s solutions
Monolithic Applications
When we look at our classic monolithic architecture, the network is typically divided in three different zones.
The first zone in our network is publicly accessible. The traffic coming to our application via the internet and reaching our load balancers.
The second zone is the traffic from our load balancers to our application. Mostly an internal network zone without direct public access.
The third zone is the closed network zone, primarily designated for data. This is considered to be an isolated zone.
Different network zones in typical application
Only the load balancers zone can reach into the application zone and only the application zone can reach into the data zone. It is a straightforward zoning system, simple to implement and manage.
Distributed Systems
The pattern changes drastically for distributed services.
Complex pattern of network traffic and routes across different services
Within our application network zone, there are multiple services which talk to each other within this network, making it a complicated traffic pattern.
Challenges
The primary challenge is that the traffic is not in any sequential flow. Unlike monolithic architecture, where the flow was defined from load balancers to the application and application to data.
Depending on the access pattern we want to support, the traffic might come from different endpoints and reaching different services.
Client essentially talks to each service within the application directly or indirectly
Given multiple services and the ability to support multiple endpoints allows us to deploy multiple service consumers and providers.
SOA demands control over trusted and untrusted sources of traffic
Controlling the flow of traffic and segmenting the network into groups or chunks will become a bigger issue. Also, making sure we have strict rules that guide us with partitioning the network based on who should be allowed to talk to whom and vice versa is also vital.
Consul’s Solutions
Consul’s solution to overall network segmentation challenge in distributed systems is by implementing service graph and mutual TLS.
Service-level policy enforcement to define traffic pattern and segmentation using Consul
Consul Connect enrolls these policies of inter-service communication that we desire and implements it as part of the service graph. So, a policy might say service A can talk to service B, but B cannot talk to C, for example.
The higher benefit of this is, it is not IP restricted. Rather it’s service level. This makes it scalable. The policy will be enforced on all instances of service and there will be no hard bound firewall rule specific to a service’s IP. Making us independent of the scale of our distributed network.
Consul Connect also handles service identity using popular TLS protocol. It distributes the TLS certificate associated with a service.
These certificates help other services securely identify each other. TLS is also helpful in making the communication between the services secure. This makes for trusted network implementation.
Consul enforces TLS using an agent-based proxy attached to each service instance. This proxy acts as a sidecar. Use of proxy, in this case, prevents us from making any change into the code of original service.
This allows for the higher level benefit of enforcing encryptions on data at rest and data in transit. Moreover, it will assist with fulfilling compliances required by laws around privacy and user identity.
Consul is shipped as a single binary download for all popular platforms. The executable can run as a client as well as server.
Each node that provides services to Consul runs a Consul agent. Each of these agents talk to one or more Consul servers.
Basic Architecture of Consul
Consul agent is responsible for health-checking the services on the node as the health-check of the node itself. It is not responsible for service discovery or maintaining key/value data.
Consul data and config storage and replication happens via Consul Servers.
Consul can run with single server, but it is recommended by HashiCorp to run a set of 3 to 5 servers to avoid failures. As all the data is stored at Consul server side, with a single server, the failure could cause a data loss.
With multi-servers cluster, they elect a leader among themselves. It is also recommended by HashiCorp to have cluster of servers per datacenter.
During the discovery process, any service in search for other service can query the Consul servers or even Consul agents. The Consul agents forward the queries to Consul servers automatically.
Consul Agent sits on a node and talks to other agents on the network synchronizing all service-level information
If the query is cross-datacenter, the queries are forwarded by the Consul server to the remote Consul servers. The results from remote Consul servers are returned to the original Consul server.
Getting Started with Consul
This section is dedicated to closely looking at Consul as a tool, with some hands-on experience.
Download and Install
As discussed above, Consul ships as a single binary downloaded from HashiCorps website or from Consul’s GitHub repo releases section.
This binary exhibits the role of both the Consul Server and Consul Client Agent. One can run them in either configuration.
$ unzip consul.zipArchive: consul.zipinflating: consul
Add it to PATH.
$ exportPATH="$PATH:/path/to/consul"
Use Consul
Once you unzip the compressed file and put the binary under your PATH, you can run it like this.
$ consul agent -dev==> Starting Consul agent...==> Consul agent running!Version: 'v1.4.2' Node ID: 'ef46ebb7-3496-346f-f67a-30117cfec0ad' Node name: 'devcube'Datacenter: 'dc1' (Segment:'<all>') Server: true (Bootstrap:false) Client Addr: [127.0.0.1] (HTTP:8500, HTTPS: -1, gRPC:8502, DNS:8600) Cluster Addr: 127.0.0.1 (LAN:8301, WAN:8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false==> Log data will now stream inasitoccurs:2019/03/0400:38:01 [DEBUG] agent: Using random ID"ef46ebb7-3496-346f-f67a-30117cfec0ad"asnodeID2019/03/0400:38:01 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:ef46ebb7-3496-346f-f67a-30117cfec0ad Address:127.0.0.1:8300}]2019/03/0400:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")2019/03/0400:38:01 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.12019/03/0400:38:01 [INFO] serf: EventMemberJoin: devcube 127.0.0.12019/03/0400:38:01 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)2019/03/0400:38:01 [INFO] consul: Handled member-join event for server "devcube.dc1"in area "wan"2019/03/0400:38:01 [DEBUG] agent/proxy: managed Connect proxy manager started2019/03/0400:38:01 [WARN] raft: Heartbeat timeout from "" reached, starting election2019/03/0400:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 22019/03/0400:38:01 [DEBUG] raft: Votes needed: 12019/03/0400:38:01 [DEBUG] raft: Vote granted from ef46ebb7-3496-346f-f67a-30117cfec0ad in term 2.Tally: 12019/03/0400:38:01 [INFO] raft: Election won. Tally: 12019/03/0400:38:01 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state2019/03/0400:38:01 [INFO] consul: cluster leadership acquired2019/03/0400:38:01 [INFO] consul: New leader elected: devcube2019/03/0400:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)2019/03/0400:38:01 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)2019/03/0400:38:01 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)2019/03/0400:38:01 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)2019/03/0400:38:01 [INFO] agent: started state syncer2019/03/0400:38:01 [INFO] connect: initialized primary datacenter CAwith provider "consul"2019/03/0400:38:01 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small2019/03/0400:38:01 [INFO] consul: member 'devcube' joined, marking health alive2019/03/0400:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically2019/03/0400:38:01 [INFO] agent: Synced node info2019/03/0400:38:01 [DEBUG] agent: Node info in sync2019/03/0400:38:01 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically2019/03/0400:38:01 [DEBUG] agent: Node info in sync
This will start the agent in development mode.
Consul Members
While the above command is running, you can check for all the members in Consul’s network.
$ consul membersNode Address Status Type Build Protocol DC Segmentdevcube 127.0.0.1:8301 alive server 1.4.02 dc1 <all>
Given we only have one node running, it is treated as server by default. You can designate an agent as a server by supplying server as command line parameter or server as configuration parameter to Consul’s config.
The output of the above command is based on the gossip protocol and is eventually consistent.
Consul HTTP API
For strongly consistent view of the Consul’s agent network, we can use HTTP API provided out of the box by Consul.
Consul also provides a DNS interface to query nodes. It serves DNS on 8600 port by default. That port is configurable.
$ dig @127.0.0.1-p 8600 devcube.node.consul; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1-p 8600 devcube.node.consul; (1 server found);; global options: +cmd;; Got answer:;; ->>HEADER<<-opcode: QUERY, status: NOERROR, id: 42215;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2;; WARNING: recursion requested but not available;; OPTPSEUDOSECTION:; EDNS: version: 0, flags:; udp: 4096;; QUESTIONSECTION:;devcube.node.consul. INA;; ANSWERSECTION:devcube.node.consul. 0INA127.0.0.1;; ADDITIONALSECTION:devcube.node.consul. 0INTXT"consul-network-segment=";; Query time: 19 msec;; SERVER: 127.0.0.1#8600(127.0.0.1);; WHEN: Mon Mar 0400:45:44IST2019;; MSGSIZErcvd: 100
Registering a service on Consul can be achieved either by writing a service definition or by sending a request over an appropriate HTTP API.
Consul Service Definition
Service definition is one of the popular ways of registering a service. Let’s take a look at one of such service definition examples.
To host our service definitions we will add a configuration directory, conventionally names as consul.d – ‘.d’ represents that there are set of configuration files under this directory, instead of single config under name consul.
$ mkdir ./consul.d
Write the service definition for a fictitious Django web application running on port 80 on localhost.
To make our consul agent aware of this service definition, we can supply the configuration directory to it.
$ consul agent -dev -config-dir=./consul.d==> Starting Consul agent...==> Consul agent running!Version: 'v1.4.2' Node ID: '810f4804-dbce-03b1-056a-a81269ca90c1' Node name: 'devcube'Datacenter: 'dc1' (Segment:'<all>') Server: true (Bootstrap:false) Client Addr: [127.0.0.1] (HTTP:8500, HTTPS: -1, gRPC:8502, DNS:8600) Cluster Addr: 127.0.0.1 (LAN:8301, WAN:8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false==> Log data will now stream inasitoccurs:2019/03/0400:55:28 [DEBUG] agent: Using random ID"810f4804-dbce-03b1-056a-a81269ca90c1"asnodeID2019/03/0400:55:28 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:810f4804-dbce-03b1-056a-a81269ca90c1 Address:127.0.0.1:8300}]2019/03/0400:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")2019/03/0400:55:28 [INFO] serf: EventMemberJoin: devcube.dc1 127.0.0.12019/03/0400:55:28 [INFO] serf: EventMemberJoin: devcube 127.0.0.12019/03/0400:55:28 [INFO] consul: Adding LAN server devcube (Addr: tcp/127.0.0.1:8300) (DC: dc1)2019/03/0400:55:28 [DEBUG] agent/proxy: managed Connect proxy manager started2019/03/0400:55:28 [INFO] consul: Handled member-join event for server "devcube.dc1"in area "wan"2019/03/0400:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)2019/03/0400:55:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)2019/03/0400:55:28 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)2019/03/0400:55:28 [INFO] agent: started state syncer2019/03/0400:55:28 [INFO] agent: Started gRPC server on 127.0.0.1:8502 (tcp)2019/03/0400:55:28 [WARN] raft: Heartbeat timeout from "" reached, starting election2019/03/0400:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 22019/03/0400:55:28 [DEBUG] raft: Votes needed: 12019/03/0400:55:28 [DEBUG] raft: Vote granted from 810f4804-dbce-03b1-056a-a81269ca90c1 in term 2.Tally: 12019/03/0400:55:28 [INFO] raft: Election won. Tally: 12019/03/0400:55:28 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state2019/03/0400:55:28 [INFO] consul: cluster leadership acquired2019/03/0400:55:28 [INFO] consul: New leader elected: devcube2019/03/0400:55:28 [INFO] connect: initialized primary datacenter CAwith provider "consul"2019/03/0400:55:28 [DEBUG] consul: Skipping self join check for "devcube" since the cluster is too small2019/03/0400:55:28 [INFO] consul: member 'devcube' joined, marking health alive2019/03/0400:55:28 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically2019/03/0400:55:28 [INFO] agent: Synced service "web"2019/03/0400:55:28 [DEBUG] agent: Node info in sync2019/03/0400:55:29 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically2019/03/0400:55:29 [DEBUG] agent: Service "web"in sync2019/03/0400:55:29 [DEBUG] agent: Node info in sync
The relevant information in the log here are the sync statements related to the “web” service. Consul agent as accepted our config and synced it across all nodes. In this case one node.
Consul DNS Service Query
We can query the service with DNS, as we did with node. Like so:
$ dig @127.0.0.1-p 8600 web.service.consul; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @127.0.0.1-p 8600 web.service.consul; (1 server found);; global options: +cmd;; Got answer:;; ->>HEADER<<-opcode: QUERY, status: NOERROR, id: 51488;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2;; WARNING: recursion requested but not available;; OPTPSEUDOSECTION:; EDNS: version: 0, flags:; udp: 4096;; QUESTIONSECTION:;web.service.consul. INA;; ANSWERSECTION:web.service.consul. 0INA127.0.0.1;; ADDITIONALSECTION:web.service.consul. 0INTXT"consul-network-segment=";; Query time: 0 msec;; SERVER: 127.0.0.1#8600(127.0.0.1);; WHEN: Mon Mar 0400:59:32IST2019;; MSGSIZErcvd: 99
We can also query DNS for service records that give us more info into the service specifics like port and node.
If you wish to update the service definition on a running Consul agent, it is very simple.
There are three ways to achieve this. You can send a SIGHUP signal to the process, reload Consul which internally sends SIGHUP on the node or you can call HTTP API dedicated to service definition updates that will internally reload the agent configuration.
$ ps aux | grep [c]onsulpranav 212892.40.317701254924 pts/2 Sl+00:550:22 consul agent -dev -config-dir=./consul.d
Send SIGHUP to 21289
$ kill -SIGHUP21289
Or reload Consul
$ consul reload
Configuration reload triggered
You should see this in your Consul log.
...2019/03/0401:10:46 [INFO] agent: Caught signal: hangup2019/03/0401:10:46 [INFO] agent: Reloading configuration...2019/03/0401:10:46 [DEBUG] agent: removed service "web"2019/03/0401:10:46 [INFO] agent: Synced service "web"2019/03/0401:10:46 [DEBUG] agent: Node info in sync...
Consul Web UI
Consul provides a beautiful web user interface out-of-the-box. You can access it on port 8500.
In this case at http://localhost:8500. Let’s look at some of the screens.
The home page for the Consul UI is services with all the relevant information related to a Consul agent and web service check.
Exploring defined services on Consul Web UI
Going into further details on a given service, we get a service dashboard with all the nodes and their health for that service.
Exploring node-level information for each service on Consul Web U
On each individual node, we can look at the health-checks, services, and sessions.
Exploring node-specific health-check information, services information, and sessions information on Consul Web UI
Overall, Consul Web UI is really impressive and a great companion for the command line tools that Consul provides.
How is Consul Different From Zookeeper, doozerd, and etcd?
Consul has a first-class support for service discovery, health-check, key-value storage, multi data centers.
Zookeeper, doozerd, and etcd are primarily based on key-value store mechanism. To achieve something beyond such key-value, store needs additional tools, libraries, and custom development around them.
All these tools, including Consul, uses server nodes that require quorum of nodes to operate and are strongly consistent.
More or less, they all have similar semantics for key/value store management.
These semantics are attractive for building service discovery systems. Consul has out-of the box support for service discovery, which the other systems lack at.
A service discovery systems also requires a way to perform health-checks. As it is important to check for service’s health before allowing others to discover it. Some systems use heartbeats with periodic updates and TTL. The work for these health checks grows with scale and requires fixed infra. The failure detection window is as least as long as TTL.
Unlike Zookeeper, Consul has client agents sitting on each node in the cluster, talking to each other in gossip pool. This allows the clients to be thin, gives better health-checking ability, reduces client-side complexity, and solves debugging challenges.
Also, Consul provides native support for HTTP or DNS interfaces to perform system-wide, node-wide, or service-wide operations. Other systems need those being developed around the exposed primitives.
Consul’s website gives a good commentary on comparisons between Consul and other tools.
Open Source Tools Around HashiCorp Consul
HashiCorp and the community has built several tools around Consul
These tools are built and maintained by the HashiCorp developers:
Consul Template (3.3k stars) – Generic template rendering and notifications with Consul. Template rendering, notifier, and supervisor for @hashicorp Consul and Vault data. It provides a convenient way to populate values from Consul into the file system using the consul-template daemon.
Envconsul (1.2k stars) – Read and set environmental variables for processes from Consul. Envconsul provides a convenient way to launch a subprocess with environment variables populated from HashiCorp Consul and Vault.
Consul Replicate (360 stars) – Consul cross-DC KV replication daemon. This project provides a convenient way to replicate values from one Consul datacenter to another using the consul-replicate daemon.
Consul Migrate – Data migration tool to handle Consul upgrades to 0.5.1+.
The community around Consul has also built several tools to help with registering services and managing service configuration, I would like to mention some of the popular and well-maintained ones –
Confd (5.9k stars) – Manage local application configuration files using templates and data from etcd or consul.
Fabio (5.4k stars) – Fabio is a fast, modern, zero-conf load balancing HTTP(S) and TCP router for deploying applications managed by consul. Register your services in consul, provide a health check and fabio will start routing traffic to them. No configuration required.
Registrator (3.9k stars) – Service registry bridge for Docker with pluggable adapters. Registrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online.
Hashi-UI (871 stars) – A modern user interface for HashiCorp Consul & Nomad.
Git2consul (594 stars) – Mirrors the contents of a git repository into Consul KVs. git2consul takes one or many git repositories and mirrors them into Consul KVs. The goal is for organizations of any size to use git as the backing store, audit trail, and access control mechanism for configuration changes and Consul as the delivery mechanism.
Spring-cloud-consul (503 stars) – This project provides Consul integrations for Spring Boot apps through autoconfiguration and binding to the Spring Environment and other Spring programming model idioms. With a few simple annotations, you can quickly enable and configure the common patterns inside your application and build large distributed systems with Consul based components.
Crypt (453 stars) – Store and retrieve encrypted configs from etcd or consul.
Mesos-Consul (344 stars) – Mesos to Consul bridge for service discovery. Mesos-consul automatically registers/deregisters services run as Mesos tasks.
Consul-cli (228 stars) – Command line interface to Consul HTTP API.
Conclusion
Distributed systems are not easy to build and setup. Maintaining them and keeping them running is an altogether another piece of work. HashiCorp Consul makes the life of engineers facing such challenges easier.
As we went through different aspects of Consul, we learnt how straightforward it would become for us to develop and deploy application with distributed or microservices architecture.
Robust production ready code (most crucial), well written detailed documentation, user friendliness, and solid community, helps in adopting HashiCorp Consul and introducing it in our technology stack straight forward.
We hope it was an informative ride on the journey of Consul. Our journey has not yet ended, this was just the first half. We will meet you again with the second part of this article that walks us through practical example close to real-life applications. Find the Practical Guide To HashiCorp Consul – Part 2 here.
Let’s us know what you would like to hear from us more or if you have any questions around the topic, we will be more than happy to answer those.
This is part 2 of 2 part series on A Practical Guide to HashiCorp Consul. The previous partwas primarily focused on understanding the problems that Consul solves and how it solves them. This part is focused on a practical application of Consul in a real-life example. Let’s get started.
With most of the theory covered in the previous part, let’s move on to Consul’s practical example.
To show how our web app would scale in this context, we are going to run two instances of Django app. Also, to make this even more interesting, we will run MongoDB as a Replica Set with one primary node and two secondary nodes.
Given we have two instances of Django app, we will need a way to balance a load among those two instances, so we are going to use Fabio, a Consul aware load-balancer, to reach Django app instances.
This example will roughly help us simulate a real-world practical application.
Example Application nodes and services deployed on them
The complete source code for this application is open-sourced and is available on GitHub – pranavcode/consul-demo.
Note: The architecture we are discussing here is not specifically constraint with any of the technologies used to create app or data layers. This example could very well be built using a combination of Ruby on Rails and Postgres, or Node.js and MongoDB, or Laravel and MySQL.
How Does Consul Come into the Picture?
We are deploying both, the app and the data, layers with Docker containers. They are going to be built as services and will talk to each other over HTTP.
Thus, we will use Consul for Service Discovery. This will allow Django servers to find MongoDB Primary node. We are going to use Consul to resolve services via Consul’s DNS interface for this example.
Consul will also help us with the auto-configuration of Fabio as load-balancer to reach instances of our Django app.
We are also using the health-check feature of Consul to monitor the health of each of our instances in the whole infrastructure.
Consul provides a beautiful user interface, as part of its Web UI, out of the box to show all the services on a single dashboard. We will use it to see how our services are laid out.
Let’s begin.
Setup: MongoDB, Django, Consul, Fabio, and Dockerization
We will keep this as simple and minimal as possible to the extent it fulfills our need for a demonstration.
MongoDB
The MongoDB setup we are targeting is in the form of MongoDB Replica Set. One primary node and two secondary nodes.
The primary node will manage all the write operations and the oplog to maintain the sequence of writes, and replicate the data across secondaries. We are also configuring the secondaries for the read operations. You can learn more about MongoDB Replica Set on their official documentation.
We will call our replication set as ‘consuldemo’.
We will run MongoDB on a standard port 27017 and supply the name of the replica set on the command line using the parameter ‘–replSet’.
As you may read from the documentation MongoDB also allows configuring replica set name via configuration file with the parameter for replication as below:
replication:replSetName: "consuldemo"
In our case, the replication set configuration that we will apply on one of the MongoDB nodes, once all the nodes are up and running is as given below:
Note: We are not forcing the set creation with any pre-defined designations on who becomes primary and secondary to allow the dynamism in service discovery. Normally, the nodes would be defined for a specific role.
We are allowing slave reads and reads from the nearest node as a Read Preference.
We will start MongoDB on all nodes with the following command:
As we need our Django app to talk to MongoDB, we will use a MongoDB connector for Django ORM, Djongo. We will set up our Django settings to use Djongo and connect with our MongoDB. Djongo is pretty straightforward in configuration.
For a local MongoDB installation it would only take two lines of code:
ENGINE: The database connector to use for Django ORM.
NAME: Name of the database.
HOST: Host address that has MongoDB running on it.
PORT: Which port is your MongoDB listening for requests.
Djongo internally talks to PyMongo and uses MongoClient for executing queries on Mongo. We can also use other MongoDB connectors available for Django to achieve this, like, for instance, django-mongodb-engine or pymongo directly, based on our needs.
Note: We are currently reading and writing via Django to a single MongoDB host, the primary one, but we can configure Djongo to also talk to secondary hosts for read-only operations. That is not in the scope of our discussion. You can refer to Djongo’s official documentation to achieve exactly this.
Continuing our Django app building process, we need to define our models. As we are building a blog-like application, our models would look like this:
from djongo import modelsclass Blog(models.Model): name = models.CharField(max_length=100) tagline = models.TextField() class Meta: abstract = Trueclass Entry(models.Model): blog = models.EmbeddedModelField( model_container=Blog, ) headline = models.CharField(max_length=255)
We can run a local MongoDB instance and create migrations for these models. Also, register these models into our Django Admin, like so:
from django.contrib import adminfrom.models import Entryadmin.site.register(Entry)
We can play with the Entry model’s CRUD operations via Django Admin for this example.
Also, to realize the Django-MongoDB connectivity we will create a custom View and Template that displays information about MongoDB setup and currently connected MongoDB host.
We will discuss Consul setup in the context of MongoDB Replica Set first – as it solves an interesting problem. At any given point of time, one of the MongoDB instances can either be a Primary or a Secondary.
The Consul agent registering and monitoring our MongoDB instance within a Replica Set has a unique mechanism – dynamically registering and deregistering MongoDB service as a Primary instance or a Secondary instance based on what Replica Set has designated it.
We achieve this dynamism by writing and running a shell script after an interval that toggles the Consul service definition for MongoDB Primary and MongoDB Secondary on the instance node’s Consul Agent.
The service definitions for MongoDB services are stored as JSON files on the Consul’s config directory ‘/etc/config.d’.
If you look closely, the service definition allows us to get a DNS entry specific to MongoDB Primary, rather than a generic MongoDB instance. This allows us to send the database writes to a specific MongoDB instance. In the case of Replica Set, the writes are maintained by MongoDB Primary.
Thus, we are able to achieve both service discovery as well as health monitoring for Primary instance of MongoDB.
Similarly, with a slight change the service definition for MongoDB Secondary instance goes like this:
Here, ‘consul_server’ represents the Consul Server running host. Similarly, we can run such agents on each of the other MongoDB instance nodes.
Note: If we have multiple MongoDB instances running on the same host, the service definition would change to reflect the different ports used by each instance to uniquely identify, discover and monitor individual MongoDB instance.
Consul on nodes running Django App
For the Django application, Consul setup will be very simple. We only need to monitor Django app’s port on which Gunicorn is listening for requests.
The Consul service definition would look like this:
Once we have the Consul service definition for Django app in place, we can run the Consul agent sitting on the node Django app is running as a service. To run the Consul agent we would fire the following command:
We are running the Consul cluster with a dedicated Consul server node. The Consul server node can easily host, discover and monitor services running on it, exactly the same way as we did in the above sections for MongoDB and Django app.
To run Consul in server mode and allow agents to connect to it, we will fire the following command on the node that we want to run our Consul server:
There are no services on our Consul server node for now, so there are no service definitions associated with this Consul agent configuration.
Fabio
We are using the power of Fabio to be auto-configurable and being Consul-aware.
This makes our task of load-balancing the traffic to our Django app instances very easy.
To allow Fabio to auto-detect the services via Consul, one of the ways is to add a tag or update a tag in the service definition with a prefix and a service identifier `urlprefix-/<service>`. Our Consul’s service definition for Django app would now look like this:</service>
In our case, the Django app or service is the only service that will need load-balancing, thus this Consul service definition change completes the requirement on Fabio setup.
Dockerization
Our whole app is going to be deployed as a set of Docker containers. Let’s talk about how we are achieving it in the context of Consul.
Dockerizing MongoDB Replica Set along with Consul Agent
We need to run a Consul agent as described above alongside MongoDB on the same Docker container, so we will need to run a custom ENTRYPOINT on the container to allow running two processes.
Note: This can also be achieved using Docker container level checks in Consul. So, you will be free to run a Consul agent on the host and check across service running in Docker container. Which, will essentially exec into the container to monitor the service.
To achieve this we will use a tool similar to Foreman. It is a lifecycle management tool for physical and virtual servers – including provisioning, monitoring and configuring.
To be precise, we will use the Golang adoption of Foreman, Goreman. It takes the configuration in the form of Heroku’sProcfile to maintain which processes to be kept alive on the host.
In our case, the Procfile looks like this:
# Mongomongo: /opt/mongo.sh# Consul Client Agentconsul: /opt/consul.sh# Consul Client Health Checksconsul_check: whiletrue; do/opt/checks_toggler.sh && sleep 10; done
The `consul_check` at the end of the Profile maintains the dynamism between both Primary and Secondary MongoDB node checks, based on who is voted for which role within MongoDB Replica Set.
The shell scripts that are executed by the respective keys on the Procfile are as defined previously in this discussion.
Our Dockerfile, with some additional tools for debug and diagnostics, would look like:
Note: We have used bare Ubuntu 18.04 image here for our purposes, but you can use official MongoDB image and adapt it to run Consul alongside MongoDB or even do Consul checks on Docker container level as mentioned in the official documentation.
Dockerizing Django Web Application along with Consul Agent
We also need to run a Consul agent alongside our Django App on the same Docker container as we had with MongoDB container.
# Djangodjango: /web/tweeter.sh# Consul Client Agentconsul: /opt/consul.sh
Similarly, we will have the Dockerfile for Django Web Application as we had for our MongoDB containers.
We are maintaining the same flow with Consul server node to run it with custom ENTRYPOINT. It is not a requirement, but we are maintaining a consistent view of different Consul run files.
Also, we are using Ubuntu 18.04 image for the demonstration. You can very well use Consul’s official image for this, that accepts all the custom parameters as are mentioned here.
We are using Compose to run all our Docker containers in a desired, repeatable form.
Our Compose file is written to denote all the aspects that we mentioned above and utilize the power of Docker Compose tool to achieve those in a seamless fashion.
Docker Compose file would look like the one given below:
Django App can now connect MongoDB Primary instance and start writing data to it.
We can use Fabio load-balancer to connect to Django App instance by auto-discovering it via Consul registry using specialized service tags and render the page with all the database connection information we are talking about.
Our load-balancer is sitting at ‘33.10.0.100’ and ‘/web’ is configured to be routed to one of our Django application instances running behind the load-balancer.
Fabio auto-detecting the Django Web Application end-points
As you can see from the auto-detection and configuration of Fabio load-balancer from its UI above, it has weighted the Django Web Application end-points equally. This will help balance the request or traffic load on the Django application instances.
When we visit our Fabio URL ‘33.10.0.100:9999’ and use the source route as ‘/web’ we are routed to one of the Django instances. So, visiting ‘33.10.0.100:9999/web’ gives us following output.
Django Web Application renders the MongoDB connection status on the home page
We are able to restrict Fabio to only load-balance Django app instances by only adding required tags to Consul’s service definitions of Django app services.
This MongoDB Primary instance discovery helps Django app to do database migration and app deployment.
One can explore Consul Web UI to see all the instances of Django web application services.
Django Web Application services as seen on Consul’s Web UI
Similarly, see how MongoDB Replica Set instances are laid out.
MongoDB Replica Set Primary service as seen on Consul’s Web UI MongoDB Replica Set Secondary services as seen on Consul’s Web UI
Let’s see how Consul helps with health-checking services and discovering only the alive services.
We will stop the current MongoDB Replica Set Primary (‘mongo_2’) container, to see what happens.
MongoDB Primary service being swapped with one of the MongoDB Secondary instances MongoDB Secondary instance set is now left with only one service instance
Consul has started failing the health-check for previous MongoDB Primary service. MongoDB Replica Set has also detected that the node is down and the re-election of Primary node needs to be done. Thus, getting us a new MongoDB Primary (‘mongo_3’) automatically.
Our checks toggle has kicked-in and swapped the check on ‘mongo_3’ from MongoDB Secondary check to MongoDB Primary check.
When we take a look at the view from the Django app, we see it is now connected to a new MongoDB Primary service (‘mongo_3’).
Switching of the MongoDB Primary is also reflected in the Django Web Application
Let’s see how this plays out when we bring back the stopped MongoDB instance.
Failing MongoDB Primary service instance is now cleared out from service instances as it is now healthy MongoDB Secondary service instance Previously failed MongoDB Primary service instance is now re-adopted as MongoDB Secondary service instance as it has become healthy again
Similarly, if we stop the service instances of Django application, Fabio would now be able to detect only a healthy instance and would only route the traffic to that instance.
Fabio is able to auto-configure itself using Consul’s service registry and detecting alive service instances
This is how one can use Consul’s service discovery capability to discover, monitor and health-check services.
We can use Consul’s Key/Value store to share configuration across both the instances of Django app.
We can use Consul’s HTTP interface to store key/value pair and retrieve them within the app using the open-source Python client for Consul, called python-consul. You may also use any other Python library that can interact with Consul’s KV store if you want.
Let’s begin by looking at how we can set a key/value pair in Consul using its HTTP interface.
# Flag to run Django app in debug modecurl -XPUT-d 'True'consul_server:8500/v1/kv/web/debug# Dynamic entries into Django app configuration # to denote allowed set of hostscurl -XPUT-d 'localhost, 33.10.0.100'consul_server:8500/v1/kv/web/allowed_hosts# Dynamic entries into Django app configuration# to denote installed appscurl -XPUT-d 'tweetapp'consul_server:8500/v1/kv/web/installed_apps
Once we set the KV store we can consume it on Django app instances to configure it with these values.
We can capture and configure our Django app accordingly using the ‘python-consul’ library.
# Set DEBUG flag usingConsul KV storeindex, data = consul_client.kv.get('web/debug')DEBUG= data.get('Value', True)# Set ALLOWED_HOSTS dynamically usingConsul KV storeALLOWED_HOSTS= []index, hosts = consul_client.kv.get('web/allowed_hosts')ALLOWED_HOSTS.append(hosts.get('Value'))# Set INSTALLED_APPS dynamically usingConsul KV storeINSTALLED_APPS= ['django.contrib.admin','django.contrib.auth','django.contrib.contenttypes','django.contrib.sessions','django.contrib.messages','django.contrib.staticfiles',]index, apps = consul_client.kv.get('web/installed_apps')INSTALLED_APPS+= (bytes(apps.get('Value')).decode('utf-8'),)
These key/value pair from Consul’s KV store can also be viewed and updated from its Web UI.
Consul KV store as seen on Consul Web UI with Django app configuration parameters
The code used as part of this guide for Consul’s service configuration section is available on ‘service-configuration’ branch of pranavcode/consul-demo project.
That is how one can use Consul’s KV store and configure individual services in their architecture with ease.
Connect provides service-to-service connection authorization and encryption using mutual TLS.
To use Consul you need to enable it in the server configuration. Connect needs to be enabled across the Consul cluster for proper functioning of the cluster.
{"connect": {"enabled": true }}
In our context, we can define that the communication is to be TLS identified and secured we will define an upstream sidecar service with a proxy on Django app for its communication with MongoDB Primary instance.
Along with Connect configuration of sidecar proxy, we will also need to run the Connect proxy for Django app as well. This could be achieved by running the following command.
We can add Consul Connect Intentions to create a service graph across all the services and define traffic pattern. We can create intentions as shown below:
$ consul connect proxy -sidecar-for web
Intentions for service graph can also be managed from Consul Web UI.
Define access control for services via Connect and service connection restrictions
This defines the service connection restrictions to allow or deny them to talk via Connect.
We have also added ability on Consul agents to denote which datacenters they belong to and be accessible via one or more Consul servers in a given datacenter.
The code used as part of this guide for Consul’s service segmentation section is available on ‘service-segmentation’ branch of velotiotech/consul-demo project.
That is how one can use Consul’s service segmentation feature and configure service level connection access control.
Conclusion
Having an ability to seamlessly control the service mesh that Consul provides makes the life of an operator very easy. We hope you have learnt how Consul can be used for service discovery, configuration, and segmentation with its practical implementation.
As usual, we hope it was an informative ride on the journey of Consul. This was the final piece of this two part series. This part tries to cover most of the aspects of Consul architecture and how it fits into your current project. In case you miss the first part, find it here.
We will continue our endeavors with different technologies and get you the most valuable information that we possibly can in every interaction. Let’s us know what you would like to hear from us more or if you have any questions around the topic, we will be more than happy to answer those.
In today’s fast-evolving world, technology has redefined how we interact with our environments. From smart homes to automated workplaces, the demand for integrated systems that offer safety, convenience and enhanced user experience is higher than ever. As part of our ongoing mission to designs and build next-gen products, we recently took on a fascinating project: developing a mobile app (Android and iOS) for audio and video calls that works seamlessly with voice entry panels supporting VoIP (Voice over Internet Protocol).
The Client’s Vision: Safety, Accessibility and Innovation
The client approached us and requested an app that could serve as a comprehensive communication hub between residents, visitors, and building security systems. They wanted a mobile application that would:
Enable smooth audio and video communication between users and VoIP-enabled voice entry panels.
Improve building security by allowing residents to verify visitors before granting access.
Increase convenience by integrating this communication into a single, easy-to-use app.
Allow the mobile device to provide keyless entry.
Should be available for both iOS and Android devices.
In essence, the app would become the digital key to a smarter, safer living space.
Why VoIP? The Backbone of the Future
Voice over Internet Protocol (VoIP) has revolutionized modern communication by allowing voice and multimedia sessions over the internet. Its use in smart buildings, particularly in voice entry panels, provides several key benefits:
Real-time communication: VoIP ensures fast, reliable, real-time communication between users and security systems.
Cost-effectiveness: Unlike traditional landlines, VoIP uses existing internet infrastructure, lowering costs and providing more flexibility for building owners.
Scalability: VoIP systems can easily be scaled up to accommodate new users and features as a building’s needs evolve.
By leveraging VoIP, we were able to design an app that offers superior connectivity, reliability and security.
Key Features of Our Audio/Video Call App
Our team focused on creating a solution that would offer a seamless user experience while enhancing security and control over building access. Below are the key features of the app we developed:
Two-Way Audio/Video Communication
The app allows residents to receive calls from visitors at building entry panels, enabling real-time audio and video communication. This allows users to verify visitors’ identities visually and audibly, adding an extra layer of security.
Remote Door Access
After verifying a visitor, the user can grant or deny access to the building via the app. This feature is particularly useful for those who may not be home but still need to authorize entry, such as in the case of package deliveries or housekeepers.
VoIP-Based Calling
Since the app is built around VoIP technology, users benefit from high-quality calls that bypass traditional phone networks. The app integrates with voice entry panels that support VoIP, ensuring clear communication even over long distances.
BLE Functionality: Turning Your Mobile Device into a Key
One of this app’s most groundbreaking features is the integration of Bluetooth Low Energy (BLE) technology, which allows the mobile device to function as a digital key. BLE is known for its ability to facilitate short-range communication with minimal power consumption, making it an ideal solution for mobile access control.
Instead of using traditional keycards, fobs, or physical keys, employees and residents can use their smartphones to open gates and doors. BLE communicates with the gate system within proximity, enabling automatic or one-tap entry. This improves convenience and reduces the need for physical keys, which can be lost or stolen.
Cloud-Based Validation: Security in the Digital Age
To ensure that only authorized individuals can gain access, the app integrates with highly secure cloud servers for communication and validation. Every time a user attempts to open a gate or door using the app, the system verifies their credentials in real-time through cloud communication. This means that building managers or security teams can update or revoke access permissions instantly, without needing to reissue physical keys or cards.
Cloud-based validation offers multiple security layers, including encryption and authentication, ensuring that access control is both flexible and safe. It also allows for seamless monitoring and reporting, as every access attempt is logged in the system, providing valuable data for building management and security audits.
This feature also lets users manage communication logs, call history, and security footage from anywhere in the world. This feature is critical for building managers who want a real-time overview of all entry points.
Streamlined Access for Employees and Residents
This app is designed with both employees and residents in mind. In office settings, employees can easily move through secured areas, using the app to access parking garages, secure floors, and meeting rooms. For residents of gated communities or apartment buildings, the app provides a hassle-free solution to enter gates and shared spaces like gyms or pools, without needing to fumble with keys or keycards and verifying visitors’ identities visually and audibly.
Customizable Alerts and Notifications
Users can set up custom notifications for different types of events, such as when a delivery arrives or a family member enters the building. These alerts ensure that users are always aware of what’s happening around their living space.
Scalability
This app can be scaled for use in various environments, from small residential buildings to large corporate complexes.
Enhancing the Future of Smart Buildings
This app’s development is part of a broader trend towards creating intelligent living environments where technology and security work hand-in-hand. Smart buildings equipped with VoIP-enabled voice entry panels, paired with mobile applications like ours not only enhance both security and user experience but also offer a glimpse into the future-a future where buildings are not just physical spaces, but adaptive, interactive ecosystems designed around the needs of their occupants.
We’re proud to contribute to this vision of safer, smarter, and more livable buildings. This project is a step forward in reshaping how residents interact with their living spaces, making life more convenient while ensuring peace of mind.
The Road Ahead
As we continue to refine and improve our app, we are excited about the potential to incorporate even more innovative features, such as Early Media Access, AI-driven facial recognition, advanced analytics for building managers and further integrations with home automation systems like Emergency Lighting System, Apartment Intercom System, Fire Detection and Alarm System including Smoke detectors, heat detectors and carbon monoxide detectors.
Our commitment remains strong: to create digital solutions which empower our clients and make the world a better, safer place to live in.
Stay tuned for more updates as we continue to push the boundaries of what smart building technology can achieve!
Once upon a time, in the vast and ever-expanding world of data storage and processing, a new hero emerged. Its name? Delta Lake. This unsung champion was about to revolutionize the way organizations handled their data, and its journey was nothing short of remarkable.
The Need for a Data Savior
In this world, data was king, and it resided in various formats within the mystical realm of data lakes. Two popular formats, Parquet and Hive, had served their purposes well, but they harbored limitations that often left data warriors frustrated.
Enterprises faced a conundrum: they needed to make changes, updates, or even deletions to individual records within these data lakes. But it wasn’t as simple as it sounded. Modifying schemas was a perilous endeavor that could potentially disrupt the entire data kingdom.
Why? Because these traditional table formats lacked a vital attribute: ACID transactions. Without these safeguards, every change was a leap of faith.
The Rise of Delta Lake
Amidst this data turmoil, a new contender emerged: Delta Lake. It was more than just a format; it was a game-changer.
Delta Lake brought with it the power of ACID transactions. Every data operation within the kingdom was now imbued with atomicity, consistency, isolation, and durability. It was as if Delta Lake had handed data warriors an enchanted sword, making them invincible in the face of chaos.
But that was just the beginning of Delta Lake’s enchantment.
The Secrets of Delta Lake
Delta Lake was no ordinary table format; it was a storage layer that transcended the limits of imagination. It integrated seamlessly with Spark APIs, offering features that left data sorcerers in awe.
Time Travel: Delta Lake allowed users to peer into the past, accessing previous versions of data. The transaction log became a portal to different eras of data history.
Schema Evolution: It had the power to validate and evolve schemas as data changed. A shapeshifter of sorts, it embraced change effortlessly.
Change Data Feed: With this feature, it tracked data changes at the granular level. Data sorcerers could now decipher the intricate dance of inserts, updates, and deletions.
Data Skipping with Z-ordering: Delta Lake mastered the art of optimizing data retrieval. It skipped irrelevant files, ensuring that data requests were as swift as a summer breeze.
DML Operations: It wielded the power of SQL-like data manipulation language (DML) operations. Updates, deletes, and merges were but a wave of its hand.
Delta Lake’s Allies
Delta Lake didn’t stand alone; it forged alliances with various data processing tools and platforms. Apache Spark, Apache Flink, Presto, Trino, Hive, DBT, and many others joined its cause. They formed a coalition to champion the cause of efficient data processing.
In the vast landscape of data management, Delta Lake stands as a beacon of innovation, offering a plethora of features that elevate your data handling capabilities to new heights. In this exhilarating adventure, we’ll explore the key features of Delta Lake and how they triumph over the limitations of traditional file formats, all while embracing the ACID properties.
ACID Properties: A Solid Foundation
In the realm of data, ACID isn’t just a chemical term; it’s a set of properties that ensure the reliability and integrity of your data operations. Let’s break down how Delta Lake excels in this regard.
A for Atomicity: All or Nothing
Imagine a tightrope walker teetering in the middle of their performance—either they make it to the other side, or they don’t. Atomicity operates on the same principle: either all changes happen, or none at all. In the world of Spark, this principle often takes a tumble. When a write operation fails midway, the old data is removed, and the new data is lost in the abyss. Delta Lake, however, comes to the rescue. It creates a transaction log, recording all changes made along with their versions. In case of a failure, data loss is averted, and your system remains consistent.
C for Consistency: The Guardians of Validity
Consistency is the gatekeeper of data validity. It ensures that your data remains rock-solid and valid at all times. Spark sometimes falters here. Picture this: your Spark job fails, leaving your system with invalid data remnants. Consistency crumbles. Delta Lake, on the other hand, is your data’s staunch guardian. With its transaction log, it guarantees that even in the face of job failure, data integrity is preserved.
I for Isolation: Transactions in Solitude
Isolation is akin to individual bubbles, where multiple transactions occur in isolation, without interfering with one another. Spark might struggle with this concept. If two Spark jobs manipulate the same dataset concurrently, chaos can ensue. One job overwrites the dataset while the other is still using it—no isolation, no guarantees. Delta Lake, however, introduces order into the chaos. Through its versioning system and transaction log, it ensures that transactions proceed in isolation, mitigating conflicts and ensuring the data’s integrity.
D for Durability: Unyielding in the Face of Failure
Durability means that once changes are made, they are etched in stone, impervious to system failures. Spark’s Achilles’ heel lies in its vulnerability to data loss during job failures. Delta Lake, however, boasts a different tale. It secures your data with unwavering determination. Every change is logged, and even in the event of job failure, data remains intact—a testament to true durability.
Time Travel: Rewriting the Past
Now, let’s embark on a fascinating journey through time. Delta Lake introduces a feature that can only be described as “time travel.” With this feature, you can revisit previous versions of your data, just like rewinding a movie. All of this magical history is stored in the transaction log, encapsulated within the mystical “_delta_log” folder. When you write data to a Delta table, it’s not just the present that’s captured; the past versions are meticulously preserved, waiting for your beck and call.
In conclusion, Delta Lake emerges as the hero of the data world, rewriting the rules of traditional file formats and conquering the challenges of the ACID properties. With its robust transaction log, versioning system, and the ability to traverse time, Delta Lake opens up a new dimension in data management. So, if you’re on a quest for data reliability, integrity, and a touch of magic, Delta Lake is your trusted guide through this thrilling journey beyond convention.
More Features of Delta Lake Are:
UPSERT
Schema Evolution
Change Data Feed
Data Skipping with Z-ordering
DML Operations
The Quest for Delta Lake
Setting up Delta Lake was like embarking on a quest. Data adventurers ventured into the cloud, AWS, GCP, Azure, or even their local domains. They armed themselves with the delta-spark spells and summoned the JARs of delta-core, delta-contrib, and delta-storage, tailored to their Spark versions.
Requirements:
Python
Delta-spark
Delta jars
You can configure in a Spark session and define the package name so it will be downloaded at the run time. As I said, I am using Spark version 3.3. We will require these things: delta-core, delta-contribs, delta-storage. You can download them from here: https://github.com/delta-io/delta/releases/
To use and configure various cloud storage options, there are separate .jars you can use: https://docs.delta.io/latest/delta-storage.html. Here, you can find .jars for AWS, GSC, and Azure to configure and use their data storage medium.
Run this command to install delta-spark first:
pip install delta-spark
(If you are using dataproc or EMR, you can install this while creating cluster as a startup action, and if you are using serverless env like Glue or dataproc batches, you can create docker build or pass the .whl file for this package.)
You must also do this while downloading the .jar. If it is serverless, download the .jar, store it in cloud storage, like S3 or GS, and use that path while running the job. If it is a cluster like dataproc or EMR, you can download this on the cluster.
One can also download these .jars at the run time while creating the Spark session as well.
Now, create the Spark session, and you are ready to play with Delta tables.
Environment Setup
How do you add the Delta Lake dependencies to your environment?
You can directly add them while initializing the Spark session for Delta Lake by passing the specific version, and these packages or dependencies will be downloaded during run time.
You can place the required .jar files in your cluster and provide the reference while initializing the Spark session.
You can download the .jar files and store them in cloud storage, and you can pass them as a run time argument if you don’t want to download the dependencies on your cluster.
You have to add the following properties to use delta in Spark:-
Spark.sql.extensions
Spark.sql.catalog.spark_catalog
You can see these values in the above code snippet. If you want to use cloud storage like reading and writing data from S3, GS, or Blob storage, then we have to set some more configs as well in the Spark session. Here, I am providing examples for AWS and GSC only.
The next thing that will come to your mind: how will you be able to read or write the data into cloud storage?
For different cloud storages, there are certain .jar files available that are used to connect and to do IO operations on the cloud storage. See the examples below.
You can use the above approach to make this .jar available for Spark sessions either by downloading at a run time or storing them on the cluster itself.
There are two modes available in Delta Lake and Spark (Append and Overwrite) while writing the data in the Delta tables from any source.
For now, we have enabled the Delta catalog to store all metadata-related information. We can also use the hive meta store to store the metadata information and to directly run the SQL queries over the delta tables. You can use the cloud storage path as well.
Here, you can see the folder structure, and after writing the data into Delta Tables, it creates one delta log file, which keeps track of metadata, partitions, and files.
Option 2: Create Delta Table and insert data using Spark SQL.
spark.sql("CREATE TABLE orgs_data(index String, c_name String, organization_id String, name String, website String, country String, description String, founded String, industry String, num_of_employees String, remarks String) USING DELTA")
Insert the data:
df.write.mode('append').format("delta").option("mergeSchema", "true").saveAsTable("orgs_data")spark.sql("SELECT * FROM orgs_data").show()spark.sql("DESCRIBE TABLE orgs_data").show()
This way, we can read the Delta table, and you can use SQL as well if you have enabled the hive.
Schema Enforcement: Safeguarding Your Data
In the realm of data management, maintaining the integrity of your dataset is paramount. Delta Lake, with its schema enforcement capabilities, ensures that your data is not just welcomed with open arms but also closely scrutinized for compatibility. Let’s dive into the meticulous checks
Delta Lake performs when validating incoming data against the existing schema:
Column Presence: Delta Lake checks that every column in your DataFrame matches the columns in the target Delta table. If there’s a single mismatch, it won’t let the data in and, instead, will raise a flag in the form of an exception.
Data Types Harmony: Data types are the secret language of your dataset. Delta Lake insists that the data types in your incoming DataFrame align harmoniously with those in the target Delta table. Any discord in data types will result in a raised exception.
Name Consistency: In the world of data, names matter. Delta Lake meticulously examines that the column names in your incoming DataFrame are an exact match to those in the target Delta table. No aliases allowed. Any discrepancies will lead to, you guessed it, an exception.
This meticulous schema validation guarantees that your incoming data seamlessly integrates with the target Delta table. If any aspect of your data doesn’t meet these strict criteria, it won’t find a home in the Delta Lake, and you’ll be greeted by an error message and a raised exception.
Schema Evolution: Adapting to Changing Data
In the dynamic landscape of data, change is the only constant. Delta Lake’s schema evolution comes to the rescue when you need to adapt your table’s schema to accommodate incoming data. This powerful feature offers two distinct approaches:
Overwrite Schema: You can choose to boldly overwrite the existing schema with the schema of your incoming data. This is an excellent option when your data’s structure undergoes significant changes. Just set the “overwriteSchema” option to true, and voila, your table is reborn with the new schema.
Merge Schema: In some cases, you might want to embrace the new while preserving the old. Delta Lake’s “Merge Schema” property lets you merge the incoming data’s schema with the existing one. This means that if an extra column appears in your data, it elegantly melds into the target table without throwing any schema-related tantrums.
Should you find the need to tweak column names or data types to better align with the incoming data, Delta Lake’s got you covered. The schema evolution capabilities ensure your dataset stays in tune with the ever-changing data landscape. It’s a smooth transition, no hiccups, and no surprises, just data management at its finest.
The above code will overwrite the existing delta table with the new schema along with the new data.
Delta Lake has support for automatic schema evolution. For instance, if you have added two more columns in the Delta Lake tables and still tried to access the existing table, you will be able to read the data without any error.
There is another way as well. For example, if you have three columns in a Delta table but the incoming table has four columns, you can set up spark.databricks.delta.schema.autoMerge.enabled is true. It can be done for the entire cluster as well.
spark.sql("DESCRIBE TABLE orgs_data").show()
Let’s add one more column and try to access the data again:
As you can see, that column has been added but has not impacted the data. You can still smoothly and seamlessly read the data. It will set null to a newly created column.
What happens if we receive the extra column in an incoming CSV that we want to append to the existing delta table? You have to set up one config here for that:
input_df = spark.read.format('csv').option('header', 'true').load("../Desktop/Data-Engineering/data-samples/input-data/organizations-11111.csv")input_df.printSchema()input_df.write.mode('append').format("delta").option("mergeSchema", "true").saveAsTable("orgs_data")spark.sql("SELECT * FROM orgs_data").show()spark.sql("DESCRIBE TABLE orgs_data").show()
You have to add this config mergeSchem=true while appending the data. It will merge the schema of incoming data that is receiving some extra columns.
The first figure shows the schema of incoming data, and in the previous one, we have already seen the schema of our delta tables.
Here, we can see that the new column that was coming in the incoming data is merged with the existing schema of the table. Now, the delta table has the latest updated schema.
Time Travel
Basically, Delta Lake keeps track of all the changes in _delta_log by creating a log file. By using this, we can fetch the data of the previous version by specifying the version number.
Here, we can see the first version of the data, where we have not added any columns. As we know, the Delta table maintains the delta log file, which contains the information of each commit so that we can fetch the data till the particular commit.
Upsert, Delete, and Merge
Unlocking the Power of Upsert with Delta Lake
In the exhilarating realm of data management, upserting shines as a vital operation, allowing you to seamlessly merge new data with your existing dataset. It’s the magic wand that updates, inserts, or even deletes records based on their status in the incoming data. However, for this enchanting process to work its wonders, you need a key—a primary key, to be precise. This key acts as the linchpin for merging data, much like a conductor orchestrating a symphony.
A Missing Piece: Copy on Write and Merge on Read
Now, before we delve into the mystical world of upserting with Delta Lake, it’s worth noting that Delta Lake dances to its own tune. Unlike some other table formats like Hudi and Iceberg, Delta Lake doesn’t rely on the concepts of Copy on Write and Merge on Read. These techniques are used elsewhere to speed up data operations.
Two Paths to Merge: SQL and Spark API
To harness the power of upserting in Delta Lake, you have two pathways at your disposal: SQL and Spark API. The choice largely depends on your Delta version. In the latest Delta version, 2.2.0, you can seamlessly execute merge operations using Spark API. It’s a breeze. However, if you’re working with an earlier Delta version, say 1.0.0, then Spark SQL is your trusty steed for upserts and merges. Remember, using the right Delta version is crucial, or you might find yourself grappling with the cryptic “Method not found” error, which can turn into a debugging labyrinth.
In the snippet below, we showcase the elegance of upserting using Spark SQL, a technique that ensures your data management journey is smooth and error-free:
-- Insert new data and update existing data based on the specified keyMERGEINTO targetTable AS targetUSING sourceTable AS sourceON target.id = source.idWHENMATCHEDTHENUPDATESET*WHENNOTMATCHEDTHENINSERT*WHENNOTMATCHEDBYSOURCETHENDELETE;
today_data_df = spark.read.format('csv').option('header', 'true').load("../Desktop/Data-Engineering/data-samples/input-data/organizations-11111.csv")today_data_df.show()spark.sql("select * from orgs_data where organization_id = 'FAB0d41d5b5ddd'").show()# Reading Existing Delta tabledeltaTable = DeltaTable.forPath(spark, "orgs_data")today_data_df.createOrReplaceTempView("incoming_data")
Here, we are loading the incoming data and showing what is inside. The existing data with the same primary key appears in the Delta table so that we can compare after upserting or merging the data.
spark.sql("""MERGE INTO orgs_dataUSING incoming_dataON orgs_data.organization_id = incoming_data.organization_idWHENMATCHEDTHENUPDATESET organization_id = incoming_data.organization_id, name = incoming_data.name""")spark.sql("select * from orgs_data where organization_id = 'FAB0d41d5b5ddd'").show()
This is the example of how you can do upsert using Spark APIs. The merge operation creates lots of small files. You can control the number of small files by setting up the following properties in the spark session.
spark.delta.merge.repartitionBeforeWrite true
spark.sql.shuffle.partitions 10
This is how merge operations work. Merge supports one-to-one mapping. What we want to say is that only rows should try to update the one row in the target delta table. If multiple rows try to update the rows in the target Delta table, it will fail. Delta Lake matches the data on the basis of a key in case of an update operation.
Change Data Feed
This is also a useful feature of Delta Lake and tracks and maintains the history of all records in the Delta table after upsert or insert at the row level. You can enable these things at the beginning while setting up the Spark session or using Spark SQL by enabling “change events” for all the data.
Now, you can see the whole journey of each record in the Delta table, from its insertion to deletion. It introduces one more extra column, _change_type, which contains the type of operations that have been performed on that particular row.
Now, after deleting something, you will be able to see the changes, like what is deleted and what is updated. If you are doing upserts on the same Delta table after enabling the change data feed, you will be able to see the update as well, and if you insert anything, you will be able to see what is inserted in your Delta table.
If we overwrite the complete Delta table, it will mark all past records as a delete:
If you want to record each data change, you have to enable this before creating the table so that we can see the data changes for each version. If you’ve already created one table, you won’t be able to see the changes for the previous version once you enable the change data feed, but you will be able to see the changes in all versions that came after this configuration.
Data Skipping with Z-ordering
Data skipping is the technique in Delta Lake where, if you have a larger number of records stored in many files, it will read the data from the files that contain required information, but apart from that, other files will get skipped. This makes it faster to read the data from the Delta tables.
Z-ordering is a technique used to colocate the information in the same dataset files. If you know the column that will be more in use in the select statement and has A cardinality, you can use Z-order by that particular column. It will reduce the large number of files from being read. We can give you multiple columns for Z-order by separating them from commas.
Let’s suppose you have two tables, a and b, and there is one column that is most frequently used. You can increase the number of files to be skipped, and you can use the columns files running that query. Normal order works linearly, whereas Z-order works in multi- dimensionally.
OPTIMIZE eventsWHERE date >=current_timestamp() -INTERVAL1 dayZORDERBY (eventType)
DML Operations
Delta Lake has capabilities to run all the DML operations of SQL in the data lake as well as update, delete, and merge operations.
Integrations and Ecosystem Supported in Delta Lake
Read Delta Tables
Unlock the Delta Tables: Tools That Bring Data to Life
Reading data from Delta tables is like diving into a treasure trove of information, and there’s more than one way to unlock its secrets. Beyond the standard Spark API, we have a squad of powerful allies ready to assist: SQL query engines like Athena and Trino. But they’re not just passive onlookers; they bring their own magic to the table, empowering you to perform data manipulation language (DML) operations that can reshape your data universe.
Athena: Unleash the SQL Sorcery
Imagine Athena as the Oracle of data. With SQL as its spellbook, it delves deep into your Delta tables, fetching insights with precision and grace. But here’s the twist: Athena isn’t just for querying; like a skilled blacksmith, it can help you hammer your data into a new shape, creating a masterpiece.
Trino: The Shape-Shifting Wizard
Trino, on the other hand, is the shape-shifter of the data realm. It glides through Delta tables, allowing you to perform an array of DML operations that can transform your data into new, dazzling forms. Think of it as a master craftsman who can sculpt your data, creating entirely new narratives and visualizations.
So, when it comes to Delta tables, these tools are not just readers; they are your co-creators. They enable you to not only glimpse the data’s beauty but also mold it into whatever shape serves your purpose. With Athena and Trino at your side, the possibilities are as boundless as your imagination.
Read Delta Tables Using Spark APIS
from delta.tables import*delta_df =DeltaTable.forPath(spark,"./Documents/DE/Delta/test-db/organisatuons")<br>delta_df.toDf().show()
Steps to Set Up Delta Lake with S3 on EC2 Or EMR and Access Data through Athena
Data Set Used – We have generated some dummy data of around 100gb and written that into the Delta tables.
Step 1- Set up a Spark session along with AWS cloud storage and Delta – Spark. Here, we have used an EC2 instance with Spark 3.3 and Delta version 2.1.1. Here, we are setting up Spark config for Delta and S3.
Spark Version – You can use any Spark version, but Spark 3.3.1 came along with the pip install. Just make sure whatever version you are using is compatible with the Delta Lake version that you are using; otherwise, most of the features won’t work.
Step 2 – Here, we are creating a Delta table with an S3 path. We can directly write the data into an S3 bucket as a Delta table, but it is better to create a table first and then write it into S3 to make sure the schema is correct.
Set the Delta location path if it exists to run the Spark SQL query and create the Delta table along with the S3 path.
# If table is already theredelta_path ="s3a://abhishek-test-01012023/delta-lake-sample-data/"spark.conf.set('table.location', delta_path)# Creating new delta table on s3 locationspark.sql("CREATE TABLE delta.`s3://abhishek-test-01012023/delta-lake-sample-data/`(id INT, first_name String, ""last_name String, address String, pincocde INT, net_income INT, source_of_income String, state String, ""email_id String, description String, population INT, population_1 String, population_2 String, ""population_3 String, population_4 String, population_5 String, population_6 String, population_7 String, ""date String) USING DELTA PARTITIONED BY (date)")
Step 3 – Here, I have given one link that I have used to generate the dummy data and have written that into the S3 bucket as Delta tables. Feel free to look over this. An example code of writing is given below:
In the above link, you find the code of dummy data generation.
Step 4 – Here, we are printing the count and selecting some data from the Delta table that we have written in just right away.
Run the SQL query to check the table data and upsert using S3 bucket data:
spark.sql("select count(*) from delta.`s3://abhishek-test-01012023/delta-lake-sample-data/` group by id having count(""*) > 1").show()spark.sql("select count(*) from delta.`s3://abhishek-test-01012023/delta-lake-sample-data/`").show()############################################################################## Upsert############################################################################## upserts the starting five records. We will read first five record and will do some changes in some columns andinput_df = spark.read.csv("s3a://abhishek-test-01012023/incoming_data/delta/4e0ae9f5-8c9d-435a-a434-febff1effbc3.csv",inferSchema=True,header=True)input_df.printSchema()input_df.createOrReplaceTempView("incoming")spark.sql("MERGE INTO delta.`s3://abhishek-test-01012023/delta-lake-sample-data/` t USING (SELECT * FROM incoming) s ON t.id = s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *")
This is the output of select statement:
This is the schema of the incoming data we are planning to merge into the existing Delta table:
After upsert, let’s see the data for the particular data partition:
spark.sql(“select * from delta.`s3://abhishek-test-01012023/delta-lake-sample-data/` where id = 1 and date = 20221206”)
Access Delta table using Hive or any other external metastore:
For that, we have to create a link between them and to create this link, go to the Spark code and generate the manifest file on the S3 path where we have already written the data.
spark.sql("GENERATE symlink_format_manifest FOR TABLEdelta.`s3a://abhishek-test-01012023/delta-lake-sample-data/`")
This will create the manifest folder not go to Athena and run this query:
In a world where data continued to grow in volume and complexity, Delta Lake stood as a beacon of hope. It empowered organizations to manage their data lakes with unprecedented efficiency and extract insights with unwavering confidence.
The adoption of Delta Lake marked a new dawn in the realm of data. Whether dealing with structured or semi-structured data, it was the answer to the prayers of data warriors. As the sun set on traditional formats, Delta Lake emerged as the hero they had been waiting for—a hero who had not only revolutionized data storage and processing but also transformed the way stories were told in the world of data.
And so, the legend of Delta Lake continued to unfold, inspiring data adventurers across the land to embark on their own quests, armed with the power of ACID transactions, time travel, and the promise of a brighter, data-driven future.
Have you ever wondered how Google Maps, Starlink, Zomato, Arogya Setu, and even methods like population clustering are able to add value to the human world? Well, the common thread between these applications and technologies is the use of spatial data and analysis techniques.
Both Google Maps and Zomato use spatial techniques to provide navigation and location-based information to their users. While Arogya Setu is a contact tracing app that uses spatial data to track the spread of infectious illnesses, Starlink uses spatial data analysis to provide internet access to remote areas around the world. Population clustering is a technique that can be useful for urban planning, public health, and disaster response. Since the use of spatial data and its analysis techniques has become increasingly critical in the current scenario, let’s understand some fundamentals and explore different aspects of spatial data analytics.
So, welcome to the world of spatial data analytics, where data meets geography and insights come to life! The use of spatial data analytics has changed the way we understand and interact with the world around us, providing insights and solutions to some of the most pressing challenges facing humanity today. So, let’s cut through the process by taking a quick tour of a spatial journey that you might have never been on before.
What is spatial data analytics?
Before we start talking about the process of spatial data analytics, let’s try to understand what is special about the term “spatial data.” Spatial data, also known as “Geospatial data,” refers to data representing features or objects on the Earth’s surface. Whether it’s man-made or natural, if it has to do with a specific location on the surface of the Earth, it’s spatial. Spatial data refers to where things are now, or perhaps, where they were or will be in the future.
This data can be further classified as:
Geometric Data:
Geometric data is a type of spatial data mapped on a two-dimensional flat surface. Google Maps is an application that uses geometric data to provide accurate directions.
Geographic Data:
Geographic data is information mapped around the Earth that highlights the latitude and longitude relationships to a specific object or location. A familiar example of geographic data is a Global Positioning System (GPS).
Spatial data is not limited to structured information; it also comprises imagery from satellites and drones, address data points, and longitudinal and latitudinal data. Primarily, spatial data is classified as vector data and raster data. Vector data consists of coordinate information, while raster data is all about layers of images extracted from camera sensors.
The real world can be represented as below, where the built environment (roads, buildings) and administrative data (countries, census areas) tend to be represented as vector data. Natural environment (e.g., elevation, temperature, precipitation) is often represented using a raster grid.
Discrete data, stored according to its exact geographic location, is called vector data.
Continuous data is represented by regular grids called raster data.
Attributes
(Image source: CVRD)
Vector Data:
Points: A single dot on the layer depicts them. It can be either an x, y, or z coordinate.
Lines: This form of vector data is presented using two coordinates, i.e., either the x-y coordinate or the inverse of this, and has a definite length. These are used for rivers, roads, railways, ferry routes, and even major pipeline flows.
Polygon: The feature is defined using three or more coordinates. It is used to showcase inland water bodies like lakes, buildings, etc.
Raster Data:
Raster is all about multilayered map images from satellites, drones, and various other camera sensors (ortho-imagery).
It is stored in cell-based and color-pixel formats. These pixels are arranged in columns and rows.
Analysis can be done better than with vector-based data due to the richness of the data.
It can give you more accurate measurements than other types of data.
Attributes and Properties:
Spatial data contains more information than just a location on the surface of the Earth.
Any additional information, or non-spatial data, that describes a feature is referred to as an attribute.
In addition to locational and attribute information, spatial data inherently contains geometric and topological properties, which help to gain deeper insights.
Geometric properties include position and measurements, such as length, direction, area, and volume.
Topological properties represent spatial relationships such as connectivity, inclusion, and adjacency.
As seen above, spatial data includes information such as geographic coordinates, elevation, and demographic information. Hence, it can be used to identify patterns, correlations, and trends that are not readily apparent through other data sources. For instance, geospatial data can be used to map the distribution of air pollution across a city, identify areas at risk for natural disasters like floods or wildfires, or monitor changes in land use over time. Here is where the analytics process takes place to uncover insights that can aid in providing solutions.
Spatial data analytics involves collecting, processing, and analyzing various types of spatial data with insights to go beyond what occurs to determine not only where and when something occurs but also why it occurs at that specific place and/or time. It can be further viewed as descriptive analytics, which involves summarizing and visualizing spatial data to identify patterns and trends. Predictive analytics uses statistical models to make predictions about future events or trends based on past data. Prescriptive analytics uses optimization techniques to determine the best course of action given a specific set of circumstances.
Why is spatial data analytics important?
Spatial data analytics plays an essential role in many industries and fields, providing insights and solutions that can have a significant impact on our daily lives. It aids businesses in gaining a competitive edge through improved decision-making and time and money savings. Urban planning, telecommunications, military, public health, and emergency management are just a few examples of industries that rely heavily on spatial data analytics to make informed decisions.
(Image source: OneStopGIS)
Public Health
A patient’s location directly influences their health. Whether it’s disease prevention or clinic site selection, considering spatial aspects in healthcare analytics can have a drastic impact.
Urban Planning
An urban planner might want to assess the extent of urban fringe growth, quantify the population growth that some suburbs are witnessing, and also understand why these particular suburbs are growing, and others are not.
Environmental and Natural Resources
Protecting our world against climate change, promoting biodiversity, exploration, and conservation planning requires spatial storytelling and sophisticated environmental analysis.
Space and Navigation
Optimizing transport infrastructure and navigation spatially is key to the future of mobility. The most efficient cities are moving away from traditional methods to analyze new data.
Telecommunication
Since network signal strength fluctuates by location over time, spatial analytics helps telecommunications companies understand where anomalies occur and then resolve them.
Architecture, Engineering, and Construction
The leading AEC firms are going beyond traditional workflows to use spatial data science in urban planning and site selection, reducing costs and boosting project profitability. A geological engineer might want to identify the best localities for constructing buildings in an earthquake-prone area by looking at rock formation characteristics.
Military
Spatial predictive analytics helps the military optimize the placement of resources while using predictive analytics to assess infrastructure, situational awareness, anticipate maintenance needs, and meet deadlines.
Weather Forecasting
Rapid response to extreme weather by visualizing blizzards, wildfires, and hurricanes fast enough for effective evacuation alerts. Spatial data analytics also helps airlines with routing and gives insurance companies a better way to assess property risk.
How to perform spatial data analytics?
The process of spatial data analytics involves data gathering, data cleaning, data processing, and visualization, much like any other traditional analytics technique. The specific details of the process will be determined on the basis of the data and the goals of the analysis.
Data Collection: The initial stage in spatial data analytics is to collect the relevant data. This involves gathering data from different sources, such as remote sensing satellites, GPS-enabled devices, social media, or survey instruments. The data may include geographic coordinates, attributes of features, and other pertinent information that can help analyze the data.
Data Cleaning and Preprocessing: Once the data is collected, it needs to be cleaned and preprocessed to ensure that it is accurate and usable for further processing. This may involve eliminating duplicates, filling in missing values, and standardizing data formats.
Data Transformation: Spatial data is often obtained from numerous sources and in a variety of forms, so the next step is to transform and combine the data into a single data set. This may involve joining tables or layers based on a shared attribute or location.
Data Analysis: This part of spatial data analytics involves identifying spatial patterns and relationships in the data. This may involve various techniques such as clustering, interpolation, spatial regression, and spatial autocorrelation analysis. The analysis may also include visualizing the data using maps, charts, and graphs for spatial data exploration.
Modeling and Prediction: Based on the results of spatial analysis, it may be possible to build models to predict future patterns or trends in the data as a part of predictive analytics. This may involve using machine learning algorithms or other statistical techniques to identify patterns and make predictions.
Business Intelligence: Finally, the results of spatial data analytics can be used to support decision-making in a variety of contexts, such as urban planning, natural resource management, or emergency response. The decision-making process may involve evaluating trade-offs between different options and considering the potential impact of different decisions on the spatial patterns in the data.
Tools and Techniques:
Spatial Data Storage
Spatial data storage is a specialized form of data storage that takes into account the spatial relationships between various data points, allowing for more efficient and effective analysis and retrieval of information. There are many tools available for spatial data storage, including both open-source and proprietary software. Here are a few instances of such tools.
(Image source: Safe Software)
RDBMS (Relational Database Management Systems): RDBMS are among the most used methods for storing geographical data having extensions that enable spatial features. RDBMS examples supporting geographic data include:
Spatial File Formats: Spatial file formats are widely used for storing and sharing spatial data. Examples of spatial file formats include:
Shapefile (.shp)
GeoJSON (.geojson)
Keyhole Markup Language (KML) (.kml)
Geography Markup Language (GML) (.gml)
NoSQL: NoSQL databases are becoming increasingly popular for spatial data storage due to their ability to handle large and complex datasets, flexible schema, and scalability. Examples of NoSQL databases that support spatial data include:
Cloud-based Storage Services: Cloud-based storage services like AWS, GCP, Azure are popular options for storing spatial data, which can be termed as DataLakes. Examples of cloud-based storage services that support spatial data include:
Amazon S3 with Amazon S3 GeoSpatial Indexing
Google Cloud Storage with Google Cloud Storage Geo-Location
Microsoft Azure Blob Storage with Azure Spatial Anchors
Spatial Data Warehouses: Spatial data warehouses are specialized databases designed for spatial data analysis. Examples of spatial data warehouses include:
It can be noted that tools, such as RDBMS and NoSQL databases, can also be used for spatial data analytics and processing in addition to storage.
Spatial Data Processing
Spatial data processing is an important step in spatial data analytics to ensure that the data is properly aligned and in a consistent format before further analysis. This is a must-do step because various applications and data sources use different formats and coordinate systems, which might lead to several difficulties when analyzing the data.
Below are a few examples of processing methods in a spatial context that ensure that spatial data is compatible and consistent across different applications and data sources.
Reprojection: Reprojection is the process of converting spatial data from one map projection to another. This is frequently necessary when working with data from multiple sources that use different projections.
Coordinate System/Datum Transformation: This transformation involves converting spatial data from one coordinate system to another or from one geodetic datum to another. This is important when working with data from different sources that use different coordinate systems and information.
Resampling: Resampling involves changing the resolution or scale of spatial data. This is often necessary when handling data at different scales coming from different sources.
Geocoding: Geocoding is the process of converting a street address or other location description into a set of geographic coordinates. This allows the location to be plotted on a map and later analyzed in a spatial context.
Georeferencing: Georeferencing is the method of aligning geographic data to a specific coordinate system or reference system. This is often required when working with data from several sources, such as aerial photographs or satellite imagery.
Digitizing: Digitizing is the process of converting analog maps or other spatial data into a digital format. This involves manually tracing features such as roads, buildings, and water bodies using a computer program.
Several tools are available that can perform such data processing techniques, and a few of these tool instances are given below.
GIS (Geographic Information Systems): GIS connects data to a map, integrating location data with all types of descriptive information. It helps users understand patterns, relationships, and geographic context. The benefits include improved communication and efficiency, as well as better management and decision-making. Examples of GIS software that supports spatial data processing include:
ArcGIS – A proprietary GIS software with a comprehensive set of features and tools
QGIS – An open-source GIS software with a wide range of plugins and tools
Python Libraries: Python is a popular programming language for spatial data processing, and there are several libraries available for this purpose. Examples of Python libraries that support spatial data processing include:
GeoPandas: A library for working with geospatial data in Python
Shapely: A library for manipulation and analysis of planar geometric objects
PySAL: A library for spatial analysis and modeling
R Packages: Like Python, R is another popular programming language for spatial data processing, and there are several packages available for spatial data operations. Examples of R packages that support spatial data processing include:
SQL: SQL can be used for spatial data processing and analysis, especially when working with spatial databases with extensions like PostGIS. Examples of SQL spatial functions include:
ST_Contains: checks if one geometry contains another
ST_Distance: calculates the distance between two geometries
Command-line Tools: There are a handful of command-line tools available for spatial data processing. Examples of command-line tools that support spatial data processing include:
GDAL/OGR: a suite of tools for geospatial data processing and conversion
GRASS GIS: a command-line tool for geospatial analysis and modeling
There are a few other tools for data processing worth exploring, such as MATLAB, GeoServer, Global Mapper, and Mapbox. Tools like GIS software and Python libraries can also be used for spatial data storage and analysis in addition to processing.
(Image source: Carto)
Spatial Data Analysis
Spatial data analysis is the process of examining geographic data to spot trends, correlations, and patterns. It involves the use of statistical, computational, and visualization methods to explore spatial data and extract business insights. There are different categories of spatial data analysis, such as:
Proximity Analysis: It involves measuring the distance between two or more places in a spatial dataset. It is possible to analyze proximity using methods like Euclidean distance.
Accessibility Analysis: It is a measure of how easy it is to get to a location from other locations in the dataset. In addition to distance, Accessibility analysis takes into account other factors that affect how easy it is to travel between locations, such as traffic, road conditions, and public transportation.
Spatial Clustering: Spatial clustering is the process of identifying groups of spatially adjacent objects that have similar characteristics. Hierarchical clustering and k-means clustering are two methods that can be used to accomplish this.
Spatial Interpolation: Spatial interpolation involves estimating values for locations where data is not available based on nearby data points. This can be done using techniques such as kriging or inverse distance weighting.
Spatial Exploratory Data Analysis: It involves creating visual representations of spatial data to explore patterns and relationships. Spatial EDA helps to identify patterns and relationships that may not be immediately apparent from the data and can help guide further analysis. This can include techniques such as choropleth maps, heat maps, or scatter plots.
Spatial Simulation: This involves using simulation models to study the behavior of spatial systems over time. Spatial simulation includes techniques such as cellular automata, agent-based models, and Monte Carlo simulations. Spatial simulation is useful for predicting the future behavior of spatial systems under different scenarios.
There are other categories, like factor analysis, trajectory analysis, network analysis, etc., that can be used for fine-grained spatial data analysis. Below are a few examples of tools that can be used to devise an analysis of spatial data.
GIS (Geographic Information Systems): As seen earlier, this software can be used not only for capturing and processing the data but also to analyze and display geographically referenced data. Examples include ArcGIS, QGIS, and GRASS GIS.
Open Source Libraries and Binaries: It includes various programming languages with many packages for spatial data analysis, such as the sp package for handling spatial data and the rgdal package for reading and writing geospatial data formats in R or packages, such as GeoPandas and Shapely to provide functionality for working with geospatial data in Python. The list can go on with the GDAL framework and its dependencies.
PostGIS: PostGIS is a spatial database extender for the PostgreSQL database management system. It adds support for geographic objects, allowing you to manipulate and query geospatial data within the database for any kind of analysis purpose.
Data Visualization Tools: These tools are used to create visual representations of spatial data for exploratory data analysis. Examples include Tableau, ArcGIS Pro, and QGIS.
Mapbox: Mapbox is a mapping platform that provides APIs and SDKs for building custom maps and applications. It includes tools for data visualization, geocoding, routing, and more.
ENVI: ENVI is a software package for processing and analyzing remote sensing data. It includes tools for image classification, spectral analysis, and terrain modeling, among others.
Spatial data analysis plays an essential role in understanding complex spatial patterns and relationships and can help formulate business decisions in a wide range of areas. The choice of tool depends on the type of analysis that needs to be performed, the size of the data set, and the resources available.
How to solve spatial big data problems?
Big data refers to datasets that are too large and complex to process and analyze using the traditional methods that we discussed earlier. When dealing with spatial data, the challenges of big data are amplified due to the added dimensions of space and time. Spatial data is being captured at an unusual rate because of the growing numbers of sensors and devices, the networks of GPS satellites and cell towers, and the rise of the Internet of Things.
Spatial data analytics can leverage strategies and resources, including distributed computing, cloud computing, and parallel processing, to address the above issues. These techniques allow for the processing and analysis of large spatial datasets, enabling real-time decision-making in industries including transportation, agriculture, and public safety. For instance, massive geographic data analytics are used by real-time traffic management systems to optimize traffic flows, ease congestion, and improve safety.
(Image source: Utilizing Cloud Computing to Address Big Geospatial Data Challenges Paper)
In the telecommunications industry, spatial analysis can be used to optimize network coverage and capacity, plan new infrastructure, and identify areas of high network congestion.
Let’s consider a hypothetical telecommunications company that wants to improve its network performance and customer experience by analyzing geospatial data. Specifically, the company wants to analyze call detail records (CDRs) to identify areas of high call volume and network congestion.
(Image source: Microsoft Azure Architectures)
In a given solution by Azure in a published article, the suggested architecture involves:
Azure Data Factory, which is used to collect the CDRs from various sources (mainly geospatial databases).
Azure Data Factory stores them in Azure Data Lake Storage in formats such as GeoJSON, WKT, and Vector tiles. The bronze container holds raw data, the silver container holds semi-curated data, and the gold container holds fully curated data as the processing proceeds.
Azure Databricks and the GeoSpark/Sedona package are being used to convert data formats and efficiently load, process, and analyze large-scale spatial data across machines.
GeoPandas exports data in various formats, which are later used by GIS applications such as QGIS and ArcGIS for exploratory analysis.
Azure Machine Learning extracts insights from geospatial data, determining, for example, where and when to deploy new wireless access points.
Power BI or Azure Maps can be used to visualize the geospatial data and identify areas where network upgrades or infrastructure improvements are needed.
A log analytics system is set up to run queries against data in Azure Monitor Logs to implement a robust and fine-grained logging system to analyze events and performance.
Overall, the Azure-based solution gives an idea about how one can try to perform geospatial analysis in the telecommunications industry and improve network performance and customer experience. You can read more about this solution here.
Challenges and limitations
In conclusion, spatial data analytics is an essential component of decision-making across a range of industries. It is important to understand the techniques, infrastructure, and challenges of spatial data analytics to effectively leverage spatial data and make informed decisions. Spatial data collection can be challenging and may contain faults or inconsistencies. The data may not be available for certain geographic areas or for certain time periods. Spatial data analytics can raise privacy concerns if personal data is collected and used without consent. In addition, there may be concerns about the use of spatial data analytics for surveillance or other unethical purposes, which can lead to significant harm.
Conclusion
Spatial data analytics is a powerful tool that can help organizations make better-informed decisions and gain a competitive advantage. As the fields of machine learning (AI) and spatial data analysis intertwine, spatial data analytics looks promising and quite useful for real-life problems. The blend of both vector and raster data produces a powerful product that can tackle various economic and earth-related problems. This blog is just a high-level overview of spatial data analytics since you have just scratched the surface, but I can guarantee that this spatialride will be smoother from here on.
As marketed, Spinnaker is an open-source, multi-cloud continuous delivery platform that helps you release software changes with high velocity and confidence.
Open sourced by Netflix and heavily contributed to by Google, it supports all major cloud vendors (AWS, Azure, App Engine, Openstack, etc.) including Kubernetes.
In this blog I’m going to walk you through all the basic concepts in Spinnaker and help you create a continuous delivery pipeline using Kubernetes Engine, Cloud Source Repositories, Container Builder, Resource Manager, and Spinnaker. After creating a sample application, we will configure these services to automatically build, test, and deploy it. When the application code is modified, the changes trigger the continuous delivery pipeline to automatically rebuild, retest, and redeploy the new version.
What Spinnaker Provides?
Application management and Application Deployment are its two core features.
Application Management
Spinnaker’s application management features can be used to view and manage your cloud resources.
Modern tech organizations operate collections of services—sometimes referred to as “applications” or “microservices”. A Spinnaker application models this concept.
Applications, Clusters, and Server Groups are the key concepts Spinnaker uses to describe services. Load balancers and Firewalls describe how services are exposed to users.
Application
An application in Spinnaker is a collection of clusters, which in turn are collections of server groups. The application also includes firewalls and load balancers. An application represents the service which needs to be deployed using Spinnaker, all configuration for that service, and all the infrastructure on which it will run. Normally, a different application is configured for each service, though Spinnaker does not enforce that.
Cluster
Clusters are logical groupings of Server Groups in Spinnaker.
Note: Cluster, here, does not map to a Kubernetes cluster. It’s merely a collection of Server Groups, irrespective of any Kubernetes clusters that might be included in your underlying architecture.
Server Group
The base resource, the Server Group, identifies the deployable artifact (VM image, Docker image, source location) and basic configuration settings such as number of instances, autoscaling policies, metadata, etc. This resource is optionally associated with a Load Balancer and a Firewall. When deployed, a Server Group is a collection of instances of the running software (VM instances, Kubernetes pods).
Load Balancer
A Load Balancer is associated with an ingress protocol and port range. Traffic is balanced among the instances present in Server Groups. Optionally, health checks can be enabled for a load balancer, with flexibility to define health criteria and specify the health check endpoint.
Firewall
A Firewall defines network traffic access. It is effectively a set of firewall rules defined by an IP range (CIDR) along with a communication protocol (e.g., TCP) and port range.
Application Deployment
Pipeline
The pipeline is the key deployment management construct in Spinnaker. It consists of a sequence of actions, known as stages. Parameters can be passed from one stage to the next one in the pipeline.
You can start a pipeline manually, or you can configure it to be automatically triggered by an event, such as a Jenkins job completing, a new Docker image being pushed in your docker registry, a CRON type schedule, or maybe a stage in another pipeline.
You can configure the pipeline to emit notifications, by email, SMS or HipChat, to interested parties at various points during pipeline execution (such as on pipeline start/complete/fail).
Stage
A Stage in Spinnaker is an atomic building block for a pipeline, describing an action that the pipeline will perform. You can sequence stages in a Pipeline in any order, though some stage sequences may be more common than others. There are different types of stages in Spinnaker such as Deploy, Manual Judgment, Resize, Disable, and many more. The full list of stages and read about implementation details for each provider here.
Deployment Strategies
Spinnaker supports all the cloud native deployment strategies including Red/Black (a.k.a Blue/Green), Rolling red/black and Canary deployments, etc.
What is Spinnaker Made Of?
Spinnaker is composed of a number of independent microservices:
Gate is the API gateway. All the API calls from UI (Deck) and other API callers go to Spinnaker through Gate.
Orca is the orchestration engine. It handles all ad-hoc operations and pipelines.
Clouddriver is responsible for all mutating calls to the cloud providers and for indexing/caching all deployed resources.
Front50 is used to persist the metadata of applications, pipelines, projects and notifications.
Rosco is the bakery. It helps to create machine images for various cloud vendors (for example GCE images for GCP, AMIs for AWS, Azure VM images). It currently wraps Packer, but will be expanded to support additional mechanisms for producing images.
Igor is used to trigger pipelines via continuous integration jobs in systems like Jenkins and Travis CI, and it allows Jenkins/Travis stages to be used in pipelines.
Echo is Spinnaker’s eventing bus. It supports sending notifications (e.g. Slack, email, Hipchat, SMS), and acts on incoming webhooks from services like GitHub.
Fiat is Spinnaker’s authorization service. It is used to query a user’s access permissions for accounts, applications and service accounts.
Kayenta provides automated canary analysis for Spinnaker.
Halyard is Spinnaker’s configuration service. Halyard manages the lifecycle of each of the above services. It only interacts with these services during Spinnaker start-up, updates, and rollbacks.
By default, Spinnaker binds ports accordingly for all the above mentioned microservices. For us the UI (Deck) will be exposed onto Port 9000.
What are We Going to Do?
Set up your environment by launching Cloud Shell, creating a Kubernetes Engine cluster, and configuring your identity and user management scheme.
Download a sample application, create a Git repository, and upload it to a Cloud Source Repository.
Deploy Spinnaker to Kubernetes Engine using Helm.
Build a Docker image from the source code.
Create triggers to create Docker images when the source code for application changes.
Configure a Spinnaker pipeline to reliably and continuously deploy your application to Kubernetes Engine.
Deploy a code change, triggering the pipeline, and watch it roll out to production.
Note: This blog post uses various billable components in GCP like GKE, Container Builder etc.
Pipeline Architecture
To continuously deliver application updates to users, companies need an automated process that reliably builds, tests, and updates their software. Code changes should automatically flow through a pipeline that includes artifact creation, unit testing, functional testing, and production rollout. In some cases, they want a code update to apply to only a subset of their users, so that it is exercised realistically before pushing it to entire user base. If one of these canary releases proves unsatisfactory, the automated procedure must be able to quickly roll back the software changes.
With Kubernetes Engine and Spinnaker, we can create a robust continuous delivery flow that helps us to ensure that software is shipped as quickly as it is developed and validated. Although rapid iteration is the end goal, we must first ensure that each application revision passes through a series of automated validations before becoming a candidate for production rollout. When a given change has been vetted through automation, we can also validate the application manually and conduct further pre-release testing.
After the team decides the application is ready for production, one of the team members can approve it for production deployment.
Application Delivery Pipeline
We are going to build the continuous delivery pipeline shown in the following diagram.
Prerequisites
Fair bit of experience in GCP services like:
GKE (Google Kubernetes Engine)
Google Compute
Google APIs
Cloud Source Repository
Container Builder
Cloud Storage
Cloud Load Balancing
Knowledge in K8s terminology like Services, Deployments, Pods, etc
Familiarity with Kubectl and Helm package manager
Before Starting just enable the APIs needed on GCP
Go to the Console and scroll the left panel down to Compute->Kubernetes Engine->Kubernetes Clusters.
Click Create Cluster.
Choose a name or leave as the default one.
Under Machine Type, click Customize.
Allocate at least 2 vCPU and 10GB of RAM.
Change the cluster size to 2.
Enable Legacy Authorization while customizing the cluster.
Keep the rest of the defaults and click Create.
In a minute or two the cluster will be created and ready to go.
Configure identity and access management
Create a Cloud Identity and Access Management (Cloud IAM) service account to delegate permissions to Spinnaker, allowing it to store data in Cloud Storage. Spinnaker stores its pipeline data in Cloud Storage to ensure reliability and resiliency. If our Spinnaker deployment unexpectedly fails, we can create an identical deployment in minutes with access to the same pipeline data as the original.
1. Create the service account:
$ gcloud iam service-accounts create spinnaker-storage-account --display-name spinnaker-storage-account
2. Store the service account email address and our current project ID in environment variables for use in later commands:
$ exportSA_EMAIL=$(gcloud iam service-accounts list --filter="displayName:spinnaker-storage-account"--format='value(email)')$ exportPROJECT=$(gcloud info --format='value(config.project)')
3. Bind the storage.admin role to our service account:
4. Download the service account key. We will need this key later while installing Spinnaker and we need to also upload the key to Kubernetes Engine.
$ gcloud iam service-accounts keys create spinnaker-sa.json --iam-account $SA_EMAIL
Deploying Spinnaker using Helm
In this section, we will deploy Spinnaker onto the K8s cluster via Charts with the help of K8s package manager Helm. Helm has made it very easy to deploy Spinnaker, it can be a very painful act to deploy it manually via Halyard and configure it.
Use the Helm command-line interface to deploy the chart with the configuration set earlier. This command typically takes five to ten minutes to complete, so we will be providing a deploy timeout with ` — timeout`.
The above command exposes the Spinnaker UI onto the local machine that we’re using to run all the commands. We can use any port of our choosing instead of 8080 in above command. Now the UI can be opened onto the url http://localhost:8080.
Building the Docker image
In this section, we will configure Container Builder to detect changes to the application source code, if yes then build a Docker image, and then push it to Container Registry.
For this step we will use a sample app provided by the Google community
4. Set the username and email address for Git commits in this repository. Replace [EMAIL_ADDRESS] with Git email address, and replace [USERNAME] with Git username.
$ exportPROJECT=$(gcloud info --format='value(config.project)')$ git remote add origin https://source.developers.google.com/p/$PROJECT/r/sample-app
8. Push the code to the new repository’s master branch:
$ git push origin master
9. Check that we can see our source code in the console.
Configuring the build triggers
In this section, we configure Google Container Builder to build and push your Docker images every time we push Git tags to our source repository. Container Builder automatically checks out the source code, builds the Docker image from the Dockerfile in repository, and pushes that image to Container Registry.
In the GCP Console, click Build Triggers in the Container Registry section.
Select Cloud Source Repository and click Continue.
Select your newly created sample-app repository from the list, and click Continue.
Set the following trigger settings:
Name:sample-app-tags
Trigger type: Tag
Tag (regex): v.*
Build configuration: cloudbuild.yaml
cloudbuild.yaml location: /cloudbuild.yaml
Click Create trigger.
From now on, whenever we push a Git tag prefixed with the letter “v” to source code repository, Container Builder automatically builds and pushes our application as a Docker image to Container Registry.
Let’s build our first image:
Push the first image using the following steps:
1. Go to source code folder in Cloud Shell.
2. Create a Git tag:
$ git tag v1.0.0
3. Push the tag:
$ git push --tags
4. In Container Registry, click Build History to check that the build has been triggered. If not, verify the trigger was configured properly in the previous section.
Configuring your deployment pipelines
Now that our images are building automatically, we need to deploy them to the Kubernetes cluster.
We deploy to a scaled-down environment for integration testing. After the integration tests pass, we must manually approve the changes to deploy the code to production services.
Create the application
1. In the Spinnaker UI, click Actions, then click Create Application.
2. In the New Application dialog, enter the following fields:
Name: sample
Owner Email: [your email address]
3. Click Create.
Create service load balancers
To avoid having to enter the information manually in the UI, use the Kubernetes command-line interface to create load balancers for the services. Alternatively, we can perform this operation in the Spinnaker UI.
On the local machine where the code resides, run the following command from the sample-app root directory:
$ kubectl apply -f k8s/services
Create the deployment pipeline
Now we create the continuous delivery pipeline. The pipeline is configured to detect when a Docker image with a tag prefixed with “v” has arrived in your Container Registry.
1. Create a new pipeline named say “Deploy”.
2. Go to the Config page for the pipeline that we just created and click Pipeline Actions -> Edit as JSON.
3. Change the directory to the source code directory and update the current pipeline-deploy.json at path spinnaker/pipeline-deploy.json according to our needs.
$ exportPROJECT=$(gcloud info --format='value(config.project)')$ sed s/PROJECT/$PROJECT/g spinnaker/pipeline-deploy.json > spinnaker/updated-pipeline-deploy.json
4. Now in the JSON editor just copy the whole file spinnaker/updated-pipeline-deploy.json.
5. Click on Update Pipeline and we should have an updated pipeline config now.
6. In the Spinnaker UI, click Pipelines on the top navigation bar.
7. Click Configure in the Deploy pipeline.
8. The continuous delivery pipeline configuration appears in the UI:
Running the pipeline manually
The configuration we just created contains a trigger to start the pipeline when a new Git tag containing the prefix “v” is pushed. Now we test the pipeline by running it manually.
1. Return to the Pipelines page by clicking Pipelines.
2. Click Start Manual Execution.
3. Select the v1.0.0 tag from the Tag drop-down list, then click Run.
4. After the pipeline starts, click Details to see more information about the build’s progress. This section shows the status of the deployment pipeline and its steps. Steps in blue are currently running, green ones have completed successfully, and red ones have failed. Click a stage to see details about it.
5. After 3 to 5 minutes the integration test phase completes and the pipeline requires manual approval to continue the deployment.
6. Hover over the yellow “person” icon and click Continue.
7. Your rollout continues to the production frontend and backend deployments. It completes after a few minutes.
8. To view the app, click Load Balancers in the top right of the Spinnaker UI.
9. Scroll down the list of load balancers and click Default, under sample-frontend-prod.
10. Scroll down the details pane on the right and copy application’s IP address by clicking the clipboard button on the Ingress IP.
11. Paste the address into the browser to view the production version of the application.
12. We have now manually triggered the pipeline to build, test, and deploy your application.
Triggering the pipeline automatically via code changes
Now let’s test the pipeline end to end by making a code change, pushing a Git tag, and watching the pipeline run in response. By pushing a Git tag that starts with “v”, we trigger Container Builder to build a new Docker image and push it to Container Registry. Spinnaker detects that the new image tag begins with “v” and triggers a pipeline to deploy the image to canaries, run tests, and roll out the same image to all pods in the deployment.
1. Change the colour of the app from orange to blue:
$ sed -i 's/orange/blue/g' cmd/gke-info/common-service.go
4. Click Pipelines to watch the pipeline start to deploy the image.
5. Observe the canary deployments. When the deployment is paused, waiting to roll out to production, start refreshing the tab that contains our application. Nine of our backends are running the previous version of your application, while only one backend is running the canary. Now we should see the new, blue version of our application appear about every tenth time we refresh.
6. After testing completes, return to the Spinnaker tab and approve the deployment.
7. When the pipeline completes, application looks like the following screenshot. Note that the colour has changed to blue because of code change, and that the Version field now reads v1.0.1.
8. We have now successfully rolled out your application to your entire production environment!!!!!!
9. Optionally, we can roll back this change by reverting the previous commit. Rolling back adds a new tag (v1.0.2), and pushes the tag back through the same pipeline we used to deploy v1.0.1:
$ git revert v1.0.1$ git tag v1.0.2$ git push --tags
Now that you know how to get Spinnaker up and running in a development environment, start using it already. In this blog, we have done everything from installing a K8s cluster on GCP to deploying an End to End Pipeline just like that in a production environment. Hope you found it helpful.
