Part I of the Modern Data Stack: An Aspiring Engineering Student’s Perspective

Written by Tornike Avaliani – Beacon Intern

In an era where data is heralded as the new enterprise currency, organisations across the globe are in a relentless pursuit of leveraging their data assets to drive innovation and gain a competitive edge. At the heart of this data-driven revolution lies the modern data stack, a dynamic ecosystem engineered to Store, Manage, and Apply vast troves of data.

The modern data stack can be distilled into three fundamental phases: Store, Manage, Apply

These broad categories serve as the bedrock for all data-driven operations in an organisation. Throughout this trilogy of articles, I will explore each of the above.

My objective is to:

  • Offer context by tracing the technological evolution leading to the present state
  • Explore the current direction of these categories — how solutions are converging, which technologies are gaining prominence, and what organisations are seeking
  • Identify disruptors and examine the forces that will define the future landscape of these phases

Why am I writing these articles?

TLDR: my fascination with data has been a big part of my journey to becoming an engineer (just starting my third year!). For those doubting the immense power of data (are there any?!), just watch ‘The Great Hack’ on Netflix. It was during my summer internship with Beacon, over (lukewarm) beers with Maria, Zach, and Tom on a (hot) Friday afternoon that my enthusiasm for the transformational impact of data on enterprises landed me onto some hot but interesting water: “Great” they said “you should write a blog on the Modern Data Stack and look at how the tools that empower organisations to turn data into actions have evolved.” Served me right! And here I am….

So, in this first article, I will look at the state of storage solutions, which serve as the foundational linchpin of the modern data stack.

Journey

The journey begins with a significant milestone — the advent of cloud-native data warehousing, by Amazon Redshift in 2012. Redshift acted as the catalyst for the modern data stack as we recognize it today.

Subsequently, storage providers such as Snowflake and Databricks have risen to prominence. More recently, the landscape of data storage has not remained untouched by the hype and disruption brought about by the popularisation and proliferation of multi-modal AI — such as the infamous chatGPT.

Legacy (2000s — present)

Legacy storage solutions encompass the existing range of well-established/adopted technologies and practices at the disposal of organisations.

  • On-Premises Data Warehouse: Oracle, Microsoft SQL Server, and IBM Db2 were widely used as on-premises storage solutions for structured data, often in the form of data warehouses (more on this later). These solutions relied on physical hardware assets that required intensive maintenance and proprietary software tools, locking clients in single-vendor solutions with very limited adaptability. Scaling up data capabilities meant significant upfront investments and system downtime for hardware upgrades.

The popularity of on-premise solutions waned in the 2010s but while it once appeared that the IT landscape was undergoing a mass exodus to cloud infrastructure, there’s now a resurgence in on-premises database solutions. Heightened concerns about data privacy and security are leading organisations to incorporate on-premises options into hybrid and multi-cloud architectures.

  • Cloud Native Data Warehouse: In 2012, Amazon Redshift revolutionised data warehousing (and thereby storage), offering cloud-native solutions with minimal maintenance. This met the growing demand for scalable, high-performance cloud alternatives, delivering up to 10x faster speeds compared to on-premises options, all with cost-effective pay-as-you-go pricing. As the decade unravelled, continued progress saw cloud-native data warehousing become an increasingly attractive proposition.
  • Data Warehouse: A centralised repository for storing, analysis and reporting on large volumes of structured data from disparate sources, such as your CRM, open social media data, and internal transactional data. Data warehousing introduced the concept of single-source of truth — providing a comprehensive view of all the data at the disposal of an organisation.

Modern Times (2010s — present)

Modern storage solutions gained adoption enthusiastically:

  • Data lake: A flexible, cost-effective repository for all types of data — structured, semi-structured, and unstructured. Ideal for big data and real-time analytics, as data is stored in raw form until needed — they are cost effective (=cheap) and easy to scale. Organisations value the data lake for its versatility, ML & AI support, and real-time analysis capability.
  • Data lakehouse: A hybrid data storage solution combining the benefits of data lakes and data warehouses (hence the name!). It provides the flexibility of data lakes for ingesting raw data with the structured, query-optimised capabilities of the data warehouses. This hybrid approach allows organisations to efficiently manage and analyse both raw and structured data within a single unified platform — promising a ‘best-of-both-worlds’ solution.

Now that we’re caught up with the present, let’s look at where we are headed with storage.

What’s Next?

We are witnessing an explosion in unstructured data. IDC says that by 2025 the volume of unstructured data will reach 175 zettabytes (175 billion terabytes) and Gartner reports that 80–90% of new enterprise data is unstructured. The impact of ChatGPT should not be overlooked — it not only demonstrated but also popularised multi-modal AI. These models are trained on diverse data types, including unstructured data like images, videos, and audio. Naturally, it follows that organisations must have robust techniques for handling unstructured data.

However, unstructured data presents difficulties in storage and query within legacy solutions.

Vector Databases

Enter vector databases — a highly specialised solution that is gaining traction. Vector databases address the intricacies of managing unstructured data by embedding data into high-dimensional vectors, enabling semantic searches through a plethora of algorithms, one of the best known being Approximate Nearest Neighbour (ANN).

Recently a heated debate has been struck up about whether to create specialist vector database solutions or to integrate vector search and vector embedding capabilities into existing legacy solutions. This has been provoked by existing database providers like ClickHouse, PostgreSQL, and Redis incorporating vector search capability. The fundamental assumption is that clients’ main needs will continue to be processing traditional non-vector data (i.e. structured data) and highly advanced vector search will not constitute a necessity for the majority of clients.

I believe this strategy may be difficult to execute for two main reasons: firstly, using non-specialized solutions is likely to result in higher costs, due to the lack of optimization, leading to increased consumption of compute and storage resources. In today’s market, where CIOs are dealing with budget constraints, this approach won’t win. Secondly, organisations won’t settle for treating unstructured data as second-class citizens. Unstructured data will be so vital to business intelligence that organisations will seek high-performance, flexible, and scalable solutions. This will come only from purpose-built solutions.

Graph Databases

Graph databases are specialised data storage systems designed to represent and manage complex relationships between data points. They use a structure of nodes and edges to store data and navigate relationships. In graph databases, relationships are undoubtedly the primary focus. They offer a higher-order exploration of connections between data points, unlike in traditional databases where understanding relationships often necessitates thorough analysis.

Graph databases are by no means new. Since the mid-2010s their value has been well harnessed by social media platforms, who used them to uncover insights through relationships between users and to power recommendation systems. Despite this, many organisations did not see value in graph databases.

But a lot has changed since then, most significantly perhaps is the development and maintenance of a standard Graph Querying Language (GQL) by the same group that maintains the SQL standard. This growing maturity in graph database technology will make it much easier for organisations to adopt.

Furthermore, graph database use-cases have expanded significantly. To highlight one area in particular — master data management, where a graph structure seems perfectly suited for connecting highly distributed data. Such as that in the hybrid/multi-cloud architecture of modern organisations. This is by no means the only use-case organisations will find useful. There has been significant progress in data lineage, explainable AI, and risk modelling.

Image Source: Jim Webber — Neo4j, ‘The Top Ten Use Cases of Graph Database Technology’, 2021 Accessed Via: https://go.neo4j.com/rs/710-RRC-335/images/Neo4j-Top-10-Use-Cases-EN-US-2.pdf?_ga=2.78736851.670231025.1694447979-187436064.1694447979

More recently, graph database technology has had a surge of interest, with Neo4j leading the way as an established market player. In 2021 they became the first specialist graph database solution to be ranked in the top 20 on DB-Engines, which ranks database providers by their popularity. In that same year, they raised $325 million in VC funding and crossed $100 million in revenue.

According to Gartner’s projections, the market for graph technologies is expected to rise to $3.2 billion by 2025 from $0.8 billion in 2019, reflecting an impressive Compound Annual Growth Rate (CAGR) of 28.1%. What’s more, by 2025, graph technologies are anticipated to underpin a staggering 80% of data and analytics innovations.

Take-Away

The data storage landscape is quietly evolving, driven by the rise of vector and graph databases. These innovations promise to reshape data management and analytics, potentially enabling and accelerating the adoption of generative AI. With scalable, cost-effective solutions, businesses will be embracing these technologies and the required organisational changes to stay competitive in our data-driven world.

Stay tuned for Part 2 where I will dive into the tools that “Manage” data in the modern data stack!

← Back to Blogs
The Private Equity European Emerging Managers SummitMeetFounders VC Night – November 23rd 2023