The"Modern Data Stack" is NOT meeting the needs of modern businesses.
To become data-empowered
organizations require fast access to reliable data for multiple use cases.
To accomplish this, data must be extracted from various source systems, cleaned, consolidated, and curated into reliable data sets for business consumption. Large organizations have many use cases for data, requiring a 3-tier architecture...
Data is ingested and stored in its raw form to be used by data scientists for AI and machine learning as well as feed the subsequent layers.
Data is cleaned and consolidated into a single version of truth to deliver reliable insights to business analysts.
Data is curated into a subset of models relevant for business users from different domains (sales, marketing, finance, etc.).
We refer to this as the DATA ESTATE
Surprisingly, these architectures are still commonly built with large teams of experts, using stacks of tools, manual hand-coding, and handwritten documentation over several months or even years.