Data Mesh: Moving Away From Monolithic & Centralised Data Lakes
Data mesh is a highly decentralised data architecture to solve challenges such as lack of ownership of data, lack of quality data and removing bottlenecks to encourage organisational scaling.
The goal of data mesh is to treat data as a product, with each source having a data product owner, who could ideally be part of the cross-functional team of data engineers. Despite having a separate owner, the data should be domain-focused and should have an autonomous offering that leads to a domain-driven distributed architecture.
Read full article
Weighing Open Source’s Worth for the Future of Big Data
“If you’re trying to overcome a technology like relational databases, which have been developed over decades and had gestation from every major university in the world that does computer science research, it takes a long time to climb that hill,” Kreps says. “What’s very different for us is there hasn’t really been this incredibly well-developed infrastructure layer in the space we’re entering. We get to kind of make it up as we go along, which is a huge advantage. “
This perhaps is the reason why — despite the availability of MySQL, MariaDB, and PostgreSQL RDBMs, the advent of modern NoSQL and NewSQL solutions, and scalable Hadoop and object-storage alternatives — proprietary RDBMs continue to drive the lion’s share of enterprise spending in the data management space.
Read full article.
Phi Beta Iota: The time has come for a clean-sheet fresh start. The Internet was designed for machine to machine communication, it was never designed for humans or content. We process less than 1% of the Big Data we have in hand and that in turn is less than 1% of what is known. A post-Amazon post-Google Internet will be distributed and encrypted, including the 50% of humanity not on the Internet today, and enable paragraph level linking and weighting.