To meet the challenges of scaling systems in size, scope, and complexity, it is useful to look at new approaches and theories to analyze, design, deploy, and manage these systems. A Design Structure Matrix (DSM) is an approach that supports the management of complexity by focusing attention on the elements of complex systems and how they relate to each other. DSM‐based techniques have proven to be very valuable in understanding, designing, and optimizing product, organization, and process a...| sookocheff.com
While microservices have become the defacto architectural pattern for building modern systems, they come with their own set of challenges. Key among them is ensuring application data consistency when data is spread over multiple databases. This article discusses one solution to this problem, distributed sagas. Distributed sagas provide consistency guarantees for systems spanning multiple databases in the face of failure using a clear and manageable implementation pattern. A saga is highly rel...| sookocheff.com
Kafka is a messaging system. That’s it. So why all the hype? In reality messaging is a hugely important piece of infrastructure for moving data between systems. To see why, let’s look at a data pipeline without a messaging system. This system starts with Hadoop for storage and data processing. Hadoop isn’t very useful without data so the first stage in using Hadoop is getting data in. Bringing Data in to Hadoop So far, not a big deal. Unfortunately, in the real world data exists on many...| Kevin Sookocheff
