Event Sourcing: Choosing the Right Approach for Scalability and Consistency


Event Sourcing: Choosing the Right Approach

Event Sourcing: Choosing the Right Approach for Scalability and Consistency

Introduction

Event Sourcing (ES) is a powerful architectural pattern where system state changes are captured as a sequence of immutable events rather than directly modifying database records. This approach offers benefits such as auditability, replayability, and improved system resilience. However, implementing ES effectively depends on business requirements, scalability needs, and consistency considerations.

Direct Database Writes: When ACID Works (and When It Doesn't)

A simple way to implement ES is to write events directly to a database, ensuring transactional consistency (ACID compliance). This approach is feasible if:

  • Events occur infrequently.
  • Data must be strongly consistent.
  • Scalability is not a primary concern.

Why ACID Doesn't Scale for ES

While ACID transactions ensure strong consistency, they introduce bottlenecks in large-scale systems due to:

  • Locking overhead: Multiple concurrent writes slow down performance.
  • Single point of failure: A centralized database limits scalability.
  • Difficulty in event replaying: Traditional relational databases are not optimized for storing and querying event logs at scale.

Message Queue-Based Event Sourcing: The BASE Approach

A more scalable alternative is to write events asynchronously to a message queue (e.g., Kafka, RabbitMQ) and process them using worker services. This follows the BASE (Basically Available, Soft state, Eventually consistent) model, which prioritizes availability and scalability over immediate consistency.

How BASE Works in ES

  1. The application emits an event.
  2. The event is written to a message queue (Kafka, RabbitMQ, etc.).
  3. A consumer (worker service) processes the event asynchronously.
  4. The processed event updates a database or projection store.

Advantages of BASE in Large-Scale ES Solutions

  • High scalability: Message queues decouple event generation from processing.
  • Fault tolerance: Events are stored reliably and can be replayed if needed.
  • Improved performance: Asynchronous processing prevents database congestion.

Hybrid Approach: Combining ACID and BASE

For systems requiring a balance between consistency and scalability, a hybrid model can be used:

  1. Write the current state synchronously to an OLTP database (ACID-compliant).
  2. Emit the event asynchronously to a message queue (BASE model).
  3. Store events in an ES database (e.g., Apache Cassandra, event logs in PostgreSQL).

Example Use Case

A banking application could use this approach:

  • The current balance is updated in a transactional database (ACID).
  • The transaction event is published to a queue for auditing and analytics (BASE).

Choosing the Right Event Sourcing Approach

Approach Best for Trade-offs
Direct Database Writes (ACID) Small-scale systems, financial transactions Doesn't scale well, can introduce locking issues
Message Queue-Based (BASE) Large-scale distributed systems, event-driven architectures Eventual consistency, increased complexity
Hybrid Model (ACID + BASE) Systems needing both consistency and scalability Requires managing dual storage mechanisms

Kafka: The Best Choice for Large-Scale Event Sourcing

For large-scale ES implementations, Apache Kafka is a leading choice due to:

  • High throughput: Can handle millions of events per second.
  • Durability: Events are stored persistently and can be replayed.
  • Scalability: Supports partitioning and distributed processing.

However, Kafka is expensive to run and requires expertise to configure and maintain properly. It is best suited for enterprises handling massive event streams, such as e-commerce platforms, fintech applications, and IoT systems.

Conclusion

Event Sourcing is a powerful architectural pattern, but it should be used only when necessary due to its complexity and cost. The choice between ACID, BASE, or a hybrid approach depends on business requirements:

  • Use ACID if consistency is the top priority.
  • Use BASE for large-scale, event-driven architectures.
  • Use a hybrid model when both are needed.

For large-scale ES, Kafka is a top choice, but its cost and complexity should be considered. Ultimately, ES is a great technology—but only if your solution truly needs it.

Comments

Post a Comment

Popular posts from this blog

Maxpooling vs minpooling vs average pooling

Image
Pooling is performed in neural networks to reduce variance and computation complexity. Many times, beginners blindly use a pooling method without knowing the reason for using it. Here is a comparison of three basic pooling methods that are widely used. The three types of pooling operations are: Max pooling: The maximum pixel value of the batch is selected. Min pooling: The minimum pixel value of the batch is selected. Average pooling: The average value of all the pixels in the batch is selected. The batch here means a group of pixels of size equal to the filter size which is decided based on the size of the image. In the following example, a filter of 9x9 is chosen. The output of the pooling method varies with the varying value of the filter size. The operations are illustrated through the following figures. Average, Max and Min pooling of size 9x9 applied on an image We cannot say that a particular pooling method is better over others generally. The choice of pooling operation is made...
Read more

Understand the Softmax Function in Minutes

Image
Learning machine learning? Specifically trying out neural networks for deep learning? You likely have run into the  Softmax function, a wonderful  activation function  that turns numbers aka logits into probabilities that sum to one. Softmax function outputs a vector that represents the probability distributions of a list of potential outcomes.  It’s also a core element used in  deep learning   classification  tasks. We will help you understand the Softmax function in a beginner friendly manner by showing you exactly how it works — by coding your very own Softmax function in python. If you are implementing Softmax in Pytorch and you already know Pytorch well, scroll down to the Deep Dive section and grab the code. This article has gotten really popular: 5800+ claps. It is updated constantly. Latest update Jan 2020 added a TL;DR section for busy souls. Dec 2019 (Softmax with Numpy Scipy Pytorch functional. Visuals indicating the location of Softmax func...
Read more

Percentiles, Deciles, and Quartiles

Image
Percentiles Percentile: the value below which a percentage of data falls. Example: You are the fourth tallest person in a group of 20 80% of people are shorter than you: That means you are at the  80th percentile . If your height is 1.85m then "1.85m" is the 80th percentile height in that group. In Order Have the data  in order , so you know which values are above and below. To calculate percentiles of height: have the data in height order (sorted by height). To calculate percentiles of age: have the data in age order. And so on. Grouped Data When the data is grouped: Add up all percentages  below  the score, plus  half  the percentage  at  the score. Example: You Score a B! In the test 12% got D, 50% got C, 30% got B and 8% got A You got a B, so add up all the 12% that got D, all the 50% that got C, half of the 30% that got B, for a total percentile of 12% + 50% + 15% =  77% In other words you...
Read more