Mastering Data Consistency Across Microservices

Mastering Data Consistency Across Microservices

Mastering Data Consistency Across Microservices

Microservices architecture is a software design pattern where an application is built as a collection of small, independent services, each responsible for a specific function.

These services communicate with each other using APIs (Application Programming Interfaces) and operate independently, allowing for greater flexibility, scalability, and ease of maintenance. Think of a food delivery app with the following services:

  • The order service manages customer orders.
  • The payment service handles transactions.
  • The restaurant service updates menu availability.
  • The delivery service assigns and tracks deliveries.

Each service operates independently, allowing teams to update or scale them separately.

Challenges of Data Consistency

Due to the separation of services, a major challenge with microservices is maintaining data consistency. Unlike monolithic systems, where all functionalities share a single database, microservices have independent databases, leading to challenges such as:

  • Duplicate or Lost Data – Without proper coordination, data may be duplicated across multiple services or lost due to partial updates.
  • Network Delays – Communication between services happens over the network, which introduces latency and failure risks.
  • Concurrency Issues – Simultaneous updates by different services can lead to conflicting data states.

How Data Inconsistency Arises

1. Eventual Consistency vs. Strong Consistency

Microservices rely on eventual consistency, meaning that data updates propagate over time instead of instantly. However, in critical operations like financial transactions, strong consistency is required.

2. Distributed Transactions

Unlike monolithic systems where transactions update multiple tables atomically, microservices need a way to ensure consistency across multiple independent databases.

3. Network and Service Failures

Failures in a service or network delays can cause stale or incomplete information in dependent services.

4. Concurrent Updates

When multiple instances of the same service update shared data simultaneously, race conditions can cause conflicts.

Strategies for Ensuring Data Consistency

1. Saga Pattern (Compensating Transactions)

A saga is a sequence of transactions where each step updates a different microservice, and compensating transactions roll back changes in case of failure. Two types of sagas:

  • Choreography – Each service listens for events from other services and reacts accordingly.
  • Orchestration – A central service manages the transaction flow.

Example in a food delivery app:

  1. The order service creates an order.
  2. The payment service processes payment.
  3. The restaurant service confirms availability.
  4. If any step fails, previous actions are rolled back (e.g., refunding payment).

2. Two-Phase Commit (2PC)

Ensures all participating services confirm a transaction before it is committed. However, it's less common due to its blocking nature, which reduces scalability.

3. Event-Driven Architecture

Microservices communicate asynchronously by publishing and subscribing to events via a message broker (e.g., Kafka, RabbitMQ). This ensures eventual consistency.

4. Idempotency and Retries

Microservices should ensure that retrying a request does not create duplicate data. Using idempotency keys ensures repeated transactions have the same effect as a single one.

5. Optimistic Concurrency Control

Versioning (e.g., timestamps or version numbers) helps detect and resolve concurrent modifications before updating records.

Conclusion

Maintaining data consistency in a microservices architecture is challenging but manageable with the right strategies. While monolithic applications rely on strong consistency through single database transactions, microservices embrace eventual consistency using sagas, event-driven communication, retries, and concurrency control mechanisms. By carefully designing data interactions and handling failures proactively, developers can build resilient and scalable microservices-based applications.

Comments

Post a Comment

Popular posts from this blog

Maxpooling vs minpooling vs average pooling

Image
Pooling is performed in neural networks to reduce variance and computation complexity. Many times, beginners blindly use a pooling method without knowing the reason for using it. Here is a comparison of three basic pooling methods that are widely used. The three types of pooling operations are: Max pooling: The maximum pixel value of the batch is selected. Min pooling: The minimum pixel value of the batch is selected. Average pooling: The average value of all the pixels in the batch is selected. The batch here means a group of pixels of size equal to the filter size which is decided based on the size of the image. In the following example, a filter of 9x9 is chosen. The output of the pooling method varies with the varying value of the filter size. The operations are illustrated through the following figures. Average, Max and Min pooling of size 9x9 applied on an image We cannot say that a particular pooling method is better over others generally. The choice of pooling operation is made...
Read more

Understand the Softmax Function in Minutes

Image
Learning machine learning? Specifically trying out neural networks for deep learning? You likely have run into the  Softmax function, a wonderful  activation function  that turns numbers aka logits into probabilities that sum to one. Softmax function outputs a vector that represents the probability distributions of a list of potential outcomes.  It’s also a core element used in  deep learning   classification  tasks. We will help you understand the Softmax function in a beginner friendly manner by showing you exactly how it works — by coding your very own Softmax function in python. If you are implementing Softmax in Pytorch and you already know Pytorch well, scroll down to the Deep Dive section and grab the code. This article has gotten really popular: 5800+ claps. It is updated constantly. Latest update Jan 2020 added a TL;DR section for busy souls. Dec 2019 (Softmax with Numpy Scipy Pytorch functional. Visuals indicating the location of Softmax func...
Read more

Percentiles, Deciles, and Quartiles

Image
Percentiles Percentile: the value below which a percentage of data falls. Example: You are the fourth tallest person in a group of 20 80% of people are shorter than you: That means you are at the  80th percentile . If your height is 1.85m then "1.85m" is the 80th percentile height in that group. In Order Have the data  in order , so you know which values are above and below. To calculate percentiles of height: have the data in height order (sorted by height). To calculate percentiles of age: have the data in age order. And so on. Grouped Data When the data is grouped: Add up all percentages  below  the score, plus  half  the percentage  at  the score. Example: You Score a B! In the test 12% got D, 50% got C, 30% got B and 8% got A You got a B, so add up all the 12% that got D, all the 50% that got C, half of the 30% that got B, for a total percentile of 12% + 50% + 15% =  77% In other words you...
Read more