Mastering Data Consistency Across Microservices

Mastering Data Consistency Across Microservices

Microservices architecture is a software design pattern where an application is built as a collection of small, independent services, each responsible for a specific function.

These services communicate with each other using APIs (Application Programming Interfaces) and operate independently, allowing for greater flexibility, scalability, and ease of maintenance. Think of a food delivery app with the following services:

  • The order service manages customer orders.
  • The payment service handles transactions.
  • The restaurant service updates menu availability.
  • The delivery service assigns and tracks deliveries.

Each service operates independently, allowing teams to update or scale them separately.

Challenges of Data Consistency

Due to the separation of services, a major challenge with microservices is maintaining data consistency. Unlike monolithic systems, where all functionalities share a single database, microservices have independent databases, leading to challenges such as:

  • Duplicate or Lost Data – Without proper coordination, data may be duplicated across multiple services or lost due to partial updates.
  • Network Delays – Communication between services happens over the network, which introduces latency and failure risks.
  • Concurrency Issues – Simultaneous updates by different services can lead to conflicting data states.

How Data Inconsistency Arises

1. Eventual Consistency vs. Strong Consistency

Microservices rely on eventual consistency, meaning that data updates propagate over time instead of instantly. However, in critical operations like financial transactions, strong consistency is required.

2. Distributed Transactions

Unlike monolithic systems where transactions update multiple tables atomically, microservices need a way to ensure consistency across multiple independent databases.

3. Network and Service Failures

Failures in a service or network delays can cause stale or incomplete information in dependent services.

4. Concurrent Updates

When multiple instances of the same service update shared data simultaneously, race conditions can cause conflicts.

Strategies for Ensuring Data Consistency

1. Saga Pattern (Compensating Transactions)

A saga is a sequence of transactions where each step updates a different microservice, and compensating transactions roll back changes in case of failure. Two types of sagas:

  • Choreography (فن تصميم الرقصات) – Choreography provides to coordinate sagas with applying publish-subscribe principles. With choreography, each microservices run its own local transaction and publishes events to message broker system and that trigger local transactions in other microservices.
    Although Choreography way decouple direct dependency of microservices when managing transactions, if Saga Workflow steps increase, then it can become confusing and hard to manage transaction between saga microservices.

  • Orchestration – provides to coordinate sagas with a centralized controller microservice. This centralized controller microservice, orchestrate the saga workflow and invoke to execute local microservices transactions in sequentially.
    The orchestrator microservices execute saga transaction and manage them in centralized way and if one of the step is failed, then executes rollback steps with compensating transactions.

    Although orchestration way is good for complex workflows which includes lots of steps, it creates tight coupling — the orchestrator needs to know about all other services.
    This makes single point-of-failure with centralized controller microservices and need implementation of complex steps.

Example in a food delivery app:

  1. The order service creates an order.
  2. The payment service processes payment.
  3. The restaurant service confirms availability.
  4. If any step fails, previous actions are rolled back (e.g., refunding payment).

2. Two-Phase Commit (2PC)

Ensures all participating services confirm a transaction before it is committed. However, it's less common due to its blocking nature, which reduces scalability.

3. Event-Driven Architecture

Microservices communicate asynchronously by publishing and subscribing to events via a message broker (e.g., Kafka, RabbitMQ). This ensures eventual consistency.

4. Idempotency and Retries

Microservices should ensure that retrying a request does not create duplicate data. Using idempotency keys ensures repeated transactions have the same effect as a single one.

⚙️ How It Works

1. Client Sends a Request

The client includes a unique idempotency key in the HTTP header or request body:

POST /api/orders

Idempotency-Key: 4a1e6bcd-9f92-42b3-b6dd-7e8899934a91

2. Server Behavior:

  • Checks if it has already processed a request with this key.

    • ✅ If yes → Returns the same result as before (cached response).

    • ❌ If no → Processes the request, stores the result, and returns the response.

5. Optimistic Concurrency Control

Versioning (e.g., timestamps or version numbers) helps detect and resolve concurrent modifications before updating records.
🔹 Step 1: Create the Table Contains a "Timestamp Column"



🔹 Step 3: Reading the Data to Be Updated with the Rowversion (Timestamp) Within The Update Transaction

SELECT ProductID, ProductName, Quantity, RowVer
FROM Products
WHERE ProductID = 1;


🔹 Step 3: Update Data and Add the Rowversion (Timestamp) as a Condition

UPDATE Products
SET Quantity = 8
WHERE ProductID = 1
  AND RowVer = 0x00000000000007D3;

🔹 Step 4: Check for Success or Concurrency Conflict

IF @@ROWCOUNT = 0
    PRINT 'Update failed: concurrency conflict detected.';
ELSE
    PRINT 'Update succeeded.';

Conclusion

Maintaining data consistency in a microservices architecture is challenging but manageable with the right strategies. While monolithic applications rely on strong consistency through single database transactions, microservices embrace eventual consistency using sagas, event-driven communication, retries, and concurrency control mechanisms. By carefully designing data interactions and handling failures proactively, developers can build resilient and scalable microservices-based applications.

Comments

Post a Comment

Popular posts from this blog

Maxpooling vs minpooling vs average pooling

Image
Pooling is performed in neural networks to reduce variance and computation complexity. Many times, beginners blindly use a pooling method without knowing the reason for using it. Here is a comparison of three basic pooling methods that are widely used. The three types of pooling operations are: Max pooling: The maximum pixel value of the batch is selected. Min pooling: The minimum pixel value of the batch is selected. Average pooling: The average value of all the pixels in the batch is selected. The batch here means a group of pixels of size equal to the filter size which is decided based on the size of the image. In the following example, a filter of 9x9 is chosen. The output of the pooling method varies with the varying value of the filter size. The operations are illustrated through the following figures. Average, Max and Min pooling of size 9x9 applied on an image We cannot say that a particular pooling method is better over others generally. The choice of pooling operation is made...
Read more

Generative AI - Prompting with purpose: The RACE framework for data analysis

Image
  Prompting with purpose: The RACE framework for data analysis We've already established that the quality of your prompts directly influences the quality of your AI-generated outputs. A well-crafted prompt acts as a guiding light, leading the AI toward insightful and actionable results. But how can we consistently achieve this level of clarity and precision in our prompts? Let’s look into the RACE framework—an invaluable tool to elevate your prompts, ensuring they are clear, insightful, and evidence-based. This framework offers a structured approach to prompt engineering, helping you break down complex tasks into manageable steps and communicate your intentions effectively to the AI. This reading will assist you with developing a powerful methodology with generative AI for your data analysis tasks, leading to more accurate, relevant, and impactful results. What is the RACE framework? RACE is an acronym that stands for Research, Answer, Cite, and Expand. It serves as a valuable fram...
Read more

Best Practices for Storing and Loading JSON Objects from a Large SQL Server Table Using .NET Core

 Best Practices for Storing and Loading JSON Objects from a Large SQL Server Table Using .NET Core Introduction JSON (JavaScript Object Notation) is widely used for storing structured data in SQL Server. However, when dealing with large tables, inefficient handling of JSON data can lead to performance bottlenecks, high storage costs, and slow queries. In this article, we will cover best practices for saving and loading JSON objects from large SQL Server tables using .NET Core. We will focus on storage strategies, indexing, efficient querying, and optimized data retrieval using Entity Framework Core (EF Core) and Dapper. --- 1. Choosing the Right Storage Strategy SQL Server supports JSON natively, but choosing the correct storage method depends on your use case. Option 1: Store JSON as NVARCHAR(MAX) (Best for Flexibility & Simplicity) ✔ Ideal for semi-structured or frequently changing data. ✔ Suitable when you need to store the entire JSON object but rarely query individual fiel...
Read more