Event Sourcing

In traditional database design, applications store only the current state of their domain entities. When a user updates their billing address or deposits money, an UPDATE statement overwrites the existing row. While simple, this approach loses valuable context: we know what the current state is, but we have lost the history of how we got there. Event Sourcing fundamentally shifts this paradigm by storing every state change as an immutable, append-only sequence of events. The current state is never directly updated; instead, it is dynamically computed by replaying these events from the beginning of time.

1. Learning Objectives

• Differentiate between traditional CRUD persistence and Event Sourcing.
• Understand the concepts of an Event Store, Aggregate, and Replay.
• Learn how Snapshotting prevents performance degradation during state reconstruction.
• Analyze the mechanics of Upcasting and schema migration for evolving events.
• Evaluate the synergy between Event Sourcing and Command Query Responsibility Segregation (CQRS).
• Implement an event-sourced domain aggregate with snapshotting and Optimistic Concurrency Control in Java, Python, and C++.

2. Prerequisites

To get the most out of this lesson, you should be familiar with:

• Event-Driven Architecture: Understanding events, publishers, and consumers.
• Database Isolation levels: Concurrency concerns and lock limits.

3. Why This Topic Matters

In critical business systems (e.g., banking, healthcare, logistics), auditability is not just a feature; it is a legal requirement. In a traditional CRUD system, if a balance mismatch occurs, engineers must sift through log files to reconstruct what happened.

Event Sourcing natively solves this. Because every single transaction, change, and withdrawal is stored as an immutable event log, we have a perfect, bulletproof audit trail. Furthermore, Event Sourcing enables Time Travel: we can easily reconstruct the exact state of the system as it was on any given date, run historical analytics, and easily resolve concurrency conflicts.

4. Real-world Analogy

Consider a Double-Entry Bookkeeping Ledger:

Traditional CRUD (The Blackboard): A company writes its current net worth on a blackboard: $1,500. When they buy supplies for $200, someone erases $1,500 and writes $1,300. There is no record of where the money went, when it was spent, or why.

Event Sourcing (The Ledger Book): The company uses an ink ledger book. They write line items:
• "Dec 1: Capital injection +$1,000"
• "Dec 2: Sold goods +$500"
• "Dec 3: Paid supplies -$200"
No entry is ever erased. To calculate the current balance, they start at $0 and add up every line item: $0 + $1000 + $500 - $200 = $1300. This ledger is the source of truth.

5. Core Concepts

• Event Store: A specialized database designed specifically for appending and retrieving event streams. It only supports writes by appending new events at the end of a stream.
• Aggregate: A domain entity or boundary of related entities that encapsulates business logic and state (e.g., a BankAccount or ShoppingCart). Commands target aggregates, and aggregates emit events.
• Event Replay: The process of reading the sequence of events associated with an aggregate from the event store and applying them in order to reconstruct the aggregate's current state.
• Snapshot: A stored recording of an aggregate's state at a specific version. This is used to optimize replay times.
• Projection: A read-only representation of the system state built by subscribing to the event stream. Projections are optimized for fast queries.
• Upcasting: The process of intercepting old event formats stored in the event store and transforming them to the current schema before passing them to the application logic.

6. Visualizations

Traditional CRUD vs. Event Sourcing DB

Event Sourcing Flow (Commands, Events & Projections)

State Reconstruction with Snapshotting

Instead of replaying all events from version 0, we load the latest snapshot and replay only subsequent events:

7. How It Works Step-by-Step

1. Command Arrival: A client issues a command (e.g., WithdrawFunds).
2. Aggregate Reconstruction: The Command Handler requests the aggregate's event stream from the Event Store. The aggregate applies all retrieved events to build its current memory representation.
3. Validation Check: The aggregate checks if the command violates any business invariant (e.g., checking if withdrawal amount exceeds current balance).
4. Event Generation: If validation passes, the aggregate generates a new event (e.g., MoneyWithdrawn) and appends it to its internal list of uncommitted changes.
5. Commit to Store: The Command Handler writes the new event(s) to the Event Store using Optimistic Concurrency Control.
6. Propagation: The Event Store publishes the committed event to projection subscribers, which update read-only views for clients.

8. Internal Architecture

An Event-Sourced system consists of the following core components:

• Aggregate Root: The domain object that owns state variables, processes incoming commands, and encapsulates state mutation logic inside apply handlers.
• Append-Only Event Store: A specialized transaction log database indexing events by aggregateId and sequential version numbers.
• Snapshot Store: A cache database storing serialized aggregates at a specific historical point (e.g., every 100th event).
• Projectors (Projections): Background workers listening to the event feed and updating relational databases or search indexes optimized for application read queries.

9. Request Lifecycle

Let's walk through a fund withdrawal request:

• t0: User requests a $100 withdrawal for account ACC-999.
• t1: The request is sent to the Command Handler. The handler queries the EventStore for account ACC-999 and retrieves all events.
• t2: The handler instantiates an empty BankAccount object. It loops through all retrieved events, calling apply(event) on each one, resulting in a current state with a balance of $150 at version 3.
• t3: The aggregate checks the business invariant: $150 - $100 >= $0 (validation passes).
• t4: The aggregate generates a MoneyWithdrawn event at version 4.
• t5: The handler saves the MoneyWithdrawn event to the EventStore, specifying expectedVersion = 3. The Event Store verifies that the current version of the account is indeed 3. It appends the event and increments the stream version to 4.
• t6: The projector consumes the MoneyWithdrawn event and executes: UPDATE accounts SET balance = 50 WHERE id = 'ACC-999' in the read database.

10. Deep Dive

Snapshotting

As an entity ages, its event stream grows. If an account has been active for years and has 100,000 transactions, replaying all of them to check if a $10 withdrawal is valid would be extremely slow.

To prevent this performance degradation, we implement Snapshotting. Every $N$ events (e.g., every 100 events), the system takes a snapshot of the aggregate state and stores it in a Snapshot Store. When loading the aggregate, the system retrieves the latest snapshot, restores the aggregate state, and only retrieves and replays events from the Event Store that have a version greater than the snapshot version.

Event Schema Evolution & Upcasting

Software requirements evolve, and so do event schemas. If you change a field name in an event schema (e.g., changing userName to fullName), how do you process old events written with the old schema?

We solve this using Upcasters. An upcaster is a middleware component that runs when events are loaded from the Event Store. It intercepts old event versions and transforms their structure to match the latest schema version before they are processed by the application. This allows the event store to remain immutable, while the application can always work with the latest schema version.

Optimistic Concurrency Control (OCC)

Because event-sourced systems read state from a history log, two clients could read the same state version and attempt to modify it at the same time. For example, both User A and User B read version 3 (balance $100) and attempt to withdraw $80 simultaneously.

To prevent double-spending, the Event Store uses Optimistic Concurrency Control (OCC). Every write request must include the version number that the business decision was based on. The Event Store verifies that the current version in the store matches this version. If it does, the write succeeds. If the version has already changed, the Event Store rejects the write, and the application must retry by reloading the new state.

11. Production Examples

• EventStoreDB: A database built from the ground up for Event Sourcing. It supports high-performance appending, aggregate routing, and indexing out of the box.
• Axon Framework (Java): A framework designed for DDD, CQRS, and Event Sourcing. It handles command routing, event storage, and projection tracking, allowing developers to focus on business logic.

12. Advantages

• Audit Trail: The event stream provides a complete, immutable audit trail of every change that has occurred.
• Time Travel: The state of the system can be reconstructed as it was at any point in history.
• High-Performance Writes: Writes only append to the end of the log, avoiding complex index updates and database lock waits.
• Natural CQRS Fit: The event stream is a natural fit for building read models optimized for specific queries.

13. Limitations

• Query Complexity: Querying the event store directly for business operations is difficult; you must use projections.
• Eventual Consistency: Projections are updated asynchronously, meaning reads might be temporarily stale.
• Schema Migration: Evolving event schemas over time requires complex upcasting logic.
• Storage Growth: The event log only grows, requiring significant storage over time.

14. Trade-offs

• Snapshot Frequency: Snapshotting frequently (e.g., every 5 events) reduces loading times, but increases snapshot store writes and storage costs. Snapshotting infrequently (e.g., every 1,000 events) saves storage but increases aggregate loading times.
• Synchronous vs. Asynchronous Projections: Synchronous projections ensure immediate read consistency, but slow down writes. Asynchronous projections maximize write performance, but introduce eventual consistency.

15. Performance Considerations

• Optimize Snapshot Size: Serializing only the necessary state in snapshots reduces I/O overhead.
• Stream Indexing: Ensure the Event Store is indexed by aggregateId to allow fast retrievals.
• Caching Aggregates: Keep active aggregates in memory (e.g., in a Redis cache) to bypass reading from the Event Store for subsequent requests.

16. Failure Scenarios

• Optimistic Concurrency Collision Storms: When multiple clients write to the same aggregate simultaneously, they cause version conflicts, resulting in retries that can degrade performance.
  Mitigation: Redesign aggregate boundaries to be smaller and more isolated, or route requests to the aggregate through a single thread queue.
• Projection Replay Failure: If a bug in a projection causes it to fail, the projection falls behind, resulting in stale reads.
  Mitigation: Implement monitoring for projection lag, and design projections so they can be easily reset and rebuilt from the event log.

17. Best Practices

• Model Events around Business Intent: Name events after business concepts (e.g., OrderCheckedOut) rather than technical actions (e.g., OrderUpdated).
• Ensure Event Immutability: Once an event is written to the store, it must never be changed.
• Keep Aggregates Small: Small aggregates minimize concurrency conflicts and scale more effectively.

18. Common Mistakes

• Querying the Event Store Directly: Writing business reports or filters directly against the Event Store instead of using projections. This is extremely slow and inefficient.
• Treating Commands and Events Interchangeably: Committing commands to the Event Store instead of events. Commands represent intent, while events represent historical facts.

19. Implementation (Event Sourced Bank Account)

The code tabs below showcase a complete, robust simulation of an Event-Sourced Bank Account in Java, Python, and C++. It demonstrates state reconstruction, snapshotting, and Optimistic Concurrency Control (OCC) during conflicts.

20. Interview Questions

Easy

Q: Why does event sourcing store changes as events instead of updating the current state?

A: Storing changes as a chronological, immutable sequence of events provides a complete, audit-safe record of everything that has occurred. This makes debugging easier, allows historical queries, and prevents the loss of state context that occurs in traditional databases.

Medium

Q: What is upcasting, and why is it necessary in event-sourced systems?

A: Upcasting is the process of intercepting old event schemas retrieved from the Event Store and transforming them into the current format before they are loaded by the aggregate logic. This is necessary because event stores are append-only and immutable; existing events cannot be modified or migrated when schemas change.

Hard

Q: How do you handle GDPR "Right to Be Forgotten" delete requests in a strictly immutable append-only event store?

A: Since you cannot delete events from an immutable log, you can use Cryptographic Erasure (Crypto-Shredding). In this design, personal data within events is stored in an encrypted format. The decryption key for each user is stored in a separate key management service. When a user requests deletion, their key is deleted, rendering all their event payloads unreadable and effectively deleted. Another option is using metadata fields that point to external mutable stores, or running a data migration process that copies the stream while removing the deleted data.

21. Practice Exercises

• Easy: Extend the BankAccount aggregate implementation to emit a NameChanged event whenever the owner name is updated, and implement the corresponding apply method.
• Medium: Implement a simple upcaster method in the Event Store simulator that intercept old AccountCreated events (e.g. version 1) and adds a default currency field before passing them to the reader.
• Hard: Build a local mock in-memory projection table. Register it to the Event Store so that every time a new event is successfully saved, the projection table automatically recalculates and caches the user's total balance.

22. Challenge Problem

Problem Statement: Design a shopping cart system for a retail website. The cart must support adding items, removing items, changing item counts, and checking out. Because users often shop on unstable mobile connections, the system must support offline cart additions, merging them with server-side updates when the connection is restored.

Explain how you would design the event payloads and conflict-resolution rules using event sourcing to ensure item counts are resolved correctly when offline changes are submitted.

23. Summary

• Event Sourcing stores all state modifications as an immutable sequence of events rather than overwriting database records.
• State is reconstructed by replaying events from a base state or from the latest snapshot checkpoint.
• Upcasters handle schema updates dynamically, ensuring historical events can be loaded safely.
• Event Stores rely on Optimistic Concurrency Control (OCC) to protect against concurrent modification conflicts.

24. Cheat Sheet

25. Quiz

1. What is the single source of truth in an event-sourced application?

• The current state stored in a relational cache database.
• The central search registry logs.
• The append-only log of committed events. (Correct)
• The command processor memory.

Explanation: In Event Sourcing, the append-only event store is the absolute source of truth. All application state is derived from it.

2. Why do event-sourced systems use snapshots?

• To encrypt personal data stored in event logs.
• To backup database files to external storage.
• To optimize state reconstruction performance by avoiding replaying long event logs. (Correct)
• To prevent out-of-order execution errors.

Explanation: Snapshots save the state of an aggregate at a specific version, allowing the system to restore state up to that point quickly, replaying only subsequent events.

3. How does "Upcasting" help manage schema changes in Event Sourcing?

• It runs database migration scripts that overwrite old event tables.
• It intercepts old event schemas and transforms them to the current version before loading them. (Correct)
• It compresses older event payloads to save storage.
• It rejects old event formats.

Explanation: Upcasting acts as a middleware converter, updating old event payloads to match the latest schema version on-the-fly without modifying the immutable event store.

4. In Event Sourcing, what is an "Aggregate"?

• A collection of tables grouped for reporting.
• A domain entity or boundary of entities that encapsulates state and enforces business invariants. (Correct)
• A system tool that summarizes event data logs.
• The network channel where events are routed.

Explanation: An aggregate is a self-contained domain object that validates commands and generates state events while protecting business rules.

5. How does the Event Store detect concurrency conflicts?

• By lock-waiting every row during reads.
• By checking if the expected version sent with the write matches the current version in the store. (Correct)
• By rejecting updates if the client IP changes.
• By running a background verification script.

Explanation: The Event Store uses version-based Optimistic Concurrency Control (OCC) to verify that no writes have occurred since the client read the aggregate state.

6. What is a key disadvantage of Event Sourcing?

• Writes are slow because of complex updates.
• It is impossible to audit changes.
• Directly querying state for complex filters is difficult without projections. (Correct)
• Aggregates cannot enforce business invariants.

Explanation: Because data is stored as a sequence of events, running query filters requires building projections, which introduces read complexity.

7. Why are events stored in the past tense (e.g., MoneyDeposited)?

• Because they represent historical facts that have already occurred. (Correct)
• To distinguish them from projection databases.
• Because they are written by background threads.
• To follow SQL styling guides.

Explanation: Events represent historical state transitions that have already been validated and committed, making the past tense appropriate.

8. What is Crypto-Shredding used for in Event Sourcing?

• To speed up event serialization.
• To encrypt event payloads in transit.
• To comply with GDPR deletion requests by deleting the encryption key for a user's data. (Correct)
• To compress database files.

Explanation: Crypto-Shredding deletes the decryption key for encrypted event data, rendering the data unrecoverable to comply with deletion requests without changing the immutable event store.

9. What is a "Projection" in Event Sourcing?

• A prediction of future event volume.
• A read-only view of state built by listening to the event log, optimized for query performance. (Correct)
• A method of routing events between brokers.
• A validation step before writing events.

Explanation: Projections consume events to maintain up-to-date, read-optimized database models separate from the main Event Store.

10. What is a common mistake when designing event-sourced aggregates?

• Querying projection tables for business reports.
• Neglecting upcasting design early in system development. (Correct)
• Using UUIDs for event tracking.
• Keeping aggregates as small as possible.

Explanation: Neglecting upcasting design early leads to difficulty migrating and loading historical event data when schema definitions change.

26. Further Reading

• Implementing Domain-Driven Design by Vaughn Vernon (Chapter 12).
• Domain-Driven Design by Eric Evans.
• Greg Young's paper: "CQRS Documents and Event Sourcing".

27. Next Lesson Preview

In the next lesson, we will explore CQRS (Command Query Responsibility Segregation), learning how to separate the write and read paths of a system to optimize scalability and query flexibility.