DDD+CQRS+EventSourcing how to make sure projections are up to date?

Question

We develop application that uses DDD,CQRS+Eventsourcing.

We have an UserAggregate. This aggregate uses UserMailIndex. UserMailIndex - is a projection, it is just a list of unique emails. Each time when userCreated event pushed to event store, the projector reads this event, and adds mail to UserMailIndex.

When UserAggregate receives command CreateUser{Mail = ...}, aggregate looks for CreateUser.Mail in UserMailIndex to make sure mail was not used before. If it was not used, UserAggregate produces eventUserCreated.

public class UserAggregate
{
    private UserMailIndex _mailIndex;
    UserAggregate(UserMailIndex mailIndex)
    {
        _mailIndex = mailIndex;
    }
    ...
    public void CreateUser(string userMail)
    {
        if(_mailIndex.Contains(userMail))
            return;
        Apply(UserCreatedEvent(userMail));
    }
}
public class UserMailIndex
{
    public void When(UserCreatedEvent evnt)
    {
        index.Add(evnt.Mail);
    }
}

How to make sure UserMailIndex is up to date? What if projector has no time to update UserMailIndex projection. If I understand correctly such architecture is based on eventual consistency.

What are the ways to make sure the projection is up to date? Or how to decide if the gap is “small enough”?

Answer 1

This issue extends to any cached data scenario including Projections, Materialised Views and all Index scenarios. When we are trying to prevent duplications of data and you have a single store that is the source of truth then the only Critical pathway is the actual process that inserts / creates the data, this should also be the only pathway that raises create events.

The first problem is that your pattern describes an event source that does not actually "know" if the event it describes has actually taken place. You are deliberately adding additional uncertainty to your model that complicates things. If you raise the UserCreated event first, and the actual create process failed, then raising the event at all has actually broken the chain of eventual-consistency unless some other process tries and succeeds to create the user.

So start there, the actual CreateUser process (not the message) needs to ensure that the mailbox is unique. This might be a SQL or other backend that can ensure atomicity and uniqueness, or it might be an external system that returns a failure result if a duplicate exists.

• This actual process, the one that creates the record that MUST be unique, this is the process that should raise the UserCreated event.

To support eventual-consistency we not only allow, but we expect these failures to occur because we know that our projections or indexes will always be out of date to some degree. A cache is always a copy of the state at a previous point in time, it is never a snapshot of "now"

So we should expect that CreateUser command will be called multiple times for the same user, but it should only raise UserCreated event once, the first time that it actually succeeds to create the user.

How to make sure UserMailIndex is up to date? What if projector has no time to update UserMailIndex projection.

You don't try, we that's not true, the system efficiency depends on the projection being kept up to date. But it is important to switch your thinking from "might possibly fail" to "is expected to sometimes fail". The point of caching is to reduce round trips to the underlying store but not to replace or prevent them entirely.

• Failure to create a user due to duplication is not an exception, it should be is an expected pathway in your logic.

So when the CreateUser process fails, due to the user already existing, to support eventual consistency you might choose to ignore this error, or change the logic to update the user with the new message detail, but importantly you wouldn't raise the UserCreated event.

Or how to decide if the gap is “small enough”?

How to decide is your IP and dependant on your application domain, but the simple model of

1. Try Create
2. If exist then Update

works for any size gap. It can mean some redundant round trips to the store for very small gaps, but that is the general trade off of eventual-consistency, there is no acceptable gap size, we must always update to ensure consistency unless the rest of your solution allows you to make an assumption that a certain gap size is tolerated.

Answer 2

Create an event stream for email addresses. When a user registers try append to the stream with the event id being email address and a prefix to make it unique. This will fail if the email address exists as event id must be unique