CQRS Journey Guide
CQRS_Journey_Guide
User Manual: Pdf
Open the PDF directly: View PDF .
Page Count: 376
Download | |
Open PDF In Browser | View PDF |
Exploring CQRS and Event Sourcing A journey into high scalability, availability, and maintainability with Windows Azure Exploring CQRS and Event Sourcing A journey into high scalability, availability, and maintainability with Windows Azure Dominic Betts Julián Domínguez Grigori Melnik Fernando Simonazzi Mani Subramanian 978-1-62114-016-0 This document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. You may modify this document for your internal, reference purposes © 2012 Microsoft. All rights reserved. Microsoft, MSDN, SQL Azure, SQL Server, Visual Studio, Windows, and Windows Azure are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners. Contents What other readers are saying about this guide Foreword by Greg Young Preface Why we created this guidance now How is this guidance structured? A CQRS journey CQRS reference Tales from the trenches A CQRS journey CQRS reference Tales from the trenches Selecting the domain for the RI Arrow legend Where to go for more information The Crew Journey 1: Our Domain: Conference Management System The Contoso Corporation Who is coming with us on the journey? The Contoso Conference Management System Overview of the system Selling seats for a conference Creating a conference Nonfunctional requirements Scalability Flexibility Beginning the journey More information xvii xxi xxiii xxiii xxiii xxiv xxv xxv xxv xxvi xxvi xxvi xxvii xxviii xxix 1 1 2 3 3 4 4 4 4 5 5 5 v vi Journey 2: Decomposing the Domain Definitions used in this chapter Bounded contexts in the conference management system Bounded contexts not included The context map for the Contoso Conference Management System Why did we choose these bounded contexts? More information Journey 3: Orders and Registrations Bounded Context A description of the bounded context Working definitions for this chapter Domain definitions (ubiquitous language) Requirements for creating orders Architecture Patterns and concepts Validation Transaction boundaries Concurrency Aggregates and aggregate roots Implementation details High-level architecture 1. Querying the read model 2. Issuing commands 3. Handling commands 4. Initiating business logic in the domain 5. Persisting the changes 6. Polling the read model Inside the write model Aggregates Aggregates and process managers Infrastructure Using the Windows Azure Service Bus Delivering a command to a single recipient Why have separate CommandBus and EventBus classes? How scalable is this approach? How robust is this approach? What is the granularity of a topic and a subscription? How are commands and events serialized? Impact on testing Summary More information 7 7 8 9 10 11 11 13 13 14 15 17 18 18 23 24 25 25 25 26 27 28 28 29 29 29 31 31 34 40 42 44 48 48 48 48 49 49 52 52 vii Journey 4: Extending and Enhancing the Orders and Registrations Bounded Context Changes to the bounded context Working definitions for this chapter User stories Implement a login using a record locator Tell the registrant how much time remains to complete an order Enable a registrant to create an order that includes multiple seat types Architecture Patterns and concepts Record locators Querying the read side Storing denormalized views in a database Making information about partially fulfilled orders available to the read side CQRS command validation The countdown timer and the read model Implementation details The order access code record locator The countdown timer Using ASP.NET MVC validation for commands Pushing changes to the read side Querying the read side Refactoring the SeatsAvailability aggregate The AddSeats method Impact on testing Acceptance tests and the domain expert Defining acceptance tests using SpecFlow features Making the tests executable Using tests to help developers understand message flows A journey into code comprehension: A tale of pain, relief, and learning Testing is important Domain tests The other side of the coin Summary More information Journey 5: Preparing for the V1 Release The Contoso Conference Management System V1 release Working definitions for this chapter User stories Ubiquitous language definitions Conference Management bounded context user stories Ordering and Registration bounded context user stories 53 53 53 54 54 55 55 55 56 56 56 57 60 61 62 62 63 64 66 69 72 73 74 74 74 74 76 81 83 83 84 86 90 90 91 91 91 92 92 92 92 viii Architecture Conference Management bounded context Patterns and concepts Event sourcing Identifying aggregates Task-based UI CRUD Integration between bounded contexts Pushing changes from the Conference Management bounded context Pushing changes to the Conference Management bounded context Choosing when to update the read-side data Distributed transactions and event sourcing Autonomy versus authority Favoring autonomy Favoring authority Choosing between autonomy and authority Approaches to implementing the read side Eventual consistency Implementation details The Conference Management bounded context Integration with the Orders and Registration bounded context The Payments bounded context Integration with online payment services, eventual consistency, and command validation Event sourcing Raising events when the state of an aggregate changes Persisting events to the event store Replaying events to rebuild state Issues with the simple event store implementation Windows Azure table storage-based event store Calculating totals Impact on testing Timing issues Involving the domain expert Summary More information Journey 6: Versioning Our System Working definitions for this chapter User stories No down time upgrade Display remaining seat quantities Handle zero-cost seats Architecture 93 97 97 97 98 99 101 101 102 104 105 105 105 106 106 106 107 107 108 108 108 109 111 113 113 117 118 120 120 122 123 123 123 124 124 125 125 126 126 126 126 126 ix Patterns and concepts 127 Handling changes to events definitions 128 Mapping/filtering event messages in the infrastructure 128 Handling multiple message versions in the aggregates 128 Honoring message idempotency 128 Avoid processing events multiple times 129 Persisting integration events 131 Message ordering 133 Implementation details 133 Adding support for zero-cost orders 134 Changes to the RegistrationProcessManager class 134 Changes to the UI 134 Data migration 136 Displaying remaining seats in the UI 138 Adding information about remaining seat quantities to the read model 138 Modifying the UI to display remaining seat quantities 140 Data migration 140 De-duplicating command messages 141 Guaranteeing message ordering 142 Persisting events from the Conference Management bounded context 146 Adding additional metadata to the messages 146 Capturing and persisting messages to the message log 146 Data migration 148 Migrating from V1 to V2 150 Generating past log messages for the Conference Management bounded context 151 Migrating the event sourcing events 151 Rebuilding the read models 151 Impact on testing 151 SpecFlow revisited 152 Discovering a bug during the migration 155 Summary 155 More information 155 Journey 7: Adding Resilience and Optimizing Performance Working definitions for this chapter Architecture Adding resilience Making the system resilient when an event is reprocessed Ensuring that commands are always sent Optimizing performance UI flow before optimization Optimizing the UI UI optimization 1 UI optimization 2 157 157 158 159 161 161 162 162 163 164 165 x Optimizing the infrastructure Sending and receiving commands and events asynchronously Optimizing command processing Using snapshots with event sourcing Publishing events in parallel Filtering messages in subscriptions Creating a dedicated receiver for the SeatsAvailability aggregate Caching conference information Partitioning the Service Bus Other optimizations Further changes that would improve performance Further changes that would enhance scalability No down-time migration Rebuilding the read models Implementation details Hardening the RegistrationProcessManager class Detecting out-of-order SeatsReserved events Detecting duplicate OrderPlaced events Creating a pseudo transaction when the RegistrationProcessManager class saves its state and sends a command Optimizing the UI flow Receiving, completing, and sending messages asynchronously Receiving messages asynchronously Completing messages asynchronously Sending messages asynchronously Handling commands synchronously and in-process Implementing snapshots with the memento pattern Publishing events in parallel Filtering messages in subscriptions Creating a dedicated SessionSubscriptionReceiver instance for the SeatsAvailability aggregate Caching read-model data Using multiple topics to partition the service bus Other optimizing and hardening changes Sequential GUIDs Asynchronous ASP.NET MVC controllers. Using prefetch with Windows Azure Service Bus Accepting multiple sessions in parallel Adding an optimistic concurrency check Adding a time-to-live value to the MakeSeatReservation command Reducing the number of round-trips to the database 165 165 166 166 167 167 167 167 168 168 169 171 172 173 174 174 175 178 178 181 186 186 186 186 186 189 191 192 193 194 195 196 196 198 198 199 199 199 199 xi Impact on testing Integration tests User interface tests Summary More information 200 200 200 200 200 Journey 8: Epilogue: Lessons Learned What did we learn? Performance matters Implementing a message-driven system is far from simple The cloud has challenges CQRS is different Event sourcing and transaction logging Involving the domain expert When to use CQRS What would we do differently if we started over? Start with a solid infrastructure for messaging and persistence Leverage the capabilities of the infrastructure more Adopt a more systematic approach to implementing process managers Partition the application differently Organize the development team differently Evaluate how appropriate the domain and the bounded contexts are for the CQRS pattern Plan for performance Think about the UI differently Explore some additional benefits of event sourcing Explore the issues associated with integrating bounded contexts More information 201 201 201 202 203 204 205 206 206 207 Reference 1: CQRS in Context What is domain-driven design? Domain-driven design: concepts and terminology Domain model Ubiquitous language Entities, value objects, and services Aggregates and aggregate roots Bounded contexts Anti-corruption layers Context maps Bounded contexts and multiple architectures Bounded contexts and multiple development teams Maintaining multiple bounded contexts CQRS and DDD More information 211 212 212 213 213 214 215 215 217 218 218 219 220 220 221 207 207 208 208 208 208 208 209 209 210 210 xii Reference 2: Introducing the Command Query Responsibility Segregation Pattern What is CQRS? Read and write sides CQRS and domain-driven design Introducing commands, events, and messages Why should I use CQRS? Scalability Reduced complexity Flexibility Focus on the business Facilitates building task-based UIs Barriers to adopting the CQRS pattern When should I use CQRS? Collaborative domains Stale data Moving to the cloud When should I avoid CQRS? Summary More information 223 223 225 227 228 230 230 231 231 232 232 232 232 233 233 234 234 234 234 Reference 3: Introducing Event Sourcing What is event sourcing? Comparing using an ORM layer with event sourcing Why should I use event sourcing? Event sourcing concerns CQRS/ES Standalone event sourcing Event stores Basic requirements Underlying storage Performance, scalability, and consistency More information 235 236 236 240 242 243 245 245 245 245 245 246 Reference 4: A CQRS and ES Deep Dive Introduction Read models and write models Commands and data transfer objects Domain-driven design (DDD) and aggregates Data and normalization Events and event sourcing Eventual consistency Defining aggregates in the domain model Aggregates and object-relational mapping layers Aggregates and event sourcing 247 247 247 247 248 248 248 248 249 249 250 xiii Commands and command handlers Commands Example code Command handlers Commands and optimistic concurrency Events and event handlers Events and intent How to model intent Events Sample Code Event handlers Sample code Embracing eventual consistency Eventual consistency and CQRS Optimizing the read-side Optimizing the write side Concurrency and aggregates Messaging and CQRS Messaging considerations Duplicate messages Lost messages Out-of-order messages Unprocessed messages Event versioning Redundant events New event types Changing existing event definitions Task-based UIs Taking advantage of Windows Azure Scaling out using multiple role instances Implementing an event store using Windows Azure table storage Persisting events Retrieving events Publishing events Implementing a messaging infrastructure using the Windows Azure Service Bus A word of warning More information 252 253 253 254 256 256 256 258 259 259 260 260 261 263 266 267 267 268 268 268 269 269 269 269 270 270 270 271 272 273 Reference 5: Communicating Between Bounded Contexts Introduction Context maps The anti-corruption layer 281 281 281 281 273 274 275 276 278 279 279 xiv Integration with legacy systems Reading the database Generating events from the database Modifying the legacy systems Implications for event sourcing More information 282 282 282 282 282 283 Reference 6: A Saga on Sagas Clarifying the terminology Process Manager Messages and CQRS What is a process manager? When should I use a process manager? When should I not use a process manager? Sagas and CQRS More information 285 285 286 286 286 290 290 290 290 Reference 7: Technologies Used in the Reference Implementation Windows Azure Service Bus Queues Topics and Subscriptions Useful API features Reading messages Sending messages Expiring messages Delayed message processing Serializing messages Further information Unity Application Block Further information More information 291 291 292 293 294 294 294 294 294 295 295 296 296 296 Tales from the Trenches 297 Twilio Product overview Lessons learned Separating reads and writes Designing for high availability Idempotency No-downtime deployments Performance References More information 297 297 297 297 297 298 298 298 299 299 xv Tales from the Trenches: Lokad Hub Project overview Lessons learned Benefits of DDD Reducing dependencies Using sagas Testing and documentation Migration to ES Using projections Event sourcing Infrastructure References More information 300 300 300 301 301 301 301 301 301 301 302 302 302 Tales from the Trenches: DDD/CQRS for large financial company Project overview Lessons learned Query performance Commands Working with legacy databases Using an Inversion of Control (IoC) container Key lessons learned More information 303 303 304 304 304 304 304 305 305 Tales from the Trenches: Digital Marketing Single Responsibility of Objects More information 306 309 309 Tales from the Trenches: TOPAZ Technologies What did we hope to accomplish by using CQRS/ES? What were the biggest challenges and how did we overcome them? What were the most important lessons learned? With hindsight, what would we have done differently? Further information More information 310 310 Tales from the Trenches: eMoney Nexus eMoney Nexus: Some CQRS lessons About eMoney & the Nexus System overview The evolution of the system Lessons learned Making it better 312 312 312 313 314 320 321 310 311 311 311 311 xvi Appendix 1: Release Notes System evolution Building and running the sample code (RI) Prerequisites Obtaining the code Creating the databases SQL Express Database Windows Azure SQL Database instance Creating the Settings.xml File Building the RI Build Configurations Release Debug DebugLocal Running the RI Scenario 1. Local Web Server, SQL Event Bus, SQL Event Store Scenario 2. Local Web Server, Windows Azure Service Bus, Table Storage Event Store Scenario 3. Compute Emulator, SQL Event Bus, SQL Event Store Scenario 4. Compute Emulator, Windows Azure Service Bus, Table Storage Event Store Scenario 5. Windows Azure, Windows Azure Service Bus, Table Storage Event Store Running the Tests Running the Unit and Integration Tests Running the Acceptance Tests Known issues More information 323 323 323 324 325 325 325 325 327 327 328 328 328 328 328 Appendix 2: Migrations Migrating from the V1 to the V2 release Running the migration program to migrate the data If the data migration fails Migrating from the V2 to the V3 Release More information 331 331 331 332 333 333 Index 335 328 329 329 329 329 329 329 330 330 330 What other readers are saying about this guide This is another excellent guide from the patterns & practices team—real software engineering with no comforting illusions taken or offered. This guide provides a detailed journal of the practitioners implementing a real production system using the CQRS and Event Sourcing patterns, and also highlights the tradeoffs and teaches the principles that underlie them. The topics presented are relevant and useful, especially if you are building highly scalable Windows Azure applications. You’ll be both challenged and inspired! —Scott Guthrie, Corporate Vice-President, Azure App Platform, Microsoft Having participated and co-authored various guides from patterns & practices, the “CQRS Journey” follows the same walkthrough, scenario-based style, but adding even more fresh empirical content. It’s a true testament of a skilled development team without previous CQRS experience, going through the journey of implementing a complex system and documenting their adventures and lessons learnt in this diary. If I had to recommend to someone where to start with CQRS, I would definitely point them to this guide. —Matias Woloski, CTO, Auth10 LLC The “CQRS Journey” guide is an excellent resource for developers who want to begin developing a CQRS system or convert their current system. It’s a true “trial by fire” approach to the concepts and implementation hurdles that a team would encounter when adopting CQRS. I would recommend reading it twice as I picked up even more lessons the second time through. —Dan Piessens, Lead Software Architect, Zywave I think it’s a really big step in communication with the developer community. You not only share your development experience with a broad audience (which is very valuable by itself) but you’re also open for learning from the community. While working on real projects it’s difficult to stop, find some time to structure your knowledge, prepare it in the form understandable for others. It’s very cool that you found time and resources for such educational effort, I really appreciate this. —Ksenia Mukhortova, Business Applications Developer, Intel I’m very excited about A CQRS Journey for a number of reasons. It explores, with an even hand and a fair mind, a topic where opinions are both diverse and numerous. True to its name, the guide captures the progression of learning. Conclusions are not simply stated; they arrive as a result of experience. Additionally, the project embraced a passionate community with a spirit of inclusion and transparency. The result is friendly-to-read guidance that is both diligent in execution and rigorous in its research. —Christopher Bennage, Software Development Engineer, Microsoft xvii xviii The journey project used Windows Azure SQL Database (backing write & read models), Service Bus (for reliable messaging), and Tables (for event store). Production-quality, scalable cloud services that can be provisioned on-demand with a few mouse-clicks (or API calls) can turn some tough infrastructure problems into trivial ones. —Bill Wilder, MVP, Independent Consultant Perhaps the best lessons out of this guidance will be just how easy it is to work with Microsoft now that they are embracing more community and open source. —Adam Dymitruk, Systems Architect The work that patterns & practices is doing here is very important as it is packaging the concepts in a digestible fashion and helping developers to wade through the ambiguities of CQRS. The real world experiences captured within the journey project will be invaluable to folks looking at applying CQRS within their application development” —Glenn Block, Senior Program Manager, Microsoft, Windows Azure SDK for Node.js, Organizer at ALT.NET Seattle Chapter The p&p team’s dedication and hard work go hand-in-hand with the very high level of competency present on the team. Their attention to detail, insistence on clarity, and open collaboration with the community all led to the creation of material representing enormous value to consumers of the guidance. I definitely plan on referencing this material and code in future engagements because I think my clients will derive many benefits from it–a win-win for everyone! —Josh Elster, Principal, Liquid Electron CQRS is a very important pattern, and a tool that any cloud developer should have in his or her toolbelt. It is particularly well-suited for the cloud since it allows for the implementation of massively scalable solutions based on simple, common patterns (like queues, event handlers, and view models, to name a few). Like all patterns, there are several concrete, correct ways of implementing CQRS. A journey of the type undertaken by Microsoft’s patterns & practices team is a great way to explore the different options, tradeoffs, and even possible mistakes one can make along the way, and accelerate one’s learning of the CQRS pattern. —Shy Cohen, Principal, Shy Cohen Consulting patterns & practices assembled many of the active and key people in the CQRS community to join them on the their journey with CQRS and along the way discovered confusing terminology and concepts that created opportunities for leaders in the community to bring clarity to a broad audience. The material produced is influenced from the results of building a real world application and expresses the experiences from advisors and the patterns & practices team during the development process. By request from the community to allow outside contributions, everything has been open sourced on GitHub. Anyone interested is encouraged to take a look at the guide or implementation. The patterns & practices team has been very welcoming to anyone who wants to collaborate on covering additional areas, alternative implementations or further extending what is currently in place. —Kelly Sommers, Developer xix Congratulations on getting to what looks to be nice guidance. I know that the announcement that p&p was going to embark on this project caused a twitter firestorm but you seem to have come through it well. I’m a fan of the p&p books and think you’ve done a great job in sharing good practices with the community. —Neil Mackenzie, Windows Azure MVP CQRS is as much about architecture community as it is about concrete patterns—thus the project is aptly named “CQRS Journey.” The community involvement and engagement in this project is unprecedented for Microsoft and reflects the enthusiasm amongst the many (if may say: young) software architects from across the industry who are rediscovering proven architecture patterns and are recomposing them in new ways to solve today’s challenges. For me, one takeaway from this project is that the recipes developed here need to be carefully weighed against their alternatives. As with any software architecture approaches that promise easy scalability or evolvability of solutions, the proof will be in concrete, larger production implementations and how they hold up to changing needs over time. Thus, the results of this Journey project mark a start and not a finish line. —Clemens Vasters, Principal Technical Lead, Microsoft Corporation The experiences and conclusions of the p&p team match up very well with our own real-world experiences. Their conclusions in Chapter 8 are spot on. One of the best aspects of this guidance is that the p&p team exposes more of their thought processes and learning throughout the Journey than most write-ups that you may read. From arguments between Developer 1 and Developer 2 on the team, to discussions with experts such as Greg Young and Udi Dahan, to an excellent post-project review in Chapter 8, the thought process is out there for you to learn from. Thanks for this great work, guys. I hope you keep this style with your upcoming guidance pieces. —Jon Wagner, SVP & Chief Architect, eMoney Advisor The CQRS journey release by patterns & practices provides real world insight into the increasingly popular CQRS pattern used in distributed systems that rely upon asynchronous, message based approaches to achieve very large scale. The exploration of the issues the team faced throughout the implementation of the pattern is extremely useful for organizations considering CQRS, both to determine where the pattern is appropriate for them, and to go into the design and implementation with a baseline understanding of the complexity it will introduce. I really enjoyed the candor around the approach taken, the issues encountered, and the early design choices that the team would change in hindsight. This is a must read for any organization embarking upon CQRS, regardless of what platform they are using. —Chris Keyser, VP Engineering, CaseNetwork It is a great resource on tactical and technical aspects of building a distributed system. —Rinat Abdullin, Technology Leader, Lokad I’d like to personally thank the team for putting together such a transparent journey throughout this project. I’m very pleased with the final release. —Truong Nguyen, CEO, Nepsoft It’s a good read. Lots to learn from it. —Christian Horsdal Gammelgaard, Lead Software Architect, Mjølner Informatics Foreword by Greg Young I started off the new year on January 3rd with a few hour long meeting showing the team at patterns & practices a bit about Command and Query Responsibility Segregation (CQRS) and Event Sourcing (ES). Most of the team had previously not been exposed to these ideas. Today is almost exactly six months later and they have produced a document of over 200 pages of discussions and guidance as well as a full end to end example hosted in Windows Azure. This is certainly not a small feat. When the announcement of the project came out, the twitter stream near instantly went negative as many thought that Microsoft was building a CQRS framework; which was premature from the community. The process followed similar paths to other patterns & practices projects with a large advisor board being set up. I believe however that the most interesting part of the process was the decision to host the work on GitHub and allow pull requests which is an extremely open and transparent way of communicating during the project. One of the main benefits for the community as a whole of going through such a process is that people were forced to refine their vocabularies. There are in the DDD/CQRS/ES communities many different voices and often times, especially in younger groups, vocabularies will go down divergent paths leading to fractured community. An example of nebulous terminologies can be seen in the terms ”saga,” ”process manager,” and ”workflow”; the community as a whole I believe benefited from the discussions over defining what it actually is. One of the most interesting conversations brought up for me personally was defining the difference between an Event Store and a Transaction Log as legitimate arguments can be made that either is a higher level abstraction of the other. This has led not only to many interesting discussions in the community but to a far stricter future definition of what an Event Store is. ”For the things we have to learn before we can do them, we learn by doing them. ~Aristotle” The quote above was the team motto during the project. Many will be looking towards the guidance presented as being authoritative guidance of how things should be done. This is however not the optimal way to look at the guidance as presented (though it does contain many bits of good authoritative guidance). The main benefit of the guidance is the learning experience that it contains. It is important to remember that the team came into the ideas presented as non-experienced in CQRS and they learned in the process of doing. This gives a unique perspective throughout much of the text where things are learned along the way or are being seen through fresh eyes of someone recently having learned and attempted to apply the ideas. This perspective has also brought up many interesting conversations within the community. The patterns & practices team deserves credit for digging deep, facilitating these discussions, and bringing to light various incongruities, confusions and inconsistencies as they went along. xxi xxii Keeping in mind the origination point of the team, the most valuable bits in the text that a reader should focus on aside from general explanations are places where tradeoffs are discussed. There is an unfortunate tendency to seek authoritative answers that ”things should be done in this way” when they in fact do not exist. There are many ways to proverbially skin a cat and all have their pros and cons. The text is quite good at discussing alternative points of view that came up as possible answers, or that received heavy discussion within the advisor group, these can often be seen in the “developer 1/developer 2 discussions.” One such discussion I mentioned previously in defining the difference between event sourcing and a transaction log. Many of these types of discussions come at the end of the guidance. How might things be approached differently? One of my favourite discussions towards the end of the guidance dealing with performance is the independent realization that messaging is not equivalent to distribution. This is a very hard lesson for many people to understand and the way that it comes up rather organically and much like it would on most teams as a performance problem is a great explanation. I can say 100 times to apply the first law of distributed computing, don’t distribute; however seeing it from the eyes of a team dealing with a performance problem who has already made the mistake of equating the two is a very understandable path and a great teaching tool. This section also contains a smörgåsbord of information and insights in terms of how to build performant applications in Windows Azure. Out in the wild, there are plenty of naïve samples of CQRS/ES implementations, which are great for describing the concepts. There are details and challenges that will not surface till you work on a complex, real-world production system. The value of the p&p’s sample application is that it uses a fairly complex domain and the team went through multiple releases and focused on infrastructure hardening, performance optimizations, dealing with transient faults and versioning, etc. — many practical issues that you face when implementing CQRS and ES. As with any project, people may disagree with implementation choices and decisions made. It is important to remember the scoping of the project. The guidance is not coming from an expert viewpoint throughout the process, but that of a group “learning by doing.” The process was and remains open to contributions, and in fact this version has been reviewed, validated, and guided by experts in the community. In the spirit of OSS “send a pull request.” This guide can serve as a valuable point to start discussions, clear up misconceptions, and refine how we explain things, as well as drive improvement both in the guidance itself and in getting consistent viewpoints throughout the community. In conclusion I think patterns & practices has delivered to the community a valuable service in the presentation of this guidance. The view point the guidance is written from is both an uncommon and valuable one. It has also really been a good overall exercise for the community in terms of setting the bar for what is being discussed and refining of the vocabularies that people speak in. Combine this with the amount of previously difficult to find Windows Azure guidance and the guidance becomes quite valuable to someone getting into the ideas. Greg Young Preface Why are we embarking on this journey? “The best way to observe a fish is to become a fish.” Jacques Cousteau Why we created this guidance now The Command Query Responsibility Segregation (CQRS) pattern and event sourcing (ES) are currently generating a great deal of interest from developers and architects who are designing and building large-scale, distributed systems. There are conference sessions, blogs, articles, and frameworks all dedicated to the CQRS pattern and to event sourcing, and all explaining how they can help you to improve the maintainability, testability, scalability, and flexibility of your systems. However, like anything new, it takes some time before a pattern, approach, or methodology is fully understood and consistently defined by the community and has useful, practical guidance to help you to apply or implement it. This guidance is designed to help you get started with the CQRS pattern and event sourcing. It is not intended to be the guide to the CQRS pattern and event sourcing, but a guide that describes the experiences of a development team in implementing the CQRS pattern and event sourcing in a realworld application. The development team did not work in isolation; they actively sought input from industry experts and from a wider group of advisors to ensure that the guidance is both detailed and practical. The CQRS pattern and event sourcing are not mere simplistic solutions to the problems associated with large-scale, distributed systems. By providing you with both a working application and written guidance, we expect you’ll be well prepared to embark on your own CQRS journey. How is this guidance structured? There are two closely related parts to this guidance: • A working reference implementation (RI) sample, which is intended to illustrate many of the concepts related to the CQRS pattern and event sourcing approaches to developing complex enterprise applications. • This written guidance, which is intended to complement the RI by describing how it works, what decisions were made during its development, and what trade-offs were considered. xxiii xxiv This written guidance is itself split into three distinct sections that you can read independently: a description of the journey we took as we learned about CQRS, a collection of CQRS reference materials, and a collection of case studies that describe the experiences other teams have had with the CQRS pattern. The map in Figure 1 illustrates the relationship between the first two sections: a journey with some defined stopping points that enables us to explore a space. Figure 1 A CQRS journey A CQRS journey This section is closely related to the RI and the chapters follow the chronology of the project to develop the RI. Each chapter describes relevant features of the domain model, infrastructure elements, architecture, and user interface (UI) that the team was concerned with during that phase of the project. Some parts of the system are discussed in several chapters, and this reflects the fact that the team revisited certain areas during later stages. Each of these chapters discuss how and why particular CQRS patterns and concepts apply to the design and development of particular bounded contexts, describe the implementation, and highlight any implications for testing. xxv Other chapters look at the big picture. For example, there is a chapter that explains the rationale for splitting the RI into the bounded contexts we chose, another chapter analyzes the implications of our approach for versioning the system, and other chapters look at how the different bounded contexts in the RI communicate with each other. This section describes our journey as we learned about CQRS, and how we applied that learning to the design and implementation of the RI. It is not prescriptive guidance and is not intended to illustrate the only way to apply the CQRS approach to our RI. We have tried wherever possible to capture alternative viewpoints through consultation with our advisors and to explain why we made particular decisions. You may disagree with some of those decisions; please let us know at cqrsjourney@microsoft.com. This section of the written guidance makes frequent cross-references to the material in the second section for readers who wish to explore any of the concepts or patterns in more detail. CQRS reference The second section of the written guidance is a collection of reference material collated from many sources. It is not the definitive collection, but should contain enough material to help you to understand the core patterns, concepts, and language of CQRS. Tales from the trenches This section of the written guidance is a collection of case studies from other teams that describe their experiences of implementing the CQRS pattern and event sourcing in the real world. These case studies are not as detailed as the journey section of the guidance and are intended to give an overview of these projects and to summarize some of the key lessons learned. The following is a list of the chapters that comprise both sections of the written guidance: A CQRS journey • Chapter 1, “The Contoso Conference Management System,” introduces our sample application and our team of (fictional) experts. • Chapter 2, “Decomposing the Domain,” provides a high-level view of the sample application and describes the bounded contexts that make up the application. • Chapter 3, “Orders and Registrations Bounded Context,” introduces our first bounded context, explores some CQRS concepts, and describes some elements of our infrastructure. • Chapter 4, “Extending and Enhancing the Orders and Registrations Bounded Context,” describes adding new features to the bounded context and discusses our testing approach. • Chapter 5, “Preparing for the V1 Release,” describes adding two new bounded contexts and handling integration issues between them, and introduces our event-sourcing implementation. This is our first pseudo-production release. • Chapter 6, “Versioning Our System,” discusses how to version the system and handle upgrades with minimal down time. • Chapter 7, “Adding Resilience and Optimizing Performance,” describes what we did to make the system more resilient to failure scenarios and how we optimized the performance of the system. This was the last release of the system in our journey. • Chapter 8, “Lessons Learned,” collects the key lessons we learned from our journey and suggests how you might continue the journey. xxvi CQRS reference • Chapter 1, “CQRS in Context,” provides some context for CQRS, especially in relation to the domain-driven design approach. • Chapter 2, “Introducing the Command Query Responsibility Segregation Pattern,” provides a conceptual overview of the CQRS pattern. • Chapter 3, “Introducing Event Sourcing,” provides a conceptual overview of event sourcing. • Chapter 4, “A CQRS and ES Deep Dive,” describes the CQRS pattern and event sourcing in more depth. • Chapter 5, “Communicating between Bounded Contexts,” describes some options for communicating between bounded contexts. • Chapter 6, “A Saga on Sagas,” explains our choice of terminology: process manager instead of saga. It also describes the role of process managers. • Chapter 7, “Technologies Used in the Reference Implementation,” provides a brief overview of some of the other technologies we used, such as the Windows Azure Service Bus. • Appendix 1, “Release Notes,” contains detailed instructions for downloading, building, and running the sample application and test suites. • Appendix 2, “Migrations,” contains instructions for performing the code and data migrations between the pseudo-production releases of the Contoso Conference Management System. Tales from the trenches • Chapter 1, “Twilio,” describes a highly available, cloud-hosted, communications platform. Although the team who developed this product did not explicitly use CQRS, many of the architectural concepts they adopted are very closely related to the CQRS pattern. • Chapter 2, “Lokad Hub,” describes a project that made full use of domain-driven design, CQRS, and event sourcing in an application designed to run on multiple cloud platforms. • Chapter 3, “DDD/CQRS for large financial company,” describes a project that made full use of domain-driven design and CQRS to build a reference application for a large financial company. It used CQRS to specifically address the issues of performance, scalability, and reliability. • Chapter 4, “Digital Marketing,” describes how an existing application was refactored over time while delivering new features. This project adopted the CQRS pattern for one of its pieces as the project progressed. • Chapter 5, “TOPAZ Technologies,” describes a project that used the CQRS pattern and event sourcing to simplify the development of an off-the-shelf enterprise application. • Chapter 6, “eMoney Nexus,” describes migration project for an application that used legacy three-tier architecture to an architecture that used the CQRS pattern and event sourcing. Many of the conclusions drawn in this project are similar to our own experiences on our CQRS journey. Selecting the domain for the RI Before embarking on our journey, we needed to have an outline of the route we planned to take and an idea of what the final destination should be. We needed to select an appropriate domain for the RI. We engaged with the community and our advisory board to help us choose a domain that would enable us to highlight as many of the features and concepts of CQRS as possible. To help us select between our candidate domains, we used the criteria in the following list. The domain selected should be: xxvii • Non-trivial. The domain must be complex enough to exhibit real problems, but at the same time simple enough for most people to understand without weeks of study. The problems should involve dealing with temporal data, stale data, receiving out-of-order events, and versioning. The domain should enable us to illustrate solutions using event sourcing, sagas, and event merging. • Collaborative. The domain must contain collaborative elements where multiple actors can operate simultaneously on shared data. • End to end. We wanted to be able illustrate the concepts and patterns in action from the back-end data store through to the user interface. This might include disconnected mobile and smart clients. • Cloud friendly. We wanted to have the option of hosting parts of the RI on Windows Azure and be able to illustrate how you can use CQRS for cloud-hosted applications. • Large. We wanted to be able to show how our domain can be broken down into multiple bounded contexts to highlight when to use and when not use CQRS. We also wanted to illustrate how multiple architectural approaches (CQRS, CQRS/ES, and CRUD) and legacy systems can co-exist within the same domain. We also wanted to show how multiple development teams could carry out work in parallel. • Easily deployable. The RI needed to be easily deployable so that you can install it and experiment with it as you read this guidance. As a result, we chose to implement the conference management system that Chapter 1, “Our Domain: The Contoso Conference Management System,” introduces. Arrow legend Many illustrations in the guidance have arrows. Here is their associated meaning. Event message Command message Method call Flow of data Object relationship Figure 2 Legend for arrows xxviii Where to go for more information There are a number of resources listed in text throughout the book. These resources will provide additional background, bring you up to speed on various technologies, and so forth. For your convenience, there is a bibliography online that contains all the links so that these resources are just a click away. You can find the bibliography on MSDN at: http://msdn.microsoft.com/en-us/library/jj619274. The Crew Captain Ernest Shackleton’s Antarctic expedition recruitment ad (1913) stated: No fewer than 5000 people replied… When we embarked on our journey half a year ago, it felt almost the same. With no fewer than 70 community members (both experts and enthusiastic novices) answering the call for advisory board and offering to volunteer their time to help us steer this project! We have now reached the end of the journey. These are the members of the development team who endured the challenges of the journey and produced this guide: Vision and Program Management Grigori Melnik (Microsoft Corporation) Development Julián Domínguez (Microsoft Corporation), Daniel Cazzulino and Fernando Simonazzi (Clarius Consulting) Testing Mani Subramanian (Microsoft Corporation), Hernan de Lahitte (Digit Factory), and Rathi Velusamy (Infosys Technologies Ltd.) Documentation Dominic Betts (Content Master Ltd.), Julián Domínguez, Grigori Melnik, and Mani Subramanian (Microsoft Corporation), and Fernando Simonazzi (Clarius Consulting) Graphic Design Alexander Ustinov and Anton Rusecki (JetStyle) Editing and Production RoAnn Corbisier and Nelly Delgado (Microsoft Corporation), Nancy Michell (Content Master Ltd.), and Chris Burns (Linda Werner & Associates Inc) The development team didn’t embark on this journey by themselves and didn’t work in isolation. We actively sought input from industry experts and from a wider group of advisors to ensure that the guidance is detailed, practical, and informed by real-world experience. We would like to thank our advisory board members and the DDD/CQRS community members in general who have accompanied us on this journey for their active participation, insights, critiques, challenges, and reviews. We have learned and unlearned many things, we’ve explored and experimented a lot. The journey wasn’t easy but it was so worth it and we enjoyed it. Thank you for keeping us grounded in the real-world challenges. Thank you for your ongoing support of our effort. We hope the community will continue exploring the space, pushing the state of the practice further, and extending the reference implementation and the guidance. xxix xxx Specifically, we’d like to acknowledge the following people who have contributed to the journey in many different ways: • Greg Young for your pragmatism, patience with us, continuous mentoring and irreplaceable advice; • Udi Dahan for challenging us and offering alternative views on many concepts; • Clemens Vasters for pushing back on terminology and providing a very valuable perspective from the distributed database field; • Kelly Sommers for believing in us and bringing sanity to the community as well as for deep technical insights; • Adam Dymitruk for jumpstarting us on git and extending the RI; • Glenn Block for encouraging us to go all the way with the OSS initiative and for introducing us to many community members; • Our GM Lori Brownell and our director Björn Rettig for providing sponsorship of the initiative and believing in our vision; • Scott Guthrie for supporting the project and helping amplify the message; • Josh Elster for exploring and designing the MIL (Messaging Intermediate Language) and pushing us to make it easier to follow the workflow of messages in code; • Cesar De la Torre Llorente for helping us spike on the alternatives and bringing up terminological incongruities between various schools and thought leaders; • Rinat Abdullin for active participation at the beginning of the project and contributing a case study; • Bruno Terkaly and Ricardo Villalobos for exploring the disconnected client scenario that would integrate with the RI; • Einar Otto Stangvik for spiking on the Schedule Builder bounded context implementation in Node.js; • Mark Seemann for sending the very first pull request focusing on code quality; • Christopher Bennage for helping us overcome GitHub limitations by creating the pundit review system and the export-to-Excel script to manage iteration backlog more effectively; • Bob Brumfield, Eugenio Pace, Carlos Farre, Hanz Zhang, and Rohit Sharma for many insights especially on the perf and hardening challenges; • Chris Tavares for putting out the first CQRS experiment at p&p and suggesting valuable scenarios; • Tim Sharkinian for your perspectives on CQRS and for getting us on the SpecFlow train; • Jane Sinyagina for helping solicit and process feedback from the advisors; • Howard Wooten and Thomas Petchel for feedback on the UI style and usability; • Kelly Leahy for sharing your experience and making us aware of potential pitfalls; • Dylan Smith for early conversations and support of this project in pre-flight times; • Evan Cooke, Tim Walton, Alex Dubinkov, Scott Brown, Jon Wagner, and Gabriel N. Schenker for sharing your experiences and contributing mini-case studies. We feel honored to be supported by such an incredible group of people. Thank you! Journey 1: Our Domain: Conference Management System The starting point: Where have we come from, what are we taking, and who is coming with us? “I am prepared to go anywhere, provided it be forward.” David Livingstone This chapter introduces a fictitious company named Contoso. It describes Contoso’s plans to launch the Contoso Conference Management System, a new online service that will enable other companies or individuals to organize and manage their own conferences and events. This chapter describes, at a high-level, some of the functional and non-functional requirements of the new system, and why Contoso wants to implement parts of it using the Command Query Responsibility Segregation (CQRS) pattern and event sourcing (ES). As with any company considering this process, there are many issues to consider and challenges to be met, particularly because this is the first time Contoso has used both the CQRS pattern and event sourcing. The chapters that follow show, step by step, how Contoso designed and built its conference management application. This chapter also introduces a panel of fictional experts to comment on the development efforts. The Contoso Corporation Contoso is a startup ISV company of approximately 20 employees that specializes in developing solutions using Microsoft technologies. The developers at Contoso are knowledgeable about various Microsoft products and technologies, including the .NET Framework, ASP.NET MVC, and Windows Azure. Some of the developers have previous experience using the domain-driven design (DDD) approach, but none of them have used the CQRS pattern previously. The Conference Management System application is one of the first innovative online services that Contoso wants to take to market. As a startup, Contoso wants to develop and launch these services with a minimal investment in hardware and IT personnel. Contoso wants to be quick to market in order to start growing market share, and cannot afford the time to implement all of the planned functionality in the first releases. Therefore, it is important that the architecture it adopts can easily accommodate changes and enhancements with minimal impact on existing users of the system. Contoso has chosen to deploy the application on Windows Azure in order to take advantage of its ability to scale applications as demand grows. 1 2 Jour ney one Who is coming with us on the journey? As mentioned earlier, this guide and the accompanying RI describe a CQRS journey. A panel of experts will comment on our development efforts as we go. This panel includes a CQRS expert, a software architect, a developer, a domain expert, an IT Pro, and a business manager. They will all comment from their own perspectives. Gary is a CQRS expert. He ensures that a CQRS-based solution will work for a company and will provide tangible benefits. He is a cautious person, for good reason. “Defining the CQRS pattern is easy. Realizing the benefits that implementing the CQRS pattern can offer is not always so straightforward.” Jana is a software architect. She plans the overall structure of an application. Her perspective is both practical and strategic. In other words, she considers not only what technical approaches are needed today, but also what direction a company needs to consider for the future. Jana has worked on projects that used the domain-driven design approach. “It’s not easy to balance the needs of the company, the users, the IT organization, the developers, and the technical platforms we rely on.” Markus is a software developer who is new to the CQRS pattern. He is analytical, detail-oriented, and methodical. He’s focused on the task at hand, which is building a great application. He knows that he’s the person who’s ultimately responsible for the code. “I don’t care what architecture you want to use for the application; I’ll make it work.” Carlos is the domain expert. He understands all the ins and outs of conference management. He has worked in a number of organizations that help people run conferences. He has also worked in a number of different roles: sales and marketing, conference management, and consultant. “I want to make sure that the team understands how this business works so that we can deliver a world-class online conference management system.” Our Dom a in: Conference M a nagement System 3 Poe is an IT professional who’s an expert in deploying and running applications in the cloud. Poe has a keen interest in practical solutions; after all, he’s the one who gets paged at 3:00 AM when there’s a problem. “Running complex applications in the cloud involves challenges that are different than the challenges in managing on-premises applications. I want to make sure our new conference management system meets our published service-level agreements (SLA).” Beth is a business manager. She helps companies to plan how their business will develop. She understands the market that the company operates in, the resources that the company has available, and the goals of the company. She has both a strategic view and an interest in the day-to-day operations of the company. “Organizations face many conflicting demands on their resources. I want to make sure that our company balances those demands and adopts a business plan that will make us successful in the medium and long term.”If you have a particular area of interest, look for notes provided by the specialists whose interests align with yours. The Contoso Conference Management System This section describes the Contoso Conference Management System as the team envisaged it at the start of the journey. The team has not used the CQRS pattern before; therefore, the system that is delivered at the end of our journey may not match this description exactly because: • What we learn as we go may impact what we ultimately deliver. • Because this is a learning journey, it is more difficult to estimate what we can achieve in the available time. Overview of the system Contoso plans to build an online conference management system that will enable its customers to plan and manage conferences that are held at a physical location. The system will enable Contoso’s customers to: • Manage the sale of different seat types for the conference. • Create a conference and define characteristics of that conference. The Contoso Conference Management System will be a multi-tenant, cloud-hosted application. Business customers will need to register with the system before they can create and manage their conferences. 4 Jour ney one Selling seats for a conference The business customer defines the number of seats available for the conference. The business customer may also specify events at a conference such as workshops, receptions, and premium sessions for which attendees must have a separate ticket. The business customer also defines how many seats are available for these events. The system manages the sale of seats to ensure that the conference and sub-events are not oversubscribed. This part of the system will also operate wait-lists so that if other attendees cancel, their seats can be reallocated. The system will require that the names of the attendees be associated with the purchased seats so that an on-site system can print badges for the attendees when they arrive at the conference. Creating a conference A business customer can create new conferences and manage information about the conference such as its name, description, and dates. The business customer can also make a conference visible on the Contoso Conference Management System website by publishing it, or hide it by unpublishing it. Additionally, the business customer defines the seat types and available quantity of each seat type for the conference. Contoso also plans to enable the business customer to specify the following characteristics of a conference: • Whether the paper submission process will require reviewers. • What the fee structure for paying Contoso will be. • Who key personnel, such as the program chair and the event planner, will be. Nonfunctional requirements Contoso has two major nonfunctional requirements for its conference management system—scalability and flexibility—and it hopes that the CQRS pattern will help it meet them. Scalability The conference management system will be hosted in the cloud; one of the reasons Contoso chose a cloud platform was its scalability and potential for elastic scalability. Although cloud platforms such as Windows Azure enable you to scale applications by adding (or removing) role instances, you must still design your application to be scalable. By splitting responsibility for the application’s read and write operations into separate objects, the CQRS pattern allows Contoso to split those operations into separate Windows Azure roles that can scale independently of each other. This recognizes the fact that for many applications, the number of read operations vastly exceeds the number of write operations. This gives Contoso the opportunity to scale the conference management system more efficiently, and make better use of the Windows Azure role instances it uses. Our Dom a in: Conference M a nagement System Flexibility The market that the Contoso Conference Management System operates in is very competitive, and very fast moving. In order to compete, Contoso must be able to quickly and cost effectively adapt the conference management system to changes in the market. This requirement for flexibility breaks down into a number of related aspects: • Contoso must be able to evolve the system to meet new requirements and to respond to changes in the market. • The system must be able to run multiple versions of its software simultaneously in order to support customers who are in the middle of a conference and who do not wish to upgrade to a new version immediately. Other customers may wish to migrate their existing conference data to a new version of the software as it becomes available. • Contoso intends the software to last for at least five years. It must be able to accommodate significant changes over that period. • Contoso does not want the complexity of some parts of the system to become a barrier to change. • Contoso would like to be able to use different developers for different elements of the system, using cheaper developers for simpler tasks and restricting its use of more expensive and experienced developers to the more critical aspects of the system. Beginning the journey The next chapter is the start of our CQRS journey. It provides more information about the Contoso Conference Management System and describes some of the high-level parts of the system. Subsequent chapters describe the stages of the journey as Contoso implements the conference management system. Contoso plans to compete by being quick to respond to changes in the market and to changing customer requirements. Contoso must be able to evolve the system quickly and painlessly. This is a big challenge: keeping the system running for all our customers while we perform upgrades with no down time. More information All links in this book are accessible from the book’s online bibliography available at: http://msdn.microsoft.com/en-us/library/jj619274. There is some debate in the CQRS community about whether, in practice, you can use different development teams for different parts of the CQRS pattern implementation. 5 Journey 2: Decomposing the Domain Planning the stops. “Without stones there is no arch.” Marco Polo In this chapter, we provide a high-level overview of the Contoso Conference Management System. The discussion will help you understand the structure of the application, the integration points, and how the parts of the application relate to each other. Here we describe this high-level structure in terms borrowed from the domain-driven design (DDD) approach that Eric Evans describes in his book, Domain-Driven Design: Tackling Complexity in the Heart of Software (Addison-Wesley Professional, 2003). Although there is no universal consensus that DDD is a prerequisite for implementing the Command Query Responsibility Segregation (CQRS) pattern successfully, our team decided to use many of the concepts from the DDD approach, such as domain, bounded context, and aggregate, in line with common practice within the CQRS community. Chapter 1, “CQRS in Context,” in the Reference Guide discusses the relationship between the DDD approach and the CQRS pattern in more detail. Definitions used in this chapter Throughout this chapter we use a number of terms, which we’ll define in a moment. For more detail, and possible alternative definitions, see Chapter 1, “CQRS in Context,” in the Reference Guide. Domain: The domain refers to the business domain for the Contoso Conference Management System (the reference implementation). Chapter 1, “Our Domain: The Contoso Conference Management System,” provides an overview of this domain. 7 8 Jour ney t wo When you use the CQRS pattern, you often use events to communicate between bounded contexts. There are alternative approaches to integration, such as sharing data at the database level. Bounded context: The term bounded context comes from Eric Evans’ book. In brief, Evans introduces this concept as a way to decompose a large, complex system into more manageable pieces; a large system is composed of multiple bounded contexts. Each bounded context is the context for its own self-contained domain model, and has its own ubiquitous language. You can also view a bounded context as an autonomous business component defining clear consistency boundaries: one bounded context typically communicates with another bounded context by raising events. Context map: According to Eric Evans, you should “Describe the points of contact between the models, outlining explicit translation for any communication and highlighting any sharing.” This exercise results in what is called a context map, which serves several purposes that include providing an overview of the whole system and helping people to understand the details of how different bounded contexts interact with each other. Bounded contexts in the conference management system We discussed making the period of time that the system holds reservations a parameter that a business customer can adjust for each conference. This may be a feature that we add if we determine that there is a requirement for this level of control. The Orders and Registrations bounded context: Within the orders and registrations bounded context are the reservations, payment, and registration items. When a registrant interacts with the system, the system creates an order to manage the reservations, payment, and registrations. An order contains one or more order items. A reservation is a temporary reservation of one or more seats at a conference. When a registrant begins the ordering process to purchase a number of seats at a conference, the system creates reservations for that number of seats. Those seats are then unavailable for other registrants to reserve. The reservations are held for 15 minutes, during which time the registrant can complete the ordering process by making a payment for the seats. If the registrant does not pay for the tickets within 15 minutes, the system deletes the reservation and the seats become available for other registrants to reserve. The Conference Management bounded context: Within this bounded context, a business customer can create new conferences and manage them. After a business customer creates a new conference, he can access the details of the conference by using his email address and conference locator access code. The system generates the access code when the business customer creates the conference. Decomposing the Dom a in 9 The business customer can specify the following information about a conference: • The name, description, and slug (part of the URL used to access the conference). • The start and end dates of the conference. • The different types and quotas of seats available at the conference. Additionally, the business customer can control the visibility of the conference on the public website by either publishing or unpublishing the conference. The business customer can also use the conference management website to view a list of orders and attendees. The Payments bounded context: The payments bounded context is responsible for managing the interactions between the conference management system and external payment systems. It forwards the necessary payment information to the external system and receives an acknowledgement that the payment was either accepted or rejected. It reports the success or failure of the payment back to the conference management system. Initially, the payments bounded context will assume that the business customer has an account with the third-party payment system (although not necessarily a merchant account), or that the business customer will accept payment by invoice. Bounded contexts not included Although they didn’t make it into the final release of the Contoso Conference Management System, some work was done on three additional bounded contexts. Members of the community are working on these and other features, and any out-of-band releases and updates will be announced on the Project “a CQRS Journey” website. If you would like to contribute to these bounded contexts or any other aspect of the system, visit the Project “a CQRS Journey” website or let us know at cqrsjourney@ microsoft.com. The Discounts bounded context: This is a bounded context to handle the process of managing and applying discounts to the purchase of conference seats that would integrate with all three existing bounded contexts. The Occasionally Disconnected Conference Management client: This is a bounded context to handle management of conferences on-site with functionality to handle label printing, recording attendee arrivals, and additional seat sales. The Submissions And Schedule Management bounded context: This is a bounded context to handle paper submissions and conference event scheduling written using Node.js. Note: Wait listing is not implemented in this release, but members of the community are working on this and other features. Any out-of-band releases and updates will be announced on the Project “a CQRS Journey” website. 10 Jour ney t wo The context map for the Contoso Conference Management System A frequent comment about CQRS projects is that it can be difficult to understand how all of the pieces fit together, especially if there a great many commands and events in the system. Often, you can perform some static analysis on the code to determine where events and commands are handled, but it is more difficult to automatically determine where they originate. At a high level, a context map can help you understand the integration between the different bounded contexts and the events involved. Maintaining up-to-date documentation about the commands and events can provide more detailed insight. Additionally, if you have tests that use commands as inputs and then check for events, you can examine the tests to understand the expected consequences of particular commands (see the section on testing in Chapter 4, “Extending and Enhancing the Orders and Registrations Bounded Context” for an example of this style of test). Figure 1 and the table that follows it represent a context map that shows the relationships between the different bounded contexts that make up the complete system, and as such it provides a highlevel overview of how the system is put together. Even though this context map appears to be quite simple, the implementation of these bounded contexts, and more importantly the interactions between them, are relatively sophisticated; this enabled us to address a wide range of issues relating to the CQRS pattern and event sourcing (ES), and provided a rich source from which to capture many valuable lessons learned. Figure 1 shows the three bounded contexts that make up the Contoso Conference Management System. The arrows in the diagram indicate the flow of data as events between them. Figure 1 Bounded contexts in the Contoso Conference Management System Decomposing the Dom a in The following list provides more information about the arrows in Figure 1. You can find additional details in the chapters that discuss the individual bounded contexts. 1. Events that report when conferences have been created, updated, or published. Events that report when seat types have been created or updated. 2. Events that report when orders have been created or updated. Events that report when attendees have been assigned to seats. 3. Requests for a payment to be made. 4. Acknowledgement of the success or failure of the payment. Why did we choose these bounded contexts? During the planning stage of the journey, it became clear that these were the natural divisions in the domain that could each contain their own, independent domain models. Some of these divisions were easier to identify than others. For example, it was clear early on that the conference management bounded context is independent of the remainder of the domain. It has clearly defined responsibilities that relate to defining conferences and seat types and clearly defined points of integration with the rest of the application. On the other hand, it took some time to realize that the orders and registrations bounded context is separate from the Payments bounded context. For example, it was not until the V2 release of the application that all concepts relating to payments disappeared from the orders and registrations bounded context when the OrderPaymentConfirmed event became the OrderConfirmed event. More practically, from the perspective of the journey, we wanted a set of bounded contexts that would enable us to release a working application with some core functionality and that would enable us to explore a number of different implementation patterns: CQRS, CQRS/ES, as well as integration with a legacy, CRUD-style bounded context. Some of the events that the Conference Management bounded context raises are coarsegrained and contain multiple fields. Remember that conference management is a create, read, update and delete (CRUD)-style bounded context and does not raise fine-grained domain-style events. For more information, see Chapter 5, “Preparing for the V1 Release.” We continued to refine the domain models right through the journey as our understanding of the domain deepened. More information All links in this book are accessible from the book’s online bibliography available at: http://msdn.microsoft.com/en-us/library/jj619274. Contoso wants to release a usable application as soon as possible, but be able to add both planned features and customer-requested features as they are developed and with no down time for the upgrades. 11 Journey 3: Orders and Registrations Bounded Context The first stop on our CQRS journey. “The Allegator is the same, as the Crocodile, and differs only in Name.” John Lawson A description of the bounded context The Orders and Registrations bounded context is partially responsible for the booking process for attendees planning to come to a conference. In the Orders and Registrations bounded context, a person (the registrant) purchases seats at a particular conference. The registrant also assigns names of attendees to the purchased seats (this is described in Chapter 5, “Preparing for the V1 Release”). This was the first stop on our CQRS journey, so the team decided to implement a core, but selfcontained part of the system—orders and registrations. The registration process must be as painless as possible for attendees. The process must enable the business customer to ensure that the maximum possible number of seats can be booked, and give them the flexibility set the prices for the different seat types at a conference. Because this was the first bounded context addressed by the team, we also implemented some infrastructure elements of the system to support the domain’s functionality. These included command and event message buses and a persistence mechanism for aggregates. The Contoso Conference Management System described in this chapter is not the final version of the system. This guidance describes a journey, so some of the design decisions and implementation details change later in the journey. These changes are described in subsequent chapters. Plans for enhancements to this bounded context in some future journey include support for wait listing, whereby requests for seats are placed on a wait list if there aren’t sufficient seats available, and enabling the business customer to set various types of discounts for seat types. Wait listing is not implemented in this release, but members of the community are working on this and other features. Any out-of-band releases and updates will be announced on the Project “a CQRS Journey” website. 13 14 Jour ney thr ee Working definitions for this chapter For a discussion of some possible optimizations that also involve a slightly different definition of a command, see Chapter 6, “Versioning our System.” This chapter uses a number of terms that we will define in a moment. For more detail, and possible alternative definitions, see “A CQRS and ES Deep Dive” in the Reference Guide. Command. A command is a request for the system to perform an action that changes the state of the system. Commands are imperatives; MakeSeatReservation is one example. In this bounded context, commands originate either from the UI as a result of a user initiating a request, or from a process manager when the process manager is directing an aggregate to perform an action. A single recipient processes a command. A command bus transports commands that command handlers then dispatch to aggregates. Sending a command is an asynchronous operation with no return value. Event. An event, such as OrderConfirmed, describes something that has happened in the system, typically as a result of a command. Aggregates in the domain model raise events. Multiple subscribers can handle a specific event. Aggregates publish events to an event bus; handlers register for specific types of events on the event bus and then deliver the event to the subscriber. In this bounded context, the only subscriber is a process manager. Process manager. In this bounded context, a process manager is a class that coordinates the behavior of the aggregates in the domain. A process manager subscribes to the events that the aggregates raise, and then follow a simple set of rules to determine which command or commands to send. The process manager does not contain any business logic; it simply contains logic to determine the next command to send. The process manager is implemented as a state machine, so when it responds to an event, it can change its internal state in addition to sending a new command. Our process manager is an implementation of the Process Manager pattern defined on pages 312 to 321 of the book by Gregor Hohpe and Bobby Woolf, entitled Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (AddisonWesley Professional, 2003). It can be difficult for someone new to the code to follow the flow of commands and events through the system. For a discussion of a technique that can help, see the section “Impact on testing” in Chapter 4, “Extending and Enhancing the Orders and Registrations Bounded Contexts.” Or ders a nd R egistr ations Bounded Context The process manager in this bounded context can receive commands as well as subscribe to events. The Reference Guide contains additional definitions and explanations of CQRS-related terms. Domain definitions (ubiquitous language) The following list defines the key domain-related terms that the team used during the development of this Orders and Registrations bounded context. Attendee. An attendee is someone who is entitled to attend a conference. An Attendee can interact with the system to perform tasks such as manage his agenda, print his badge, and provide feedback after the conference. An attendee could also be a person who doesn’t pay to attend a conference such as a volunteer, speaker, or someone with a 100% discount. An attendee may have multiple associated attendee types (speaker, student, volunteer, track chair, and so on.) Registrant. A registrant is a person who interacts with the system to place orders and to make payments for those orders. A registrant also creates the registrations associated with an order. A registrant may also be an attendee. User. A user is a person such as an attendee, registrant, speaker, or volunteer who is associated with a conference. Each user has a unique record locator code that the user can use to access user-specific information in the system. For example, a registrant can use a record locator code to access her orders, and an attendee can use a record locator code to access his personalized conference agenda. Seat assignment. A seat assignment associates an attendee with a seat in a confirmed order. An order may have one or more seat assignments associated with it. Order. When a registrant interacts with the system, the system creates an order to manage the reservations, payment, and registrations. An order is confirmed when the registrant has successfully paid for the order items. An order contains one or more order items. Order item. An order item represents a seat type and quantity, and is associated with an order. An order item exists in one of three states: created, reserved, or rejected. An order item is initially in the created state. An order item is in the reserved state if the system has reserved the quantity of seats of the seat type requested by the registrant. An order item is in the rejected state if the system cannot reserve the quantity of seats of the seat type requested by the registrant. The team initially referred to the process manager class in the orders bounded context as a saga. To find out why we decided to change the terminology, see the section “Patterns and concepts” later in this chapter. We intentionally implemented a record locator mechanism to return to a previously submitted order via the mechanism. This eliminates an often annoying requirement for users to create an account in the system and sign in in order to evaluate its usefulness. Our customers were adamant about this. 15 16 Jour ney thr ee Seat. A seat represents the right to be admitted to a conference or to access a specific session at the conference such as a cocktail party, a tutorial, or a workshop. The business customer may change the quota of seats for each conference. The business customer may also change the quota of seats for each session. Reservation. A reservation is a temporary reservation of one or more seats. The ordering process creates reservations. When a registrant begins the ordering process, the system makes reservations for the number of seats requested by the registrant. These seats are then not available for other registrants to reserve. The reservations are held for n minutes during which the registrant can complete the ordering process by making a payment for those seats. If the registrant does not pay for the seats within n minutes, the system cancels the reservation and the seats become available to other registrants to reserve. Seat availability. Every conference tracks seat availability for each type of seat. Initially, all of the seats are available to reserve and purchase. When a seat is reserved, the number of available seats of that type is decremented. If the system cancels the reservation, the number of available seats of that type is incremented. The business customer defines the initial number of each seat type to be made available; this is an attribute of a conference. A conference owner may adjust the numbers for the individual seat types. Conference site. You can access every conference defined in the system by using a unique URL. Registrants can begin the ordering process from this site. Each of the terms defined here was formulated through active discussions between the development team and the domain experts. The following is a sample conversation between developers and domain experts that illustrates how the team arrived at a definition of the term attendee. Developer 1: Here’s an initial stab at a definition for attendee. “An attendee is someone who has paid to attend a conference. An attendee can interact with the system to perform tasks such as manage his agenda, print his badge, and provide feedback after the conference.” Domain Expert 1: Not all attendees will pay to attend the conference. For example, some conferences will have volunteer helpers, also speakers typically don’t pay. And, there may be some cases where an attendee gets a 100% discount. Domain Expert 1: Don’t forget that it’s not the attendee who pays; that’s done by the registrant. Developer 1: So we need to say that Attendees are people who are authorized to attend a conference? Developer 2: We need to be careful about the choice of words here. The term authorized will make some people think of security and authentication and authorization. Developer 1: How about entitled? Domain Expert 1: When the system performs tasks such as printing badges, it will need to know what type of attendee the badge is for. For example, speaker, volunteer, paid attendee, and so on. Or ders a nd R egistr ations Bounded Context 17 Developer 1: Now we have this as a definition that captures everything we’ve discussed. An attendee is someone who is entitled to attend a conference. An attendee can interact with the system to perform tasks such as manage his agenda, print his badge, and provide feedback after the conference. An attendee could also be a person who doesn’t pay to attend a conference such as a volunteer, speaker, or someone with a 100% discount. An attendee may have multiple associated attendee types (speaker, student, volunteer, track chair, and so on.) Requirements for creating orders A registrant is the person who reserves and pays for (orders) seats at a conference. Ordering is a two-stage process: first, the registrant reserves a number of seats and then pays for the seats to confirm the reservation. If registrant does not complete the payment, the seat reservations expire after a fixed period and the system makes the seats available for other registrants to reserve. Figure 1 shows some of the early UI mockups that the team used to explore the seat-ordering story. Figure 1 Ordering UI mockups 18 Jour ney thr ee A frequently cited advantage of the CQRS pattern is that it enables you to scale the read side and write side of the application independently to support the different usage patterns. In this bounded context, however, the number of read operations from the UI is not likely to hugely out-number the write operations: this bounded context focuses on registrants creating orders. Therefore, the read side and the write side are deployed to the same Windows Azure worker role rather than to two separate worker roles that could be scaled independently. “A value I think developers would benefit greatly from recognizing is the de-emphasis on the means and methods for persistence of objects in terms of relational storage. Teach them to avoid modeling the domain as if it was a relational store, and I think it will be easier to introduce and understand both domaindriven design (DDD) and CQRS.” —Josh Elster, CQRS Advisors Mail List These UI mockups helped the team in several ways, allowing them to: • Communicate the core team’s vision for the system to the graphic designers who are on an independent team at a thirdparty company. • Communicate the domain expert’s knowledge to the developers. • Refine the definition of terms in the ubiquitous language. • Explore “what if” questions about alternative scenarios and approaches. • Form the basis for the system’s suite of acceptance tests. Architecture The application is designed to deploy to Windows Azure. At this stage in the journey, the application consists of a web role that contains the ASP.NET MVC web application and a worker role that contains the message handlers and domain objects. The application uses a Windows Azure SQL Database instance for data storage, both on the write side and the read side. The application uses the Windows Azure Service Bus to provide its messaging infrastructure. While you are exploring and testing the solution, you can run it locally, either using the Windows Azure compute emulator or by running the MVC web application directly and running a console application that hosts the handlers and domain objects. When you run the application locally, you can use a local SQL Server Express database instead of SQL Database, and use a simple messaging infrastructure implemented in a SQL Server Express database. For more information about the options for running the application, see Appendix 1, “Release Notes.” Patterns and concepts The team decided to implement the first bounded context without using event sourcing in order to keep things simple. However, they did agree that if they later decided that event sourcing would bring specific benefits to this bounded context, then they would revisit this decision. For a description of how event sourcing relates to the CQRS pattern, see “Introducing Event Sourcing” in the Reference Guide. One of the important discussions the team had concerned the choice of aggregates and entities that they would implement. The following images from the team’s whiteboard illustrate some of their initial thoughts, and questions about the alternative approaches they could take with a simple conference seat reservation scenario to try and understand the pros and cons of alternative approaches. Or ders a nd R egistr ations Bounded Context 19 This scenario considers what happens when a registrant tries to book several seats at a conference. The system must: • Check that sufficient seats are available. • Record details of the registration. • Update the total number of seats booked for the conference. We deliberately kept the scenario simple to avoid distractions while the team examines the alternatives. These examples do not illustrate the final implementation of this bounded context. The first approach considered by the team, shown in Figure 2, uses two separate aggregates. Figure 2 Approach 1: Two separate aggregates These diagrams deliberately exclude details of how the system delivers commands and events through command and event handlers. The diagrams focus on the logical relationships between the aggregates in the domain. 20 Jour ney thr ee The term rehydration refers to the process of deserializing the aggregate instance from a data store. The numbers in the diagram correspond to the following steps: 1. The UI sends a command to register attendees X and Y for conference 157. The command is routed to a new Order aggregate. 2. The Order aggregate raises an event that reports that an order has been created. The event is routed to the SeatsAvailability aggregate. 3. The SeatsAvailability aggregate with an ID of 157 is rehydrated from the data store. 4. The SeatsAvailability aggregate updates its total number of seats booked. 5. The updated version of the SeatsAvailability aggregate is persisted to the data store. 6. The new Order aggregate, with an ID of 4239, is persisted to the data store. You could consider using the Memento pattern to handle the persistence and rehydration. Or ders a nd R egistr ations Bounded Context 21 The second approach considered by the team, shown in Figure 3, uses a single aggregate in place of two. Figure 3 Approach 2: A single aggregate The numbers in the diagram correspond to the following steps: 1. The UI sends a command to register Attendees X and Y for conference 157. The command is routed to the Conference aggregate with an ID of 157. 2. The Conference aggregate with an ID of 157 is rehydrated from the data store. 3. The Order entity validates the booking (it queries the SeatsAvailability entity to see if there are enough seats left), and then invokes the method to update the number of seats booked on the Conference entity. 4. The SeatsAvailability entity updates its total number of seats booked. 5. The updated version of the Conference aggregate is persisted to the data store. 22 Jour ney thr ee The third approach considered by the team, shown in Figure 4, uses a process manager to coordinate the interaction between two aggregates. Figure 4 Approach 3: Using a process manager The numbers in the diagram correspond to the following steps: 1. The UI sends a command to register Attendees X and Y for conference 157. The command is routed to a new Order aggregate. 2. The new Order aggregate, with an ID of 4239, is persisted to the data store. 3. The Order aggregate raises an event that is handled by the RegistrationProcessManager class. Or ders a nd R egistr ations Bounded Context 4. The RegistrationProcessManager class determines that a command should be sent to the SeatsAvailability aggregate with an ID of 157. 5. The SeatsAvailability aggregate is rehydrated from the data store. 6. The total number of seats booked is updated in the SeatsAvailability aggregate and it is persisted to the data store. For more information about process managers and sagas, see Chapter 6, “A Saga on Sagas” in the Reference Guide. The team identified the following questions about these approaches: • Where does the validation that there are sufficient seats for the registration take place: in the Order or SeatsAvailability aggregate? • Where are the transaction boundaries? • How does this model deal with concurrency issues when multiple registrants try to place orders simultaneously? • What are the aggregate roots? The following sections discuss these questions in relation to the three approaches considered by the team. Validation Before a registrant can reserve a seat, the system must check that there are enough seats available. Although logic in the UI can attempt to verify that there are sufficient seats available before it sends a command, the business logic in the domain must also perform the check; this is because the state may change between the time the UI performs the validation and the time that the system delivers the command to the aggregate in the domain. Process manager or saga? Initially the team referred to the RegistrationProcessManager class as a saga. However, after they reviewed the original definition of a saga from the paper “Sagas” by Hector Garcia-Molina and Kenneth Salem, they revised their decision. The key reasons for this are that the reservation process does not include explicit compensation steps, and does not need to be represented as a long-lived transaction. When we talk about UI validation here, we are talking about validation that the Model-View Controller (MVC) controller performs, not the browser. 23 24 Jour ney thr ee Undo is just one of many compensating actions that occur in real life. The compensating actions could even be outside of the system implementation and involve human actors: for example, a Contoso clerk or the business customer calls the registrant to tell them that an error was made and that they should ignore the last confirmation email they received from the Contoso system. In the first model, the validation must take place in either the Order or SeatsAvailability aggregate. If it is the former, the Order aggregate must discover the current seat availability from the SeatsAvailability aggregate before the reservation is made and before it raises the event. If it is the latter, the SeatsAvailability aggregate must somehow notify the Order aggregate that it cannot reserve the seats, and that the Order aggregate must undo (or compensate for) any work that it has completed so far. The second model behaves similarly, except that it is Order and SeatsAvailability entities cooperating within a Conference aggregate. In the third model, with the process manager, the aggregates exchange messages through the process manager about whether the registrant can make the reservation at the current time. All three models require entities to communicate about the validation process, but the third model with the process manager appears more complex than the other two. Transaction boundaries An aggregate, in the DDD approach, represents a consistency boundary. Therefore, the first model with two aggregates, and the third model with two aggregates and a process manager will involve two transactions: one when the system persists the new Order aggregate and one when the system persists the updated SeatsAvailability aggregate. The term consistency boundary refers to a boundary within which you can assume that all the elements remain consistent with each other all the time. To ensure the consistency of the system when a registrant creates an order, both transactions must succeed. To guarantee this, we must take steps to ensure that the system is eventually consistent by ensuring that the infrastructure reliably delivers messages to aggregates. In the second approach, which uses a single aggregate, we will only have a single transaction when a registrant makes an order. This appears to be the simplest approach of the three. Or ders a nd R egistr ations Bounded Context 25 Concurrency The registration process takes place in a multi-user environment where many registrants could attempt to purchase seats simultaneously. The team decided to use the Reservation pattern to address the concurrency issues in the registration process. In this scenario, this means that a registrant initially reserves seats (which are then unavailable to other registrants); if the registrant completes the payment within a timeout period, the system retains the reservation; otherwise the system cancels the reservation. This reservation system introduces the need for additional message types; for example, an event to report that a registrant has made a payment, or report that a timeout has occurred. This timeout also requires the system to incorporate a timer somewhere to track when reservations expire. Modeling this complex behavior with sequences of messages and the requirement for a timer is best done using a process manager. Aggregates and aggregate roots In the two models that have the Order aggregate and the SeatsAvailability aggregate, the team easily identified the entities that make up the aggregate, and the aggregate root. The choice is not so clear in the model with a single aggregate: it does not seem natural to access orders through a SeatsAvailability entity, or to access the seat availability through an Order entity. Creating a new entity to act as an aggregate root seems unnecessary. The team decided on the model that incorporated a process manager because this offers the best way to handle the concurrency requirements in this bounded context. Implementation details This section describes some of the significant features of the Orders and Registrations bounded context implementation. You may find it useful to have a copy of the code so you can follow along. You can download it from the Download center, or check the evolution of the code in the repository on github: mspnp/cqrs-journey-code. Do not expect the code samples to match the code in the reference implementation exactly. This chapter describes a step in the CQRS journey, the implementation may well change as we learn more and refactor the code. 26 Jour ney thr ee High-level architecture As we described in the previous section, the team initially decided to implement the reservations story in the conference management system using the CQRS pattern but without using event sourcing. Figure 5 shows the key elements of the implementation: an MVC web application, a data store implemented using a Windows Azure SQL Database instance, the read and write models, and some infrastructure components. We’ ll describe what goes on inside the read and write models later in this section. Figure 5 High-level architecture of the registrations bounded context Or ders a nd R egistr ations Bounded Context 27 The following sections relate to the numbers in Figure 5 and provide more detail about these elements of the architecture. 1. Querying the read model The ConferenceController class includes an action named Display that creates a view that contains information about a particular conference. This controller class queries the read model using the following code: public ActionResult Display(string conferenceCode) { var conference = this.GetConference(conferenceCode); return View(conference); } private Conference.Web.Public.Models.Conference GetConference(string conferenceCode) { var repo = this.repositoryFactory(); using (repo as IDisposable) { var conference = repo.Query() .First(c => c.Code == conferenceCode); var conferenceModel = new Conference.Web.Public.Models.Conference { Code = conference.Code, Name = conference.Name, Description = conference.Description }; return conferenceModel; } } The read model retrieves the information from the data store and returns it to the controller using a data transfer object (DTO) class. 28 Jour ney thr ee 2. Issuing commands The web application sends commands to the write model through a command bus. This command bus is an infrastructure element that provides reliable messaging. In this scenario, the bus delivers messages asynchronously and once only to a single recipient. The RegistrationController class can send a RegisterToConference command to the write model in response to user interaction. This command sends a request to register one or more seats at the conference. The RegistrationController class then polls the read model to discover whether the registration request succeeded. See the section “6. Polling the Read Model” below for more details. The following code sample shows how the RegistrationController sends a RegisterToConference command: var viewModel = this.UpdateViewModel(conferenceCode, contentModel); var command = new RegisterToConference { OrderId = viewModel.Id, ConferenceId = viewModel.ConferenceId, Seats = viewModel.Items.Select(x => new RegisterToConference.Seat { SeatTypeId = x.SeatTypeId, Quantity = x.Quantity }).ToList() }; this.commandBus.Send(command); All of the commands are sent asynchronously and do not expect return values. 3. Handling commands Command handlers register with the command bus; the command bus can then forward commands to the correct handler. The OrderCommandHandler class handles the RegisterToConference command sent from the UI. Typically, the handler is responsible for initiating any business logic in the domain and for persisting any state changes to the data store. The following code sample shows how the OrderCommandHandler class handles the RegisterToConference command: Or ders a nd R egistr ations Bounded Context 29 public void Handle(RegisterToConference command) { var repository = this.repositoryFactory(); using (repository as IDisposable) { var seats = command.Seats .Select(t => new OrderItem(t.SeatTypeId, t.Quantity)) .ToList(); var order = new Order( command.OrderId, Guid.NewGuid(), command.ConferenceId, seats); repository.Save(order); } } 4. Initiating business logic in the domain In the previous code sample, the OrderCommandHandler class creates a new Order instance. The Order entity is an aggregate root, and its constructor contains code to initiate the domain logic. See the section “Inside the Write Model” below for more details of what actions this aggregate root performs. 5. Persisting the changes In the previous code sample, the handler persists the new Order aggregate by calling the Save method in the repository class. This Save method also publishes any events raised by the Order aggregate on the command bus. 6. Polling the read model To provide feedback to the user, the UI must have a way to check whether the RegisterToConference command succeeded. Like all commands in the system, this command executes asynchronously and does not return a result. The UI queries the read model to check whether the command succeeded. The following code sample shows the initial implementation where the RegistrationController class polls the read model until either the system creates the order or a timeout occurs. The WaitUntilUpdated method polls the read-model until it finds either that the order has been persisted or it times out. 30 Jour ney thr ee [HttpPost] public ActionResult StartRegistration(string conferenceCode, OrderViewModel contentModel) { ... this.commandBus.Send(command); var draftOrder = this.WaitUntilUpdated(viewModel.Id); if (draftOrder != null) { if (draftOrder.State == "Booked") { return RedirectToAction( "SpecifyPaymentDetails", new { conferenceCode = conferenceCode, orderId = viewModel.Id }); } else if (draftOrder.State == "Rejected") { return View("ReservationRejected", viewModel); } } return View("ReservationUnknown", viewModel); } The team later replaced this mechanism for checking whether the system saves the order with an implementation of the Post-Redirect-Get pattern. The following code sample shows the new version of the StartRegistration action method. For more information about the Post-Redirect-Get pattern see the article Post/Redirect/Get on Wikipedia. Or ders a nd R egistr ations Bounded Context 31 [HttpPost] public ActionResult StartRegistration(string conferenceCode, OrderViewModel contentModel) { ... this.commandBus.Send(command); return RedirectToAction( "SpecifyRegistrantDetails", new { conferenceCode = conferenceCode, orderId = command.Id }); } The action method now redirects to the SpecifyRegistrantDetails view immediately after it sends the command. The following code sample shows how the SpecifyRegistrantDetails action polls for the order in the repository before returning a view. [HttpGet] public ActionResult SpecifyRegistrantDetails(string conferenceCode, Guid orderId) { var draftOrder = this.WaitUntilUpdated(orderId); ... } The advantages of this second approach, using the Post-Redirect-Get pattern instead of in the StartRegistration post action are that it works better with the browser’s forward and back navigation buttons, and that it gives the infrastructure more time to process the command before the MVC controller starts polling. Inside the write model Aggregates The following code sample shows the Order aggregate. public class Order : IAggregateRoot, IEventPublisher { public static class States { public const int Created = 0; public const int Booked = 1; public const int Rejected = 2; public const int Confirmed = 3; } 32 Jour ney thr ee private List events = new List (); ... public Guid Id { get; private set; } public Guid UserId { get; private set; } public Guid ConferenceId { get; private set; } public virtual ObservableCollection Lines { get; private set; } public int State { get; private set; } public IEnumerable Events { get { return this.events; } } public void MarkAsBooked() { if (this.State != States.Created) throw new InvalidOperationException(); this.State = States.Booked; } public void Reject() { if (this.State != States.Created) throw new InvalidOperationException(); this.State = States.Rejected; } } Or ders a nd R egistr ations Bounded Context 33 Notice how the properties of the class are not virtual. In the original version of this class, the properties Id, UserId, ConferenceId, and State were all marked as virtual. The following conversation between two developers explores this decision. Developer 1: I’m really convinced you should not make the property virtual, except if required by the object-relational mapping (ORM) layer. If this is just for testing purposes, entities and aggregate roots should never be tested using mocking. If you need mocking to test your entities, this is a clear smell that something is wrong in the design. Developer 2: I prefer to be open and extensible by default. You never know what needs may arise in the future, and making things virtual is hardly a cost. This is certainly controversial and a bit non-standard in .NET, but I think it’s OK. We may only need virtuals on lazy-loaded collections. Developer 1: Since CQRS usually makes the need for lazy load vanish, you should not need it either. This leads to even simpler code. Developer 2: CQRS does not dictate usage of event sourcing (ES), so if you’re using an aggregate root that contains an object graph, you’d need that anyway, right? Developer 1: This is not about ES, it’s about DDD. When your aggregate boundaries are right, you don’t need delay loading. Developer 2: To be clear, the aggregate boundary is here to group things that should change together for reasons of consistency. A lazy load would indicate that things that have been grouped together don’t really need this grouping. Developer 1: I agree. I have found that lazy-loading in the command side means I have it modeled wrong. If I don’t need the value in the command side, then it shouldn’t be there. In addition, I dislike virtuals unless they have an intended purpose (or some artificial requirement from an object-relational mapping (ORM) tool). In my opinion, it violates the Open-Closed principle: you have opened yourself up for modification in a variety of ways that may or may not be intended and where the repercussions might not be immediately discoverable, if at all. Developer 2: Our Order aggregate in the model has a list of Order Items. Surely we don’t need to load the lines to mark it as Booked? Do we have it modeled wrong there? Developer 1: Is the list of Order Items that long? If it is, the modeling may be wrong because you don’t necessarily need transactionality at that level. Often, doing a late round trip to get and updated Order Items can be more costly that loading them up front: you should evaluate the usual size of the collection and do some performance measurement. Make it simple first, optimize if needed. —Thanks to Jérémie Chassaing and Craig Wilson 34 Jour ney thr ee Aggregates and process managers Figure 6 shows the entities that exist in the write-side model. There are two aggregates, Order and SeatsAvailability, each one containing multiple entity types. Also there is a RegistrationProcessManager class to manage the interaction between the aggregates. The table in the Figure 6 shows how the process manager behaves given a current state and a particular type of incoming message. Or ders a nd R egistr ations Bounded Context 35 Figure 6 Domain objects in the write model The process of registering for a conference begins when the UI sends a RegisterToConference command. The infrastructure delivers this command to the Order aggregate. The result of this command is that the system creates a new Order instance, and that the new Order instance raises an OrderPlaced event. The following code sample from the constructor in the Order class shows this happening. Notice how the system uses GUIDs to identify the different entities. 36 Jour ney thr ee public Order(Guid id, Guid userId, Guid conferenceId, IEnumerable lines) { this.Id = id; this.UserId = userId; this.ConferenceId = conferenceId; this.Lines = new ObservableCollection (items); this.events.Add( new OrderPlaced { OrderId = this.Id, ConferenceId = this.ConferenceId, UserId = this.UserId, Seats = this.Lines.Select(x => new OrderPlaced.Seat { SeatTypeId = x.SeatTypeId, Quantity = x.Quantity }).ToArray() }); } To see how the infrastructure elements deliver commands and events, see Figure 7. The system creates a new RegistrationProcessManager instance to manage the new order. The following code sample from the RegistrationProcessManager class shows how the process manager handles the event. Or ders a nd R egistr ations Bounded Context public void Handle(OrderPlaced message) { if (this.State == ProcessState.NotStarted) { this.OrderId = message.OrderId; this.ReservationId = Guid.NewGuid(); this.State = ProcessState.AwaitingReservationConfirmation; this.AddCommand( new MakeSeatReservation { ConferenceId = message.ConferenceId, ReservationId = this.ReservationId, NumberOfSeats = message.Items.Sum(x => x.Quantity) }); } else { throw new InvalidOperationException(); } } The code sample shows how the process manager changes its state and sends a new MakeSeatReservation command that the SeatsAvailability aggregate handles. The code sample also illustrates how the process manager is implemented as a state machine that receives messages, changes its state, and sends new messages. When the SeatsAvailability aggregate receives a MakeReservation command, it makes a reservation if there are enough available seats. The following code sample shows how the SeatsAvailability class raises different events depending on whether or not there are sufficient seats. Notice how we generate a new globally unique identifier (GUID) to identify the new reservation. We use these GUIDs to correlate messages to the correct process manager and aggregate instances. 37 38 Jour ney thr ee public void MakeReservation(Guid reservationId, int numberOfSeats) { if (numberOfSeats > this.RemainingSeats) { this.events.Add(new ReservationRejected { ReservationId = reservationId, ConferenceId = this.Id }); } else { this.PendingReservations.Add(new Reservation(reservationId, numberOfSeats)); this.RemainingSeats -= numberOfSeats; this.events.Add(new ReservationAccepted { ReservationId = reservationId, ConferenceId = this.Id }); } } The RegistrationProcessManager class handles the ReservationAccepted and ReservationRejected events. This reservation is a temporary reservation for seats to give the user the opportunity to make a payment. The process manager is responsible for releasing the reservation when either the purchase is complete, or the reservation timeout period expires. The following code sample shows how the process manager handles these two messages. public void Handle(ReservationAccepted message) { if (this.State == ProcessState.AwaitingReservationConfirmation) { this.State = ProcessState.AwaitingPayment; this.AddCommand(new MarkOrderAsBooked { OrderId = this.OrderId }); this.commands.Add( new Envelope ( new ExpireOrder { OrderId = this.OrderId, ConferenceId = message.ConferenceId }) Or ders a nd R egistr ations Bounded Context { Delay = TimeSpan.FromMinutes(15), }); } else { throw new InvalidOperationException(); } } public void Handle(ReservationRejected message) { if (this.State == ProcessState.AwaitingReservationConfirmation) { this.State = ProcessState.Completed; this.AddCommand(new RejectOrder { OrderId = this.OrderId }); } else { throw new InvalidOperationException(); } } If the reservation is accepted, the process manager starts a timer running by sending an ExpireOrder command to itself, and sends a MarkOrderAsBooked command to the Order aggregate. Otherwise, it sends a ReservationRejected message back to the Order aggregate. The previous code sample shows how the process manager sends the ExpireOrder command. The infrastructure is responsible for holding the message in a queue for the delay of fifteen minutes. You can examine the code in the Order, SeatsAvailability, and RegistrationProcessManager classes to see how the other message handlers are implemented. They all follow the same pattern: receive a message, perform some logic, and send a message. The code samples shown in this chapter are from an early version of the conference management system. The next chapter shows how the design and implementation evolved as the team explored the domain and learned more about the CQRS pattern. 39 40 Jour ney thr ee Infrastructure The sequence diagram in Figure 7 shows how the infrastructure elements interact with the domain objects to deliver messages. Figure 7 Infrastructure sequence diagram A typical interaction begins when an MVC controller in the UI sends a message using the command bus. The message sender invokes the Send method on the command bus asynchronously. The command bus then stores the message until the message recipient retrieves the message and forwards it to the appropriate handler. The system includes a number of command handlers that register with the command bus to handle specific types of commands. For example, the OrderCommandHandler class defines handler methods for the RegisterToConference, MarkOrderAsBooked, and RejectOrder commands. The following code sample shows the handler method for the MarkOrderAsBooked command. Handler methods are responsible for locating the correct aggregate instance, calling methods on that instance, and then saving that instance. Or ders a nd R egistr ations Bounded Context public void Handle(MarkOrderAsBooked command) { var repository = this.repositoryFactory(); using (repository as IDisposable) { var order = repository.Find (command.OrderId); if (order != null) { order.MarkAsBooked(); repository.Save(order); } } } The class that implements the IRepository interface is responsible for persisting the aggregate and publishing any events raised by the aggregate on the event bus, all as part of a transaction. The only event subscriber in the reservations bounded context is the RegistrationProcessManager class. Its router subscribes to the event bus to handle specific events, as shown in the following code sample from the RegistrationProcessManager class. We use the term handler to refer to the classes that handle commands and events and forward them to aggregate instances, and the term router to refer to the classes that handle events and commands and forward them to process manager instances. The team later discovered an issue with this when they tried to use Windows Azure Service Bus as the messaging infrastructure. Windows Azure Service Bus does not support distributed transactions with databases. For a discussion of this issue, see Chapter 5, “Preparing for the V1 Release.” 41 42 Jour ney thr ee public void Handle(ReservationAccepted @event) { var repo = this.repositoryFactory.Invoke(); using (repo as IDisposable) { lock (lockObject) { var process = repo.Find (@event.ReservationId); process.Handle(@event); repo.Save(process); } } } Typically, an event handler method loads a process manager instance, passes the event to the process manager, and then persists the process manager instance. In this case, the IRepository instance is responsible for persisting the process manager instance and for sending any commands from the process manager instance to the command bus. Using the Windows Azure Service Bus The team at Contoso decided to use the Windows Azure Service Bus because it offers outof-the-box support for the messaging scenarios in the conference management system. This minimizes the amount of code that the team needs to write, and provides for a robust, scalable messaging infrastructure. The team plans to use features such as duplicate message detection and guaranteed message ordering. For a summary of the differences between Windows Azure Service Bus and Windows Azure Queues, see “Windows Azure Queues and Windows Azure Service Bus Queues - Compared and Contrasted” on MSDN. To transport command and event messages, the team decided to use the Windows Azure Service Bus to provide the low-level messaging infrastructure. This section describes how the system uses the Windows Azure Service Bus and some of the alternatives and trade-offs the team considered during the design phase. Figure 8 shows how both command and event messages flow through the system. MVC controllers in the UI and domain objects use CommandBus and EventBus instances to send BrokeredMessage messages to one of the two topics in the Windows Azure Service Bus. To receive messages, the handler classes register with the CommandProcessor and EventProcessor instances that retrieve messages from the topics by using the SubscriptionReceiver class. The CommandProcessor class determines which single handler should receive a command message; the EventProcessor class determines which handlers should receive an event message. The handler instances are responsible for invoking methods on the domain objects. A Windows Azure Service Bus topic can have multiple subscribers. The Windows Azure Service Bus delivers messages sent to a topic to all its subscribers. Therefore, one message can have multiple recipients. Or ders a nd R egistr ations Bounded Context 43 Figure 8 Message flows through a Windows Azure Service Bus topic In the initial implementation, the CommandBus and EventBus classes are very similar. The only difference between the Send method and the Publish method is that the Send method expects the message to be wrapped in an Envelope class. The Envelope class enables the sender to specify a time delay for the message delivery. Events can have multiple recipients. In the example shown in Figure 8, the ReservationRejected event is sent to the RegistrationProcessManager, the WaitListProcessManager, and one other destination. The EventProcessor class identifies the list of handlers to receive the event by examining its list of registered handlers. A command has only one recipient. In Figure 8, the MakeSeatReservation is sent to the SeatsAvailability aggregate. There is just a single handler registered for this subscription. The CommandProcessor class identifies the handler to receive the command by examining its list of registered handlers. 44 Jour ney thr ee This implementation gives rise to a number of questions: • How do you limit delivery of a command to a single recipient? • Why have separate CommandBus and EventBus classes if they are so similar? • How scalable is this approach? • How robust is this approach? • What is the granularity of a topic and a subscription? • How are commands and events serialized? The following sections discuss these questions. A separate issue is to ensure that the handler retrieves commands from the topic and processes them only once. You must ensure either that the command is idempotent, or that the system guarantees to process the command only once. The team will address this issue in a later stage of the journey. See Chapter 7, “Adding Resilience and Optimizing Performance” for more information. Delivering a command to a single recipient This discussion assumes you that you have a basic understanding of the differences between Windows Azure Service Bus queues and topics. For an introduction to Windows Azure Service Bus, see “Technologies Used in the Reference Implementation” in the Reference Guide. With the implementation shown in Figure 8, two things are necessary to ensure that a single handler handles a command message. First, there should only be a single subscription to the conference/ commands topic in Windows Azure Service Bus; remember that a Windows Azure Service Bus topic may have multiple subscribers. Second, the CommandProcessor should invoke a single handler for each command message that it receives. There is no way in Windows Azure Service Bus to restrict a topic to a single subscription; therefore, the developers must be careful to create just a single subscription on a topic that is delivering commands. It is possible to have multiple SubscriptionReceiver instances running, perhaps in multiple worker role instances. If multiple SubscriptionReceiver instances can receive messages from the same topic subscription, then the first one to call the Receive method on the SubscriptionClient object will get and handle the command. An alternative approach is to use a Windows Azure Service Bus queue in place of a topic for delivering command messages. Windows Azure Service Bus queues differ from topics in that they are designed to deliver messages to a single recipient instead of to multiple recipients through multiple subscriptions. The developers plan to evaluate this option in more detail with the intention of implementing this approach later in the project. Or ders a nd R egistr ations Bounded Context 45 The following code sample from the SubscriptionReceiver class shows how it receives a message from the topic subscription. private SubscriptionClient client; ... private void ReceiveMessages(CancellationToken cancellationToken) { while (!cancellationToken.IsCancellationRequested) { BrokeredMessage message = null; try { message = this.receiveRetryPolicy .ExecuteAction(this.DoReceiveMessage); } catch (Exception e) { Trace.TraceError( "An unrecoverable error occurred while trying to receive" + "a new message:\r\n{0}", e); throw; } try { if (message == null) { Thread.Sleep(100); continue; } this.MessageReceived(this, new BrokeredMessageEventArgs(message)); } finally { if (message != null) { message.Dispose(); } 46 Jour ney thr ee } } } protected virtual BrokeredMessage DoReceiveMessage() { return this.client.Receive(TimeSpan.FromSeconds(10)); } This code sample shows how the system uses the Transient Fault Handling Application Block to retrieve messages reliably from the topic. The Windows Azure Service Bus SubscriptionClient class uses a peek/lock technique to retrieve a message from a subscription. In the code sample, the Receive method locks the message on the subscription. While the message is locked, other clients cannot see it. The Receive method then tries to process the message. If the client processes the message successfully, it calls the Complete method; this deletes the message from the subscription. Otherwise, if the client fails to process the message successfully, it calls the Abandon method; this releases the lock on the message and the same, or a different client can then receive it. If the client does not call either the Complete or Abandon methods within a fixed time, the lock on the message is released. The MessageReceived event passes a reference to the SubscriptionReceiver instance so that the handler can call either the Complete or Abandon methods when it processes the message. The following code sample from the MessageProcessor class shows how to call the Complete and Abandon methods using the BrokeredMessage instance passed as a parameter to the MessageReceived event. private void OnMessageReceived(object sender, BrokeredMessageEventArgs args) { var message = args.Message; object payload; using (var stream = message.GetBody ()) using (var reader = new StreamReader(stream)) { Or ders a nd R egistr ations Bounded Context payload = this.serializer.Deserialize(reader); } try { ... ProcessMessage(payload); ... } catch (Exception e) { if (args.Message.DeliveryCount > MaxProcessingRetries) { Trace.TraceWarning( "An error occurred while processing a new message and" + "will be dead-lettered:\r\n{0}", e); message.SafeDeadLetter(e.Message, e.ToString()); } else { Trace.TraceWarning( "An error occurred while processing a new message and" + "will be abandoned:\r\n{0}", e); message.SafeAbandon(); } return; } Trace.TraceInformation("The message has been processed and will be completed."); message.SafeComplete(); } This example uses an extension method to invoke the Complete and Abandon methods of the BrokeredMessage reliably using the Transient Fault Handling Application Block. 47 48 Jour ney thr ee Why have separate CommandBus and EventBus classes? Although at this early stage in the development of the conference management system the implementations of the CommandBus and EventBus classes are very similar and you may wonder why we have both, the team anticipates that they will diverge in the future. There may be differences in how we invoke handlers and what context we capture for them: commands may want to capture additional runtime state, whereas events typically don’t need to. Because of these potential future differences, I didn’t want to unify the implementations. I’ve been there before and ended up splitting them when further requirements came in. There are no costs associated with having multiple topics, subscriptions, or queues. Windows Azure Service Bus usage is billed based on the number of messages sent and the amount of data transferred out of a Windows Azure sub-region. How scalable is this approach? With this approach, you can run multiple instances of the SubscriptionReceiver class and the various handlers in different Windows Azure worker role instances, which enables you to scale out your solution. You can also have multiple instances of the CommandBus, EventBus, and TopicSender classes in different Windows Azure worker role instances. For information about scaling the Windows Azure Service Bus infrastructure, see Best Practices for Performance Improvements Using Service Bus Brokered Messaging on MSDN. How robust is this approach? This approach uses the brokered messaging option of the Windows Azure Service Bus to provide asynchronous messaging. The Service Bus reliably stores messages until consumers connect and retrieve their messages. Also, the peek/lock approach to retrieving messages from a queue or topic subscription adds reliability in the scenario in which a message consumer fails while it is processing the message. If a consumer fails before it calls the Complete method, the message is still available for processing when the consumer restarts. What is the granularity of a topic and a subscription? The current implementation uses a single topic (conference/commands) for all commands within the system, and a single topic (conference/ events) for all events within the system. There is a single subscription for each topic, and each subscription receives all of the messages published to the topic. It is the responsibility of the CommandProcessor and EventProcessor classes to deliver the messages to the correct handlers. In the future, the team will examine the options of using multiple topics—for example, using a separate command topic for each bounded context; and multiple subscriptions—such as one per event type. These alternatives may simplify the code and facilitate scaling of the application across multiple worker roles. Or ders a nd R egistr ations Bounded Context How are commands and events serialized? The Contoso Conference Management System uses the Json.NET serializer. For details on how the application uses this serializer, see “Technologies Used in the Reference Implementation” in the Reference Guide. Impact on testing Because this was the first bounded context the team tackled, one of the key concerns was how to approach testing given that the team wanted to adopt a test-driven development approach. The following conversation between two developers about how to do TDD when they are implementing the CQRS pattern without event sourcing summarizes their thoughts: 49 “You should consider whether you always need to use the Windows Azure Service Bus for commands. Commands are typically used within a bounded context and you may not need to send them across a process boundary (on the write side you may not need additional tiers), in which case you could use an in memory queue to deliver your commands.” —Greg Young, conversation with the patterns & practices team Developer 1: If we were using event sourcing, it would be easy to use a TDD approach when we were creating our domain objects. The input to the test would be a command (that perhaps originated in the UI), and we could then test that the domain object fires the expected events. However if we’re not using event sourcing, we don’t have any events: the behavior of the domain object is to persist its changes in data store through an ORM layer. Developer 2: So why don’t we raise events anyway? Just because we’re not using event sourcing doesn’t mean that our domain objects can’t raise events. We can then design our tests in the usual way to check for the correct events firing in response to a command. Developer 1: Isn’t that just making things more complicated than they need to be? One of the motivations for using CQRS is to simplify things! We now have domain objects that need to persist their state using an ORM layer and raise events that report on what they have persisted just so we can run our unit tests. Developer 2: I see what you mean. Developer 1: Perhaps we’re getting stuck on how we’re doing the tests. Maybe instead of designing our tests based on the expected behavior of the domain objects, we should think about testing the state of the domain objects after they’ve processed a command. Developer 2: That should be easy to do; after all, the domain objects will have all of the data we want to check stored in properties so that the ORM can persist the right information to the store. Developer 1: So we really just need to think about a different style of testing in this scenario. 50 Jour ney thr ee Developer 2: There is another aspect of this we’ll need to consider: we might have a set of tests that we can use to test our domain objects, and all of those tests might be passing. We might also have a set of tests to verify that our ORM layer can save and retrieve objects successfully. However, we will also have to test that our domain objects function correctly when we run them against the ORM layer. It’s possible that a domain object performs the correct business logic, but can’t properly persist its state, perhaps because of a problem related to how the ORM handles specific data types. For more information about the two approaches to testing discussed here, see Martin Fowler’s article “Mocks Aren’t Stubs” and “Point/ Counterpoint” by Steve Freeman, Nat Pryce, and Joshua Kerievsky. The tests included in the solution are written using xUnit.net. The following code sample shows two examples of tests written using the behavioral approach discussed above. These are the tests we started with, but we then replaced them with state-based tests. public SeatsAvailability given_available_seats() { var sut = new SeatsAvailability(SeatTypeId); sut.AddSeats(10); return sut; } [TestMethod] public void when_reserving_less_seats_than_total_then_succeeds() { var sut = this.given_available_seats(); sut.MakeReservation(Guid.NewGuid(), 4); } [TestMethod] [ExpectedException(typeof(ArgumentOutOfRangeException))] public void when_reserving_more_seats_than_total_then_fails() { var sut = this.given_available_seats(); sut.MakeReservation(Guid.NewGuid(), 11); } Or ders a nd R egistr ations Bounded Context 51 These two tests work together to verify the behavior of the SeatsAvailability aggregate. In the first test, the expected behavior is that the MakeReservation method succeeds and does not throw an exception. In the second test, the expected behavior is for the MakeReservation method to throw an exception because there are not enough free seats available to complete the reservation. It is difficult to test the behavior in any other way without the aggregate raising events. For example, if you tried to test the behavior by checking that the correct call is made to persist the aggregate to the data store, the test becomes coupled to the data store implementation (which is a smell); if you want to change the data store implementation, you will need to change the tests on the aggregates in the domain model. The following code sample shows an example of a test written using the state of the objects under test. This style of test is the one used in the project. public class given_available_seats { private static readonly Guid SeatTypeId = Guid.NewGuid(); private SeatsAvailability sut; private IPersistenceProvider sutProvider; protected given_available_seats(IPersistenceProvider sutProvider) { this.sutProvider = sutProvider; this.sut = new SeatsAvailability(SeatTypeId); this.sut.AddSeats(10); this.sut = this.sutProvider.PersistReload(this.sut); } public given_available_seats() : this(new NoPersistenceProvider()) { } [Fact] public void when_reserving_less_seats_than_total_then_seats_become_unavailable() { this.sut.MakeReservation(Guid.NewGuid(), 4); this.sut = this.sutProvider.PersistReload(this.sut); Assert.Equal(6, this.sut.RemainingSeats); } 52 Jour ney thr ee [Fact] public void when_reserving_more_seats_than_total_then_rejects() { var id = Guid.NewGuid(); sut.MakeReservation(id, 11); Assert.Equal(1, sut.Events.Count()); Assert.Equal(id, ((ReservationRejected)sut.Events.Single()).ReservationId); } } The two tests shown here test the state of the SeatsAvailability aggregate after invoking the MakeReservation method. The first test tests the scenario in which there are enough seats available. The second test tests the scenario in which there are not enough seats available. This second test can make use of the behavior of the SeatsAvailability aggregate because the aggregate does raise an event if it rejects a reservation. Summary In the first stage in our journey, we explored some of the basics of implementing the CQRS pattern and made some preparations for the next stages. The next chapter describes how we extended and enhanced the work already completed by adding more features and functionality to the Orders and Registrations bounded context. We will also look at some additional testing techniques to see how they might help us on our journey. More information All links in this book are accessible from the book’s online bibliography available at: http://msdn.microsoft.com/en-us/library/jj619274. Journey 4: Extending and Enhancing the Orders and Registrations Bounded Context Further exploration of the Orders and Registrations bounded context. “I see that it is by no means useless to travel, if a man wants to see something new.” Jules Verne, Around the World in Eighty Days Changes to the bounded context The previous chapter described the Orders and Registrations bounded context in some detail. This chapter describes some changes that the team made in this bounded context during the second stage of our CQRS journey. The specific topics described in this chapter include: • Improvements to the way message correlation works with the RegistrationProcessManager class. This illustrates how aggregate instances within the bounded context can interact in a complex manner. • Implementing a record locator to enable a registrant to retrieve an order that she saved during a previous session. This illustrates adding some additional logic to the write side that enables you to locate an aggregate instance without knowing its unique ID. • Adding a countdown timer to the UI to enable a registrant to track how much longer they have to complete an order. This illustrates enhancements to the write side to support the display of rich information in the UI. • Supporting orders for multiple seat types simultaneously. For example, a registrant requests five seats for a preconference event and eight seats for the full conference. This requires more complex business logic on the write side. • CQRS command validation. This illustrates how to make use of the model validation feature in MVC to validate your CQRS commands before you send them to the domain. The Contoso Conference Management System described in this chapter is not the final version of the system. This guidance describes a journey, so some of the design decisions and implementation details change in later steps in the journey. These changes are described in subsequent chapters. Working definitions for this chapter This chapter uses a number of terms, which we will describe next. For more detail, and possible alternative definitions, see Chapter 4, “A CQRS and ES Deep Dive,” in the Reference Guide. 53 54 Jour ney four From the business perspective it was important for us to be as user-friendly as possible: we don’t want to block or unnecessarily burden anyone who is trying to register for a conference. Therefore, we have no requirement for a user to create an account in the system prior to registration, especially since users must enter most of their information in a standard checkout process anyway. Command. A command is a request for the system to perform an action that changes the state of the system. Commands are imperatives; for example, MakeSeatReservation. In this bounded context, commands originate from either the UI as a result of a user initiating a request, or from a process manager when the process manager is directing an aggregate to perform an action. A single recipient processes a command. A command bus transports commands that command handlers then dispatch to aggregates. Sending a command is an asynchronous operation with no return value. Event. An event, such as OrderConfirmed, describes something that has happened in the system, typically as a result of a command. Aggregates in the domain model raise events. Multiple subscribers can handle a specific event. Aggregates publish events to an event bus; handlers register for specific types of events on the event bus and then deliver the events to the subscriber. In this bounded context, the only subscriber is a process manager. Process manager. In this bounded context, a process manager is a class that coordinates the behavior of the aggregates in the domain. A process manager subscribes to the events that the aggregates raise, and then follows a simple set of rules to determine which command or commands to send. The process manager does not contain any business logic, only logic to determine the next command to send. The process manager is implemented as a state machine, so when the process manager responds to an event, it can change its internal state in addition to sending a new command. The process manager in this bounded context can receive commands as well as subscribe to events. Our process manager is an implementation of the Process Manager pattern defined on pages 312 to 321 in the book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf (Addison-Wesley Professional, 2003). User stories This chapter discusses the implementation of two user stories in addition to describing some changes and enhancements to the Orders and Registrations bounded context. Implement a login using a record locator When a registrant creates an order for seats at a conference, the system generates a five-character order access code and sends it to the registrant by email. The registrant can use her email address and the order access code on the conference web site as a record locator to retrieve the order from the system at a later date. The registrant may wish to retrieve the order to review it, or to complete the registration process by assigning attendees to seats. E xtending a nd Enh a ncing the Or ders a nd R egistr ations Bounded Context 55 Tell the registrant how much time remains to complete an order When a registrant creates an order, the system reserves the seats requested by the registrant until the order is complete or the reservations expire. To complete an order, the registrant must submit her details, such as name and email address, and make a successful payment. To help the registrant, the system displays a countdown timer to inform her how much time remains to complete the order before the seat reservations expire. Enable a registrant to create an order that includes multiple seat types When a registrant creates an order, she may request different numbers of different seat types. For example, a registrant may request five seats for the full conference and three seats for the preconference workshop. Architecture The application is designed to deploy to Windows Azure. At this stage in the journey, the application consists of a web role that contains the ASP.NET MVC web application and a worker role that contains the message handlers and domain objects. The application uses Windows Azure SQL Database (SQL Database) instances for data storage, both on the write side and the read side. The application uses the Windows Azure Service Bus to provide its messaging infrastructure. Figure 1 shows this high-level architecture. Figure 1 Contoso Conference Management System high-level architecture 56 Jour ney four While you are exploring and testing the solution, you can run it locally, either using the Windows Azure compute emulator or by running the MVC web application directly and running a console application that hosts the handlers and domain objects. When you run the application locally, you can use a local SQL Server Express database instead of SQL Database, and use a simple messaging infrastructure implemented in a SQL Server Express database. For more information about the options for running the application, see Appendix 1, “Release Notes.” Patterns and concepts This section describes some of the key areas of the application that the team visited during this stage of the journey and introduces some of the challenges met by the team when we addressed these areas. Record locators The system uses access codes instead of passwords so the registrant is not forced to set up an account with the system. Many registrants may use the system only once, so there is no need to create a permanent account with a user ID and a password. The system needs to be able to retrieve order information quickly based on the registrant’s email address and access code. To provide a minimum level of security, the access codes that the system generates should not be predictable, and the order information that registrants can retrieve should not contain any sensitive information. Querying the read side The previous chapter focused on the write-side model and implementation; in this chapter we’ll explore the read-side implementation in more detail. In particular, we’ll explain how we implemented the read model and the querying mechanism from the MVC controllers. In this initial exploration of the CQRS pattern, the team decided to use SQL views in the database as the underlying source of the data queried by the MVC controllers on the read side. To minimize the work that the queries on the read side must perform, these SQL views provide a denormalized version of the data. These views currently exist in the same database as the normalized tables that the write model uses. The team will split the database into two and explore options for pushing changes from the normalized write side to the denormalized read side in a later stage of the journey. For an example of using Windows Azure blob storage instead of SQL tables for storing the readside data, see the SeatAssignmentsViewModelGenerator class. E xtending a nd Enh a ncing the Or ders a nd R egistr ations Bounded Context Storing denormalized views in a database One common option for storing the read-side data is to use a set of relational database tables to hold the denormalized views. You should optimize the read side for fast reads, so there is typically no benefit in storing normalized data because this will require complex queries to construct the data for the client. This implies that goals for the read side should be to keep the queries as simple as possible, and to structure the tables in the database in such a way that they can be read quickly and efficiently. An important area for consideration is the interface whereby a client such as an MVC controller action submits a query to the readside model. Application scalability and a responsive UI are often explicit goals when people choose to implement the CQRS pattern. Optimizing the read side to provide fast responses to queries while keeping resource utilization low will help you to achieve these goals. Figure 2 The read side storing data in a relational database In Figure 2, a client, such as an MVC controller action, invokes a method on the ViewRepository class to request the data it needs. The ViewRepository class in turn runs a query against the denormalized data in the database. The team at Contoso evaluated two approaches to implementing the ViewRepository class: using the IQueryable interface and using non-generic data access objects (DAOs). Using the IQueryable interface One approach to consider for the ViewRepository class is to have it return an IQueryable instance that enables the client to use languageintegrated query (LINQ) to specify its query. It is very easy to return an IQueryable instance from many ORMs such as Entity Framework or NHibernate. The following code snippet illustrates how the client can submit such queries. A normalized database schema can fail to provide adequate response times because of the excessive table JOIN operations. Despite advances in relational database technology, a JOIN operation is still very expensive compared to a single-table read. The Repository pattern mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects. For more info see Martin Fowler, Catalog of Patterns of Enterprise Application Architecture, Repository. 57 58 Jour ney four var ordersummary = repository .Query () .Where(LINQ query to retrieve order summary); var orderdetails = repository .Query () .Where(LINQ query to retrieve order details); This approach has a number of advantages: Simplicity • This approach uses a thin abstraction layer over the underlying database. Many ORMs support this approach and it minimizes the amount of code that you must write. • You only need to define a single repository and a single Query method. • You don’t need a separate query object. On the read side, the queries should be simple because you have already denormalized the data from the write side to support the read-side clients. • You can make use of Language-Integrated Query (LINQ) to provide support for features such as filtering, paging, and sorting on the client. In the RI, using Entity Framework, we didn’t need to write any code at all to expose the IQueryable instance. We also had just a single ViewRepository class. Testability • You can use LINQ to Objects for mocking. There are possible objections to this approach including that: • It is not easy to replace the data store with a non-relational database that does not expose an IQueryable object. However, you can choose to implement the read model differently in each bounded context using an approach that is appropriate to that bounded context. • The client might abuse the IQueryable interface by performing operations that can be done more efficiently as a part of the denormalization process. You should ensure that the denormalized data fully meets the requirements of the clients. • Using the IQueryable interface hides the queries away. However, since you denormalize the data from the write side, the queries against the relational database tables are unlikely to be complex. • It’s hard to know if your integration tests cover all the different uses of the Query method. E xtending a nd Enh a ncing the Or ders a nd R egistr ations Bounded Context 59 Using non-generic DAOs An alternative approach is to have the ViewRepository expose custom Find and Get methods, as shown in the following code snippets. var ordersummary = dao.FindAllSummarizedOrders(userId); var orderdetails = dao.GetOrderDetails(orderId); You could also choose to use different DAO classes. This would make it easier to access different data sources. var ordersummary = OrderSummaryDAO.FindAll(userId); var orderdetails = OrderDetailsDAO.Get(orderId); This approach has a number of advantages: Simplicity • Dependencies are clearer for the client. For example, the client references an explicit IOrderSummaryDAO instance rather than a generic IViewRepository instance. • For the majority of queries, there are only one or two predefined ways to access the object. Different queries typically return different projections. Flexibility • The Get and Find methods hide details such as the partitioning of the data store and the data access methods such as an object relational mapping (ORM) or executing SQL code explicitly. This makes it easier to change these choices in the future. • The Get and Find methods could use an ORM, LINQ, and the IQueryable interface behind the scenes to get the data from the data store. This is a choice that you could make on a methodby-method basis. Performance • You can easily optimize the queries that the Find and Get methods run. • The data access layer executes all queries. There is no risk that the client MVC controller action tries to run complex and inefficient LINQ queries against the data source. Testability • It is easier to specify unit tests for the Find and Get methods than to create suitable unit tests for the range of possible LINQ queries that a client could specify. Maintainability • All of the queries are defined in the same location, the DAO classes, making it easier to modify the system consistently. 60 Jour ney four Possible objections to this approach include: • Using the IQueryable interface makes it much easier to use grids that support features such as paging, filtering, and sorting in the UI. However, if the developers are aware of this downside and are committed to delivering a task-based UI, then this should not be an issue. The team decided to adopt the second approach because of the clarity it brings to the code; in this context, we did not see any significant advantage in the simplicity of the first approach. For examples, see the ConferenceDao and OrderDao classes in the Registration project. Making information about partially fulfilled orders available to the read side The UI displays data about orders that it obtains by querying the model on the read side. Part of the data that the UI displays to the registrant is information about partially fulfilled orders: for each seat type in the order, the number of seats requested and the number of seats that are available. This is temporary data that the system only uses while the registrant is creating the order using the UI; the business only needs to store information about seats that were actually purchased, not the difference between what the registrant requested and what the registrant purchased. The consequence of this is that the information about how many seats the registrant requested only needs to exist in the model on the read side. A further consequence is that the underlying storage on the read side cannot be simple SQL views because it includes data that is not stored in the underlying table storage on the write side. Therefore, you must pass this information to the read side using events. Figure 3 shows all the commands and events that the Order and SeatsAvailability aggregates use and how the Order aggregate pushes changes to the read side by raising events. You can’t store this information in an HTTP session because the registrant may leave the site between requesting the seats and completing the order. E xtending a nd Enh a ncing the Or ders a nd R egistr ations Bounded Context Figure 3 The new architecture of the reservation process The OrderViewModelGenerator class handles the OrderPlaced, OrderUpdated, OrderPartiallyReserved, OrderRegistrantAssigned, and OrderReservationCompleted events and uses DraftOrder and DraftOrderItem instances to persist changes to the view tables. CQRS command validation When you implement the write model, you should try to ensure that commands very rarely fail. This gives the best user experience, and makes it much easier to implement the asynchronous behavior in your application. One approach, adopted by the team, is to use the model validation features in ASP.NET MVC. If you look ahead to Chapter 5, “Preparing for the V1 Release,” you’ll see that the team extended the use of events and migrated the Orders and Registrations bounded context to use event sourcing. 61 62 Jour ney four The Transient Fault Handling Application Block from Microsoft patterns & practices is designed to make it easier to implement consistent retry behavior for any transient faults. It comes with a set of built-in detection strategies for Windows Azure SQL Database, Windows Azure storage, Windows Azure Caching, and Windows Azure Service Bus, and it also allows you to define your own strategies. Similarly, it comes with a set of handy built-in retry policies and supports custom ones. For more information, see The Transient Fault Handling Application Block. You should be careful to distinguish between errors and business failures. Examples of errors include: • A message is not delivered due to a failure in the messaging infrastructure. • Data is not persisted due to a connectivity problem with the database. In many cases, especially in the cloud, you can handle these errors by retrying the operation. A business failure should have a predetermined business response. For example: • If the system cannot reserve a seat because there are no seats left, then it should add the request to a wait list. • If a credit card payment fails, the user should be given the chance to either try a different card, or set up payment by invoice. The countdown timer and the read model The countdown timer that displays how much time remains to complete the order to the registrant is part of the business data in the system, and not just a part of the infrastructure. When a registrant creates an order and reserves seats, the countdown begins. The countdown continues, even if the registrant leaves the conference website. The UI must be able to display the correct countdown value if the registrant returns to the site; therefore, the reservation expiry time is a part of the data that is available from the read model. Implementation details This section describes some of the significant features of the implementation of the Orders and Registrations bounded context. You may find it useful to have a copy of the code so you can follow along. You can download a copy from the Download center, or check the evolution of the code in the repository on GitHub: https://github.com/ mspnp/cqrs-journey-code. Your domain experts should help you to identify possible business failures and determine the way that you handle them: either using an automated process or manually. Note: Do not expect the code samples to match exactly the code in the reference implementation. This chapter describes a step in the CQRS journey, but the implementation may well change as we learn more and refactor the code. E xtending a nd Enh a ncing the Or ders a nd R egistr ations Bounded Context 63 The order access code record locator A registrant may need to retrieve an order, either to view it, or to complete the assignment of attendees to seats. This may happen in a different web session, so the registrant must supply some information to locate the previously saved order. The following code sample shows how the Order class generates a new five-character order access code that is persisted as part of the Order instance. public string AccessCode { get; set; } protected Order() { ... this.AccessCode = HandleGenerator.Generate(5); } To retrieve an Order instance, a registrant must provide her email address and the order access code. The system will use these two items to locate the correct order. This logic is part of the read side. The following code sample from the OrderController class in the web application shows how the MVC controller submits the query to the read side using the LocateOrder method to discover the unique OrderId value. This Find action passes the OrderId value to a Display action that displays the order information to the registrant. [HttpPost] public ActionResult Find(string email, string accessCode) { var orderId = orderDao.LocateOrder(email, accessCode); if (!orderId.HasValue) { return RedirectToAction( "Find", new { conferenceCode = this.ConferenceCode }); } return RedirectToAction( "Display", new { conferenceCode = this.ConferenceCode, orderId = orderId.Value }); } 64 Jour ney four The countdown timer When a registrant creates an order and makes a seat reservation, those seats are reserved for a fixed period of time. The RegistrationProcessManager instance, which forwards the reservation from the SeatsAvailability aggregate, passes the time that the reservation expires to the Order aggregate. The following code sample shows how the Order aggregate receives and stores the reservation expiry time. public DateTime? ReservationExpirationDate { get; private set; } public void MarkAsReserved(DateTime expirationDate, IEnumerable seats) { ... this.ReservationExpirationDate = expirationDate; this.Items.Clear(); this.Items.AddRange( seats.Select( seat => new OrderItem(seat.SeatType, seat.Quantity))); } The ReservationExpirationDate is initially set in the Order constructor to a time 15 minutes after the Order is instantiated. The RegistrationProcessManager class may revise this time based on when the reservations are actually made. It is this time that the process manager sends to the Order aggregate in the MarkSeatsAsReserved command. When the RegistrationProcessManager sends the MarkSeatsAsReserved command to the Order aggregate with the expiry time that the UI will display, it also sends a command to itself to initiate the process of releasing the reserved seats. This ExpireRegistrationProcess command is held for the expiry duration plus a buffer of five minutes. This buffer ensures that time differences between the servers don’t cause the RegistrationProcessManager class to release the reserved seats before the timer in the UI counts down to zero. In the following code sample from the RegistrationProcessManager class, the UI uses the Expiration property in the MarkSeatsAsReserved command to display the countdown timer, and the Delay property in the ExpireRegistrationProcess command determines when the reserved seats are released. E xtending a nd Enh a ncing the Or ders a nd R egistr ations Bounded Context 65 public void Handle(SeatsReserved message) { if (this.State == ProcessState.AwaitingReservationConfirmation) { var expirationTime = this.ReservationAutoExpiration.Value; this.State = ProcessState.ReservationConfirmationReceived; if (this.ExpirationCommandId == Guid.Empty) { var bufferTime = TimeSpan.FromMinutes(5); var expirationCommand = new ExpireRegistrationProcess { ProcessId = this.Id }; this.ExpirationCommandId = expirationCommand.Id; this.AddCommand(new Envelope (expirationCommand) { Delay = expirationTime.Subtract(DateTime.UtcNow).Add(bufferTime), }); } this.AddCommand(new MarkSeatsAsReserved { OrderId = this.OrderId, Seats = message.ReservationDetails.ToList(), Expiration = expirationTime, }); } ... } The MVC RegistrationController class retrieves the order information on the read side. The DraftOrder class includes the reservation expiry time that the controller passes to the view using the ViewBag class, as shown in the following code sample. 66 Jour ney four [HttpGet] public ActionResult SpecifyRegistrantDetails(string conferenceCode, Guid orderId) { var repo = this.repositoryFactory(); using (repo as IDisposable) { var draftOrder = repo.Find (orderId); var conference = repo.Query () .Where(c => c.Code == conferenceCode) .FirstOrDefault(); this.ViewBag.ConferenceName = conference.Name; this.ViewBag.ConferenceCode = conference.Code; this.ViewBag.ExpirationDateUTCMilliseconds = draftOrder.BookingExpirationDate.HasValue ? ((draftOrder.BookingExpirationDate.Value.Ticks - EpochTicks) / 10000L) : 0L; this.ViewBag.OrderId = orderId; return View(new AssignRegistrantDetails { OrderId = orderId }); } } The MVC view then uses JavaScript to display an animated countdown timer. Using ASP.NET MVC validation for commands You should try to ensure that any commands that the MVC controllers in your application send to the write model will succeed. You can use the features in MVC to validate the commands on both the client side and server side before sending them to the write model. The following code sample shows the AssignRegistrantDetails command class that uses DataAnnotations to specify the validation requirements; in this example, the requirement is that the FirstName, LastName, and Email fields are not empty. Client-side validation is primarily a convenience to the user in that it avoids the need for round trips to the server to help the user complete a form correctly. You still need to implement server-side validation to ensure that the data is validated before it is forwarded to the write model. E xtending a nd Enh a ncing the Or ders a nd R egistr ations Bounded Context 67 using System; using System.ComponentModel.DataAnnotations; using Common; public class AssignRegistrantDetails : ICommand { public AssignRegistrantDetails() { this.Id = Guid.NewGuid(); } public Guid Id { get; private set; } public Guid OrderId { get; set; } [Required(AllowEmptyStrings = false)] public string FirstName { get; set; } [Required(AllowEmptyStrings = false)] public string LastName { get; set; } [Required(AllowEmptyStrings = false)] public string Email { get; set; } } The MVC view uses this command class as its model class. The following code sample from the SpecifyRegistrantDetails.cshtml file shows how the model is populated. @model Registration.Commands.AssignRegistrantDetails ... @Html.LabelFor(model class="editor-field">@Html.EditorFor(model class="editor-label">@Html.LabelFor(model class="editor-field">@Html.EditorFor(model class="editor-label">@Html.LabelFor(model class="editor-field">@Html.EditorFor(model => => => => => => model.FirstName)model.FirstName)