Cross aggregate references to child entities

[Alex Farcas]

My issue is similar to the one discussed here: https://groups.google.com/forum/#!topic/dddcqrs/m4Lr9Nzk_T0. Reposting with the hope of getting more detailed responses on how to refactor/redesign around the problem.

My project's domain is order fulfillment. For fulfillment purposes, an Order can be split into multiple Shipments. There are situations when the client orders N items of the same Product (represented as a single OrderLine qith qty = N) and in order to optimize shipping costs, those N items are split into M Shipments. Not all of those M Shipments might be created at the same time (some several days apart, depending on inventory availability). 

Given the above, the OrderLine needs to hold some counters with the number of items (initial_qty, part_of_shipment_qty) and it also has a status ("unfulfilled", "partially fulfilled" and "fulfilled"). Maintaining a part_of_shipment_qty is useful when attempting to create a new Shipment, because i can easily calculate the max quantity that can be put into the new Shipment (initial_qty - part_of_shipment_qty). Without such a counter i would be forced to loop through all the Shipments of an Order and calculate the quantities that are present in ShipmentLines. Although i would have preferred to make OrderLine a VO, all this forces me to make it an Entity since it has a lifecycle. ShipmentLines on the other hand can be plain VOs and in my implementation they just copy data from the OrderLines they represent (although they don't have the exact same fields).

A Shipment goes through several state changes and mutations before being actually shipped. So now let's assume that a Shipment needs to be cancelled (prior to being actually shipped, after which cancellation is no longer be possible). What happens to the OrderLines that shipment was made for? I need to be able to modify the part_of_shipment_qty  and possibly the status too. How do i know which OrderLine a ShipmentLine refers to if i do not maintain some sort of id within the ShipmentLine? The only way it could do that was if OrderLine was an agg root itself but that doesn't seem right (an OrderLine does not make sense without an Order, so the Order must be the agg root). 

I'm pretty sure some of my assumptions or modeling decisions are wrong. Can you please shed some light on my confusion?

[Alex Farcas]

Searching for a solution, got me to the following article: http://blog.sapiensworks.com/post/2014/05/20/Is-OrderLine-an-Entity-or-A-Value-Object.aspx/. It offers various reasons for why OrderLine should not be an Entity (mainly because it's not part of the ubiquitous language). He suggests that instead of referring to OrderLines we should refer to Products (or some related concept).

So in my particular case, i would not need to maintain a reference to the OrderLine inside the ShipmentLine but instead just a reference to a Product. Because the OrderLine would too maintain a reference to the Product, identifying the OrderLine would be done via the Product. But let me throw a wrench in this: what if there can be multiple OrderLines inside an Order for the same Product? Let's say that you import these Orders from a third party system you have no control over. That third party system also gives you some foreign IDs for the order lines so that when you report back with a fulfillment (partial or complete Shipment) you also need to tell them the original foreign IDs for the order lines (because otherwise they do not know which order line within their system you are referring to).

Suddenly i am no able to use the Product to identify OrderLines and i am back to square one. I need some sort of ID.

[Johanna Belanger]

If you write out how the process would be done without computers first, that can help illuminate what is really necessary in *your* domain. So you get this Order with multiple order lines for the same product. What happens next? (If you are able to share this.)

[Alex Farcas]

We've never done this without computers so for a good part of the process the most i can do is imagine what a paper based one would look like.

The Order would probably look pretty much like an invoice. The order lines would be displayed in a table like structure with the following columns: Line Number, SKU, Product Name, Quantity, Price. The line number would just be an ordinal number (starting from 1) and not an ID. The order would be given to a Shipper (warehouse employee) to start processing. The Shipper creates a Pick List by extracting the essential information from the Order (SKU and quantity) and asks a Runner (another warehouse employee) to locate and fetch them (obviously this operation is more efficiently done in bulk which is what happens in reality). The Runner locates the items based on a mapping list (SKU to warehouse location), picks up the items from the shelves and returns them to the Shipper. The Shipper decides if the returned items can be packed together in a single shipment or if they need to be part of multiple shipments. Assuming he needs multiple shipments, he chooses appropriate boxes for them and starts packing. As soon as a package is complete, he creates a document listing the items in that package. Say he calls that document a Shipment and it contains the order number it was built for and a grid with the following columns: Line Number, SKU, Quantity. No information about the order line is present in this document. The fact that the same product is present in multiple order lines is irrelevant to him (he effectively sums them grouped by SKU). He also weighs the package and fills in the dimensions and weight on the Shipment document. He creates another document (called a Shipping Slip) that contains the shipping and the return addresses and glues it to the package. After he finishes packing all the packages for that order, he calls the Courier to come pick them up and places the order into the "to be picked up" pile. The Runner then takes the package to a warehouse location designated for pick up.

It may happen that after the packages are complete and placed in the pick up location, the store faxes us an updated version of the order that is to replace the old one. Say this new order contains fewer items and some order lines that were present in the previous order are missing in this new one. The Shipper looks over the existing Shipment documents and compares their content with the items in the new Order. If some of the packages need to be returned to stock or rebuilt (just a few items removed) he marks the Shipment document of those packages as cancelled, and asks the Runner to return the items in the packages to stock. He then proceeds to create new packages and Shipment documents for them following the same process that was previously described.

It's pretty obvious that as far as this process is concerned, the Shipper does not need to know the IDs or line numbers of order lines. He does however maintain those counters i was talking about in his mind while packaging. If the number of packages is big he might use some paper based external memory either by scribbling on the order document itself or on some disposable piece of paper.

The problem comes after the packages are shipped and is caused by the requirements of the store the order came from. The Shipper needs to notify the store with the packages he created, including the order lines numbers and the quantity present in that shipment from each order line. Presumably the store would use this information to compute how many items and from which order lines are still to be shipped (in case the order was only partially fulfilled). No non-computer based stores would ask for this of course, but alas, platforms like Shopify do! In order to accommodate this, during the packaging process the Shipper would most probably need to maintain a new column Order Line Number in his Shipment document. This way, when the notification time comes he would be able to just fax the Shipment documents and it would all be done.

--
You received this message because you are subscribed to a topic in the Google Groups "DDD/CQRS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dddcqrs/bnl9d51kiPc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dddcqrs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johanna Belanger]

Wow clear and thorough description, thanks!

"The Shipper needs to notify the store with the packages he created, including the order lines numbers and the quantity present in that shipment from each order line."

Can the order be updated/cancelled after this notification?

Can the shipment lines be assigned to order lines at the end of the process? So the shipper just works with the SKU totals during the shipment assembly and then at the end someone checks the shipment against the order and records "This shipment line fills order line 1 and partially fills order line 3"? That keeps the "corruption" from the 3rd party format at the boundary of your system.

[Alex Farcas]

The order can be modified at any time before being completely fulfilled (before receiving notifications about all order lines and their quantity having been shipped). The order can be completely refunded/cancelled at any point in time (even after it is fully fulfilled). Of course shipments that have left the warehouse cannot be "cancelled" but they would eventually return to the warehouse as Returns. There's an entire workflow for processing returns the details of which are not of interest atm. A cancelled order cannot be modified any further in any way.

Your suggestion could work if we could notify the store about all the shipments of the order within a single notification. But say you have an order with 2 shipments and those 2 shipments are created and shipped in different days. When you notify about the first one being shipped you tell the store a story that you made up at that moment (if i understood you correctly). When you want to notify about the second one being shipped you need to remember what you notified the first time (the combination and matches that you made) so that you don't end up telling a lie. Telling a lie ends up with your lie being rejected by the store.

[Johanna Belanger]

This suggests to me that you need a component which consumes Order and Shipment events for a given OrderID and maintains/publishes the current overall state of the order. Examine the consistency requirements around this responsibility. This will help determine your aggregate boundaries. When in the overall process does the store need to be notified? What happens if the notification is published after an order is updated or a shipment is created but before the component receives the event?

[Johanna Belanger]

More precisely, it suggests a component which ensures all notifications about a particular Order ID are consistent with each other, because that consistency is what the store is checking.

[Alexandre Potvin Latreille]

Would it make any sense to make a ShipmentProcess/ShipmentObligation concrete aggregate root, which would be created once an Order is ready to get shipped. The ShipmentObligation would hold the orderId as well as what's to be shipped and could be fulfilled through a series of Shipments. Therefore, the ShipmentObligation would make sure that invariants spanning across multiple Shipments are protected.

[Michael Bergman]

Thank you for sharing the link. Very interesting reading and I believe the proposed solution of avoiding a reference to OrderLineId by instead referencing (OrderId, ProductId) is an interesting one.

In your case however, I cannot yet see how you can avoid the reference to OrderLineId since you have an ExternalOrderLineId to take into account.

Michael

[Michael Bergman]

I believe a ShipmentObligation could be beneficial. This would make it perfectly fine for a ShipmentLine to then hold a ShipmentObligationId reference. This is similar to my re-design attempt (http://yuml.me/0144304f) in a previous post. There I named it “Allotment” instead of ShipmentObligation.

In my case I needed Allotment mostly for the requirement that one order line must be able to be shipped to multiple addresses (20 units to AddressA, 10 units to AddressB). For Alex, the requirement seems to be that there is only a single shipping address for the Order.

In Allotment I have a “soft string reference” named ReferenceNumber (that can be anything unique). I don’t like this part since it is not very robust and clear. However, it opens up to be able to deliver other things than “Product” which we actually have a use case for (shipping unfinished products and raw material etc.).

[Kijana Woodard]

In a battle between models, being explicit is better.

You're already seeing that shipping has a different conception of an order than sales. From your other post, there's tension between shipping retail orders and other items.

If you have the resources to keep digging, do so. There may very well be an abstraction that works for both, but that may be yet another model: One for people moving material in a warehouse/trucks another for managing the shipping details.

---

From: Michael Bergman
Sent: ‎7/‎17/‎2015 4:59 AM
To: ddd...@googlegroups.com
Subject: [DDD/CQRS] Re: Cross aggregate references to child entities

--
You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

[Alex Farcas]

By "component" you mean something that is not part of the initial domain (a subdomain / bounded context)? Or is it something that is outside of the domain (a 'on the wire' translator that is used only when talking to that specific store)? 

[Johanna Belanger]

I used the word component because it's generic and doesn't refer yet to a specific implementation. It's good to tease the responsibility out, get a solid sense of how it connects to and where it can be disconnected from everything else, and then figure out the best way to implement it. Going back to the paper process, imagine it's a person in charge of notifying the store of shipments. How do they need to do their job?

[Ben Kloosterman]

Whats the issue line items and picked are both value types.. they have no ID concept ( think XML inside an order not a separate table) .  At best you would have a work order to pick a order which consists of line items  but line items are not entities.  

eg Warehouse gets the order and creates a picket list and copy in the line items creating new picked line items  , picking and shipping is different from the order and sometimes even a different BC ..   

How you handle the partially shipped order ?   You can get a shipped event of what has been shipped  ( or possibly get the delivered from the courier service) and update it but from a business logic point of view , are the other items cancelled shipped later etc ?

Dont think in older normalized relational DB terms. Having copies or value types is fine.  

There was a very similar question  2 months ago.  

Regards,

Ben

--

You received this message because you are subscribed to the Google Groups "DDD/CQRS" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dddcqrs+u...@googlegroups.com.

[Alex Farcas]

Thank you everybody for your suggestions. I don't think that the problem of having a foreign/external ID in your model can get solved by making use of new entities/concepts within that model (or at least within that bounded context). It seems to me that if we're to maintain a pure model those external IDs should not even be in our model to begin with. Johanna hinted on this to some extent.

I've been reading up on bounded contexts this weekend (from IDDD by Vaughn Vernon) and it became clear to me that my problem was in fact caused by a leaky boundary. 

Taking a closer look at my model revealed that this was not the only leakage caused by Shopify. When we initially designed the system we assumed that we would import products based on the SKU and import orders based on some external order ID. You need to have some sort of unique IDs to be able to talk to the third party systems, so these two seemed sufficient. Mind you that initially we had no hint of foreign IDs for order lines. But then we had to integrate Shopify, which apparently brought the need for these order line IDs. Additionally, Shopify was maintaining two order IDs: an ID which was internal to their system and another which was displayed to users and third party systems (namely us). But their REST API is not consistent; it sometimes requires you to use the internal ID and other times the "display" ID. We thought we had no choice but to make an exception for Shopify, so instead of storing a single external order ID like we initially planned, we were now storing two IDs. It should have been pretty obvious that this approach was not scalable as it was exposing us to all the quirks and issues of third party systems.

The solution seems to lie in a better application of bounded contexts. Each third party system we interact with is a bounded context on it's own. IDDD (chapter 3) tells about various autonomy levels when integrating with other bounded contexts. One way is to always make synchronous RPC style calls via a domain service or a repository interface, and the other is to create a local bounded context that would mirror the remote one (containing the minimal amount of information needed). This mirror would get updated via REST polling calls and webhooks published by the remote system (if available).

If we were to employ the mirror approach we could completely eliminate any external contamination from our domain and make that new mirror bounded context compensate for the lackings of the third party system. Thus, in the case of Shopify, we could have that mirror bounded context communicate with our bounded context based on a single type of order ID instead of two. The mirror bounded context would indeed maintain two IDs but it would make the translation between them as needed by the Shopify API. The mirror bounded context would also receive shipment/fulfillment events/notifications from our bounded context and ensure the consistency when it comes to external order line IDs (it would associate the shipping notifications with the IDs on the fly and remember that association for further notifications).

Of course, all the above requires a lot of work when compared to the current implementation, but at least now i have a better idea of where we need to go.

It would help if you guys could validate this approach based on your past experiences. Critique is always useful.

On Wednesday, 15 July 2015 18:02:10 UTC+3, Alex Farcas wrote:

[Kijana Woodard]

This is why adding 3rd party services is not a clear a win as it first appears, especially if you're using a small set of features. Ditto your experience for Salesforce and many many more.

In order to be viable, SaaS products tend to keep adding features such that they want to own your entire business. When you're starting out, this seems reasonable. At some point [hopefully], you grow to where your needs/use cases diverge from those of the 3rd party. If you have not built a boundary which explicitly manages the integration, you'll find the 3rd party *concepts* have leaked throughout your application. 

I've seen it be so bad that the business was unable to sell new products/services without first spending major time and money detaching the 3rd party tendrils. The 3rd party was, effectively, dictating the business model through leaky abstractions.

Be happy you've figured this out early.

My advice for 3rd party services is either go "all in" and completely run your business in the SaaS, firewall around the integration and heavily code review changes, or build the minimum feature set yourself.

--