is this a sensible problem for perdistence?

[Tim Pigden]

Hi

I'm looking at modelling a large parcel delivery system with approximately 300k - 1m parcels per day depending on time of year. Each of those parcels has its own life cycle from assignment to a trip through eventual delivery. If parcels are on a route, then, for example, a delivery event at position 5 on the route will cause an update of the ETA for parcels 6 - n where n could be 100 or so.

A parcel in position 50 might receive 60 such updates up until it is successfully delivered and therefore "retired" from active consideration.

I'm estimating that with 0.5m parcels and an 8 hour working day, I'll get around 20 delivery notifications per second, each generating a cascade of up to 100 new ETAs - so around 2000 updates/second. In the morning, around 8am and 9am these numbers will peak as high priority orders tend to cluster well geographically so the drivers hit more deliveries in a shorter space of time - so say at Christmas peak we could get up to 5000 updates/second for brief periods.

Now I'm not a big data guy. To me this is immense, but realistically is this going to cause any problem for Akka persistence? Is up to 1m separate actor processors a sensible thing to have? There are alternative ways to model it - the updates are internally generated at the vehicle level (so theoretically I only need to persist the incoming vehicle signals) but the parcel life cycle extends beyond the duration of the vehicle trip so that could get messy.

[Tim Pigden]

Hmm - on reading other threads and reflection I think I may have the wrong model here. The information about the ETA is possibly not something that should be persisted. It's not hard data. And perhaps my parcel should be split between a persisting actor (still need 1m of them) and some sort of query object that reflects current ETA and the real object is only updated in response to materialisation of the ETA (i.e. sending it to the customer as advice) in which case it's the advice history that's the thing to persist.

Is anyone encountering similar modelling issues?

[Patrik Nordwall]

On Thu, Jan 8, 2015 at 11:59 AM, Tim Pigden <tim.p...@optrak.com> wrote:

Hmm - on reading other threads and reflection I think I may have the wrong model here. The information about the ETA is possibly not something that should be persisted. It's not hard data. And perhaps my parcel should be split between a persisting actor (still need 1m of them) and some sort of query object that reflects current ETA and the real object is only updated in response to materialisation of the ETA (i.e. sending it to the customer as advice) in which case it's the advice history that's the thing to persist.

That sounds good. 

/Patrik

 

Is anyone encountering similar modelling issues?

On Thursday, January 8, 2015 10:34:10 AM UTC, Tim Pigden wrote:

Hi

I'm looking at modelling a large parcel delivery system with approximately 300k - 1m parcels per day depending on time of year. Each of those parcels has its own life cycle from assignment to a trip through eventual delivery. If parcels are on a route, then, for example, a delivery event at position 5 on the route will cause an update of the ETA for parcels 6 - n where n could be 100 or so.

A parcel in position 50 might receive 60 such updates up until it is successfully delivered and therefore "retired" from active consideration.

I'm estimating that with 0.5m parcels and an 8 hour working day, I'll get around 20 delivery notifications per second, each generating a cascade of up to 100 new ETAs - so around 2000 updates/second. In the morning, around 8am and 9am these numbers will peak as high priority orders tend to cluster well geographically so the drivers hit more deliveries in a shorter space of time - so say at Christmas peak we could get up to 5000 updates/second for brief periods.

Now I'm not a big data guy. To me this is immense, but realistically is this going to cause any problem for Akka persistence? Is up to 1m separate actor processors a sensible thing to have? There are alternative ways to model it - the updates are internally generated at the vehicle level (so theoretically I only need to persist the incoming vehicle signals) but the parcel life cycle extends beyond the duration of the vehicle trip so that could get messy.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Greg Young]

Its also very common to CALCULATE the ETA changes and raise them as events without necessarily persisting them.

As a thought experiment if your notification system lost 1/10000000 would it really matter?

[Tim Pigden]

Hi Greg

Well the history of ETA changes might be useful but certainly is never essential. 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/HTyC75jnOEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Tim Pigden
Optrak Distribution Software Limited
+44 (0)1992 517100
http://www.linkedin.com/in/timpigden
http://optrak.com
Optrak Distribution Software Ltd is a limited company registered in England and Wales.
Company Registration No. 2327613 Registered Offices: Suite 6,The Maltings, Hoe Lane, Ware, SG12 9LR England 
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Optrak Distribution Software Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.