two tips for module authors
14 views
Skip to first unread message
Michiel de Jong
Sep 5, 2012, 12:32:58 PM9/5/12
to unhosted
Hi,
While working on the money module today i was realised that the data
structure choices are far from trivial.
First of all, i started by storing transaction :id as
:year/:month/:day/:tab/transaction/:id
and the balance for participant :peer as
:year/:month/:day/:tab/balance/:peer
Later i realised that tabs can probably be kept separate from each
other, so that it would be more efficient to put :tab a the front. So
I changed that. Luckily we don't have an installed base here yet,
because i can only imagine having to write migration code every time
we change something like this. so my first note to self and tip to
other module authors, make sure you get the data organisation right
the first time. :)
Later on i realised that if you add a transaction, then you have to
update the balance for that day and all subsequent days. There is
basically no good way to do that (we don't have any locking or
transactions). So then i thought of something that actually eliminates
the need for a solution, by changing the problem.
Let's just prohibit random access to past data. only the current day
can be edited. and only past days have a balance calculated. This
slightly influences the functionality, but i think it's an acceptable
interpretation that history can't be changed, so you get a sort of
temporally incremental data design. anyway, that's my second tip that
can probably be useful for other modules too, so i thought i would
share it. :)
Cheers!
Michiel
While working on the money module today i was realised that the data
structure choices are far from trivial.
First of all, i started by storing transaction :id as
:year/:month/:day/:tab/transaction/:id
and the balance for participant :peer as
:year/:month/:day/:tab/balance/:peer
Later i realised that tabs can probably be kept separate from each
other, so that it would be more efficient to put :tab a the front. So
I changed that. Luckily we don't have an installed base here yet,
because i can only imagine having to write migration code every time
we change something like this. so my first note to self and tip to
other module authors, make sure you get the data organisation right
the first time. :)
Later on i realised that if you add a transaction, then you have to
update the balance for that day and all subsequent days. There is
basically no good way to do that (we don't have any locking or
transactions). So then i thought of something that actually eliminates
the need for a solution, by changing the problem.
Let's just prohibit random access to past data. only the current day
can be edited. and only past days have a balance calculated. This
slightly influences the functionality, but i think it's an acceptable
interpretation that history can't be changed, so you get a sort of
temporally incremental data design. anyway, that's my second tip that
can probably be useful for other modules too, so i thought i would
share it. :)
Cheers!
Michiel
Thad Guidry
Sep 5, 2012, 12:40:54 PM9/5/12
to unho...@googlegroups.com
Michiel,
Applied to the database world, you've hit on the differences between transactional logging style tables VERSUS aggregate style tables. (schemaless, schemaless, schemaless designs are your friends sometimes here....and your worst enemies, if you expect millions of users :)
Michiel de Jong
Sep 5, 2012, 12:47:14 PM9/5/12
to unho...@googlegroups.com
yeah, all these issues like replication, versioning and consistency
are just classics repeating themselves :)
they are important topics though for RemoteStorageJs module authors,
so hopefully sharing our experiences helps us a little bit to get
things right.
> --
>
>
>
are just classics repeating themselves :)
they are important topics though for RemoteStorageJs module authors,
so hopefully sharing our experiences helps us a little bit to get
things right.
>
>
>
Melvin Carvalho
Sep 5, 2012, 12:51:07 PM9/5/12
to unho...@googlegroups.com
i dont store balances in the data, only in the app after aggregation
the reason is that you cannot guarantee the app has access to all your transactions in an open world assumption which allows the possibility of privacy
Cheers!
Michiel
--
Michiel de Jong
Sep 5, 2012, 12:57:23 PM9/5/12
to unho...@googlegroups.com
On Wed, Sep 5, 2012 at 6:51 PM, Melvin Carvalho
<melvinc...@gmail.com> wrote:
> the reason is that you cannot guarantee the app has access to all your
> transactions in an open world assumption which allows the possibility of
> privacy
sure, but they are per-tab balances. a tab is like a piece of paper on
<melvinc...@gmail.com> wrote:
> the reason is that you cannot guarantee the app has access to all your
> transactions in an open world assumption which allows the possibility of
> privacy
which you keep track of who owes what. So the module doesn't assume it
has access to all your tabs, there might be other tabs on which there
are other transactions and other balances (even among the same peers).
Melvin Carvalho
Sep 5, 2012, 1:01:41 PM9/5/12
to unho...@googlegroups.com
yes thats fine ... this is how the web was designed with documents containing data items
just remember that if you want to mix things then the balances may not be accurate, tho there's one ingenious feature in that a tab can be collapsed into a single transaction
if you want a conside shorthand, istr reuters had a data format using "FIDS" that was at one time popular in financial circles, am not sure what the start of the art is now, tho something like json ld / webcredits is translatable to any format
--
Daniel Brolund
Sep 5, 2012, 1:06:59 PM9/5/12
to unho...@googlegroups.com
For friendsunhosted, we are looking for doing event sourcing.
Everything that happens is stored as an event, with its own key. Once
written we don't change it. Then we create relevant aggregates for
parts we want to retrieve often or fast, such as last access, friends
list, latest status updates etc. That way we always keep the original
event, but we can change the aggregated data that we use the most to
adapt to new circumstances.
However, we haven't started doing this just yet since we don't have
key listing in the remoteStorage we use, we're waiting for the
upgrade. :-)
Cheers
Daniel
2012/9/5 Michiel de Jong <mic...@unhosted.org>:
> --
>
>
>
--
---------------------------------------------------------
Daniel Brolund
Agical AB - www.agical.com
work: daniel....@agical.com
phone: +46708754002
blog:http://danielbrolund.wordpress.com
twitter: @danielbrolund
private: daniel....@gmail.com
Everything that happens is stored as an event, with its own key. Once
written we don't change it. Then we create relevant aggregates for
parts we want to retrieve often or fast, such as last access, friends
list, latest status updates etc. That way we always keep the original
event, but we can change the aggregated data that we use the most to
adapt to new circumstances.
However, we haven't started doing this just yet since we don't have
key listing in the remoteStorage we use, we're waiting for the
upgrade. :-)
Cheers
Daniel
2012/9/5 Michiel de Jong <mic...@unhosted.org>:
>
>
>
--
---------------------------------------------------------
Daniel Brolund
Agical AB - www.agical.com
work: daniel....@agical.com
phone: +46708754002
blog:http://danielbrolund.wordpress.com
twitter: @danielbrolund
private: daniel....@gmail.com
Neophyte Representative
Sep 6, 2012, 9:13:00 AM9/6/12
to unho...@googlegroups.com
> While working on the money module today i realized that the data
is a discipline called Data Analysis that needs to be done parallel to
the Functional Analysis and the data structure should be designed just
as flexibly as the functional structure.
> First of all, i started by storing transaction :id as
>
> :year/:month/:day/:tab/transaction/:id
>
> and the balance for participant :peer as
>
> :year/:month/:day/:tab/balance/:peer
>
> Later i realized that tabs can probably be kept separate from each
> other, so that it would be more efficient to put :tab at the front. So
pinched off bubble of the Multiverse. Luckily, you are working on
something that a lot of folks have spent enormous time making right
because money is at the core of their day job. Perhaps you could look
at some of the existing money systems for guidance on how it has best
been done. Surely there is accessible documentation somewhere. Stand
on those shoulders, not in the wilderness.
> because i can only imagine having to write migration code every time
> we change something like this.
Try thinking of the smallest units of data that can stand alone or be
combined with other isolated data to report a fact. You'll run into
fewer needs to restructure something that is built dynamically as
needed. While you eventually won't want to accumulate all
transactions to establish a current balance, because it will
eventually take too long, you should know what is statically accurate
versus what is derived.
> so my first note to self and tip to
> other module authors, make sure you get the data organization right
> the first time. :)
Let this be your lesson, you never know everything at the beginning of
your design process. So, nothing is ever "Right the First Time."
Design for maintenance.
> Later on i realized that if you add a transaction, then you have to
Be careful, here be bugs waiting to happen.
> So then i thought of something that actually eliminates
> the need for a solution, by changing the problem.
Again, be careful. You can't eliminate all problems by wishing they
didn't exist. Often a new algorithm is exactly what you need, but
don't eliminate required functionality to achieve simplified design.
It'll just come back to you as a harder to implement requirement.
> Let's just prohibit random access to past data.
Define "past data" carefully. Does it mean data about the past that
you didn't have at the time? Or the data that you entered some time
in the past?
> only the current day
> can be edited.
Only what you have now can be entered correctly?
> and only past days have a balance calculated.
A provisional balance?
> This
> slightly influences the functionality, but i think it's an acceptable
> interpretation that history can't be changed,
You certainly can't go back and change what you knew then, but you can
add recently acquired information now. The question seems to be, how
does one add it?
> so you get a sort of
> temporally incremental data design. anyway, that's my second tip that
> can probably be useful for other modules too, so i thought i would
> share it. :)
>
>
> Cheers!
> Michiel
It's good to see someone who is willing to "Talk out loud" about his
analysis. Bugs are much easier to eliminate at this stage than later
and sharing with co-workers is a way to eliminate the bats in the
belfry that may prevent clear vision of the requirements.
Keep thinking. Carefully.
> structure choices are far from trivial.
>
The data structures are ALWAYS "far from trivial". That's why there
>
is a discipline called Data Analysis that needs to be done parallel to
the Functional Analysis and the data structure should be designed just
as flexibly as the functional structure.
> First of all, i started by storing transaction :id as
>
> :year/:month/:day/:tab/transaction/:id
>
> and the balance for participant :peer as
>
> :year/:month/:day/:tab/balance/:peer
>
> other, so that it would be more efficient to put :tab at the front. So
> I changed that.
> Luckily we don't have an installed base here yet,
Only the Big Bang had no installed base and even it may have been a
> Luckily we don't have an installed base here yet,
pinched off bubble of the Multiverse. Luckily, you are working on
something that a lot of folks have spent enormous time making right
because money is at the core of their day job. Perhaps you could look
at some of the existing money systems for guidance on how it has best
been done. Surely there is accessible documentation somewhere. Stand
on those shoulders, not in the wilderness.
> because i can only imagine having to write migration code every time
> we change something like this.
combined with other isolated data to report a fact. You'll run into
fewer needs to restructure something that is built dynamically as
needed. While you eventually won't want to accumulate all
transactions to establish a current balance, because it will
eventually take too long, you should know what is statically accurate
versus what is derived.
> so my first note to self and tip to
> the first time. :)
Let this be your lesson, you never know everything at the beginning of
your design process. So, nothing is ever "Right the First Time."
Design for maintenance.
> Later on i realized that if you add a transaction, then you have to
> update the balance for that day and all subsequent days. There is
> basically no good way to do that (we don't have any locking or
> transactions).
But this scenario is one of the drivers of the invention of locking.
> basically no good way to do that (we don't have any locking or
> transactions).
Be careful, here be bugs waiting to happen.
> So then i thought of something that actually eliminates
> the need for a solution, by changing the problem.
didn't exist. Often a new algorithm is exactly what you need, but
don't eliminate required functionality to achieve simplified design.
It'll just come back to you as a harder to implement requirement.
> Let's just prohibit random access to past data.
you didn't have at the time? Or the data that you entered some time
in the past?
> only the current day
> can be edited.
> and only past days have a balance calculated.
> This
> slightly influences the functionality, but i think it's an acceptable
> interpretation that history can't be changed,
add recently acquired information now. The question seems to be, how
does one add it?
> so you get a sort of
> temporally incremental data design. anyway, that's my second tip that
> can probably be useful for other modules too, so i thought i would
> share it. :)
>
>
> Cheers!
> Michiel
analysis. Bugs are much easier to eliminate at this stage than later
and sharing with co-workers is a way to eliminate the bats in the
belfry that may prevent clear vision of the requirements.
Keep thinking. Carefully.
Reply all
Reply to author
Forward
0 new messages