Ideas/help for entity-to-entity mapping/conversion

[Chris Nicola]

I am currently working on a project to take a data feed which comes in
as a a set of ASCII CSV file and needs to be processed to import data
to a number of tables in an RDBMS. For anyone who might be
experienced these are ISM data, which contain information on
securities and trading for financial institutions.

I've started with loading the CSV records as entities and for that I
found FileHelpers (http://www.filehelpers.com) which was unbelievably
useful for me. So I can load all of the records in the CSV files into
collections of entities (I have a set of 12 entities one for each type
of record in the files) so first problem solved for the most part.

Next, I have created nHibernate mappings to the RDBMS tables and
created entities for those tables so I can do CRUD operations on these
entities necessary to save the updated values and new transactions
from the ISM entities.

Now, essentially I need to run a process of conversion of these ISM
transactions and securities values into the database entities. I
could perhaps just write some ugly procedural code to do this, but I
really want to try to keep this nice and clean, also it should be
modular. The conversion logic is quite complex in several places and
there will probably be a lot of need for additional logic over time.

I also know for a fact that the RDBMS will be completely replaced
within a year. I also know that it will be required that this
application be able to import the ISM data into both the new and old
database for a period of time.

Finally, I know there will be other sources of data that this
application should handle. Some will be ISM with different ASCII file
formatting, other sources will be fairly different. So again all this
says I need to be able to be able to "plug" these sources and sinks
in, in some way.

So I am wondering if anyone has any suggestions or ideas for
approaches I should consider using (learning) or things I should take
into consideration when creating the conversion parts of this
application?

Thanks,
Chris

[Adam D.]

You can source the data by generating events to which the new system
can subscribe to. You can google Martin Fowler and Event Sourcing. The
key is that the messages will contain the "contracts" that represent
the data that is important to you. This should save you from over-
engineering the existing system when you know it will be put off line
in the near future. I think we touched on that today :)

Adam

[Chris Nicola]

Thanks Adam, I am looking at the Event Sourcing documentation now. 

This seems like it could be a very useful approach.  At the moment don't currently have a requirement for having replay functionality which seems to be a big reason for using Event Sourcing, nonetheless that could change at any time.  I'm gonna read the events patterns in more detail first.  Thanks for the tip.

Chris

[Adam D.]

No problem. I was hoping that would lead you down the pub/sub
architecture anyway.

[Chris Nicola]

I've spent some time reading over some stuff on pub/sub.  In particular I have been looking at Rhino Service Bus.  I do think that at this point going directly for a pub/sub solution might be trying too bite off too much initially.  I am considering just implementing something using the Observer pattern to start with.

It is likely the application that reads these ISM data files will be just a simple client based app and not part of a SOA, though in the long run that will probably be the goal.  Initially, I am think I need to get something simple done that allows for testing the data loading and conversion logic.  The sooner I can get them involved in that process the better I will be able to understand exactly how the data should be handled.

Thanks,
Chris

[Adam Dymitruk]

Yup. No need to be async if you don't have to. What you will get out
of this still is a good contract about the data that's important and
it's structure. Should that information be needed elsewhere as well,
you will start to get pain points and a move to a bus architecture
would make sense then.

--
Adam Dymitruk
Director of IT
SCANit International (a division of APARA Systems)
mobile: 604.831.7804