Advice on whether retlang would be appropriate for this scenario

[AgileBoy]

Hi Mike,

I came across retlang when reading around the forums for Parallel
Extensions. I wonder if you could advice on the processing below if it
would be a candidate for retlang.

Our scenario is essentially processing a queue of XML items as quickly
as possible (We could have +50K queue items). The queue is held in a
SQL Server table and at the moment a service is processing the queue.
At some point we would like to run the service on multiple machines so
we can scale out the processing if necessary (although maybe not for
this conversation) - Here is the processing flow:



1) Read off 'n' entries off the queue (essentially an XML or flat
File) and set a lock flag on each item (DB update) - this is to
preempt the multi machine processing so we do not get 2 services
processing the same queue entry.

2) Each entry then

a) Runs a .NET component against the queue entry to produce an XML
file (File A). It also generates some properties of the document
(Properties C).

b) A compiled XSLT runs against File A to produce a marked up XML file
(File B).

c) File B is then run against a remote (rendering) server. We are
eventually going to render this document to a PDF but here we are just
trying to return the number of pages this document would be as a PDF
so we return a Page Count D from this remote call.

d) Finally, we take File B, Page Count D and Properties C and persist
these back to SQL Server.

3) The last step is to unlock the entry in the SQL table and set the
processed indicator on that queue item.

There are also a few other services which follow a simmilar pattern
but the gist of the services is that there is a mix of local
processing (.NET Component and XSLT) and remote processing (rendering
server).

Our remit is to process these queue items as fast as possible i.e.
Suck in batches of queue items and then fire them of to be processed
by the above steps maximising the underlying hardware.

Is this something do you think that would lend itself well to retlang?
Is there some sort of throttle mechanism so the machine does not get
swamped with requests (I can imagine the queue is read very quickly
off the SQL Server and requests to process are fired off very
frequently). Also, one other thing for me to consider is that the
remote server could become a bottleneck if lots of requests are
pending - How would these timeouts get handled in retlang if the
processing has already gone through 2 or 3 pipes?

Any help, advice or thoughts would be greatly appreciated

Many thanks in advance

Mike

[Mike Rettig]

The system could be designed as a set of pipes and filters.

http://www.enterpriseintegrationpatterns.com/PipesAndFilters.html

Retlang channels take the place of pipes.

http://code.google.com/p/retlang/wiki/MessageChannels

A Retlang Process Context can provide the filtering.

http://code.google.com/p/retlang/wiki/ThreadingDesign

Also, Retlang provides batching for subscriptions. This may be helpful when writing to external systems and file systems.

However, Retlang is purely an in process threading library. If you want distribution for your system, then you should look at MSMQ or NServiceBus(http://nservicebus.com/). Those messaging solutions work well as complementary technologies with Retlang.

Let me know if you have any questions.

Mike