Batch Processing

[Sean]

We have a business requirement where the client wishes to receive HL7 messages in batches.  For example, we would have a HL7 workflow where the final destination would be a EnsLib.HL7.Operation.TCPOperation operation.  However, they dont want them sent in real time.  Instead they would want them to queue and receive them in hourly batches or at the end of the day.

I have reviewed the schedules (http://docs.intersystems.com/ens20131/csp/docbook/DocBook.UI.Page.cls?KEY=ECONFIG_reusable#ECONFIG_reusable_schedule) and it appears I can schedule the operation to stop and start.  However, I have a couple of concerns:
* If I schedule the operation to start and stop 30 minutes after the start, what if it hasn't processed all of the queued message?  Does it wait to stop until all the messages are confused.   Or does it stop after all the queued messages are processed.
* Can I havethe operation turn it self off? For the interval, I can see maybe setting a start every 15 minutes.  But this would only work if it stops after it has processed the queued messages?
* Will it send messages that were received after the start that should be part of the next batch.

Should I use something besides a schedule to achieve my goal?  I will do some testing today and see if I can get answers but if someone has some previous experience it would be much appreciated.

Thanks
Sean

[Ted Peck]

We don't recommend trying to use an Operation's Schedule setting to make
it send something at a particular time. That setting is intended to
disable operations during time periods when you know they shouldn't be
active, for example during a nightly backup, or on weekends.

Ensemble does contain batch support for HL7 messages. Since batched
messages are most commonly used in file transfers, the File operation
has rich support for batch construction. It works like this:
- you construct a batch header object
- you send a bunch of messages to the Operation where each one has the
header object as its parent; the Operation detects this and writes the
header segment (but not the trailer segments) before writing the
contents of the first message.
- finally you send the header object to the File Operation; the
Operation recognizes that this is the parent of all the other messages
it wrote and takes this as its signal to write the trailer segment. It
also automatically inserts the batch count in the trailer segment.

Since you want to send batched messages over TCP, there are two ways you
could do this.

1) you could use a File Operation to construct the batch. If you give a
timestamp specifier as part of the filename setting it will roll over
automatically whenever the timestamp calculation causes the filename to
change. Then you could use a different Service to detect the rollover,
pull in the file, and send the resulting complete batch object out to a
TCP Operation.

2) you could use a BP that remembers all the messages it receives and
then, when the specified time interval rolls over, constructs a batch
parent object, adds all the messages to it, and forwards the parent
object to a TCP Operation.

Ted

> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: Ensemble in Healthcare Community" group.
> To post to this group, send email to
> Ensemble-in...@googlegroups.com
> To unsubscribe from this group, send email to
> Ensemble-in-Healt...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/Ensemble-in-Healthcare?hl=en
> ---
> You received this message because you are subscribed to the Google
> Groups "InterSystems: Ensemble in Healthcare" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to Ensemble-in-Healt...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Sean Phelan]

Ted-

I had a feeling there was a better way. 

Thanks

Sean

On Thu, Sep 12, 2013 at 10:42 AM, Ted Peck <ted...@intersystems.com> wrote:

We don't recommend trying to use an Operation's Schedule setting to make it send something at a particular time.  That setting is intended to disable operations during time periods when you know they shouldn't be active, for example during a nightly backup, or on weekends.

Ensemble does contain batch support for HL7 messages. Since batched messages are most commonly used in file transfers, the File operation has rich support for batch construction. It works like this:
- you construct a batch header object
- you send a bunch of messages to the Operation where each one has the header object as its parent; the Operation detects this and writes the header segment (but not the trailer segments) before writing the contents of the first message.
- finally you send the header object to the File Operation; the Operation recognizes that this is the parent of all the other messages it wrote and takes this as its signal to write the trailer segment. It also automatically inserts the batch count in the trailer segment.

Since you want to send batched messages over TCP, there are two ways you could do this.

1) you could use a File Operation to construct the batch. If you give a timestamp specifier as part of the filename setting it will roll over automatically whenever the timestamp calculation causes the filename to change.  Then you could use a different Service to detect the rollover, pull in the file, and send the resulting complete batch object out to a TCP Operation.

2) you could use a BP that remembers all the messages it receives and then, when the specified time interval rolls over, constructs a batch parent object, adds all the messages to it, and forwards the parent object to a TCP Operation.

Ted

On 9/12/2013 10:47 AM, Sean wrote:

We have a business requirement where the client wishes to receive HL7 messages in batches.  For example, we would have a HL7 workflow where the final destination would be a EnsLib.HL7.Operation.TCPOperation operation.  However, they dont want them sent in real time.  Instead they would want them to queue and receive them in hourly batches or at the end of the day.

I have reviewed the schedules (http://docs.intersystems.com/ens20131/csp/docbook/DocBook.UI.Page.cls?KEY=ECONFIG_reusable#ECONFIG_reusable_schedule) and it appears I can schedule the operation to stop and start. However, I have a couple of concerns:
* If I schedule the operation to start and stop 30 minutes after the start, what if it hasn't processed all of the queued message?  Does it wait to stop until all the messages are confused.   Or does it stop after all the queued messages are processed.
* Can I havethe operation turn it self off? For the interval, I can see maybe setting a start every 15 minutes.  But this would only work if it stops after it has processed the queued messages?
* Will it send messages that were received after the start that should be part of the next batch.

Should I use something besides a schedule to achieve my goal?  I will do some testing today and see if I can get answers but if someone has some previous experience it would be much appreciated.

Thanks
Sean

--
You received this message because you are subscribed to the Google Groups "InterSystems: Ensemble in Healthcare Community" group.

To post to this group, send email to Ensemble-in-Healthcare@googlegroups.com
To unsubscribe from this group, send email to Ensemble-in-Healthcare-unsub...@googlegroups.com

For more options, visit this group at http://groups.google.com/group/Ensemble-in-Healthcare?hl=en
---
You received this message because you are subscribed to the Google Groups "InterSystems: Ensemble in Healthcare" group.

To unsubscribe from this group and stop receiving emails from it, send an email to Ensemble-in-Healthcare+unsub...@googlegroups.com.

[John Zarate]

I don't disagree with Ted, as his solution is more elegant.  However we use the Ensemble Task scheduler to accomplish batch sending.

The FILE operation that generates the .TXT batch of HL7  stays disabled throughout the day so it queues all the messages and at 11pm it is started via the Task, so as to Enable / Disable the process at a specific time.

STOP Command:

d ##class(Ens.Director).EnableConfigItem("To_HL7FILEORU",0,1)

START Command:

d ##class(Ens.Director).EnableConfigItem("To_HL7FILEORU",1,1)

To post to this group, send email to Ensemble-in...@googlegroups.com
To unsubscribe from this group, send email to Ensemble-in-Healt...@googlegroups.com

For more options, visit this group at http://groups.google.com/group/Ensemble-in-Healthcare?hl=en
---
You received this message because you are subscribed to the Google Groups "InterSystems: Ensemble in Healthcare" group.

To unsubscribe from this group and stop receiving emails from it, send an email to Ensemble-in-Healt...@googlegroups.com.