saving and replaying messages received over TCP/IP

[David Loveluck]

I want (in some sense) to archive a history of thousands of incoming HL7 messages forwarded from a specific HL7 TCP Business service, so that I can delete the entire message warehouse and replay them in a clean environment.  [This is an Ensemble HL7 application and not a classic interface engine]

 

There are a number of ways to do this but none that come to mind this evening are going to be as easy and intuitive as I would like.

 

My first thought was to simply export the messages to one or more files, but I either create an enormous file, or have to worry about the files being replayed in the correct order. If I knew for certain the order that files would be processed from a directory, perhaps I could manage the filenames as I export.

 

I could move the information to separate globals and then replay them by reading and recreating the original TCP/IP feeds but that sounds like too much work.

 

Does anyone have any experience or creative ideas about how this could be done with minimal work?

 

Dave

[Dale du Preez]

Hi Dave,

It sounds like this is a test environment, so why don't you do something like the following?

Shut Ensemble down and then copy the cache.dat to a new folder.
Restart Ensemble and "create" a new database using that cache.dat, and then create a new Ensemble namespace using that database.
Clear out your current message warehouse.
Add an HL7 TCP Operation to a production in your new namespace that loops back to an HL7 TCP service in your main namespace/production. Write a service in your new namespace to loop over the message  header global/table, open the message objects with MessageBodyClassname = "EnsLib.HL7.Message" and SourceConfigName="YourBusinessService", and then send those objects to the business operation.

When you've finished, you can then remove the duplicated namespace and database, as it now has extra information.

I am sure there are other options as well, though. Does anyone have other ideas?

Dale

[Jim Dolson]

Dave,

Are these existing messages which are already in the database or is this something that you want to start doing with new messages?

Jim

[David Loveluck]

A history of message built up over the last three months.

 

Essentially they have been processed and filed into a data model but the data model has changed so I want to refile them.

 

Thanks

 

Dave

 

David Loveluck

617 225 3122

---

[Andrew Harris]

Have you considered a business operation which writes each message to a separate file, where the file name is built from a specified directory, prefix and sequence number, starting at 1 (or specified number)? Then create a business service which does the opposite – reads files staring at sequence 1 (or specified number) until a specified end number or there are no more. Then you get the files in order and don’t have to read a directory.

 

Andrew

 

---

[Ted Peck]

There are 2 things that come to mind regarding your specific questions:

 

- what's wrong with a huge file?

- the File Adapter uses a directory query (%File:FileSet) to get the files. It currently sorts by DateModified. That should be OK for you if you write your files out in order (and preserve their dates if using a Unix copy to move them).

 

Another option would be to copy your Cache.DAT to a new namespace and make a new Production with a Service that just does a query and feeds out the desired messages in order to a TCP Operation, sending them to the original Production in the cleaned-out,  original namespace with its new data model.

 

Ted

[Ted Peck]

Whoops, I just noticed Dale had already suggested that "other option".

[Ben]

Here is a method we developed to for quering the DB, and writing HL7
messages to file.

-Ben


/// This Method queries HL7 message body IDs for Services and
Operations from the Ens.MessageHeader table within the date range that
is specified.
/// The Date range is by default T-1. If sDate1 is supplied and
sDate2 is not, the date ranges is sDate1.
/// If both sDate1 and sDate2 are supplied, the date range spans the
two dates.
/// The archive path defaults to d:\archive\, but it can be overridden
as the 3rd parameter if needed.
/// Once the message body IDs are acquired, the RawContect of each
message is written to a file that is named CCYY-MM-DD.
[ItemName].hl7.txt
ClassMethod ArchiveHL7(sDate1 As %String = "", sDate2 As %String = "",
ArchivePath As %String = "D:\Archive\") As %Status
{
set InfoMsg = "Start Archive at: "_$ZDT($H)_" | "


If (sDate1="")&&(sDate2=""){
set sDate1 = $ZD($H-1,3)
set sDate2 = $ZD($H-1,3)
} elseif (sDate1'="")&&(sDate2=""){
set sDate2=sDate1
}

set InfoMsg = InfoMsg_"sDate1: "_sDate1_" | sDate2: "_sDate2_" |
AchivePath: "_ArchivePath_" | "

If (sDate1?4N1"-"2N1"-"2N)&&(sDate2?4N1"-"2N1"-"2N) {

set StartStamp = sDate1_" 00:00:00"
set EndStamp = sDate2_" 23:59:59"
//set EndStamp = sDate2_" 01:00:00"

set SQL = ""
Set rset = ##class(%ResultSet).%New()
set rset.RuntimeMode = "1"

set SQL = SQL_"SELECT MessageBodyId, MessageBodyClassName,
SourceBusinessType, SourceConfigName, TargetBusinessType,
TargetConfigName, TimeCreated "
set SQL = SQL_"FROM Ens.MessageHeader "
set SQL = SQL_"WHERE TimeCreated >= '"_StartStamp_"' "
set SQL = SQL_"AND TimeCreated <= '"_EndStamp_"' "
set SQL = SQL_"AND (SourceBusinessType = 1 OR TargetBusinessType =
3) "

Do rset.Prepare(SQL)
Do rset.Execute()

Kill ^CacheTempMsgArchive
set ^CacheTempMsgArchive = ""

While (rset.Next()) {

If rset.Data("MessageBodyClassName") = "EnsLib.HL7.Message" {

If rset.Data("SourceBusinessType")=1 {
set ^CacheTempMsgArchive(rset.Data("SourceConfigName"),
$P(rset.Data("TimeCreated")," ",1),rset.Data("MessageBodyId")) = ""
} ElseIf rset.Data("TargetBusinessType")=3 {
set ^CacheTempMsgArchive(rset.Data("TargetConfigName"),
$P(rset.Data("TimeCreated")," ",1),rset.Data("MessageBodyId")) = ""
}

}

}

//w "end while",!

Set ItemName=""
For {
Set ItemName=$O(^CacheTempMsgArchive(ItemName))
Q:ItemName=""
//w ItemName,!
Set TimeCreated=""
For {
Set TimeCreated=$O(^CacheTempMsgArchive(ItemName,TimeCreated))
Q:TimeCreated=""

Set file=##class(%File).
%New(ArchivePath_TimeCreated_"."_ItemName_".hl7.txt")
If $ISOBJECT(file) = 1 {
Do file.Open("WSN")

//w file.Name,!

Set MsgID=""
For {
Set MsgID=$O(^CacheTempMsgArchive(ItemName,TimeCreated,MsgID))
Q:MsgID=""

set hl7 = ##class(EnsLib.HL7.Message).%OpenId(MsgID)
If $ISOBJECT(hl7) = 1 {
Do file.WriteLine(hl7.RawContent)
} else {
do ##class(Ens.Util.Log).LogWarning("IIBAENSEMBLE.Utilities",
"ArchiveHL7", "Could not create hl7 object: "_MsgID)
}
}

//Close file
do file.Close()
} else {
do ##class(Ens.Util.Log).LogWarning("IIBAENSEMBLE.Utilities",
"ArchiveHL7", "Could not create file:
"_ArchivePath_TimeCreated_"."_ItemName_".hl7.txt")
}
}
}
} Else {
set ErrorMsg = "Arhive HL7 Date 1: "_sDate1_" or 2: "_sDate2_" is
not correctly formatted as ODBC date YYYY-MM-DD"
do ##class(Ens.Util.Log).LogError("IIBAENSEMBLE.Utilities",
"ArchiveHL7", ErrorMsg)
}

set InfoMsg = InfoMsg_"End Archive at: "_$ZDT($H)

do ##class(Ens.Util.Log).LogInfo("IIBAENSEMBLE.Utilities",
"ArchiveHL7", InfoMsg)

Quit $$$OK

>     Dave- Hide quoted text -
>
> - Show quoted text -

[Jim Dolson]

Hi Ben.

I know enough to be impressed with your code but not enough to fully understand it  :-)

Is the statement "If $ISOBJECT(hl7) = 1 { Do file.WriteLine(hl7.RawContent)" accessing the RawContent field that contains just the first 10,000 characters of the HL7 message?

Jim

[Graham, Ben]

Yes it is, and for us the 10K is sufficient.

I think you could use hl7.OutputToString() to get the complete message.

 

You would need to test this a bit, but it should work.

 

-Ben

 

From: Ensemble-in...@googlegroups.com [mailto:Ensemble-in...@googlegroups.com] On Behalf Of Jim Dolson

Sent: Wednesday, June 11, 2008 8:44 AM
To: Ensemble-in...@googlegroups.com

This electronic transmission and any documents accompanying this electronic transmission may contain information that is confidential and/or legally privileged. The information is intended only for the use of the individual or entity named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the taking of any action in reliance on or regarding the contents of this electronically transmitted information is strictly prohibited. If you have received this e-mail in error, please notify the sender and delete this message immediately.

[Ted Peck]

Good catch - the right thing to do is

 

  If $ISOBJECT(hl7) = 1 { Do hl7.OutputToLibraryStream(file)

 

You need to use the hl7 methods to get the whole content out. As Jim pointed out the property is potentially truncated. It also gives less control over separators and framing.

Ted

 

----- Original Message -----

[Andy]

We have to replay messages lots of time for all sorts of reason

In our production we created a service call resendin, its just a tcp
inbound but is not started (doesn't need to be for this)
We create an outbound Operation probably HL7 file base or use an
existing operation or process that I want to send these replayed
messages to.

then I have this code that I run from a terminal windows. Yes it's
crude but it works fine

change the SourceConfigName to be the name of the source Business
service and the TimeCreated to be your start, add a end date if
required
change the "name of target Process or Operation" to the name of the
Business process or Business Operation you want send them to.

ClassMethod Resend() As %String
{
set x = 1
set sqll = "SELECT MessageBodyId FROM Ens.MessageHeader"_
" WHERE SourceConfigName = 'Name of Source service' AND
TimeCreated > '2008-05-15 15:13:38.659'"_
" ORDER BY TimeCreated"
w sqll
set rs = ##class(%ResultSet).%New()
set status = rs.Prepare(sqll)
set rs.RuntimeMode = 1
w status,!
set status = rs.Execute()
w status,!
while rs.Next() {
// just because I like to see something happening
if $P((x/1000),".",2)
= "" {
w "Replaying "_x,!
}
set x = x + 1
set mymessageid = rs.GetData(1)
set mymessagepointer = ##class(EnsLib.HL7.Message).
%OpenId(mymessageid)
// I have the message
so if I want to filter on message content I can here
set tService =
##class(Ens.Director).CreateBusinessService("resendin",.service)
do service.SendRequestAsync("name of target Process or
Operation",mymessagepointer)
}
}

You can use this to replay messages, or send to a file to load in
later using a filebased inbound. I have processed anything up to
500000 messages through this without problems

Andy Richards
Mater Health Services

[David Loveluck]

Thanks to everyone who answered. In this case, it was decided that we could reduce the number of messages resent by an order of magnitude. So exporting everything to a file worked well - we had a 10 MB file. We read that in with a standard HL7 file adapter with no problem.

 

My concern about one big file was the ability to throttle the throughput and also to restart in the case of failure.

 

I have written routines before that find and resend messages, but I have always added in a small hang every few messages and kept track of how far I have gone.

 

These can all be resolved with a little thought but I didn't need to in this case.

 

dave

 

 

David Loveluck

 

-----Original Message-----
From: Ensemble-in...@googlegroups.com [mailto:Ensemble-in...@googlegroups.com] On Behalf Of Andy
Sent: Thursday, June 19, 2008 9:03 PM
To: InterSystems: Ensemble in Healthcare
Subject: [InterSystems-EnsHlth] Re: saving and replaying messages received over TCP/IP

 

 

We have to replay messages lots of time for all sorts of reason