Re: Parse PHINMS Blobs

[Frans de Wet]

Yes, what do you want to do?

Frans

On 28 Nov 2012 20:03, "Ray Humphrys" <hump...@gmail.com> wrote:

Has anyone tried to parse the BLOBs that are components of the PhinMS queue tables?

--
 
 
 

[Lowe, Phillip (DOH)]

Sure.  We had a system running that would read blobs out of selected records and pass them on to another component for processing.

 

We also have tools for unloading and loading the PHINMS tables directly so we can link other applications to PHINMS.

 

In a couple of cases, we are using SQL to take an inbound message and turn it into an outbound message.  We use our development PHINMS server to move certain data from the development network to our production network and then forward it to the CDC.  All great fun once you see what PHINMS is doing with the tables J

 

 

Phill Lowe (360) 236-4261 Philli...@DOH.WA.GOV

Informatics Office
"The Department of Health Works to Protect and Improve the Health of People in Washington State"
(DOH email filters occasionally block legitimate email. Call me if you don't receive a response within 24 hours.)

[Preacher Man]

To be clear, those "BLOBs" are simply the PHIN-MS payloads stored within the dB API in binary format.  The content is identical to what you get if you simply write the payload to the file system.  So any "parse" is dependent on the expected payload content, regardless of how you stored it.

Personally, I never saw much point in a "BLOB" storage type for dBs.  There is actually LESS meta information available from the dB than you get with the file system (time stamps, groups, owners, permissions, etc. unless you build it into your schema as additional fields like PHIN-MS does...  but then the meta data is not really managed by the dB), and under the hood a lot of dB's simply use the file system for BLOB storage anyway (where the BLOB is just a pointer to an inode for example).  And you always have to use a special function to actually get the content (sorry, can't just "select blob" and get the data).  

But it does allow one to stay with the SQL API, and at this point it becomes a "religious" issue (my API is the one and only true API :-).  I'd rather parse say an HL7 payload with anything OTHER THAN SQL (C, Java, Python, even Javascript), so I find the file system a more convenient payload  interface.  YMMV.

Tom

---

Date: Wed, 28 Nov 2012 12:29:33 -0800
From: hump...@gmail.com

[Frans de Wet]

If you check the box that says it is Text Payload for a receiver the payload will be written to the payloadtextcontent column.  It is much easier to get it from there, if it matters to you.  You can access it directly via selects.  You may run into 8000 byte limits here and there though, so be careful ...

I have mechanisms for MySQL and for SQL Server to easily convert the data into something usable within the select statements.  

We also have Mirth Connectors that actually allow you to read and write to and from these tables.  That makes it a snap too ;-)

If you are merely pulling the data from the payloadbinarycontent column you can just use your favorite Oracle function to base64 decode the data.  You may need some other type conversions to get at the data too.  

Let me know if you need with any specific part of this.  Simplest may be to just check the box in the PHINMS configuration to say the payload is of the type text.  

Frans

Thanks,

Frans de Wet
Integration Engineer
(850) 583-0041

On Thu, Nov 29, 2012 at 1:08 PM, Ray Humphrys <hump...@gmail.com> wrote:

I want to either parse the payload (Blob) and write the contents into rows in an Oracle table, or grab the contents before they are converted to a Blob and write them to a table for further processing.  Supposedly they would have to in an ASCII format early on in the process, I think.  Since they start as ASCII they have to be this way somewhere.

--
 
 
 

[Lowe, Phillip (DOH)]

No, you really don’t want to do that!

 

There are a couple of cases where you will receive messages that are still encrypted – think of an error in the yearly certificate updates (on both ends).  The metadata in the receiving table is your friend!  It is much smarter to do a poll on a regular basis and look for messages that have arrived (applicationstatus = null) and encryption is no and transportstatus = success.  You know that this message has been received (not still coming in) and that it was successfully decrypted.  Once you process this message, you set applicationstatus to something and you are ready for the next step. 

 

I have a series of jobs that look for specific applicationstatus values – when I find them and they are more than 7 days old, they get deleted.  A bit of scheduling will keep your PHINMS database from growing at all. This also means that I can go back and reset applicationstatus and deliver the message again if needed.  Any messages that fail to decrypt won’t become garbage in the database because they won’t be processed.  This is a real plus in some applications where deleting data is a nightmare.

 

You can also use the database with similar queries to Tee data so that it goes to multiple destinations.  We have some data that comes in, gets Teed with one copy going to disk (using a tool) and archived and the second copy gets sent back out to one of our partners – mostly managed with SQL jobs.

 

I also monitor our data flows by looking at the metadata fields in the phinms tables.  I know when a partner’s PHINMS locks up because I don’t see expected traffic.  I know when we are failing to successfully deliver data to partners.  All it takes it a bit of time watching the tables as PHINMS runs to see what the various metadata fields are and you can build scripts to your heart’s content.  J

 

Phill Lowe (360) 236-4261 Philli...@DOH.WA.GOV

Informatics Office
"The Department of Health Works to Protect and Improve the Health of People in Washington State"
(DOH email filters occasionally block legitimate email. Call me if you don't receive a response within 24 hours.)

 

From: phi...@googlegroups.com [mailto:phi...@googlegroups.com] On Behalf Of Ray Humphrys
Sent: Thursday, November 29, 2012 10:09 AM
To: phi...@googlegroups.com
Subject: Re: Parse PHINMS Blobs

 

I want to either parse the payload (Blob) and write the contents into rows in an Oracle table, or grab the contents before they are converted to a Blob and write them to a table for further processing.  Supposedly they would have to in an ASCII format early on in the process, I think.  Since they start as ASCII they have to be this way somewhere.

 

On Wednesday, November 28, 2012 8:04:01 PM UTC-5, Frans de Wet wrote:

--
 
 
 

[Frans de Wet]

You do not want to do what?

Frans

Thanks,

Frans de Wet
Integration Engineer
(850) 583-0041

--
 
 
 

[Lowe, Phillip (DOH)]

Sorry, I was replying to an earlier message in between other work…  This is was I was responding to. I don’t want to grab the contents before they are converted to a BLOB.

 

I want to either parse the payload (Blob) and write the contents into rows in an Oracle table, or grab the contents before they are converted to a Blob and write them to a table for further processing.  Supposedly they would have to in an ASCII format early on in the process, I think.  Since they start as ASCII they have to be this way somewhere.

 

 

Phill Lowe (360) 236-4261 Philli...@DOH.WA.GOV

Informatics Office
"The Department of Health Works to Protect and Improve the Health of People in Washington State"
(DOH email filters occasionally block legitimate email. Call me if you don't receive a response within 24 hours.)

 

--
 
 
 

[Frans de Wet]

That is what I was thinking ... just wanted it clear ;-)

Thanks,

Frans de Wet
Integration Engineer
(850) 583-0041

--
 
 
 

[Frans...@doh.state.fl.us]

Orion Rhapsody does have actual comm-points.  But they are basically intended to replace PHINMS.  So, you can send from within Rhapsody to a remote PHINMS (route not read or standard receiver) without every installing PHINMS.  Same goes for incoming connections.  We have configured this a time or two before.  The certificate part can be a challenge, but it works at many states and other facilities.

Thanks,

Frans de Wet
Integration Engineer
Florida Department of Health - Applications Development
frans...@doh.state.fl.us
850.245.4444 ext. 3257
Mission: To protect, promote & improve the health of all people in Florida through integrated state, county, & community efforts.

Please note: Florida has a very broad public records law. Most written communications to or from state officials regarding state business are public records available to the public and media upon request. Your email communication may therefore be subject to public disclosure.

 

---

From: phi...@googlegroups.com [mailto:phi...@googlegroups.com] On Behalf Of Ray Humphrys

Sent: Thursday, November 29, 2012 1:31 PM

To: phi...@googlegroups.com
Subject: Re: Parse PHINMS Blobs

Hey,   thanks let me take a look at our Phinms configs.  

" Mirth Connectors" - we use Orion's Rhapsody.  I wonder if they have a similar widget/commpoint?

On Thursday, November 29, 2012 1:24:26 PM UTC-5, Frans de Wet wrote:

If you check the box that says it is Text Payload for a receiver the payload will be written to the payloadtextcontent column.  It is much easier to get it from there, if it matters to you.  You can access it directly via selects.  You may run into 8000 byte limits here and there though, so be careful ...

I have mechanisms for MySQL and for SQL Server to easily convert the data into something usable within the select statements.  

We also have Mirth Connectors that actually allow you to read and write to and from these tables.  That makes it a snap too ;-)

If you are merely pulling the data from the payloadbinarycontent column you can just use your favorite Oracle function to base64 decode the data.  You may need some other type conversions to get at the data too.  

Let me know if you need with any specific part of this.  Simplest may be to just check the box in the PHINMS configuration to say the payload is of the type text.  

Frans

Thanks,

Frans de Wet
Integration Engineer
(850) 583-0041

On Thu, Nov 29, 2012 at 1:08 PM, Ray Humphrys <hump...@gmail.com> wrote:

I want to either parse the payload (Blob) and write the contents into rows in an Oracle table, or grab the contents before they are converted to a Blob and write them to a table for further processing.  Supposedly they would have to in an ASCII format early on in the process, I think.  Since they start as ASCII they have to be this way somewhere.

On Wednesday, November 28, 2012 8:04:01 PM UTC-5, Frans de Wet wrote:

Yes, what do you want to do?

Frans

On 28 Nov 2012 20:03, "Ray Humphrys" <hump...@gmail.com> wrote:

Has anyone tried to parse the BLOBs that are components of the PhinMS queue tables?

--
 
 
 

--
 
 
 

--
 
 
 

---

Spam
Not spam
Forget previous vote

[Preacher Man]

Frans,

If you could Post an SOP for configuring a Rhapsody PHIN-MS compatible sender that would be REALLY great.  I've had clients that wanted to do this, but I couldn't help them (since I don't have Rhapsody),  they didn't seem to get much traction from Orion, and the only thing I could find was a pre-configured file for sending to CDC... essentially a black box plugin.

I know this is supposed to be doable, but all the demos I've ever seen stuck PHIN-MS in front of Rhapsody (sigh).

Tom

---

Subject: RE: Parse PHINMS Blobs
Date: Thu, 29 Nov 2012 13:50:44 -0500
From: Frans...@doh.state.fl.us
To: phi...@googlegroups.com

--
 
 
 

[Preacher Man]

re:widgets....

Mirth is almost, but not quite as pointy clicky as Rhapsody, but includes a whole bunch of pre-defined operations.  You can plug in your own Javascript and/or Java brokering as well.

Tried both of them (as well as Interfaceware's product) for data brokering.  I ended up using the OHF HL7 Java toolkit embedded inside a PHIN-MS receiver servlet.  Fewer moving parts.  YMMV.

Tom

---

Date: Thu, 29 Nov 2012 10:31:28 -0800
From: hump...@gmail.com

To: phi...@googlegroups.com
Subject: Re: Parse PHINMS Blobs


Hey,   thanks let me take a look at our Phinms configs.  

" Mirth Connectors" - we use Orion's Rhapsody.  I wonder if they have a similar widget/commpoint?

--
 
 
 

[Frans de Wet]

I have accessed them via mirth.

Frans

On Nov 29, 2012 3:01 PM, "Ray Humphrys" <hump...@gmail.com> wrote:

Since we are talking about the internal PhinMs storage - has anyone dealt with the CLOBS? Parse them, and write them into a table.

On Thursday, November 29, 2012 1:31:28 PM UTC-5, Ray Humphrys wrote:

Hey,   thanks let me take a look at our Phinms configs.  

" Mirth Connectors" - we use Orion's Rhapsody.  I wonder if they have a similar widget/commpoint?

--
 
 
 

[Frans de Wet]

Tom,

I will poke around and see what I have, or put one together.  

Frans

Thanks,

Frans de Wet
Integration Engineer
(850) 583-0041

--
 
 
 

[Preacher Man]

Same girl, different dance. A CLOB is really just a BLOB that can only have character data.  Since PHIN-MS is payload agnostic it has to support both binary and character payload content.

As an aside I'll note that size can be problematic for some SQL interfaces (as Frans  pointed out earlier).  Most ODBC and JDBC drivers impose limits on field size that are independent of the dB (e.g. even if your dB will support BIG fields, they get squished by the API on the remote side... Yup, I've had that fight too).  Another reason I stick with the file system for payload storage if I can.

Tom

---

Date: Thu, 29 Nov 2012 11:15:33 -0800
From: hump...@gmail.com

To: phi...@googlegroups.com
Subject: Re: Parse PHINMS Blobs

Since we are talking about the internal PhinMs storage - has anyone dealt with the CLOBS? Parse them, and write them into a table.

On Thursday, November 29, 2012 1:31:28 PM UTC-5, Ray Humphrys wrote:

Hey,   thanks let me take a look at our Phinms configs.  

" Mirth Connectors" - we use Orion's Rhapsody.  I wonder if they have a similar widget/commpoint?

--
 
 
 

[Preacher Man]

Teams?  You get Teams???!!!!  (:-)

Seriously, I'm curious, how many developers do you have working on messaging/data brokering?

Tom (I are the team... ahuck... ahuck..  :-)

---

Date: Thu, 29 Nov 2012 05:19:30 -0800
From: mark.h...@state.mn.us

To: phi...@googlegroups.com
Subject: Re: Parse PHINMS Blobs

Phil,
Your last sentence is very interesting. At MDH, we put all received PHINMS messages into Oracle DB tables and code (java or perl) has been given to the teams that want the data. We work with both CLOBs and BLOBs. But when you said, "All great fun once you see what PHINMS is doing with the tables" I was very interested. Could you explain that a little.

As an aside, we use the embedded HSQLDB for outgoing messages since we send so few. But I have one team that needed to add extra parameters in a series of sends that was a nightmare to think of using Folder Polling, so I write those directly to the HSQLDB outgoing table.

[Hollock, Mark (MDH)]

Tom, Funny. When I said teams, I didn’t mean developers. I, too, am a one man army. But we have people who actually care about the content of the data, epidemiologists, and they are the teams I referred to. Like immunization, disease reporting. I’m lucky enough that I don’t have to look into the content of files, just deliver them. J

Mark

 

 

From: phi...@googlegroups.com [mailto:phi...@googlegroups.com] On Behalf Of Preacher Man
Sent: Friday, November 30, 2012 8:03 AM
To: phi...@googlegroups.com
Subject: RE: Parse PHINMS Blobs

 

Teams?  You get Teams???!!!!  (:-)

--
 
 
 

[david....@nebraska.gov]

Tom,

I am interested in this.  Much of what I have found online so far about OHF is outdated (as in 2007).  Is it still active somewhere I haven't looked?  For that matter, I'd be interested in looking at the source of this OHF Java library even if it is outdated, but I haven't found source or binaries yet.

Thanks, cheers,

--Dave

On Thursday, November 29, 2012 1:43:09 PM UTC-6, tom wrote:

[...] 

I ended up using the OHF HL7 Java toolkit embedded inside a PHIN-MS receiver servlet.

[...] 

[Preacher Man]

It WAS part of the Eclipse projects list (http://www.eclipse.org/proposals/eclipse-ohf/).  However the download page appears to have disappeared (404's).  I don't know what's up with that.

I've got some Zips somewhere that I can provide.  Let me know.  

However, if the OHF project is dead you might want to consider looking at some alternatives.  I've used the hl7light library available from nule.org.  I've also written a "tiny" HL7 implementation that is part of my PhinmsX project (https://github.com/tdunnick/PhinmsX). It doesn't support XML, but is about 100x faster than OHF, as is the nule library.  And the API is way easier to learn.  OHF is so big you really have to invest in the  "learning curve".  Unless you must have XML/v3 support you really don't need all that.

Meanwhile, maybe I bug my contact at IBM about OHF...

Cheers

Tom

---

Date: Fri, 18 Jan 2013 08:47:47 -0800
From: david....@nebraska.gov

To: phi...@googlegroups.com
Subject: Re: Parse PHINMS Blobs

--
 
 
 

[Loyall, David]

Regarding PhinmsX, nice to meet you.  I starred your repo on github a couple of days ago. :)  “It breaks beginning with 2.8.01sp1 where the PHIN-MS API changes.”  We’re running a 2.7 receiver, but I intended to upgrade to 2.8.02 soon.  Should I not?

 

Regarding hl7light, let’s see if I can CC the author.  Mike, you there?  I see nule.org is recovering from the fallout from the wordpress problems.  When you get a chance, is there any chance you could publish a list of links to your SVN repos?

 

Regarding OHF, sure, I’m not surprised to hear that it is slow.  But then, I wouldn’t be surprised if it were complete, either*.  http://en.wikipedia.org/wiki/Completeness#Computing (I’d like a library that could load any possible (valid) hl7 message.  In the mean time, data keeps flowing and I’ll be happy with a subset.)

 

Cheers,

--Dave

 

* assuming that they finished it or that someone will finish it one day

 

From: phi...@googlegroups.com [mailto:phi...@googlegroups.com] On Behalf Of Preacher Man
Sent: Friday, January 18, 2013 11:09 AM
To: phi...@googlegroups.com
Subject: RE: Parse PHINMS Blobs

 

It WAS part of the Eclipse projects list (http://www.eclipse.org/proposals/eclipse-ohf/).  However the download page appears to have disappeared (404's).  I don't know what's up with that.

--
 
 
 

[Frans...@doh.state.fl.us]

We use HAPI (http://hl7api.sourceforge.net/) quite a bit ... it loads just about any message I have come across.  Or am I just reading the tail end of the story ...

Thanks,

Frans de Wet
Integration Engineer

Florida Department of Health - Applications Development

frans...@doh.state.fl.us
850.245.4444 ext. 3257

Mission: To protect, promote & improve the health of all people in Florida through integrated state, county, & community efforts.

Please note: Florida has a very broad public records law. Most written communications to or from state officials regarding state business are public records available to the public and media upon request. Your email communication may therefore be subject to public disclosure.

 

---

From: phi...@googlegroups.com [mailto:phi...@googlegroups.com] On Behalf Of Loyall, David
Sent: Friday, January 18, 2013 2:44 PM
To: phi...@googlegroups.com
Cc: mi...@thot.us

Spam
Not spam
Forget previous vote

--
 
 
 

[Loyall, David]

Thanks, Frans.

 

I have had the HAPI Test Panel for some time, but I hadn’t looked at the API in detail.  Now I have, and I like it.

 

But if anyone knows what happened to OHF, I am still interested in knowing.

 

Cheers and thanks again,

 

--Dave

--
 
 
 

[Loyall, David]

Here is a message from the hl7light author. (below my signature.)

Mike, FYI the mailing list web interface is here: https://groups.google.com/forum/?fromgroups=#!forum/phinms ...

Cheers,

--Dave

-----Original Message-----
From: mi...@thot.us [mailto:mi...@thot.us]
Sent: Friday, January 18, 2013 16:43 PM
To: Loyall, David
Cc: phi...@googlegroups.com; michael.l...@gmail.com
Subject: RE: Parse PHINMS Blobs

Hi,



The link for the LightHl7Library SVN Repo is:

https://svn.thot.us/svn/nule.org_LightHl7Lib/



Its philosophy for handling HL7 is akin to the concept of a "well-formed"

XML document. As long as the delimiters are correct, it really doesn't

care the content. It tends to be fast and robust for that reason.



Let me know if you run into any problems, I'm always happy to lend a hand.

I'll also be trying to update the website with some more of the missing

information from my wordpress implosion in coming days.



Thanks,

Mike







On Fri, 18 Jan 2013 19:43:46 +0000, "Loyall, David"

> From: david....@nebraska.gov<mailto:david....@nebraska.gov>

> To: phi...@googlegroups.com<mailto:phi...@googlegroups.com>

[Preacher Man]

Re upgrading to 2.8.02:  I wouldn't let  PhinmsX stop you.  For one thing, you could simply drag the 2.7 jars into the PhinmsX context and still be running 2.8.02 in the PHIN-MS "receiver" context. Or it wouldn't be a huge deal to re-engineer PhinmsX for the new(er) API.  Logging and a couple of the encryption methods changed IRRC.  Its the folk who don't have a earlier version of PHIN-MS that are cut out of the game currently.

In development I was at a point where I either needed to create two versions of PhinmsX, or resort to Java reflection to code to the API in a version independent manner.  If CDC were to make ebxml.jar distributable I would probably just bundle whatever version I target and provide a complete WAR, along with the "build your own" distribution.  But I think going forward a "PHIN-MS free" version that was not dependent on any of the CDC binaries would be the way to go.  That would eliminate both the version dependencies and the distribution issues.

Note the PhinmsX HL7 (and other parts like the XML interface) are already "PHIN-MS" free and can stand alone if you want to use them separately.

Re OHF:  Yes, very complete.  The down side is that is also very strict, especially when parsing or formating XML.  Of the two dozen HL7 stream sources I've dealt with coming from a variety of LIS/LIMS and HL7 interface vendors, almost none will successfully load without some "pre-processing" of the data.  There are some big names there that I can't mention,  but almost nobody can satisfy an ORU XSD in my experience.  I implemented a FilterInputStream  to manage that.  If I were starting over, I'd go with a small and loose.  After all, the only reason to load/parse HL7 is to broker the data anyway.  Its easier to fix things inside the HL7 framework, than in a pre-processor.

Tom

---

From: david....@nebraska.gov
To: phi...@googlegroups.com
CC: mi...@thot.us

Subject: RE: Parse PHINMS Blobs

Date: Fri, 18 Jan 2013 19:43:46 +0000

--
 
 
 

[Preacher Man]

Wow Frans... don't know how I missed that one when I started down this road in 2005.  HAPI looks both mature and active.  Are you actually coding to their API, or just using the pre-built tools?

Tom

---

Subject: RE: Parse PHINMS Blobs

Date: Fri, 18 Jan 2013 14:50:57 -0500
From: Frans...@doh.state.fl.us
To: phi...@googlegroups.com
CC: mi...@thot.us

--
 
 
 

[Loyall, David]

I found some answers of my own regarding OHF.

 

Last message on the OHF developers mailing list:

http://dev.eclipse.org/mhonarc/lists/ohf-dev/msg01022.html

 

Full archive of their old website, including a copy of all their source repositories (5.3gb file!)

http://archive.eclipse.org/technology/archives/ohf.tgz

 

But since much of it apparently migrated to other projects, maybe the useful information I need isn’t the OHF data itself, but a map of where all the pieces went when it was split up and migrated to other projects...  I’ll CC the last known-active OHF dev.  (Oh, it turns out that he helped shape the hl7 standard!)

 

Grahame, Hello.  I have a question.  What HL7 parser would you use if thoroughness is the primary requirement?  (The messages below should provide some context.)

 

Cheers, and thanks,

 

Dave Loyall

State of Nebraska

Office of the CIO

Web Development Team

david....@nebraska.gov

(402) 471-0677

--
 
 
 

[Preacher Man]

Interesting.  I briefly checked out the OHT website.  Looks like they took "huge" and made it "really huge".  The usual bloat associated with IBM and their adopted projects.  The OHF HL7 core is still relatively big, but just a mote in the OHT "eye".
 
I was somewhat challenging when I first picked up the OHF HL7 core, to find it among all the other stuff.  The source that I have (from the Eclipse CVS repository 8/30/2011) runs a bit over 14M.  My eclipse workspace for that and a few other tid bits is 66M (71.5M on disk).  I'd be surprised if it has changed much since.  IBM really didn't do much with it other that leverage it for creating their IHE suite.  Here is specifically what I'm using:
 
org.eclipse.ohf.hl7v2.core
org.eclipse.ohf.utilities
 
As I illuded to previously, "thoroughness" is a double edged sword.  The OHF HL7 core can challange your notion of what proper HL7 looks like (:-).
 
Tom
 

---

From: david....@nebraska.gov
To: phi...@googlegroups.com

CC: gra...@healthintersections.com.au

Subject: RE: Parse PHINMS Blobs

Date: Tue, 22 Jan 2013 23:39:48 +0000

--