Rome being rebuilt in a day

[Kevin Toppenberg]

Hey all,

There has been discussion in the past about how much VistA code is old and could stand some refreshing.  But because of IT decisions by the VA it wasn't being done.  And there has always been the concern of cost and the philosophy of "If it ain't broke, don't fix it."  So we have been a bit stuck in the past.  

I have heard it said that "Rome wasn't built in a day."  The message I took from this in regards to VistA was that it took a long time to get where we are today, but we should just keep moving forward and we can achieve big things over time. 

Well, I think AI is turning this idea on its head. 

There are many things about CPRS that show its age and pedigree.  For example, in the Notes tab, it is very inefficient.  We have some patients with a vary large number of note titles.  The default is to just bring in the last 100 notes.  But if you want to go back further, I must select it to load 1000 notes.  And this can take almost 30 seconds!  This has always bothered me as being SO SLOW!!!  So I decided to use AI to get to the bottom of it.  

The situation, as i recall it (I may be off on a few details) was as follows:

-- The note selector on the left side of the tab is not the true selector.  It seems to have been added on later.  The true selector was the listbox that shows up at the top of the page when a group is selected.  This GUI arrangement look similar to the CHUI interface in the VistA commandline note editor, and my theory is that when moving from CHUI to GUI they copied that, and then later implemented the side treeview notes selector (tvNotes).     

-- The action taken when  a user would select a note in the treeview (TV) was multi-stepped.  There was an event from the TV, which would then manually create a new event for the listbox etc.  The end result was that sometimes the view would update and then update again, leading to form flicker at times.  Very inefficient.  

-- There was a hidden control on the notes tab, lstNotes, that was being used as a data store.  And as I recall there was duplication of data in several places.  And not only was the raw string data being stored, it was also being parsed into a formatted dataobject for each entry. This involves instantiating the object (fetching memory) and then parsing the string (using Piece() ) into the various fields.  

-- Furthermore, the AI said that the parsing was recursing such that compute time was quadratic (time^2).  

To address these issues, I did the following in my CPRS.  And by the way, this is not yet my official version.  I have been moving TMG-CPRS code such that it can be compiled by Delphi 12 (community edition).  This involved lots of changes due to strings all being multi-byte by default.  So all this is in a development build. 

-- First I got rid of using a form control as the data store (the "model" in data parlance).  I created a pure data object to store the note data info, as returned by the RPC.  

-- I got rid of the listbox selector that shows up when a group node is selected in the treeview (TV).  FYI, a group node is something like "All signed notes" or "Notes currently being edited".  Now when such a  node is selected in the TV, it now just drops to the first child node.  

-- I then changed the treeview on the left panel (tvNotes) to implement lazy loading, such that when loading tvNotes, it parses only the title name for display, but leaves the rest unparsed.  Then, when the user clicks a particular node, it just-in-time parses and saves off the compiled data.  Thus if I load 1000 note title, so I can find the operative report from 5 yrs ago, the parsing of all 1000 note data is skipped in favor of parsing just the one that the user selects.  

-- I also spent time on the Piece() functions because that is used heavily in parsing.  Much of the code will put the datastring line in variable x, and then compose code like this:  ouput = piece(x,U,3) + piece (x,U,7) + piece (x,U,8) ...   And for each of these parse efforts, a new scan of x must be started from the start .  We addressed this in a few ways.  Only for times where there was going to be a huge parse, such as with loading 1000 notes, we did a prescan of the string and saved off the start position and length of each piece, so that subsequent fetches would be much faster.  There was already a function for PiecesToList that output into a TStringList (SL), but there is overhead there of fetching memory for the object and duplicating the source strings into the SL.    We also created some other utility functions such as PieceEquals for use when comparing a piece to a value.  Rather than copying the source string to a result string and then comparing that to the value, it would compare directly inside the source string.    In the end, all of these improvements made only a small improvement in parsing time (sadly). 

-- We also got rid of the recursive parsing.  I didn't fully understand what the AI was describing as the source of the recursion.  Perhaps it was some sort of data FindByID() which would have to scan through every item each time is processes a new one.  But it fixed it (see times below).

-- Another issue was that because the loading was so long, we had implemented a progress bar.  In order for that progress bar to be refreshed and repainted, we would have to call Application.ProcessMessages().  This gives the application time to address any work (such as repainting) that it needs to do.   But it can also slow things down if it decides to do other things.   So we changed this such that it would update only after a certain number of notes have been processed.  We started at every 10 notes, but as things got faster, I think I increased that number perhaps as high as every 100 notes. 

And now the results...  (drum roll please)

Our load time dropped from 27 seconds for 1 patient's notes down to 6 seconds!    That feels like a huge improvement when sitting at the desktop waiting.  

So I am writing this because I feel like it has opened a new vista (pun intended) of opportunity.  There is a lot of cruft that has built up over the years, and it has never been worth the time and effort to clean it up.  But now it is possible.  And that makes me very happy.  

Another cleanup I did to TMG-CPRS was related to the embedded web browser.  When I first started, I didn't know the boundary between what the underlying Internet Explorer (IE) was doing and what Delphi wrapper code was doing.  So I had brought in these open source project to make the web browser act like a text editor.  But with time, I ended up not using all that stuff and it was just lots of files with perhaps one function being used.  So I had AI go through and trace the code paths, and move everything that was actually used into 1 file and was able to get rid of many useless code files.  Much nicer!

My next project is as follows:

-- In the reports tab, there is an Imaging report that shows radiology reports, for example a Chest X-ray (CXR) report.  We get these reports via HL7 from the hospital.  But there are also times when we get faxed radiology reports from other facilities we don't have an interface with.  What to do with those?  We have been scanning them in as TIU notes with a title like RADIOLOGY STUDY.  (For those that don't know, TMG-CPRS supports notes in HTML format such that we can view images directly inside notes.)  This works, but it also means we have to look in 2 locations for studies, and often we may forget and look in only 1 place -- missing information. 

-- So my goal is to include the TIU titles in the radiology report area, and have the display window intelligently change from a TMemo area (current situation) to a TWebBrowser when needed such that it can display the scanned radiology report (which we implement via HTML).   Such a project would be a long undertaking and I frankly have been unwilling to start it before this.  But now I fairly confident I can achieve this fairly quickly. 

Is AI making me a bit hazy on the exact details of what is done?  Probably (as demonstrated above). But does it help me achieve my goals?  Definitely!

Perhaps Rome can be built in a day. 

Kevin

[Kekoa]

This is awesome to hear. I do enjoy learning about these types of accomplishments.

[Sam Habiel]

>  This GUI arrangement look similar to the CHUI interface in the VistA commandline note editor, and my theory is that when moving from CHUI to GUI they copied that, and then later implemented the side treeview notes selector (tvNotes).

That's a major insight. That corroborates my theory that the design of CPRS has not changed much since it was written in 1998. One thing that always annoyed me is that the Problem List/Encounters uses diagnoses list from the Encounter Forms package, and that has been very difficult to use.

Back to a discussion of AI: It is completely upending the IT industry. Many projects that were too difficult to accomplish are now doable. And various ideas about replacing legacy systems because of people who maintain them retiring are no longer true. But it's not magical. It's just a super-charged programmer who can do things 1000 times faster; and just as capable of making mistakes.

--Sam

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hardhats/33c653af-8e4d-4584-81a5-4b45b6739da1n%40googlegroups.com.