A clean VistA database file
Jawad Shafiq
I am working on preparing one clean VistA database file. As we all know the CACHE.DAT file placed at VA ftp site is actually pulled out from a running environment and they Destroy/Delete all their proprietary and confidential data before placing it on ftp site.
Now that CACHE.DAT file has some erroneous data in its Database.
I performed a Data Dictionary Check and a Data check on the Database file that i downloaded from VA ftp site. i found quite some errors in both Data dictionary and Data.
I need you gurus to guide me on how should i proceed to accomplish this task.
Q1- Should i try to fix Data Dictionary?? if Yes, what approach would be best? what i think is to first take a look at all those DD problems and categories them. After that one can think on how to fix a specific type (category) of DD problem.
Q2- For the Data fixing part, i was wondering would it harm if i delete all the entries from all files that does not contain reference data. And by reference data i mean zip codes, cpts, LAB tests etc.
Actually what i would like to have is a fresh database which has all the reference data but none of the data that actually gets into the system/database when the system is running/live.
To start with, what if i delete all data from DOMAIN, INSTITUTION, MEDIAL CENTER Division, Hospital Location and NEW PERSON FILE...but will keep POSTMASTER and CAMERON account.
Any suggestions/help will be appreciated.
Greg Woodhouse
from scratch, and it probably (!) still is, if you have access to all
the patches. But remember that this is thousands upon thousands of
patches that need to be installed. The first time I ever set up a VistA
system (on MSM), the basic process was simple: install Fileman, install
tandalone KIDS (an application for bootstrapping Kernel ONLY, not a
full implementation of KIDS), instally Kernel, Toolkit and Mailman.
Make sure everything is up to date, and then proceed to install (and
patch) the applications. Because of cross-package dependencies, this is
a bit tricky, and you need to work out the right installation order
(which can be a non-trivial task). But originally, each package had a
maintenance release very two years. This is very valuable to people
beinging up new systems, but that practice has been abandoned in favor
of continuous patching. I don't know if anyone has successfully
installed from scratch in recent years.
Of course, my hope is that a release cycle can be developed for non-VA
adopters, making it easier to do what you propose. Other people are
working on this.
In the meantime, well, you probably need to start systematically
removing what you don't want, cleaning up as you go. You'll be breaking
new ground, and it's very important to have experienced VistA
developers working with you.
As to your specific questions:
01- What do you mean by 'fix the data dictionary"? Are you talking
about removing files or fields? Fileman provides utilities to do this,
and you should always use the Fileman utilities to modify the data
dictionary. If for some reason this doesn't seem workable, please
explain why, and maybe we can help you. Editing the data dictionary by
hand is very tricky business.
)2 - When deleting data, you need to proceed carefully and slowly.
There are several stumbling blocks here, including triggers, MUMPS
cross-references, and integrity constraints built into options or
routines. There are several files that just should not even edited in
Fileman. The individual packages provide APIs to work with those
filles, and deleting data can compromise the integrity of packages,
even when you don't think it will. I've mentioned, for example, that
you find IENs embedded in free text fields, and cleaning them up can
take, well, some pretty detailed knowledge of the "anatomy" of VistA.
Keep in mind that there are also files that are populated as part of
the installation process for a package or patch, and you DON'T want to
simply delete the data in those files. There may be no way to recover
it.
Encouraging, aren't I? I'm not saying you shouldn't undertake this
project. On the contrary, I think it could be very valuable. But it is
something you need to go into with your eyes open, and epect to make
mistakes and run into problems along the way. Think of it as surgery:
you may have a general plan, but you're likely to run into unexpected
problems forcing you to adjust and adapt as you go.
Jawad Shafiq
i think i didn't clarify my goal in the previous mail. i do not want to delete/edit any files that are being populated during the patching and module installation process. i just want to fix all those 'wrong pointers' and "falls data" that come with the VistA database file that VA has placed on their FTP site. so clearly i do not want to do a virgin installation. (that's what VA calls it in their manuals :P )
1) Data Dictionary: Yes, i will delete/edit files and fields if i need to. For example, NEW PERSON file has a filed called 'Person' which points to PERSON file. PERSON file WAS the file which used to hold all users accounts data, i suppose. Then VA introduced NEW PERSON file and they wanted to keep a link b/w both PERSON and NEW PERSON file for transition stage, i guess. But PERSON file does not exists any more in VistA database file and 'person' pointer filed still exists in NEW PERSON file. So i deleted 'Person' field from NEW PERSON file.
There are some quite serious errors, like 2 Fields of a file have the same subscript location in the global. May be there is a logic behind it that i don't know of.
2) Data: Yes, i am aware of all those "Sharks" that i have to dodge or kill during this mission :). i am carrying out this project in a testing environment. Thankx for sharing your knowleedge with me and "boosting" up my moral :P..
Could any one please suggest on "what if i delete all data from DOMAIN, INSTITUTION, MEDIAL CENTER Division, Hospital Location and NEW PERSON FILE...but will keep POSTMASTER and CAMERON account."
Ragards
Jawad Shafiq
JohnLeo Zimmer, MD
>... so clearly i do not want to do a virgin
> installation. (that's what VA calls it in their manuals :P )
>
Not a "virgin" installation,
but a surgical reconstruction of that state.
Cameron
for example, that the copyrighted LOINC codes are not removed as their
copyright permits its redistribution as long as the original copyright
statement is intact with the distribution ... it's in the file's DD.)
The hashing algorithm for the Access and Verify codes is removed and
the corresponding codes in the New Person file are removed. Actually,
the only "running" part of the account from which the VistA FOIA is
created are the packages, VA FileMan, Kernel, Toolkit, MailMan and
recently, the Health Level Seven packages. No other package has been
"implemented". Only sufficient information is entered for any given
package to be able to complete the packages' installs. All the
"broken" values are in fact remnants of things not cleaned up in the
course of various packages' releases. I've always recognized the
presence of those remnants and have always encouraged the various teams
to clean things up as they go along. Unfortunately, over a decade ago
VA management directed that development of new versions of packages was
to cease (except in a few selected and high profile packages) and only
changes for "serious bug fixes" were to be allowed. The repair and
refactoring of code that previously was done every 6 to 18 months
ceased in the early to mid 90's.
Also, the Check/Fix DD option is run prior to the posting of the VistA
FOIA (or at least, most of the time I rememberd to do that!) Of
course, that doesn't remove things like an old DD element that used to
"point" to the old PERSON file 16. :) In fact, that was left behind
on purpose 7 years ago as a "just in case something went wrong" with
the conversion to file 200 and subsequent cleanup.
Since part of the installs for many packages requires data in tables
such as Institution, Medical Center Division, etc, deleting those
existing entries will definitely cause some packages behaviors to fail
when you try to "implement" them in sometimes mysterious ways. Rather
than deleting the entries, I suggest you simply re-name the values in
the .01 fields and edit any otherr fields according to the packages'
instructions for implementation. VistA-Office EHR is being prepared to
be setup in much the same way, i.e. with "place-holder" table entries
for sites to be able to edit and/or copy into additional entries.