Corrupt database
Ivan Metzlar
I have been using OpenVistA with FMQL and OVID for a couple of weeks
on a test run in a small medical clinic but now it seems I have a
corrupt database. Whenever I try to do something (Fileman, terminal
menu's, openvista-cis) I get the following error and the process exits
to the GT.M prompt:
Setting up programmer environment%GTM-E-REQRUNDOWN, Error accessing
database /opt/openvista/EHR/globals/mumps.dat. Must be rundown on
cluster node VistA.,%GTM-I-TEXT, Error with database control
shmctl,%SYSTEM-E-ENO22, Invalid argument
At M source location GETFILE+5^DIC0
I tried several mupip rundown and integ on the database, they all
succeed without any error but the above problem is not solved. The
strange thing too is I did not reboot the server (on an open database)
but the problem started showing up after I made some FileMan changes
to security keys and RPC options to enable patient photo attachments.
I have been trying to solve this for a couple of days now without any
success and I hit myself in the head for not creating regular backups
since it was just a test run. But I would really like to find out how
this came to happen. Has anyone any idea on how to fix this? Any help
or suggestions are greatly appreciated.
--Ivan
David Whitten
$gtm_dist/muipip rundown -r "*"
a simple rundown that doesn't include the region clears out the memory semaphors
but doesn't change the header of the mumps.dat file.
David
--Ivan
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com
Ivan Metzlar
this is the output I get when I run mupip rundown -r "*"
%GTM-I-MUFILRNDWNSUC, File /opt/openvista/EHR/globals/mumps.dat
successfully rundown
%GTM-I-MUFILRNDWNSUC, File /opt/openvista/EHR/globals/scratch.dat
successfully rundown
%GTM-E-FTOKERR, Error getting ftok of the file
/opt/openvista/EHR/globals/db.repl
%SYSTEM-E-ENO2, No such file or directory
%GTM-W-MUNOTALLSEC, WARNING: not all global sections accessed were
successfully rundown
It completed almost instantly which seems not OK. What about the
warning on the missing file?
I am still getting the same error:
ASTRON>D ^XUP
Setting up programmer environment%GTM-E-REQRUNDOWN, Error accessing
database /opt/openvista/EHR/globals/mumps.dat. Must be rundown on
cluster node VistA.,%GTM-I-TEXT, Error with database control
shmctl,%SYSTEM-E-ENO22, Invalid argument
At M source location GETFILE+5^DIC0
Any other ideas?
--Ivan
Ivan Metzlar
time I ran D ^XUP immediately after the rundown:
Setting up programmer environment%GTM-E-MUTEXERR, Mutual Exclusion
subsystem failure,%GTM-I-TEXT, Error with mutex socket
bind,%SYSTEM-E-ENO13, Permission denied
At M source location GETFILE+5^DIC0
This looks like a permission problem. Am I running the GT.M prompt as
the wrong user? I am using the openvistaEHR user for all VistA related
processes. Can I change the permissions of the database and module
files?
Cheers,
--Ivan
David Whitten
MUMPS> ZP GETFILE^DIC0:GETFILE+6
GETFILE(DIC,DIFILE,DIENS) ; Return file number, global references, IEN string and KEY fields data.
S DIFILE="" I $G(DIC)="" Q
I +$P(DIC,"E")'=DIC N DIDIC M DIDIC=DIC N DIC S DIDIC=$$CREF^DILF(DIDIC),DIDIC=$NA(@DIDIC),DIDIC=$$OREF^DILF(DIDIC) M DIC=DIDIC K DIDIC
N DA
I +$P(DIC,"E")=DIC D
. S DIFILE=DIC,DIC=$G(^DIC(DIC,0,"GL")) Q:DIC]""
. S DIC=DIFILE,DIFILE="" Q
Above in pink, I have highlighted the line for the error you mentioned.
This is a very simple code line that could only get a permission error if
the data file for the MUMPS system is radically wrong.
The rest of the error mentions that a Mutual Exclusion subsystem mutex socket
bind failed. Again this is a rather radically wrong error.
To debug this, I would look at things like the permissions for the gtmsecshr shared
object segment to make sure it is running as root, or look at the permissions on the
database file that the ^DIC global is stored in.
It is hard to diagnose long distance like this, but perhaps if you list these permissions
out that it might give us some information.
Ivan Metzlar
it seems no active mumps process can actually run because of the
error. If I run vistastart.sh (as openvistaEHR) the script fails with
the following errors:
%GTM-I-MUJNLSTAT, Initial processing started at Mon Apr 2 15:09:15 2012
%GTM-E-JNLFILEOPNERR, Error opening journal file
/opt/openvista/EHR/globals/mumps.mjl
%SYSTEM-E-ENO2, No such file or directory
%GTM-E-MUNOACTION, MUPIP unable to perform requested action
%GTM-I-MUJNLSTAT, End processing at Mon Apr 2 15:09:15 2012
%GTM-I-MUFILRNDWNSUC, File /opt/openvista/EHR/globals/mumps.dat
successfully rundown
Created file /opt/openvista/EHR/globals/mumps.dat
Created file /opt/openvista/EHR/globals/scratch.dat
%GTM-E-REQRUNDOWN, Error accessing database
/opt/openvista/EHR/globals/mumps.dat. Must be rundown on cluster node
VistA.
%GTM-I-TEXT, Error with database control shmctl
%SYSTEM-E-ENO22, Invalid argument
%GTM-E-MUNOFINISH, MUPIP unable to finish all requested actions
%GTM-E-REQRUNDOWN, Error accessing database
/opt/openvista/EHR/globals/mumps.dat. Must be rundown on cluster node
VistA.,%GTM-I-TEXT, Error with database control
shmctl,%SYSTEM-E-ENO22, Invalid argument
At M source location INIT^ZTMB
%GTM-E-INVCMD, Invalid command keyword encountered
y
^-----
%GTM-E-REQRUNDOWN, Error accessing database
/opt/openvista/EHR/globals/mumps.dat. Must be rundown on cluster node
VistA.,%GTM-I-TEXT, Error with database control
shmctl,%SYSTEM-E-ENO22, Invalid argument
At M source location GETFILE+5^DIC0
%GTM-E-REQRUNDOWN, Error accessing database
/opt/openvista/EHR/globals/mumps.dat. Must be rundown on cluster node
VistA.,%GTM-I-TEXT, Error with database control
shmctl,%SYSTEM-E-ENO22, Invalid argument
At M source location GETFILE+5^DIC0
%GTM-E-CMD, Command expected but not found
^
^-----
%GTM-E-CMD, Command expected but not found
/opt/openvista/EHR/bin/run.sh
^-----
How do I check the permissions? I'm pretty sure all routines are owned
by openvistaEHR and are executable by root and openvistaEHR.
Thanks for your help,
--Ivan
Ivan Metzlar
created a new empty database using mupip create. Surprisingly the same
error keeps showing up....
Should I reinstall the routines and KID files but keep the old
database? Is that even possible?
Any help is greatly appreciated!
--Ivan
Nancy Anthracite
--
Nancy Anthracite
Bhaskar, K.S
Let me offer an analogy. You have a patient with appendicitis on whom you have performed a liver transplant and whom you now plan to send for cardiac rehab. Mupip rundown is like giving said patient something like morphine or codeine.
You need to be thinking about setting up your VistA system with journaling and recovery. Please contact whoever set up your VistA system the first time and have them set it up with journaling and recovery. If you set it up yourself, please work through the GT.M Acculturation Workshop (search the archives of this list for discussion on the topic) before you attempt to set it up again.
I am on vacation for most of this week, and therefore unable to give you better advice than this message. But the symptoms of your patient and his treatment have been covered before on this list, and Google is your friend - as is the Acculturation Workshop and/or someone who knows how to set up a GT.M database for recovery with journaling.
Regards
-- Bhaskar
-- GT.M - Rock solid. Lightning fast. Secure. No compromises.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Chris
To unsubscribe, send email to Hardhats+unsubscribe@googlegroups.com-- http://groups.google.com/group/Hardhats To unsubscribe, send email to Hardhats+unsubscribe@googlegroups.com-- http://groups.google.com/group/Hardhats To unsubscribe, send email to Hardhats+unsubscribe@googlegroups.com-- http://groups.google.com/group/Hardhats To unsubscribe, send email to Hardhats+unsubscribe@googlegroups.com
Ivan Metzlar
thank you for the nice analogy. Apparently I had journaling set up and
I did restore a backup but this did not solve the problem. Also
creating a new empty database did change nothing. I got a very
interesting e-mail from Todd Garland:
"""
Been reading about your issues and proposed solutions. I had same
crash you had, just over a week ago using openvista.
It appears from your posts that you have installed openvista via
astronautvista installer—I recognize the astron>….anyway, if you have
been using it for a couple of weeks the system is set up to
automatically journal and backup from the get-go.
In my case, I just did a restore backup, instead of going through a ton of time.
So, having said that if you go into your file system you should find
the backups in /usr/share/vista_parts/backup and in
/usr/share/vista_parts/bin you will find restore.sh which is used to
restore a backup. In addition you should see the nightlybak.sh in same
directory and this (if you left your machine on 24 hours a day) should
have created a backup at 5:00am every day. It will also show where the
backup was placed—which essentially is the same directory and in
folder /backup.
In addition, you will find your journaling backups in
/opt/(whatevervistaversion)/(EHR)/journals/….this directory should
contain the journal backups.
"""
But unfortunately this did not fix my problem. This worries me a
little. What I really want to know is how did this happen and how can
it be prevented from happening again. As I said I had journaling
turned on but even a recovery (restore) did not resolve anything. Or
am I doing something wrong?
As an experienced software engineer, hacker and *nix user I am getting
very nervous about this problem and how it seems unsolvable especially
now that I found out I am not the only one. If we want to integrate
VistA into more clinics and hospitals this kind of
'unrecoverable-loss-of-data-issue' is a definite deal breaker.
I am still hoping somebody knows a solution and/or preferably a way to
prevent this problem from happening again while I start over with
setting up VistA, installing FMQL, OVID, adding users, adding patients
etc.
Cheers,
--Ivan
Bhaskar, K.S
Regards
-- Bhaskar
Jignesh Patel
Bhaskar, K.S
Using rundown on a crashed database is like giving morphine to the patient with appendicitis - it can make the pain go away, but does not cure anything.
Mupip rundown only cleans up structures in the file header. It does not repair database damage - which is definitely possible in the event of a crash. The correct solution is to use journaling and use journal files to recover databases.
To everyone who is using rundown on crashed databases: please work through and understand the exercises in the Acculturation Workshop. It represents many weeks of personal work, specifically with the intention of educating people setting up and running GT.M. I find it quite disheartening to see people make the same mistake over and over and over again - in spite of my posts telling people what needs to be done.
PLEASE USE JOURNALING ON ANY DATABASE THAT YOU CARE ABOUT. THE ONLY DATABASES WITHOUT JOURNALING SHOULD BE THOSE WITH DATA YOU DON'T CARE ABOUT - DATABASE FILES THAT YOU ARE PREPARED TO DELETE AND REPLACE WITH FRESHLY CREATED NEW EMPTY DATABASE FILES AFTER A SYSTEM CRASH.
Thanks for listening.
Regards
-- Bhaskar