Debugging ^XTER and taskman issues.

34 views
Skip to first unread message

Kevin Toppenberg

unread,
Jun 25, 2026, 2:40:06 PM (3 days ago) Jun 25
to Hardhats
2nd attempt at sending this message. The first time I included a name, but no actual data.   So I asked for it to be deleted. 

Below is an ^XTER output.  This was an error from a process that put about 100+ alerts into my CPRS when I was seeing patients. 

The ultimate problem was that where the code was expecting a DUZ, it was finding "217^Doctor,Unspecified^- Physician".  Eddie mitigated the problem by wrapping the value in +$GET().  But I would like to get to the root of the problem.

We have recently implemented a new lab interface and I think there is some place that we are storing wrong info.  

Questions:
1) Can anyone tell what process was triggering this error over and over?  I see ABNORMAL LAB RESULTS (ACTION) which makes me think there is some process that scans labs after they have been entered.  

2) What techniques do others use when tracking down a problem like this?  I know how to read the stack and trace to the point of error.  But this process depends on inputs that are passed to it in the taskman variables handoff.  So I need to be able to track further upstream. 

Thanks
Kevin


  2)  <(NULSUBSC)>MAKEVP+4^XPAR1      08:24:33  EHR:ubuntu64   688332  0
  1)  <(NULSUBSC)>MAKEVP+4^XPAR1      08:24:26  EHR:ubuntu64   687927  0

No screened error

     Enter '^' to quit listing, <RETURN> to continue...: 1
Process ID:  687927  (687927)           JUN 22, 2026 08:24:26
UCI/VOL: [EHR:ubuntu64]
$ZA:   0
Current $IO: 0                          Current $ZIO: ^0^0^0
$ZE= MAKEVP+4^XPAR1, Access or update failed because Null subscripts are not allowed for database file: /opt/worldvista/
EHR/g/mumps.dat,150373498,-%YDB-E-NULSUBSC
Last Global Ref: ^%ZOSF("OS")
 Q ";"_$P($G(^DIC(FN,0,"GL")),U,2)

Which symbol? > ^L
$DEVICE=
$ECODE=,Z150373498,
$ESTACK=17
$ETRAP=D ERROR2^%ZTMS0 HALT
$QUIT=0
$STACK=17
$STACK(000)=-direct
$STACK(000,"ECODE")=
$STACK(000,"PLACE")=SUBMGR+3^%ZTMS1
$STACK(000,"MCODE")= D PROCESS^%ZTMS2 G:$D(ZTQUIT) QUIT^%ZTMS
$STACK(001)=DO
$STACK(001,"ECODE")=
$STACK(001,"PLACE")=PROCESS+9^%ZTMS2
$STACK(001,"MCODE")= D TASK^%ZTMS3 I ZTYPE="C"!$D(ZTNONEXT) Q
$STACK(002)=DO
$STACK(002,"ECODE")=
$STACK(002,"PLACE")=4+10^%ZTMS3
$STACK(002,"MCODE")= D RUN
$STACK(003)=DO
$STACK(003,"ECODE")=
$STACK(003,"PLACE")=RUN+4^%ZTMS3
$STACK(003,"MCODE")= D @ZTRTN
$STACK(004)=DO
$STACK(004,"ECODE")=
$STACK(004,"PLACE")=ZTSK+1^ORB3
$STACK(004,"MCODE")= D START
$STACK(005)=DO
$STACK(005,"ECODE")=
$STACK(005,"PLACE")=DOALERT+28^ORB3
$STACK(005,"MCODE")= I $D(ORBADUZ)>1 D PKGDUZS ;pkg-supplied recips
$STACK(006)=DO
$STACK(006,"ECODE")=
$STACK(006,"PLACE")=PKGDUZS+7^ORB3
$STACK(006,"MCODE")= F  S ORBPDUZ=$O(ORBADUZ(ORBPDUZ)) Q:ORBPDUZ=""  S ORBDUZ=ORBPDUZ D USER
$STACK(007)=DO
$STACK(007,"ECODE")=
$STACK(007,"PLACE")=USER+1^ORB3
$STACK(007,"MCODE")= D USER^ORB3USER(.XQA,ORBDUZ,ORN,.ORBU,.ORBUI,ORBDFN,+$G(ORNUM))
$STACK(008)=DO
$STACK(008,"ECODE")=
$STACK(008,"PLACE")=USER+14^ORB3USER
$STACK(008,"MCODE")= S ORBTM=$P(ORBDUZ,U,2) I $L(ORBTM) D  ;if user recip via team
$STACK(009)=DO
$STACK(009,"ECODE")=
$STACK(009,"PLACE")=USER+15^ORB3USER
$STACK(009,"MCODE")= .S ORBTMF=$$GET^XPAR(ORBTM_";OR(100.21,","ORB PROCESSING FLAG",ORN,"I")
$STACK(010)=$$
$STACK(010,"ECODE")=
$STACK(010,"PLACE")=GET+9^XPAR
$STACK(010,"MCODE")= D INTERN^XPAR1 I ERR Q ""
$STACK(011)=DO
$STACK(011,"ECODE")=
$STACK(011,"PLACE")=INTERN+18^XPAR1
$STACK(011,"MCODE")= I '(+ENT&(ENT[";")) D ENTEXT(.ENT) D:ENT=""  G C1 ;resolve external vptr fmt
$STACK(012)=DO
$STACK(012,"ECODE")=
$STACK(012,"PLACE")=ENTEXT+8^XPAR1
$STACK(012,"MCODE")= S ENT=$$FIND1^DIC(FN,"","X",X)_$$MAKEVP(FN)
$STACK(013)=$$
$STACK(013,"ECODE")=,Z150373498,
$STACK(013,"PLACE")=MAKEVP+4^XPAR1
$STACK(013,"MCODE")= Q ";"_$P($G(^DIC(FN,0,"GL")),U,2)
$STACK(014)=DO
$STACK(014,"ECODE")=
$STACK(014,"PLACE")=ERR2+7^%ZTMS
$STACK(014,"MCODE")= I '$$SCREEN^%ZTER(%ZTME) D
$STACK(015)=DO
$STACK(015,"ECODE")=
$STACK(015,"PLACE")=ERR2+8^%ZTMS
$STACK(015,"MCODE")= . D ^%ZTER ;Kernel error file
$X $Y=0 0
$ZV=GT.M V7.0-005 Linux x86_64
%=12
%ZISRL=5
%ZT("^XUTL(""XQ"",$J)")=
%ZTIME=5853025466
%ZTME=MAKEVP+4^XPAR1, Access or update failed because Null subscripts are not allowed for database file: /opt/worldvista/EHR/g/mumps.dat,150373498,-%YDB-E-NULSUBSC
%ZTMEH=67743,30266
%ZTMETSK=2514236
%ZTPFLG("BalLimit")=100
%ZTPFLG("HOME")=EHR:ubuntu64
%ZTPFLG("LOCKTM")=3
%ZTPFLG("MIN")=2
%ZTPFLG("RT")=20
%ZTPFLG("USER")=211
%ZTPFLG("XUSCNT")=54
%ZTPFLG("ZTPN")=2
%ZTPFLG("ZTREQ")=0
DFN=33050
DIERR=1^1
DILOCKTM=3
DISYS=19
DT=3260622
DTIME=1
DUZ=168
DUZ(0)=@
DUZ(1)=
DUZ(2)=69
DUZ("AG")=O
DUZ("AUTHENTICATION")=AVCODES
DUZ("AUTO")=1
DUZ("BUF")=1
DUZ("LANG")=1
DUZ("LOA")=2
ENT=
ERR=0
FMT=I
FN=
INST=14
IO=ORB NOTIFICATION RESOURCE
IO(0)=ORB NOTIFICATION RESOURCE
IO(1,"ORB NOTIFICATION RESOURCE")=RES
IOBS=$C(8)
IOF=#
IOHG=
IOM=132
ION=ORB NOTIFICATION RESOURCE
IOPAR=
IOS=25
IOSL=64
IOST=P-OTHER
IOST(0)=16
IOT=RES
IOUPAR=
IOXY=
ORBADT=0
ORBADUZ("217^Doctor,Unspecified^- Physician")=
ORBATTD=0
ORBDFN=33050
ORBDUZ=217^Doctor,Unspecified^- Physician
ORBENT=DIV^SYS^PKG
ORBID=;3260622;0;CH;6739377
ORBIDX=
ORBLOCK=0
ORBN=ABNORMAL LAB RESULTS (ACTION)^OR^Abnormal lab results.^PKG^R^RPTLAB^ORB3FUP2^
ORBNOW=3260622.082426
ORBPDATA=0@OR|;3260622;0;CH;6739377@LRCH
ORBPDUZ=217^Doctor,Unspecified^- Physician
ORBPMSG=Abnormal lab results: - [Jun 22, 2026]
ORBPRIM=0
ORBTM=Doctor,Unspecified
ORBUI=1
ORDGPMA=
ORN=14
ORNUM=0
PAR=48
POP=0
PRE=Doctor,Unspecified;OR(100
U=^
VA("BID")=6038
VA("PID")=409-02-6038
VA200=
VADM(1)=LNAME,FNAME
VADM(2)=XXXXXXXXX^XXX-XX-XXXX
VADM(3)=2110101^JAN 1,1911
VADM(4)=68
VADM(5)=M^MALE
VADM(6)=
VADM(7)=
VADM(8)=
VADM(9)=
VADM(10)=2^MARRIED
VADM(11)=1
VADM(11,1)=2^NOT HISPANIC OR LATINO
VADM(11,1,1)=3^OBSERVER
VADM(12)=1
VADM(12,1)=13^WHITE
VADM(12,1,1)=3^OBSERVER
VADM(13)=
VADM(13,1)=
VAERR=0
VAIN(1)=
VAIN(2)=
VAIN(3)=
VAIN(4)=
VAIN(5)=
VAIN(6)=
VAIN(7)=
VAIN(8)=
VAIN(9)=
VAIN(10)=
VAIN(11)=
X=21,
XPARGET=
XPARSYS=5;DIC(4.2,
XQY=8552
XQY0=OR CPRS GUI CHART
ZT4=8
ZTDESC= Send Alert Notification (14) ABNORMAL LAB RESULTS (ACTION)  for (33050) LNAME,FNAME
ZTDTH=67743,30266
ZTIO=ORB NOTIFICATION RESOURCE
ZTLKTM=3
ZTPFLG("BalLimit")=100
ZTPFLG("HOME")=EHR:ubuntu64
ZTPFLG("LOCKTM")=3
ZTPFLG("MIN")=2
ZTPFLG("RT")=20
ZTPFLG("USER")=211
ZTPFLG("XUSCNT")=54
ZTPFLG("ZTPN")=2
ZTPFLG("ZTREQ")=0
ZTQUEUED=2514236
ZTREC02=1^EHR^^
ZTRTN=ZTSK^ORB3
ZTSK=2514236
ZTSTAT=1 General error
ZTSYNCFL=
ZTYPE=G
^XUTL("XQ",$J,0)=3260622.082426
^XUTL("XQ",$J,"DUZ")=168
^XUTL("XQ",$J,"DUZ(0)")=@
^XUTL("XQ",$J,"DUZ(1)")=
^XUTL("XQ",$J,"DUZ(2)")=69
^XUTL("XQ",$J,"DUZ(AG)")=O
^XUTL("XQ",$J,"DUZ(AUTHENTICATION)")=AVCODES
^XUTL("XQ",$J,"DUZ(AUTO)")=1
^XUTL("XQ",$J,"DUZ(BUF)")=1
^XUTL("XQ",$J,"DUZ(LANG)")=1
^XUTL("XQ",$J,"DUZ(LOA)")=2
^XUTL("XQ",$J,"IO")=ORB NOTIFICATION RESOURCE
^XUTL("XQ",$J,"IOBS")=$C(8)
^XUTL("XQ",$J,"IOF")=#
^XUTL("XQ",$J,"IOM")=132
^XUTL("XQ",$J,"ION")=ORB NOTIFICATION RESOURCE
^XUTL("XQ",$J,"IOPAR")=
^XUTL("XQ",$J,"IOS")=25
^XUTL("XQ",$J,"IOSL")=64
^XUTL("XQ",$J,"IOST")=P-OTHER
^XUTL("XQ",$J,"IOST(0)")=16
^XUTL("XQ",$J,"IOT")=RES
^XUTL("XQ",$J,"IOUPAR")=
^XUTL("XQ",$J,"IOXY")=
^XUTL("XQ",$J,"ZTSK")= Send Alert Notification (14) ABNORMAL LAB RESULTS (ACTION)  for (33050) LNAME,FNAME
^XUTL("XQ",$J,"ZTSKNUM")=2514236
$ZE= MAKEVP+4^XPAR1, Access or update failed because Null subscripts are not allowed for database file: /opt/worldvista/EHR/g/mumps.dat,150373498,-%YDB-E-NULSUBSC
Last Global Ref: ^%ZOSF("OS")
 Q ";"_$P($G(^DIC(FN,0,"GL")),U,2)

Sam Habiel

unread,
Jun 25, 2026, 3:32:13 PM (2 days ago) Jun 25
to hard...@googlegroups.com
> Can anyone tell what process was triggering this error over and over?

In days of yore, I would have looked in the code myself to see who sets the specific ZTDESC. But I asked Claude. It initially had no clue, but I told it to look hard at LEDI, because that's where I think it's coming from. I think it found something

---
  The native verify → alert chain (this is "the process scanning labs")

  In stock VistA there is no code in the HL7/LEDI result handlers (LA7V*, LR7O*) that calls ORB3 directly. The abnormal-lab notification (#14) has exactly one producer: OR^LR7ORB3 →
  EN^ORB3(LRTYPE,…). And OR^LR7ORB3 is reached through only one chain — result verification:

  result lands in #63
    → verify pass:  LRVER3 (CH)  /  LRMIEDZ2 (MI)
        → ASKXQA^LRVER3            (LRVER3:255)
            S LRDEFAULT=$$GET^XPAR("USR^DIV^PKG","LR CH VERIFY CPRS ALERT",1,"Q")   (LRVER3:262)
            I LRDEFAULT>0 D ASKXQA^LR7ORB3(LRDFN,"CH",LRIDT,LRUID,LRDEFAULT)         (LRVER3:263)
                → SETUP^LR7ORB3
                    → GETDOCS  (builds recipient list = ordering provider + PCP/attending)  (LR7ORB3:59)
                    → OR^LR7ORB3 → EN^ORB3(14,DFN,LROIFN,.LRXQA,LRMSG,LRIENS)        (LR7ORB3:228)

  So the "process that scans labs after they're entered" is really verification, and the abnormal alert is a side-effect of verifying. The on/off switch is the parameter LR CH VERIFY CPRS ALERT
  (and LR MI VERIFY CPRS ALERT for micro). That's the knob.

  Where LEDI fits: LEDI files reference-lab results back into #63 and drives them through this same verify pass (often auto-verified, no human). So yes — natively, LEDI causes abnormal-lab
  alerts via verification, gated by that parameter. The recipient list comes from GETDOCS, which reads the ordering provider off the CH result ($P(^LR(LRDFN,"CH",LRIDT,0),"^",10), LR7ORB3:66)
  and formats each recipient as IEN^NAME^role (LR7ORB3:67).

  But your specific alert did not come from this baseline code

  Three fingerprints in your dump don't match what this FOIA-baseline chain emits:

  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────┐
  │                                               your dump                                                │                        what baseline OR^LR7ORB3 produces                        │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ ORBADUZ("217^Doctor,Unspecified^- Physician") (whole string is the subscript)                          │ SETUP does S LRXQA(+LRDOCS(LRC))=… (LR7ORB3:35) — the + forces a bare DUZ       │
  │                                                                                                        │ subscript; can't produce this                                                   │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ role piece = - Physician                                                                               │ GETDOCS emits role Ordering Provider (LR7ORB3:67)                               │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
  │ ORBPMSG="Abnormal lab results: - [Jun 22, 2026]" (date in brackets);                                   │ OR puts the test name in brackets (…" - ["_$P(LRTST,"^",2)_"]", LR7ORB3:218)    │
  │ ORBPDATA="0@OR|;3260622;0;CH;6739377@LRCH" (order ifn 0, empty order piece)                            │ and fills the order pieces (LR7ORB3:208)                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────┘

  That + at LR7ORB3:35 is precisely the guard that prevents the <NULSUBSC> crash we traced — and your alert bypassed it. Conclusion: the version of this chain running on your box is newer than,
  or locally modified from, this 2013 baseline, and its GETDOCS/SETUP/OR equivalent (or a LEDI auto-verify wrapper) builds the recipient array using the whole IEN^NAME^role string as the
  ORBADUZ subscript instead of +DUZ. That's the actual defect, and it lives in code that isn't in this FOIA tree. The repeated firing is because LEDI-received results keep resolving the
  ordering provider to the placeholder "Doctor,Unspecified" (#217) — one unmapped provider, one alert+crash per result.

  Confirm and trace it on the live system

  Since the offending routine isn't in this baseline, do this on your running EHR:

  1. Find the divergent caller. On the live system grep your routine set for who builds the recipient array and calls into the alert:
  grep 'OR^LR7ORB3\|SETUP^LR7ORB3\|LRXQA(' your routines
  1. Compare your live LR7ORB3/LRVER3 to this baseline (D PRINT^XINDEX or just diff). The caller that sets LRXQA(<whole string>) / role - Physician is your bug.
  2. Kill switch / confirm the trigger: check the parameter that gates it —
  W $$GET^XPAR("USR^DIV^PKG","LR CH VERIFY CPRS ALERT",1,"Q"). Setting it to 0 stops the flood immediately while you fix root cause (confirms the verify-time path is the source).
  3. Inspect the source record from the dump's fingerprints: S X=$$CHECKUID^LRWU4(6739377) → ^LRO(68,…) accession → the #63 result; look at the CH ordering-provider field GETDOCS reads (piece
  10 of ^LR(LRDFN,"CH",LRIDT,0)). You'll find 217 / "Doctor,Unspecified".
  4. Inbound HL7: pull the ORU for these accessions from the message logs (#776/#777, and #772/#773) and look at OBR-16 (ordering provider). The reference lab is sending a provider your filer
  can't map, so it substitutes Doctor,Unspecified — that mapping step is the upstream "storing wrong info" you suspected.

  So: native LEDI → verify → LR CH VERIFY CPRS ALERT → OR^LR7ORB3 is genuinely the engine. Your crash is a modified version of GETDOCS/SETUP in that chain dropping the + on the recipient
  subscript, combined with provider mapping that defaults to #217. Want me to write up the exact LRXQA contract (subscript = +DUZ or "G.mailgroup", value = display role) so whoever owns your
  LEDI build can patch their GETDOCS equivalent?
---

2. What techniques do others use when tracking down a problem like this?

Do you know about ^R in the error trap?

VEHU>D ^XTER


In response to the DATE prompt you can enter:
     'S' to specify text to be matched in error or routine name


No error logged on 6/25/2026

Which date? > T-1
1 error logged on 6/24/2026
  1)  LR588 : Application Error! please disregard NO need to report13:40:24  RO


No screened error

     Enter '^' to quit listing, <RETURN> to continue...: 1
Process ID:  184  (184)                 JUN 24, 2026 13:40:24
UCI/VOL: [ROU:vehu]                    
$ZA:   0                                $ZB: \013
Current $IO: /dev/pts/0                 Current $ZIO: /dev/pts/0^36^34^/dev/pts/
0
$ZE= LR588 : Application Error! please disregard NO need to report
Last Global Ref: ^LAM(0)
 S @%ZTERRT@("LINE")=$STACK(%2,"MCODE")

Which symbol? > ?

Enter:
     ^Q to EXIT
     '^' to return to the last question
     Leading character(s) of symbol(s) you wish to examine
     ^L to obtain a list of all symbols
     ^P to select a printer and print this error
     ^M to capture the current error in a mail message
     ^I to obtain information on key package variables
     ^R to restore the symbol table and ... and enter direct mode
     $ to get a display of the $ system variables
$ZE= LR588 : Application Error! please disregard NO need to report
Last Global Ref: ^LAM(0)
 S @%ZTERRT@("LINE")=$STACK(%2,"MCODE")


Which symbol? > ^R
VARIABLES RESTORED


Once you have the symbol table, you can run what's in ZTRTN in direct mode.

I have a more advanced technique where you stop Taskman and run the submanager in the foreground, and put break points there. Rick Marshall (the author of Taskman) taught me how to do it. It's on VistApedia, but VistApedia is down right now.

--Sam

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hardhats/4570e024-cd30-4093-8554-f37230419515n%40googlegroups.com.

Kevin Toppenberg

unread,
Jun 25, 2026, 6:03:48 PM (2 days ago) Jun 25
to Hardhats
Sam, 

This is at once amazing and also terrifying.  

I didn't realize that AI could do such a deep dive into VistA. 

Thanks
Kevin

Sam Habiel

unread,
Jun 25, 2026, 6:07:34 PM (2 days ago) Jun 25
to hard...@googlegroups.com
>This is at once amazing and also terrifying.  
I agree!!!

>I didn't realize that AI could do such a deep dive into VistA. 
I found it figures out things pretty well if you give it real source code to work with, and not ask it to answer a question from information it "memorized" from the Internet. In analyzing this error trap, I had a copy of VistA-M repo which had the latest source code from FOIA. So it could read your error trap and and read the source code and compare.

I am really curious to see if it actually got the answer right. Let us know.

--Sam


Reply all
Reply to author
Forward
0 new messages