Kevin Toppenberg <kdt...@gmail.com>: Jun 28 01:45PM -0700
On Thursday, June 25, 2026 at 6:07:34 PM UTC-4 Sam Habiel wrote:
>This is at once amazing and also terrifying.
I agree!!!
>I didn't realize that AI could do such a deep dive into VistA.
I found it figures out things pretty well if you give it real source code
to work with, and not ask it to answer a question from information it
"memorized" from the Internet. In analyzing this error trap, I had a copy
of VistA-M repo which had the latest source code from FOIA. So it could
read your error trap and and read the source code and compare.
I am really curious to see if it actually got the answer right. Let us know.
To be honest, I don't know if it is correct or not. We don't use LEDI or
standard HL7 processing. I couldn't get it to work reliably. I have my
own HL7 handler and filer. So the only thing that could possibly trigger
the code it is describing would be input transforms. But honestly, since
we came up with a mitigation and the error is not causing problems any
more, I haven't waded into that area.
What I was hoping was that there was some part of taskman that stored
information about the process that created the task. But it seems that is
not the case.
Thanks
Kevin
--Sam
On Thu, Jun 25, 2026 at 6:03 PM Kevin Toppenberg <kdt...@gmail.com> wrote:
Sam,
This is at once amazing and also terrifying.
I didn't realize that AI could do such a deep dive into VistA.
Thanks
Kevin
On Thursday, June 25, 2026 at 3:32:13 PM UTC-4 Sam Habiel wrote:
> Can anyone tell what process was triggering this error over and over?
In days of yore, I would have looked in the code myself to see who sets the
specific ZTDESC. But I asked Claude. It initially had no clue, but I told
it to look hard at LEDI, because that's where I think it's coming from. I
think it found something
---
The native verify → alert chain (this is "the process scanning labs")
In stock VistA there is no code in the HL7/LEDI result handlers (LA7V*,
LR7O*) that calls ORB3 directly. The abnormal-lab notification (#14) has
exactly one producer: OR^LR7ORB3 →
EN^ORB3(LRTYPE,…). And OR^LR7ORB3 is reached through only one chain —
result verification:
result lands in #63
→ verify pass: LRVER3 (CH) / LRMIEDZ2 (MI)
→ ASKXQA^LRVER3 (LRVER3:255)
S LRDEFAULT=$$GET^XPAR("USR^DIV^PKG","LR CH VERIFY CPRS
ALERT",1,"Q") (LRVER3:262)
I LRDEFAULT>0 D
ASKXQA^LR7ORB3(LRDFN,"CH",LRIDT,LRUID,LRDEFAULT) (LRVER3:263)
→ SETUP^LR7ORB3
→ GETDOCS (builds recipient list = ordering provider +
PCP/attending) (LR7ORB3:59)
→ OR^LR7ORB3 →
EN^ORB3(14,DFN,LROIFN,.LRXQA,LRMSG,LRIENS) (LR7ORB3:228)
So the "process that scans labs after they're entered" is really
verification, and the abnormal alert is a side-effect of verifying. The
on/off switch is the parameter LR CH VERIFY CPRS ALERT
(and LR MI VERIFY CPRS ALERT for micro). That's the knob.
Where LEDI fits: LEDI files reference-lab results back into #63 and
drives them through this same verify pass (often auto-verified, no human).
So yes — natively, LEDI causes abnormal-lab
alerts via verification, gated by that parameter. The recipient list
comes from GETDOCS, which reads the ordering provider off the CH result
($P(^LR(LRDFN,"CH",LRIDT,0),"^",10), LR7ORB3:66)
and formats each recipient as IEN^NAME^role (LR7ORB3:67).
But your specific alert did not come from this baseline code
Three fingerprints in your dump don't match what this FOIA-baseline chain
emits:
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────┐
│ your dump
│ what baseline
OR^LR7ORB3 produces │
├────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ ORBADUZ("217^Doctor,Unspecified^- Physician") (whole string is the
subscript) │ SETUP does S LRXQA(+LRDOCS(LRC))=…
(LR7ORB3:35) — the + forces a bare DUZ │
│
│ subscript; can't produce this
│
├────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ role piece = - Physician
│ GETDOCS emits role Ordering Provider
(LR7ORB3:67) │
├────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ ORBPMSG="Abnormal lab results: - [Jun 22, 2026]" (date in brackets);
│ OR puts the test name in brackets (…" -
["_$P(LRTST,"^",2)_"]", LR7ORB3:218) │
│ ORBPDATA="0@OR|;3260622;0;CH;6739377@LRCH" (order ifn 0, empty order
piece) │ and fills the order pieces
(LR7ORB3:208) │
└────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────┘
That + at LR7ORB3:35 is precisely the guard that prevents the <NULSUBSC>
crash we traced — and your alert bypassed it. Conclusion: the version of
this chain running on your box is newer than,
or locally modified from, this 2013 baseline, and its GETDOCS/SETUP/OR
equivalent (or a LEDI auto-verify wrapper) builds the recipient array using
the whole IEN^NAME^role string as the
ORBADUZ subscript instead of +DUZ. That's the actual defect, and it lives
in code that isn't in this FOIA tree. The repeated firing is because
LEDI-received results keep resolving the
ordering provider to the placeholder "Doctor,Unspecified" (#217) — one
unmapped provider, one alert+crash per result.
Confirm and trace it on the live system
Since the offending routine isn't in this baseline, do this on your
running EHR:
1. Find the divergent caller. On the live system grep your routine set
for who builds the recipient array and calls into the alert:
grep 'OR^LR7ORB3\|SETUP^LR7ORB3\|LRXQA(' your routines
1. Compare your live LR7ORB3/LRVER3 to this baseline (D PRINT^XINDEX or
just diff). The caller that sets LRXQA(<whole string>) / role - Physician
is your bug.
2. Kill switch / confirm the trigger: check the parameter that gates it —
W $$GET^XPAR("USR^DIV^PKG","LR CH VERIFY CPRS ALERT",1,"Q"). Setting it
to 0 stops the flood immediately while you fix root cause (confirms the
verify-time path is the source).
3. Inspect the source record from the dump's fingerprints: S
X=$$CHECKUID^LRWU4(6739377) → ^LRO(68,…) accession → the #63 result; look
at the CH ordering-provider field GETDOCS reads (piece
10 of ^LR(LRDFN,"CH",LRIDT,0)). You'll find 217 / "Doctor,Unspecified".
4. Inbound HL7: pull the ORU for these accessions from the message logs
(#776/#777, and #772/#773) and look at OBR-16 (ordering provider). The
reference lab is sending a provider your filer
can't map, so it substitutes Doctor,Unspecified — that mapping step is
the upstream "storing wrong info" you suspected.
So: native LEDI → verify → LR CH VERIFY CPRS ALERT → OR^LR7ORB3 is
genuinely the engine. Your crash is a modified version of GETDOCS/SETUP in
that chain dropping the + on the recipient
subscript, combined with provider mapping that defaults to #217. Want me
to write up the exact LRXQA contract (subscript = +DUZ or "G.mailgroup",
value = display role) so whoever owns your
LEDI build can patch their GETDOCS equivalent?
---
2. What techniques do others use when tracking down a problem like this?
Do you know about ^R in the error trap?
VEHU>D ^XTER
In response to the DATE prompt you can enter:
'S' to specify text to be matched in error or routine name
No error logged on 6/25/2026
Which date? > T-1
1 error logged on 6/24/2026
1) LR588 : Application Error! please disregard NO need to report13:40:24
RO
No screened error
Enter '^' to quit listing, <RETURN> to continue...: 1
Process ID: 184 (184) JUN 24, 2026 13:40:24
UCI/VOL: [ROU:vehu]
$ZA: 0 $ZB: \013
Current $IO: /dev/pts/0 Current $ZIO:
/dev/pts/0^36^34^/dev/pts/
0
$ZE= LR588 : Application Error! please disregard NO need to report
Last Global Ref: ^LAM(0)
S @%ZTERRT@("LINE")=$STACK(%2,"MCODE")
Which symbol? > ?
Enter:
^Q to EXIT
'^' to return to the last question
Leading character(s) of symbol(s) you wish to examine
^L to obtain a list of all symbols
^P to select a printer and print this error
^M to capture the current error in a mail message
^I to obtain information on key package variables
^R to restore the symbol table and ... and enter direct mode
$ to get a display of the $ system variables
$ZE= LR588 : Application Error! please disregard NO need to report
Last Global Ref: ^LAM(0)
S @%ZTERRT@("LINE")=$STACK(%2,"MCODE")
Which symbol? > ^R
VARIABLES RESTORED
Once you have the symbol table, you can run what's in ZTRTN in direct mode.
I have a more advanced technique where you stop Taskman and run the
submanager in the foreground, and put break points there. Rick Marshall
(the author of Taskman) taught me how to do it. It's on VistApedia, but
VistApedia is down right now.
--Sam
On Thu, Jun 25, 2026 at 2:40 PM Kevin Toppenberg <kdt...@gmail.com> wrote:
2nd attempt at sending this message. The first time I included a name, but
no actual data. So I asked for it to be deleted.
Below is an ^XTER output. This was an error from a process that put about
100+ alerts into my CPRS when I was seeing patients.
The ultimate problem was that where the code was expecting a DUZ, it was
finding "217^Doctor,Unspecified^- Physician". Eddie mitigated the problem
by wrapping the value in +$GET(). But I would like to get to the root of
the problem.
We have recently implemented a new lab interface and I think there is some
place that we are storing wrong info.
Questions:
1) Can anyone tell what process was triggering this error over and over? I
see ABNORMAL LAB RESULTS (ACTION) which makes me think there is some
process that scans labs after they have been entered.
2) What techniques do others use when tracking down a problem like this? I
know how to read the stack and trace to the point of error. But this
process depends on inputs that are passed to it in the taskman variables
handoff. So I need to be able to track further upstream.
Thanks
Kevin
2) <(NULSUBSC)>MAKEVP+4^XPAR1 08:24:33 EHR:ubuntu64 688332 0
1) <(NULSUBSC)>MAKEVP+4^XPAR1 08:24:26 EHR:ubuntu64 687927 0
No screened error
Enter '^' to quit listing, <RETURN> to continue...: 1
Process ID: 687927 (687927) JUN 22, 2026 08:24:26
UCI/VOL: [EHR:ubuntu64]
$ZA: 0
Current $IO: 0 Current $ZIO: ^0^0^0
$ZE= MAKEVP+4^XPAR1, Access or update failed because Null subscripts are
not allowed for database file: /opt/worldvista/
EHR/g/mumps.dat,150373498,-%YDB-E-NULSUBSC
Last Global Ref: ^%ZOSF("OS")
Q ";"_$P($G(^DIC(FN,0,"GL")),U,2)
Which symbol? > ^L
$DEVICE=
$ECODE=,Z150373498,
$ESTACK=17
$ETRAP=D ERROR2^%ZTMS0 HALT
$QUIT=0
$STACK=17
$STACK(000)=-direct
$STACK(000,"ECODE")=
$STACK(000,"PLACE")=SUBMGR+3^%ZTMS1
$STACK(000,"MCODE")= D PROCESS^%ZTMS2 G:$D(ZTQUIT) QUIT^%ZTMS
$STACK(001)=DO
$STACK(001,"ECODE")=
$STACK(001,"PLACE")=PROCESS+9^%ZTMS2
$STACK(001,"MCODE")= D TASK^%ZTMS3 I ZTYPE="C"!$D(ZTNONEXT) Q
$STACK(002)=DO
$STACK(002,"ECODE")=
$STACK(002,"PLACE")=4+10^%ZTMS3
$STACK(002,"MCODE")= D RUN
$STACK(003)=DO
$STACK(003,"ECODE")=
$STACK(003,"PLACE")=RUN+4^%ZTMS3
$STACK(003,"MCODE")= D @ZTRTN
$STACK(004)=DO
$STACK(004,"ECODE")=
$STACK(004,"PLACE")=ZTSK+1^ORB3
$STACK(004,"MCODE")= D START
$STACK(005)=DO
$STACK(005,"ECODE")=
$STACK(005,"PLACE")=DOALERT+28^ORB3
$STACK(005,"MCODE")= I $D(ORBADUZ)>1 D PKGDUZS ;pkg-supplied recips
$STACK(006)=DO
$STACK(006,"ECODE")=
$STACK(006,"PLACE")=PKGDUZS+7^ORB3
$STACK(006,"MCODE")= F S ORBPDUZ=$O(ORBADUZ(ORBPDUZ)) Q:ORBPDUZ="" S
ORBDUZ=ORBPDUZ D USER
$STACK(007)=DO
$STACK(007,"ECODE")=
$STACK(007,"PLACE")=USER+1^ORB3
$STACK(007,"MCODE")= D
USER^ORB3USER(.XQA,ORBDUZ,ORN,.ORBU,.ORBUI,ORBDFN,+$G(ORNUM))
$STACK(008)=DO
$STACK(008,"ECODE")=
$STACK(008,"PLACE")=USER+14^ORB3USER
$STACK(008,"MCODE")= S ORBTM=$P(ORBDUZ,U,2) I $L(ORBTM) D ;if user recip
via team
$STACK(009)=DO
$STACK(009,"ECODE")=
$STACK(009,"PLACE")=USER+15^ORB3USER
$STACK(009,"MCODE")= .S ORBTMF=$$GET^XPAR(ORBTM_";OR(100.21,","ORB
PROCESSING FLAG",ORN,"I")
$STACK(010)=$$
$STACK(010,"ECODE")=
$STACK(010,"PLACE")=GET+9^XPAR
$STACK(010,"MCODE")= D INTERN^XPAR1 I ERR Q ""
$STACK(011)=DO
$STACK(011,"ECODE")=
$STACK(011,"PLACE")=INTERN+18^XPAR1
$STACK(011,"MCODE")= I '(+ENT&(ENT[";")) D ENTEXT(.ENT) D:ENT="" G C1
;resolve external vptr fmt
$STACK(012)=DO
$STACK(012,"ECODE")=
$STACK(012,"PLACE")=ENTEXT+8^XPAR1
$STACK(012,"MCODE")= S ENT=$$FIND1^DIC(FN,"","X",X)_$$MAKEVP(FN)
$STACK(013)=$$
|