RSX-11 system crash (Driver not loaded)

[Eric N]

I've had a very stable RSX-11 system running on PiDP-11 for years now, but over the last few months, the system randomly crashes and takes me to the XDT prompt (see below).  The system can sit idle for a few hours or a few days, then it just randomly halts.  I've tried restoring from a backup image with the same results. 

The only changes I've made are the usual sudo apt-get upgrades on Raspbian v10 (buster) to current releases.  Any ideas what's causing this crash?

Here's the output from the command line:

$
BE:122250
XDT>x
CRASH -- CRASH DRIVER NOT LOADE
HALT instruction, PC: 020670 (BR 20666)
sim>

[Eric N]

One additional bit of info:  The breakpoint is always PC: 020670.

Eric

[Anton Lavrentiev]

Hi Eric,

Sorry to hear about your troubles.

I am running RSX-11M+ on buster with all the recent Linux updates (and the original PiDP11's systems.tgz) without any issues, and it's been running for a month now (a few latest updates didn't even require a reboot)....

Did you try a new SD card? To me it looks like you've experiencing a data corruption of some sorts...

As for the PC being the same each time, I guess it traps to a service routine (because of the absent driver) and so that halts at that very same address.  It'd be cool if someone of RSX gurus on here could advise on how to figure out which exactly the driver was missing, as I see no detailed information to that extent in the message.

Anton

--
You received this message because you are subscribed to the Google Groups "[PiDP-11]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pidp-11+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/a54a0b67-3ac0-4747-a1dc-dcd0c58dfe2fn%40googlegroups.com.

[Johnny Billquist]

Well. A couple of comments.

The message about crash driver not loaded have nothing to do with the
crash. That is just a message about when you have a crash, there is no
crash driver loaded. The crash driver is capable of/reponsible for
making a crash dump in the event of a crash.

Assuming crash dump support have been generated into the kernel, you can
select what crash driver to use by just giving the command:

> SET /CRASH_DEVICE=MU0:

for example, for having the crash dump written to MU0:

Or check what it is currently set to by:

> SET /CRASH_DEVICE

If you have a crash, then it can be analyzed properly if you have the
crash dump, and the .STB file of the kernel, which you usually can find
at LB:[1,54]RSX11M.STB

There is a whole manual about the crash dump analysis tool. There is
also a whole manual about XDT, which is the kernel debugger.

As it is, there is too little information in here to really tell what
the problem is.

BE is however, a breakpoint. Basically, the kernel executed a BPT
instruction in kernel mode.

Johnny

> --
> You received this message because you are subscribed to the Google
> Groups "[PiDP-11]" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pidp-11+u...@googlegroups.com

> <mailto:pidp-11+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pidp-11/fd6dd997-c991-4fdb-9aa1-28e831c61fd7n%40googlegroups.com
> <https://groups.google.com/d/msgid/pidp-11/fd6dd997-c991-4fdb-9aa1-28e831c61fd7n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol

[Mark Matlock]

Eric,

   I want to add one additional detail to Johnny Billquist’s explanation of setting a crash dump device. Since this is RSX-11M+ on a PiDP-11/70 (RSX-11M+ on it was sysgened with the XX: crash device) and not a real PDP-11, in addition to the 

>SET /CRASH_DEVICE=MU0:

you also need, first, to ^E at the console to get a sim> prompt and type

sim>attach tq0 crash.dmp

This is the software analogy of putting a blank tape on your TU81 (or other MSCP disk) so that when the crash occurs the crash media is ready. You could also add that attach command to the PiDP-11/70 RSX-11M+ (console switch bit 0 up) boot.ini file located in:

 /opt/pidp11/systems/rsx11mplus/boot.ini

After the SET /CRASH_DEVICE=MU0: you should see:

SET — Crash Device MU000: has been successfully loaded

At your next crash you should see XDT>

then type X and you’ll see

CRASH — CONT WITH SCRATCH MEDIA ON MU0

at that point the crash dump will take place. Then you can reboot attach the crash.dmp file to tq0 and run CDA as Johnny explained to analyze the crash.

With CDA you can determine which task was active at the time of the crash and even what instruction was being executed when it occurred on.

One source of potential crashes is open serial lines. I don’t know if this happens much on simh based emulation, but RSX-11M/M+ on real PDP-11s was susceptible to a non-logged in serial port getting a high speed slug of unsolicted characters. Since the line is not logged in, and if no task has a QIO read posted against it, RSX thinks someone is trying to log in and replies. This can happen very fast and the outstanding I/O piles up and a crash can happen. Normally any line that you attach to a data source should be SET /SLAVE=TTnn: to avoid this.

Good Luck,

Mark

-- 
You received this message because you are subscribed to the Google Groups "[PiDP-11]" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pidp-11+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/a54a0b67-3ac0-4747-a1dc-dcd0c58dfe2fn%40googlegroups.com.

[Robert Evans]

  A few years ago, using XDT, I analyzed a crashing RSX-11M 3.2 kernel on my PiDP-11.  I point you to this as another way the problem can be examined live without writing the contents of memory to a storage device.  In my case, I made errors during SYSGEN that were not obvious to me until I did the crash analysis.

[Eric N]

Appreciate the tips and I will post my results of the crash dump this weekend.

Eric

[Mark Matlock]

Robert,
Thanks for the link to your RSX-11M V3.2 crash analysis with XDT. It was an interesting read that refreshed my memory about using XDT. I also checked out the RSX11M v3.2 Redux page which was also a bit of a trip down memory lane.

Best,
Mark

> To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/77746fea-acd0-4761-8a62-0cd23fbc0e85n%40googlegroups.com.

[Eric N]

Everything is set and I'm just waiting for another crash to occur.  Of course, it will probably behave fine now.  In either case, I'll also prepare a replacement flash.

Eric

[Anton Lavrentiev]

Eric,
I'm just curious for no follow up -- nothing happened?
:-)

> To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/4983d2e1-4f8a-442b-84d6-1932aa2e5877n%40googlegroups.com.

[Eric N]

The darn thing has been running with no issues since Saturday.  I haven't replaced the flash card yet either.  As soon as it crashes, I'll post an update.

Eric

[Eric N]

Just updating everyone that the darn thing is still running with no crashes. 

One interesting thing I did see that I believe is unrelated to this issue is that the LED panel froze (the PiDP sim was still running with no issues).  I've seen this occur in the past regardless of which PDP OS I'm running.  The PiDP sits idle for a few weeks, then the LED's freeze up and I need to restart PiDP to get them working again.  The PiDP OS is not affected.  Has anyone else seen this?

Eric

[David Johnson]

My PiDP will sometimes stop updating the LEDs. Have not been able to troubleshoot anything,

usually just reboot the Pi.  

To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/1890ff04-dac2-45fa-a463-b95a9137b09an%40googlegroups.com.

[Johnny Billquist]

On 2022-12-11 19:38, Eric N wrote:
> Just updating everyone that the darn thing is still running with no
> crashes.

:-)

> One interesting thing I did see that I believe is unrelated to this
> issue is that the LED panel froze (the PiDP sim was still running with
> no issues).  I've seen this occur in the past regardless of which PDP OS
> I'm running.  The PiDP sits idle for a few weeks, then the LED's freeze
> up and I need to restart PiDP to get them working again.  The PiDP OS is
> not affected.  Has anyone else seen this?

Yes. It's sortof a known bug. I haven't had time or energy to read up on
the code that drives the front panel, but there is some kind of bug in
there that triggers occasionally.

Johnny

> <https://groups.google.com/d/msgid/pidp-11/a54a0b67-3ac0-4747-a1dc-dcd0c58dfe2fn%40googlegroups.com>.

>
> >> >
> >> >
> >> > --
> >> > You received this message because you are subscribed to
> the Google Groups "[PiDP-11]" group.
> >> > To unsubscribe from this group and stop receiving emails
> from it, send an email to pidp-11+u...@googlegroups.com.
> >> > To view this discussion on the web visit

> https://groups.google.com/d/msgid/pidp-11/77746fea-acd0-4761-8a62-0cd23fbc0e85n%40googlegroups.com
> <https://groups.google.com/d/msgid/pidp-11/77746fea-acd0-4761-8a62-0cd23fbc0e85n%40googlegroups.com>.

>
> >>
> > --
> > You received this message because you are subscribed to the
> Google Groups "[PiDP-11]" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send an email to pidp-11+u...@googlegroups.com.
> > To view this discussion on the web visit

> https://groups.google.com/d/msgid/pidp-11/4983d2e1-4f8a-442b-84d6-1932aa2e5877n%40googlegroups.com
> <https://groups.google.com/d/msgid/pidp-11/4983d2e1-4f8a-442b-84d6-1932aa2e5877n%40googlegroups.com>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "[PiDP-11]" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pidp-11+u...@googlegroups.com

> <mailto:pidp-11+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pidp-11/1890ff04-dac2-45fa-a463-b95a9137b09an%40googlegroups.com
> <https://groups.google.com/d/msgid/pidp-11/1890ff04-dac2-45fa-a463-b95a9137b09an%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Anton Lavrentiev]

Yes, the freeze happened to me too just this morning (hmm, a weird coincidence with yours?).... RSX-11M+ was running fine (for like a month), and in fact, it kept running just fine when I opened the console, but the blinkenlights stopped.... I had to shut down RSX before restarted the entire pidp11 "service", and it was back on again.

It happens rather sporadically.  Last time it happened was probably no less than half a year ago for me.  And RSX uptime is in the range of months between the restarts, and there were no freezes lately until today.

To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/1890ff04-dac2-45fa-a463-b95a9137b09an%40googlegroups.com.

[Rick Reynolds]

Just chiming in to say that his has happened several times to me also. I’ll come into my office and notice that the lights are no longer moving. I usually just reboot the Pi at that point because I really don’t have my pidp11 doing much other than showing pretty blinkenlights.

To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/CAAo%3Dyr3hTPTxcW09D2zrXL%3DLTYR6xn10cTY2Qh-WiJq3Dy%3DBqA%40mail.gmail.com.

[Steve Tockey]

Blinkenlight freezing happens on my PiDP11/70 frequently as well. I can't say for sure, but it at least seems associated with running the Chromium web browser. If I don't run Chromium it seems to rarely happen, if at all. On the other hand, if I do run Chromium it appears to happen quite frequently--on the order of 75% of the time. It may be just a coincidence, but that's what I'm seeing.

Note that I don't leave mine on for long periods of time. It's on for a couple of hours when I want to do something with it and off otherwise. When I do turn it on, it freezes within the first 5-10 minutes of starting Chromium.My habit is to boot everything (RPi and 11/70) then do everything I need to do in Chromium. If it doesn't freeze, fine, I continue with the 11/70. If it does freeze, I reboot the RPi and skip Chromium because I don't need it anymore.

And, FWIW, you may not need to completely reboot the RPi to recover. I usually do just reboot anyway but I seem to recall being able to CTRL/E to drop into SIMH and then:

.CTRL/E

simh> cd ../<your system directory, e.g., idled>

simh> do boot.ini

It could be a useful data point if this does un-freeze SIMH vs. if it doesn't.

-- steve

[Anton Lavrentiev]

You can just restart the service from Pi's shell:

$ sudo service pidp11 restart

but it's always best to shut down whatever is running under SimH first (e.g. "run $shutup" for RSTS or RSX)

--
You received this message because you are subscribed to the Google Groups "[PiDP-11]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pidp-11+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pidp-11/CAHF_aiz6C7ejs14EhxOrk4DaQEwjX_uvTyx9Btpzkqduv0KybQ%40mail.gmail.com.

[Eric N]

With my RSX system running for over two weeks now with no driver crashes, I'm going to try one more experiment and not set the crash device MU0: this time (original configuration) to see if I can replicate the original issue.

Eric