Fixing the LED panel freeze issue

[Eric N]

After implementing Oscar's changes to historybuffer, I still saw the LED freezes occurring after a month or so of use, so I decided to dive in and look for other potential race conditions.  I think I've found the culprit.  There is a double-buffer hand-off between the LED pattern thread and the GPIO mux thread that should also be wrapped in a mutex to be thread safe.  The calls occur in gpio.c and gpiopattern.c.  I mutex'd the calls, and I've been running an instance of RSX-11M for two months now with no freezes.

Here are the changes, and I've included Oscar's fixes as well:

>>Oscar's changes
07.0_blinkenlight_api
 - historybuffer.c
 - historybuffer.h

>>My changes
 11_pidp_server/pidp11
 - gpio.c
 - gpio.h
 - gpiopattern.c
 - gpiopattern.h

Once you have replaced the files, make sure you have killed the pidp server process, then run the following build commands on your Pi:

- sudo ./makeclient.sh

- sudo ./makeserver.sh

Restart pidp (or reboot your Pi) and you should be good to go.

It would be great to get feedback on these fixes, and if things look good, finally put this LED issue to bed!

Eric

[Glenn Babecki]

Out of perhaps unwarranted caution, does this issue and fix also apply to the PiDP-8?

Somehow I had a notion that the PiDP-8 also suffered from the LEDs freezing after a prolonged period. However a search of the PiDP-8 forum didn't turn up a discussion on this issue, so I believe my concern may be unwarranted.

--
You received this message because you are subscribed to the Google Groups "[PiDP-11]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pidp-11+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pidp-11/dee48ddc-4d03-48e3-ac58-b05b763a7605n%40googlegroups.com.

[Eric N]

I only have the PiDP-11 hardware, so I can't confirm for the PiDP-8.  However, the gpio source files I modified are within the pidp_server code, and don't make any specific references to PDP models.

Eric

[Glenn Babecki]

Thanks.  I'll check in to the software relationship when I get a chance.

To view this discussion visit https://groups.google.com/d/msgid/pidp-11/965f1087-41c4-439f-baa5-2baccc54d15dn%40googlegroups.com.

[timr...@gmail.com]

My PiDP-8 has been up and running TSS/8 for 140 days.  So I would say No.  I did have the issue on my PiDP-11 a few times.  After about 30 days I seem to remember.

On Sunday, June 22, 2025 at 9:47:35 AM UTC-5 glenn....@gmail.com wrote:

[terry-...@glaver.org]

Please take a look at this thread: https://groups.google.com/g/pidp-11/c/LWJ2PJSfiLU

I and at least one other user have problems when your patches are installed - after a reboot, the system comes up and appears to immediately halt / shut down. I'm on a Pi 5, I don't know what model the other user reporting this problem has.

If I'm bored later I'll do a diff between your files and the ones in Oscar's distribution. I still don't see how a user-mode program could halt the system, but here we are...

[Eric N]

I'm running Rasbpian Buster on a Pi 4.  My changes were minimal. I wrapped a couple of calls in a mutex.  When you diff the files, you will see what I've changed.  Perhaps trying running the native checkout of PiDP with no changes on the Pi 5 and see if there are issues.  Happy to help here if want me to test anything.  I've been running three months now with no issues.

Eric

[terry-...@glaver.org]

On Saturday, July 5, 2025 at 11:45:53 PM UTC-4 eric...@gmail.com wrote:

I'm running Rasbpian Buster on a Pi 4.  My changes were minimal. I wrapped a couple of calls in a mutex.  When you diff the files, you will see what I've changed.  Perhaps trying running the native checkout of PiDP with no changes on the Pi 5 and see if there are issues.  Happy to help here if want me to test anything.  I've been running three months now with no issues.

The  stock install works, both in the  "use precompiled binaries" and "compile from source" variations. I'm using Bookworm on a Pi 5.

It is only when your patches are applied that the problem appears. I tried both your suggested rebuild of the of the client and server apps as well as putting your updated files into the source tree between the "git pull" and the "install.sh" steps and then selecting compilation from sources. All of these cause a shutdown as soon as the Pi boots and the emulator starts.

Your files do not seem to be based on the versions found in the GitHub repo - for example:
https://github.com/obsolescence/pidp11/blob/main/src/11_pidp_server/pidp11/gpio.c

That has a latest comment of:

18-Dec-2023  OV    new GPIO interface using the code of pinctrl, needed for Pi 5

Your gpio.c seems to be lacking that, with the latest comment being:

14-Aug-2019  OV    fix for Raspberry Pi 4's different pullup configuration
which is one older than the latest comment in the GitHub repo.

So it seems your files are not aware of the changes Oscar made for Pi 5 support. That probably explains why things are failing on the Pi 5 (at least). A diff between your gpio.c and Oscar's from GitHub is massive at 404 lines:

(1:216) gate:/data1/Temp/PiDP11 LED fix round 2% diff gpio.c ~terry | wc -l
     404

Can you post diffs between the unmodified versions from Oscar that you based your patches on, and your patched versions? That should let me try to re-base your patches on Oscar's current code.

[Eric N]

Well, bummer. I didn't realize there had been a recent  update for Pi 5, but I see the code changes now.  Here are diffs of my changes based on the "Pre-Pi5" code updates:

[terry-...@glaver.org]

On Sunday, July 6, 2025 at 9:18:01 AM UTC-4 eric...@gmail.com wrote:

Well, bummer. I didn't realize there had been a recent  update for Pi 5, but I see the code changes now.  Here are diffs of my changes based on the "Pre-Pi5" code updates:

I've merged your changes into the latest version of the emulator, as well as making a number of other

fixes for things that have been nagging me:

1) Added comments to gpio.c to tell users what to change if the knobs cause the display to cycle backwards.

2) Worked around a bug in Oscar's install script that didn't install the terminal emulators at all if you selected the "Compile from source" option at install time.

3) Use the same logic as the install script to remove old executables before compiling the fixed versions.

4) Fixed up the botched file permissions on the recompiled executables which prevented them from (at a minimum) being able to set their realtime priority.

5) Improved the ~pi/pdp.sh to say something more useful than "0" if the emulator is not running, or if more than one screen master session is detected. Note that this might break stuff that depends on the output from pdp.sh - see below.

I have NOT run this on an actual PiDP-11, just a Raspberry Pi 5 with no PiDP-11 attached. It may be broken in blatantly obvious (or subtle) ways when run on an actual PiDP-11.

I do see the following issues, which I think are unrelated to any of these changes. These are happening with a VNC desktop and X11, and may not happen on a real HDMI display connected to the Pi, or with Wayland. These will need to be investigated and fixed:

1) The VT52 emulators exit with an unseen error message: "SDL_CreateWindowAndRenderer() failed: No available displays". However, there are available displays:
pi@pool11:~/Desktop $ xrandr
Screen 0: minimum 320 x 200, current 1920 x 1080, maximum 8192 x 8192
HDMI-1 disconnected primary (normal left inverted right x axis y axis)
HDMI-2 disconnected (normal left inverted right x axis y axis)

2) The Tek emulator puts up a window which instantly vanishes:

tek4010 version 1.9
Screen dimensions: 1920 x 1080
Window dimensions: 1024 x 780
Process has been terminated

Neither terminal emulator has source code included in the PiDP-11 distribution and sty (ASR33 emulation) works fine.

To install the fixes:

1) Back up your entire /opt/pidp11 directory tree (just in case)

2) Make sure the emulator is not running by using pidpcontrol and selecting Stop if the emulator is running

3) Place the attached file in your login (~pi) directory

4) Extract with: tar -xpvf LED-Fix-2.tgz

5) Change to that directory: cd LED-fix-2  (note the intentional difference in capitalization)

6) Type: ./recompile.sh

7) Sit back and wait (how long depends on the speed of your Pi and your storage - it takes about a minute and a half on my Pi 5 with an SDXC card)

8) Start the emulator as usual with pidpcontrol

9) Report your successes (or my failures) here

[Tony Nicholson]

On Monday, July 7, 2025 at 1:30:52 PM UTC+10 terry-...@glaver.org wrote:

On Sunday, July 6, 2025 at 9:18:01 AM UTC-4 eric...@gmail.com wrote:

Well, bummer. I didn't realize there had been a recent  update for Pi 5, but I see the code changes now.  Here are diffs of my changes based on the "Pre-Pi5" code updates:

I've merged your changes into the latest version of the emulator, as well as making a number of other

fixes for things that have been nagging me:

[snip]

9) Report your successes (or my failures) here

Terri,

As it happens my PiDP-11 entered the panel freeze mode some time

earlier today - giving me the opportunity to try your LED-Fix-2.tgz changes.

Following your instructions yielded a successful compile/update.

Re-booting my PI 3B controlled panel and it came up correctly running

my RSTS/E V10.1 system.

For reference the PI 3B is running 64-bit Bookworm with all the most

recent updates -

pi@pi3:~ $ uname -a
Linux pi3 6.12.34+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.34-1+rpt1~bookworm (2025-06-26) aarch64 GNU/Linux

Time will tell if this has worked.

Thank you.

Tony

 

[Tony Nicholson]

On Monday, July 7, 2025 at 3:10:02 PM UTC+10 Tony Nicholson wrote:

On Monday, July 7, 2025 at 1:30:52 PM UTC+10 terry-...@glaver.org wrote:

On Sunday, July 6, 2025 at 9:18:01 AM UTC-4 eric...@gmail.com wrote:

Well, bummer. I didn't realize there had been a recent  update for Pi 5, but I see the code changes now.  Here are diffs of my changes based on the "Pre-Pi5" code updates:

I've merged your changes into the latest version of the emulator, as well as making a number of other

fixes for things that have been nagging me:

[snip]

9) Report your successes (or my failures) here

Terri,

As it happens my PiDP-11 entered the panel freeze mode some time

earlier today - giving me the opportunity to try your LED-Fix-2.tgz changes.

Following your instructions yielded a successful compile/update.

Re-booting my PI 3B controlled panel and it came up correctly running

my RSTS/E V10.1 system.

Spoke too soon.   The system did not start DECnet properly.

I had to re-apply my fix to be able to get it to use libpcap

as non-root (as I use the raw eth0 device and not a bridge).

This should also be included in your update just in case other

simulations have an 'att xu eth0' in their boot.ini file.

#!/bin/sh
#
# Set capabilities on the PiDP11 client to allow pcap to attach to raw eth0
# interface.
#
sudo setcap cap_net_raw,cap_net_admin=+ep /opt/pidp11/src/02.3_simh/4.x+realcons/bin-rpi/pdp11_realcons
sudo getcap /opt/pidp11/src/02.3_simh/4.x+realcons/bin-rpi/pdp11_realcons

Tony

 

[terry-...@glaver.org]

Spoke too soon.   The system did not start DECnet properly.

I had to re-apply my fix to be able to get it to use libpcap

as non-root (as I use the raw eth0 device and not a bridge).

This should also be included in your update just in case other

simulations have an 'att xu eth0' in their boot.ini file.

#!/bin/sh
#
# Set capabilities on the PiDP11 client to allow pcap to attach to raw eth0
# interface.
#
sudo setcap cap_net_raw,cap_net_admin=+ep /opt/pidp11/src/02.3_simh/4.x+realcons/bin-rpi/pdp11_realcons
sudo getcap /opt/pidp11/src/02.3_simh/4.x+realcons/bin-rpi/pdp11_realcons

I missed that in Oscar's install.sh. He has eip, not just ep. However, I'm utterly unfamiliar with Linux capability flags (my background is in FreeBSD). I added:

# to use PCAP as non-root
sudo setcap cap_net_raw,cap_net_admin=+eip /opt/pidp11/src/02.3_simh/4.x+realcons/bin-rpi/pdp11_realcons

to the recompile script. I'm not going to post an updated .tgz here until I get some more reports in on other things that might be broken.

Can you try another ./recompile.sh (to get back to missing the capabilities) and then my command line above and let me know if it fixes the issue for you? 

[Tony Nicholson]

On Monday, July 7, 2025 at 4:20:01 PM UTC+10 terry-...@glaver.org wrote:

I missed that in Oscar's install.sh. He has eip, not just ep. However, I'm utterly unfamiliar with Linux capability flags (my background is in FreeBSD). I added:

# to use PCAP as non-root
sudo setcap cap_net_raw,cap_net_admin=+eip /opt/pidp11/src/02.3_simh/4.x+realcons/bin-rpi/pdp11_realcons

The “i” in the “eip” is to grant inheritance of the capability for any program executed via an execve() call from the simulator.  I guess this is safe to allow network interface setups to be spawned from SIMH pdp11_realcons when executing commands from the boot.ini or sim> prompt.

I ran ./recompile.sh again followed by the above setcap command and it works.

Tony

[terry-...@glaver.org]

On Monday, July 7, 2025 at 7:30:43 AM UTC-4 agn...@gmail.com wrote:

I ran ./recompile.sh again followed by the above setcap command and it works.

Great!

I'm now running my Pi 5 in a PiDP-11 case I finished way back in December 2018.

I needed the same fix as you did, and I also changed the knob rotation direction to be correct for my particular PiDP-11 (apparently they vary based on some unknown condition).

My PiDP-11 has  4 physical serial ports on the back, connected to the Pi via the MAX232 converters and USB-serial adapters. Right now the power cord is sticking out the top - I'm waiting for a USB-C internal bulkhead extension cord as well as my NVMe bottom plate for the Pi. Until those are installed and 4 terminals are up and running without losing the LEDs, I'll consider it finally finished (gee, only 6.5 years!).

Here's a picture of it running RSTS/E with the temporary power cord and a single VT420 terminal running DISPLY:

[terry-...@glaver.org]

On Monday, July 7, 2025 at 7:30:43 AM UTC-4 agn...@gmail.com wrote:

I ran ./recompile.sh again followed by the above setcap command and it works.

Attached is the updated recompile.sh file. Just copy it into the LED-fix-2 directory after unpacking the .tgz in my earlier post.

That should hopefully let everyone get up-to-date with this latest set of patches and improvements. 

[Eric N]

I'll try this on my Pi4 and PiDP-11 system this weekend, thanks for merging everything.

Eric

[Eric N]

Terry, are you starting with the PIDP-11 source code located here (prior to applying your changes): https://github.com/slcasner/pidp11

I'm getting a number of compile errors and missing items when I run recompile.  Before I dive into them, I want to make sure I'm starting from the right source tree.

Eric

[Eric N]

Everything appeared to compile correctly, however, I get the following error when starting the pdp.sh script, and there are no lights on my PDP-11:

PiDP-11 FP 20250706
[21:07:24.205] unable to register (BLINKENLIGHTD, BLINKENLIGHTD_VERS, udp).
PDP-11 simulator V4.0-0 Current  REALCONS build Jul 12 2025
Disabling XQ
Searching realcons controller "11/70" ...
Connecting to host localhost ...
Connect to host localhost failed.
boot.ini-5> set realcons connected
Unit not attachable

Eric

[terry-...@glaver.org]

On Saturday, July 12, 2025 at 5:13:19 PM UTC-4 eric...@gmail.com wrote:

Everything appeared to compile correctly, however, I get the following error when starting the pdp.sh script, and there are no lights on my PDP-11:

[Update from after I typed and proofread the reply below - as they said in "Raiders of the Lost Ark", 'You're digging in the wrong place'. The URL you gave in your prior reply, https://github.com/slcasner/pidp11, seems to be an unknown fork of older PiDP-11 software. Start with the instructions as I quote below which use Oscar's official GitHub repo and see if that works better.]

I started with a clean install via the instructions found here:
https://obsolescence.dev/pidp-11-building-instructions.html

In particular, this:
cd /opt
sudo git clone https://github.com/obsolescence/pidp11
/opt/pidp11/install/install.sh 

Once that is done, unpack the patch .tgz I posted in this thread. Then download the updated recompile.sh file into the same directory as the updated patches. Then do ./recompile and everything should be good to go (in theory).

Potential gotchas:
1) I broke something (the updated recompile.sh fixes a problem with not being able to attach the Ethernet device)

2) You're on a Pi other than a 5, but I don't think any of the changes are Pi-model-specific

3) You took a non-default answer to Oscar's  /opt/pidp11/install/install.sh script and it's doing something I'm not handling properly

4) You're running a Pi OS other than the latest Bookworm (Oscar's restriction, not mine)
5) Some unknown error I haven't seen yet

6) Evil spirits 8-}

I suggest renaming /opt/pidp11 to something like /opt/pidp11-save and then starting from scratch with Oscar's script. Make sure everything works (including the LEDs). Then use the desktop pdpcontrol to stop the emulator, run my recompile.sh and try again.

I apologize for being a part of the problem (old chemistry joke: "If you're not part of the solution, you're part of the precipitate"). This has been going on for 5.5 years now and I'm really trying hard to get the LED fixes (along with my other fixes in recompile.sh) to work for good. I've already filed GitHub reports on the other issues, so hopefully they'll be addressed in the official repository at some point in the future.

If it still doesn't work, say "script bugreport.txt" in a console window, run recompile.sh, then say exit to exit the script command. Then email me the bugreport.txt ("reply to author" should work).  Of course (these things always happen like this) I'm away on a weeklong vacation with my partner 150 miles from home. I can remote in to the Pi, but can't tell if the lights are working (they were when I left).

[Tony Nicholson]

The official repository on GitHub for the PiDP-11 is at

https://github.com/obsolescence/pidp11

Tony

Tony Nicholson <tony.ni...@computer.org>

--
You received this message because you are subscribed to a topic in the Google Groups "[PiDP-11]" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pidp-11/osWq3mfln_g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pidp-11+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pidp-11/0a686404-d71e-4f36-bd99-8fea79237511n%40googlegroups.com.

[terry-...@glaver.org]

Deleted/reposted because while the preview was good, Google damaged the post.

Can I get feedback from anyone else who has tried my LED-Fix-2.tgz with updated recompile.sh a few posts further down?

[terry-...@glaver.org]

On Tuesday, August 26, 2025 at 1:54:30 AM UTC-4 terry-...@glaver.org wrote:

Can I get feedback from anyone else who has tried my LED-Fix-2.tgz with updated recompile.sh a few posts further down?

Bill Ezell emailed me directly to tell me about another LED hang fix that he submitted and Oscar accepted, but which didn't make it into the GitHub repo. That one is in hisorybuffer.c.

I'm debating whether I should:
A) Create a new LED-fix release with this and other fixes
B) Build a complete kit with this and other fixes, replacing the "git clone" with a curl or wget install
C) Fork Oscar's PiDP-11 repo and do all work in there  

Those are in order of easiest to hardest for me. But option A can't fix an error in Oscar's install script because users have already run it by the time they run my LED-fix release. Option B is good for me, but provides an opaque "black box" to users unless they compare each file against Oscar's repo - I'm essentially saying "trust me" (although I've been involved with PDP-11s since 1982 and I am well-known in the DECUS community. Option C is more work, but the world doesn't need Yet Another Fork of Oscar's repo - there are 9 already. Eight are moribund (behind Oscar's current repo) and the remaining one is one commit ahead but idle for 3+ months.

I'd be interested in hearing people's thoughts on this, including an additional option D - "It works fine for me now, I don't care what you do". Please use the "Reply to author" if you're just voting, so it doesn't clog this forum. OTOH, if you want to say something beyond simply voting, feel free to reply to the group. 

[Tony Nicholson]

On Friday, August 29, 2025 at 10:39:25 AM UTC+10 terry-...@glaver.org wrote:

On Tuesday, August 26, 2025 at 1:54:30 AM UTC-4 terry-...@glaver.org wrote:

Can I get feedback from anyone else who has tried my LED-Fix-2.tgz with updated recompile.sh a few posts further down?

Bill Ezell emailed me directly to tell me about another LED hang fix that he submitted and Oscar accepted, but which didn't make it into the GitHub repo. That one is in hisorybuffer.c.

I'm debating whether I should:
A) Create a new LED-fix release with this and other fixes
B) Build a complete kit with this and other fixes, replacing the "git clone" with a curl or wget install
C) Fork Oscar's PiDP-11 repo and do all work in there

I've had no LED panel freeze since incorporating your fixes.

Actually I'd prefer option C with the updates being eventually pushed back to the GitHub repository main branch and merged (by Oscar).  The install.sh script would also need to be able to detect and remedy any system mods that need to be undone/fixed from its previous incarnation.  That way people just do a git pull into their previously cloned directory and re-run install.sh (which hopefully doesn't touch anything in the systems directory - where I've added operating systems and modified the selections and various boot.ini files).

If this isn't possible, then an update procedure would involve preserving the old /opt/pidp11 heirarchy, cloning the repository, and restoring local systems and boot file mods back into the new heirarchy - similar to what I did recently when I upgraded Raspberry Pi OS

Tony

[Adam Thornton]

It would certainly be easiest for me to have a git repo I can clone and build from.  I'd like a commit history so I know how my sources have diverged from the official ones.  I don't want a prebuilt version because I'm not running Raspian, I'm running stock Debian, and I think on a Pi 3B, although I don't remember the actual model, but not one of the new shiny ones.  I don't have any idea whether or not I have all the fixes so far (but probably not since both my pidp-10 and -11 had panel freezes in the last few days), because I am not good at keeping up with my email, and if I read it and I'm not sitting on my home machine, which I often am not, I'm not going to remember to download and apply the patches when I get home.

I don't mind (once I have a fully updated source tree) doing yet another fork and making a PR and working through any of Oscar's feedback (although I don't promise timeliness), if you don't want to deal with the PR process.  Ultimately an installer that preserved local mods (and let you set a user, rather than hardcoding "pi") would be nice to have.

In the longer term, a monorepo that combined the -8, the -11, the -10, and the -1 (I'm sure the underlying panel libraries have more in common than they do different) would be lovely to have, but that's definitely not something I'm volunteering to do now.  Maybe once my observatory has been on-sky a while and work has settled down it's something I could take on.

Adam

--

You received this message because you are subscribed to the Google Groups "[PiDP-11]" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pidp-11+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pidp-11/c934d586-60dc-49bf-afa4-c26feecf1b8bn%40googlegroups.com.

[terry-...@glaver.org]

The votes are in! The winner was "Create a GitHub fork" and the runner-up was "No opinion".

The fork is located at: https://github.com/Terri-Kennedy/pidp11

I will be committing changes over the coming months. Until I announce that it is ready for use, please assume that this fork will toggle between "no improvements" and "badly broken" and should not be used.

[Steven A. Falco]

That is excellent news, Terry. Thanks for taking it on.

Steve

> --
> You received this message because you are subscribed to the Google Groups "[PiDP-11]" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to pidp-11+u...@googlegroups.com <mailto:pidp-11+u...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pidp-11/15f1ec71-045b-4fa6-afa7-4ba4099824een%40googlegroups.com <https://groups.google.com/d/msgid/pidp-11/15f1ec71-045b-4fa6-afa7-4ba4099824een%40googlegroups.com?utm_medium=email&utm_source=footer>.

[John Hudak]

Thanks for the update.  Looks like 'yet another fork' and a ton of work on your end.   Do you think having a secondary contact/back up person would be helpful to help with/carry on with future maintenance?

Might be helpful to document the issues that lead to this, and the design approach/architecture used to fix the issues.  Seems like useful (key?) design docs were absent in the current fork.

J

--

You received this message because you are subscribed to the Google Groups "[PiDP-11]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pidp-11+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/pidp-11/15f1ec71-045b-4fa6-afa7-4ba4099824een%40googlegroups.com.

[Bert Driehuis]

Let's not overengineer the solution before it gets off the ground... Terry volunteered to take patches. Github is an excellent platform to collaborate on a project, and if it turns out that handling the pull requests becomes a big task, there are many ways of tackling that issue. But let's get the pull requests flowing first. I should probably get started.

Terry, a heartfelt thanks for championing this!

With kind regards,

Bert

To view this discussion visit https://groups.google.com/d/msgid/pidp-11/CAE%3DviVqNawT22dse4KmV8xqBUwMBFytbPE3qk2hWspp6gOXFJA%40mail.gmail.com.