Medley hangs while processing fonts

[Herb Jellinek]

Hi all,

As I mentioned in our meeting this morning, Medley hangs when I have it process a number of HTMLSTREAM font metrics files.

As requested, I ran the code again and looked at lsof, FileWatch, and the VM size.

I'm running under SDL on an Intel MacBook Pro, macOS Sequoia (15.7.3).
VM size never exceeded 12%, which is the same value it had when I started the computation.
There were only 3 open files when Medley hung.

The effective process command line was

exec ~/Projects/IL/medley/scripts/medley/medley.sh -y - -e -k ~/il/vmem/lisp.virtualmem -r ~/IL/INIT -ps 2 -x ~/Projects/IL

That ran Maiko via

~/Projects/IL/medley/../maiko/darwin.x86_64/lde /Users/hjellinek/il/vmem/lisp.virtualmem -id default -title Medley -g 1462x922 -sc 1440x900 -pixelscale 2

I've attached the source code I executed and output of the stack sampler and lsof.  Note that Medley hangs more quickly without the call to BLOCK.

On the screen:





I'm happy to work with any volunteers to debug this literal showstopper.

                Herb

[Nick Briggs]

I believe it was trying to say:

    if (dtd68k->dtd_free & 0x8000001) error("bad entry on free chain.");

based on the stack trace from N_OP_createcell at mkcell.c:76, since below that it was in error() and trying to print something in URaid, but it failed miserably.

-- Nick

[Herb Jellinek]

Great, that narrows things down a bit.

How can we work backwards from here to find the bad free chain entry, or trap when it's added to the chain?

            Herb

[Nick Briggs]

What was visible in the terminal window from which you started medley?  It should have looked like

*Error* bad entry on free chain.

Enter the URaid

CL:NIL

You could run it under lldb (attach to the running ldesdl process before you start your font metric file mashing) and put a breakpoint at error (break set -n error) and see what the pointer was that it was complaining about ( it might be 0x1 ).

-- Nick

<screenshot_1155.png>

<screenshot_1154.png>

I'm happy to work with any volunteers to debug this literal showstopper.

                Herb

--
You received this message because you are subscribed to the Google Groups "Medley Interlisp core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lispcore+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lispcore/8f541195-24d7-4c96-b2ad-99c5b00378ed%40newscenter.com.

<code.lisp>

<stack-and-files.zip>

[Nick Briggs]

For a start, recompile mkcell.c with -DDTDDEBUG added to the debug flags in makefile-darwin.x86_64-sdl

DEBUGFLAGS = -DDTDDEBUG # -DDEBUG -DOPTRACE

You'll need to touch maiko/src/mkcell.c and then in the maiko/bin directory run ./makeright sdl to get your new executables.

It might not find anything, but if it does, it should shed some more light on what's going on.

-- Nick

[Herb Jellinek]

I have been running it from my macOS dock, sending stdout to /dev/null.  I just ran it from the terminal and here's what I saw.

*Error* bad entry on free chain.
Enter the URaid
CL:NIL

< ▮

It's waiting for me to tell it what to do.

            Herb

[Herb Jellinek]

Command 'l' shows this on the stack:

  0 :    0x12f3c : IL:\ALLOCHUNK
  1 :    0x12f1c : IL:\MAIKO.ALLOCBLOCK
  2 :    0x12ee4 : IL:ALLOCSTRING
  3 :    0x12eba : IL:UTF8TOMSTRING
  4 :    0x12e92 : SI:*UNWIND-PROTECT*
  5 :    0x12e7a : IL:\UFS.RECOGNIZE.FILE
  6 :    0x12e60 : IL:\UFSGetFileName
  7 :    0x12e42 : IL:\GETFILENAME
  8 :    0x12e2c : IL:INFILEP
  9 :    0x12e02 : IL:\HTML.FONTEXISTS?
 10 :    0x12ddc : IL:FONTEXISTS?
 11 :    0x12db6 : IL:FONTCREATE1
 12 :    0x12d9c : IL:FONTCREATE
 13 :    0x12d76 : IL:COMPLETE.FONT
 14 :    0x12d56 : IL:HTML.COMPLETE-FONT-FAMILY
 15 :    0x12d34 : IL:\EVALFORM
 16 :    0x12d16 : IL:FAULTEVAL
 17 :    0x12cf8 : IL:\EVALFORM
 18 :    0x12ce4 : IL:EVAL
 19 :    0x12cc0 : IL:EVAL-INPUT
 20 :    0x12c4e : IL:DO-EVENT
 21 :    0x12c18 : XCL:EXECA0001A0002
 22 :    0x12be0 : XCL:EXECA0001
 23 :    0x12b3a : IL:\DO.PROGV
 24 :    0x12abc : XCL:EXEC
 25 :    0x12a94 : IL:\PROC.REPEATEDLYEVALQT
 26 :    0x12a74 : IL:\EVALFORM
 27 :    0x12a26 : IL:\MAKE.PROCESS0
 28 :    0x11802 : CL:T

@Nick, we could do a screen share at some point, if that would work for you.
 
          Herb

[Nick Briggs]

It might be interesting to know what the UTF8 string was that it had its hands on - though it's probably NOT this string that was broken, it would have been some previous string.

I hope there's not something bad going on in the UTF8TOMSTRING code that could be walking off the end of the string.  There aren't any weird characters in the file names that translate into multi-byte MCCS characters are there?

I might be able to screen share, but I don't know much about this code - so it might not be that useful.

f number

will show you the stack frame for the number down the left of the stack.

-- Nick

[Ron Kaplan]

But, how could a Medley error give rise to the spinning beachball?

To view this discussion visit https://groups.google.com/d/msgid/lispcore/A97E3676-0440-403C-B6DB-7CC444B896AE%40gmail.com.

[Nick Briggs]

It was sitting in URaid trying to read - if no I/O was possible (not sure where stdin/stderr got connected in this case of it being started from the dock)  then I think it could beachball.  It's just an I/O wait state.

[Ron Kaplan]

Can an error in Medley cause URAID to get into such a state?

[Herb Jellinek]

Can an error in Medley cause URAID to get into such a state?

Yes, in that when you drop into URaid, the macOS app's event loop is suspended.  That causes macOS to show the beach ball cursor whenever the mouse is in the app's window.  Since I was running Medley without a visible terminal, any trips into URaid looked like Medley was hung.

That aside, further experimentation demonstrates that the BLOCK call is unnecessary, and that Maiko discovers the bad entry on the free chain in a variety of places.  For instance, see the stack trace below.

I suspect that something's overwriting storage in the course of storing charset info, probably in my code.  I will stick some assertions in my code and see what I find and report back.

< l
  0 :    0x12e90 : IL:\ALLOCHUNK
  1 :    0x12e70 : IL:\MAIKO.ALLOCBLOCK
  2 :    0x12e4c : IL:BITMAPCREATE
  3 :    0x12e2e : IL:MAKE-CHARSET-BIT-ARRAY
  4 :    0x12dea : IL:\HTML.FONTCREATE
  5 :    0x12dd2 : IL:\CREATEFONT
  6 :    0x12db6 : IL:FONTCREATE1
  7 :    0x12d9c : IL:FONTCREATE
  8 :    0x12d76 : IL:COMPLETE.FONT
  9 :    0x12d56 : IL:HTML.COMPLETE-FONT-FAMILY
 10 :    0x12d34 : IL:\EVALFORM
 11 :    0x12d16 : IL:FAULTEVAL
 12 :    0x12cf8 : IL:\EVALFORM
 13 :    0x12ce4 : IL:EVAL
 14 :    0x12cc0 : IL:EVAL-INPUT
 15 :    0x12c4e : IL:DO-EVENT
 16 :    0x12c18 : XCL:EXECA0001A0002
 17 :    0x12be0 : XCL:EXECA0001
 18 :    0x12b3a : IL:\DO.PROGV
 19 :    0x12abc : XCL:EXEC
 20 :    0x12a94 : IL:\PROC.REPEATEDLYEVALQT
 21 :    0x12a74 : IL:\EVALFORM
 22 :    0x12a26 : IL:\MAKE.PROCESS0
 23 :    0x11802 : CL:T

            Herb

To view this discussion visit https://groups.google.com/d/msgid/lispcore/E92E13AB-1459-4FEC-88EF-107704089384%40post.harvard.edu.

[Ron Kaplan]

I run with ARRAYBLOCKCHECKING set to T.  Maybe that will catch something?

To view this discussion visit https://groups.google.com/d/msgid/lispcore/dbdac706-66c0-4ff8-92b8-2500434b21a4%40newscenter.com.

[Herb Jellinek]

I executed (SETQ ARRAYBLOCKCHECKING T)and ran the problematic code again.  I ended up in URaid again with no additional information.

What would I have seen if ARRAYBLOCKCHECKING had had the desired effect?  What can we infer from the fact that setting it to T didn't seem to change anything?

            Herb

[Ron Kaplan]

I think it scans for certain (but not all) bad arrayblock configurations.

[Nick Briggs]

You would have seen about the same behavior, except the message on entering URaid might have been different.

What was the URaid message?  What were the contents of the most recent stack frames? (the "f <n>" command after you've done an "l" command)

-- Nick

[Herb Jellinek]

The location of the URaid call kept changing, which led me to think the root problem lay elsewhere, and that it was somewhere in my code.

I commented out (actually, (AND NIL ...)ed out) the two pieces of code in \HTML.FONTCREATE that touch array blocks and ran the suspect code without issue.  I re-enabled one, cleared the font cache, and ran the test again.  It didn't fall into URaid.  Enabling the other piece of code did provoke the problem.  The purpose of that bit of code was to create a "slug charsetinfo."  I found code Ron had written to do that, \BUILDSLUGCSINFO, and updated \HTML.FONTCREATE to call that instead.  And... success!  No more visits to URaid.

            Herb

To view this discussion visit https://groups.google.com/d/msgid/lispcore/1CD56E3F-9FA2-4336-ACEE-4FA1CB1E4456%40gmail.com.

[Nick Briggs]

I thought I'd said this, but I guess I didn't actually hit send:

Compile maiko with -DDTDDEBUG (set in maiko/bin/makefile-darwin.x86_64-sdl )

DEBUGFLAGS = -DDTDDEBUG # -DDEBUG -DOPTRACE

then touch maiko/src/{gcrcell.c,mkcell.c} and redo the maiko build (./bin/makeright sdl) and it should recompile just those two files and give you a new ldesdl

It does a little more on checking over the pages used for allocating named datatypes.

-- Nick

On Jan 20, 2026, at 14:04, Herb Jellinek <jell...@newscenter.com> wrote:

[Nick Briggs]

Ah, excellent.  Have you diagnosed what was wrong with your slug charsetinfo code that broke the DTD space?

[Herb Jellinek]

Actually, I did get your note with directions for compiling Maiko with extra debugging, but wanted to experiment more with Medley before doing that.

My code assumed that the slug charset's widths array block had already been created and just set values into it.  I had found that to be the case in some early experiments.  Ron's code creates that array block it and then sets it.  I'm happy to use that stable API instead of my homebrew hackery.

            Herb