What does it mean when the debugger buttons are all grayed out?

[Matthew Boulter]

Hi all,

TL;DR

Recently all my debug sessions started from callback breakpoints do not allow me to control the execution from that point. (i.e., Into/Over/.../Define are always, all grayed out. Modifying code and saving in the debugger does not take me back to the start of the frame to continue)??

I would like to understand the conditions that essentially 'neuter' the debugger.

I know debugging certain things can be sensitive.
We have some very complex, real-time (constantly updating) UIs for certain operations, that I have learnt to not set breakpoints in (it always ends with a Task Man kill of VAST due to I believe database operations in timers that cause memory inconsistencies, etc etc).

This question is not about that.

Recently I have been adding advanced, extended features to guis in different parts of the system that do not have those background timers and (AFAI believe) don't suffer from those cautionary debug 'things'... and for the most part that has proved true, even in this case, VAST is not "dead in the water" (this doesn't require a hard-reset of the IDE), but the debug session is nothing more than a 'fancy-insitu-inspect' (useful as that is).

The class I have been trying to perfect, hooks into #drawBackgroundRequested and uses the callback's #renderContext to perform specialised rendering of the contents of that boundingBox which is *from* an EwTableCell.

I've never had this issue when my breakpoints, for the same Cell objects, were in handlers for #cellValueRequested, but I'm learning that draw background seems to gives me a lot more flexibility than cell value does. 

For example: my trials completely over-paint the cell bounding box with different sized fonts/colours for text that in essence act like "badges" for the cell's content.

All of that long diatribe said. Really just ...

Why are my debugger frame progression buttons disabled?

Thanks in advance as always.

Regards, Matthew.

[Mariano Martinez Peck]

Hi Matthew,

Debug features like 'stepping' and 'save & modify' are disabled while in the process of 'call-in' (for example, a callback).

Hope this helps, 

--
You received this message because you are subscribed to the Google Groups "VAST Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/8b60f1e8-a797-45c6-9183-799d898195b3n%40googlegroups.com.

--

Mariano Martinez Peck VAST Team Lead Senior Software Engineer  mp...@instantiations.com  @MartinezPeck  /mariano-martinez-peck  instantiations.com

[Richard Sargent]

On Monday, March 18, 2024 at 7:28:39 AM UTC-7 Mariano Martinez Peck wrote:

Hi Matthew,

Debug features like 'stepping' and 'save & modify' are disabled while in the process of 'call-in' (for example, a callback).

Mariano, can you elaborate on the why of that? What are the consequences if they were usable during a call-in?

[Mariano Martinez Peck]

On Mon, Mar 18, 2024 at 1:28 PM Richard Sargent <richard...@gemtalksystems.com> wrote:

On Monday, March 18, 2024 at 7:28:39 AM UTC-7 Mariano Martinez Peck wrote:

Hi Matthew,

Debug features like 'stepping' and 'save & modify' are disabled while in the process of 'call-in' (for example, a callback).

Mariano, can you elaborate on the why of that? What are the consequences if they were usable during a call-in?

Hi Richard,

Good question.

This limitation goes back to IBM times :)  AFAIK, this is how the VAST VM was designed. Technically, a call-in is another instance of the Smalltalk execution engine within the same Smalltalk process, invoked via the VAST API EsSendMessage call. While the debugger only supports one 'root' Smalltalk instance.

If you look at the DbgDebugger and StsDebugger classes, and more specifically the #menuProcess method, you will see that the buttons to go over, into, etc. are only allowed if the DbgDebugSession is resumable. And DbgDebugSession is resumable if the session itself is resumable and if the process is also resumable. See DbgDebugSession >> resumable.

The next piece to look at is the method Process>>reportError: errorString resumable: aBoolean is the one that (eventually) calls the debugger. And there, notice these lines:

System entryCount > 1 ifTrue: [
(Processor forkAt: 7) isNil ifTrue: [Processor activeProcess terminate].
Processor activeProcess markNonResumable.
System startUpClass errorReporter reportError: errorString resumable: false startBP: bp "should never return"
].

If entryCount is > 1 (there are call-ins), then the process is marked as non-resumable.  And this, of course, causes the debugger buttons to be disabled.

However, if you want to debug something from a call-in (for example, a GUI callback), there is a workaround you can do. I will explain this with an example:

1) Set a one-shot breakpoint in OSScintillaEditor >> windowProc: msg with: arg1 with: arg2.

2) This should immediately open a debugger. Of course the buttons are disabled as explained.

3) Go one frame up the stack (in this example OSEventManager windowProc: hwnd msg: msg with: arg1 with: arg2) and select the piece of code that calls the next frame (this should have happened automatically).

4) Right click on the selected text and choose "Debug". This should have opened a second stack on your debugger with a Halt and buttons enabled.

5) Now click "over", "into", etc. as desired and debug your actual callback with all buttons enabled :)

For your convenience, I have attached pictures of the 5 steps.

So what this workaround allows you to do is to execute the callback code, but instead from a call-in, from normal execution.

Best regards,

[mar...@feldtmann.online]

I've been asking this for myself over lots of years ... :-)