Test coverage

[htk...@bosch.com]

Hello, are there test coverage tools available for VAST apart from McCoverage? As I understand McCoverage only provides method coverage. I am in need for something more granular such as line or branche coverage. Is anyone using another test coverage tool? thank you

[Richard Sargent]

I did some brute force experimental work for this. Install a breakpoint at every possible point in all the methods. The breakpoints do nothing.
Run the test suite.
Examine the breakpoints to identify which have never been visited.
Unfortunately, it substantially increases the run time of a test suite. But, it's also something that doesn't need to be run every time.

Depending on how you want to work, you can remove the breakpoints with non-zero hit counts, revise tests. run the test suite or a subset to exercise the changes, and repeat. The next full coverage run then theoretically will show everything being covered. (It probably won't, but ...)

[Richard Sargent]

An optional variation would probably speed things up:

Have the breakpoint trigger expression remove the breakpoint before answering false. As the test suite runs, there would be fewer and fewer breakpoints being evaluated and so run faster. I wouldn't start with this until you have run it once the simple way and analysed the number of times visited breakpoints were visited. If the majority are only visited a small number of times, the complexity of removing visited breakpoints probably isn't going to yield a significant performance improvement.

[Simon Franz]

Hi Richard,
could you perhaps share the snippet that makes a breakpoint "remove itself"?

I think that would be significantly more efficient than having to search for it again afterward. Maybe it helps someone anytime later.

Best regards

[Richard Sargent]

I don't have a snippet. I suspect it would require some stack spelunking to find it.

[Richard Sargent]

And to reiterate, optimizing by removing the hit breakpoint would only make sense after you determine that it is hit often enough to have a performance impact on running the test suite.

[Mariano Martinez Peck]

Hi everyone,

Not sure if this is exactly what you are looking for but I used below script to add/remove breakpoints:

| methodsToBreak breakpoints  |

methodsToBreak := OrderedCollection new.
breakpoints := OrderedCollection new.
methodsToBreak
    add: (Object >> #deprecated:);
    add: (Object >> #deprecated:in:).
    "add...."

methodsToBreak do: [:aMethod |
    | breakpoint |
    breakpoint := System image breakpointManager
                    createBreakpointFor: aMethod
                    nearestPosition: 1
                    lineDelimiter: nil
                    ifExists: [:existingBP | self error: 'Breakpoint already exists' ].
    breakpoint ifNil: [ self error: 'Cannot install breakpoint' ].
    breakpoint activate.
    breakpoints add: breakpoint.
].

breakpoints do: [:each | each clear ]

--
You received this message because you are subscribed to the Google Groups "VAST Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/va-smalltalk/2ba0d8c5-d552-40e1-90e3-3bf2e31fc9c0n%40googlegroups.com.

--

Mariano Martinez Peck VAST Lead Consultant Senior Software Engineer  mp...@instantiations.com  @MartinezPeck  /mariano-martinez-peck  instantiations.com

[Richard Sargent]

On Thu, Jul 10, 2025 at 10:13 AM 'Mariano Martinez Peck' via VAST Community Forum <va-sma...@googlegroups.com> wrote:

Hi everyone,

Not sure if this is exactly what you are looking for but I used below script to add/remove breakpoints:

The challenge is gaining access to the breakpoint itself while evaluating the trigger expression (and/or the class, method, IP, etc to be able to get the BP from the BP Manager).

 

You received this message because you are subscribed to a topic in the Google Groups "VAST Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/va-smalltalk/G9rMhApGmWc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to va-smalltalk...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/va-smalltalk/CAOUkibHz7NnOHQL5EX%2B72reicWe4LACWV_OCikUpjiH3teDWPQ%40mail.gmail.com.

[htk...@bosch.com]

Hello,

thanks for the suggestions but I would like to remind you that my question is if there are already tools available or if somebody has already found a way that he uses successfully. You can of course go on with the ongoing discussion but this is not what I am asking for.

[Marcus Wagner]

It is on the edge of the realisability in Smalltalk to demand s.th. more granular. So: no, it is not likely to find s.th. here.

Interesting, as the question somehow feels personally a little bit strange for me... concerning the granularity, out of standpoint and sight of the fundamental Smalltalk architecture. 

From the outside of the VM: the next (and last) available atomar level at runtime would be the byte codes, targetable by break points, as already discussed - line or branches are not reflected at that level. 

Not yet covered here is the potential of the parse tree (statements, expressions, blocks, annotations), naturally because of its normal intentionally absence at runtime (as a packaging consequence).

Outside approaches are based on proxies (the design pattern). McCoverageFramework follows this approach, seems likely to be extendable  easily.

From the inside of the VM: this was kept internal, except what is published about the VM API. This area is touched by ENVY/Stats, memory allocation and runtime behaviour and processes. One may also hook on primitives, but I do not know anything utilising this (in Smalltalk) similar to DLL hooks elsewhere (example the processExplorer on operating system level).

I admit, I also do not consider further possible areas following out of the JIT LLVM aspects which have been applied since VAST 9 or 10. Inside approaches are even thinkable to be based on hardware, not limited by Smalltalk's architecture. I remember two interprocess hardware assited monitoring approaches to inspect and collect the process register (of another job/process) and to analyse this traces offline. was named PBHIST (80ies) and SPY (60/70ies). Side aspect: the more granular this runtime inspection was, the more biased where the results...

Theoretically but coming more and more in sight is simulation of this system which is definitely and practically already done every day: 

the ST VM being run in a VM on a server/in the cloud.

-

M

[Marcus Wagner]

There is a hidden, potentially severe limit concerning break points: 

DbgBreakpointManager>>#maxBreakpoints -> 16rFFFF.

That might hurt in a tool based approach if break points are setup brute force.

-

M

[Marcus Wagner]

A hint how to find that: a break point is signalled as ExHalt with an identifying break point number ($NLS$ Breakpoint #%1)

-

M

[Richard Sargent]

"...  or if somebody has already found a way that he uses successfully."

I think my discussion of having successfully used breakpoints to get step-point tracing met that criterion.

[Richard Sargent]

On Sunday, July 13, 2025 at 9:46:18 AM UTC-7 Marcus Wagner wrote:

There is a hidden, potentially severe limit concerning break points: 

DbgBreakpointManager>>#maxBreakpoints -> 16rFFFF.

Thanks. I did not know that. I guess because we hadn't hit that limit.

Question for Instantiations: While I assume 16 bits was a reasonable limitation for 32-bit VMs, is there anything that forces the same limitation in 64-bit VMs?

[Marcus Wagner]

Some insights about break points (technically): when running a (special) byte code, the value gives the identy (which bp has been reached).

The VM enters the ST image via the core registry, where DbgBreakpoint>>breakpointEncountered:  is registered (Core registry at: 55). 

So the answer to the question is  the op code format is the limitation - any other representation requires a change, a new byte code format.

Even if only this instruction is envolved, I leave it open to you to estimate the likeliness of such a change (byte codes are abound - all images, all repositories, since ages).

-

M

[Henry Johansen]

Marcus is correct - not without changing the bytecode format.
To execute a break bytecode, the VM calls the breakpoint handler with an ID constructed from the following two bytes as a 2-byte tagged SmallInteger - meaning, the maximum is really 16r7FFF - not 16rFFFF.

I'm not quite sure what happens if you try to add than 32K breakpoints - most likely, they will registered in the Manager, but never be called,
as in the bytecode the following bytes will wrap around to zero, and thus end up never triggering breakpoints with ID  mod 32K.

We'll add a case to change the documented limit, but as far as a new bytecode format - that's not likely to happen anytime soon, but I'll write it down as a thing to consider if we do.

Cheers,

Henry Johansen VAST Consultant Senior Software Engineer  hjoh...@instantiations.com  instantiations.com

--

You received this message because you are subscribed to the Google Groups "VAST Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/va-smalltalk/5786f71b-fa46-4526-b203-429a70ec7799n%40googlegroups.com.

[Richard Sargent]

Interesting information, Henry.

I found the following, which looks like there might be three different breakpoint byte codes. 8-bit, 16, bit, and possibly 32-bit.

_PRAGMA_BytecodeValues
<snip>
(name: BCbreakW isConstant: true valueExpression: '155')
(name: BCbreakWP isConstant: true valueExpression: '156')
<snip>
(name: BCbreakB isConstant: true valueExpression: '192')

Admittedly, that may be wishful thinking on my part.

[Marcus Wagner]

Hi Richard,

due to the given (propietary) VM architecture many opcodes appear in 3 styles, suffix B, W or WP. 

This is independent of the architecture's bitness (32 or 64).

B, W or WP denote the capacity of the argument (byte, word or -I guess- word pointer - address of an object), that to ensure that the value of the argument can be allocated within this instruction.

This is a basic design pattern to offer performance, well known in the *near* and *far* pointer architectures, to give an example.

In Smalltalk, traditionally that pattern is applied to offsets (to instructions in jumps or to slots, literal ones - static in the method byte codes - or dynamic - relativily to the stack frame) or to direct arguments like characters and numbers and object references.

As a break point opcode (temporarily) replaces the original op code, so that the breakpoint can become active.

To achieve this replacement, three different versions of break op codes, B, W, WP,  have to be reserved - though doing always the same thing, a break.

In this situation, the effective argument of the break point itself (the break point number) cannot follow recursivly this style:

you cannot seriously demand that a break point named 10000 cannot replace an op code with a byte argument (e.g. the argument capacity of the break point op code must be able to replace any target). 

And remember, in the current design, break point numbers are created automatically.

So a break point op code must be generic in this sense, as the break point number will be created on demand and the break point op code must accept any number, independent of which target op code is to be replaced.

I also knew another architecture  precautiously reserving NOPs at all potential target break point places. In this environment a single format of a break point op code replaces a NOP to become active. There the break point always has to be of the size of a NOP.

However this approach follows the traditional compilation work flow (providing code with or without debug information). 

Not to say that such a code gets larger (additional NOP per statement) and slower (even a NOP costs execution time) just to be prepared to be debugged.

There were good reasons not to follow this approach in an interactive environment.

Summary:

As a (fictitious) break point numbered 266372627 must be able to replace the tiniest possible target opcode (which intentionally was carefully designed to become so small to be performant) [in particular in the discussed scenario, when automatically generating break points at all possible locations to be able to measure the coverage].

So this historically required the choice of a limit - obviously that one could not have been extreme in both directions - e.g. maximum 255 break points as byte code or as in my extreme example, a WP numbered one). Thus resulted in the existing compromise.

Even an indirection (a reference to a place actually holding the indentity/number/name of a break point) would not solve the problem, as then instead one has to care about space of the indirect number/names and as the number of references can grow and become large, it would end up in the same dilemma.

Instead of adopting op codes another idea could be followed: what if a bit is spent for every instruction supported by the vm, not necessarily in the passive generated code but only at execution time, after being loaded and before being run. 

Any execution will turn on this bit. A way of its examination has to be designed and provided anew.

if s.b. considers this to be science fiction, think about the existance of readonly bit of all objects, also nonstandard but present and very useful. Might be adopted for this purpose.

The result be a new feature as add on, which would not endanger existing material, replacing the existing development vm, as choice, when needed.

Can potentially be arranged in the existing ENVY/Stats, EsbSampler.

-

M 

PS to all insiders: as the VM is propietary and being an outside, I hope I did not infringe internals - and please neglect my inaccuracies. 

[Henry Johansen]

There are indeed three different bytecodes, but 
- B means the argument is of byte (8 bits) length, while W means the argument is of word length (16 bits).
P means there's a padding byte before the argument (for aligned access).

The code responsible for generating a breakpoint bytecode, is in CodeStream >> breakpoint:.
there you can see we currently only emit the W variant(s) (WP is emitted in nextWordCodePut: if misaligned, all P bytecode variants are 1 higher).

A short experiment of what will happen when we compile a method with the 16r8000th breakpoint:

|cs| cs := CodeStream new. cs instVarNamed: 'collection' put: (EsByteCodeArray new: 100). cs breakpoint: 16r8000 reveals the code generator will generate an error - not with the useful 'Breakpoint out of range' error - but with  value out of range, since we are nextWordPut:'ing a value higher than 16rFFFF. Cheers,

Henry Johansen VAST Consultant Senior Software Engineer  hjoh...@instantiations.com  instantiations.com

To view this discussion visit https://groups.google.com/d/msgid/va-smalltalk/b3c82034-54f4-4179-99eb-2b15ce691d2bn%40googlegroups.com.

[Marcus Wagner]

Hi Henry

thanks for your information about B, W and WP forms, devoted to the capacity of the argument but misunderstood by me under a different sense.

So I will refrain from distributing any -guess- marked nonsense from me here - and I hope that you will forgive me for that.

-

M

[Richard Sargent]

On Wednesday, July 23, 2025 at 3:57:20 AM UTC-7 Marcus Wagner wrote:

Hi Richard,

due to the given (propietary) VM architecture many opcodes appear in 3 styles, suffix B, W or WP. 

This is independent of the architecture's bitness (32 or 64).

B, W or WP denote the capacity of the argument (byte, word or -I guess- word pointer - address of an object), that to ensure that the value of the argument can be allocated within this instruction.

This is a basic design pattern to offer performance, well known in the *near* and *far* pointer architectures, to give an example.

In Smalltalk, traditionally that pattern is applied to offsets (to instructions in jumps or to slots, literal ones - static in the method byte codes - or dynamic - relativily to the stack frame) or to direct arguments like characters and numbers and object references.

As a break point opcode (temporarily) replaces the original op code, so that the breakpoint can become active.

Ah! Thanks for reminding me of that breakpointing technique. I've been working a lot in VisualWorks lately which compiles a new method when you add a breakpoint. 

[Richard Sargent]

On Wednesday, July 23, 2025 at 8:49:00 AM UTC-7 Henry Johansen wrote:

There are indeed three different bytecodes, but 
- B means the argument is of byte (8 bits) length, while W means the argument is of word length (16 bits).
P means there's a padding byte before the argument (for aligned access).

The code responsible for generating a breakpoint bytecode, is in CodeStream >> breakpoint:.
there you can see we currently only emit the W variant(s) (WP is emitted in nextWordCodePut: if misaligned, all P bytecode variants are 1 higher).

A short experiment of what will happen when we compile a method with the 16r8000th breakpoint:

|cs|
cs := CodeStream new.
cs instVarNamed: 'collection' put: (EsByteCodeArray new: 100).
cs breakpoint: 16r8000

reveals the code generator will generate an error - not with the useful 'Breakpoint out of range' error - but with 
value out of range, since we are nextWordPut:'ing a value higher than 16rFFFF.

That's a logic error in CodeStream>>#breakpoint:.
It's not *we* who are putting a value greater than 16rFFFF, but the result of makeInteger: is.

CodeStream>>#breakpoint: should check for < 16r8000 (and should also check for >= 0).

[Marcus Wagner]

Summary to fix this, two methods should look like:

DbgBreakpointManager>>#maxBreakpoints

maxBreakpoints
"Answer the number of breakpoints currently supported by the breakpoints implementation."
^16r7FFF

and CodeStream>>#breakpoint:

breakpoint: breakpointNumber
"Set the breakpoint identified by breakpointNumber to be run.  Report an error if the
breakpoint number cannot be represented by an unsigned word."
breakpointNumber isNil ifTrue: [^self].
(breakpointNumber between: 0 and: 16r7FFF) ifFalse: [^self fatalError: 'Breakpoint out of range'].
self
nextWordCodePut: BCbreakW;
nextWordPut: (CodeStream makeSmallInteger: breakpointNumber)

However, in the case that a break point number 0 is also invalid, then this should be (debugger starts always with #1)

breakpoint: breakpointNumber
"Set the breakpoint identified by breakpointNumber to be run.  Report an error if the
breakpoint number cannot be represented by an unsigned word."
breakpointNumber isNil ifTrue: [^self].
(breakpointNumber between: 1 and: 16r7FFF) ifFalse: [^self fatalError: 'Breakpoint out of range'].
self
nextWordCodePut: BCbreakW;
nextWordPut: (CodeStream makeSmallInteger: breakpointNumber)

[Marcus Wagner]

The last question is also t.b. clarified. There is a special breakpoint number zero, it is reserved as the null (non firing) breakpoint. Without knowing its details, the CodeStream>breakPoint:

check fix now depends on whether this special (non firing) breakpoint might be set explicitely or serves only internally without being allowed to be set externally.

-

M