Idea for a new function: hb_HrbDoFromBuf

[Eric Lendvai]

Thank you again  Przemek for enhancing hb_CompileFromBuf(). The recent improvements make embedded compilation far more usable.

I would like to propose an additional helper function that would complement this work and simplify debugger integration in dynamic-code scenarios.

Proposal: hb_HrbDoFromBuf( <cSourceBuffer>, <cVirtualFileName>, <pHrb>, … )

The motivation is the following:

When an HRB is produced via HB_CompileFromBuf(), we can now specify a “virtual” source file name that is used for debug information. This is excellent for breakpoints triggered by ALTD(), because the debugger will attempt to load the corresponding source file.

However, since the source file does not actually exist on disk unless we explicitly write it, the debugger cannot show the source unless the embedding application manually writes the content to that filename. This undermines the performance benefit of HB_CompileFromBuf(), whose purpose is to avoid filesystem I/O.

To address this, I would like to propose a new function, tentatively named:

hb_HrbDoFromBuf( cSource, cFileName, pHrb, [args…] )

Intended behavior

At runtime, before invoking the debugger (e.g., upon ALTD()),
the function would write cSource to cFileName only if needed.

Under normal execution (no breakpoints), the function behaves identically to hb_HrbDo(), with no additional file writes.

This keeps execution fast and avoids unnecessary disk output, while still ensuring that the debugger can display the correct source code when breakpoints occur.

Why this is useful

Today, any application using HB_CompileFromBuf() and the debugger must:

1. Compile from memory,
2. Manually write the same source code to disk,
3. Load and execute the HRB.

The proposed function collapses steps (2) and (3) into a safer, more efficient API that is aligned with the intent of in-memory compilation. It also ensures that the debugger will always find the correct file, without requiring every embedding application to re-implement this logic individually.

Thanks for your consideration, Eric

[Aleksander Czajczynski]

A note that the debugger is kind of replaceable module, so that you
should also plan/expose an API to get the source. Should it be like
ProcFile( <n> ) ... hb_ProcFileBuffer( <n> ) -> <cBuffer>/<hBuffer> or
VF filter driver that overlays the filesystem in debug versions with
known ".prg" files, so they are just accessible with any other function.

Best regards, Aleksander

[Lailton Fernando Mariano]

It is an excellent idea, and very useful. 

Will be amazing if you can implement that.

Thanks

[Antonio Linares]

Gemini 3 answer:

Based on the src/compiler/hbcmplib.c source code and the existing Harbour architecture, here is a proposal for the hb_HrbDoFromBuf implementation.

This solution addresses your need to lazily write the source file to disk only when the debugger is active, preserving the performance benefits of in-memory compilation (HB_CompileFromBuf) while ensuring ALTD() and breakpoints work correctly.

Proposed Implementation

You can implement hb_HrbDoFromBuf as a wrapper that checks the debugger state before executing the HRB.

/* * Proposed helper function: hb_HrbDoFromBuf
 * * Arguments:
 * cSource   - The original source code buffer
 * cFileName - The virtual filename (used in debug info)
 * pHrb      - The loaded HRB pointer (or buffer, depending on usage)
 * ...       - Arguments to be passed to the HRB
 */

#include "hbapi.h"
#include "hbvm.h"
#include "hbapifs.h"

HB_FUNC( HB_HRBDOFROMBUF )
{
   const char * cSource = hb_parc( 1 );
   const char * cFileName = hb_parc( 2 );
   PHB_HRB pHrb = ( PHB_HRB ) hb_parptr( 3 ); /* Assuming passed as pointer */
   
   /* * 1. Lazy Write: Check if the debugger is active.
    * We only incur the I/O cost if debugging is actually enabled.
    * 'hb_vmIsDebug()' is the internal check (or checks if pDebugInfo is present).
    */
   if( cSource && cFileName && hb_vmIsDebug() )
   {
      /* Check if file already exists to avoid overwriting (optional)
         or just overwrite to ensure sync with memory buffer */
      if( ! hb_fsFileExists( cFileName ) )
      {
         HB_FILEHANDLE fh = hb_fsCreate( cFileName, HB_FO_WRITE );
         if( fh != HB_FILEHANDLE_INVALID )
         {
             hb_fsWriteLarge( fh, cSource, ( HB_SIZE ) hb_parclen( 1 ) );
             hb_fsClose( fh );
         }
      }
   }

   /* * 2. Execute the HRB
    * We need to pass the remaining arguments (4 to PCount) to the HRB.
    * Since we are in a C-level HB_FUNC, we can shift parameters
    * or use hb_hrbRun() logic.
    */
   
   /* * Note: Implementation of forwarding variable arguments
    * depends on how hb_HrbDo/hb_hrbRun is exposed in your version.
    * A simple approach is to rely on hb_hrbRun if it handles the stack.
    */
   if( pHrb )
   {
      /* Push the HRB codeblock/pointer to stack and execute */
      /* This is pseudo-code for the execution part, as it mimics hb_hrbDo */
      hb_hrbRun( pHrb );
   }
   
   /* Return value handling */
}

Key Components Explained

1. hb_vmIsDebug(): This is the critical check. By calling this (or checking hb_stackBase()->pDebugInfo if the function isn't public), you ensure the file is only written when the debugger is attached and listening. In production (Release builds) or when the debugger is off, this returns false, skipping the hb_fs* calls completely.

2. hb_fsCreate / hb_fsWriteLarge: These are the native Harbour file system APIs. Using hb_fsWriteLarge ensures safe handling of large source files (bigger than 32KB/64KB).

3. Virtual Filename: The cFileName passed here must match the one provided to HB_CompileFromBuf( ..., cVirtualFileName ). This links the in-memory debug symbols to the physical file we just created.

Alternative: Virtual File System (VF) API

You also mentioned an API to "get the source" (like hb_ProcFileBuffer). This is architecturally cleaner because it avoids disk I/O entirely.

Instead of writing to disk, you could implement a Debugger Source Provider:

• Proposal: hb_dbgSetSourceProvider( HB_BOOL (*callback)( const char * cName, char ** cBuf, HB_SIZE * nLen ) )

• Mechanism: When the debugger hits an ALTD() and needs to display virtual_code.prg, it calls this callback. Your application checks its in-memory map, finds the buffer for virtual_code.prg, and returns it directly to the debugger.

• Benefit: This makes the "Virtual File" truly virtual. The debugger reads directly from RAM, eliminating the need for hb_HrbDoFromBuf to write temporary files.

Summary

The hb_HrbDoFromBuf function effectively patches the gap between HB_CompileFromBuf and the file-based debugger. By wrapping the file write in a hb_vmIsDebug() check, you satisfy the requirement for zero-overhead in normal execution while ensuring seamless debugging when needed.

[Antonio Linares]

Deepwiki analysis:

Thank you for this detailed proposal. Let me analyze your suggestion in the context of the existing Harbour architecture.

## Current Architecture

### Compilation from Buffer
The current `HB_COMPILEFROMBUF()` function accepts source code as a string parameter and compiles it to an HRB buffer: [1](#0-0)

When compiling from a buffer, the compiler uses `"{SOURCE}"` as a default virtual filename, which gets passed through to the internal compilation process: [2](#0-1)

The virtual filename is used to create the preprocessor buffer and becomes part of the module's debug information: [3](#0-2)

### HRB Execution
The `hb_HrbDo()` function executes loaded HRB modules with parameters: [4](#0-3)

### Debugger Source Loading
When the debugger needs to show source code at a breakpoint, it attempts to load the file from disk: [5](#0-4)

The debugger searches for files using `LocatePrgPath()` if they're not found in the current directory: [6](#0-5)

### Module Name Tracking
Module names are stored via `hb_vmRegisterSymbols()` during HRB loading and can be retrieved for stack trace information: [7](#0-6)

The `ProcFile()` function retrieves the module name for a given stack level: [8](#0-7)

## Analysis of Your Proposal

Your proposal of `hb_HrbDoFromBuf()` addresses a real gap in the current architecture. However, the feedback about exposing an API to get the source is architecturally more sound for several reasons:

### Why an API Approach is Better

1. **Separation of Concerns**: The debugger is a replaceable module. Writing files to disk couples the HRB execution layer with filesystem semantics, which violates this design principle.

2. **Virtual File System Support**: Harbour already has infrastructure for virtual file I/O (see `src/rtl/vfile.c`). This suggests the codebase is designed to support abstraction over physical filesystem operations.

3. **Multiple Debugger Scenarios**: Different debuggers or debugging contexts might want to access source code differently (display in GUI, send over network, etc.). An API provides flexibility.

### Recommended Approach

Instead of `hb_HrbDoFromBuf()`, consider implementing:

**Option 1: Source Buffer API**
- `hb_ProcFileBuffer(n)` → returns source buffer for stack level `n`
- This mirrors the existing `ProcFile(n)` pattern: [9](#0-8)

**Option 2: Virtual File Filter Driver**
- Register compiled buffers with a VF handler
- The debugger would access sources through standard file I/O functions
- The VF layer would intercept requests for known virtual filenames and return the in-memory buffer

The VF approach is more powerful because:
- It requires no debugger changes
- It works with any code that reads source files (debuggers, code analyzers, etc.)
- It's consistent with Harbour's existing virtual I/O infrastructure

### Implementation Considerations

To support either approach, you would need to:

1. **Store source buffers**: Modify `hb_hrbLoad()` to optionally store the source buffer alongside the HRB: [10](#0-9)

2. **Associate buffers with module names**: Link the source buffer to the module name stored during symbol registration

3. **Expose retrieval mechanism**: Either through a direct API or VF handler registration

## Notes

- Your motivation is valid: the current architecture requires applications to duplicate the source-to-disk write that `HB_CompileFromBuf()` was designed to avoid.

- The VF filter driver approach is architecturally cleaner and more flexible than conditionally writing files in `hb_HrbDoFromBuf()`.

- Consider that source buffers need lifecycle management—they should be freed when the HRB is unloaded: [11](#0-10)

- The debugger's file loading mechanism would need minimal changes to first check for virtual sources before attempting physical file access.

A hybrid solution might be most practical: expose `hb_ProcFileBuffer()` as the primary API, with optional VF registration for backward compatibility with existing debugger code.

[Antonio Linares]

FUNCTION hb_HrbDoFromBuf( cSource, cVirtualFileName, xHrb, ... )
   
   IF hb_vmIsDebug() .AND. !hb_FileExists( cVirtualFileName )
      hb_MemoWrit( cVirtualFileName, cSource )
   ENDIF

   RETURN hb_hrbRun( xHrb, ... )

[Eric Lendvai]

Hello Antonio,

My idea was to not write out the source file, unless the debugger was called.
hb_CompileFromBuf does not use I/O, so why should we, unless needed by the debugger. By the way I use the Harbour VSCode debugger.

And in regards to the parameters, I can confirm the following example that is aware of the latest change in  hb_CompileFromBuf   :

l_oHrb := HB_CompileFromBuf( ;

                l_cSourceCodeFull, ;

                .T.,;

                "mingw32", ;      // Provide the compiler info that the app was built with, informational only

                "-b", ;                // generate debug info

                "-n", ;

                "-I" + l_cHBheaders1, ;

                "-I" + l_cHBheaders2, ;

                "-I" + l_cHBheaders3, ;

                "-I" + hb_GetEnv( "HB_INCLUDE" ),;

                l_cSourceFilePath)   // pseudo filename for debugger

I will be pushing changes to my Local Harbour Sandbox Repo in the next few days so you could see the complete example, including how to catch compilation and runtime errors.

Thanks, Eric

Example for detecting compilation errors in Local Harbour Sandbox

--
You received this message because you are subscribed to the Google Groups "Harbour Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-deve...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/harbour-devel/e97f9da4-8bdd-4e2b-be98-669043aa0199n%40googlegroups.com.