hbmk2 - Undefined reference link error building pCode DLL

[Leandro Damasio - 2D Info]

Hi,

 

I'm trying to build a (not-self-contained) pCode DLL, but I get undefined reference link error to any standard harbour function called from it.

Maybe hbmk2 should parametrize the linker to mount the dll despite the undefined references, because they will be solved just in runtime on the host VM.

 

The DLL content is:

<code>

function test()
? "test"
return "test"

</code>

 

The code above causes undefined reference link error message about QOUT function. 

Below the hbmk2 output.

 

Regards,

Leandro

 

 

D:\2d\teste\speed>hbmk2 teste -hbdyn -lhbmaindllp
hbmk2: Processando opções do ambiente: -compiler=mingw
hbmk2: Processando arquivo de configuração: D:\HARBOUR\BIN\hbmk.cfg
Harbour 2.1.0beta1 (Rev. 14702)
Copyright (c) 1999-2010, http://harbour-project.org/
Compiling 'teste.prg'...
Lines 3, Functions/Procedures 1
Generating C source output to 'C:\Users\Leo\AppData\Local\Temp\hbmk_isc7be.dir\t
este.c'... Done.
C:\Users\Leo\AppData\Local\Temp\hbmk_isc7be.dir\teste.o:teste.c:(.data+0x38): un
defined reference to `HB_FUN_QOUT'
collect2: ld returned 1 exit status
hbmk2: Erro: Executando comando de linkagem da biblioteca dinâmica. 1
gcc.exe -shared -o teste.dll C:\Users\Leo\AppData\Local\Temp\hbmk_isc7be.dir\tes
te.o  -Wl,--start-group -lhbmaindllp -lkernel32 -luser32 -lgdi32 -ladvapi32 -lws
2_32 -lwinspool -lcomctl32 -lcomdlg32 -lshell32 -luuid -lole32 -loleaut32 -lmpr
-lwinmm -lmapi32 -limm32 -lmsimg32 -lwininet  -Wl,--end-group  -Ld:/harbour/lib/
win/mingw

 

D:\2d\teste\speed>

 

[Viktor Szakáts]

Hi Leandro,

You currently have to add '-nohblib-' option to make it work.

Quite edgy place of hbmk2, and I'm not yet 100% happy with this
recent change, so keep an eye on ChangeLog for any changes here.

Viktor

[Leandro Damasio - 2D Info]

> Hi Leandro,
>
> You currently have to add '-nohblib-' option to make it work.
>
> Quite edgy place of hbmk2, and I'm not yet 100% happy with this
> recent change, so keep an eye on ChangeLog for any changes here.
>
> Viktor

Hi Viktor,

Forgive me for taking so long to get back to this subject. I appreciate your
help very much. Thank you.

I tried hbmk2 mydll.prg -hbdyn -lhbmaindllp -nohblib-, but the linker then
acuses double definition of hb_vmProcessSymbols and hb_vmExecute by hvmall.o
from libhbvm.a because both were first defined at dllpcode.o from
libhbmaindllp.a

By the way, could you explain the meaning of the minus [-] after the
parameter -nohblib please?

Regards,
Leandro

[Viktor Szakáts]

Hi,

> Forgive me for taking so long to get back to this subject. I appreciate your help very much. Thank you.
>
> I tried hbmk2 mydll.prg -hbdyn -lhbmaindllp -nohblib-, but the linker then acuses double definition of hb_vmProcessSymbols and hb_vmExecute by hvmall.o from libhbvm.a because both were first defined at dllpcode.o from libhbmaindllp.a

I'm at the moment (well still) completely confused about whole
dynamic lib issue. I reread Przemek's detailed instructions form
2010.02.09, and those instructions don't work (anymore?).

To create .dlls like I did for contribs, use:
hbmk2 -hbdyn -shared -lhbmaindllp <lib>.hbp

This links, and also works in many situations, but not in all,
plus it's only one possible scenario, so it's far from being
finalized yet.

I hope Przemek will drop by and check on current state of
details.

[ Oh and I've only dealt with Windows yet... others will have
to be implemented and tested as well. ]

> By the way, could you explain the meaning of the minus [-] after the parameter -nohblib please?

It's negating the meaning of the option.

Viktor

[Leandro Damasio - 2D Info]

Hi Viktor,

> I'm at the moment (well still) completely confused about whole
> dynamic lib issue. I reread Przemek's detailed instructions form
> 2010.02.09, and those instructions don't work (anymore?).
>
> To create .dlls like I did for contribs, use:
> hbmk2 -hbdyn -shared -lhbmaindllp <lib>.hbp
>
> This links, and also works in many situations, but not in all,
> plus it's only one possible scenario, so it's far from being
> finalized yet.

I didn't understand what is the scenario that uses above hbmk2
parametrization, but apparently it builds a harbour pcode DLL to be used
with a harbour dll (dynamically loaded hvm), most probably by a non-harbour
application, did I get it right?

I have in mind a apparently lesser complex scenario, that would be a harbour
executable loading symbols from a harbour pcode DLL. Of course it doesn't
necessarily means that this is easier to implement than the former scenario,
but I believe this would be the most basic pcode DLL use.

> I hope Przemek will drop by and check on current state of
> details.

Yes, Premek's help would be essencial here. I believe he knows better
everything on this subject. :)

I also noticed that using "hbmk2 mydll.prg -hbdyn -lhbmaindllp" the linking
fails by not to resolve the references to functions out of the dll. But once
there is no HVM in pCode DLLs, what would be the use on resolving all these
references in link-time? LD.exe has an option --allow-shlib-undefined that
seems to fit this purpose of building a shared library
without with undefined references and I believe other linkers support it
too, so maybe when building pcode DLLs hbmk2 by default parametrizes the
linker to mount the pCode DLL without to resolve all the references. What do
you think?

Regards,
Leandro

[Viktor Szakáts]

Hi Leandro,

>> I'm at the moment (well still) completely confused about whole
>> dynamic lib issue. I reread Przemek's detailed instructions form
>> 2010.02.09, and those instructions don't work (anymore?).
>>
>> To create .dlls like I did for contribs, use:
>> hbmk2 -hbdyn -shared -lhbmaindllp <lib>.hbp
>>
>> This links, and also works in many situations, but not in all,
>> plus it's only one possible scenario, so it's far from being
>> finalized yet.
>
> I didn't understand what is the scenario that uses above hbmk2 parametrization, but apparently it builds a harbour pcode DLL to be used with a harbour dll (dynamically loaded hvm), most probably by a non-harbour application, did I get it right?

No, it's used to move prg/c code into a .dll and use that
dll at link time. Meaning the final app will automatically
require and load <lib>.dll and harbour*.dll when launched.

> I have in mind a apparently lesser complex scenario, that would be a harbour executable loading symbols from a harbour pcode DLL. Of course it doesn't necessarily means that this is easier to implement than the former scenario, but I believe this would be the most basic pcode DLL use.

As I mentioned I'm confused about these, since I cannot
adapt the February e-mail of Przemek to current state,
and those examples didn't work for me (at least the first
three cases) for some reason, which may simply be because of
changes in hbmk2/Harbour since then (or maybe typos or just
plain misunderstanding from my side, I didn't test those
suggestions back then).

>> I hope Przemek will drop by and check on current state of
>> details.
>
> Yes, Premek's help would be essencial here. I believe he knows better everything on this subject. :)
>
> I also noticed that using "hbmk2 mydll.prg -hbdyn -lhbmaindllp" the linking fails by not to resolve the references to functions out of the dll. But once there is no HVM in pCode DLLs, what would be the use on resolving all these references in link-time? LD.exe has an option --allow-shlib-undefined that seems to fit this purpose of building a shared library
> without with undefined references and I believe other linkers support it too, so maybe when building pcode DLLs hbmk2 by default parametrizes the linker to mount the pCode DLL without to resolve all the references. What do you think?

I think DYNAMIC keyword is needed for that. IMO no linker
trick would help, but I pass the word to Przemek.

Now we have make.hbs capable to build .dlls out of contribs,
where the build parameters are configured in contrib/hbpost.hbm,
so we can finally experiment with real examples.

BTW at current state _all_ contribs are successfully built
on Windows when using HB_BUILD_CONTRIB_DYN=yes option. And
not all, but some of these can be successfully used (not all
of them though).
(* the only contribs I haven't tested is OpenSSL dependent
ones). Even hbqtcore.dll and friends are built now.

See more notes in recent ChangeLogs about cmdline examples
and error reports.

Viktor

[Leandro Damasio - 2D Info]

Hi Viktor,

>> I didn't understand what is the scenario that uses above hbmk2
>> parametrization, but apparently it builds a harbour pcode DLL to be used
>> with a harbour dll (dynamically loaded hvm), most probably by a
>> non-harbour application, did I get it right?
>
> No, it's used to move prg/c code into a .dll and use that
> dll at link time. Meaning the final app will automatically
> require and load <lib>.dll and harbour*.dll when launched.
>

Thank you for explaining. So this allows to reduce the executable size to
nothing but the application prg "core", because HVM will be loaded from
harbour*.dll and "everything else" from <lib[s]>.dll, right?
BTW, does harbour*.dll contain only HVM or also RTL, RDDs, etc?
This scenario is interesting but how does the Harbour licence deal with this
case where one has to distribute harbour*.dll with proprietary application?

>> I have in mind a apparently lesser complex scenario, that would be a
>> harbour executable loading symbols from a harbour pcode DLL. Of course it
>> doesn't necessarily means that this is easier to implement than the
>> former scenario, but I believe this would be the most basic pcode DLL
>> use.
>
> As I mentioned I'm confused about these, since I cannot
> adapt the February e-mail of Przemek to current state,
> and those examples didn't work for me (at least the first
> three cases) for some reason, which may simply be because of
> changes in hbmk2/Harbour since then (or maybe typos or just
> plain misunderstanding from my side, I didn't test those
> suggestions back then).
>

(...)

> I think DYNAMIC keyword is needed for that. IMO no linker
> trick would help, but I pass the word to Przemek.

Ok, lets wait Przemek can join this thread again. I also reread his
instructions from february, rebuilt harbour with -DHB_DYNAMIC cFlag, and
used "hbmk2
mydll.prg -nohblib -strip -hbdyn -lhbmaindllp -cflag=-DHB_DYNLIB -n -w -es2"
but it didn't work. I get undefined error to hb_vmProcAddress link error
building the pCode dll.

For now I just need to implement the 4th case mentioned by Przemek then,
that would be "Main application is linked in static mode with harbour static
libraries but PCODE dll is not linked statically at link time but rather
loaded dynamically by HB_LIBLOAD()", so I can compare the performance of
pCode DLLs versus HRB modules.

> Now we have make.hbs capable to build .dlls out of contribs,
> where the build parameters are configured in contrib/hbpost.hbm,
> so we can finally experiment with real examples.
>
> BTW at current state _all_ contribs are successfully built
> on Windows when using HB_BUILD_CONTRIB_DYN=yes option. And
> not all, but some of these can be successfully used (not all
> of them though).
> (* the only contribs I haven't tested is OpenSSL dependent
> ones). Even hbqtcore.dll and friends are built now.

Thanks and congratulations Viktor, sounds like a big gain of flexibility to
all contribs users!

Regards,
Leandro

[Viktor Szakáts]

Hi Leandro,

>> No, it's used to move prg/c code into a .dll and use that
>> dll at link time. Meaning the final app will automatically
>> require and load <lib>.dll and harbour*.dll when launched.
>>
>
> Thank you for explaining. So this allows to reduce the executable size to nothing but the application prg "core", because HVM will be loaded from harbour*.dll and "everything else" from <lib[s]>.dll, right?

Yes.

> BTW, does harbour*.dll contain only HVM or also RTL, RDDs, etc?

Yes. Everything besides hbcplr and hbdebug.

> This scenario is interesting but how does the Harbour licence deal with this case where one has to distribute harbour*.dll with proprietary application?

It doesn't contain pure GPL code (hbcplr), so it's
freely distributable.

> (...)
>> I think DYNAMIC keyword is needed for that. IMO no linker
>> trick would help, but I pass the word to Przemek.
>
> Ok, lets wait Przemek can join this thread again. I also reread his instructions from february, rebuilt harbour with -DHB_DYNAMIC cFlag, and used "hbmk2 mydll.prg -nohblib -strip -hbdyn -lhbmaindllp -cflag=-DHB_DYNLIB -n -w -es2" but it didn't work. I get undefined error to hb_vmProcAddress link error building the pCode dll.
>
> For now I just need to implement the 4th case mentioned by Przemek then, that would be "Main application is linked in static mode with harbour static libraries but PCODE dll is not linked statically at link time but rather loaded dynamically by HB_LIBLOAD()", so I can compare the performance of pCode DLLs versus HRB modules.

I tried this with current .dlls (using mingw) and
the load is successful, but it cannot find the loaded
symbols, so it RTEs. I've documented it in ChangeLog.

>> Now we have make.hbs capable to build .dlls out of contribs,
>> where the build parameters are configured in contrib/hbpost.hbm,
>> so we can finally experiment with real examples.
>>
>> BTW at current state _all_ contribs are successfully built
>> on Windows when using HB_BUILD_CONTRIB_DYN=yes option. And
>> not all, but some of these can be successfully used (not all
>> of them though).
>> (* the only contribs I haven't tested is OpenSSL dependent
>> ones). Even hbqtcore.dll and friends are built now.
>
> Thanks and congratulations Viktor, sounds like a big gain of flexibility to all contribs users!

Thank you.

Just some warning lights: It's still highly experimental, and
I just reached the stage to find a (well yet another) pretty large
miss from current implementation which will have to be sorted out
to make the whole thing work properly on all compilers. For now,
if you want to experiment, use mingw compiler. [ the problem is
related to how to add lib dependency references without adding
self. ]

Viktor

[druzus]

On 3 Sie, 00:33, Viktor Szakáts <harbour...@syenar.hu> wrote:

Hi,

> >> I think DYNAMIC keyword is needed for that. IMO no linker
> >> trick would help, but I pass the word to Przemek.

DYNAMIC disable hardcoded bindings to given function.
Final binaries can be linked without this function and dynamic
HRB modules or shared libraries can be loaded even if such
function does not exists. Bindings is done automatically
on function call without any performance overhead.
It's also usefule when you have to dynamic PCODE modukles
with cross references because it allows to load them without
missing function RTE.

> I tried this with current .dlls (using mingw) and
> the load is successful, but it cannot find the loaded
> symbols, so it RTEs. I've documented it in ChangeLog.

Functions written in C without any references from at
least one PRG module are not registered automatically
in HVM. This is the reason why you have "Undefined
function: CHARADD" RTE in your example in ChangeLog.
To resolve this problem we added hbextern to harbour
shared library. The same should be done for HBCT.
I.e. you can add to HBCT library such simple PRG file

ANNOUNCE __HBCT_EXTERN__
#include "ctextern.ch"

PRG functions or functions which are accessed from
some PRG modules which are part of HBCT library
are registered in HVM automatically. It happens because
each compiled PRG module contains symbol table
with references to all defined and used functions (with
the exception to the ones marked as DYNAMIC) and
this symbol table is automatically registered by startup
code when applications starts or library is loaded.

In theory we can add code with symbol table to each
.c file which defines functions for PRG code and they
will be registered automatically (we have such tables
in RDD code for each RDD, i.e. src/rdd/sdf1.c from
HB_INIT_SYMBOLS_BEGIN(...) to the end of file)
but then we will have to sync them manually. Trick
with adding file including with {hb,ct}extern.ch seems
to be much easier to manage.

Please also note that the problem is only with C code
loaded as platform specific dynamic libraries like .dll,
.so .dyn, ... so it does not exists with PRG code and
HRB files. On some platform the format of dynamic
libraries is known and even some API to scan public
symbols in such libraries exists. For these platforms
we can create code which will register all functions
with name starting with HB_FUN_ prefix in HVM global
symbol table just after library is loaded. It would resolve
the problem too in really nice way but it's not well
portable solution so sooner or later we will have to
implement some alternative version described above.

BTW Exactly the same PCODE is generated for HRB
modules and shared libraries so there is no speed
difference for both versions though function encapsulation
is slightly different.

HTH

best regards,
Przemek

[Viktor Szakáts]

Hi Przemek,

>> I tried this with current .dlls (using mingw) and
>> the load is successful, but it cannot find the loaded
>> symbols, so it RTEs. I've documented it in ChangeLog.
>
> Functions written in C without any references from at
> least one PRG module are not registered automatically
> in HVM. This is the reason why you have "Undefined
> function: CHARADD" RTE in your example in ChangeLog.
> To resolve this problem we added hbextern to harbour
> shared library. The same should be done for HBCT.
> I.e. you can add to HBCT library such simple PRG file
>
> ANNOUNCE __HBCT_EXTERN__
> #include "ctextern.ch"

It solved it perfectly.

> Please also note that the problem is only with C code
> loaded as platform specific dynamic libraries like .dll,
> .so .dyn, ... so it does not exists with PRG code and
> HRB files. On some platform the format of dynamic
> libraries is known and even some API to scan public
> symbols in such libraries exists. For these platforms
> we can create code which will register all functions
> with name starting with HB_FUN_ prefix in HVM global
> symbol table just after library is loaded. It would resolve
> the problem too in really nice way but it's not well
> portable solution so sooner or later we will have to
> implement some alternative version described above.

Above solution really raises some manageability
problems. Maintaining hbextern.ch is already a heavy
burden and doing the same for all libs is certainly
not very good, but for sure error prone.

So pbly the question is how to generate these "extern"
(rather "request") files automatically? If it's not
possible to do reliable from object, it may be done
from source, and question remains: when? hbmk2 could
do such thing, we have hbextern to do it (though it
needs rework to operate inside the build system, and
now also inside hbmk2). Plus there is the other side,
DYNAMIC, which is easy to create form the same list,
even with a simple #define. Of course any source parsing
solution is slow and quite error prone in general,
because of the comment/macro parsing needs (or errors).
If we generate it from objects (or libs), it makes
build more difficult because you need two passes. So
it'd be better to keep it as a pre-build task.

Any thoughts are welcome.

Viktor

[druzus]

On 3 Sie, 12:13, Viktor Szakáts <harbour...@syenar.hu> wrote:

Hi Viktor,

> So pbly the question is how to generate these "extern"
> (rather "request") files automatically? If it's not
> possible to do reliable from object, it may be done
> from source, and question remains: when? hbmk2 could
> do such thing, we have hbextern to do it (though it
> needs rework to operate inside the build system, and
> now also inside hbmk2). Plus there is the other side,
> DYNAMIC, which is easy to create form the same list,
> even with a simple #define. Of course any source parsing
> solution is slow and quite error prone in general,
> because of the comment/macro parsing needs (or errors).
> If we generate it from objects (or libs), it makes
> build more difficult because you need two passes. So
> it'd be better to keep it as a pre-build task.
> Any thoughts are welcome.

From .a library:
nm libhbct.a |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/
g"
From .o files:
nm *.o |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/g"

best regards,
Przemek

[Viktor Szakáts]

Thanks Przemek. How to tell grep and sed to accept an optional
leading underscore before 'HB_FUN_' ? mingw adds it.

[ This solution still needs a lot of work as it is post,
so another build pass has to be added, plus similar parser
written for all supported C compilers :( ]

Viktor

[Tamas TEVESZ]

On Tue, 3 Aug 2010, Viktor Szakáts wrote:

> > From .a library:
> > nm libhbct.a |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/
> > g"
> > From .o files:
> > nm *.o |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/g"

is there a difference between these two? i can't see any :)

> Thanks Przemek. How to tell grep and sed to accept an optional
> leading underscore before 'HB_FUN_' ? mingw adds it.

nm $file | sed -r '/ T _?HB_FUN_/{s/.*_?HB_FUN_(.*)/EXTERNAL \1/;p;d};d'

--
[-]

mkdir /nonexistent

[druzus]

On 3 Sie, 12:52, Viktor Szakáts <harbour...@syenar.hu> wrote:

Hi Viktor,

> > From .a library:
> >  nm libhbct.a |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/
> > g"
> > From .o files:
> >  nm *.o |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/g"
>
> Thanks Przemek. How to tell grep and sed to accept an optional
> leading underscore before 'HB_FUN_' ? mingw adds it.

simpler version using only nm and sed:
nm -g --defined-only -C libhbct.a|sed -e '/ _*HB_FUN_/ !d' -e 's/.*
_*HB_FUN_\(.*\)/\1/g'

> [ This solution still needs a lot of work as it is post,
> so another build pass has to be added, plus similar parser
> written for all supported C compilers :( ]

It's enough to regenerate such extern files from
time to time using MinGW or Linux GCC builds
and commit them. It can be even part of default
MinGW and Linux/GCC builds activated by some
envvar i.e. similar to HB_REBUILD_PARSER=yes
so few of us can set it and extern files in contrib
will be updated automatically. Then on next commit
it will be enough to only add small description to
ChangeLog entry.

best regards,
Przemek

[Viktor Szakáts]

>>> From .a library:
>>> nm libhbct.a |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/
>>> g"
>>> From .o files:
>>> nm *.o |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/g"
>
> is there a difference between these two? i can't see any :)
>
>> Thanks Przemek. How to tell grep and sed to accept an optional
>> leading underscore before 'HB_FUN_' ? mingw adds it.
>
> nm $file | sed -r '/ T _?HB_FUN_/{s/.*_?HB_FUN_(.*)/EXTERNAL \1/;p;d};d'

Superb, thanks.

Continued in reply to Przemek's answer.

Viktor

[Leandro Damasio - 2D Info]

Hi Viktor

> Above solution really raises some manageability
> problems. Maintaining hbextern.ch is already a heavy
> burden and doing the same for all libs is certainly
> not very good, but for sure error prone.
>
> So pbly the question is how to generate these "extern"
> (rather "request") files automatically? If it's not
> possible to do reliable from object, it may be done
> from source, and question remains: when? hbmk2 could
> do such thing, we have hbextern to do it (though it
> needs rework to operate inside the build system, and
> now also inside hbmk2). Plus there is the other side,
> DYNAMIC, which is easy to create form the same list,
> even with a simple #define. Of course any source parsing
> solution is slow and quite error prone in general,
> because of the comment/macro parsing needs (or errors).
> If we generate it from objects (or libs), it makes
> build more difficult because you need two passes. So
> it'd be better to keep it as a pre-build task.
>
> Any thoughts are welcome.

Maybe the whole solution can become simpler if pCode DLLs are treated as
symbol containers only, without the need to treat the references to foreign
symbols. When the DLL is mounted despite the unresolved references, the
undefined symbols can be detected by the host HVM in run-time. Borland
ilink32 for example uses the magic word "CAFEDEAD" as the address for
unresolved references, so when the DLL symbol table is being registered it
is possible to replace those "DEAD" adresses by adresses already resolved in
the host HVM.

Regards,
Leandro

[Viktor Szakáts]

Hi Przemek and Tamas,

>>> From .a library:
>>> nm libhbct.a |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/
>>> g"
>>> From .o files:
>>> nm *.o |grep " T HB_FUN_"|sed "s/.*HB_FUN_\(.*\)/EXTERNAL \1/g"
>>
>> Thanks Przemek. How to tell grep and sed to accept an optional
>> leading underscore before 'HB_FUN_' ? mingw adds it.
>
> simpler version using only nm and sed:
> nm -g --defined-only -C libhbct.a|sed -e '/ _*HB_FUN_/ !d' -e 's/.*
> _*HB_FUN_\(.*\)/\1/g'

regex p$rn!

>> [ This solution still needs a lot of work as it is post,
>> so another build pass has to be added, plus similar parser
>> written for all supported C compilers :( ]
>
> It's enough to regenerate such extern files from
> time to time using MinGW or Linux GCC builds
> and commit them. It can be even part of default
> MinGW and Linux/GCC builds activated by some
> envvar i.e. similar to HB_REBUILD_PARSER=yes
> so few of us can set it and extern files in contrib
> will be updated automatically. Then on next commit
> it will be enough to only add small description to
> ChangeLog entry.

That could work.

I've added this to make.hbs, and similar could
be added to postinst.hbs to create hbextern.ch.

Pls check it, I chose .hbx extension for this
kind of generated file. It has ANNOUNCE right
there, so it's directly compilable (though now
I'm not sure about the ANNOUNCE thing).

[ Central one has some tricky implications like
multiple input libs, plus the split nature of
files, and some extra #define tricks. I'll see. ]

Viktor

[Leandro Damasio - 2D Info]

Hi Viktor,

Sorry, I'm not sure I understood what you've decided.
A header file declaring all harbour's libraries exported functions will be
parsed and distributed with harbour sources to be #included on pCode DLL
modules to declare DYNAMIC references to harbour's libraries exported
functions? This #inclusion will be implemented automatically by hbmk2?

Regards,
Leandro

[Viktor Szakáts]

Hi Leandro and All,

> Sorry, I'm not sure I understood what you've decided.
> A header file declaring all harbour's libraries exported functions will be parsed and distributed with harbour sources to be #included on pCode DLL modules to declare DYNAMIC references to harbour's libraries exported functions? This #inclusion will be implemented automatically by hbmk2?

Moving forward step by step, I haven't decided
yet about these things, for now I'm concentrating
on clearing the obstacle from the way.

If the automatically generated headers will be in
place, we can start to use them to pull in all
.prg code into .dlls so that they can be loaded
and called dynamically, and they can also be used
in user code which plans to dynamically call these
functions. [BTW with hbct this should work already. ]

For the first purpose, all libs will have to feature
a new .prg which includes the .hbx file. This can
be included in all builds unconditionally (AFAICS).
For the second purpose there has be added a logic
to translate EXTERNAL to DYNAMIC.

Probably it will be good to change EXTERNAL to
DYNAMIC by default in .hbx code and make the
translation to EXTERNAL on request, f.e. when
compiling them for lib or dll.

So:

--- hblib.hbx
#ifdef __HBLIB_EXTERN_REQ__
ANNOUNCE __HBLIB_EXTERN__
#command __EXTDECL <func> => EXTERNAL <func>
#else
#command __EXTDECL <func> => DYNAMIC <func>
#endif
__EXTDECL HBLIB_MYFUNC1
__EXTDECL HBLIB_MYFUNC2
__EXTDECL HBLIB_MYFUNCn
---

--- hblibext.prg
#define __HBLIB_EXTERN_REQ__
#include "hblib.hbx"
---

- - - - - - - - - - - - - - - - - - - - - -

--- userapp.prg
#include "hblib.hbx"
...code...
---

Before I implement this for 50 contribs I'd like to
hear from others and certainly from Przemek, if this
seems like a feasible solution to solve all needs
in an efficient way. I don't want to waste my time
with trial and error.

Viktor

[druzus]

On 4 Sie, 14:47, Viktor Szakáts <harbour...@syenar.hu> wrote:

Hi Viktor,

> If the automatically generated headers will be in
> place, we can start to use them to pull in all
> .prg code into .dlls so that they can be loaded
> and called dynamically, and they can also be used
> in user code which plans to dynamically call these
> functions. [BTW with hbct this should work already. ]
>
> For the first purpose, all libs will have to feature
> a new .prg which includes the .hbx file. This can
> be included in all builds unconditionally (AFAICS).

To be precise.
This is necessary only for .c files which have Harbour
functions never accessed from any PRG code and
this .c files after compilation are part of some .DLL.
It's not necessary for .prg code and users creating
their own DLLs from compiled .PRG code can ignore
this thread because it will not change anything for
them.

> For the second purpose there has be added a logic
> to translate EXTERNAL to DYNAMIC.

If we create files with complete list of function (also
PRG ones) then they will be useful also for dynamic
programing. Anyhow if we resolve the problem with
automatic registering Harbour functions written in .c
which are not accessed from any .prg code in HVM
global symbol table then anyone can easy generate
function list at runtime using small .prg code so this
problem will be resolved too.
Now we generate complete list of functions from
compiled libraries so using the same files with full
function list seems to be good idea.

ANNOUNCE should not be part of .hbx file.
Instead it should be added to hblibext.prg
Inside .hbc file. In such case we do not need
any hardcoded PP rules in .hbx file and programmer
can simply make:

#command EXTERN <fncs,...> => DYNAMIC <fncs>
#include "hblib.hbx"
#uncommand EXTERN <fncs,...> => DYNAMIC <fncs>

> Before I implement this for 50 contribs I'd like to
> hear from others and certainly from Przemek, if this
> seems like a feasible solution to solve all needs
> in an efficient way. I don't want to waste my time
> with trial and error.

The only one problem I can see is list of some
exceptions for functions which should not be added
to .hbx files automatically. Probably this part will have
to be updated manually, i.e. we can have optional
exclude.hbx in each contrib directory which needs them.
For some functions like HB_GT_* we can always exclude
but it's possible that some libraries will use their own
functions which should not be registered in HVM when
library is loaded from DLL.

best regards,
Przemek

[Viktor Szakáts]

Hi Przemek,

>> If the automatically generated headers will be in
>> place, we can start to use them to pull in all
>> .prg code into .dlls so that they can be loaded
>> and called dynamically, and they can also be used
>> in user code which plans to dynamically call these
>> functions. [BTW with hbct this should work already. ]
>>
>> For the first purpose, all libs will have to feature
>> a new .prg which includes the .hbx file. This can
>> be included in all builds unconditionally (AFAICS).
>
> To be precise.
> This is necessary only for .c files which have Harbour
> functions never accessed from any PRG code and
> this .c files after compilation are part of some .DLL.
> It's not necessary for .prg code and users creating
> their own DLLs from compiled .PRG code can ignore
> this thread because it will not change anything for
> them.

Yes, I stand corrected it's .c not .prg.

The #uncommand makes it simpler (though what if
users have this command already overridden?), but
I think it should by default be DYNAMIC, since this
is the only reason (AFAICS) users might include this
header in app code. To use the "EXTERN" functionality,
they can just REQUEST __HBLIB_EXTERN__, which shall
also be more efficient.

Or is there any other situation when user want to
include it in EXTERN mode? (given they can REQUEST
the whole list using one command?)

>> Before I implement this for 50 contribs I'd like to
>> hear from others and certainly from Przemek, if this
>> seems like a feasible solution to solve all needs
>> in an efficient way. I don't want to waste my time
>> with trial and error.
>
> The only one problem I can see is list of some
> exceptions for functions which should not be added
> to .hbx files automatically. Probably this part will have
> to be updated manually, i.e. we can have optional
> exclude.hbx in each contrib directory which needs them.
> For some functions like HB_GT_* we can always exclude
> but it's possible that some libraries will use their own
> functions which should not be registered in HVM when
> library is loaded from DLL.

Yes, for this maybe the simplest is to include
these exceptions in .hbx file itself using some
special notation, so the generator will have to
be extended to first parse it to find these
exceptions. F.e. "// HB_NO_EXTERN <function_name>"

How does this sound?

Viktor

[druzus]

On 4 Sie, 15:40, Viktor Szakáts <harbour...@syenar.hu> wrote:

Hi VIktor,

> >   #command EXTERN <fncs,...> => DYNAMIC <fncs>
> >   #include "hblib.hbx"
> >   #uncommand EXTERN <fncs,...> => DYNAMIC <fncs>
>
> The #uncommand makes it simpler (though what if
> users have this command already overridden?), but

Nothing wrong, unlike #define commands and translations
are pushed on the stack and coresponding #un* directive
pop it from the stack restoring previous user definition.

> I think it should by default be DYNAMIC, since this
> is the only reason (AFAICS) users might include this
> header in app code. To use the "EXTERN" functionality,
> they can just REQUEST __HBLIB_EXTERN__, which shall
> also be more efficient.

OK,
It could be DYNAMIC though personally I'd prefer to user
extern because I use such files to force linking functions
from given libraries with my application and list with DYNAMIC
definitions I'm generating automatically from final binaries so
I do not use them at all.

> Or is there any other situation when user want to
> include it in EXTERN mode? (given they can REQUEST
> the whole list using one command?)

see above,

> Yes, for this maybe the simplest is to include
> these exceptions in .hbx file itself using some
> special notation, so the generator will have to
> be extended to first parse it to find these
> exceptions. F.e. "// HB_NO_EXTERN <function_name>"
> How does this sound?

Any form is good for me, though using the same file
makes the procedure more complicated because you
cannot simply overload existing file. Anyhow chose
the method it's better for you.

best regards,
Przemek

[Leandro Damasio - 2D Info]

> Moving forward step by step, I haven't decided
> yet about these things, for now I'm concentrating
> on clearing the obstacle from the way.

Ok Viktor, thank you.
Regards,
Leandro