Modifications of standard headers for PECOFF
TAMURA Kent
PECOFF format, on which the PEACE project [1] bases.
- Add /usr/include/sys/cdefs_pe.h
- Add LIBC_EXTERN definitions to cdefs_elf.h, cdefs_aout.h and cdefs_pe.h like:
cdefs_elf.h cdefs_aout.h:
#define LIBC_EXTERN extern
cdefs_pe.h:
#ifdef LIBC
#define LIBC_EXTERN __declspec(dllexport) extern
#else
#define LIBC_EXTERN __declspec(dllimport) extern
#endif
- Modify cdefs.h so that it includes cdefs_pe.h if __PECOFF__ is defined.
- For global variables defined in libc and exposed to the outside of libc,
add LIBC_EXTERN to their declaration. For example,
stdio.h:
extern FILE __sF[];
->
LIBC_EXTERN FILE __sF[];
The modifications are required because of the special treatment
for global variables exported/imported over DLLs. In the PEACE
project, we provide PECOFF versions of some standard libraries
such as libc.dll. If the standard headers in /usr/include
supports PECOFF, we don't have to copy the headers and patch
them.
Any comments?
[1] http://chiharu.hauN.org/peace/
--
TAMURA Kent <kent...@hauN.org> <ke...@netbsd.org>
Jason R Thorpe
> - Add /usr/include/sys/cdefs_pe.h
> - Add LIBC_EXTERN definitions to cdefs_elf.h, cdefs_aout.h and cdefs_pe.h like:
> cdefs_elf.h cdefs_aout.h:
> #define LIBC_EXTERN extern
> cdefs_pe.h:
> #ifdef LIBC
> #define LIBC_EXTERN __declspec(dllexport) extern
> #else
> #define LIBC_EXTERN __declspec(dllimport) extern
> #endif
> - Modify cdefs.h so that it includes cdefs_pe.h if __PECOFF__ is defined.
> - For global variables defined in libc and exposed to the outside of libc,
> add LIBC_EXTERN to their declaration. For example,
> stdio.h:
> extern FILE __sF[];
> ->
> LIBC_EXTERN FILE __sF[];
If these are going to be in header files, it must be in the implementation
namespace. Might I suggest:
__libc_extern FILE __sF[];
...or possibly:
__lib_extern ...
because I guess libraries other than libc would need similar treatment?
Anyway, other than that, I don't see any reason not to allow these
changes.
> The modifications are required because of the special treatment
> for global variables exported/imported over DLLs. In the PEACE
> project, we provide PECOFF versions of some standard libraries
> such as libc.dll. If the standard headers in /usr/include
> supports PECOFF, we don't have to copy the headers and patch
> them.
>
> Any comments?
...it might be nice to have to have the run-time linker in the
main source tree :-)
--
-- Jason R. Thorpe <tho...@wasabisystems.com>
Christos Zoulas
TAMURA Kent <ke...@netbsd.org> wrote:
I don't have a problem with that. But is that limited to libc?
Perhaps a better name than LIBC_EXTERN would be __export?
christos
TAMURA Kent
> If these are going to be in header files, it must be in the implementation
> namespace. Might I suggest:
>
> __libc_extern FILE __sF[];
Ok, I agree on this.
> __lib_extern ...
>
> because I guess libraries other than libc would need similar treatment?
Actually we have to define macro for each library which exposes
global variables. While building libstdc++.dll, "__sF[]" should
be "__declspec(dllimport) extern FILE __sF[]" and "cout" should
be "__declspec(dllexport) extern _IO_ostream_withassign cout".
This is a kind of DLL Hell ;-(
> ...it might be nice to have to have the run-time linker in the
> main source tree :-)
The run-time linker is also a PECOFF executable. So we need
build environment targetting PECOFF. This change is the first
step to build it in the main source tree.
Jim Wise
Hash: SHA1
On Thu, 16 May 2002, TAMURA Kent wrote:
>> ...it might be nice to have to have the run-time linker in the
>> main source tree :-)
>
>The run-time linker is also a PECOFF executable. So we need
>build environment targetting PECOFF. This change is the first
>step to build it in the main source tree.
We could perhaps provide `blessed' uuencoded versions of the PECOFF
runtime linker, in the source tree in uuencoded format, much as we do
for the a.out linker on i386 and elsewhere...
- --
Jim Wise
jw...@draga.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (NetBSD)
iD8DBQE86I31N71lEcOYcw4RAk7+AJ9hCXEXXLnHkaI4agdi+Ab4cvDFlgCgmlWU
lIIWsWNOBPEDN9wOjN4X4/k=
=0QR0
-----END PGP SIGNATURE-----
Bang Jun-Young
> PECOFF format, on which the PEACE project [1] bases.
BTW, what advantages can we get by having support for PE/COFF in
the base system, compared to leaving it as separate package like
PEACE?
Jun-Young
--
Bang Jun-Young <juny...@mogua.com>
TAMURA Kent
In message "Re: Modifications of standard headers for PECOFF"
on 02/05/24, Bang Jun-Young <juny...@mogua.com> writes:
> BTW, what advantages can we get by having support for PE/COFF in
> the base system, compared to leaving it as separate package like
> PEACE?
We have a plan to implement some kernel32.dll functions in
compat_pecoff. It should be more efficient than server approach
like Wine.
Bang Jun-Young
>
> In message "Re: Modifications of standard headers for PECOFF"
> on 02/05/24, Bang Jun-Young <juny...@mogua.com> writes:
> > BTW, what advantages can we get by having support for PE/COFF in
> > the base system, compared to leaving it as separate package like
> > PEACE?
>
> We have a plan to implement some kernel32.dll functions in
> compat_pecoff. It should be more efficient than server approach
> like Wine.
Hmm...I'm skeptical that it's the right direction for us to have
PE/COFF support in the base source. IMHO, it'd be better to keep it
under sys/compat/pecoff and provide pkgsrc entries for PEACE.
More specifically, some reasons why I think so:
* PEACE need cross toolchain to build.
* PE/COFF has never been the native binary format of NetBSD, and never
will. It's only designed for Windows, not Unix.
* There are quite a number of API differences between MSVCRT and
NetBSD libc. Any binaries compiled with and successfully running with
PEACE libc.dll are not guaranteed to run on Windows with MSVCRT.DLL,
or vice versa.
* We don't want to fork off NetBSD/i386pecoff, NetBSD/alphapecoff, ...
from existing ports (do we?).
Bang Jun-Young
> for global variables exported/imported over DLLs. In the PEACE
> project, we provide PECOFF versions of some standard libraries
> such as libc.dll. If the standard headers in /usr/include
> supports PECOFF, we don't have to copy the headers and patch
> them.
IMHO, the idea of providing libc.dll as replacement to MSVCRT.DLL
is not good. In a project I have been involved in, we wrote msvcrt.dll
around mingw-runtime headers and it seemed to work quite well.
There was no need for touching NetBSD headers in /usr/include.
Bill Studenmund
>
> In message "Re: Modifications of standard headers for PECOFF"
> on 02/05/24, Bang Jun-Young <juny...@mogua.com> writes:
> > BTW, what advantages can we get by having support for PE/COFF in
> > the base system, compared to leaving it as separate package like
> > PEACE?
>
> We have a plan to implement some kernel32.dll functions in
> compat_pecoff. It should be more efficient than server approach
> like Wine.
I don't think that answered his question. I think the question is why
should NetBSD's userland includes be modified for PEACE?
Take care,
Bill
Jason R Thorpe
> I don't think that answered his question. I think the question is why
> should NetBSD's userland includes be modified for PEACE?
Because he needs to be able to build PECOFF versions of some system
libraries which will help interface to the in-kernel emulator.
Bill Studenmund
> On Fri, May 24, 2002 at 01:04:13AM +0900, TAMURA Kent wrote:
> >
> > In message "Re: Modifications of standard headers for PECOFF"
> > on 02/05/24, Bang Jun-Young <juny...@mogua.com> writes:
> > > BTW, what advantages can we get by having support for PE/COFF in
> > > the base system, compared to leaving it as separate package like
> > > PEACE?
> >
> > We have a plan to implement some kernel32.dll functions in
> > compat_pecoff. It should be more efficient than server approach
> > like Wine.
As the other message mentioned, I don't think that answered the question.
> Hmm...I'm skeptical that it's the right direction for us to have
> PE/COFF support in the base source. IMHO, it'd be better to keep it
> under sys/compat/pecoff and provide pkgsrc entries for PEACE.
On the flip side, I don't think you quite understood the suggestion. Or at
least I got a different understanding than you did. :-)
> More specifically, some reasons why I think so:
>
> * PEACE need cross toolchain to build.
>
> * PE/COFF has never been the native binary format of NetBSD, and never
> will. It's only designed for Windows, not Unix.
>
> * There are quite a number of API differences between MSVCRT and
> NetBSD libc. Any binaries compiled with and successfully running with
> PEACE libc.dll are not guaranteed to run on Windows with MSVCRT.DLL,
> or vice versa.
I don't think that's what the PEACE project is doing.
From the ancient install I have, what they are doing is:
1) providing a loader which will load & link PECOFF executables.
2) providing DLLs that have the same name as standard windows ones, that
windows apps will link against. Like say kernel32.dll, ddraw.dll,
ole32.dll, user32.dll, etc.
3) Implementing said dlls using a unix-like environment. i.e. the innards
of these dlls look like (Net)BSD libraries. They just get bundled as dlls.
4) providing dll-versions of NetBSD libraries for use in 3). i.e. the
libc.dll is for the PEACE dlls, not the windows apps.
While libc mightn't seem like much, looking at the (old) PEACE install I
have, I have libX11.dll, libXext.dll, libXmu.dll, and libXt.dll (among
others). While the peace folks could probably get away w/o libc, making
the X libs work without libc is not something I would wish on anyone.
Since X is the windowing system we use, PEACE dlls have to turn Windows
DirectDraw calls into X calls.
My understanding of what Tamura-san was wanting is to make life easier for
these dlls; these dlls that are based off of NetBSD code (makes sense).
However routines that are in dlls need a "I'm inside your dll" or "I'm
outside your dll" attribute on the prototype.
So all of the standard NetBSD headers could be used in the PECOFF
envirnoment if they had these attributes.
The PEACE folks have macros set up so that the right thing happens when
you're building different DLLs; the prototypes in that dll say "i'm in
your dll". The same prototypes say "I'm in dll foo" when compiling other
code.
From everything I can tell, to add these attributes/macros is a manual
task. You have to find EACH prototype, and add the right one.
More imortantly, over time as we add stuff to the NetBSD headers, the new
prototypes we add need to get these new macros. Manually.
That looks like a maintainance nightmare, which is why TAMURA-san wanted
to have the NetBSD-source tree contain the macros.
Of course, in the NetBSD build envirnment, the macros expand to nothing.
:-)
We can choose to add these macros or not. But let's understand what we're
adding and why.
> * We don't want to fork off NetBSD/i386pecoff, NetBSD/alphapecoff, ...
> from existing ports (do we?).
I don't think that was suggested. ??
Oh, while I think it would be a bit weird to see in our code, I'm up for
adding this stuff to our tree.
Take care,
Bill
TAMURA Kent
In message "Re: Modifications of standard headers for PECOFF"
on 02/05/24, Bang Jun-Young <juny...@mogua.com> writes:
> PE/COFF support in the base source. IMHO, it'd be better to keep it
> under sys/compat/pecoff and provide pkgsrc entries for PEACE.
It is one of reasonable choices.
Anyway, what we want to do at the moment is to make building
PEACE easier. We need tons of patches if we had no header
modifications.
Bang Jun-Young
> - Add LIBC_EXTERN definitions to cdefs_elf.h, cdefs_aout.h and cdefs_pe.h like:
> cdefs_elf.h cdefs_aout.h:
> #define LIBC_EXTERN extern
> cdefs_pe.h:
> #ifdef LIBC
> #define LIBC_EXTERN __declspec(dllexport) extern
> #else
> #define LIBC_EXTERN __declspec(dllimport) extern
> #endif
> - Modify cdefs.h so that it includes cdefs_pe.h if __PECOFF__ is defined.
> - For global variables defined in libc and exposed to the outside of libc,
> add LIBC_EXTERN to their declaration. For example,
> stdio.h:
> extern FILE __sF[];
> ->
> LIBC_EXTERN FILE __sF[];
These changes don't even look reasonable to me. Why don't you use
--enable-auto-export and --enable-auto-import features of recent
binutils (the former is default now, AFAIK)?
TAMURA Kent
In message "Re: Modifications of standard headers for PECOFF"
on 02/05/24, Bang Jun-Young <juny...@mogua.com> writes:
> --enable-auto-export and --enable-auto-import features of recent
> binutils (the former is default now, AFAIK)?
Oh, I have not known --enable-auto-import. I'll try it. Thank
you.
TAMURA Kent
In message "Re: Modifications of standard headers for PECOFF"
> > These changes don't even look reasonable to me. Why don't you use
> > --enable-auto-export and --enable-auto-import features of recent
> > binutils (the former is default now, AFAIK)?
>
> Oh, I have not known --enable-auto-import. I'll try it. Thank
> you.
I have tried it and found --enable-auto-import does not work in
many cases. It seems to work for simple variables but not for
arrays and struct variables.
I'll commit __libc_extern change later.
Perry E. Metzger
Jason R Thorpe <tho...@wasabisystems.com> writes:
> On Thu, May 23, 2002 at 04:06:44PM -0700, Bill Studenmund wrote:
> > I don't think that answered his question. I think the question is why
> > should NetBSD's userland includes be modified for PEACE?
>
> Because he needs to be able to build PECOFF versions of some system
> libraries which will help interface to the in-kernel emulator.
That makes considerable sense. Myself, I think PEACE is a fairly
important project, and I'd like to see impediments to development lowered...
--
Perry E. Metzger pe...@wasabisystems.com
--
NetBSD: The right OS for your embedded design. http://www.wasabisystems.com/
Bill Studenmund
> In message "Re: Modifications of standard headers for PECOFF"
> on 02/05/24, TAMURA Kent <ke...@netbsd.org> writes:
> > > These changes don't even look reasonable to me. Why don't you use
> > > --enable-auto-export and --enable-auto-import features of recent
> > > binutils (the former is default now, AFAIK)?
> >
> > Oh, I have not known --enable-auto-import. I'll try it. Thank
> > you.
>
> I have tried it and found --enable-auto-import does not work in
> many cases. It seems to work for simple variables but not for
> arrays and struct variables.
Can you describe how it doesn't work? It could be we can get binutils
fixed. :-)
I think we all want to make it so that PEACE doesn't need to keep all
those header diffs. If it's easy to get auto-import and auto-export to
work, we should do that.
Hmmm... One thing Tamura-san mentioned is that it's __libc_extern since
the macros expand to different things if you're compiling libc.dll or some
other dll. Jun-Young, how would auto-import and auto-export help that?
Take care,
Bill
Bang Jun-Young
>
> In message "Re: Modifications of standard headers for PECOFF"
> on 02/05/24, TAMURA Kent <ke...@netbsd.org> writes:
> > > These changes don't even look reasonable to me. Why don't you use
> > > --enable-auto-export and --enable-auto-import features of recent
> > > binutils (the former is default now, AFAIK)?
> >
> > Oh, I have not known --enable-auto-import. I'll try it. Thank
> > you.
>
> I have tried it and found --enable-auto-import does not work in
> many cases. It seems to work for simple variables but not for
> arrays and struct variables.
Arrgh. It's a known problem. ld(1) manual says (sorry for long
excerption):
--enable-auto-import
Do sophisticated linking of _symbol to __imp__symbol for DATA
imports from DLLs, and create the necessary thunking symbols
when building the DLLs with those DATA exports. This generally
will 'just work' - but sometimes you may see this message:
"variable '<var>' can't be auto-imported. Please read the
documentation for ld's --enable-auto-import for details."
This message occurs when some (sub)expression accesses an
address ultimately given by the sum of two constants (Win32
import tables only allow one). Instances where this may occur
include accesses to member fields of struct variables imported
from a DLL, as well as using a constant index into an array
variable imported from a DLL. Any multiword variable (arrays,
structs, long long, etc) may trigger this error condition.
However, regardless of the exact data type of the offending
exported variable, ld will always detect it, issue the warning,
and exit.
There are several ways to address this difficulty, regardless
of the data type of the exported variable:
One solution is to force one of the 'constants' to be a
variable - that is, unknown and un-optimizable at compile time.
For arrays, there are two possibilities: a) make the indexee
(the array's address) a variable, or b) make the 'constant'
index a variable. Thus:
extern type extern_array[];
extern_array[1] -->
{ volatile type *t=extern_array; t[1] }
or
extern type extern_array[];
extern_array[1] -->
{ volatile int t=1; extern_array[t] }
For structs (and most other multiword data types) the only
option is to make the struct itself (or the long long, or the
...) variable:
extern struct s extern_struct;
extern_struct.field -->
{ volatile struct s *t=&extern_struct; t->field }
or
extern long long extern_ll;
extern_ll -->
{ volatile long long * local_ll=&extern_ll; *local_ll }
A second method of dealing with this difficulty is to abandon
'auto-import' for the offending symbol and mark it with
__declspec(dllimport). However, in practice that requires using
compile-time #defines to indicate whether you are building a
DLL, building client code that will link to the DLL, or merely
building/linking to a static library. In making the choice
between the various methods of resolving the 'direct address
with constant offset' problem, you should consider typical
real-world usage:
Original:
--foo.h
extern int arr[];
--foo.c
#include "foo.h"
void main(int argc, char **argv){
printf("%d\n",arr[1]);
}
Solution 1:
--foo.h
extern int arr[];
--foo.c
#include "foo.h"
void main(int argc, char **argv){
/* This workaround is for win32 and cygwin; do not "optimize" */
volatile int *parr = arr;
printf("%d\n",parr[1]);
}
Solution 2:
--foo.h
/* Note: auto-export is assumed (no __declspec(dllexport)) */
#if (defined(_WIN32) || defined(__CYGWIN__)) && \
!(defined(FOO_BUILD_DLL) || defined(FOO_STATIC))
#define FOO_IMPORT __declspec(dllimport)
#else
#define FOO_IMPORT
#endif
extern FOO_IMPORT int arr[];
--foo.c
#include "foo.h"
void main(int argc, char **argv){
printf("%d\n",arr[1]);
}
A third way to avoid this problem is to re-code your library to
use a functional interface rather than a data interface for the
offending variables (e.g. set_foo() and get_foo() accessor
functions).
> I'll commit __libc_extern change later.
I still believe __declspec(dllexport) is not necessary at all since
--enable-auto-export is enabled by default, and __declspec(dllimport)
can be omitted for simple variables if --enable-auto-import is given
to ld(1).
BTW, how many multiword variables are exported from libc to
outer modules?
TAMURA Kent
> > I have tried it and found --enable-auto-import does not work in
> > many cases. It seems to work for simple variables but not for
> > arrays and struct variables.
> Can you describe how it doesn't work? It could be we can get binutils
> fixed. :-)
We think it is NOT a bug. According to Oki-san, It is
essentially impossible to resolve this problem only by ld.
TAMURA Kent
> I still believe __declspec(dllexport) is not necessary at all since
> --enable-auto-export is enabled by default, and __declspec(dllimport)
> can be omitted for simple variables if --enable-auto-import is given
> to ld(1).
I think so.
> BTW, how many multiword variables are exported from libc to
> outer modules?
The following is a list of exported variables of libc. The
option --enable-auto-import is effective to h_errorno,
svc_maxfd, sys_nsig, __mb_cur_max, daylight, timezone, opterr,
optind, optopt and optreset, I think. Other variables need
__declspec(dllimport).
ctype.h _ctype_
ctype.h _tolower_tab_
ctype.h _toupper_tab_
errno.h __sys_errlist14
gmon.h _gmonparam
netdb.h h_errno
nsswitch.h __nsdefaultsrc
resolv.h __p_class_syms
resolv.h __p_type_syms
resolv.h _res
resolv.h _res_ext
rpc/auth.h _null_auth
rpc/clnt.h rpc_createerr
rpc/cvs.h svc_fdset
rpc/raw.h __rpc_rawcombuf
rpc/svc.h svc_maxfd
signal.h __sys_siglist14
signal.h __sys_signame14
signal.h sys_signame
stdio.h __sF
stdlib.h __mb_cur_max
sys/localedef.h _CurrentMessagesLocale
sys/localedef.h _CurrentMonetaryLocale
sys/localedef.h _CurrentNumericLocale
sys/localedef.h _CurrentTimeLocale
sys/localedef.h _DefaultMessagesLocale
sys/localedef.h _DefaultMonetaryLocale
sys/localedef.h _DefaultNumericLocale
sys/localedef.h _DefaultTimeLocale
time.h __timezone13
time.h _daylight
time.h _tzname
unistd.h __sys_siglist14
unistd.h optarg
unistd.h opterr
unistd.h optind
unistd.h optopt
unistd.h optreset
unistd.h suboptarg