hmm: "unstable" transfer thunk
BGB / cr88192
the same code may be compiled and run on both Windows and Linux, and it was
my consideration to use Win64 for both OS's on x86-64...
however, the code using Win64 may closely interoperate with code using the
SysV/AMD64 calling convention.
problem:
in not all cases can it be statically determined what is the correct calling
convention.
consider, for example, a function pointer which is stuffed into a structure.
it may not be clear which calling convention the caller will used (AKA:
which side of the "border" it will be called from). however, I don't want to
make them "special", I would want them to remain as good old function
pointers.
so, here is a thought:
how about all non-native function pointers are essentially "wrapped" with an
"unstable" wrapper, which rather than simply calling the function (or even
simply converting arguments), will attempt to determine the calling
convention of the caller, and handle as appropriate.
the question is then, how to go about this?...
one possibility:
a hash table is used to hash prior results (based on return RIP);
the first step is to hash the RIP, and see if the spot is filled (if so, it
contains the "appropriate" transfer thunk);
otherwise, it preserves the state, and transfers control to a "figure it
out" handler.
Win64 generally mandates the use of prologues and epilogues, which are
presumably "out of place" in SysV land, so the handler would likely, as its
first step (or at least an early step, maybe after checking if the function
came from a DLL), would be to search for an epilogue.
the presence of an epilogue could be used to indicate the status of Win64.
also possible:
memory location, namely, OS-controlled=SysV, DLL=Win64.
this leaves the dynamically-assembled region, which annoyingly is more of a
"grab bag" (dynamically compiled code, various auto-generated thunks, ...).
sadly, it would be problematic to look for epilogues in the dynamic region,
since the wrong epilogue may be easily found.
this leaves reverse-lookup, which will be able to at least identify the
signature of the calling function (and, similarly, it could be possible to
infer the caller's size, ...).
in this case, I could make use of "annotations" or similar.
note that composing Win64<->SysV thunks actually needs a decent amount of
info.
at least in the case of my "XCall" name mangling scheme, this info is
generally embedded into the symbol names (subtly, this would mean leaving
internal symbols when composing DLL's, which annoyingly MS's linker does not
do).
the issue then is for the possibility of MSVC compiled code, which will not
preserve this level of information (at this point, I am at a loss as to how
to automatically glue together MSVC compiled code and "native" code in
Linux-land...).
simpler options:
possibly, the function pointer is either requested in, or marshalled to, a
particular calling convention (requires complete information).
example:
foo_f=dyllWrapNative(p);
another possiblilty is to use explicit callers:
dyllApply(p, "(ii)v", 2, 3);
which, conviniently, sidesteps the whole auto-thunking issue (and it is,
after all, much easier to figure out which calling convention the target is
using, and the info needed for marshalling is provided in this case).
the great ugly cost though, is needing to use a function call to call
function pointers.
it really doesn't help that I am considering possibly throwing bytecode into
the mix as well.
summary (of bytecode):
bytecode is derived from EBC (EFI ByteCode) and x86/x86-64 machine code.
opcodes are similar to EBC (and, thus far, no multi-byte opcodes or
prefixes).
a suffix I call the "TCB" (Type Control Block) is used on many opcodes,
which essentially encodes info about the type and addressing mode of opcodes
(originally, I had imagined this as a prefix, with it being absent for
integer opcodes). as such, this structure is "novel", but oh well.
(I could move it to a prefix though, at the cost of an additional byte for
all non-integer opcodes, but saving 1 byte for most integer opcodes).
both ModRM and SIB were merged into a single 2-byte form I am calling the
RMB (or RegMemBlock). this was done because I had noted that, combining
them, I could do a bit more with 16 bits than with ModRM+SIB (16-registers,
additional addressing modes, although, no AVX style opcodes, ...).
the great cost though, is that even for the simple 'reg, reg' case, it takes
2 bytes...
thus, on average, opcodes are bigger than x86 equivalents...
(very few opcodes are < 4 bytes...). (however, SIMD and FP ops would be
generally smaller...).
if needed though, some operations could be "shorthanded", allowing for
2-byte reg/reg integer ops.
one thing it "should" retain from EBC though, is that it is being designed
to work in either 32 or 64-bit mode, as well as a partial 128-bit "wide"
mode (32/64+128).
the register set may make an issue though, as it is likely that for a naive
JIT, memory-mapped registers would be used.
another slight issue (mostly for my compiler), is that the register
naming/numbering differs from x86 and x86-64, namely, R15 is SP (R14 is BP,
all others solely use numeric names).
as well, the "flexible" addressing could be a hassle (since many things do
not have fixed offsets/addresses, which is an assumption made in many cases
in my compiler).
similarly, opcode naming is not exactly the same either, ...
FWIW, I am almost left thinking it would be just as easy to hack x86-64
machine code into a bytecode (x86-64 but with "flexible" addressing hacked
on...). this would reduce the cost of compiling to it, but would complicate
the JIT design (would need to, more or less, JIT 64-bit x86 to 32-bit
x86...).
the other possibility involves hacking flexible addressing into 32-bit x86,
and allowing it to be JITT'ed to 64-bit mode, but this is possibly uglier.
so, the bytecode may still be worth the cost...
James Harris
> well, here is the issue:
> the same code may be compiled and run on both Windows and Linux, and it was
> my consideration to use Win64 for both OS's on x86-64...
>
> however, the code using Win64 may closely interoperate with code using the
> SysV/AMD64 calling convention.
>
> problem:
> in not all cases can it be statically determined what is the correct calling
> convention.
>
> consider, for example, a function pointer which is stuffed into a structure.
> it may not be clear which calling convention the caller will used (AKA:
> which side of the "border" it will be called from). however, I don't want to
> make them "special", I would want them to remain as good old function
> pointers.
Just some thoughts...
Perhaps you could type the pointers to functions. If so, the type
would include the expected data and the mechanism required.
If you can avoid explicit pointers that may be better. Pointers are,
IMHO, the gotos of the data structure world.
>
> so, here is a thought:
> how about all non-native function pointers are essentially "wrapped" with an
> "unstable" wrapper, which rather than simply calling the function (or even
> simply converting arguments), will attempt to determine the calling
> convention of the caller, and handle as appropriate.
As a general approach this may be too slow.
Better may be to have specific prologues and epilogues to the meat of
each function. Or, where the function is small, have separate copies
of the function each with aposite prologues and epilogues.
BGB / cr88192
"James Harris" <james.h...@googlemail.com> wrote in message
news:68092978-c60f-4155...@y20g2000vbk.googlegroups.com...
> On 24 Sep, 17:05, "BGB / cr88192" <cr88...@hotmail.com> wrote:
>> well, here is the issue:
>> the same code may be compiled and run on both Windows and Linux, and it
>> was
>> my consideration to use Win64 for both OS's on x86-64...
>>
>> however, the code using Win64 may closely interoperate with code using
>> the
>> SysV/AMD64 calling convention.
>>
>> problem:
>> in not all cases can it be statically determined what is the correct
>> calling
>> convention.
>>
>> consider, for example, a function pointer which is stuffed into a
>> structure.
>> it may not be clear which calling convention the caller will used (AKA:
>> which side of the "border" it will be called from). however, I don't want
>> to
>> make them "special", I would want them to remain as good old function
>> pointers.
>
> Just some thoughts...
>
> Perhaps you could type the pointers to functions. If so, the type
> would include the expected data and the mechanism required.
>
> If you can avoid explicit pointers that may be better. Pointers are,
> IMHO, the gotos of the data structure world.
>
would be problematic in C, since C expects raw pointers, and raw function
pointers...
the best I could really hope for would be for code to call function pointers
via API calls, but this isn't all that super likely either...
it really doesn't help that, GCC only knows about SysV in this case, and
MSVC only knows Win64, so I can't use approach of annotating everything.
granted, producing code which uses SysV on Linux is an option, but it is not
a "general" solution if cross-platform Windows/Linux code is to be under
consideration.
matters are made a bit simpler though if I disallow Windows/Linux binary
compatibility, where a newer and simpler option presents itself:
just use SysV...
>>
>> so, here is a thought:
>> how about all non-native function pointers are essentially "wrapped" with
>> an
>> "unstable" wrapper, which rather than simply calling the function (or
>> even
>> simply converting arguments), will attempt to determine the calling
>> convention of the caller, and handle as appropriate.
>
> As a general approach this may be too slow.
>
> Better may be to have specific prologues and epilogues to the meat of
> each function. Or, where the function is small, have separate copies
> of the function each with aposite prologues and epilogues.
>
performance is where the hash comes in.
note, none of this is used in cases where the compiler can statically
determine all this, which is likely to be the majority case anyways.
;rax, r10, and r11 are safe in both conventions
foo:
mov rax, [rsp] ;return RIP
call transfer_dispatch
lea rax, ... ;actual function entry point (also used for target
identification)
jmp rax
transfer_dispatch:
mov r10, rax
imul r10, 65521
shr r10, 16
and r10, 4095
mov r11, [transfer_hash+r10*8]
and r11, r11
jnz transfer_go
imul r10, 127
and r10, 4095
mov r11, [transfer_hash+r10*8]
and r11, r11
jnz transfer_go
...
jmp transfer_fail
transfer_go:
jmp r11
transfer_fail:
;fun begins here...
...
now:
transfer_win64: ;stuffed in hash if caller is Win64
ret ;control goes back to function
now, as for the bytecode:
I am now thinking, given I would be JIT'ing it anyways, I may as well just
use a slightly higher-level TAC-based bytecode (and forsake all this
pseudo-x86 flexible-mode crap...), and essentially then use my existing full
codegen to do the JIT (rather than some braindead JIT).
this way, when the codegen is compiling to bytecode, it is essentially just
sort of operating in a "dummy" mode, more or less producing output it,
itself, would later accept and compile...
granted, this TAC bytecode could be interpreted as well, hmm...
granted, it would still be fairly low-level:
register-based, ...
I may make bytecode then be the sole option for cross-OS binary code...
Maxim S. Shatskih
> "unstable" wrapper, which rather than simply calling the function (or even
> simply converting arguments), will attempt to determine the calling
> convention of the caller, and handle as appropriate.
Just mandate the correct calling convention using the compiler default, or, even better, provide it explicitly with __stdcall or __cdecl keyword.
Another issue of interest is the field alignment for the structures used in the API.
--
Maxim S. Shatskih
Windows DDK MVP
ma...@storagecraft.com
http://www.storagecraft.com
BGB / cr88192
"BGB / cr88192" <cr8...@hotmail.com> wrote in message
news:h9g97k$fqq$1...@news.albasani.net...
>
>
> ;rax, r10, and r11 are safe in both conventions
> foo:
> mov rax, [rsp] ;return RIP
> call transfer_dispatch
> lea rax, ... ;actual function entry point (also used for target
> identification)
> jmp rax
>
> transfer_dispatch:
> mov r10, rax
> imul r10, 65521
> shr r10, 16
>
> and r10, 4095
> mov r11, [transfer_hash+r10*8]
> and r11, r11
> jnz transfer_go
>
correction:
xor r11, rax
jz transfer_go
'transfer_go' then grabs the target from another table.
>
> transfer_go:
mov r11, [transfer_hash_dst+r10*8]
jmp r11
Maxim S. Shatskih
> a "general" solution if cross-platform Windows/Linux code is to be under
> consideration.
I think that the task of having a cross-platform (Linux/Win32) _binary_ is unsolvable.
James Harris
wrote:
> > granted, producing code which uses SysV on Linux is an option, but it is not
> > a "general" solution if cross-platform Windows/Linux code is to be under
> > consideration.
>
> I think that the task of having a cross-platform (Linux/Win32) _binary_ is unsolvable.
Not sure what you mean but that cannot be the case as stated, can it?
A layer of middleware would do it. What matters is preventing apps
from bypassing the middleware layer.
As for the _binary_ part dynamic linking is an option.
James
Maxim S. Shatskih
> A layer of middleware would do it. What matters is preventing apps
> from bypassing the middleware layer.
For me, the goal does not justify the means.
_Source_ compat across Win32/Linux is doable, but what is the need in loading a Win32 binary in Linux?
More so, Wine is already here, and full support of loading Win32 binaries in Linux will be Wine remake :-) this is the "layer of middleware" you want.
And, if we are speaking about the source, then just note that a) structure alignment b) calling convention of all functions, callbacks included - are parts of any API contract.
BGB / cr88192
yes.
basically, as I am imagining it there would be several major parts:
a "native" part, which would be using good old SysV and friends;
an "island", which would essentially fake some portions of Windows, but
would by no means be a Windows clone. most of the rest is stuff provided
directly by the "framework".
it would mostly be to limit having to recompile stuff.
granted, it should all work fairly well for 32-bit x86.
the problem then is for x64 / x86-64, where Windows and Linux diverge on
many fronts...
hence, I am starting to almost think that bytecode will be a better option:
compile once, run multiple places.
granted, I want a bytecode which effectively runs C, and for other reasons,
I am keeping the bytecode at a "moderate" to "low" level of abstraction.
hence, at this level, it would not be anything like .NET bytecode or even
JBC.
but, the exact level of abstraction is a harder issue.
I am torn between a lower level "machine-like" approach (fixed number of
virtual fixed-sized registers, more-or-less untyped operations, ...);
a slightly higher level, slightly more abstract model, where operations are
properly and statically typed.
there are benefits and costs in both cases.
the cost is that, the higher up the chain of abstraction one moves, the ever
more responsibilities are put on the low-level codegen or interpreter...
(and also the more risk there is of things changing...).
I like things being a lot more solid than this...
Rod Pemberton
>
> how about all non-native function pointers are essentially "wrapped" with
an
> "unstable" wrapper, which rather than simply calling the function (or even
> simply converting arguments), will attempt to determine the calling
> convention of the caller, and handle as appropriate.
>
Ok, so for dynamicly called functions, you're suggesting a special
prolog/epilog wrapper to identify the needed calling convention, then setup
and call the appropriate function via the correct calling convention: SysV
or Win64. It seems from the snipped comments that identifying which will be
used isn't that easy. Yes, there will have to be a table or something that
gets patched with corrections.
> the issue then is for the possibility of MSVC compiled code, which will
not
> preserve this level of information (at this point, I am at a loss as to
how
> to automatically glue together MSVC compiled code and "native" code in
> Linux-land...).
Ugh! Why ARE you doing that? Wouldn't it be easier just to not use MSVC
and it's code? I.e., use GCC only, for both? Or, vice-versa?
> which, conviniently, sidesteps the whole auto-thunking issue (and it is,
Ok, not that I spell it correctly all the time either, but hopefully
convincingly, it's "conveniently"...
> auto-thunking issue
What's wrong with the programmer specifying the convention via a keyword or
pragma? The function calls on Win64 aren't going to work the same as on
SysV even if named the same. So, I don't understand why you need
"auto-thunking" for dynamicly called functions... What are you trying to
do? Are you trying to compile one piece of code for two platforms, where it
calls the same DLL function, where the DLL library is compiled in different
formats? Why is the main program in one format and the DLL in another?
Rod Pemberton
Rod Pemberton
>
> a "general" solution if cross-platform Windows/Linux code is to be under
> consideration.
>
Does Agner Fog's "Object file converter" help? How does he manage to get
code for one platform to work for the other?... Isn't this the same issue?
"This utility can be used for converting object files between COFF/PE, OMF,
ELF and Mach-O formats for all 32-bit and 64-bit x86 platforms. Can modify
symbol names in object files. Can build, modify and convert function
libraries across platforms. Can dump object files and executable files. Also
includes a very good disassembler supporting the SSE4, AVX, FMA and XOP
instruction sets. Source code included (GPL). Manual.
File name: objconv.zip, size: 710609, last modified: 2009-Jul-16."
http://www.agner.org/optimize/#objconv
Rod Pemberton
BGB / cr88192
"Maxim S. Shatskih" <ma...@storagecraft.com.no.spam> wrote in message
news:h9gnec$178k$1...@news.mtu.ru...
> how about all non-native function pointers are essentially "wrapped" with
> an
> "unstable" wrapper, which rather than simply calling the function (or even
> simply converting arguments), will attempt to determine the calling
> convention of the caller, and handle as appropriate.
<--
Just mandate the correct calling convention using the compiler default, or,
even better, provide it explicitly with __stdcall or __cdecl keyword.
-->
does not work on x86-64.
for x86, everything is cdecl, so no problem there...
for x86-64, I would need __win64 and __sysv, which for obvious reasons do no
good with either MSVC or GCC...
my compiler could figure it out, but it is not for code compiled in my
compiler where I really need to figure this out (since, for these cases, it
already figures it out).
so, the main area for concern is function pointers which may be called from
code compiled via GCC, but in which I am not certain.
however...
keywords could, at least, partially resolve the issue of function pointers
in structures, but still leaves the problem of fetching function pointers
via API calls...
<--
Another issue of interest is the field alignment for the structures used in
the API.
-->
simple point of note:
consider it "unwise" to make any structures where the size will vary between
OS's...
in a practical note, this means staying well clear of "long" on x86-64.
it actually is an issue in my compiler even on Windows, since MSVC uses a
32-bit long, but my compiler uses a 64-bit long (actually, for technical
reasons I have been tempted by making this the case even in 32-bits, mostly
as Java and C# also use a 64-bit 'long').
otherwise, much like calling convention, sizeof(long) would end up varying
with CPU, language, and OS, which is a messy situation...
hence, "long" is not a safe type, but oh well, I don't mind...
luckily, besides 'long', sizing and alignment are acceptably well agreed
on...
ok, there "is" short, but either the reference I have is wrong, or my
compiler disagrees with MSVC, as I see no benefit of using 32-bit alignment
for a 16-bit type (after all, this would defeat the whole point of using
short...).
in practice, I have not run into issues as of yet, but then again, I tend
not to use 'short' so much in structs either...
BGB / cr88192
"Rod Pemberton" <do_no...@nohavenot.cmm> wrote in message
news:h9gscs$pd0$1...@aioe.org...
> "BGB / cr88192" <cr8...@hotmail.com> wrote in message
> news:h9g97k$fqq$1...@news.albasani.net...
>>
>> a "general" solution if cross-platform Windows/Linux code is to be under
>> consideration.
>>
>
> Does Agner Fog's "Object file converter" help? How does he manage to get
> code for one platform to work for the other?... Isn't this the same
> issue?
>
would not help with 64-bit code, since you would essentially have to rewrite
the code in the conversion process...
granted, all this is mostly specific to Win64 and Linux x86-64, since it
just so happens that Win32 and Linux x86 generally use the same calling
convention (cdecl, except for the occasional 'stdcall' but we can ignore
this, as it plagues Windows code just as bad as cross-platform code...).
as far as getting the code loaded, I have my own PE/COFF loader, which would
be used to get DLLs loaded on Linux.
the problem is, however, getting DLL's to be able to use anything not
declared in DLL's (or, for that matter, to get native Linux code to be able
to make use of the content loaded in from DLL's).
the natural tendency would be for the DLL's to form their own closed
ecosystem, which may be undesirable for other reasons...
in fact, it would be much easier to have the code simply operate in "fake
Windows land" than to integrate in a tight-knit manner with "native" code...
so, getting the code to run is not the problem, but getting it to
interoperate is...
> "This utility can be used for converting object files between COFF/PE,
> OMF,
> ELF and Mach-O formats for all 32-bit and 64-bit x86 platforms. Can modify
> symbol names in object files. Can build, modify and convert function
> libraries across platforms. Can dump object files and executable files.
> Also
> includes a very good disassembler supporting the SSE4, AVX, FMA and XOP
> instruction sets. Source code included (GPL). Manual.
>
> File name: objconv.zip, size: 710609, last modified: 2009-Jul-16."
>
> http://www.agner.org/optimize/#objconv
>
you can convert the files, but can you rewrite the code?...
that is the bigger issue.
as I see it, this tool is not addressing the correct issue...
>
> Rod Pemberton
>
>
BGB / cr88192
"Rod Pemberton" <do_no...@nohavenot.cmm> wrote in message
> "BGB / cr88192" <cr8...@hotmail.com> wrote in message
> news:h9g5d0$9io$1...@news.albasani.net...
>>
>> how about all non-native function pointers are essentially "wrapped" with
> an
>> "unstable" wrapper, which rather than simply calling the function (or
>> even
>> simply converting arguments), will attempt to determine the calling
>> convention of the caller, and handle as appropriate.
>>
>
> Ok, so for dynamicly called functions, you're suggesting a special
> prolog/epilog wrapper to identify the needed calling convention, then
> setup
> and call the appropriate function via the correct calling convention: SysV
> or Win64. It seems from the snipped comments that identifying which will
> be
> used isn't that easy. Yes, there will have to be a table or something
> that
> gets patched with corrections.
>
yep.
although, it will be via disjoint "transfer thunks" rather than via
prologues or epilogues (these are used by the exception-handling mechanisms,
but not for transfer mechanics).
in this case, the "transfer thunk" would be referenced whenever attempting
to retrieve the function pointer (possibly statically, but more likely,
dynamically and via the runtime...).
>> the issue then is for the possibility of MSVC compiled code, which will
> not
>> preserve this level of information (at this point, I am at a loss as to
> how
>> to automatically glue together MSVC compiled code and "native" code in
>> Linux-land...).
>
> Ugh! Why ARE you doing that? Wouldn't it be easier just to not use MSVC
> and it's code? I.e., use GCC only, for both? Or, vice-versa?
>
GCC on Win64 is still a "work in progress"...
although, granted, it is probably a little better now, but last I looked it
was bughappy...
otherwise, it would mean using purely my own compiler...
as-is, I trust MSVC's stability a little more than my compiler's, and so
trying to use my compiler "seriously" would probably force me to fix wave
after wave of bugs...
>> which, conviniently, sidesteps the whole auto-thunking issue (and it is,
>
> Ok, not that I spell it correctly all the time either, but hopefully
> convincingly, it's "conveniently"...
>
spelling is not always my strong area...
>> auto-thunking issue
>
> What's wrong with the programmer specifying the convention via a keyword
> or
> pragma? The function calls on Win64 aren't going to work the same as on
> SysV even if named the same. So, I don't understand why you need
> "auto-thunking" for dynamicly called functions... What are you trying to
> do? Are you trying to compile one piece of code for two platforms, where
> it
> calls the same DLL function, where the DLL library is compiled in
> different
> formats? Why is the main program in one format and the DLL in another?
>
probably, I would be compiling the DLL's on Windows, and using them on both
OS's...
on Linux, however, the app, and core facilities, would be compiled using
using ELF64 and SysV and similar.
hence, there will be a mismatch...
the only other real option, would be to essentially compile damn near the
whole damn app on Windows (or, a faked Windows), and essentially run in a
vaguely Wine-like manner (or, the Windows equivalent: Cygwin...).
however, this is an ugly mess, since it would essentially make a Linux port
a high-maintainence task (one not only needs to keep the "core" up and
running, but an entire complement of libraries and tools as well...).
the other extreme, is that I don't bother loading DLL's on Linux, and
instead require all code to be compiled on Linux and/or using SysV...
>
> Rod Pemberton
>
>
Rod Pemberton
>
> as-is, I trust MSVC's stability a little more than my compiler's, and so
> trying to use my compiler "seriously" would probably force me to fix wave
> after wave of bugs...
>
Ah.
Well, if MSVC is your current preference, I think you should work from that
perspective to find a solution.
> probably, I would be compiling the DLL's on Windows, and using them on
> both OS's...
>
> on Linux, however, the app, and core facilities, would be compiled using
> using ELF64 and SysV and similar.
>
> hence, there will be a mismatch...
>
> the only other real option, would be to essentially compile damn near the
> whole damn app on Windows (or, a faked Windows), and essentially run in a
> vaguely Wine-like manner (or, the Windows equivalent: Cygwin...).
>
> however, this is an ugly mess, since it would essentially make a Linux
> port
> a high-maintainence task (one not only needs to keep the "core" up and
> running, but an entire complement of libraries and tools as well...).
What about writing your library to work on top of an existing cross-platform
library or libraries? I.e., let someone else figure out the platform
specifics, you use the standardized API(s). I know the following links are
a limited subset of needed functionality and they apply more to applications
that perhaps a DLL, but it may be a place to start.
libSDL (cross-platform mouse, keyboard, graphics, sound)
http://www.libsdl.org
Applications, like Ethereal, have been written to work on Windows and *nix.
Ethereal
http://www.ethereal.com
Ethereal's "about" lists:
Glib (cross-platform, low level non-GUI and lots of misc and OS related.)
GTK+ (cross-platform GUI object toolkit)
WinPcap (port of libpcap, low-level networking)
libz (data compression)
libpcre (PERL regex library)
Net-SNMP (email)
ADNS (DNS resolver)
GTK+, Glib, WinPcap/libpcap might be useful. These three plus SDL should be
sufficient. (?)
Anyway, lots of stuff you've talked about is listed on Wikipedia's Glib
page:
http://en.wikipedia.org/wiki/GLib
Rod Pemberton
BGB / cr88192
"Rod Pemberton" <do_no...@nohavenot.cmm> wrote in message
> "BGB / cr88192" <cr8...@hotmail.com> wrote in message
> news:h9gvfe$ihc$1...@news.albasani.net...
>>
>> as-is, I trust MSVC's stability a little more than my compiler's, and so
>> trying to use my compiler "seriously" would probably force me to fix wave
>> after wave of bugs...
>>
>
> Ah.
>
> Well, if MSVC is your current preference, I think you should work from
> that
> perspective to find a solution.
>
I use gcc typically when building on Linux, and MSVC on Win64, mostly as
GCC-Win64 is not exactly working so well as of yet...
I had been using MinGW on Win32, but my makefiles are probably out of date
again...
<snip>
>>
>> however, this is an ugly mess, since it would essentially make a Linux
>> port
>> a high-maintainence task (one not only needs to keep the "core" up and
>> running, but an entire complement of libraries and tools as well...).
>
> What about writing your library to work on top of an existing
> cross-platform
> library or libraries? I.e., let someone else figure out the platform
> specifics, you use the standardized API(s). I know the following links
> are
> a limited subset of needed functionality and they apply more to
> applications
> that perhaps a DLL, but it may be a place to start.
>
this totally misses the point...
basically, if I were to actually allow a harsh barrier between the native
code and the "island", it would be necessary to supply (within the island) a
large number of core libraries (as DLLs).
for example:
a C runtime (such as newlib), as well as any other core libraries;
wrapper libraries to allow much of anything to interface with the outside
world;
...
it would be, in all, a similar issue to that addressed by Wine:
writing a whole bunch of DLLs which operate within their own "layer", and
which then interface with the underlying Linux system.
this would also create a large maintainence problem, as well as greatly
increase the total project size (as I would have to distribute special-built
versions of various libraries, ...).
my problem is not, in the first place, a matter of APIs, rather, it is a
problem that goes right down to the level of the machine code...
all this would be a horrid PITA, just the sort of thing I originally
designed my VM to avoid.
so, faced with these terrible prospects, I went and implemented SysV.
now, for the original goal of cross-OS binaries, I am looking into a
different route: bytecode.
or, more correctly, right now I am looking into the route of doing something
more extreme:
a full x86 interpreter.
this will be similar to an emulator, but would only do a subset of x86
(32-bit userspace / flat + ring3).
yesterday and this morning, I had focused mostly on core machinery and a
"big switch", but it can be quickly realized that this is a very difficult
route (involves several layers of such big switches).
I thought about possible "easier" ways to do this, and another thing which
comes to mind was to start adapting some of the machinery from my
assembler/disassembler.
this allows a possible "shortcut" of partially "disassembling" the opcode
(into an internal form), and then making use of much more generic logic (as
another advantage, it also allows for more informative messages for
simulated #UD and #GP exceptions...).
however, I am also quickly realizing that with everything, the interpreter
is likely to be very slow (at least 100x or more slowdown...).
a proper bytecode would be less slow (less twiddling), but a bytecode would
still require a lot of work (beating support into the compiler and
assembler, writing an interpreter, ...).
<snip>
>
> GTK+, Glib, WinPcap/libpcap might be useful. These three plus SDL should
> be
> sufficient. (?)
>
> Anyway, lots of stuff you've talked about is listed on Wikipedia's Glib
> page:
> http://en.wikipedia.org/wiki/GLib
>
not really...
this is all about source-level / porting issues.
this is not my problem.
the issue is actually about binary compatibility.
generating code which is, at the same time, binary compatible with both
Windows and Linux, would be, problematic.
or, at least for 64-bit code, 32-bit is not so bad as both Win32 and Linux
use cdecl.
now, as for my source, this usually does build ok on both OS'es (apart from
endless fiddling, mostly with stale makefiles, ...). my usual approach is
mostly to have machinery which is swapped out (typically via whole source
files, or often via '#ifdef'...).
Rod Pemberton
>
> this totally misses the point...
It's that time of year...
> so, faced with these terrible prospects, I went and implemented SysV.
Bravo!
> now, for the original goal of cross-OS binaries, I am looking into a
> different route: bytecode.
Sadly, I've begun considering an interpreter because I just don't want to
deal with x86-64...
> or, more correctly, right now I am looking into the route of doing
something
> more extreme:
> a full x86 interpreter.
Lot's of work ahead... Look at: QEMU, or Scitech Soft's x86emu. This is
still for MSVC code on Linux? You want something like WoW64, but for Linux,
but able to run MSVC code like a port Windows WoW64 to Linux. QEMU? KQEMU?
> yesterday and this morning, I had focused mostly on core machinery and a
> "big switch", but it can be quickly realized that this is a very difficult
> route (involves several layers of such big switches).
Nested big switches: the only useful place in C for a goto...
> I thought about possible "easier" ways to do this, and another thing which
> comes to mind was to start adapting some of the machinery from my
> assembler/disassembler.
>
> this allows a possible "shortcut" of partially "disassembling" the opcode
> (into an internal form), and then making use of much more generic logic
(as
> another advantage, it also allows for more informative messages for
> simulated #UD and #GP exceptions...).
I can't help you here, my very incomplete x86 interpreter is very brute
force, i.e., minimal instruction encode/decode logic and a few big switches.
> a proper bytecode would be less slow (less twiddling), but a bytecode
would
> still require a lot of work (beating support into the compiler and
> assembler, writing an interpreter, ...).
Can't you use Java VM? Both environments will have Java installed. JVM,
JBC, Java DLL's, etc. I.e., no extra DLLs required.
Java assembler:
http://web.archive.org/web/20000621000830/http://www.sbktech.org/jas.html
> the issue is actually about binary compatibility.
The "Linux Unified Kernel" supposedly allows both Linux and Windows binaries
to run, but not 64-bit...
Anyway, you have access to the Linux code which should do what you need, if
you want GCC compiled code to run on Windows. But, MS code which does what
you want on Linux is closed source. Binary compatibility between
differently structured OSes is difficult. Similarly structured OSes, say
Linux and BSD, allows a simple conversion layer. e.g., lxrun allows Linux
binaries to run on similarly structured *nix OSes.
The best cross-platform execution of an application I've seen was the FX!32
emulation for Dec Alpha's - which I'm likely to have mentioned previously.
It was really, really slow at first, but after a few runs the applications
ran really quick. x86 Emulation + Binary Translation + Binary Optimization
= fast native application... Oh, you thought my comment to compile to
SUBLEQ, use a SUBLEQ-to-x86 binary translator, followed by an x86 binary
optimizer, on comp.lang.misc was joke, didn't you? ;)
QEMU works, albeit not so fast, but not absurdly slow either. KQEMU?
> generating code which is, at the same time, binary compatible with both
> Windows and Linux, would be, problematic.
...
Rod Pemberton
BGB / cr88192
"Rod Pemberton" <do_no...@nohavenot.cmm> wrote in message
> "BGB / cr88192" <cr8...@hotmail.com> wrote in message
> news:h9on2c$dn8$1...@news.albasani.net...
>>
>> this totally misses the point...
>
> It's that time of year...
>
>> so, faced with these terrible prospects, I went and implemented SysV.
>
> Bravo!
>
yep...
it was not as bad as I would have thought...
partly makes me wonder why I have spent so much time off going and trying to
get TAC and SSA working...
still faced with the idea of cross-platform binary portability though.
>> now, for the original goal of cross-OS binaries, I am looking into a
>> different route: bytecode.
>
> Sadly, I've begun considering an interpreter because I just don't want to
> deal with x86-64...
>
yeah, it is all a bit of a pain, and one quickly finds that the further one
goes the harder things get.
had I known everything in advance, I would have probably taken a route
similar to the Java VM.
bytecode and interpretation to make it work, JIT to make it faster, and
magic glue code to glue things together...
if anything, targetting a (single) bytecode is a lot less work than
targetting multiple architectures.
my original idea was "well, I can just rewrite the lower end of the compiler
for each possible target". alas, it did not work out this way...
>> or, more correctly, right now I am looking into the route of doing
> something
>> more extreme:
>> a full x86 interpreter.
>
> Lot's of work ahead... Look at: QEMU, or Scitech Soft's x86emu. This is
> still for MSVC code on Linux? You want something like WoW64, but for
> Linux,
> but able to run MSVC code like a port Windows WoW64 to Linux. QEMU?
> KQEMU?
>
with 32-bit x86, I have a lot more options WRT the compiler (gcc works, MSVC
works, other options work, ...).
I am beating together the interpreter quickly enough (errm... maybe... still
several days and it doesn't quite interpret anything as of yet, but using
its internal structures it is working like a crude disassembler...), but, it
is probably going to be "teh slow".
but, alas, getting it beaten together is more important than optimizing...
>> yesterday and this morning, I had focused mostly on core machinery and a
>> "big switch", but it can be quickly realized that this is a very
>> difficult
>> route (involves several layers of such big switches).
>
> Nested big switches: the only useful place in C for a goto...
>
this also starts to turn very ugly very quickly...
x86 may be too complicated for this strategy...
>> I thought about possible "easier" ways to do this, and another thing
>> which
>> comes to mind was to start adapting some of the machinery from my
>> assembler/disassembler.
>>
>> this allows a possible "shortcut" of partially "disassembling" the opcode
>> (into an internal form), and then making use of much more generic logic
> (as
>> another advantage, it also allows for more informative messages for
>> simulated #UD and #GP exceptions...).
>
> I can't help you here, my very incomplete x86 interpreter is very brute
> force, i.e., minimal instruction encode/decode logic and a few big
> switches.
>
yep.
well, I ended up sort of doing this, mostly reworking logic from my
disassembler (and a few other places), and modifying it into interpreter
logic.
actually, it is more of a 'disassembler' than my actual disassembler, since
it actually more closely inverts the logic of my assembler (converting
opcodes into structures), and then re-dispatching them according to generic
form.
actually, it is expanding into the same sort of logic-tree structure as used
in my compiler.
so, as oposed of stepping through stuff simply to format and display the
results, now it puts the info into structures and sets up other things.
as an odd bit of trivial, the instruction decoding is capable of the full
instruction set, as well as 16-bit and 64-bit addressing modes, ... however,
the interpreter will not likely deal with all this (instead likely limiting
itself to 32-bit userspace).
>> a proper bytecode would be less slow (less twiddling), but a bytecode
> would
>> still require a lot of work (beating support into the compiler and
>> assembler, writing an interpreter, ...).
>
> Can't you use Java VM? Both environments will have Java installed. JVM,
> JBC, Java DLL's, etc. I.e., no extra DLLs required.
>
> Java assembler:
> http://web.archive.org/web/20000621000830/http://www.sbktech.org/jas.html
>
Java is not C, however...
the JVM is also not capable of effectively handling C, or doing a lot of
other stuff I would want from it, and hacking this stuff on is not so good
either.
however, the JVM has influenced many other parts of my framework.
I also have a JBC interpreter floating around, ...
>> the issue is actually about binary compatibility.
>
> The "Linux Unified Kernel" supposedly allows both Linux and Windows
> binaries
> to run, but not 64-bit...
>
was messing with this, but it does not seem to be working so well in my
case...
then again, I am also using it on an Asus EEE, and maybe it would be better
suited for a more full-featured computer.
pure Wine should work better on a 64-bit install, which is what I have on my
main computer, but I have not gone and done so as of yet (LUK I guess does
not yet support 64-bits, but Wine can do 32-bit Windows on 64-bit Linux...).
> Anyway, you have access to the Linux code which should do what you need,
> if
> you want GCC compiled code to run on Windows. But, MS code which does
> what
> you want on Linux is closed source. Binary compatibility between
> differently structured OSes is difficult. Similarly structured OSes, say
> Linux and BSD, allows a simple conversion layer. e.g., lxrun allows Linux
> binaries to run on similarly structured *nix OSes.
>
I have enough documentation, and I don't care about MS's closed-source
portions.
basically, I am making part of my codebase run on top of itself, where it
would exist as several major portions:
a native portion, which runs directly on the host OS;
a virtual portion, which runs on top of the native portion.
from the POV of the virtual portion, the native portion will about resemble
the OS, and in the case of the interpreter, this similarity would be more
severe, as basically a simulated 32-bit process would be running inside a
likely 64-bit world.
granted, the interpreter is being designed to allow x86-32 on x86-32, or
x86-32 on non-x86, but on a 32-bit system the space would be a bit more
cramped (effective operation on a 32-bit system would likely involve using a
full MMU and a page-file).
granted, as-is, I am using a sort of partial "fake MMU" strategy, where
basically it is using a virtual address space, but I am currently doing it
via spans rather than page tables.
page tables may later be an option...
initially, it was the design of my MMU logic which gave me performance
doubts, but then I started on the interpreter, which proceeded to give me
much bigger doubts.
> The best cross-platform execution of an application I've seen was the
> FX!32
> emulation for Dec Alpha's - which I'm likely to have mentioned previously.
> It was really, really slow at first, but after a few runs the applications
> ran really quick. x86 Emulation + Binary Translation + Binary
> Optimization
> = fast native application... Oh, you thought my comment to compile to
> SUBLEQ, use a SUBLEQ-to-x86 binary translator, followed by an x86 binary
> optimizer, on comp.lang.misc was joke, didn't you? ;)
>
SUBLEQ: I don't really see how this would be practical...
then again, most of my optimization experience revolves around "big complex
logic trees..." (AKA: maybe a more-or-less "top-down" optimization strategy,
with some small amount of stateful bottom-up optimizations at the leaves).
partly, another weight in favor of my current interpreter design, even
though likely to be a very slow interpreter (the instruction decoding likely
being a major issue), is that it "should" make it easier to later modify it
into a JIT if needed (and it also raises the level of abstraction somewhat).
a big mass of switches, though probably faster, would be very one-off...
some idle thoughts have been the possible idea of a "decode cache", which
could help with loops (basically, recently decoded opcodes are cached, and
if still in cache are used directly) rather than decoded again (an
uncertainty though would be how to most effectively manage this cache,
thoughts are floating around variants of hash tables).
> QEMU works, albeit not so fast, but not absurdly slow either. KQEMU?
>
maybe, but this seems more to be a full system emulator, rather than an
interpreter...
for example, mine would not do, nor have much use for, kernel-mode stuff
(nor for HW emulation or anything else of the sort).
instead, everything "kernel" would be handled by the host app (such as
handling faults and interrupts, ...).
Rod Pemberton
>
> > The best cross-platform execution of an application I've seen was the
> > FX!32
> > emulation for Dec Alpha's - which I'm likely to have mentioned
previously.
> > It was really, really slow at first, but after a few runs the
applications
> > ran really quick. x86 Emulation + Binary Translation + Binary
> > Optimization
> > = fast native application... Oh, you thought my comment to compile to
> > SUBLEQ, use a SUBLEQ-to-x86 binary translator, followed by an x86 binary
> > optimizer, on comp.lang.misc was joke, didn't you? ;)
> >
>
> SUBLEQ: I don't really see how this would be practical...
>
The point was one can construct fast native code from a simple
representation. FX!32 produced fast Dec Alpha code from x86 instructions.
So, there is no reason one couldn't compile code to a much simpler turing
complete language, e.g., SUBLEQ or BrainFuck, and then produce very fast x86
from it. SUBLEQ and BrainFuck are portable everywhere. It shifts were the
work needs to be done to produce fast code and changes the areas of compiler
complexity. This may be better or may be worse. It won't solve your SysV
vs. Win64 issue.
> partly, another weight in favor of my current interpreter design, even
> though likely to be a very slow interpreter (the instruction decoding
likely
> being a major issue), is that it "should" make it easier to later modify
it
> into a JIT if needed (and it also raises the level of abstraction
somewhat).
>
> a big mass of switches, though probably faster, would be very one-off...
I ended up deciding on switches because 1) I already had a list of x86
instructions by completed first byte and/or second byte - i.e., no decoding
needed 2) shift operations and if() conditionals used to decode an
instruction take more time than the jump table usually produced for a
compiled switch 3) I only needed real mode instructions upto 386/486. I am
hoping for reasonable speed, if I ever get around to completing it.
> some idle thoughts have been the possible idea of a "decode cache", which
> could help with loops (basically, recently decoded opcodes are cached, and
> if still in cache are used directly) rather than decoded again (an
> uncertainty though would be how to most effectively manage this cache,
> thoughts are floating around variants of hash tables).
IIRC, all x86 instructions except control-flow instructions (jmp, jcc, call,
ret, iret, int) are position independent. E.g., it doesn't matter where you
execute "mov ax, 0x32" because the eip/ip isn't involved. I'm not sure if
this is true for x86-64 instructions. So, you can save them on a stack
until you need them again. Then, you can pull them off, write them to your
execution buffer, and execute them. You just need to filter out the
control-flow instructions from the execution stream and handle them
separately.
Rod Pemberton
BGB / cr88192
"Rod Pemberton" <do_no...@nohavenot.cmm> wrote in message
I am not really sure how one would do much with it though.
my intuition on the matter would be that it would be difficult to produce
code in this form, and that an optimizer would likely get stuck somewhere
well before even being able to make up for the added awkwardness and
verbosity.
TAC+SSA would be far preferable, or even RPN...
>> partly, another weight in favor of my current interpreter design, even
>> though likely to be a very slow interpreter (the instruction decoding
> likely
>> being a major issue), is that it "should" make it easier to later modify
> it
>> into a JIT if needed (and it also raises the level of abstraction
> somewhat).
>>
>> a big mass of switches, though probably faster, would be very one-off...
>
> I ended up deciding on switches because 1) I already had a list of x86
> instructions by completed first byte and/or second byte - i.e., no
> decoding
> needed 2) shift operations and if() conditionals used to decode an
> instruction take more time than the jump table usually produced for a
> compiled switch 3) I only needed real mode instructions upto 386/486. I
> am
> hoping for reasonable speed, if I ever get around to completing it.
>
the only lists I had was in the Intel doc PDF's, and this was awkward...
the code was getting really long really fast, and I noted that this thing
would be teh-huge before I even got a "reasonable" amount of the instruction
set implemented (I thought for a moment about just how many opcodes there
were in x86, and the relative rate of progress).
...
granted, the instruction decoder is likely to be teh-slow, since it was
based directly on my disassembler, which was itself not exactly designed for
raw speed (it does a reverse-lookup on the encoder tables to match the
opcode).
I have mostly been idly considering ideas to speed up the decoder
(hash-tables and chain-linking, ...). but have not come up with any "good"
strategy yet (issues of performance, complexity, and correctness).
another strategy would be to basically cache the decoded opcodes such that
having to re-decode them can be greatly reduced (however, I would have to
flush this cache in the case of self-modifying code).
this is most likely to help in the case of tight loops (and recursion), but
would not much help for long non-looping code (size overhead could also be
an issue).
another considered optimization (not strictly decoder related though), was
to decode certain sequences (such as 'rep movsb') as a single instruction,
mostly so that the interpreter can implement them via a memcpy or similar.
>> some idle thoughts have been the possible idea of a "decode cache", which
>> could help with loops (basically, recently decoded opcodes are cached,
>> and
>> if still in cache are used directly) rather than decoded again (an
>> uncertainty though would be how to most effectively manage this cache,
>> thoughts are floating around variants of hash tables).
>
> IIRC, all x86 instructions except control-flow instructions (jmp, jcc,
> call,
> ret, iret, int) are position independent. E.g., it doesn't matter where
> you
> execute "mov ax, 0x32" because the eip/ip isn't involved. I'm not sure if
> this is true for x86-64 instructions. So, you can save them on a stack
> until you need them again. Then, you can pull them off, write them to
> your
> execution buffer, and execute them. You just need to filter out the
> control-flow instructions from the execution stream and handle them
> separately.
>
my issue though is more how to avoid having to re-decode the same
instructions, rather than how to pipeline them (as is, I am not sure if
pipelining would help much anyways).
some idle thoughts:
a hash table, where this would then basically hash a few pieces of info, and
check the hash a few times for the requested opcode, before falling back to
decoding the instruction. this would be fairly simple and fairly fast, but
would have a strict limit to its effectiveness (mostly related to hash table
size and max number of cached opcodes, which is mostly an issue as my
decoded opcode struct is a little large, ~32 bytes at present).
a binary tree, where this can scale a little nicer, but is not likely to be
as fast, and would have a larger memory overhead.
another idle though was "tracing", where basically streams of opcodes are
decoded and "pre-cooked" (into an internal bytecode or JITted), and then
when jumping to one of these locations, this stream is executed (falling
back to more generic mechanisms mostly during calls or jumps). the great
enemy of this case is self-modifying code, which would essentially destroy
any cached streams.
an acceptable optimization though would be to allow making executable areas
read-only, and all other areas non-executable, which could allow more freely
using such a strategy.
however, as-is, the higher priority is that of making it work acceptably,
rather than trying to make it fast...
>
> Rod Pemberton
>
>
>
Rod Pemberton
> >> >
> >> > compile to
> >> > SUBLEQ, use a SUBLEQ-to-x86 binary translator, followed by an x86
> >> > binary optimizer
> >>
> >
> > The point was one can construct fast native code from a simple
> > representation.
>
> my intuition on the matter would be that it would be difficult to produce
> code in this form, and that an optimizer would likely get stuck somewhere
> well before even being able to make up for the added awkwardness and
> verbosity.
>
> TAC+SSA would be far preferable, or even RPN...
Well, my thoughts are if you've got C-to-assembly, and
assembly-to-"whatever," you can then can translate and optimize "whatever".
E.g., the FRAK assembler assembles instructions for a virtual cpu (8-bit) to
BrainFuck. So, if you implemented a FRAK assembly backend for a C compiler,
then you could compile C to FRAK and FRAK to BrainFuck. In this case, FRAK
is useless as it's 8-bits and a virtual cpu. But, the basic
proof-of-concept is there. The basic proof of concept for simplified C into
SUBLEQ is here:
http://mazonka.com/subleq/hsq.html
Of course, that still leaves translation to x86 and optimization... A
working example of non-native-to-highly optimized native instruction
conversion was done by FX!32. Once you have the functional essence of a
program represented, it's possible to convert it to other forms, e.g., C to
an AST, albeit differently.
> my issue though is more how to avoid having to re-decode the same
> instructions, rather than how to pipeline them (as is, I am not sure if
> pipelining would help much anyways).
How are you defining ""same instructions""? Do you mean each, say "popa",
instruction no matter where or when it occurs? Or, do you mean a block of,
perhaps the last 20, executed instructions? Even with only a 4KB buffer you
could store alot of either... It's a no brainer to do the no decode
instructions: one byte or two byte fixed size, no offsets, no sib, no modrm.
You can't completely decode instructions with offsets and branches. e.g.,
you could decode the "mov" part of "mov 0x32" and "mov eax, 0x64" and store
that. But, you might not want to store a full decode with 0x32 or 0x64.
With a 64KB buffer, you should be able to handle most of the 32-bit x86
instruction set (based on FX!32 which did the entire 16-bit set in 64k...).
IIRC, 32-bit basically adds just the sib addressing extensions... (?)
Rod Pemberton
BGB
compiling to x86 and interpreting this works, no need for more
funkiness. it would have been "enough" extra just to support a
conventional bytecode...
>> my issue though is more how to avoid having to re-decode the same
>> instructions, rather than how to pipeline them (as is, I am not sure if
>> pipelining would help much anyways).
>
> How are you defining ""same instructions""? Do you mean each, say "popa",
> instruction no matter where or when it occurs? Or, do you mean a block of,
> perhaps the last 20, executed instructions? Even with only a 4KB buffer you
> could store alot of either... It's a no brainer to do the no decode
> instructions: one byte or two byte fixed size, no offsets, no sib, no modrm.
> You can't completely decode instructions with offsets and branches. e.g.,
> you could decode the "mov" part of "mov 0x32" and "mov eax, 0x64" and store
> that. But, you might not want to store a full decode with 0x32 or 0x64.
> With a 64KB buffer, you should be able to handle most of the 32-bit x86
> instruction set (based on FX!32 which did the entire 16-bit set in 64k...).
> IIRC, 32-bit basically adds just the sib addressing extensions... (?)
>
the current pattern is to decode instructions per virtual address
(CS:EIP), as it now does partial segmentation.
it doesn't "merge" instructions as this would likely cost more time than
it saves,however, a static translator could do this.
note that it decodes all instructions into an internal struct, even
trivial ones, and the opcode number is internal (assigned by some of the
tools which automatically process the listing and generate a lot of the
code).
more efficient decoding could be done though by doing additional
processing on the listing, essentially allowing a per-byte lookup rather
than a pattern-matching lookup.
FWIW: I did partly add the 'rep movsvb' and friends special cases...
or such...
>
> Rod Pemberton
>
>