Project structure changed

[Petr Kobalíček]

Hi devs,

I'd like to inform all of you that the project structure (file naming)
and root directories in svn have been changed to use lowercase. I'm
doing this change in all my projects to follow more consistent file
naming and compatibility.

This means that if you use SVN it's important to do a new checkout
since trunk/AsmJit has been changed to trunk/asmjit.

This change will be included in next beta version as well.

Best regards
Petr Kobalicek

[Razvan Aciu]

Hi,

I want to use AsmJit to compile each function in its own segment, like this:

map<string,void(*)()> compiledFunctions; //name,executable code
void compile(Function &func)
{
X86Assembler a86;
//... compile func into a86 ...
executableMemFree(compiledFunctions[func.name]);
compiledFunctions[func.name]=executableMemNewCopy(a86.getCodeSize(),a86.getCode());
fixDependencies(func.name);
}

If I have to compile a new function, this should be done in its own
executable segment (kept in compiledFunctions) because I don't want to
recompile again all the other functions in one big segment. In the same
time, a function can have multiple versions, for example in a REPL
interpreter command line where the user can enter again and again the same
function with different code.

For this probably I need to tell to AsmJit that the immediate operands can
be "external labels", much like the ordinary Label, and if I use such
"external label", AsmJit can put its occurence in the generated code in a
relocation list, which I can use later, something like this:

map<string,ExternalRelocations> relocations;

void compile(Function &func)
{
//...same code as above, possibly including:

//ExternLabel *eLabel=a86.newExternLabel("a_function_name");
//a86.call(eLabel);

relocations[func.name]=a86.externalRelocations;
}

void fixDependencies(const string &name)
{
void(*funcAddr)()=compiledFunctions[name];
//Algorithm:
// 1. iterate all functions
// 1.1 for each function iterate its relocations list
// 1.2 update all relocations named name with funcAddr
}

Is this scenario possible now with AsmJit, or support can it can be
added...maybe in another form?

Thank you,
Razvan Aciu

[Petr Kobalíček]

Hi Razvan,

the feature you described is definitely nice and I think that AsmJit
should support it. However, I'm not sure about the current status;
most of the time I focused on Compiler (and I'm at this time rewriting
main part of it to fix some bugs and make it better/more lightweight).

Recently I introduced a concept of Runtime (previous name was
Context/JitContext, this was removed or will be, I'm not sure whether
it has been committed). Now it's quite simple interface, but this is
the class where the problem should be solved in a global way (I mean
mainly patching).

I introduced it mainly to allow something like "symbols", so the code
inside one code segment (or just one generated function) can link to
others, possibly not yet generated symbols. I have to say that it's
not implemented yet and I'm still thinking about the best design,
because ideally if we have good interface we can generate not only
JIT, but it should be possible to generate an executable/shared
library by this way.

But to be in reality, we should start with a simple example of
generating, let's say two functions, and creating a solution to fix
the links to a function(s) when one of them is being recreated. There
is currently no API to fetch code relocations, so a good start would
be to make all relocations accessible through a public API and make
them visible to Runtime class which is on the top.

What do you think?

Best regards
Petr Kobalicek

[Razvan Aciu]

I also think the best solution is the one which allows future
developments...these are always required :)

In our case, I think a good solution is to implement some kind of linker, as
in this model (top-down):

//a class which holds all the segments and has ownership over them
class Binary{
- flags; //processor type, architecture, ...
- segments; //collection of segments

Segment *createSegment(); //create an empty segment
//create a new segment from the relevant code and symbols from assembler
//internal symbols like Label, used only inside that code, are not
necessary to be taken
//instead, an Assembler must define special kinds of Labels, which acts
like:
//Symbols: with a bound address inside that code
//References: with an address outside the code
void createSegmentFrom(Assembler &assembler);
void erase(Segment *s); //erase a segment

void linkAll(); //link all the symbols from all segments
//considers all de symbols from s as possibly modified and relocate them
in all other segments
void link(Segment *s);
Segment *findSegmentBySymbol(char *symName); //finds the Segment
which defines symName
};

//has ownership over symbols, references, data, exeCode
class Segment{
- flags; //data/code, read_only, initialized, ...
- symbols; //collection of symbols defined in this segment
- references; //collection of extern symbols needed to relocate the code
from this segment
- data; //the actual code/data of segment
- exeCode; //the executable linked version of data, needed for code
segments used for JIT; not needed for writing files

Symbol *findSymbol(char *symName); //find the symbol with the
given name
void *getExeAddr(char *symName); //get the address in exeCode
of symName
};

class Symbol{
- flags; //if it is exported (like a function for DLL creation), ...
- name; //symbol name
- offset; //symbol offset in data
};

//has ownership over relocations
class Reference{
- flags //if it is imported (like a DLL function), ...
- name
- relocations //collection of the relocation of this reference
};

class Relocation{
- type //absolute, relative, ...
- size //1,4,8 - its size in data (maybe size can be a subtype of
type)
- offset //offset in data
};

With this kind of linker, someone can do:

X86Binary bin86;
void (*Call)();
Call compile(Function &f)
{
X86Assembler a86;
//generate code
bin86.createSegmentFrom(a86);
Segment *s=bin86.findSegmentBySymbol(f.name);
bin86.link(s);
return (Call)s->getExeAddr(f.name);
}

other possible usages:

//write an executable/library/object file
COFFWriter coff(bin86);
coff.write("app.exe");

//save the Binary to a cache, with all its segments, symbols, references,
etc
//and reload it later, in order to avoid recompilations
BinaryWriter bw(bin86);
bw.write("cache.bin");
...
bw.read("cache.bin");

I think on this architecture can be built later and it is sufficiently
powerful and extensible...or some variants of it.
If you handle the interface with the Assembler and linkage, I can also work
on on writers/loaders :)

Thank you,
Razvan Aciu