ODEX File Format

[msg555]

Is there any documentation on the odex file format available? I was
able to find some information by snooping through libdex/DexFile.h
about the opt header format but it doesn't really answer all of my
questions.

1. The opt header points to a 'deps' and 'aux' sections. What is the
format of these sections? I assume deps contains what files this odex
file depends on and some meta-data on those files but what information
does aux contain?

2. How do I translate from object offsets to object members? Is this
information contained in the dex file or is this something I can just
compute from the field lists?

3. I have similar questions about virtual table offsets. Virtual
table offsets won't change by simply adding an instruction to a
method's bytecode, correct?

4. How does the execute-inline function work? What do the inline
offsets represent?

Basically I would like to be able to extract as much information from
the odex file as I can. In an ideal world I would be able to
translate back to a dex file.

Thanks
-Mark

[msg555]

Does anyone have an answer to at least some of these questions?

Thanks
Mark

[Dan Bornstein]

On Sun, Aug 8, 2010 at 9:50 PM, msg555 <msg...@gmail.com> wrote:
> Is there any documentation on the odex file format available?

The only documentation of the odex format is in the code, which it
looks like you already found.

> Virtual table offsets won't change by simply adding an instruction to a
> method's bytecode, correct?

Correct. They'll only change if a method is added to or removed from
the class or one of its superclasses.

> 4. How does the execute-inline function work? What do the inline
> offsets represent?

It's not an offset, per se. It's a function number. See InlineNative.c
for more details. And you can see how it's hooked up in
mterp/c/OP_EXECUTE_INLINE.c.

Cheers,

-dan

[Javed Absar]

you may get some answers at http://www.netmite.com/android/mydroid/dalvik/docs/dexopt.html

 

dex and odex  -> I think the main difference between the two is that

DEX: has symbolic link (e.g. to constant pool)

ODEX: has instantiated values.

In other words

DEX :   virtual_call "myfunction"

 

ODEX: virtual_call ptr_to_DVM_data

but i am still learning so i may be wrong

 

 

 

 

 

 

--
You received this message because you are subscribed to the Google Groups "android-platform" group.
To post to this group, send email to android-...@googlegroups.com.
To unsubscribe from this group, send email to android-platfo...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-platform?hl=en.

--
my homepage: http://www.javedabsar.com

[msg555]

Thanks, that was quite helpful. Hopefully I'll be able to figure out
the rest in a similar manner.

[fadden]

On Aug 12, 8:57 pm, msg555 <msg...@gmail.com> wrote:
> Thanks, that was quite helpful.  Hopefully I'll be able to figure out
> the rest in a similar manner.

You may want to look at http://code.google.com/p/smali/, in particular
the "de-odex" portion.

[msg555]

Thanks, I think I've found all of the information I need. deodexing
isn't my only goal, I also want to be able to write out odex files.
It seems that the interpretation of the odex file is highly dependent
on the dalvik vm version that created it, though hopefully I can
support those that are in use today.

At this point I understand how the instance variables are aligned and
methods are set up and what the inline indexes mean though I haven't
written any code to actually interpret these yet. I have at this
point been successful in reading in the odex files and am working on
writing them back out. Just need to generate the class lookup table
and register maps. Which brings me to a few question.

Are the expanding or reducing index maps used in production yet?
What would happen if I replaced a bootstrap odex file like /system/
framework/framework.odex with another mostly equivalent odex file that
didn't have a class lookup table or didn't have register maps? I
imagine there would be a performance hit if it works at all?

On Aug 16, 6:20 pm, fadden <fad...@android.com> wrote:
> On Aug 12, 8:57 pm, msg555 <msg...@gmail.com> wrote:
>
> > Thanks, that was quite helpful.  Hopefully I'll be able to figure out
> > the rest in a similar manner.
>

> You may want to look athttp://code.google.com/p/smali/, in particular
> the "de-odex" portion.

[fadden]

On Aug 17, 9:03 am, msg555 <msg...@gmail.com> wrote:
> Thanks, I think I've found all of the information I need.  deodexing
> isn't my only goal, I also want to be able to write out odex files.
> It seems that the interpretation of the odex file is highly dependent
> on the dalvik vm version that created it, though hopefully I can
> support those that are in use today.

Why? Why not write a simple DEX and let dexopt do the work?

> At this point I understand how the instance variables are aligned and
> methods are set up and what the inline indexes mean though I haven't
> written any code to actually interpret these yet.  I have at this
> point been successful in reading in the odex files and am working on
> writing them back out.  Just need to generate the class lookup table
> and register maps.  Which brings me to a few question.

Field position and vtable ordering a subject to change from release to
release. In practice these don't change much, but the execute-inline
table has changed with every release (and not in a backward-compatible
way).

Optimized DEX files are expected to be generated and used by the same
version of the VM running on the same device. They're not meant to be
copied between devices or work across an upgrade.

> Are the expanding or reducing index maps used in production yet?

No. This was a mostly dead end.

> What would happen if I replaced a bootstrap odex file like /system/
> framework/framework.odex with another mostly equivalent odex file that
> didn't have a class lookup table or didn't have register maps?  I
> imagine there would be a performance hit if it works at all?

You'll need to keep the SHA-1 signature the same or the embedded
dependency lists won't match. The class lookup table needs to be
present. Register maps are optional; the GC will fall back to a
conservative scan if they're not present.

[msg555]

On Aug 17, 5:16 pm, fadden <fad...@android.com> wrote:
> On Aug 17, 9:03 am, msg555 <msg...@gmail.com> wrote:
>
> > Thanks, I think I've found all of the information I need.  deodexing
> > isn't my only goal, I also want to be able to write out odex files.
> > It seems that the interpretation of the odex file is highly dependent
> > on the dalvik vm version that created it, though hopefully I can
> > support those that are in use today.
>
> Why?  Why not write a simple DEX and let dexopt do the work?

The use case is when I have to work with an already optimized odex
file (most phones it seems have the bootstrap classes pre-optimized
and these are what I want to work with). For instance I would want to
be able to add function hooks to some of the methods. I have this
working for dex files by inserting new hooking methods and putting an
invoke-direct/range instruction at the beginning of each function I
want to hook.

I feel like it would be easier to write back out an odex file rather
than trying to go from odex->dex then writing out a dex file. This
way I can treat the inline indexes, object offsets, and vtable indexes
abstractly and just write them out as I read them in assuming I do
nothing that would change their alignment. This option would also
result in the extra files being written to the dalvik-cache.

>
> > At this point I understand how the instance variables are aligned and
> > methods are set up and what the inline indexes mean though I haven't
> > written any code to actually interpret these yet.  I have at this
> > point been successful in reading in the odex files and am working on
> > writing them back out.  Just need to generate the class lookup table
> > and register maps.  Which brings me to a few question.
>
> Field position and vtable ordering a subject to change from release to
> release.  In practice these don't change much, but the execute-inline
> table has changed with every release (and not in a backward-compatible
> way).
>

I understand. And hopefully most of the time I can just treat these
numbers as black boxes and not know what they mean. If I want to try
to reverse the process I think I should be able to check the vm
version in the odex file to know the format (or at least know if I
support the format).

> Optimized DEX files are expected to be generated and used by the same
> version of the VM running on the same device.  They're not meant to be
> copied between devices or work across an upgrade.
>

I understand, this is not my intention. Same odex file for the same
phone.

> > Are the expanding or reducing index maps used in production yet?
>
> No.  This was a mostly dead end.

Ok, less work for me :)

>
> > What would happen if I replaced a bootstrap odex file like /system/
> > framework/framework.odex with another mostly equivalent odex file that
> > didn't have a class lookup table or didn't have register maps?  I
> > imagine there would be a performance hit if it works at all?
>
> You'll need to keep the SHA-1 signature the same or the embedded
> dependency lists won't match.  The class lookup table needs to be
> present.  Register maps are optional; the GC will fall back to a
> conservative scan if they're not present.

Ok, so it sounds like this should be able to work in theory. I'll see
how it goes.

Thanks for the help
-Mark

[msg555]

Well I haven't implemented the register maps yet but I was successful
in adding hooks to an existing odex file. Is the cost of not
including the register maps important when I'm working with one of the
bootstrapclasses?

[fadden]

On Aug 18, 1:30 pm, msg555 <msg...@gmail.com> wrote:
> Well I haven't implemented the register maps yet but I was successful
> in adding hooks to an existing odex file.  Is the cost of not
> including the register maps important when I'm working with one of the
> bootstrapclasses?

A precise GC scan may be able to free up objects that a conservative
scan retains. Depending on what the code is doing, it could mean
holding on to a bunch of objects that should be discarded, or it could
have no effect at all.

The register maps are generated as part of verifying the class. The
verifier doesn't know how to deal with the "optimized" instructions
though, so unfortunately you can't just re-verify the class unless you
un-optimize it first.