Help on writing extensions

[hhm]

I am a bit overwhelmed and confused by the extension API,since I am not so familiar with C codebases yet, and so I am not able to understand the API from reading the code only. Are there any docs on the API somewhere?

If not, here is what I want to do, if anyone could give any tips:

A "mount emulation" using FUSE, so basically to mount things without being the kernel, by emulating the proper syscalls; FUSE is *made* for filesystems in userspace, so it seems that in essence it should be simple: mount() just needs to be emulated, and, of course, the file-related system calls, and the rest of the functionality can be provided by (maybe slightly patched) fuse. Some of this filename functionality is obviously already implemented for bind mounts; however probably file reading and writing will need to be implemented.

(See http://kernel.org/doc/Documentation/filesystems/fuse.txt towards the end of the page, for docs on the fuse<->kernel interface).

So basically I don't know how to make an extension to implement these things in real code :-).

If anyone has any info or suggestions, then that would be great!

(by the way, this is for the Android rootfs project I wrote about previously on this list, which thank G-d is coming along really well, mostly thanks to PRoot!)

[Cédric VINCENT]

Hello,

> I am a bit overwhelmed and confused by the extension API, since I am

> not so familiar with C codebases yet, and so I am not able to
> understand the API from reading the code only. Are there any docs on
> the API somewhere?

So far, the only available documentation is:

https://github.com/cedric-vincent/PRoot/blob/v2.3.1/src/extension/extension.h#L34

I think it's time to write something more understable (the control
flow is missing, for instance), I'll keep you informed once it's done.
In the mean time you could have a look at src/extension/fake_id0 and
src/extension/kompat, and ask for details also.

> If not, here is what I want to do, if anyone could give any tips: A
> "mount emulation" using FUSE, so basically to mount things without
> being the kernel, by emulating the proper syscalls; FUSE is *made* for
> filesystems in userspace, so it seems that in essence it should be
> simple: mount() just needs to be emulated, and, of course, the
> file-related system calls, and the rest of the functionality can be
> provided by (maybe slightly patched) fuse. Some of this filename
> functionality is obviously already implemented for bind mounts;
> however probably file reading and writing will need to be implemented.
> (See http://kernel.org/doc/Documentation/filesystems/fuse.txt towards
> the end of the page, for docs on the fuse<->kernel interface). So
> basically I don't know how to make an extension to implement these
> things in real code :-).
>
> If anyone has any info or suggestions, then that would be great!

I'm not sure to understand, why don't you just use fusermount?

Cédric.

[Hee Hoo]

On Tue, Feb 5, 2013 at 6:56 AM, Cédric VINCENT <cedric....@gmail.com> wrote:
> Hello,
>
>> I am a bit overwhelmed and confused by the extension API, since I am
>> not so familiar with C codebases yet, and so I am not able to
>> understand the API from reading the code only. Are there any docs on
>> the API somewhere?
>
> So far, the only available documentation is:
>
> https://github.com/cedric-vincent/PRoot/blob/v2.3.1/src/extension/extension.h#L34

OK, thanks.

>
> I think it's time to write something more understable (the control
> flow is missing, for instance), I'll keep you informed once it's done.
> In the mean time you could have a look at src/extension/fake_id0 and
> src/extension/kompat, and ask for details also.
>

Ok, thank you!

>> If not, here is what I want to do, if anyone could give any tips: A
>> "mount emulation" using FUSE, so basically to mount things without
>> being the kernel, by emulating the proper syscalls; FUSE is *made* for
>> filesystems in userspace, so it seems that in essence it should be
>> simple: mount() just needs to be emulated, and, of course, the
>> file-related system calls, and the rest of the functionality can be
>> provided by (maybe slightly patched) fuse. Some of this filename
>> functionality is obviously already implemented for bind mounts;
>> however probably file reading and writing will need to be implemented.
>> (See http://kernel.org/doc/Documentation/filesystems/fuse.txt towards
>> the end of the page, for docs on the fuse<->kernel interface). So
>> basically I don't know how to make an extension to implement these
>> things in real code :-).
>>
>> If anyone has any info or suggestions, then that would be great!
>
> I'm not sure to understand, why don't you just use fusermount?

Soory, I wasnt very clear. I want to be able to use FUSE without even
*installing* it as root; in other words, even the kernel-space
interface would be implemented in user space instead (maybe by
simulating the /def/fuse device with syscalls?). This way, on systems
like Android without root, files can be made executable and loopmounts
can be used. This way e.g. the SD card can be used to keep a rootfs
on, which is useful in a bunch of ways, especially for some phones
which have barely any internal storage memory.

>
> Cédric.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "PRoot" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to proot_me+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Cédric VINCENT]

> Soory, I wasnt very clear. I want to be able to use FUSE without
> even *installing* it as root; in other words, even the kernel-space
> interface would be implemented in user space instead (maybe by
> simulating the /def/fuse device with syscalls?). This way, on
> systems like Android without root, files can be made executable and
> loopmounts can be used. This way e.g. the SD card can be used to
> keep a rootfs on, which is useful in a bunch of ways, especially for
> some phones which have barely any internal storage memory.

As my understanding, you want to mount a rootfs image through PRoot + a
FUSE-compatible extension. Maybe something like:

proot --fuse-extension [...] fusermount [...] /media/sdcard/my_rootfs.ext2

Am I right? Why don't you just use PRoot with an extracted rootfs:

proot /media/sdcard/my_rootfs/

Maybe it's because your SD card is VFAT formatted (thus, it doesn't
support Unix file-system attributes)? In that case you may try
another format (UMSDOS?), however I don't know what are the FS supported
by Android.

Regards,
Cédric.

[hhm]

On Wed, Feb 6, 2013 at 9:11 AM, Cédric VINCENT <cedric....@gmail.com> wrote:
>> Soory, I wasnt very clear. I want to be able to use FUSE without
>> even *installing* it as root; in other words, even the kernel-space
>> interface would be implemented in user space instead (maybe by
>> simulating the /def/fuse device with syscalls?). This way, on
>> systems like Android without root, files can be made executable and
>> loopmounts can be used. This way e.g. the SD card can be used to
>> keep a rootfs on, which is useful in a bunch of ways, especially for
>> some phones which have barely any internal storage memory.
>
> As my understanding, you want to mount a rootfs image through PRoot + a
> FUSE-compatible extension. Maybe something like:
>
> proot --fuse-extension [...] fusermount [...] /media/sdcard/my_rootfs.ext2
>
> Am I right? Why don't you just use PRoot with an extracted rootfs:
>
> proot /media/sdcard/my_rootfs/

You are right, basically I want to make the extension provide the
response for any FUSE file syscall (or even just provide /dev/fuse, I
don't know enough about it yet to know which way(s)), instead of the
kernel providing it.

I would just use PRoot with an extracted rootfs, and that is the way I
use it for the internal storage currently, but this does not work when
the rootfs is on the sdcard, as explained below.

>
> Maybe it's because your SD card is VFAT formatted (thus, it doesn't
> support Unix file-system attributes)? In that case you may try
> another format (UMSDOS?), however I don't know what are the FS supported
> by Android.
>

Yes :-). My sdcard *is* vfat formatted, but even if it was not, it
still would not work; Android mounts the sdcard with 'noexec' by
default, and root is necessary to remount it without 'noexec' (see
http://stackoverflow.com/questions/10133274/how-to-run-c-application-in-android-shell).
Also, even where noexec is not set, vfat cannot have symlinks (for
busybox etc.), and if distributing, it would be a pain for everyone to
reformat their sdcards, since as far as I know, many sdcards are by
default formatted with vfat :-)

Therefore it would be useful for me to use FUSE, that way we have
loopmounting and a whole bunch of other stuff. I don't know if you'd
want to distribute this with proot by default, however, because then
it may start becoming a Linux-API user space compatibility layer,
instead of just chroot-like utility ;-).

It may be simpler to make an extension that just fakes executable
permissions for all files, and emulates symlinks (using text files
with the target path or something, ala git's "ref:filepath") though.
However from the little I know it seems that there may be more to
being executable than just the bit :-), like RAM loading etc..

> Regards,

> Cédric.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "PRoot" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to proot_me+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Thank you very much for your being of assistance!

[Cédric VINCENT]

On Thu, Feb 7, 2013 at 7:51 AM, hhm <heeh...@gmail.com> wrote:
> Therefore it would be useful for me to use FUSE, that way we have
> loopmounting and a whole bunch of other stuff. I don't know if you'd
> want to distribute this with proot by default, however, because then
> it may start becoming a Linux-API user space compatibility layer,
> instead of just chroot-like utility ;-).

I really like this idea ;)

> However from the little I know it seems that there may be more to
> being executable than just the bit :-), like RAM loading etc..

Right, it seems the "noexec" attribute is way stronger than what I
thought:

noexec Do not allow direct execution of any binaries on the
mounted filesystem. (Until recently it was possible to
run binaries anyway using a command like /lib/ld*.so
/mnt/binary. This trick fails since Linux 2.4.25 / 2.6.0.)
-- man 2 mount

It means it's not possible to cheat anymore by using a user-space
loader :(

Although, even if I really like your idea of a FUSE-compatible
extension in PRoot, I think Goanna is more suitable for your needs:

http://www.fsl.cs.sunysb.edu/project-goanna.html

> Thank you very much for your being of assistance!

You're welcome! :)
Cédric.

[hhm]

From reading about Goanna, I realized that it should be possible to
override noexec by avoiding using the kernel mmap implementation for
mapping executables when they are being executed; the mmaper is the
one that disallows it, by checking if the file being mmap'ed is on an
executable mount, as far as i know. libsigsegv
[http://www.gnu.org/software/libsigsegv/] might be helpful if
implementing this.

also, I realized that I wouldn't care to use the fuse implementation,
but just to use something which is compatible with fuse, so all the
existing fuse filesystem implementations can be used transparently. I
would guess the best way to do this is to reimplement libfuse, but
emulating /dev/fuse and other fuse interfaces may be needed too.

[Cédric VINCENT]

Hello,

> From reading about Goanna, I realized that it should be possible to
> override noexec by avoiding using the kernel mmap implementation for
> mapping executables when they are being executed; the mmaper is the
> one that disallows it, by checking if the file being mmap'ed is on an
> executable mount, as far as i know. libsigsegv
> [http://www.gnu.org/software/libsigsegv/] might be helpful if
> implementing this.

It should be possible to bypass the noexec limitation on mmap by
replacing file-backed mappings with copies of the expected contents
into anonymous mappings. This might be quite inefficient and some
corner cases [about file-backed mappings] are not easy to simulate
(c.f. chapter 49 "Memory Mappings" from the great "Linux Programming
Interface" book http://man7.org/tlpi/)

In a PRoot extension, this could be achieved this way:

// pseudo code, no error checking!
// extension event: SYSCALL_ENTER_*
hook_mmap_enter(tracee)
{
prot = peek_reg(tracee, CURRENT, SYSARG_3);
flags = peek_reg(tracee, CURRENT, SYSARG_4);

if ((flags & MAP_ANONYMOUS) != 0)
return; // not a file-backed mmap

prot |= PROT_WRITE;
prot &= ~PROT_EXEC;
flags |= MAP_ANONYMOUS;

poke_reg(tracee, SYSARG_3, prot);
poke_reg(tracee, SYSARG_4, flags);
poke_reg(tracee, SYSARG_5, -1);
}

This modifies the mmap parameters before this syscall is handled by
the kernel. For instance:

- mmap(NULL, 183055, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0)
+ mmap(NULL, 183055, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)

Then, once this syscall is completed by the kernel, such an extension
could copy the expected content to the expected address:

// pseudo code, no error checking!
// extension event: SYSCALL_EXIT_*
hook_mmap_exit(tracee)
{
remote_address = peek_reg(tracee, CURRENT, SYSARG_RESULT);
remote_size = peek_reg(tracee, ORIGINAL, SYSARG_2);
remote_fd = peek_reg(tracee, ORIGINAL, SYSARG_5);

local_fd = open("/proc/%d/fd/%d", tracee->pid, remote_fd);
local_address = mmap(NULL, remote_size, PROT_READ,
MAP_PRIVATE, local_fd, 0);

write_data(tracee, remote_address, local_address, remote_size);

}


> also, I realized that I wouldn't care to use the fuse implementation,
> but just to use something which is compatible with fuse, so all the
> existing fuse filesystem implementations can be used transparently. I
> would guess the best way to do this is to reimplement libfuse, but
> emulating /dev/fuse and other fuse interfaces may be needed too.

I don't know what would be easier: writing libfuse-like interface or
writing a "/dev/fuse"-like interface.


Regards,
Cédric.


PS: for information, the project umview (similar to PRoot in some
aspects) has a user-mode dev/loop interface. Although, I don't
know if this could help.

[Cédric VINCENT]

err,

On Thu, Apr 18, 2013 at 10:15 AM, Cédric VINCENT
<cedric....@gmail.com> wrote:
> prot &= ~PROT_EXEC;

No need to remove PROT_EXEC, it's now an anonymous mapping. Sorry for
the confusion.

[Cédric VINCENT]

On Wed, Apr 17, 2013 at 11:07 AM, hhm <heeh...@gmail.com> wrote:
> libsigsegv [http://www.gnu.org/software/libsigsegv/] might be
> helpful if implementing this.

Note it is possible with ptrace to analyse (PTRACE_GETSIGINFO) and
discard signals (ptrace restart with signal = 0) received by a
process. That means libsigsegv is not useful if you are in PRoot.

// pseudo code, no error checking!

// extension event: NEW_STATUS
hook_sigsegv(tracee, new_status)
{
if (!WIFSTOPPED(new_status) || ((new_status & 0xfff00) >> 8) != SIGSEGV)
return 0; // not a sigsegv, let PRoot handle this event

ptrace(PTRACE_GETSIGINFO, tracee->pid, 0, &siginfo);

// do stuff according to siginfo
...

// discard the signal and restart the process.
ptrace(PTRACE_SYSCALL, tracee->pid, 0, 0);

return 1; // tell PRoot to do nothing
}

Cédric.

[hhm]

Thank you for this information!

 

I don't know what would be easier: writing libfuse-like interface or
writing a "/dev/fuse"-like interface.

I would probably prefer to write whichever one is on a lower layer; this way the higher layers are automatically implemented and the code is more portable and often less fidgety (basically to me it is similar to the ptrace-vs.-LD_PRELOAD option  :-) )

 

Regards,
Cédric.


PS: for information, the project umview (similar to PRoot in some
    aspects) has a user-mode dev/loop interface.  Although, I don't
    know if this could help.

Thanks! Checked it out, but was too hard for me to compile this for arm and android at that time:-).One of the things I really like about PRoot is its ease of compilation and moving around.

I am sorry that it took me such a long time to respond :-)!

Thanks!

[Cédric VINCENT]

On Thursday, April 18, 2013 10:15:58 AM UTC+2, Cédric VINCENT wrote:

It should be possible to bypass the noexec limitation on mmap by
replacing file-backed mappings with copies of the expected contents
into anonymous mappings.  This might be quite inefficient and some
corner cases [about file-backed mappings] are not easy to simulate
(c.f. chapter 49 "Memory Mappings" from the great "Linux Programming
Interface" book http://man7.org/tlpi/)

Here is a proof-of-concept (quick n' dirty, see commit message):

    https://github.com/cedric-vincent/PRoot/commit/ea2c72493ff1622e6690f6338af6661a3beda79b

Regards,
Cédric.

[hhm]

awesome!!!!!!!!

testing now.

[Corbin Champion]

How has this worked out?  Did it work for you?  Any performance improvements from initial commit?

[Corbin Champion]

I have now tried this on an Android device with everything moved to sdcard except the interpreter and static executables.  The android sdcard is mounted noexec.  I get the following error:

error while loading shared libraries: /lib/arm-linux-gnueabi/libexpat.so.1: cannot apply additional memory protection after relocation: Permission denied.

hhm, did you ever try it?  Did you see similar?

Corbin

[h hm]

Yes; also tried, forget the error right not though; need to try again :-)

--

[h hm]

Yup tried it, and kept errors this time :-). Replied on different thread.