Question about where does Pal load the application binary

85 views
Skip to first unread message

Ou, Botong

unread,
Oct 22, 2021, 5:43:52 AM10/22/21
to sup...@graphene-project.io
Hi there,

I am currently doing a research project on graphene. As I was reading the codes, I become confused about how Pal loads the application binary into the memory.
In my case, I am looking into the folder under /Pal/src/host/Linux without SGX support. There is a main function in db_main.c file which is called pal_linux_main.

I was hoping that this function would do something such as loading the binary and preallocating the memory but I can't find where are the corresponding codes.
Therefore, I am wondering where does Pal load the executable binary(e.g. HelloWorld) into the memory. Could you give me some advice about the associated codes?

Thanks!
Regards,
Botong

Michał Kowalczyk

unread,
Oct 22, 2021, 5:50:42 AM10/22/21
to Ou, Botong, sup...@graphene-project.io
Hi,

PAL doesn't load the user binary, it loads LibOS binary (libsysdb.so) and then the LibOS loads the user binary. You can find the loader code for both of them in source files with "rtld" (runtime loader) in their names.

Best,
Michał

Kuvaiskii, Dmitrii

unread,
Oct 22, 2021, 6:06:18 AM10/22/21
to Kowalczyk, Michal, Ou, Botong, sup...@graphene-project.io

@Ou, Botong We have a PR that makes our “rtld” code in PAL much more readable: https://github.com/gramineproject/gramine/pull/152

 

You may want to browse the code of this version. To answer your question in a bit more detail:

Enjoy.

--

Dmitrii

--
You received this message because you are subscribed to the Google Groups "Graphene Support Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graphene-suppo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graphene-support/7e05b095-382a-b53d-6dec-0f619aa56d35%40invisiblethingslab.com.

Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva  
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Ou, Botong

unread,
Oct 24, 2021, 3:38:11 PM10/24/21
to Kuvaiskii, Dmitrii, Kowalczyk, Michal, sup...@graphene-project.io
Hi Dmitrii and Michal,

After reading the codes, I have the following questions:
  1.  Does g_process.exec always refer to the object of application binary?
  2. I see there is RUN_INIT(init_stack, argv, envp, &new_argp, &new_auxv); in shim_init(). Is this stack prepared for the user application to use?
    https://github.com/rssys/graphene/blob/master/LibOS/shim/src/shim_init.c#:~:text=RUN_INIT(init_stack%2C%20argv%2C%20envp%2C%20%26new_argp%2C%20%26new_auxv)%3B

  3. ​ shim_init() also called RUN_INIT(init_elf_objects); where I see it called load_elf_object. Is this where LibOS load the ELF object of application into memory?
    https://github.com/rssys/graphene/blob/master/LibOS/shim/src/shim_init.c#:~:text=RUN_INIT(init_elf_objects)%3B
    https://github.com/rssys/graphene/blob/master/LibOS/shim/src/shim_rtld.c#:~:text=ret%20%3D%20load_elf_object(exec%2C%20%26g_exec_map)%3B

Sorry for having so many questions. I appreciate it much if you can help me specify these things.

Regards,
Botong
From: Kuvaiskii, Dmitrii <dmitrii....@intel.com>
Sent: Friday, October 22, 2021 3:05 AM
To: Kowalczyk, Michal <mk...@invisiblethingslab.com>; Ou, Botong <b...@purdue.edu>; sup...@graphene-project.io <sup...@graphene-project.io>
Subject: RE: [graphene-support] Question about where does Pal load the application binary
 

Kuvaiskii, Dmitrii

unread,
Oct 25, 2021, 3:02:25 AM10/25/21
to Ou, Botong, Kowalczyk, Michal, sup...@graphene-project.io

(Take my answers with a grain of salt, because I only briefly looked at the codes – I generally remember the logic but don’t remember the details.)

 

> Does g_process.exec always refer to the object of application binary?

 

Yes.

 

> I see there is RUN_INIT(init_stack, argv, envp, &new_argp, &new_auxv); in shim_init(). Is this stack prepared for the user application to use?

 

Yes.

 

> shim_init() also called RUN_INIT(init_elf_objects); where I see it called load_elf_object. Is this where LibOS load the ELF object of application into memory?

 

Yes. But here pay attention to `interp`: almost all ELF executables do not start executing immediately, instead the “interpreter” (`interp`) is started first – this interpreter prepares all the environment for ELF executable to run (loads needed libraries, relocates the ELF executable itself, etc.), and only then passes control to the ELF executable. Almost always, the “interpreter” is ld.so.

 

You can notice here that if there is an interpreter specified by one of the special ELF fields in the executable, then the interpreter is run first: https://github.com/rssys/graphene/blob/fb71e4376a1fa797697832ca5cbd7731dc7f8793/LibOS/shim/src/shim_rtld.c#L892

 

But in general, all your understanding is correct.

 

--

Dmitrii

Kuvaiskii, Dmitrii

unread,
Oct 29, 2021, 3:35:17 AM10/29/21
to Ou, Botong, Kowalczyk, Michal, sup...@graphene-project.io

You are using the old Graphene repository. Please use the newest Gramine repository (“Graphene” was renamed to “Gramine” about 1.5 months ago).

 

You will notice that e.g. RTLD code in PAL is much cleaner and easier to follow in the latest Gramine version:

https://github.com/gramineproject/gramine/blob/master/Pal/src/db_rtld.c

 

--

Dmitrii

 

From: Ou, Botong <b...@purdue.edu>
Sent: Thursday, October 28, 2021 7:17 PM
To: Kuvaiskii, Dmitrii <dmitrii....@intel.com>; Kowalczyk, Michal <mk...@invisiblethingslab.com>; sup...@graphene-project.io
Subject: Re: [graphene-support] Question about where does Pal load the application binary

 

Hi Dmitrii and Michal,

 

I have a question about the function file_map() under path pal/src/host/linux. Why the mem address for the loading segment can't be changed. If I tried to map it to another region by creating another mmap and memcpy the data, there will be an error called unexpected memory fault. I am a little confused about this. Am I understanding something wrong? Please let me know.

https://github.com/gramineproject/graphene/blob/master/Pal/src/host/Linux/db_files.c#:~:text=mem%20%3D%20(void*)DO_SYSCALL(mmap%2C%20mem%2C%20size%2C%20prot%2C%20flags%2C%20fd%2C%20offset)%3B


Also, I found that the map_elf_object_by_handle doesn't have much comments in the code which is quite hard to understand. It would be great if someone can let me know how it maps each segment one by one into the memory.  What I am pursuing here is to map the elf object into a different memory region fixed by myself. Here is the link of the function:
https://github.com/gramineproject/graphene/blob/master/Pal/src/db_rtld.c#:~:text=static%20struct%20link_map*%20map_elf_object_by_handle,size_t%20fbp_len%2C%20bool%20do_copy_dyn)%20%7B

 

 

Thanks for your help in advance! I hope I am not bothering you too much.

Ou, Botong

unread,
Oct 29, 2021, 7:52:10 AM10/29/21
to Kuvaiskii, Dmitrii, Kowalczyk, Michal, sup...@graphene-project.io
Hi Dmitrii and Michal,

I have a question about the function file_map() under path pal/src/host/linux. Why the mem address for the loading segment can't be changed. If I tried to map it to another region by creating another mmap and memcpy the data, there will be an error called unexpected memory fault. I am a little confused about this. Am I understanding something wrong? Please let me know.
https://github.com/gramineproject/graphene/blob/master/Pal/src/host/Linux/db_files.c#:~:text=mem%20%3D%20(void*)DO_SYSCALL(mmap%2C%20mem%2C%20size%2C%20prot%2C%20flags%2C%20fd%2C%20offset)%3B

Also, I found that the map_elf_object_by_handle doesn't have much comments in the code which is quite hard to understand. It would be great if someone can let me know how it maps each segment one by one into the memory.  What I am pursuing here is to map the elf object into a different memory region fixed by myself. Here is the link of the function:
https://github.com/gramineproject/graphene/blob/master/Pal/src/db_rtld.c#:~:text=static%20struct%20link_map*%20map_elf_object_by_handle,size_t%20fbp_len%2C%20bool%20do_copy_dyn)%20%7B


Thanks for your help in advance! I hope I am not bothering you too much.

Regards,
Botong
From: Kuvaiskii, Dmitrii <dmitrii....@intel.com>
Sent: Monday, October 25, 2021 12:02 AM
To: Ou, Botong <b...@purdue.edu>; Kowalczyk, Michal <mk...@invisiblethingslab.com>; sup...@graphene-project.io <sup...@graphene-project.io>
Reply all
Reply to author
Forward
0 new messages