How can I map OSv kernel image to high memory address space?

[persi...@gmail.com]

OSv can run unmodified Linux applications which are complied with `-fPIC` or `-pie`. However, position dependent executables are not support.

Currently, OSv kernel image is mapped at low address space (0x 20 0000). Many position dependent executable executables also start at a low address, such as 0x 40 0040.

OSv Linear Address Space:

 ---------  0x 0000 0000 0000 0000

 |       |

 ---------  0x 0000 0000 0020 0000  kernel_base    --\

 |       |                                           |- Kernel (Core ELF) - 22M

 ---------  0x 0000 0000 0180 0000  lzkernel_base  --/

 |       |

 |--------  0x 0000 1000 0000 0000  program_base  --\

 |       |                                          |- s_program - 8G

 |-------|  0x 0000 1002 0000 0000  --\           --X

 |       |                             |- Program   |

 |-------|  0x 0000 1004 0000 0000  --/             |

 |       |                                          |- ELF Namespaces(max: 32) - 256G

 |       |  ......................                  |

 |       |                                          |

 |-------|  0x 0000 1042 0000 0000                --/

 |       |

 ---------  0x ffff 8000 0000 0000  phys_mem  --\

 |       |                                      |- Main Area - 16T

 ---------  0x ffff 9000 0000 0000            --X

 |       |                                      |- Page Area - 16T

 ---------  0x ffff a000 0000 0000            --X

 |       |                                      |- Mempool Area - 16T

 ---------  0x ffff b000 0000 0000            --X

 |       |                                      |- Debug Area - 80T

 ---------  0x ffff ffff ffff ffff            --/

Linux Linear Address Space:

 ---------  0x 0000 0000 0000 0000  --\

 |       |                            |- Reserved (Catch Exceptions of NULL/Little Pointers)

 ---------                          --X

 |       |                            |- Text Segment

 ---------                          --X

 |       |                            |- Data Segment

 ---------                          --X

 |       |                            |- [Random]

 ---------                          --X

 |       |                            |- Heap

 ---------                          --/   \/

 |       |

 ---------                          --\

 |       |                            |- Memory Mapping Segment

 ---------                          --/

 |       |                            |- [Random]

 ---------                          --/

 |       |

 ---------                          --\   /\

 |       |                            |- Stack

 ---------                          --X

 |       |                            |- [Random]

 ---------                          --X

 |       |                            |- Command-line Arguments

 ---------                          --X

 |       |                            |- Environment Variables

 |       |                          --/

 ---------

 |       |

 ---------  0x 0000 7fff ffff ffff

 |       |

 |       |  -- Unused Space

 |       |

 ---------  0x ffff 8000 0000 0000  --\

 |       |                            |

 |       |                            |- Kernel Space - 128T

 |       |                            |

 ---------  0x ffff ffff ffff ffff  --/

If OSv kernel image can be mapped to high address space like Linux ( 0x ffff 8000 0000 0000 ?), then low address space can be used for applications.

So, single position dependent executable can be supported like shared objects and PIE.

I have tried to do this, but it seems difficult for me. Can you give some advice?

Thank you!

[Nadav Har'El]

On Mon, Dec 24, 2018 at 10:40 AM <persi...@gmail.com> wrote:

OSv can run unmodified Linux applications which are complied with `-fPIC` or `-pie`. However, position dependent executables are not support.

Currently, OSv kernel image is mapped at low address space (0x 20 0000). Many position dependent executable executables also start at a low address, such as 0x 40 0040.

The ability to run one position-dependent executable was proposed in https://github.com/cloudius-systems/osv/issues/190

But so far nobody has done this, because it's usually quite easy to recompile as shared object, and because running a regular executable will come with strings attached: you can only run one of them (so you can't expect completely normal Linux packages, including shell scripts and other stuff, to just work out of the box), and you'll also have all the regular problems of PIE - see https://github.com/cloudius-systems/osv/issues/352 and https://github.com/cloudius-systems/osv/issues/689 (shared objects have more complete support).

 I haven't tried to do this, or thought it through, but it should be possible. Right now in 0xfff.... we map the entire physical memory linearly, to return through malloc(). We can start this mapping in a slightly higher address, i.e., increase "phys_mem"  and leave a lower range in 0xfff... for the other stuff - kernel, mmap(), and shared objects. We'll also need to change the loader to load the kernel to a different address.

Nadav.

[persi...@gmail.com]

I have tried to rebuild osv with `-mcmodel=large`. A strang error happened:

  ...
  LINK loader.elf
ld.bfd: dynamic STT_GNU_IFUNC symbol `memcpy' with pointer equality in `build/release.x64/arch/x64/string.o' can not be used when making an executable; recompile with -fPIE and relink with -pie

But when I see the original osv kernel image.

file build/release.x64/loader-stripped.elf

build/release.x64/loader-stripped.elf: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked, stripped

readelf -ls build/release.x64/loader-stripped.elf | grep IFUNC
  1859: 0000000000388120    76 IFUNC   GLOBAL DEFAULT    2 memcpy
  5685: 00000000003f48f0    32 IFUNC   GLOBAL DEFAULT    2 _ZN5sched10set_fsbaseEm
  5768: 0000000000396ce0    32 IFUNC   GLOBAL DEFAULT    2 _ZN5sched17fpu_state_rest
 10518: 0000000000388360    32 IFUNC   GLOBAL DEFAULT    2 memset
 11790: 0000000000396ca0    32 IFUNC   GLOBAL DEFAULT    2 _ZN5sched14fpu_state_init
 13365: 0000000000396cc0    32 IFUNC   GLOBAL DEFAULT    2 _ZN5sched14fpu_state_save

It certainly is an executable with dynamic STT_GNU_IFUNC symbols. I don't know the reason of the error, I just changed code model from small to large.

在 2018年12月24日星期一 UTC+8下午5:30:30，Nadav Har'El写道：

[Waldek Kozaczuk]

I think if we map OSv kernel code "high enough" and leave most of the 1st GB of virtual memory free we should be able to run most of the position dependant executables that typically start at 4MB. This would allow OSv handle apps like no-pie java, python, etc (non-pie tiny bootstrapping loaders) easily.

The reason I am suggesting "high-enough" instead of "very-high" (like Linux) is because it might make implementation easier. The OSv boot.S (https://github.com/cloudius-systems/osv/blob/c8395118cb580f2395cac6c53999feb217fd2c2f/arch/x64/boot.S#L23-L41) maps 1st 1GB one to one. So if we changed that static mapping in a way that maps 2MB:end_of_kernel to say roughly last 10MB of 1GB then very early on our kernel would be mapped high enough. We could remap the kernel later but that would be more complicated I think. Also as long as kernel is mapped within first 2GB we would not need to use `-mcmodel=large`, right?