Upgrading musl - I can see light in the tunnel

[Waldek Kozaczuk]

It has been almost a month and I think I can see light in the tunnel. After all the patches I have sent recently (including the libc/time-related ones), I think we are at the point where we can proceed with the real upgrade.

As far as libc/ folder goes, we are down from roughly 215 C files month ago to 104 ones where only following 54 + 8 (under libc/arch) at some point came from musl and are actual candidates for some kind of upgrade:

libc/arch/aarch64/setjmp/sigrtmin.c

libc/arch/aarch64/setjmp/block.c

libc/arch/aarch64/setjmp/siglongjmp.c

libc/arch/aarch64/setjmp/sigrtmax.c

libc/arch/x64/setjmp/sigrtmin.c

libc/arch/x64/setjmp/block.c

libc/arch/x64/setjmp/siglongjmp.c

libc/arch/x64/setjmp/sigrtmax.c

libc/env/__environ.c

libc/env/secure_getenv.c

libc/errno/strerror.c

libc/internal/libc.c

libc/locale/freelocale.c

libc/locale/strtod_l.c

libc/locale/strtof_l.c

libc/locale/strtold_l.c

libc/locale/uselocale.c

libc/misc/ffs.c

libc/network/getaddrinfo.c

libc/network/gethostbyname_r.c

libc/network/getifaddrs.c

libc/network/getnameinfo.c

libc/network/inet_addr.c

libc/network/inet_aton.c

libc/network/inet_ntop.c

libc/network/__ipparse.c

libc/prng/__rand48_step.c

libc/stdio/__fdopen.c

libc/stdio/fgetc.c

libc/stdio/flockfile.c

libc/stdio/fmemopen.c

libc/stdio/__fopen_rb_ca.c

libc/stdio/fputc.c

libc/stdio/ftrylockfile.c

libc/stdio/funlockfile.c

libc/stdio/getc.c

libc/stdio/__lockfile.c

libc/stdio/open_memstream.c

libc/stdio/open_wmemstream.c

libc/stdio/putc.c

libc/stdio/remove.c

libc/stdio/sscanf.c

libc/stdio/stderr.c

libc/stdio/stdin.c

libc/stdio/__stdio_read.c

libc/stdio/__stdio_write.c

libc/stdio/stdout.c

libc/stdio/vdprintf.c

libc/stdio/vfprintf.c

libc/stdio/vfscanf.c

libc/stdio/vfwscanf.c

libc/stdio/vsnprintf.c

libc/stdio/vsscanf.c

libc/stdio/vswprintf.c

libc/stdio/vswscanf.c

libc/stdlib/strtod.c

libc/stdlib/strtol.c

libc/stdlib/wcstol.c

libc/string/strerror_r.c

libc/syslog.c

libc/time/ftime.c

libc/time/__tz.c

The libc/arch ones should be simple makefile replacements to point to a new version of musl. Many other files (~10-20) from the 54 ones should also be as easy.

Most of the files that will require manual changes are those under libc/stdio (28, mostly .lock vs .locking changes in FILE and syscalls) and libc/network. For both stdio and locale and I should also have some android-based unit tests.

I have looked at locale related changes in a new version of musl and we might need to make some changes in other places of our code (linux.cc).

For all the files we will have to keep changing the original musl files, I suggest we actually store the .patch files as they make it easier/quicker to see what changes we make and automatically detect we need to update those as we upgrade to future versions of musl. For that, we will have to slightly change the build process. 

Now the worst part to deal with might actually be the headers. Out of over ~220 *.h files under include/api, most of those symlink to musl/include folder but many are still copies from older/newer version of musl and often manually modified. Some might require simple change to symlinks, but others might require special attention.

Here are all the files (not symlinks to musl/include) I identified as candidates for an upgrade:

include/api/sched.h

include/api/stdint.h

include/api/link.h

include/api/time.h

include/api/mqueue.h

include/api/aarch64/bits/stat.h

include/api/aarch64/bits/stdint.h

include/api/aarch64/bits/posix.h

include/api/aarch64/bits/limits.h

include/api/aarch64/bits/errno.h

include/api/aarch64/bits/float.h

include/api/aarch64/bits/shm.h

include/api/aarch64/bits/syscall.h

include/api/aarch64/bits/mman.h

include/api/aarch64/bits/fenv.h

include/api/aarch64/bits/statfs.h

include/api/aarch64/bits/termios.h

include/api/aarch64/bits/setjmp.h

include/api/aarch64/bits/ipc.h

include/api/aarch64/bits/fcntl.h

include/api/aarch64/bits/socket.h

include/api/aarch64/bits/ioctl.h

include/api/aarch64/bits/endian.h

include/api/aarch64/bits/signal.h

include/api/pthread.h

include/api/fenv.h

include/api/string.h

include/api/sys/stat.h

include/api/sys/mount.h

include/api/sys/shm.h

include/api/sys/types.h

include/api/sys/sem.h

include/api/sys/time.h

include/api/sys/mman.h

include/api/sys/sysinfo.h

include/api/sys/uio.h

include/api/sys/statfs.h

include/api/sys/eventfd.h

include/api/sys/resource.h

include/api/sys/sysmacros.h

include/api/sys/socket.h

include/api/sys/ioctl.h

include/api/sys/inotify.h

include/api/sys/__socket.h

include/api/sys/poll.h

include/api/sys/sysctl.h

include/api/sys/statvfs.h

include/api/printf.h

include/api/x64/bits/stat.h

include/api/x64/bits/stdint.h

include/api/x64/bits/limits.h

include/api/x64/bits/syscall.h

include/api/x64/bits/mman.h

include/api/x64/bits/fenv.h

include/api/x64/bits/socket.h

include/api/x64/bits/ioctl.h

include/api/x64/bits/msg.h

include/api/stdio.h

include/api/stddef.h

include/api/math.h

include/api/assert.h

include/api/net/__if.h

include/api/net/if.h

include/api/fcntl.h

include/api/utmpx.h

include/api/endian.h

include/api/arpa/inet.h

include/api/netinet/in.h

include/api/netinet/__in.h

include/api/netinet/ip_icmp.h

include/api/stdlib.h

include/api/paths.h

include/api/libaio.h

include/api/wchar.h

include/api/unistd.h

The main problem, I anticipate, is that there will be conflicts when we change musl/ to the new version (1.1.24) and all of a sudden most of the headers under include/api will be new and other ones will be old. This will very likely affect the compilation of my C files under new musl/, libc/ and even other ones. Shall we have two include/api folders like  include/api_old and  include/api (this may not work either)? Given this situation with headers, how shall we go about executing the upgrade incrementally, ideally one musl/src/ and libc/ subfolder at a time? Is this even possible? Any suggestions?

Finally, how shall we name the directories? Rename current musl/ to musl.0.9.12/ and add musl/ subproject that would directly point to 1.1.14 tag of the original musl git repo (git://git.musl-libc.org/musl)? I know that right now all git subprojects point to the copies of the relevant repos in cloudous git org. Shall we copy the 1.1.14 tag of musl somewhere and make github subproject point to it? I have created at some point new https://github.com/osvunikernel org where we can create the musl 1.1.14 copy project (original musl is not on github).

Any more suggestions regarding mechanics? Order of things to do - here is orignal Nadav's plan (with my some adjustments):

• Bring the latest Musl version into a "musl-1.1.24" subdirectory in OSv (or rename musl to musl-0.9.12 and bring 1.1.24 into musl directory).
• Leave the old "musl/" and "libc/" directories as well.
• In Makefile, make a new list of objects (say, "nmusl") which will take code from musl-1.1.24/ instead of musl/.
• Start to switch individual files and directories from "musl += ..." to "nmusl += ...".  We can start with the math functions needed for aarch64. Eventually, everything can be converted to nmusl and hopefully, nothing or little will break and need to be fixed.
• Replace our include/api with musl-1.1.24/include. Would be even better to drop include/api, and just use musl-1.1.24/include directly if we can. I think, though, this is low priority, and might take quite a bit of work ("git log include/api" shows we modified this quite a bit since we took it from musl). I'm hoping that the old header files will work correctly also for the newer musl.
  • I think this might be tricky one at this point

Waldek

[Nadav Har'El]

On Wed, Aug 26, 2020 at 7:58 AM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

It has been almost a month and I think I can see light in the tunnel.

:-)

Very nice work.

 

For all the files we will have to keep changing the original musl files, I suggest we actually store the .patch files as they make it easier/quicker to see what changes we make and automatically detect we need to update those as we upgrade to future versions of musl. For that, we will have to slightly change the build process. 

Yes. ".patch" are not a great solution because they are hard to edit, but they are good enough for changes we just need to create once and never edit (like those stdio locking issues).

In some cases it might be possible to avoid the .patch because we come up with other solutions - like that "aliases.ld" thing I did which made a lot of patches unnecessary, or macros you did for syscall(), and so on.

The main problem, I anticipate, is that there will be conflicts when we change musl/ to the new version (1.1.24) and all of a sudden most of the headers under include/api will be new and other ones will be old. This will very likely affect the compilation of my C files under new musl/, libc/ and even other ones.

It might, but then again, it might not - or - hopefully - only have small localized effects.

The reason why I'm saying this is that the header files are supposed to be compatible with *standard* C and Linux, so they could not have changed too much.

Sure they could have added things, but it's not like a header could define unsigned int sleep(unsigned int) and then suddenly changed to some other signature.

So it is possible that you could compile the old .c files with new header files, or vice versa. I'm sure some things will break, I just hope not everything breaks.

 

Shall we have two include/api folders like  include/api_old and  include/api (this may not work either)? Given this situation with headers, how shall we go about executing the upgrade incrementally, ideally one musl/src/ and libc/ subfolder at a time? Is this even possible? Any suggestions?

I don't really know. I think at this point you have a better understanding, and more experience, at this musl-upgrade stuff than I do, so you probably have better insights on how to best do it.

You can also try doing the upgrade now, and see what actually happens - what breaks and what might be the easiest way to unbreak it.

Finally, how shall we name the directories? Rename current musl/ to musl.0.9.12/ and add musl/ subproject that would directly point to 1.1.14 tag of the original musl git repo (git://git.musl-libc.org/musl)? I know that right now all git subprojects point to the copies of the relevant repos in cloudous git org. Shall we copy the 1.1.14 tag of musl somewhere and make github subproject point to it? I have created at some point new https://github.com/osvunikernel org where we can create the musl 1.1.14 copy project (original musl is not on github).

Any more suggestions regarding mechanics? Order of things to do - here is orignal Nadav's plan (with my some adjustments):

• Bring the latest Musl version into a "musl-1.1.24" subdirectory in OSv (or rename musl to musl-0.9.12 and bring 1.1.24 into musl directory).

Maybe take 1.2.1? Or are you deliberately targetting an older release?

 

• Leave the old "musl/" and "libc/" directories as well.
• In Makefile, make a new list of objects (say, "nmusl") which will take code from musl-1.1.24/ instead of musl/.
• Start to switch individual files and directories from "musl += ..." to "nmusl += ...".  We can start with the math functions needed for aarch64. Eventually, everything can be converted to nmusl and hopefully, nothing or little will break and need to be fixed.
• Replace our include/api with musl-1.1.24/include. Would be even better to drop include/api, and just use musl-1.1.24/include directly if we can. I think, though, this is low priority, and might take quite a bit of work ("git log include/api" shows we modified this quite a bit since we took it from musl). I'm hoping that the old header files will work correctly also for the newer musl.
  • I think this might be tricky one at this point

Waldek

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/a49e2e1c-89fc-49b5-a87b-88b3737097e9n%40googlegroups.com.