Porting ninja to USS on z/OS

[Sean Perry]

Hi, I have just ported ninja over to z/OS and would like to contribute the changes to the master branch.  Before I create a PR I thought we should talk about this since it is a little different than most changes..  I guess I'll start by giving an overview of the changes I had to make to get ninja working.  The changes can be summarized as:

1) EBCDIC support.  This is fair easier than you would expect.  I built ninja with the compiler in "ASCII" mode and used few calls to fcntl() to convert characters from EBCDIC to ASCII while reading input by the file system.  This should mean no one needs to worry about EBCDIC when making changes on other platforms.  All I did was add one function call per important file descriptor to enable the conversion and the implementation of that function.

2) Using exec/fork for the subprocess management.  z/OS doesn't have the same level of support for spawn as ninja is using.  I had to implement the spawning using the traditional fork/exec sequence.

3) A few smaller places where z/OS has different interfaces to handle system interaction (eg. determining the number of processors).

4) Add support for the z/OS compiler

5) other misc things where the interfaces in system headers were slightly different than on other platforms.

I have not looked at re2c yet.  So right now you can't modify the grammar on z/OS.  I did find one curious bug in configure.py related to this that may occur on other platforms and may require a newer python level  (or just someone with more python knowledge than I have).   I found the code to run "re2c -V" would cause duplicate lines in the build.ninja file when it failed.  Changing "stdout=subprocess.PIPE" to "stdout=subprocess.DEVNULL" solved the problem.

I do work for IBM.  I have been working on compilers for many platforms for many years.   You can reach me at perry at ca.ibm.com as well as this email address.  

Thanks

[Nico Weber]

Hi Sean,

I'm glad you find ninja useful enough to want to port it to your system.

Having said that, the changes you list sound fairly terrifying.

For (1), I'd say we don't want anything mentioning EBCDIC near upstream. Sorry. If it's a small patch, it shouldn't cause too many issues.

For (2), we moved ninja to posix_spawn() a while ago since that's a major perf win on many systems. What other ports do (e.g. termux/android, which also doesn't have posix_spawn) is to link against a lib that implements it on top of fork/exec. E.g. https://github.com/termux/termux-packages/blob/master/packages/libandroid-spawn/posix_spawn.cpp / https://github.com/termux/termux-packages/blob/master/packages/ninja/build.sh . Maybe this is an option for you.

(3)-(5) depend on how big the diff is. If it's small cosmetic things, that's generally fine.

[Michael Jones]

@Sean Perry

Consider whether std::filesystem would be a more appropriate solution for EBCDIC compatibility, at least as far as file paths are concerned.

I have a "port" of the Ninja codebase for C++17, including std::filesystem support. I would welcome collaboration.

https://github.com/splinter-build/splinter/tree/16-std-filesystem

[Sean Perry]

Hi Mike thanks for the heads up.  I'll definitely take a look at your port.  

The file system wasn't a problem with porting to z/OS.  The z/OS provides what is called "unix system services", or USS, that is a standard Unix shell.   This shell puts you into a korn shell or bash shell at login and you do everything just like you would on Linux.  All of the commands we are used to are available.   The biggest issue I had porting ninja was really the spawn calls.  I have a suggestion on how to handle that.  The only other thing is adding a system call for each open file to enable some codepage convertions on I/O.  Hopefully, I can convince everyone to accept these changes.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/c23285fc-bc83-4f15-83b9-bed4fe07a16e%40googlegroups.com.

--

Sean Perry

[Fran Wong]

Hi Sean,

Have you successfully port ninja to USS? I am currently working on an open source python project on z/OS which requires NumPy and Pandas. I need to build the NumPy from source on z/OS which requires ninja. Could you tell me where I can find the z/OS version of ninja? I am struggling on how to set the C++ compiler options to make ninja package build on z/OS.

Thank you very much!

Fran