Support for gRPC on msys2?

[Lukács T. Berki]

Are you guys planning to support gRPC on msys2 eventually (to some degree?)

We are considering using gRPC as the RPC mechanism between the client and the server of Bazel (http://bazel.io), and Bazel currently compiles only under msys2 on Windows.

I tried to get gRPC and all its dependencies to work under msys2 and I *almost* succeeded to get it to compile (the current roadblock is a few missing symbols, e.g. _vscprintf, _localtime_s and _putenv_s). Do you know of anyone who succeeded in doing so?

Eventually, we are planning to migrate away from msys2, but that won't happen for a while and in the meantime, it would be nice to get gRPC up and running.

The patch to gRPC (release-0_13_0) I needed is attached. It's only two lines of code.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[Lukács T. Berki]

I managed to get it to compile by using the {env,time,string,log}_posix.c instead of their win32 versions. It seems that the win32 versions of these files use functions from MSVCRT.DLL which are not available under msys2 (as far as I can tell).

When running a simple client (without a server, for now) it loops with the message:

E0408 17:08:40.218937000 25769803872 tcp_client_windows.c:211] Unable to set socket options: The attempted operation is not supported for the type of object referenced.

Does this ring a bell? I'm running Windows 8.1 and compiled gRPC with my little homebrew BUILD files at the tag "release-0_13_0" with the following command line options:

"-Wno-implicit-function-declaration",

"-D_WIN64",

"-D_WIN32_WINNT=0x0600",

"-DGPR_POSIX_LOG",

"-DGPR_POSIX_STRING",

"-DGPR_POSIX_ENV",

"-DGPR_POSIX_TIME",

"-DUSE_SYS_TYPES_FD_SET",

"-D__USE_W32_SOCKETS",

The code line above does some error message formatting after something that looks like initializing a socket.

[Lukács T. Berki]

Hello there,

Apologies for the somewhat incoherent e-mails before. I got the C++ implementation of gRPC to compile on msys2 and I can run at least a "Hello, World" client and server. Patch attached. Most of it is only a stanza in port_platform.h. The only nontrivial part is that setting SO_REUSEADDR on server sockets seems to return an error (I haven't investigated why), so I had to #ifdef it out.

Would it be possible to upstream this patch so that we the Bazel people don't have to maintain a patch?

[Craig Tiller]

It certainly would be possible - I don't see an attachment though?

Regardless - would you be willing to turn it into a pull request and work with us to get it into a submittable state?

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAOu%2B0LVR5xj3-bbdNrOfjooQgjchjoxKGV7QgAGFt4XUx_83_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Lukács T. Berki]

Apologies -- it seems that I forgot to click on the little paperclip button.

I attached it now and also created a pull request: https://github.com/grpc/grpc/pull/6140

[Lukács T. Berki]

Ping? Github says that there were some failures, but they seem to be entirely unrelated.

[Nicolas Noble]

Ah, sorry, I didn't see that thread, because firehose. So, I replied on the github thread, but here is a more extensive braindump.

First, we do have limited msys2 support sprinkled in the code. The problem being we don't have any good testing story for it yet, so it may break from time to time. The main issue is that while the core understands msys2, the tests architecture doesn't. We'd need to add msys2 as a test platform for that. And design something in Jenkins to properly support it. We've already migrated Jenkins from cygwin to msys2, so we got one small step done here, but there's several more to fully get there.

Now, msys2, unlike cygwin, doesn't actually provide a true posix-facing interface. It only provides a very thin gcc layer around the Windows API. And the Windows API really isn't posix. We shouldn't be using it as if it was posix. Doing so would make a lot of things fail in weird ways. For example, we wouldn't be calling WSAStartup if we were compiling grpc using posix-exclusive.

We've been successful in the past in using msys2, so the proper path here would be to fix whichever deltas have been missing, and not switch to the posix API.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAOu%2B0LUVerWCp6T%3DPE%3DX-ccCnXqUvbATdceR%2BVsLwaEOdi01uQ%40mail.gmail.com.

[Nicolas Noble]

So, I just successfully compiled and ran h2_full_test.exe and h2_ssl_test.exe under msys2 using our Makefile with CPPFLAGS="-D_WIN32_WINNT=0x600 -DUNICODE -D_UNICODE". I had to pound at test_config.c to fully remove and disable error checking because, as I said, our test infrastructure isn't properly set to handle tests under msys2 at the moment, but otherwise it linked and worked fine for me.

Any specific things you've seen actually breaking ?

[Nicolas Noble]

More specifically:

nnoble@NNOBLE0-W MINGW64 /c/work/grpc

diff --git a/test/core/util/port_windows.c b/test/core/util/port_windows.c

index 2b6d3dd..356ce75 100644

--- a/test/core/util/port_windows.c

+++ b/test/core/util/port_windows.c

@@ -114,7 +114,7 @@ static int is_port_available(int *port, int is_tcp) {

   /* Try binding to port */

   addr.sin_family = AF_INET;

   addr.sin_addr.s_addr = INADDR_ANY;

-  addr.sin_port = htons(*port);

+  addr.sin_port = htons((u_short)*port);

   if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {

     gpr_log(GPR_DEBUG, "bind(port=%d) failed: %s", *port, strerror(errno));

     closesocket(fd);

@@ -127,7 +127,7 @@ static int is_port_available(int *port, int is_tcp) {

     closesocket(fd);

     return 0;

   }

-  GPR_ASSERT(alen <= sizeof(addr));

+  GPR_ASSERT(alen <= (socklen_t)sizeof(addr));

   actual_port = ntohs(addr.sin_port);

   GPR_ASSERT(actual_port > 0);

   if (*port == 0) {

diff --git a/test/core/util/test_config.c b/test/core/util/test_config.c

index 3155a4e..e4e7d05 100644

--- a/test/core/util/test_config.c

+++ b/test/core/util/test_config.c

@@ -50,10 +50,10 @@ static unsigned seed(void) { return (unsigned)getpid(); }

 #if GPR_GETPID_IN_PROCESS_H

 #include <process.h>

-static unsigned seed(void) { return _getpid(); }

+static unsigned seed(void) { return (unsigned)_getpid(); }

 #endif

-#if GPR_WINDOWS_CRASH_HANDLER

+#if 0

 #include <windows.h>

 #include <tchar.h>

nnoble@NNOBLE0-W MINGW64 /c/work/grpc

$ CPPFLAGS="-D_WIN32_WINNT=0x600 -DUNICODE -D_UNICODE" make -j h2_ssl_test h2_full_test >& /dev/null && echo success || echo failure

success

nnoble@NNOBLE0-W MINGW64 /c/work/grpc

$ ./bins/opt/h2_full_test.exe >& /dev/null && echo success || echo failure

success

nnoble@NNOBLE0-W MINGW64 /c/work/grpc

$ ./bins/opt/h2_ssl_test.exe >& /dev/null && echo success || echo failure

success

nnoble@NNOBLE0-W MINGW64 /c/work/grpc

$ uname -a

MINGW64_NT-6.1 NNOBLE0-W 2.4.1(0.294/5/3) 2016-02-03 10:57 x86_64 Msys

[Nicolas Noble]

Ah. One thing worth mentioning is that there's several msys2 environments.

That's the one I've always been using. I'm guessing you've been using "MSYS2 Shell" instead ? That probably would break - no work has been done for that one yet, but your PR would still break the "good" one. Mmmh... let's see...

[Nicolas Noble]

Nope. I just tried with the MSYS2 Shell, and I was surprisingly able to do the same exact sequence of things, and build and run the same couple of tests. Only a minor detail in boringssl had to be tweaked for the MSYS2 Shell environment.

[Lukács T. Berki]

I'll take a closer look tomorrow (I'm in Europe, so it's already late here)

My guess is that you used the MinGW compiler, and I used the msys2 compiler, because that's how Bazel compiles at the moment.

I tried 0.13.0, and the first thing I found was that a preprocessor error was raised that said "Autoconfiguration failed". Then I manually defined WIN32, _WIN32_WINNT and maybe a few other things and using various weird tweaks I managed to get it to compile, but a "Hello, World" server I put together wouldn't open a socket for some mysterious reason, which I took as an indication that I'm holding it wrong.

Then I tried going the POSIX route and simply changed the autoconfig header so that it defined all the POSIX preprocessor defines and it compiled and ran basically flawlessly, the only minor issue being that the POSIX emulation msys provides apparently doesn't support SO_REUSEADDR, so I had to comment it out.

re: difference between MinGW and msys: my understanding is that the MinGW compiler builds native Windows binaries and msys2 builds Unixy tools that can be used during the build of said native Windows binaries. Thus, msys2 provides POSIX emulation and MinGW doesn't [1]. Since Bazel is a tool used for builds, msys2 seemed like a good fit (and since we have lot of POSIXisms, it was easier to get it to work to a rudimentary level). Here is a bit longer treatise. I haven't checked but my guess is that the MinGW compiler defines WIN32 and friends and the msys2 one doesn't.

Why do you think this patch would break your Windows support? It doesn't change anything if __MSYS__ is not defined, and I suppose that's not defined under MinGW (I haven't actually checked, though. I'll try it once I have access to a suitable computer.)

[1]: And that's why there are gcc toolchains and make binaries under msys2: one targets msys2, the other, MinGW.

[Nicolas Noble]

Yes, I was surprised to see you replying that late :-)

Actually, after digging a bit more, I just realized I was using strawberry perl gcc, which I have installed because of OpenSSL and BoringSSL, and it polluted the environment enough to percolate into msys2's environment. Joy. Environments are hard, and I wish I had docker under Windows.

So, yes, msys's gcc doesn't indeed compile grpc properly. The delta of changes I needed to do however shouldn't be too horrible. I'll see what I can do for you - most of the changes should be peripheral to grpc anyway.

Regarding the difference between msys and mingw, yes, this is close to the reality. In fact, the msys and mingw compilers and environments are originally the same, but the msys environment has a few more things added to it for some added posix compatibility, by bringing that extra msys dll dependency.

A good discriminatory test is to try to create a socket using the typical socket() call. If you are using msys's layer, it should return something like 3, because it translated the winsock socket into an internal structure to do its work (and 0, 1 and 2 are stdin, stdout and stderr). But if you are using the win32 api layer directly, it should return a higher number, like 216, because these are actual Winsock sockets. And it is possible to use the Winsock API when using msys's compiler instead of mingw32. It's just a matter of including the proper headers.

When compiling actual Windows code however, msys and mingw compilers and environments are pretty much interchangeable, with the delta that msys currently has an older environment - hence the reason there are things missing and breaking compilation for grpc at the moment. But both are providing the win32 API, really, and any piece of Windows code that wants to compile and run within msys should be able to, and there are enough subtle but annoying differences in the socket API that I'd rather be fully using the Windows API instead of the posix-emulation layer added by msys. For instance, I am not sure msys would have proper trailing bytes support when closing tcp connections, and that one bit us enough already that I'd rather make sure we're doing the right thing there. I wouldn't even be sure that the sequence we're using in posix would work to properly support trailing bytes when using msys's posix layer. And we don't need an extra layer anyway - we already have the code working for Windows :-) So I'd still would prefer we use the win32 api instead. As I said, let me see what I can do for you, this may or may not be trivial.

Note that when compiling win32 code with msys's gcc, one need to use the flag -mwin32, in order to enable usage of the windows API. This will define WIN32 and _WIN32 for you, and potentially a few other things that are needed. That's not something you need to do with mingw's compiler. However, by design, we are leaving _WIN32_WINNT up to the user to define in their project - which may or may not be a good idea.

[Nicolas Noble]

So it turned out to be a bit more time-consuming than I thought, and I discovered a fun msys bug on the way. The msys environment really lacks a few of the more recent API additions, a lot of the secure ones, and it doesn't seem to be well tested overall.

But at the end, I crafted this: https://github.com/grpc/grpc/pull/6183

Let me know if that helps you.

[Lukács T. Berki]

Patching it on top of bcd49994e23 and running "CPPFLAGS="-mwin32 -D_WIN32_WINNT=0x600" make" yields no errors other than "cc: error: gpr.def: No such file or directory", which is probably caused by the differences between how linking works on Windows and on Linux, so it's a good sign. Which means that if I integrate this with Bazel, it may actually work, because we have that part sorted out to some degree already. Let me just run some more experiments...

[Nicolas Noble]

I made a few changes to that PR in the meantime - it's a bit of a several pronged issue, and I seem to have broken something when compiling with Visual Studio, but I'm not sure what.

The gpr.def thing shouldn't exist. It's a fluke. Only grpc.def should.

[Lukács T. Berki]

No worries, I actually was about to write you that it doesn't work when I realized that you updated the pull request in the meantime.  I *hope* I am working with the latest version now. I'll get back to you once I manage to either integrate it with Bazel successfully or come up with a reason why I think it can't be done. I'm working on it now, so I should be done in a few minutes.

[Nicolas Noble]

I can't seem to locally reproduce the Visual Studio issue reported by Jenkins remotely, and otherwise things seem fine on my local machine with Visual Studio, msys and mingw. I'll try to debug the Visual Studio thing tomorrow from the office.

I'll also see if we can add a test or something for these.

[Lukács T. Berki]

With this patch, gRPC builds under Bazel without any extra changes to the command line and a "Hello, "World" client and server work correctly. "-mwin32 -D_WIN32_WINNT=0x600" needs to be added to the list of C++ compiler flags and "-lws2_32" needs to be added to the list of linker flags, but these are all expected.

So, it would be awesome if it could be merged :) (for the time being, we'll make do with our patched 0.13.0)

[Nicolas Noble]

Glad to hear that. And these changes are making gRPC much more Windows-y than the typical posix emulation layer, so that makes me more comfortable.

I'll look at that spurious Visual Studio thing tomorrow and we'll see to it that everything's merged.

Thanks for your patience :-)