[windows] no sman on msys2/mingw64

[Qian Yun]

On msys2/mingw64, there is no sman, because there is no fork.

So, first, we should tweak 'fricas' script a little so that
it can work out of box when 'sman' is not presented.

Second, the replacement of sman, which I remember was discussed
long long time ago. Anyway, the replacement of sman should be
more cross platform.

- Qian

[Waldek Hebisch]

On Thu, Mar 04, 2021 at 06:53:52PM +0800, Qian Yun wrote:
> On msys2/mingw64, there is no sman, because there is no fork.

Real problem is pty-s, we depend on them to get output from
FRICASsys.

> So, first, we should tweak 'fricas' script a little so that
> it can work out of box when 'sman' is not presented.

Yes, we could do this. OTOH this is really install time
thing, we probaby should install different version of
'fricas' (or none at all). AFAICS currently 'fricas'
on Windows buys nothing compared to just using
FRICASsys.

--
Waldek Hebisch

[Qian Yun]

Wait a second, there are numerous packages on msys2/mingw64
that uses fork and pty, how does that happen?

[Dima Pasechnik]

On Thu, Mar 4, 2021 at 4:26 PM Qian Yun <oldk...@gmail.com> wrote:
>
> Wait a second, there are numerous packages on msys2/mingw64
> that uses fork and pty, how does that happen?

Could you point at any such package, with an URL?

you probably mix it up with cygwin (which does support fork() etc)

>
> On 3/4/21 11:51 PM, Waldek Hebisch wrote:
> > On Thu, Mar 04, 2021 at 06:53:52PM +0800, Qian Yun wrote:
> >> On msys2/mingw64, there is no sman, because there is no fork.
> >
> > Real problem is pty-s, we depend on them to get output from
> > FRICASsys.
> >
> >> So, first, we should tweak 'fricas' script a little so that
> >> it can work out of box when 'sman' is not presented.
> >
> > Yes, we could do this. OTOH this is really install time
> > thing, we probaby should install different version of
> > 'fricas' (or none at all). AFAICS currently 'fricas'
> > on Windows buys nothing compared to just using
> > FRICASsys.
> >
>

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/96de2689-efa0-ee03-1ad7-0f06341c78ff%40gmail.com.

[Waldek Hebisch]

On Fri, Mar 05, 2021 at 12:25:52AM +0800, Qian Yun wrote:
> Wait a second, there are numerous packages on msys2/mingw64
> that uses fork and pty, how does that happen?

IIUC msys uses (or maybe re-implements) Cygwin emulation
of fork and pty. Fork is emulated in rather crude way
and tends to fail from time to time. I did not look
how pty emulation works.

My point was that we can with moderate effort avoid
use of fork. ATM avoidning pty would be much more tricky.

--
Waldek Hebisch

[oldk1331]

THANK YOU both, Dima and Waldek.

Now I understand the difference between "msys2 shell" and
"mingw 64bit shell".

Indeed, in "msys2 shell", there are "fork".  (They partly
re-implement cygwin?).

So, in "msys2 shell", we can build sman successfully.
But when building algebra, SBCL encounters strange memory
fault, which I'll look into later.  And right now, sman
from msys2 shell does not mix well with FriCASsys from
mingw64 shell.

I'll continue explore the possibility from "msys2 shell".

On 3/5/21 12:31 AM, Dima Pasechnik wrote:
> On Thu, Mar 4, 2021 at 4:26 PM Qian Yun <oldk...@gmail.com> wrote:
>>
>> Wait a second, there are numerous packages on msys2/mingw64
>> that uses fork and pty, how does that happen?
> Could you point at any such package, with an URL?
>
> you probably mix it up with cygwin (which does support fork() etc)
>

[Qian Yun]

On 3/5/21 12:27, oldk1331 wrote:
> THANK YOU both, Dima and Waldek.
>
> Now I understand the difference between "msys2 shell" and
> "mingw 64bit shell".
>
> Indeed, in "msys2 shell", there are "fork".  (They partly
> re-implement cygwin?).
>
> So, in "msys2 shell", we can build sman successfully.
> But when building algebra, SBCL encounters strange memory
> fault, which I'll look into later.  And right now, sman
> from msys2 shell does not mix well with FriCASsys from
> mingw64 shell.

The memory fault happens if FriCASsys loads "libspad.so".
Probably the same thing for cygwin.

Obviously FFI works for SBCL on Windows. Now I need to
find out where went wrong in our build process.

- Qian

[Qian Yun]

On 6/18/22 11:35, Qian Yun wrote:
>
>> So, in "msys2 shell", we can build sman successfully.
>> But when building algebra, SBCL encounters strange memory
>> fault, which I'll look into later.  And right now, sman
>> from msys2 shell does not mix well with FriCASsys from
>> mingw64 shell.
>
> The memory fault happens if FriCASsys loads "libspad.so".
> Probably the same thing for cygwin.
>
> Obviously FFI works for SBCL on Windows.  Now I need to
> find out where went wrong in our build process.

FFI works for SBCL on Windows, I have made GMP support work
on Windows.

The problem is sbcl will seg fault shortly after it
"load-foreign-object" with anything that links to "mys-2.0.dll".

Summary: MingW64 toolchain allows you to build native win32
programs. Msys2 provides a minimal POSIX environment to assist
you to do that.

sman is inherent POSIX and not win32. So it is a dead end
to try to build sman on mingw64/msys2.

But I am satisfied with current win32 native "FriCASsys.exe".

SBCL on cygwin will probably meet same problem when it
tries to load cygwin dll. So I'm not going to try this
path and stick to "sbcl+mingw64 native win32 build".

- Qian

[Waldek Hebisch]

On Sun, Jun 19, 2022 at 09:54:29PM +0800, Qian Yun wrote:
>
>
> On 6/18/22 11:35, Qian Yun wrote:
> >
> >>So, in "msys2 shell", we can build sman successfully.
> >>But when building algebra, SBCL encounters strange memory
> >>fault, which I'll look into later.  And right now, sman
> >>from msys2 shell does not mix well with FriCASsys from
> >>mingw64 shell.
> >
> >The memory fault happens if FriCASsys loads "libspad.so".
> >Probably the same thing for cygwin.
> >
> >Obviously FFI works for SBCL on Windows.  Now I need to
> >find out where went wrong in our build process.
>
> FFI works for SBCL on Windows, I have made GMP support work
> on Windows.
>
> The problem is sbcl will seg fault shortly after it
> "load-foreign-object" with anything that links to "mys-2.0.dll".

No wonder. sbcl is depending on catching signals and a lot
of system features. System-style C code can easily interfer
with this. For example, for some time we had trouble with Ctrl-C
handling in sbcl. It turned out that sbcl used SIGPIPE to
handle Ctrl-C and our code set handler for SIGPIPE, so
sbcl never saw it. This feature of sbcl was completely
undocumented, I solved it comparing traces of system calls
for sbcl without our code and with our code.

One could try to ask sbcl and MingW64 folks about their
system assumptions/dependencies, but I am affraid that
they will be unwilling to give any specification.

> Summary: MingW64 toolchain allows you to build native win32
> programs. Msys2 provides a minimal POSIX environment to assist
> you to do that.
>
> sman is inherent POSIX and not win32. So it is a dead end
> to try to build sman on mingw64/msys2.

I think this is too simplistic point of view. All we need in
FRICASsys is sockets or equivalent like named pipes. We need
some Posix features in sman. Note that there are various
Posix level and with eception of pty-s other things should
be available on Windows. For example, we start programs by
'spawn_of_hell'. 'spawn' is Windows function and should be
sufficient here (ATM we use 'fork' to implement 'spawn'
but we could use different implementation). Similarly
sockets are available on Windows (and we should be able
to do with named pipes). So probably the only real
trouble are pty-s. Currently we need pty-s to collect text
output from Lisp. But once our output routines are abstract
enough we will be able to redirect them to other place like
socket or named pipe. We are not far from this.

> But I am satisfied with current win32 native "FriCASsys.exe".
>
> SBCL on cygwin will probably meet same problem when it
> tries to load cygwin dll.

You could ask sbcl folks. Unless sbcl was ported to Cygwin
it is unlikely to work with it...

--
Waldek Hebisch

[Waldek Hebisch]

On Sun, Jun 19, 2022 at 09:54:29PM +0800, Qian Yun wrote:
>
>

> On 6/18/22 11:35, Qian Yun wrote:
> >
> >>So, in "msys2 shell", we can build sman successfully.
> >>But when building algebra, SBCL encounters strange memory
> >>fault, which I'll look into later.  And right now, sman
> >>from msys2 shell does not mix well with FriCASsys from
> >>mingw64 shell.

<snip>

> sman is inherent POSIX and not win32. So it is a dead end
> to try to build sman on mingw64/msys2.

As I wrote, biggest trouble is due to pty-s. Simple test
for feasiblity of using msys2 pty-s is to build 'clef' under
msys2 using it Posix feature and then try if it can drive
Windows executable like sbcl or FRICASsys. I understand that
you do not like 'clef' but it is rather simple program that
depends on pty-s. If that works, then whole FriCAS should
be able to run in similar way (that is msys2 sman + mingw
FRICASsys). If not, then we need to eliminate use of pty-s
first.

--
Waldek Hebisch

[Qian Yun]

OK, I wonder if you know that there is no libX11 in msys2/mingw64.

Isn't the point of sman to support hyperdoc and graphics? (and clef).

Just want to make sure if your "whole FriCAS on win32" vision contains
hyperdoc and graphics.

- Qian

[Waldek Hebisch]

I did not look at 'msys2/mingw64'. 32-bit msys DLL was just fork
of Cygwin DLL so difficulities to get them running were probably
similar. Anyway, once we get past trouble with pty-s we can use
Cygwin hyperdoc and graphics. But emulating pty-s may be tricky...

> Just want to make sure if your "whole FriCAS on win32" vision contains
> hyperdoc and graphics.

Yes.

--
Waldek Hebisch

[Qian Yun]

On 6/28/22 06:42, Waldek Hebisch wrote:
>> OK, I wonder if you know that there is no libX11 in msys2/mingw64.
>>
>> Isn't the point of sman to support hyperdoc and graphics? (and clef).
>
> I did not look at 'msys2/mingw64'. 32-bit msys DLL was just fork
> of Cygwin DLL so difficulities to get them running were probably
> similar. Anyway, once we get past trouble with pty-s we can use
> Cygwin hyperdoc and graphics. But emulating pty-s may be tricky...
>
>> Just want to make sure if your "whole FriCAS on win32" vision contains
>> hyperdoc and graphics.
>
> Yes.
>

OK, from https://www.msys2.org/wiki/How-does-MSYS2-differ-from-Cygwin/ :

====
Cygwin and MSYS2 -- as projects -- have significantly different goals.

Cygwin tries to bring a POSIX-compatible environment to Windows so that
most software that runs on unices will build and run on Cygwin without
any significant modifications.

MSYS2 tries to provide an environment for building *native* Windows
software.

The shell and core tools exist mainly to allow porting Unix programs to
run *natively* on Windows (i.e. without requiring a POSIX emulation
layer). MSYS2 doesn't try to duplicate Cygwin's efforts more than
necessary, so the number of provided POSIX-emulated software is very
small.
====

So, for Cygwin, we can treat it as a Linux distro. Currently we use
ECL to build FriCAS. I didn't bother to build hypertex and graphics,
but I'm 99.9% sure it just works. Also Cygwin doesn't use win32
sockets.

Now, for MSYS2. My goal is to have *native* windows binary. Now we
have native windows binary without hyperdoc/graphics. But frankly
speaking, current native windows build almost has zero C part, it is
almost a pure lisp image.

sman/hyperdoc/graphics uses too much POSIX, it is hard (and not worth
it) to make them cross-platform.

- Qian

[Waldek Hebisch]

On Tue, Jun 28, 2022 at 08:51:31AM +0800, Qian Yun wrote:
>
>
> On 6/28/22 06:42, Waldek Hebisch wrote:
> >>OK, I wonder if you know that there is no libX11 in msys2/mingw64.
> >>
> >>Isn't the point of sman to support hyperdoc and graphics? (and clef).
> >
> >I did not look at 'msys2/mingw64'. 32-bit msys DLL was just fork
> >of Cygwin DLL so difficulities to get them running were probably
> >similar. Anyway, once we get past trouble with pty-s we can use
> >Cygwin hyperdoc and graphics. But emulating pty-s may be tricky...
> >
> >>Just want to make sure if your "whole FriCAS on win32" vision contains
> >>hyperdoc and graphics.
> >
> >Yes.
> >
>
> OK, from https://www.msys2.org/wiki/How-does-MSYS2-differ-from-Cygwin/ :

<snip>

> So, for Cygwin, we can treat it as a Linux distro. Currently we use
> ECL to build FriCAS. I didn't bother to build hypertex and graphics,
> but I'm 99.9% sure it just works. Also Cygwin doesn't use win32
> sockets.
>
> Now, for MSYS2. My goal is to have *native* windows binary. Now we
> have native windows binary without hyperdoc/graphics. But frankly
> speaking, current native windows build almost has zero C part, it is
> almost a pure lisp image.
>
> sman/hyperdoc/graphics uses too much POSIX, it is hard (and not worth
> it) to make them cross-platform.

A am affraid you did not notice what I wrote several times:
there is no pricipal reason for graphic/hyperdoc to be
compiled in the same way as FRICASsys. I know that some
Windows folks strongly dislike Cygwin. But Cygwin is
reasonable way to get graphical parts to Windows. Given
that sbcl seem to be incompatible with msys2.dll, we
should build FRICASsys without linking to it or Cygwin
DLL. But this does not exclude using Cygwin program
as helpers (in similar spirit to using msys2 tools for build).

Actually, cmucl port have similar spirit: cmucl is still 32-bit,
but C executables are build as 64-bit code.

Of course, real Windows lover would replace X11 calls by calls
to Windows API.

--
Waldek Hebisch

[Qian Yun]

On 6/28/22 18:40, Waldek Hebisch wrote:
>
> A am affraid you did not notice what I wrote several times:
> there is no pricipal reason for graphic/hyperdoc to be
> compiled in the same way as FRICASsys. I know that some
> Windows folks strongly dislike Cygwin. But Cygwin is
> reasonable way to get graphical parts to Windows. Given
> that sbcl seem to be incompatible with msys2.dll, we
> should build FRICASsys without linking to it or Cygwin
> DLL. But this does not exclude using Cygwin program
> as helpers (in similar spirit to using msys2 tools for build).
>
> Actually, cmucl port have similar spirit: cmucl is still 32-bit,
> but C executables are build as 64-bit code.
>
> Of course, real Windows lover would replace X11 calls by calls
> to Windows API.
>

I tried mixing sman linked with msys2.dll with native sbcl loading
libspad.so with win32sockets.

It hangs. I didn't debug why. Maybe because win32sockets doesn't
work with path like "/tmp/.xxxx" well. Maybe switching to TCP
sockets helps.

If people want to get graphical part on windows, they will have
to use ECL instead of sbcl then.

Just saying, the replacement of hyperdoc/graphics should be cross
platform.

- Qian