[Caml-list] GODI News: RocketBoost Beta
Gerd Stolpmann
In the past weeks I've worked hard to finish the next GODI release,
focusing on portability. A beta version of the new bootstrap and
godi_console, called RocketBoost, is now available, and it would be
great if it were tested at large.
There are not many new features in this release, so there is no reason
to switch to it if you already have GODI.
The big news is that GODI now supports the MinGW port of OCaml for
Windows (besides the Cygwin port). This means that it is now possible to
create native Windows applications with GODI. Note, however, that the
porting effort is still in its beginning - GODI itself works, but most
packages aren't ported yet, and won't work.
The MinGW support has become possible because large parts of the GODI
core have been rewritten. In particular, the C programs accompanying
godi_console are now coded in O'Caml (godi_make, godi_add, etc.), so
porting was possible by enhancing a few O'Caml libraries. This
refactoring of GODI also increases the portability in the Unix world -
effectively it should now run on all platforms where OCaml runs, where
gcc is available, and where POSIX-compliant shell utilities are
available.
If you would like to test it, follow these instructions. For all ports,
you need the bootstrap tarball from:
http://download.camlcity.org/download/godi-rocketboost-20080615.tar.gz
--- Installation for Unix ---
For Unix platforms (including MacOSX), just download
godi-rocketboost-20080615.tar.gz, unpack it, run ./bootstrap, and follow
the instructions.
--- Installation for Windows ---
For Windows you always need Cygwin, even for the MinGW port (it is
needed as scripting environment, please don't question that). Get it
from cygwin.com. Install everything that is needed (binutils, bzip2,
diffutils, file, findutils, gawk, gcc-core, gcc-mingw-core, grep, groff,
gzip, m4, make, man, mingw-runtime, patch, rxvt, sed, tar, w32api, wget
- hope this list is complete).
IMPORTANT: When you install software you need for GODI, choose paths
that do not contain space characters. This is incompatible with the
shell scripts. So don't install into "My Files".
Download godi-rocketboost-20080615.tar.gz, and unpack it:
$ tar xzf godi-rocketboost-20080615.tar.gz
$ cd godi-rocketboost-20080615
Now invoke bootstrap. You probably want to set the path where it is
going to be installed with -prefix. Furthermore, you can now select
whether you want the Cygwin or the MinGW port. For the latter, pass
"-w32port mingw" as extra argument.
IMPORTANT: Remember that GODI itself relies on Cygwin scripting. If you
pass paths to GODI (including godi_console, godi_add, etc.) these are
Cygwin paths, even if you build the MinGW port.
$ ./bootstrap -prefix /home/gerd/godi -w32port mingw
This will build boot_console. Then you need an Internet connection, and
do:
$ export PATH=/home/gerd/godi/bin:/home/gerd/godi/sbin:$PATH
$ ./bootstrap_stage2
This will install the minimal GODI core.
IMPORTANT: godi_console isn't ported to the Windows console window, and
for now only supports terminal windows that can deal with ANSI control
codes. Use Cygwin's rxvt as terminal (or any other capable terminal like
xterm). Furthermore, there is an issue with the MinGW port setting some
terminal features. As workaround, you have to set the TTY environment
variable to the Cygwin tty device, i.e. TTY=`tty`.
Note that the MinGW port is only available for ocaml-3.10. Passing
"-section 3.09" to bootstrap won't work.
--- List of problems ---
The final part of this message lists some problems that popped up during
the MinGW porting effort. I have often found workarounds, but maybe the
reader knows better solutions.
* Cygwin interoperability
A lot of the porting effort was about Cygwin interoperability.
godi_console (which is a native Win32 binary) can translate Cygwin paths
to native Windows paths (by reading the Cygwin mount table in the
registry). However, some problems turned out to be hard:
Starting a Cygwin binary with CreateProcess from godi_console turned out
to be unreliable when godi_console was itself called from a Cygwin
program. So the calling chain Cygwin_pgm -> Win32_pgm -> Cygwin_pgm
did not always work (symptom: the called program immediately exits). A
workaround was to use cmd.exe as intermediate instance:
Cygwin_pgm -> Win32_pgm -> cmd.exe -> Cygwin_pgm
Maybe related to that, it wasn't possible to run stty without explicit
device argument. (godi_console needs to call it to talk to the Cygwin
tty driver.) stty (and tty, too) always complained that stdin wasn't a
tty. The workaround is to ask the user to set the environment variable
TTY to the tty device.
* Cygwin vs. Windows file names
The approach is that GODI itself is seen as a Cygwin program - although
godi_console is a native program. As already mentioned, godi_console is
capable of translating Cygwin paths without using Cygwin.
At some time, it is required to translate paths to the Windows style,
because programs like ocaml, ocamlfind, etc. are Windows executables
without Cygwin path translation capability. In GODI Makefiles, there is
now always the variable $(LOCALBASE_NATIVE), set to the Windows path
(using forward slashes). The porting effort was effectively quite low -
in a few places I had to use ${LOCALBASE_NATIVE} instead of
${LOCALBASE}.
* CR/LF
In godi_console, it was required to look carefully over the code, and to
switch between ASCII and binary channel modes where needed. As native
Windows executable, godi_console uses CR/LF as EOL characters.
It turned out that Cygwin's bash has sometimes problems with CR/LF.
Especially, something like
stdlib=`ocamlc -where`
does not work - the CR character remains as part of the result value.
The workaround was to use Cygwin's alternate shell, ash, in these cases,
which handles that better.
* cygpcre DLL, and DLL lookup by PATH
Cygwin includes a PCRE DLL. Of course, this DLL cannot be used in
non-Cygwin programs, so GODI always installs its own PCRE DLL.
It turns out that the MinGW install of PCRE produces a DLL with the same
name as the Cygwin version (cygpcre-0.dll, as far as I remember). As
DLL's are looked up by PATH, and we _need_ the Cygwin directories in
PATH, the problem arises how to direct MinGW executables to use the
right DLL.
For now the workaround is that the MinGW version has a different version
number (cygpcre-7.dll). It would be nice, however, to get rid of the
"cyg" prefix for a clearer separation of DLL spaces.
* Ocaml does not install ocamldep.opt.exe
in the MinGW port. Maybe an oversight?
* No support for ocamlmklib
A lot of work has been put into working around the missing ocamlmklib.
The difficulty is that stub libraries need to be compiled with different
flags when a DLL is to be produced, i.e. the C compiler is invoked
differently in this case. This is incompatible with many build Makefiles
that can only produce one version of .o files from a .c file. Because of
this difficulty, ocamlmklib isn't available.
For ocamlnet, I worked around by using two scripts, stubcc, and
mkstublib, available from
https://godirepo.camlcity.org/svn/godi-build/trunk/godi/godi-ocamlnet/files/
Effectively, stubcc calls the C compiler twice, and produces a .d.o
intended for DLL inclusion, and a normal .o for the static library. I'm
wondering whether such a tool could be included in the ocaml
distribution, or whether ocamlc could have a special -for-stublib option
so that it invokes gcc twice for .c files.
* Windows "execv" is broken with respect to "wait"
When a windows process calls execv to invoke another binary, it is
signaled to the parent process that the program is finished. (The
correct POSIX behavior is to wait until the second binary is finished.)
In many places, it was easy to work around the problem by using the
spawn calls. At one place, this is not possible, and godi_console has to
use "execv" to avoid deadlocks. As consequence, it may happen that the
caller of godi_console thinks that the program is done although it is
still running.
No workaround yet.
* No real argv support in Windows
There are no clear rules how to quote arguments in Windows (cmd.exe).
(Actually, there is no list of arguments (argv) in Windows, but only a
command line string.) In GODI, it sometimes happens that quoted
arguments need to be quoted (and more complicated), and this did not
work in Windows. The workaround is to write complex commands to
temporary files, and to run these by /bin/sh.
* No signals in Windows
There is no clear way how to signal a process to terminate itself. As a
consequence, aborting builds is broken in godi_console.
* argv.(0) is broken
I don't know who is to blame for this, either Windows, or Cygwin. At
least it wasn't possible to pass a name as argv.(0) that differed from
the executable name.
The workaround was to use wrapper programs.
Anyway, I hope you like the new port. And remember: be patient. Windows
is very slow when used for shell-style scripting.
Gerd
--
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany
ge...@gerd-stolpmann.de http://www.gerd-stolpmann.de
Phone: +49-6151-153855 Fax: +49-6151-997714
------------------------------------------------------------
_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs
Adrien
First, thanks for your work for the windows platform.
I've just tried this beta. Overall it worked quite well (it compiled
during about one hour on a quite recent machine and installed 260MB of
data) but failed in the end. Here are the last lines (I haven't logged
everything but this could be enough) :
> + /home/Administrateur/godi-rocketboost-20080615/run/ocamlrun-3.09.3/byterun/ocamlfatrun /home/Administrateur/godi-rocketboost-20080615/godi-tools-boot/boot_console delete -f godi-tools
> Warning: Package godi-tools is protected against deletion.
> Deleting anyway.
> + /home/Administrateur/godi-rocketboost-20080615/run/ocamlrun-3.09.3/byterun/ocamlfatrun /home/Administrateur/godi-rocketboost-20080615/go
> di-tools-boot/boot_console add /ocaml/build/packages/All/godi-tools-2.0test31.tgz
> + exec /ocaml/sbin/godi_console perform -tmpdir-state /tmp/godi4632-0:1
> Fatal error: cannot find file perform
> Failure!
I have time to install godi from scratch so if you need anything,
don't hesitate. :)
Btw, is there any chance the PATH gets automatically updated ?
I've installed ocaml by hand several times but I keep on forgetting this step.
Thanks.
---
Adrien Nader
2008/6/15 Gerd Stolpmann <in...@gerd-stolpmann.de>:
Gerd Stolpmann
Am Sonntag, den 15.06.2008, 18:32 +0200 schrieb Adrien:
> Hi,
>
> First, thanks for your work for the windows platform.
>
> I've just tried this beta. Overall it worked quite well (it compiled
> during about one hour on a quite recent machine and installed 260MB of
> data) but failed in the end. Here are the last lines (I haven't logged
> everything but this could be enough) :
>
> > + /home/Administrateur/godi-rocketboost-20080615/run/ocamlrun-3.09.3/byterun/ocamlfatrun /home/Administrateur/godi-rocketboost-20080615/godi-tools-boot/boot_console delete -f godi-tools
> > Warning: Package godi-tools is protected against deletion.
> > Deleting anyway.
> > + /home/Administrateur/godi-rocketboost-20080615/run/ocamlrun-3.09.3/byterun/ocamlfatrun /home/Administrateur/godi-rocketboost-20080615/go
> > di-tools-boot/boot_console add /ocaml/build/packages/All/godi-tools-2.0test31.tgz
> > + exec /ocaml/sbin/godi_console perform -tmpdir-state /tmp/godi4632-0:1
> > Fatal error: cannot find file perform
> > Failure!
That's quite strange. I have no idea what it is, but it is harmless.
Actually, this last command does not nothing, so GODI is installed.
Can you check whether the newly installed godi_console works? A good
test is
godi_console perform -build godi-tools
so it builds itself again using the new godi_console (and not the
bootstrap version).
> I have time to install godi from scratch so if you need anything,
> don't hesitate. :)
Thanks for your help.
> Btw, is there any chance the PATH gets automatically updated ?
> I've installed ocaml by hand several times but I keep on forgetting this step.
Well, you would need to do it in your login setup, e.g. .bashrc. The
bootstrap script just cannot do it.
What we can talk about is that stage2 is automatically called.
Gerd
Adrien
>
> Am Sonntag, den 15.06.2008, 18:32 +0200 schrieb Adrien:
>> Hi,
>>
>> First, thanks for your work for the windows platform.
>>
>> I've just tried this beta. Overall it worked quite well (it compiled
>> during about one hour on a quite recent machine and installed 260MB of
>> data) but failed in the end. Here are the last lines (I haven't logged
>> everything but this could be enough) :
>>
>> > + /home/Administrateur/godi-rocketboost-20080615/run/ocamlrun-3.09.3/byterun/ocamlfatrun /home/Administrateur/godi-rocketboost-20080615/godi-tools-boot/boot_console delete -f godi-tools
>> > Warning: Package godi-tools is protected against deletion.
>> > Deleting anyway.
>> > + /home/Administrateur/godi-rocketboost-20080615/run/ocamlrun-3.09.3/byterun/ocamlfatrun /home/Administrateur/godi-rocketboost-20080615/go
>> > di-tools-boot/boot_console add /ocaml/build/packages/All/godi-tools-2.0test31.tgz
>> > + exec /ocaml/sbin/godi_console perform -tmpdir-state /tmp/godi4632-0:1
>> > Fatal error: cannot find file perform
>> > Failure!
>
> That's quite strange. I have no idea what it is, but it is harmless.
> Actually, this last command does not nothing, so GODI is installed.
>
> Can you check whether the newly installed godi_console works? A good
> test is
>
> godi_console perform -build godi-tools
>
> so it builds itself again using the new godi_console (and not the
> bootstrap version).
Well, it doesn't work :
Administrateur@Chewbacca ~
$ godi_console perform -build godi_tools
Fatal error: cannot find file perform
I had some weird feeling about this so I tried strings on godi_console
and in fact...
Administrateur@Chewbacca ~
$ cat `which godi_console`
#! /bin/sh
exec /home/Administrateur/godi-rocketboost-20080615/run/ocamlrun-3.09.3/byterun/ocamlfatrun
/home/Administrateur/godi-rocketboost-20080615
/godi-tools-boot/boot_console "$@"
I guess it failed at some point during installation but I really don't
know when. I'll try another installation on tomorrow.
>> I have time to install godi from scratch so if you need anything,
>> don't hesitate. :)
>
> Thanks for your help.
>
>> Btw, is there any chance the PATH gets automatically updated ?
>> I've installed ocaml by hand several times but I keep on forgetting this step.
>
> Well, you would need to do it in your login setup, e.g. .bashrc. The
> bootstrap script just cannot do it.
>
> What we can talk about is that stage2 is automatically called.
I think it's alright this way. I just always forget the PATH thing. I
think I'll have a tatoo or something like that, that should do the
trick.
Alain Frisch
> The big news is that GODI now supports the MinGW port of OCaml for
> Windows (besides the Cygwin port). This means that it is now possible to
> create native Windows applications with GODI.
That's really great news!
> A lot of the porting effort was about Cygwin interoperability.
> godi_console (which is a native Win32 binary) can translate Cygwin paths
> to native Windows paths (by reading the Cygwin mount table in the
> registry).
This feature might be useful for other projects. Do you plan to make it
available as a stand-alone library (maybe together with other
Windows-related hacks)? Does it support Cygwin symlinks?
> * Ocaml does not install ocamldep.opt.exe
>
> in the MinGW port. Maybe an oversight?
Already fixed in CVS HEAD (many of the Makefiles are now shared between
Unix and Windows ports).
> * No support for ocamlmklib
>
> A lot of work has been put into working around the missing ocamlmklib.
> The difficulty is that stub libraries need to be compiled with different
> flags when a DLL is to be produced, i.e. the C compiler is invoked
> differently in this case.
This is no longer true with the version in CVS HEAD, to become 3.11. Now
there is some hackery at link time to allow using the same set of object
files for linking statically or dynamically (for stub code dlls). Also,
ocamlmklib is available for all the Windows ports.
> There are no clear rules how to quote arguments in Windows (cmd.exe).
Indeed. I've found that it is sometimes necessary (and it seems to be
always sufficient) to add extra quotes around the whole command line.
E.g. to call emacs on a file "foo bar" (without the quotes) from OCaml,
here what I do:
Sys.command "\"\"C:\\Program Files\\emacs-22.1\\bin\\emacs\" \"foo bar\"\""
That is, one passes this command line to cmd.exe:
""C:\Program Files\emacs-22.1\bin\emacs" "foo bar""
As weird as it may look, the quotes around the program and argument
names are not escaped.
-- Alain
Gerd Stolpmann
Am Montag, den 16.06.2008, 08:39 +0200 schrieb Alain Frisch:
> Gerd Stolpmann wrote:
> > The big news is that GODI now supports the MinGW port of OCaml for
> > Windows (besides the Cygwin port). This means that it is now possible to
> > create native Windows applications with GODI.
>
> That's really great news!
>
> > A lot of the porting effort was about Cygwin interoperability.
> > godi_console (which is a native Win32 binary) can translate Cygwin paths
> > to native Windows paths (by reading the Cygwin mount table in the
> > registry).
>
> This feature might be useful for other projects. Do you plan to make it
> available as a stand-alone library (maybe together with other
> Windows-related hacks)?
Currently I don't have such plans. For now, just get the library from
https://godirepo.camlcity.org/svn/godi-bootstrap/godi-tools/trunk/console-src/godi-sys-win32/ (module Godi_file, in the parallel dir godi-sys-unix there is an implementation for POSIX).
> Does it support Cygwin symlinks?
It can read and resolve symlinks. For creating symlinks it just calls
Cygwin's /bin/ln for now.
> > * Ocaml does not install ocamldep.opt.exe
> >
> > in the MinGW port. Maybe an oversight?
>
> Already fixed in CVS HEAD (many of the Makefiles are now shared between
> Unix and Windows ports).
Cool.
> > * No support for ocamlmklib
> >
> > A lot of work has been put into working around the missing ocamlmklib.
> > The difficulty is that stub libraries need to be compiled with different
> > flags when a DLL is to be produced, i.e. the C compiler is invoked
> > differently in this case.
>
> This is no longer true with the version in CVS HEAD, to become 3.11. Now
> there is some hackery at link time to allow using the same set of object
> files for linking statically or dynamically (for stub code dlls). Also,
> ocamlmklib is available for all the Windows ports.
That's great. Thanks for doing that. I'll see how to integrate flexdll -
is it needed before ocaml is built, or can it be installed later? (That
looks like a bootstrap problem.)
> > There are no clear rules how to quote arguments in Windows (cmd.exe).
>
> Indeed. I've found that it is sometimes necessary (and it seems to be
> always sufficient) to add extra quotes around the whole command line.
> E.g. to call emacs on a file "foo bar" (without the quotes) from OCaml,
> here what I do:
>
> Sys.command "\"\"C:\\Program Files\\emacs-22.1\\bin\\emacs\" \"foo bar\"\""
>
> That is, one passes this command line to cmd.exe:
>
> ""C:\Program Files\emacs-22.1\bin\emacs" "foo bar""
>
> As weird as it may look, the quotes around the program and argument
> names are not escaped.
Interesting. Nevertheless, really helpful is only something predictable.
There is no good information from Microsoft.
Gerd
--
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany
ge...@gerd-stolpmann.de http://www.gerd-stolpmann.de
Phone: +49-6151-153855 Fax: +49-6151-997714
------------------------------------------------------------
Alain Frisch
> That's great. Thanks for doing that. I'll see how to integrate flexdll -
> is it needed before ocaml is built, or can it be installed later? (That
> looks like a bootstrap problem.)
Yes, it is currently assumed that a binary version of flexdll is
available when ocaml is built. I guess it is simpler to just install a
binary version (installing is just unzipping the distribution where you
want and extend the PATH) rather than trying to bootstrap flexdll itself.
-- Alain
Gerd Stolpmann
Am Sonntag, den 15.06.2008, 23:42 +0200 schrieb Adrien:
> > Can you check whether the newly installed godi_console works? A good
> > test is
> >
> > godi_console perform -build godi-tools
> >
> > so it builds itself again using the new godi_console (and not the
> > bootstrap version).
>
> Well, it doesn't work :
> Administrateur@Chewbacca ~
> $ godi_console perform -build godi_tools
> Fatal error: cannot find file perform
I think I got it. It may happen that there are both godi_console (as
shell script) and godi_console.exe, and depending on the Cygwin version
one or the other is called, and for some versions invocation fails
completely.
Just delete the shell scripts (in the bin and sbin directories, when
there is also an .exe), and it should work.
I've released a new version of rocketboost that fixes this:
http://download.camlcity.org/download/godi-rocketboost-20080616.tar.gz
Gerd
Adrien
"godi_console perform -build godi-tools" now works.
If there is any test I can run, I'll be happy to do so.
---
Adrien Nader
2008/6/16 Gerd Stolpmann <in...@gerd-stolpmann.de>:
Pierre-Evariste Dagand
> In the past weeks I've worked hard to finish the next GODI release,
> focusing on portability. A beta version of the new bootstrap and
> godi_console, called RocketBoost, is now available, and it would be
> great if it were tested at large.
Great work, thanks !
So, this works for me :
$ uname -m -r -s
FreeBSD 7.0-RELEASE i386
$ gcc -v
Using built-in specs.
Target: i386-undermydesk-freebsd
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 4.2.1 20070719 [FreeBSD]
$ ocaml -version
The Objective Caml toplevel, version 3.10.1
Although, I had (and still have) troubles installing the "standard"
Godi system on FreeBSD 7.0: godi_create segfaults. Two months ago, I
spent some time solving the issues until the point where godi_create
behaves segfaultlessly but, then, that was yet another godi_* tool
that gone into an infinite loop. Sadly, I did not find the time to
bust the bug or even to make a bug report. Now, I have some free time
so if this is still of interest I can try again to make it work.
Regards,
--
Pierre-Evariste DAGAND
http://perso.eleves.bretagne.ens-cachan.fr/~dagand/