preview of a cross-compilation tool

[Matthew Flatt]

While `raco exe` in v8.1 very nearly supports cross-builds of Racket
executables[*], it's not easy to pass the right flags and set up the
needed target-platform distributions.

The `raco cross` command provided by the new "raco-cross" package wraps
Racket tools to simplify all of that management. For example,

raco cross --target x86_64-linux --vm bc exe example.rkt

is like running

raco exe example.rkt

but it cross-builds for x86_64 Linux. That build process involves
downloading a minimal racket tarball for the target platform,
installing "compiler-lib" there, downloading a matching minimal build
of Racket for the host platform, and then running the host Racket in
cross-build mode with installed target distribution --- but, again,
`raco cross` takes care of all that.

Just like using `raco exe` on Linux, the generated executable won't
actually work on other machines until you package it in a distribution
using `raco dist`:

raco cross --target x86_64-linux --vm bc dist example-dist example

Naturally, this second invocation of `raco cross` for the same target
uses the installations prepared by the first `raco cross`, so it's
relatively fast.


I expect that `raco cross` will become more useful after the v8.2
release, which will repair cross-compilation for CS executables and
executables with native libraries (like GUI executables). Meanwhile,
it's possible to use `raco cross` with snapshot builds, but you have to
point it at a snapshot site; see the documentation for more
information.


Because juggling multiple installations and versions is a lot of the
work for cross-compilation, `raco cross` can also be a way to get to a
different version of Racket in minimal form. For example,

raco cross --version 8.0 racket

starts a Racket v8.0 prompt. (The `racket` command is treated specially
by `raco cross`. All other commands are prefixed by `raco`.)


For more information see

https://docs.racket-lang.org/raco-cross/index.html


[*] Running a cross-built CS executable fails at the very last step of
startup, and that's due to an expander bug for handling top-level
`begin` forms compiled in machine-independent form. We didn't
detect this problem before, because it wasn't part of Bogdan's use
case for cross builds, and because cross-compilation was painful
enough that we didn't actually try other cases.

[schle...@gmail.com]

This is like being a child, waking up and realizing its christmas (and easter at the same time),

thank you for working on this, I will test out this shiny new toy/gift (meant in the best possible way).

Because I am on linux, I will try to cross compile to windows (time to dust off my windows vm for testing ;))

[kamist...@gmail.com]

I ran raco-cross

(installed on racket cs 8.0 [does this cause trouble for racket cs < 8.2 or is it fine because host and target are both bc?])

to run exe and dist:

;; install dependencies
raco cross --workspace /tmp/todays-snapshot --target i386-win32 --vm bc pkg install
;; create exe
raco cross --workspace /tmp/todays-snapshot --target i386-win32 --vm bc exe myapp.rkt
;; create dist
raco cross --workspace /tmp/todays-snapshot --target i386-win32 --vm bc dist myapp-i386-win32-bc myapp.exe

>> Cross configuration
 Target:    i386-win32
 Host:      x86_64-linux
 Version:   8.0
 VM:        bc
 Workspace: /tmp/todays-snapshot

On windows:

1. I had to add the lib subfolder in the dist to the path (else the exe would not find the dlls in there).

2. Copy the dlls from /tmp/todays-snapshot/i386-win32-bc/lib/ to the lib subfolder.

[I also tried `raco exe --embed-dlls` but then the libs from 2. are missing from the executable,

do I need to run future raco-cross at version cs 8.2 so that this part works, or could I use bc to make it work?]

After that the program ran for a bit allowed me to select a directory and then crashed with an exception,

probably this is a bug in my application that is windows specific. I need to check that.

I also tried the utah snapshots but they don't have i386 builds and the windows7 vm is 32bit,

I might also test with 64bit windows10 vm, I am currently downloading it.

https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

(in case other people need the link to the test vms, they are for ie testing but you can test other things with it)

Anyways so far the experience has been quite good compared to my tries without raco-cross maybe a year ago.

[Matthew Flatt]

At Sat, 15 May 2021 03:10:55 -0700 (PDT), "kamist...@gmail.com" wrote:
> I ran raco-cross
> (installed on racket cs 8.0 [does this cause trouble for racket cs < 8.2 or
> is it fine because host and target are both bc?])

Yes, it's fine because the target is BC (and a host BC is set up as
needed).

> On windows:
> 1. I had to add the lib subfolder in the dist to the path (else the exe
> would not find the dlls in there).

The intent in a Windows distribution is that the executable and "lib"
folder stay together, as they are in the archive. Was the executable
not able to find the folder content when kept alongside in the same
directory?

> 2. Copy the dlls from /tmp/todays-snapshot/i386-win32-bc/lib/ to the lib
> subfolder.

Which libraries were needed? They should have been pulled along in the
distribution's "lib" folder automatically, assuming they're referenced
in a Racket module with `define-runtime-path` (which modules like
`racket/draw` and `racket/gui` do for their libraries).

> I also tried the utah snapshots but they don't have i386 builds and the
> windows7 vm is 32bit,

The Utah snapshot's 32-bit Windows builds have been having trouble, and
it looks like they will fail again today. The Northwestern site
provides 32-bit Windows builds, but it's Linux builds have a naming
issue (so `raco cross` can't find things) that probably will be
resolved in the next build. Unless you need something new for your
code, though, the v8.l release builds should work for BC.


Matthew

[kamist...@gmail.com]

> On windows:
> 1. I had to add the lib subfolder in the dist to the path (else the exe
> would not find the dlls in there).

The intent in a Windows distribution is that the executable and "lib"
folder stay together, as they are in the archive. Was the executable
not able to find the folder content when kept alongside in the same
directory?

Yes it didn't find the libiconv-2.dll in the distribution "lib" sub-folder, only after I added that to my path.

Is there a specific function like `current-library-collection-paths` I should output from my executable,

to get insight into what might be wrong?

It seems as if "./lib" is meant to be a default search path, but isn't for some reason.

> 2. Copy the dlls from /tmp/todays-snapshot/i386-win32-bc/lib/ to the lib
> subfolder.

Which libraries were needed? They should have been pulled along in the
distribution's "lib" folder automatically, assuming they're referenced
in a Racket module with `define-runtime-path` (which modules like
`racket/draw` and `racket/gui` do for their libraries).

 I copied a few one by one, with the error message telling me which one it didn't have next, but eventually I got impatient and copied all that I had in that folder:

libatk-1.0-0.dll     libfribidi-0.dll      libiconv-2.dll           libpixman-1-0.dll
libcairo-2.dll       libgio-2.0-0.dll      libintl-9.dll            libpng16-16.dll
libeay32.dll         libglib-2.0-0.dll     libjpeg-9.dll            longdouble.dll
libexpat-1.dll       libgmodule-2.0-0.dll  libpango-1.0-0.dll       myssink.dll
libffi-6.dll         libgobject-2.0-0.dll  libpangocairo-1.0-0.dll  sqlite3.dll
libfontconfig-1.dll  libgthread-2.0-0.dll  libpangoft2-1.0-0.dll    ssleay32.dll
libfreetype-6.dll    libharfbuzz-0.dll     libpangowin32-1.0-0.dll  zlib1.dll

these were present before the copy: libiconv-2.dll  libracket3m_d8ary8.dll  longdouble.dll

I am not too sure where to start debugging this, any tips about values to look at would be appreciated.

[Matthew Flatt]

At Sat, 15 May 2021 05:18:22 -0700 (PDT), "kamist...@gmail.com" wrote:
> Yes it didn't find the libiconv-2.dll in the distribution "lib" sub-folder,
> only after I added that to my path.

Oh, right... I had forgotten already that the current release does
*not* cross-build correctly, even for a BC build, when the target needs
native libraries. That problem should be fixed in snapshots.

So, you were on the right track trying to use a snapshot, and it just
happens that neither site at the moment manages to cover the
combination of Linux and 32-bit Windows. We'll get the snapshot sites
in line soon, I hope.


Matthew

[kamist...@gmail.com]

utah has 32-bit Windows snapshots now again.

I did a cross build with:

>> Cross configuration
 Target:    i386-win32
 Host:      x86_64-linux

 Version:   current
 VM:        bc
 Workspace: /tmp/todays-snapshot

with `pkg install` `exe ...` `dist ...` moved dist over to windows vm and it ran fine,

until it hit a sqlite syntax that was too new for the version that is included in racket i386-win32

(it reported sql-lite version 3.22.0 from 2018-01-22).

I then downloaded 3.35.5 (current release) for windows replaced the sqlite3.dll in /tmp/todays-snapshot/i386-win32-bc/lib/ with the newer one,

repeated the exe and dist, after that the application worked without problems.

(So I guess the only question that comes up is: is it possible to update the sqlite version?)

[vaguely remember seeing a mailing list thread about this but, currently I only find other ones]

The build was pretty easy and working great (I guess with the next release it becomes even simpler, not having to use snapshots, but that is just one more command line argument anyway),

so thank you again, definitively makes distributing racket applications easier!

Simon

[Bogdan Popa]

kamist...@gmail.com writes:

> (So I guess the only question that comes up is: is it possible to update
> the sqlite version?)
> [vaguely remember seeing a mailing list thread about this but, currently I
> only find other ones]

My libsqlite3[1] package provides recent versions (currently 3.35.5) of
SQLite for Linux, macOS and Windows. If you add it to your app's
`deps`, everything should just work.

[1]: https://pkgd.racket-lang.org/pkgn/package/libsqlite3

[kamist...@gmail.com]

 

This package works great!

Now I could run my app without copying files.

Some caveats though (am I doing something wrong here?):

because db-lib already has an old version of the sqlite library `raco pkg install` complains that there is a conflict if I depend on db-lib and libsqlite3

I used `raco pkg install --force` to make it ignore the conflict

I wonder whether racket has a way to tell db-lib to not include its version of sqlite.

(would this require to split db-lib into smaller packages so that you can depend on hypothetical: db-interface-lib and not on db-sqlite-lib?)

If the conflict is present tools like `raco setup --check-pkg-deps --unused-pkg-deps ...`

don't understand that the db-lib version of sqlite is not wanted and has a different version, so it thinks libsqlite3 is not needed:

raco setup: unused dependency detected
  for package: "myapp"
  on package:
   "libsqlite3"