changes to the racket repo

[Matthew Flatt]

Changes are coming to the https://github.com/racket/racket repo.

Within a few days, when you `make` from the Git repo, you'll end up
running Racket CS as `racket` (and DrRacket CS as `drracket`, etc.)
instead of Racket BC. This does *not* mean that the snapshots or the
release will immediately change to Racket CS, although it's part of a
plan to get there.

Even this part of the plan requires two steps. The first step will
happen right away, and it's mostly about improving the Racket CS build
process and removing the dependency of building Racket BC first:

* The https://github.com/racket/ChezScheme repo will be included as a
subtree of https://github.com/racket/racket.

Combining the two repos makes it easier to keep them in sync for
various purposes, such as designating releases or when a pull
request needs to span the two repos. Git submodules are another
possible approach to that problem, but both past experience and new
experiments suggest that submodules will create problems. The
subtree pattern still allows changes to be pulled from and
backported to the ChezScheme repo.

The ChezScheme repo's history will be preserved while merging it
into the Racket repo, but boot files will be pruned from the
history, so all of the commit IDs will change --- but the old
commit IDs will be preserved as amendments to each commit message.

The submodules already in the ChezScheme repo are also merged, but
as squashed subtrees, because history and potential backporting are
not really a concern there.

* Racket CS will build directly without first building Racket BC.

The key here is a new pb (portable bytecode) backend for Chez
Scheme, where a single set of boot files work for all supported
platforms, and those boot files can be used to build the native
boot files on each platform. A single set of pb boot files is an
alternative to the multiple sets of boot files (for a subset of
supported platforms) that were originally included in the Chez
Scheme repo.

The new pb boot files are not added to the `racket` repo, however.
They're checked into a different repo
https://github.com/racket/pb
and every revision will go into a fresh branch of that repo (i.e.,
no branch has history). This strategy works something like
submodules, but managed more explicitly by the top-level makefile
in the `racket` repo to avoid submodule pitfalls.

The pb boot files will be included in a source bundle, so that the
source can be built on any supported platform using only
conventional tools (like `make`, a C compiler, etc.).

Alternatively, Racket CS can be bootstrapped without pb boot files
by running an old Racket --- at least v7.1, but not necessarily the
same version as the one being built. In this sense, the pb boot
files really aren't needed to recreate a build easily, so it's
reasonable for them to be in a separate repository.

* Schemified variants of the Racket-implemented layers of Racket CS
will be checked into the repository.

This is another component of building Racket CS directly. Layers
like "thread", "io", and "expander" are implemented in Racket and
compiled to Chez Scheme. Unlike boot files, the compiled ".scm"
forms are human-readable and -editable, and they diff nicely: small
changes to the source tend to create small changes in the ".scm"
form.

Also unlike boot files, they must be created by a similar Racket
version, so all the traditional bootstrapping trickiness applies,
mitigated only somewhat by the fact that the output is editable.
They're not easily recreated for an old version (at least, without
going through BC), so they make sense to check in.

This first step will happen tomorrow, pending any objections or
problems discovered before then.

The second step is where `racket` starts meaning Racket CS. It will
take some time (likely measured in days rather than hours or weeks),
but it will be the more significant change if you're not actively
developing Racket internals:

* A `make` at the top level of a repo checkout will build Racket CS
instead of Racket BC.

The generated Racket CS executable will be `racket` by default, and
a new `bc` target will create a Racket BC executable named
`racketbc` by default. The `both` target will create `racket` and
`racketbc`.

An executable with a `bc` suffix will put ".zo" files in a
subdirectory of "compiled", similar to the way that `racketcs`
currently puts compiled files in a subdirectory, so the two
variants can coexist in a build.

* The "racket/src/racket" directory within the repo will move to
"racket/src/bc/racket".

Some sibling directories, like "gracket" and "foreign", probably
also move into "bc" to clarify that they are specific to the BC
build.

We can take these steps without shifting the snapshot or release
builds to Racket CS, but this will already be a big change for anyone
working from the Git repo.


Matthew

[Matthew Flatt]

These changes are now all in place. A top-level `make` in a checkout of
the `racket` Git repo will now build Racket CS as `racket`.

The snapshot builds will still be as before --- except that in new
source bundles, the `configure` script selects a CS build by default.
From a source bundle, configure with `--enable-bcdefault` to build
Racket BC, instead.


Hopefully, a `make` after `git pull` will update an existing checkout,
but it will certainly leave some obsolete files behind.

If you DO NOT have any linked clones for packages, and it you don't
have any other installation state that you want to keep, then consider
using `git clean -d -x -f .` to reset your checkout. If you have a
"racket/src/build/ChezScheme" directory, it will stick around even
after `git clean`, since that's a checkout of another repository,;
you'll have to delete it manually.

A more conservative approach to clean up your checkout is to manually

* discard "racket/src/build",

* remove any subdirectories of a "compiled" subdirectory, such as
"compiled/ta6le", and

* remove "racket/bin/*.cs" and/or "racket/*CS.app" and/or
"racket\*CS.exe".

Matthew

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/20200728194813.2ce%40sirmail.smtp.
> cs.utah.edu.

[Laurent]

A more conservative approach to clean up your checkout is to manually

 * discard "racket/src/build",

 * remove any subdirectories of a "compiled" subdirectory, such as
   "compiled/ta6le", and

 * remove "racket/bin/*.cs" and/or "racket/*CS.app" and/or
   "racket\*CS.exe".

I followed this first (though I actually removed every `compiled` subdirectory too), then `git pull`, then `make`, but it failed with:

checking for unistd.h... yes
checking uuid/uuid.h usability... no
checking uuid/uuid.h presence... no
checking for uuid/uuid.h... no
Could not find uuid/uuid.h, needed for Chez Scheme
make[6]: *** [Makefile:289: racket/src/build/cs/c/Makefile] Error 1

So then I did `git clean -d -x -f .` but it fails with the same error.

What's missing?

[Matthew Flatt]

At Sun, 2 Aug 2020 16:54:33 +0100, Laurent wrote:
> I followed this first (though I actually removed every `compiled`
> subdirectory too), then `git pull`, then `make`, but it failed with:
> checking for unistd.h... yes
> checking uuid/uuid.h usability... no
> checking uuid/uuid.h presence... no
> checking for uuid/uuid.h... no
> Could not find uuid/uuid.h, needed for Chez Scheme
> make[6]: *** [Makefile:289: racket/src/build/cs/c/Makefile] Error 1

This is a new dependency for Racket CS. Try installing "uuid-dev" or
"libuuid-dev" with your OS's package manager.

I'll add a note about the dependency, but I'm not sure there's anything
else we could do. Unlike something like libffi, say, which we have
traditionally pulled it into the Racket repo, the uuid library varies
across platforms. And even if uuid was always the same implementation,
pulling that kind of library into the Racket repo hasn't clearly been a
good idea.

Matthew

[Laurent]

That works, thanks. I don't mind the dependency (it's very small), but maybe a clearer error message would be useful (and less of a burden to you eventually).

[Philip McGrath]

There's an issue about this on the upstream Chez Scheme repository, and I mentioned in a comment that generating Type 4 UUIDs is not especially hard: I have a Racket package that does it. The sense from the Chez maintainers seemed to be that libuuid is convenient, but the considerations may weigh differently for Racket. At least for Racket, we already require platforms to provide a source of randomness to implement `crypto-random-bytes`.

-Philip

 

[Matthew Flatt]

At Sun, 2 Aug 2020 13:06:05 -0400, Philip McGrath wrote:
> There's an issue about this on the upstream Chez Scheme repository, and I
> mentioned in a comment

> <https://github.com/cisco/ChezScheme/issues/458#issuecomment-560061318>

> that generating Type 4 UUIDs is not especially hard: I have a Racket package

> <https://github.com/LiberalArtist/uuid> that does it. The sense from the

> Chez maintainers seemed to be that libuuid is convenient, but the
> considerations may weigh differently for Racket. At least for Racket, we
> already require platforms to provide a source of randomness to implement
> `crypto-random-bytes`.

Yes, that sounds like a good option, and I'll look into it more.

Thanks!
Matthew