[ANN] Core Suite 112.24

[Thomas Refis]

I am pleased to announce the 112.24 release of the Core suite.

The following packages were upgraded:

- async
- async_extended
- async_extra
- async_kernel
- async_ssl
- async_unix
- bignum
- bin_prot
- core
- core_extended
- core_kernel
- custom_printf
- jenga
- ocaml_plugin
- pa_ounit
- pa_test
- patdiff
- patience_diff
- rpc_parallel
- sexplib
- typerep

Files for this release are available on our website and all packages
are in opam:

https://ocaml.janestreet.com/ocaml-core/112.24/files

Here is the list of changes for this version:

## 112.24.00

### async

Keep up to date with interface changes in `Async_kernel`, `Async_extra` and
`Async_unix`.

### async_extended

- Fixed misspelling in `Command_rpc.Connection`, renaming `propogate_stderr` as
`propagate_stderr`.

### async_extra

- Changed `Log` to not eagerly run the rotation loop when an
`Output.Rotating_file` is created.
- Changed `Log.Output.combine` to write log outputs in sequence rather than
parallel, to give the correct semantics when the same output is included
multiple times in `Log.create`.

This fixed a test that was failing in `lib_test/log_test.ml`.

- Remove `Log.Rotation.t_of_sexp`.
- Made `Command.async*` functions flush `stdout` and `stderr` before calling
shutdown, to avoid timeouts causing data to be dropped.

For now, we're making this change in `Command` rather than `Writer`.
`Writer` already has `at_shutdown` handlers. We've observed that they
don't behave well for command-line programs w.r.t. stderr. So, the
thinking of this feature is to try out a different `at_shutdown`
behavior, just for `Command` executables and just for `stdout` and
`stderr`. If it works out, maybe we move it into `Writer` proper.
Putting the change in `Command` for now reduces the scope of what is
affected by the experiment, and hopefully correlates well with where
the change is likely to help.

- In `Rpc`, catch exceptions raised by blocking-RPC implementations.

- Added functionality to `Versioned_typed_tcp.Repeater`.

Added to `create` an argument `is_client_allowed : Client_name.t -> bool`.

Added to `start` an argument:

on_connecting_error : (client_name : Client_name.t
-> server_name : Server_name.t
-> Error.t -> unit)

- Fixed a race in `Versioned_typed_tcp` in which a message can be dropped
between `Server.create` and `Server.listen`

- Simplified the implementation of `Rpc`.

Rpc has an internal Response_handler module, which is just a record containing
a response-handling function and an `already_removed` boolean field. It turns
out that this is unnecessary: `already_removed` is set to true when the
function returns ``remove`, but if it returns ``remove` then it will also be
removed from a hash table, and we only call the function immediately after
looking it up in that hash table.

This wasn't always pointless: this function used to return deferred
values and
run inside a throttle. The simplification is only possible because we made it
synchronous a while ago.

- Added `Tcp.Server.num_connections` function.

- Added creation functions for `Versioned_rpc` menus, for use in clients of an
RPC proxy.

In `Menu`:

val create : Implementation.Description.t list -> t

In `Connection_with_menu`:
val create_directly : Connection.t -> Menu.t -> t

These are for use in clients of an RPC proxy, which can't use the
regular menu mechanism since they each need to have many menus (one
for each potential target) but should only need to have one connection
(to the proxy).

- Added to `Rpc` expert submodules, `Implementations.Expert` and `Rpc.Expert`,
with low-level access for implementing a proxy that can handle queries without
knowing their names, types, etc. in advance.

- Renamed `Rpc.Implementation.Description` as `Rpc.Description`.

- Added `Rpc.{Rpc,Pipe_rpc,State_rpc}.description` accessor functions.

- Added `Rpc.Implementation.descriptions`, which returns all RPCs in an
`Rpc.Implementations.t`.
This was needed for the `rpc_discovery` library: given an
`Implementations.t` we want to advertise at

prefix/<rpc_name>/<rpc_version>/host_and_port = <host_and_port>

- Added combinators to `Rpc.Implementations`: `lift`, `add`, `add_exn`.

### async_kernel

- Now depends on `Core_kernel` instead of `Core`.

`Async_kernel.Clock` uses `Core_kernel.Time_ns` and
`Core_kernel.Timing_wheel_ns` rather than `Core.Time` and
`Core.Timing_wheel_float`.

- Added `Async_kernel.Types` module to deal with the mutual recrsion of
`Async_kernel`'s types.

This should help eliminate the complexity and make it easier to make changes
without running into as many constraints due to module/type ordering.

Merged `Types.Jobs` into `Types.Scheduler`.

- Improved the performance of `Deferred.bind`, eliminating an allocation in
`Ivar.connect`.

- Optimized `Deferred.bind`, removing a closure allocation by inlining
`Deferred.create`.

- Added `Pipe.interleave_pipe`, which is like `interleave`, but takes a pipe
rather than a list.

### async_ssl

- By default OpenSSL ignores the result of certificate validation, so we need to
tell it not to.

- Expose session details such as checked certificates and negotiated version.
Add session resumption.

### async_unix

- Made `Process.env` type equal `Core.Std.Unix.env` type, effectively adding the
``Replace_raw` variant.
- Renamed `Process.wait` as `collect_output_and_wait`, and added a `wait`
function that is a thin wrapper around `waitpid`.

Also renamed:

wait_stdout --> collect_stdout_and_wait
wait_stdout_lines --> collect_stdout_lines_and_wait

- Added `Unix.getgrouplist`, a wrapper around the eponymous function in core
- Change the Async scheduler to run external actions immediately upon
dequeueing them, rather than first enqueueing them in the normal job queue.

Also, made external actions be jobs rather than closures.

- Changed `Unix.Inet_addr.of_string_or_gethostbyname` to not use a sequencer.

We had used a sequencer to workaround bugs in winbind, which we don't
use anymore.

Reported on github: https://github.com/janestreet/async_unix/issues/4

### bignum

- Fixed exception raised by `Bignum.sexp_of_t` when the denominator is zero.

### bin_prot

Minor commit: comments.

### core

- Renamed `Dequeue` as `Deque`.
- Added `Fdeque`, a functional deque (a double-ended `Fqueue`, or a
functional `Deque`).
- Changed `Fqueue`'s bin-io format, and added a stable type.

Deprecated deque-like functions in favor of `Fdeque`.

- Added `Fheap`, a functional heap implementation based on pairing heaps.
- Reverted the change to `Or_error`'s bin-io format made in 112.17, going back
to the format in 112.16 and before.
- Added to `Unix.env` type a ``Replace_raw` variant, as used in `exec` and
`fork_exec`.
- Added `Array.Permissioned` module, which has a permissioned array type and
permissioned versions of all the regular array functions.
- Added `Date.week_number : t -> int`.
- Added values to `Day_of_week`: `of_int_exn`, `iso_8601_weekday_number`,
`weekdays`.

val of_int_exn : int -> t
val iso_8601_weekday_number : t -> int
val weekdays : t list (- [ Mon; Tue; Wed; Thu; Fri ] *)

- Switched `Float` IEEE functions to use `Int63.t` for the mantissa rather than
`int`, so they work on 32-bit platforms.
- Added a `length` field to the `Map.t` record, making `Map.length` `O(1)`
rather than `O(n)`.
- Moved a fragment of `Time_ns` from `Core` to `Core_kernel`, enough so that
`Async_kernel` can use `Core_kernel.Time_ns` and ultimately only depend on
`Core_kernel`.
- Fixed compilation of `Time_ns` 32-bit Linux.
- Added `Bounded_int_table.clear`.
- Fixed the `module_name` passed to `Identifiable.Make` for a number of modules.

The module name must be an absolute module path.

Reported here: https://github.com/janestreet/core/issues/52

- Added `Tuple.Binable` functor, for making binable tuples.
- Sped up a `Time_stamp_counter` unit test.

`Time_stamp_counter` unit test has an 18s unit test, which seems
excessive. Take a couple of orders of magnitude off the number of
iterations.

- Added `Time_ns.pause`, whose implementation is the same as `Time.pause`.

This involved moving the `nanosleep` C code from `Core` to
`Core_kernel`.

This was necessary so that `Async_kernel` can pause without introducing
a dependence of Async on Core.

- Made `Core_kernel.Time_ns.Alternate_sexp` use a similar format to
`Core.Time_ns`.

This was needed so that `Async_kernel` can use a nice sexp format for
time spans.

- Changed `Timing_wheel` implementation to use `Time_ns`, and moved to
`Core_kernel.Timing_wheel_ns`; made `Core.Timing_wheel` a wrapper around
`Timing_wheel_ns`.

Generalized the timing-wheel interface to be parametric in `Time`, so
that one interface applies to both `Timing_wheel` and
`Timing_wheel_ns`.

Generalized the timing-wheel unit tests to a functor,
`Timing_wheel_unit_tests.Make`, that is used to test both
`Timing_wheel_ns` and `Timing_wheel_float`. Moved a few tests that
depend on `Time` and `Date` from the functor into
`timing_wheel_float_unit_tests.ml`.

Split out `Timing_wheel.Debug` into a separate functor,
`Timing_wheel_debug.Make`.

This was done in so that `Async_kernel` can depend only on
`Core_kernel` and not `Core`.

- Added optional arguments to `Command.group`: `?body` and
`?preserve_subcommand_order`.

`preserve_subcommand_order : unit` causes subcommands to be in the order
they are specified, rather than sorted.

`body : (path:string list -> unit)` is called when no additional
arguments are passed.

- Added accessor function `Command.summary : t -> string`.
- Fixed a bug in `Time.Span` robust comparison.
- Changed `Command`'s tab-completion bash code so that it is possible for
programs to return completions containing spaces.

Actually knowing when and how to do so is difficult, because of
course there's escaping to
worry about. Adding helper functions to make that sort of thing
manageable is left for
future work.

- In `Command`, made the `-version` and `-build-info` flags work at the top
level when there are subcommands.
- Added `Sequence.interleaved_cartesian_product`, which implements cartesian
product of potentially infinite sequences.

### core_extended

- Added to `Shell.set_defaults` a `?preserve_euid:bool` argument, which causes
`Shell` to use `bash -p`.
- Removed `Array.Access_control`, now that there is
`Core.Std.Array.Permissioned`.
- Removed `Fast_int_div`.

### core_kernel

- Added `Time_ns` module.

A fragment of `Core.Std.Time_ns` is now in `Core_kernel.Std.Time_ns` such that
`Async_kernel` can use `Time_ns` and only depend on `Core_kernel`.

- Renamed `Dequeue` as `Deque`.
`Dequeue` remains for backward compatibility, but should not be used anymore.
Use `Deque` instead.

- Added `Fdeque` module, a functional version `Deque`.
Deprecate deque-like functions in `Fqueue`.

### custom_printf

- Added syntax to use `Sexp.to_string_mach` rather than `Sexp.to_string_hum`.

Added syntax `%{sexp#mach:<type>}`, which is like `%{sexp:<type>}`,
except it calls `Sexplib.Sexp.to_string_mach` instead of
`Sexplib.Sexp.to_string_hum`.

In fact, you can put any identifier after the `sexp#` and it will call
`Sexplib.Sexp.to_string_<that identifier>`; however, there are no other
such functions right now.

### jenga

- Interns strings, significantly reducing memory use.

### ocaml_plugin

Minor update: follow Async evolution.

### pa_ounit

- Added `-verbose` switch to time unit tests, so we can easily see which ones
are taking a long time.

### pa_test

- Reduce code generated by `pa_test` in favor of more code in `pa_test_lib`.
+ Less generated code means less time spent compiling it.
+ Reducing code in `pa_test.ml` is good, because:
a) Staging semantics make this code hard to understand.
b) This code is written using the less familiar revised OCaml syntax.
b) We get less code to migrate to syntax extensions.

- Re-used location code in `pa_here`; side benefit that we get full path names
in (Loc...) instead of just the file's basename.

### patdiff

Minor update: doc.

### patience_diff

Update references to `Core.Std.Dequeue` to refer to `Core.Std.Deque`

### rpc_parallel

- Added `Parallel.State.get` function, to check whether `Rpc_parallel` has been
initialized correctly.
- Added `Map_reduce` module, which is an easy-to-use parallel
map/reduce library.

It can be used to map/fold over a list while utilizing multiple
cores on multiple machines.

Also added support for workers to keep their own inner state.

- Fixed bug in which zombie process was created per spawned worker.

Also fixed shutdown on remote workers

- Made it possible for workers to spawn other workers, i.e. act as masters.

- Made the connection timeout configurable and bumped the default to 10s.

### sexplib

Minor update: documentation.

### typerep

- Remove unused "bin_proj" rewriter.

--
Thomas Refis, for the Core team