[ANN] Core Suite 112.01.00 + rpc_parallel

[Jeremie Dimino]

I am pleased to announce the 112.01.00 release of the Core suite.

Starting with this releases core packages require OCaml 4.02.

We are also releasing a new package: rpc_parallel. From the package

description:

  Rpc_parallel offers an API to define various workers and protocols,

  spawn workers as separate processes, and communicate with them using

  Async Rpc.

The following packages were upgraded:

- async

- async_extended

- async_extra

- async_kernel

- async_unix

- bignum

- bin_prot

- core_bench

- core

- core_extended

- core_kernel

- custom_printf

- jenga

- ocaml_plugin

- rpc_parallel

- sexplib

- textutils

- type_conv

Files for this release are available on our website and all packages

are in opam:

  https://ocaml.janestreet.com/ocaml-core/112.01.00/individual/

Unfortunately the documentation is not there yet due to opam-doc not

working with 4.02.

Here is list of changes for this version:

# 112.01.00

## async

- update tests

## async_extended

- Clarified an error in `Rpc_proxy`.

## async_extra

- Changed `Persistent_rpc_client.connected` to avoid returning a

  connection that is closed at the time it was called.

- Optimized `Rpc.implement` so that if a server's implementation

  returns a determined deferred, then the output is immediately

  serialized and written out for the client.

  This reduces memory consumption, improves throughput and latency.

  Measurements with the `pipe_rpc_test program` showed that a server

  went from processing 600\_000 msg/sec, to 2\_200\_000 msg/sec before

  pegging the CPU.

- Changed `Log`'s output processor's batch size from `1_000` to `100`.

- Added `Persistent_rpc_client.close` and `close_finished`.

- In `Rpc.Connection.client` and `with_client`, used the

  `handshake_timeout` as the `timeout` passed to `Tcp.connect`.

  `handshake_timeout` was previously used only for the `Rpc` module's

  handshake timeout.

- Changed `Rpc.create`'s `on_unknown_rpc` argument, renaming

  `\`Ignore` as `\`Close_connection`, and requiring `\`Call` to return

  `\`Close_connection` or `\`Continue`.

  `\`Ignore` was renamed because it was a poor name, since in fact it

  closed the connection.

  Added a `\`Continue` option, whic allows one to keep the connection

  open.

  Changed `\`Call` to return `\`Continue` or `\`Close_connection`,

  where the old `unit` return value meant `\`Close_connection`.

- In `Versioned_typed_tcp`, enabled the use of "credentials" in the

  "Hello" message.

  Propagate credentials to the user code when it arrives on the wire.

## async_kernel

- Optimized `Monitor.try_with ~run:\`Now f` to return a determined

  deferred if `f ()` returns a determined deferred.

  Previously, `Monitor.try_with ~run:\`Now f` always introduced a

  `Deferred.map`, which made it impossible to do some optimizations

  that bypass the scheduler overhead.

- Added an `ASYNC_CONFIG` field that causes the program to dump core

  if Async jobs are delayed too much.

  The new field is `dump_core_on_job_delay`.

- Switched `Async_kernel` from using `Core.Sys` to `Pervasives.Sys`

  eliminating one of the dependencies on `Core`.

## async_unix

- Changed `Writer.transfer write pipe` to close `pipe` when the

  `writer`, is closed.

  Previously, `Writer.transfer` did not close the pipe when the

  underlying writer is closed.  This was strange because:

  1. Callers would have to consistently check for the writer being

     closed and close the `Pipe.Reader`t= themselves

  2. The analogous function `Pipe.transfer` closes the reader on

     similar circumstances.

  The absence of the close was noticed as a bug in `Rpc`, which

  assumed that `Writer.transfer` did the close.

- Fixed a bug in `Scheduler.yield` that caused it to pause for 50ms if

  there is no other pending work and no I/O.

- Exposed type equivalence between `Unix.Passwd.t` and

  `Core.Std.Unix.Passwd.t`.

- Changed `Writer.write_bin_prot` to use the new

  `Bigstring.write_bin_prot`.

## bignum

- Added `Bignum.Bigint` module, with arbitrary-precision integers

  based on `Zarith`, which is significantly faster than the

  `Num.Big_int` library.

## bin_prot

- In `Write`, improved some OCaml macros to name values and avoid

  calling C functions multiple times.

## core

- Removed vestigial code supporting OCaml 4.00.

- Added `Command` support for flags that are passed one or more times.

  Added `Command.Spec.one_or_more` and

  `Command.Spec.non_empty_sequence` to deal with the cases where you

  expect a flag or anonymous argument (respectively) to be passed one

  or (optionally) more times.  This is common enough and distinct from

  the case where you want the argument passed zero or more times that

  it seems like we should canonize it in the library.

- In `Lock_file`, made stale lock detection more robust.

  Made `Lock_file.create foo` succeed if `foo` is absent and

  `foo.nfs_lock` file is present and stale.  Previously, it would

  fail.

- Removed `Syslog.syslog`'s `add_stderr` argument; use the `PERROR`

  option instead.

- Fixed `unix_stubs.c` compilation on NetBSD

  Closes #45

- Added `Filename` operators `/^` and `/@`, and `of_parts`, like the

  same functions for Catalog paths.

- Changed `Iobuf` functions that advance the iobuf to not also return

  a redundant number of bytes processed.

  This avoids a small allocation (in the case of the `int option`

  functions) and normalizes the result (so the same information isn't

  returned two ways).  Actually, it doesn't yet avoid the allocation in

  the implementation, as the corresponding `Bigstring` functions must

  still return the number of bytes processed, and currently do so as an

  option.  We hope to eventually change that.

  In the future I expect we will change `unit` to some `error` variant

  to also avoid the exception construction for `EWOULDBLOCK/EAGAIN`.  We

  can even make Unix syscalls `noalloc` if we're careful.

- In `Unix` module, added unit tests for `Cidr.does_match`.

## core_bench

- fixed legacy format string

## core_extended

- Added `Float_ref` module, which is like `float ref` but faster for

  sets due to bypassing the write barrier.

  Benchmark results on Sandy Bridge:

  | [float\_ref.ml:] float ref set  |    2\_886.94ns |   8.00w |            |

  | [float\_ref.ml:] Float\_ref.set |       355.76ns |   6.00w |            |

  | [float\_ref.ml:] float ref get  |       415.52ns |   6.00w |            |

  | [float\_ref.ml:] Float_ref.get  |       416.19ns |   6.00w |            |

- Added `Bin_io_utils.Wrapped.t`, which defines an `'a t with bin_io`

  that supports size-prefixed serialization and deserialization.

  `Wrapped` has two useful submodules, `Opaque` and `Ignored`, for

  efficient handling of size-prefixed bin-io values in cases where

  serialization can be bypassed.  See the comments in the module for

  more details.

## core_kernel

- Removed vestigial code supporting OCaml 4.00.

- Used `{Hashable,Comparable}.S_binable` in `Day_of_week` and `Month`.

- Improved the performance of `Set_once.set`.

- Added `Type_equal.Lift3` functor.

- Replaced occurrences of `Obj.magic 0` with `Obj.magic None`.

  With the former the compiler might think the destination type is

  always an integer and instruct the GC to ignore references to such

  values.  The latter doesn't have this problem as options are not

  always integers.

- Made `String_id.of_string` faster.

- Added `Bigstring` functions for reading and writing the

  size-prefixed bin-io format.

  - `bin_prot_size_header_length`

  - `write_bin_prot`

  - `read_bin_prot`

  - `read_bin_prot_verbose_errors`

- Added `{Info,Error}.to_string_mach` which produces a single-line

  sexp from an `Error.t`.

- Added `{Info,Error}.createf`, for creation from a format string.

- Added new `Perms` module with phantom types for managing access

  control.

  This module supersedes the `read_only`, `read_write`, and

  `immutable` phantom types, which are now deprecated, and will be

  removed in the future.  This module uses a different approach using

  sets of polymorphic variants as capabilities, and contravariant

  subtyping to express dropping capabilities.

  This approach fixes a bug with the current phantom types used for

  `Ref.Permissioned` in which `immutable` types aren't guaranteed to

  be immutable:

  ```ocaml

  let r = Ref.Permissioned.create 0

  let r_immutable = (r :  (int, immutable) Ref.Permissioned.t)

  let () = assert (Ref.Permissioned.get r_immutable = 0)

  let () = Ref.Permissioned.set r 1

  let () = assert (Ref.Permissioned.get r_immutable = 1)

  ```

  The bug stems from the fact that the phantom-type parameter is

  covariant, which allows OCaml's relaxed value restriction to kick

  in, which allows one to create a polymorphic value, which can then

  be viewed as both immutable and read write.  Here's a small

  standalone example to demonstrate:

  ```ocaml

  module F (M : sig

              type +'z t

              val create : int -> _ t

              val get : _ t -> int

              val set : read_write t -> int -> unit

            end) : sig

    val t : _ M.t

  end = struct

    let t = M.create 0

    let t_immutable = (t :  immutable M.t)

    let () =

      assert (M.get t_immutable = 0);

      M.set t 1;

      assert (M.get t_immutable = 1);

    ;;

  end

  ```

  The new approach fixes the problem by making the phantom-type

  parameter contravariant, and using polymorphic variants as

  capabilities to represent what operations are allowed.

  Contravariance allows one to drop capabilities, but not add them.

- Added `Int.Hex` module, which has hexadecimal sexp/string

  conversions.

- Added `Gc.major_plus_minor_words`, for performance reasons.

## custom_printf

- Fixed uses of `printf=-style`format strings that have unspecified

  behavior in OCaml 4.02 and will become errors.

- Support substitution in format string (lost with 4.02 compatibility)

## jenga

- Don't show noisy `glob..changed` messages except with `-show-glob-changed` flag.

- Support shared build rules via `${jenga}/share`.

- Detect cycle in dep scheme instead of hanging.

- Made standalone actions atomic, just like actions associated with

  target files.

  Running actions and recording the result in the persistent

  `.jenga.db` should be performed atomically for standalone actions,

  as it is for actions which are associated with target files

## ocaml_plugin

- Changed to not use `rm -r` when it is expected to remove one file.

## rpc_parallel

Initial import.

## sexplib

- Replaced occurrences of `Obj.magic 0` with `Obj.magic None`.

  With the former the compiler might think the destination type is

  always an integer and instruct the GC to ignore references to such

  values.  The latter doesn't have this problem as options are not

  always integers.

## textutils

- Added `Ascii_table.Table_char`, to expose the special table border

  characters.

## type_conv

- Updated ast matching for 4.02

-- 

Jeremie Dimino, for the Core team