[ANNOUNCE] urbit-v0.10.8

[joe bryan]

urbit-v0.10.8

Release binaries:

(macOS)
https://bootstrap.urbit.org/urbit-v0.10.8-darwin.tgz

(linux64)
https://bootstrap.urbit.org/urbit-v0.10.8-linux64.tgz

Release notes:

This is a significant refactoring (approaching a rewrite) of the i/o layer
in the urbit runtime.

* The IPC protocol between the "king" (urbit: i/o & event persistence) and
"serf" (urbit-worker: compute & state persistence) processes has been replaced;
the new protocol avoids head-of-line blocking, includes full error reports for
all events, efficiently replays events, and supports arbitrary reads from the
urbit namespace.

* The C reference implementations of the king and serf (colloquially, "vere")
have been rewritten to support and take advantage of the new protocol. Notable
features include prioritized event scheduling, capped packet queue, periodic
crash reports, i/o status reports with SIGINFO (SIGUSR1 on linux), namespace
reads (with -X and -Y), and partial replay (with -n and -r).

* The ncurses/terminfo dependency has been removed; release bundles no longer
include a terminfo database, and urbit can be run via $PATH.

* The haskell rewrite of the "king" process (colloquially, "king-haskell")
has reached approximate feature-parity with vere, and is now included in
release bundles as "urbit-king". This is but an initial, soft release:
king-haskell is intended to supplant the vere "king" (if the latter lives on
it will be as a development sandbox and lightweight reference implementation).
Try it out!

Changelog:

Brendan Hay (3):
      build: add pthread dependency to LDFLAGS
      u3: fix format-truncation errors in GCC 8
      serf: fix format-truncation errors in GCC 8

Elliot Glaysher (1):
      king: Build release tarballs with both king and vere.

Fang (12):
      Merge branch 'master' into m/eyre-scries
      vere: add -X flag for running a scry
      vere: warn on invalid behn doze
      vere: scry out next behn timer for backstop
      vere: rename behn.c's alm -> alm_o
      vere: if behn scry fails, don't try again
      vere: u3_nul in place of c3__null
      vere: support saving scry jam to directory
      Merge pull request #3049 from urbit/m/scry-command
      vere: rename behn.c's alm -> alm_o
      vere: more readable http server status
      Merge branch 'master' into ipc-redux

Jared Tobin (1):
      vere: increase LMDB mapsize to 1TB on macOS, linux

Joe Bryan (151):
      vere: WIP serf
      u3: adds u3r_safe_* accessors
      vere: WIP adds auto.c (i/o driver abstraction)
      vere: WIP adds lord.c (serf controller)
      u3: properly disables stack-overflow handling via NO_OVERFLOW
      vere: WIP serf protocol tweaks
      vere: WIP pier/serf rewrite
      vere: distinguishes lmdb error messages
      vere: refactors event replay, fixing bugs
      vere: improves lord's graceful serf shutdown
      vere: WIP moves behn.c to u3_auto
      vere: WIP moves ames.c to u3_auto
      vere: WIP moves unix.c to u3_auto
      vere: WIP moves cttp.c to u3_auto
      vere: removes reverse, reverse tcp proxy
      vere: WIP moves http.c to u3_auto
      vere: WIP moves term.c to u3_auto
      vere: prevents term.c from retrying tcsetattr() indefinitely
      vere: removes reck.c, all i/o drivers use u3_auto
      vere: disposes of ovum when done and on exit
      vere: plugs leaks in cttp
      vere: refactored event batching to serf, fixing snapshots
      c3: use c3_assert in c3_stub
      vere: refactors lmdb.c, separating it from u3 and uv
      vere: refactors disk.c
      vere: refactors lord.c
      vere: re-enables event spinner
      vere: refactors init/boot/play state transitions
      vere: refactors auto.c and i/o driver integrations
      vere: WIP moar serf refactoring, leaks appear fixed
      vere: WIP refactors serf
      vere: adds hack to fix u3l_log printing across the pipe
      vere: making i/o driver bail callbacks optional
      vere: sets up i/o driver async initialization
      vere: renames newt callback typedefs
      vere: begins vere.h header cleanup
      vere: WIP refactor types, enable peek
      vere: misc tweaks
      blah i/o driver integrations
      vere: free i/o drivers on exit
      vere: much more pier refactoring
      vere: refactors ovum lifecycle callbacks
      wip rewrites peek
      vere: refactors event replay and failure printfs
      TMP kill -9 urbit in test
      u3: switches GMP import assert to conditional
      u3: fixes rock:load size printf
      serf: tweaks pack for recovery ops
      serf: fix bug on auto-reclaim on replay
      serf: implements %pack writ
      vere: fixes early replay termination
      vere: adds -r to load rock (jammed kernel) on startup if present
      vere: adds -n replay barrier
      vere: print replay batch completions
      vere: bumps replay batch size to 500 events
      serf: auto-pack every 50K events
      vere: HACK pack on -n completion
      blah roc
      vere: use separate timers for each disk read request
      serf: measure memory after %pack writ
      serf: fixes -r load by tracking event number
      vere: fix ames port in startup printfs
      serf: clean up auto reclaim/pack conditions, pack every 20k
      WIP kh ipc protocol
      u3: fixes use of system malloc in u3i_chubs
      u3: cleans up imprison.h/c
      u3: rewrites u3i_chubs() for efficiency
      u3: adds allocator profiling labels to u3i_* atom functions
      u3: implements u3i_list()/u3nl() varargs list constructor
      u3: adds failing u3r_mug_words tests
      u3: rewrites u3r_mug_words, correct for arbitrary input
      u3: use u3r_mug_words for all atoms
      newt: print errors to stderr
      newt: cleans up api, allocates less on write
      newt: delivers inbound messages asynchronously
      newt: adds synchronous read, used in serf with blocking writes
      serf: plugs leak of error-notification event
      serf: refactors %work $writ handling
      serf: disables auto |pack, refactors loading from rock (-r)
      Merge branch 'master' into ipc-redux-kh
      term: bypass terminfo, use direct ANSI escape sequences
      vere: removes ncurses dependency
      build: remove duplicate zlib dependency declaration
      vere: track ovum queue depth per i/o driver
      ames: cap ovum queue at 1k, dropping oldest packets first
      serf: disables incomplete "static grab" on boot
      Merge branch 'release/next-vere' into ipc-redux
      ames: fix null-deref in capped queue
      vere: fixes u3_auto_drop list-unlink implementation
      vere: stop leaking effects
      vere: moves i/o drivers
      vere: switch lmdb back to the u3 prefix
      vere: adds common structure management in ward.c
      vere: properly dispose disk resources on exit
      vere: adds u3_ovum init/dispose to ward.c
      vere: renames "daemon" to king, removes command socket
      newt: adds stop/dispose functions for read and write
      vere: refactors king/pier/lord shutdown
      term: ignore SIGWINCH if there's no active terminal driver
      vere: cleans up i/o driver shutdown
      ames: count dropped packets (and print every 1k with -v)
      pier: further limit activity in emergency exit (SIGABRT)
      Revert "TMP kill -9 urbit in test"
      vere: disable core dumps in king
      vere: fixes doubly-linked list handling in u3_auto_drop
      Merge branch 'khrc' into ipc-redux
      vere: removes now-unused sniproxy dependency
      vere: removes SIGQUIT handler
      vere: free pier on graceful shutdown
      vere: fix use-after-free on exit in cttp
      vere: adds basic info printfs on SIGINFO/SIGUSR1
      vere: completes daemon->king re-renaming
      vere/king/serf: adds and enables event timeouts
      king: fixes --serf worker-path cli parser
      vere: skip SIGINFO on linux (unavailable)
      vere: handle partial writes and retry in the terminal
      vere: initialize king timer early (for reliable early-exit)
      vere: factors common _mcut functions into ward
      vere: silence ames errors unless -v, prints stats every 1k
      vere: tweaks pier/lord info printfs
      vere: adds pending ipc message to lord status printfs
      vere: restores arbitrary event injection with -I
      vere: fixes re-entrancy bug in term.c write() wrapper
      Merge branch 'ipc-redux' into jb/ipc-redux
      vere/king: tweaks serf startup args (unconditionally send all)
      Merge branch 'master' into jb/ipc-redux
      Revert "Merge pull request #3050 from urbit/m/behn-improvements"
      vere: adds/improves error messages for term syscall retry loops
      nix: add explicit shutdown to solid-pill derivation
      vere/serf/king: support timeouts and error response in %peek
      vere: retry %behn %born event failures
      vere: refactors %ames packet failure callback
      vere: refactors ovum scheduling api
      vere: corrects all uses of [vod_p] to [ptr_v]
      vere: refactors ovum and callbacks to support retry
      vere: retry behn timer failures
      vere: remove stale function declarations
      vere: refactor %behn's retries of failed events
      Merge branch 'master' into release/next-vere
      serf: fix incorrect structure in %peek response
      vere: replaces "irrelevant" uses of strncpy
      Merge branch 'release/next-vere' into ipc-redux
      vere: fixes driver exit in u3_pier_bail()
      vere: wires |pack to new compaction; renames old |pack to |cram
      serf: restores autopack in response to memory pressure
      serf: adds subcommands to urbit-worker
      u3: refactors u3m_reclaim() into noun modules, works on any road
      u3: moves u3a_compact to u3m_pack, refactors internals
      serf: tweaks |pack and |mass printfs
      u3: note that u3a_rewrit* doesn't yet support south roads
      Merge pull request #3099 from urbit/philip/compact

Philip Monk (11):
      Merge remote-tracking branch 'origin/master' into ford-fusion
      noun: road-size printfs on u3m_fall
      Merge pull request #3012 from urbit/philip/road-size
      Merge remote-tracking branch 'origin/master' into release/next-vere
      Merge remote-tracking branch 'origin/master' into ford-fusion
      Merge branch 'ford-fusion' (#3060)
      noun: first pass at memory compaction
      noun: clean up memory compaction
      noun: don't add identical values to song_x_cape's hashtable
      Merge remote-tracking branch 'origin/master' into release/next-vere
      Merge pull request #3131 from urbit/philip/sing

Ted Blackman (2):
      ci: fix herb tests; update pills
      -test: support other desks (full beams in args)

b (32):
      urbit-king: Library for getting current terminal size (SIGWINCH).
      urbit-king: urbit `urbit-termsize` library.
      king: Command Line Parsing for multi-tenet king.
      king: Support running multiple ships at the same time.
      Merge branch 'master' of github.com:urbit/urbit into two-ship-king
      king: First stab at handling ship crashes gracefully.
      Correctly shutdown on SIGTERM/SIGINT (fixing bad exception handling code).
      king: CLI parsing for ship-shared HTTP(S) server.
      king: Started implementing multi-tenet HTTP.
      king: Further sketch of shared http.
      king: Factored out UDP flow from Ames driver.
      king: Factored out DNS flow from Ames driver.
      Begin factoring out Raw HTTP logic from Eyre.
      king: Renamed %eyre modules.
      king: Factor out HTTP server lifecycle from Eyre + impl multi-tenet HTTP.
      king: Finish factoring out HTTP server lifecycle from Eyre.
      king: eyre: Finished multi-tenet HTTP flow (not yet started by king).
      king: eyre: Hooked up multi-tenet HTTP. Start-up is a bit hacky. Still totally untested.
      king: eyre: Got multi-tenet HTTP working.
      king: eyre: [WIP] Got multi-tenet HTTPS working.
      king: Flags to disable HTTP/HTTPS/UDP per-ship.
      king: eyre: [WIP] Got multi-tenet HTTPS working.
      king: Use same logger from all contexts.
      king: Logging cleanup.
      king: Use random numbers for request IDs.
      RIO environment cleanup.
      king: Skecthed out lowest layer of new IPC protocol.
      HasKingEnv + HasPierEnv + misc cleanup
      king: Get tests to compile.
      Merge pull request #2831 from urbit/two-ship-king
      Merge branch 'khrc' of github.com:urbit/urbit into ipc-redux-kh
      Fix `make test` + no logging during king-haskell tests.

pilfer-pandex (6):
      king: drop ames packets when >1k are unprocessed
      king: fix goshdarn typo
      king: ames bounded q, now with logging and fifo
      king: fix comment about ames q behavior
      king: it was too clever of me to use stateTVar; compiler can't help
      king: fix ames tests

ryjm (1):
      nix: fix boot-ship.nix

~siprel (51):
      king: Fleshed out some IPC flows for new protocol.
      king: Got new IPC working, still needs lots of polish.
      king: Handle replacement events correctly (dont try to parse them).
      king: Review and Cleanup w/ Harrison
      king: Better handling of edge-cases around IPC failure.
      king: Misc Small Cleanup.
      king: Slogs routed to both terminal and log output.
      king: IPC Batching during replay and normal operation.
      king: Make sure spinner callbacks are handled correctly from serf.
      king: Cleanup IPC batching in `swimming` flow.
      king: Remove `GoodParse` hack in `newRunCompute`.
      king: Urbit.Vere.King.IPC cleanup + handle scry/pack.
      Docs and cleanup for `Urbit.Vere.Serf.IPC`.
      king: Re-Implemenet collectFX and log events as passed to the serf.
      king: Each event comes with error callback, but all do nothing for now.
      king: snapshot requests block until event log catches up to serf.
      king: TODO.md cleanup.
      ^C brings king/serf down cleanly and promptly.
      Got bulshit scry working and hooked up replay progress callback.
      king: Cleanup date serialization and move to Urbit.Time.
      king: In non-daemon-mode, ^D now actually brings down the executable.
      king: Shutdown TMVars now live in KingEnv/PierEnv.
      king: Pier cleanup pass.
      king: Progress bars on replay.
      King: Fixed a few bugs.
      king: Module re-org in prepraration for new IO-Driver interface.
      king: TODO notes about how to break into packages.
      king: Spin off `urbit-noun-core` and `urbit-noun` packages.
      king: Snapshot on shutdown; no snapshot on start if no events to replay.
      king: Separate package for `racquire` + small cleanup.
      king: Minor cleanup.
      king: Separate out urbit-eventlog-lmdb package.
      king: Urbit.Vere.Pier cleanup.
      king: Event prioritization and error handling for born events.
      king: Fix spinner bugs.
      king: handle tanks that are just a cord.
      king: TODO.md updates.
      king: ^C from connected terminals kills current event.
      king: serf stderr goes to terminals.
      In non-daemon mode, initial serf slogs are user-visiable.
      king: Handle %vega/%exit, %wack on boot, and stop logging secrets.
      CLI option for serf exe + serf exe/flags live in PierConfig.
      king: fix behn tests.
      king: disable "bullshit scry tester"
      travis: Cache test results in cachix.
      Less verbose log output.
      king: Command line flag to enable verbose logging.
      travis: Various fixes and improvements.
      Merge branch 'master' of https://github.com/urbit/urbit into siprel/sigwinch
      Merge branch 'khrc' of https://github.com/urbit/urbit into siprel/sigwinch
      urbit-termsize: Change test executable name.

[joe bryan]

I missed one of the most important notes:

* |pack (memory compaction) has been replaced with a new implementation;
up to orders of magnitude faster, and always safe to run.

The old implementation used +jam and +cue, performing full deduplication,
serialization, and deserialization. The new implementation defragments
the heap in place, without deduplication or allocation. In short, the old
implementation was superior when it worked, but it didn't always work,
and it could be very slow in some scenarios. The new implementation always
works.

The old implementation lives on in the new partial replay feature (the -n and
-r arguments). For ship recovery purposes (and as a temporary measure),
both implementations are available directly in the urbit-worker binary,
simply run it to see usage details.

More to come soon!