[ANNOUNCE] urbit-v1.14

[joe bryan]

urbit-v1.14

Arvo 417K
Vere 1.14

This release implements demand paging for the snapshot, improves
efficiency and error handling in snapshot updates, and adds a
replay command with improved error handling and retry logic.

Release Notes

- adds demand paging, implemented with a file-backed, copy-on-write
mapping of the home-road heap (north.bin). this drastically reduces
effective memory usage and improves performance under memory pressure,
in most situations. demand paging is on by default where supported
(everywhere but Windows), and can be disabled with --no-demand.

- adds the `play` command, for improved replay of large event logs,
with automatic retry of bail:meme after pack (or meld with --auto-meld),
snapshot updates after errors or interruption (via ctrl-c), and
optional full replay (without editing files in the pier).

- rewrites snapshot updates to involve far fewer syscalls, thanks to
explicitly-positioned i/o (half as many syscalls in patch application)
and coalesced memory protection changes (from one syscall per page
to a fixed upper bound of five). i/o errors (and partial success) are
retried where possible, reducing unnecessary fatal errors and improving
error messages.

- simplifies the guard page implementation, preventing premature bail:meme.

- updates the vendored libwhereami dependency, fixing the OpenBSD build.

Contributions:

Joe Bryan (79):
u3: restages #6062 for release
u3: misc cleanup of snapshot system
u3: coalesces memory protection when loading a snapshot
u3: refactors _ce_loom_blit_north(), replacing loop with a single read
u3: coalesces memory protection when saving a snapshot
u3: implements full demand paging for the home-road heap
u3: adds printf ("live: mapped ...") for demand paging
vere: adds --no-demand argument to disable demand paging
vere: supports --no-demand for all relevant subcommands
Merge tag 'urbit-v1.13' into jb/restage/pread-all
Merge tag 'urbit-v1.13' into jb/restage/demand
Merge branch 'jb/restage/pread-all' into jb/release/vere
u3: updates memory protections and page metadata when moving guard page
Merge branch 'jb/release/vere' into jb/restage/demand
u3: document snapshot system invariants
u3: cleanup comments about snapshot system limitations
u3: adds new, batch page tracking implementations + tests
u3: enforce snapshot invariants on segment truncation
u3: switch page size constants to macros to avoid VLAs
Merge pull request #6108 from urbit/philip/willoc-comment
Merge pull request #6121 from urbit/ja/openbsd-7.2-fix
Merge pull request #6125 from urbit/jb/ward-safely
Merge pull request #6126 from urbit/jb/snapshot-invariants
Merge branch 'jb/restage/demand' into jb/release/vere
u3: protect guard page if necessary after remapping loom
u3: protect guard page if necessary after reprotecting loom
Merge pull request #6127 from urbit/jb/guard-always-again
u3: updates guard-page assertion to account for variable loom sizes
test: re-disables ames decryption test to avoid bail:evil in ci
test: re-disables failing grq test
test: renames nock-tests to meme-tests
u3: removes u3e_foul()
u3: stop using u3P.dir_c outside of events.c
u3: move directory creation out of events.c
u3: automatically place the guard page in u3e_live()
u3: simplifies road stack adjustment after snapshot load
u3: adds snapshot facade to manage.c
u3: simplifies u3e_live() signature
u3: fixes compile-time conditionals around guard page impl
u3: refactors loom protect-south page-length calculation
u3: factors out individual page protections
u3: reimplements guard page w/out posts
u3: adds facade for u3e_fault(), refactors error handling
u3: refactors u3e_save() facade and watermark calculations
u3: adds facade for u3e_ward(), refactors watermarks
u3: adds comments to road pave implementations
Merge pull request #6152 from urbit/jb/events-interface
vere: bumps version
u3: ports minor noun/ cleanup
u3: factors out |mass-style memory measurement
u3: ports new arvo invocations for boot and +poke
u3: ports meld measurement printfs
vere: factors out sift/etch functions event serialization
vere: ports synchronous lmdb iterator
vere: ports synchronous event log iterator
vere: ports u3_disk_read_list()
mars: adds replay implementation
vere: updates command docs
vere: adds dedicated replay command: play
vere: adds full replay option to play subcommand
vere: adds --auto-meld for play subcommand
Merge pull request #6159 from urbit/jb/mars/play
mars: fix bail:meme retry counter
vere: bumps version
ci: only save version string and setup gcp if we're uploading binaries
ci: updates gcp integration for setting release version strings
ci: updates gcp integration for uploading tarballs
vere: handle ctrl-z like ctrl-c in play command
mars: cleanup event log on exit
u3: make the fault handler robust against initialization reorder
term: skip prompt refresh on ^c under -t
Merge pull request #6167 from joemfb/ctrl-z-replay
vere: make full replay (play -f) command bypass corruption
vere: fix "queu" command argument parsing
Merge pull request #6168 from urbit/jb/replay-tools
vere: bumps version
vere: fix queu short-args parsing
vere: refactors http server init for better error messages
vere: bumps version

James Acklin (1):
vere: updates whereami dep to support OpenBSD

Philip Monk (1):
u3: add comment explaining willoc logic

—
~master-morzod
https://tlon.io