[ANNOUNCE] urbit-os-v2.123

219 views
Skip to first unread message

phi...@tlon.io

unread,
Jun 1, 2022, 8:51:19 PM6/1/22
to d...@urbit.org
urbit-os-v2.123

This is a release of the base Urbit OS, which takes us to Lull kelvin
version 329 and Zuse kelvin version 418. Highlights include:

- There is a new vane called Khan, for running threads from either
inside Urbit or from Unix
- Clay
- Clay allows you to tombstone old files, reclaiming memory
- Clay's namespace has been further implemented, so that you can now
mount, build files on, and run code from past revisions and foreign
desks.
- Ford's build cache has been overhauled to share builds between desks
- Clay no longer stores %delta blobs. Its object store is now pages.
- Clay no longer has an ankh cache.
- Comets are now able to communicate with each other
- There is a new system for recording commit messages called "story"
- Aqua has been revived and improved
- The trie data structure `axal` has been extended with several new
operations and renamed to use the `+of` core

Base hash will be one of:

0v1c.332cn.4a748.f54t9.gs4nj.4fte8.36jjg.09gq8.fh113.akpik.us8e5
0v1v.p3a22.iv754.lt3ot.hpbee.0elcg.orl5e.fakce.r1pb5.sg0kt.46vqk

Other changes:

- New command +vat displays information about an individual desk
- The roller now recognizes oversized batches and reduces their size so
they will be accepted by common ethereum nodes
- `-naive-csv` has been added to summarize information about L2 batches
on chain in a CSV format
- Spider's child threads have been revived
- The longstanding "dangling-bone" issue is fixed
- A longstanding memory leak in the compiler has been fixed, and
structural sharing has been improved
- A bug has been fixed in `+uno:by` when called with an empty map
- `;<` now uses a hygienic expansion, so that it doesn't shadow `$` when
looping
- There is now a `$+` rune for adding type "names" to improve printing
types
- Ring signatures in lib/ring have been revived
- Some fixes to the LSP support
- Getting up to date with azimuth is now significantly faster
- Various other fixes and improvements

Breaking changes:

There are very few actually breaking changes in this release. We
sampled about 30 apps from the network, and only two required any
changes beyond updating the sys.kelvin.

- This reduces the Lull kelvin to 329 and the Zuse kelvin to 418. This
means every app needs to release a new version that at a minimum
changes sys.kelvin to reflect the new kelvin. Any changes which are
made to deal with other breaking changes should be made in the same
commit if possible.

- Prior to this release, you could assume that any file referred to by
Clay was available to be read. Now, it may have been tombstoned,
unless it's at the head of the desk.

- Clay's blob type has been removed, so anything that refers to it
should use the page instead. This is mostly internal to Clay, so apps
are unlikely to be affected.

- The trie (`axal`) operations have changed, so any apps which used it
should be modified to reflect the new interface.

- The type of the `dome` returned from a Clay `%cv` read or scry has
been altered to remove the ankh cache.

Further details:

Khan is a new vane which has two purposes:

- To provide a request/response interface to running threads
- To allow the runtime to trigger running a thread and receiving the
response, giving a cleaner option for controlling your ship from Unix.

Prior to this release, running a thread required interacting directly
with the spider app, through a complex sequence of poke, watch,
poke-ack, watch-ack, fact, and kick. This replaces that with a simple
request/response pattern. To run a thread, use a move like:

[%k %fard %base %hi %noun !>(`~zod)]

Once it completes, you will receive a move like:

[%k %arow %& %noun !>("hi ~zod successful")]

(Note the `-hi` thread happens to produce a tape, but a thread can
produce anything).

A later release will add runtime support so that these can be triggered
from Unix as well.

Clay is primarily made up of files and commits. Each filepath
in a commit has a pointer to a file. A "tombstone" is when that file
does not exist in our object store, so the pointer has no target -- it's
just a marker saying "the data that was here is gone". This saves
the memory that the file would have taken.

Clay has no way to remove commits or any other metadata, but it does
have a way to remove files. However, since files may be referred to by
multiple commits, this must be handled with care.

The paradigm is that each commit has a set of configurable rules
("norms") determining whether to preserve the files it points to, or
whether they're candidates for tombstoning.

Note that these norms governs future actions, but you can't infer from
the norms whether or not you actually have a blob; you must check that
directly. If the norms says a file can be deleted, it may not actually
be deleted if the norms for another commit require it to be preserved.
Similarly, if a file has already been tombstoned, setting the norm to
require it to be preserved can't resurrect it.

In addition to the per-commit norms, each desk has a "default norm",
which applies to any commits which have not been given specific norms,
including new commits.

Finally, we always guarantee that the head of each local desk has no
tombstones. Any operation which would violate this will crash.

The norms are specified as a trie of paths to loobeans, where "yes"
means to keep the file. These are interpeted as a set of overlapping
rules. For example:

=| =norm:clay
=. norm (~(put of norm) / %&)
=. norm (~(put of norm) /app/js %|)
=. norm (~(put of norm) /app/js/special %&)
norm

This means "keep everything under /, except don't keep anything under
/app/js, but if it's in /app/js/special, then keep it anyway". The most
specific rule applies.

This gives the following interface changes:

If you try to scry for a tombstoned file with %x, that will crash, just
as if the file didn't exist or was on a foreign desk that we don't have
cached.

%z works correctly on commits with tombstoned files, since the tombstone
contains the hash.

If you %warp for a tombstoned file with %x, it will not respond, same as
if the file was on a foreign desk that it couldn't contact.

We recommend being conservative in tombstoning files, especially at
first. In particular, until everyone has received this release, ships
on the previous version won't be able to ingest tombstones from ships on
the new release. For this reason, we recommend don't yet recommend
tombstoning anything on galaxies, stars, or ships which distribute apps.

If you do want to tombstone files, these are the most common commands:

- `|norm ~ship %desk /path %|`. Modify the default norm for the that
ship/desk to include/exlude the given path. This is the primary way
we expect to tombstone files.

- `|pick` runs the current norms, tombstoning everything it can. This
does not happen automatically.

- `+norms`. Print default norms for all desks.

- `|tomb /=base/5/path/to/file`. Sets the minimum norms required to
remove that file. Add `, =dry &` to see what it's going to do before
it does. Note that it sets the norms for commits individually;
sometimes it's simpler to set the default norm for the entire desk.

These are additional fundamental commands:

- `|pass [%c %tomb %norm ~ship %desk (~(put de *norm:clay) / %|)`. Set
the default norm for the given desk. This example norm represents
"tombstone everything".

- `|pass [%c %tomb %worn ~ship %desk 0vtako *norm:clay]`. Set the norm
for a specific commit.

- `|pass [%c %tomb %seek ~ship %desk ud+5]`. Downloads the blobs
associated with a specific foreign desk revision. For use when you
have tombstoned parts of a foreign desk that you now want to re-fetch.

- For emergency use only: `|pass [%c %tomb %all ~]`. Tombstone
everything possible. Does not change norms.

This introduces the first version of an optional system for commit
messages in Clay. They are stored in a `.story` file on each desk,
which stores a map of commit hashes to commit messages.

These are the primary commands. All of them accept an optional `=desk
%desk` argument and some have other optional arguments.

- `|story-init` creates a blank story file

- `|story-write 'title' 'content'` writes the commit message to the
current desk for the current version

- `|story-remove ud+5` removes the commit message from the given
revision.

- `+story-read` prints a summary of the commits messages, in recency
order.

In case of a merge where two different commit messages are stored for
the same commit, both will be preserved.

We have a trie data structure called `axal`, which has been greatly
expanded in this release. It now contains the following operations in
the +of core:

- `get`: retrieve the item at a path
- `fit`: retrieve the item at the longest prefix match for a path
- `has`: check whether a path has an item
- `put`: put an item at a path
- `del`: delete the item at a path
- `lop`: delete all items under a path
- `dip`: descend to a path
- `tap`: serialize to a list
- `tar`: serialize to a map from path to items
- `gas`: put several items from a list

Contributions:
DaneBettis (2):
vere: add %nara %hela hints, add a hoon test module for hints
changes in response to code review

Joe Bryan (3):
hoon: updates %mcgl with an hygienic expansion
hoon: plug type leak in vase literals by properly burping %hint
hoon: preserve structural sharing by testing subject/product equality in +burp

John Franklin (2):
vat: add +vat, a single-desk version of +vats
show-vat: change output formatting to match +vats

Jōshin (59):
khan: vane skeleton
khan: basic +call, types
khan: %done task
khan: closer to starting threads
khan: wip fyrd
khan: wip threads v0
khan: cleanup
khan: accepts thread arguments with marks
khan: functionally done
khan: del in %kick
khan: cleanup
khan: strip inner faces from +bear
khan: PR feedback, wip tests
khan: +bear allows a full +beak
test: khan test cleanup
spider: blank lines
khan: add %fard/%arow for in-arvo threads
test: rewrite khan start-args unit test
test: wip full-run test
khan: docs, lift arg vase into (unit)
khan: parameterized types, nested implementation
khan: cleanup, namespace
test: full run, happy path
test: %khan full run, %fyrd
test: remaining %fard failures
test: refactor, strip out inner %fard call
test: %fyrd fail case
test: cleanup
test: cleanup
test: missing mark cases, further cleanup
test: tlv
khan: name $cast type
khan: move =/ down
arvo: introduces $crag
test: fix
khan: cleanup
khan: randomize tids
test: fix failure introduced by random tids
khan: alignment
khan: better wire production
khan: structural slap, not general slam
khan: initial state version is %0
khan: %fard/%arow use $cage, not $vase
khan: update docs
test: 1162
arvo: remove redundant *
khan: use +en-beam / +de-beam, wide form
khan: separate arms by blank comments
khan: handle multiple %fards in one event
khan: don't reset tic
test: confirm different tids on same eny
arvo: $card is not a cask
arvo: remove $crag, promote $page
khan: docs
khan: saner tic rollover, remove our
hoon: fix comment whitespace
lull, zuse: cool
khan: minor whitespace cleanup
arvo: $card is (cask)

Liam Fitzgerald (5):
eyre: include mark of fact in channel json
eyre: address mark review, update pill
zuse: add ++ns:dejs:format
clay: break out ford memory usage in more detail
clay: be careful with subject in ford results

Philip Monk (75):
clay: Add support for representing tombstones
clay: refactor +wake
ted: remove unused -azimuth-tracker
helm: add poke to helm to allow self-breaching moons
clay: use XX for things that really need to change
clay: align comments
clay: add basic %tomb
clay: prefer life
clay: add upgrade to %11
hoon: fix +uno:by when map is null
clay: require non-dead files at head of remote
clay: extend %tomb to delete all unneeded blobs
clay: style
clay: bump kelvin, refactor
clay: unify all trie implementations
clay: address review comments
clay: add norm state, remove mark from %delta blobs
clay: add gc routines
clay: make +pick fast
clay: trying to find %dead bug
clay: address review comments
arvo: revert axal to use $
clay: add norm to +pick cache
clay: allow to set norm for past revision
clay: add %tako scry
clay; fix overly cautios tombstoning
clay: add |tomb
clay: add |norm
clay: don't download unneeded blobs
clay: seek wip
clay: convert all blobs to pages
clay: use old requests for now when backfilling
clay: address review comments
clay: wip
arvo: rename +de to +of
clay: cleanup
clay: remove ankh
clay: enable mounting revisions of desks
clay: collapse version 12 into 11
clay: new ford cache
clay: remove +build-leak
clay: remove -k arms
clay: rename ford-cache->flow
clay: add sprig cache
clay: rename poor->mist
clay: reduce printfs
clay: add dependencies to stack on sprig hit
clay: address review comments
clay: add %sweep scry to verify refcounts
clay: comment
clay: add |pick
clay: fix |pick
hoon: in molds, unify result with input
clay: tweak |mass output
clay: validate files in separate roads
clay: only maintain mime cache if mounted
kiln: use %only-that for vats merges
helm: emit moon-breach in new event
eth-watcher: fix upgrade
tests: fix various
various: make all files build correctly
arvo: reduce priority of lifecycle printfs
clay: fix stack while building directories
clay: re-export page to not break apps
clay: tweaks from walkthrough
clay: fixes
clay: don't require head of foreign desks
clay: fetch entire latest foreign commit
clay: progress download when receiving tombstone
clay: make trim clear ford cache
clay: add tomb scry
clay: don't add lobe to wire to avoid flow leak
ames: fix comet communication with breached ship
clay: add logging commands
clay: update tests

Ted Blackman (2):
ames: |ames-prod to reset congestion control
ames: pop the next packet, not the first packet again

Yaseen (43):
clay: add new `%s` scry which gives the `tako` for a given beak
story: add generator `+story-list`
story: add generator `|story-add`
story: add generator `|story-remove`
story: rename generator `|story-add` to `|story-set`
story: let |story-set take a case and make it optional
story: add generator |story-init
story: amend |story-set to use the tako from appropriate desk
clay: implement `%base-tako` scry under the `%s` care
story: make `|story-init` use new types and make overwriting optional
story: update `|story-set` to use new types
story: make `|story-list` use new types
story: add generator `+story-log`
story: misc changes
story: update `|story-remove` to use new types
story: misc changes
story: make `+story-log` use arms from lib
story: cleanup
story: fix to-text arms
story: misc changes
story: generators improvements
story: fix |story-remove and allow it to remove a specific prose
story: make desk an optional param in story-list
story: shorten recur statement in +story-log
story: |story-set: better error message on missing desk
story: add generator help text
clay: add scry for case existence check
story: make |story-{set,remove} use $cash instead of $case
story: check for storyfile existence and use case from beak where appl.
story: fix generator bugs
story: generators: add check for invalid case
story: generators: fix case scry
story: generators: add desk existence everywhere
story: fix scry path in |story-set
story: generators: fix case check
story: fix output formatting in `+story-log`
clay: fix bug in fuse
story: rename generators
review: align end comments
review: symlink base-dev files into arvo
story: allow `|story-write` to also accept a lone title
story: enforce 80 cols, fix formatting, update comments
story: marginal output formatting improvement to `+story-list`

drbeefsupreme (37):
naive: l2 csv thread initial commit: timestamps
naive: L2 csv thread refactor filters
squash! naive: L2 csv thread refactor filters
naive: L2 csv transaction receipts
naive: L2 tx csv gas price request
naive: L2 csv swap lists to maps
naive: l2 csv first data collation
naive: l2 csv add roller address to collated data
naive: l2 csv - names for address, hash, blocknum
squash! naive: l2 csv - names for address, hash, blocknum
squash! squash! naive: l2 csv - names for address, hash, blocknum
naive: l2 csv get naive effects
naive: l2 csv all relevant data in one map
naive: l2 csv flatten data into list of tx
naive: l2 csv export csv file
naive: l2 csv refactoring for readability
squash! naive: l2 csv refactoring for readability
naive: l2 csv add ^sein:title field
naive: l2 csv change gasUsed to effectiveGasPrice
naive: l2 csv add @da to saved file
naive: l2 csv factor out %eth-get-timestamps
naive: l2 csv factor out %eth-get-tx-receipts
naive: l2 csv more informative printfs & comments
squash! naive: l2 csv more informative printfs & comments
squash! squash! naive: l2 csv more informative printfs & comments
spider: fix child threads
squash! squash! squash! naive: l2 csv more informative printfs & comments
spider: mv %thread-done %thread-fail to base-dev
squash! spider: mv %thread-done %thread-fail to base-dev
naive: l2 csv style changes
naive: l2 csv error handling for child threads
naive: remove name collision for block
naive: l2 csv remove wip comment
naive: l2 csv correct tx-data initialization
naive: l2 csv remove space from line >80 char
naive: l2 csv more comment spacing
naive: l2 csv remove scag on logs for testing

fang (18):
hoon: hook up $+ for shorthand type rendering
spider: factor spider-arg type out into interface
ph: remove deprecated migration testing threads
pill: use %pill mark, not %boot-pill
aqua: cache results of booting ships from pill
aqua: use cache for fake ships only
aqua: fix most tests
azimuth: allow configuring of the refresh rate
aqua: support injecting arbitrary tasks
aqua: increase azimuth refresh rate for real ships
aqua: cache real ships as well
aqua: stub out azimuth snapshot in pills
aqua: remove todo list note
ph: clarify comment in moon test
aqua: support using real moons
aqua: disable cache loading for real ships for now
hoon: make manual shorthand type printing prettier
aqua: revert erroneously committed changes

lukechampine (5):
ames: don't enqueue blobs sent to unknown comets
ames: fix comet self-attestation handling
ames: add sendkeys-packet
tests: add message flow tests for comets
ames: add resend timer for attestation requests

pkova (2):
eth-watcher: implement optional to-block parameter
eth-watcher: remove obsolete zoom-margin handling

ryjm (3):
%home -> %base
arvo: language-server desk support
fix typo and beak

yosoyubik (26):
roller: fix wrong ship quota
roller: add support for forced transactions
roller-rpc: make force field optional
roller: keep force flag in sending transactions
roller: check for existence in resend nonce
roller: include signing address in sending txs
roller: bug fix in looking for sending diff
roller: add batch-trimming functionality
roller: make output less loud
roller: fix parsing nonces with dot separator
ames: fix dangling-bone error
ames: fix cached state in larval upgrade
ames: address review
ames: don't drop events with old wires
ames: fix emit of parsing old wire log
ames: remove sigwut
ames: use +trace to print logs
ames: remove unused ames-to-ames %plea logic
azimuth: revert azimuth change in 60cc44e
ames: hardcode rift for comets in state upgrade
azimuth: use %kick poke to restart subscriptions
eth-watcher: discard future pend-logs in watchdog
ames: fix 4-to-5 state migration
azimuth: restart %eth-watcher if not running
ames: start drainage timer if regressed from adult
ames: indent

~datnut-pollen (1):
naive: l2 csv apply ~wicdev's style suggestions

~nallux-dozryl (1):
Fix comment indent for abs:si

John Franklin

unread,
Jun 7, 2022, 1:34:50 AM6/7/22
to urbit-dev, phi...@tlon.io
I upgraded to 1.9 and got the OTA.

I have two planets and a comet tracking `~wanzod`:
```
> +vat %base
%base
  /sys/kelvin:      [%zuse 418]
  base hash:        0v1p.tn1a4.27ia9.8qdbl.bpog2.88qcc.29n8s.fcrh9.jibl9.oe756.i9173
  %cz hash:         0v1p.tn1a4.27ia9.8qdbl.bpog2.88qcc.29n8s.fcrh9.jibl9.oe756.i9173
  app status:       running
  force on:         ~
  force off:        ~
  publishing ship:  ~
  updates:          tracking
  source ship:      ~wanzod
  source desk:      %kids
  source aeon:      122
  pending updates:  ~
::
```

Another comet tracking `~binzod`:
```
> +vat %base
%base
  /sys/kelvin:      [%zuse 418]
  base hash:        0v1p.tn1a4.27ia9.8qdbl.bpog2.88qcc.29n8s.fcrh9.jibl9.oe756.i9173
  %cz hash:         0v1p.tn1a4.27ia9.8qdbl.bpog2.88qcc.29n8s.fcrh9.jibl9.oe756.i9173
  app status:       running
  force on:         ~
  force off:        ~
  publishing ship:  ~
  updates:          tracking
  source ship:      ~binzod
  source desk:      %kids
  source aeon:      122
  pending updates:  ~
::
```

And another comet tracking `~marzod`:
```
> +vat %base
%base
  /sys/kelvin:      [%zuse 418]
  base hash:        0v1p.tn1a4.27ia9.8qdbl.bpog2.88qcc.29n8s.fcrh9.jibl9.oe756.i9173
  %cz hash:         0v1p.tn1a4.27ia9.8qdbl.bpog2.88qcc.29n8s.fcrh9.jibl9.oe756.i9173
  app status:       running
  force on:         ~
  force off:        ~
  publishing ship:  ~
  updates:          tracking
  source ship:      ~marzod
  source desk:      %kids
  source aeon:      123
  pending updates:  ~
::
```

The hashes match each other, but don't match what's published above.
Also, the `source aeon` from `~marzod` is `123`, but I don't know if that's weird.

~rabsef-bicrym

unread,
Jul 10, 2022, 2:12:36 PM7/10/22
to urbit-dev, John Franklin, phi...@tlon.io
I have this same experience - is this a problem?

Jack Fox

unread,
Jul 12, 2022, 3:31:47 PM7/12/22
to urbit-dev, ~rabsef-bicrym, John Franklin, phi...@tlon.io
I lost the thread on why I posted this.

Could be I mucked around in some desks changing their base hash.

Philip Monk

unread,
Jul 12, 2022, 4:07:17 PM7/12/22
to Jack Fox, urbit-dev, ~rabsef-bicrym, John Franklin
Just realized my original response went directly to John:

The fact that they're at 418 is pretty conclusive that it worked out fine. There are several possible hashes because of a quirk in how the OTA logic used to work, and we actually fixed that in this update so that in future the hash should be much more consistent, but that only applies to subsequent updates.

Reply all
Reply to author
Forward
0 new messages