A package may include an app or several apps or any other set of files. These are stored in a desk, which also contains a bunch of other files needed to run the app, including the standard library. # Running 3rd party software Prior to the mechanical questions (eg "how to discover" or "how to download") is the question of where to run 3rd party packages relative to each other and base software. There's two basic schools of thought here: ## Containerizing "Every package is a new container, with no mistakes in it... yet" Containerization is the idea that each package should specify exactly its dependencies and always run against those in an environment it has full control over. This means from the perspective of the base layer, the container is a child in its own world that interacts through defined interfaces -- often networking interfaces. The biggest advantage this gives is flexibility in your dependencies. Since you're the only one using them, you don't have to support multiple versions of them -- you can even modify them if you wish. You can make the whole system work exactly how you want it to, and a whole class of error conditions disappears. Of course, there are limits. You can specify the exact version of your database client, but unless your database is only for your own use, you won't bundle the database server with it. This means you have to support version mismatches between the client and the server, in the usual ways (define a protocol, avoid breaking backward compatibility, etc). This leads us to an important point: stateful dependencies *must* be coordinated. This is irreducible complexity. Note by "stateful dependencies" I mean only those which other apps care about -- you can of course bundle a dependency which maintains some state as long as nobody else needs to talk to it. This means if your app depends on someone else's app, they need to be able to talk to each other. The standard ways of making sure two pieces of code can interact are: - Version negotiation: have an agreed-upon scheme for determining if client and server are compatible. Semver is one such scheme. Another is to ad-hoc specify a range of versions you support on either side, and proceed if either claims to support the other. - Defensive programming: related to duck typing, start using something and be ready for any part of it to fail unexpectedly. Convenient, but allows complexity creep. - Static typing: define the protocol using static types that both sides understand. If the compile step succeeds, then both sides should be able to handle any valid value. Note this can include explicit versions, especially a tagged union of previous versions of the protocol which we know how to forward-port. Containerizing a package makes it harder to interact with the rest of the system. This is by design, but it's a tradeoff. The disadvantage is that each package ends up in its own world, which limits their power considerably. One of the biggest disadvantages of SaaS model for users is their apps can't interact with each other except through occasional specially-defined interfaces. One of the biggest advantages of containers is the ability to manage big balls of mud. On Earth, this is super important. For example, python devs virtually always use at least virtualenv because their dependencies are uniquely poorly managed. However, on Mars it's super important that we not create big balls of mud in the first place. So if containers only provide that benefit, we shouldn't use them. However, containers have other benefits, and we must avoid the fallacy of "X was chosen because Y, and now Y is false so X must be the wrong choice". It's possible to be good for two reasons. In Urbit, the most prominent container-style solution is "theory of the moon". There are many variations on it, ranging from "just run a moon in a whole other process for each 3rd party package" to "run each desk as a container inside a single arvo instance". I'll describe one version here; though much variation is possible, the complexity tradeoff is almost identical. Your base arvo is a hypervisor, which contains many desks. Each desk is defined from a pill and a set of files (including all static dependencies) and then evolves on its own. Since it has its own compiler, its types are incommensurable with those of the host or any other desk -- if we need typed messages, we must clam them. Each desk has its own kernel and vanes, which communicate with the host's vanes through a stable and version-controlled interface, similar to the one used to communicate with the runtime. Each desk has its own Ames, and those Ames's may have signficant differences -- eg they may use different congestion control algorithms. Because their states are different, each desk's Ames must be assigned a unique identity -- a moon. To communicate between each other, desks send each other Ames messages -- after learning about each other from the planet, their host. Version negotiation is identical to that between different planets. For efficiency, the host Ames recognizes when a message can be forwarded directly to another colocated desk/moon instead of being serialized, packetized, encrypted, and going out on the network. This is quite clean and allows radically different environments for each package. For example, they needn't use the same programming language. If the runtime supports it (say, with a jet), you could even run a non-nock deterministic desk as long as it produces nouns. ## Blending The second school of thought is that containerization is "giving up" on building a well-designed system, and that the convenience isn't worth the restrictions it imposes, particularly since the convenience is primarily a license to created complicated code structures. This school says that your personal cloud computer should be *a* computer, not a balkanized collection of independent information appliances. This has largely failed on Earth, but maybe we should have taken that as a sign that Earth is doomed instead of building (fallout shelters)[https://www.youtube.com/watch?v=ybSzoLCCX-Y]. This means you have one language, one kernel, and one set of vanes. Packages are installed from other desks onto your home desk, and simply merged together. If the result compiles, running apps upgrade successfully, and all tests pass, then the merge is committed and you start running with that. This is much cleaner if it works, since your entire system is compiled together. You can take advantage of a static type system to help protect even your interface with stateful dependencies. Of course, on Linux this you don't get much advantage from this since it doesn't have a powerful native type system. Notes: - "Merging" doesn't imply a 3rd party package can overwrite base files. It's easy to imagine a merge strategy that says "merge %foo to %home, failing if that would overwrite any file in %base". - Merging allows you to use the natural DAG structure of commits and derive versions from this. In general, the version of %foo (which could be the base or a 3rd party package) that exists in %home is the mergebase of the two desks. IOW it's the latest commit which is present in both ancestries. - If you really need an incompatible language/kernel/vanes, you can of course run it as a moon. - Another way to describe this is that we use the same mechanism for 3rd party packages as for the base distro. ## Comparison - Both systems allow you to bundle your own static dependencies. - Blending allows more atomic updates, since the entire system can be compiled together and updates rejected if something goes wrong. - Blending allows you to take some of the nice tools we have for static dependencies (eg static type system, unit tests, integration tests) and use them for stateful dependencies as well. If your actual `sur/foo.hoon` file doesn't work to compile `app/bar.hoon`, that's probably because %bar was written against a incompatible version of the %foo app. Containers would by defualt allow the upgrade and then things start failing at runtime. Blending will recognize the incompatibility and reject the update. Of course you can and should introduce explicit version negotiation for containers in this situation, but that doesn't invalidate the point. - In 2020 we've taken a hard turn toward "compile everything at once", especially with Ford Fusion, and the result has been much more stable and less complex. - Theory of the moon provides natural scalability, since you can spin out any moon into a separate process or machine without changing the semantics. Perhaps this isn't important for most ships, since there's little that individuals want to do that would benefit by this -- don't build for industrial use cases if it adds complexity -- but it's very natural. - Theory of the moon would require significant work in Clay, and we would need to define a large number of protocols up front that describe the structure of a desk-moon and how it interacts with the host. By comparison blending works today -- just `|sync %home ~ship %app`. I've spent most of the year preferring theory of the moon, but in the past couple months, encouraged by the success of Ford Fusion, I've come to believe blending is worth trying first. I believe it will be much preferable if it turns out to be practical, and it's straightforward to try it for a little while in an unofficial capacity. Since blending works now, I'll describe the rest as though that's what we're doing, but most of it applies regardless. # Getting 3rd party software While you can simply install a package if you know its name, you usually want to get a little bit of info about it before you do, and you may wish to browse for packages you didn't know about. The simplest version of this is simply a publish notebook with entries for each package, with a description and the name of the ship/desk to sync from. A real package manager would let you add a package that you host by providing the desk name and a description, and then if you specify another ship it will ask that ship if it has any available, and you can click a button to `|sync` that app into your %home desk. It should show all packages you've installed this way, their version (mergebase) and whether the latest update succeeded. You should be able to share an app to a group as well, so that if you're a part of that group you see that package available to install. This is probably the primary method of discovery. The package manager should also have an uninstall button. First, it needs to identify which files should be removed. I'm not sure exactly the correct formulation for this, but I think it's something like the inversion of `diff(mergebase(%base,%foo),%foo)`. This isn't ideal since if a file is added by two packages, this will either remove the file, or if the file was modified by one of the packages, it may cause a conflict. Another option is `diff(merge(all-other-desks),%foo)` but this is complex in the general case. We can eliminate most of the troublesome cases by specifying which files "belong" to a particular desk, which is probably a good idea anyway -- we don't want packages to clobber other packages' files. There may also be space for "weak" files, where we include a file if it doesn't exist but we don't own it. This would be appropriate for an app's "header files". The second thing needed to uninstall an app is to suspend or delete the app. To suspend an app, call its +on-save to produce a vase of its state, then throw away the app code. Then we have three choices, any of which could be useful: - Keep the state, maintain subscriptions, and queue up cards sent to the app. This can be resuscitated by loading the state in +on-load. - Keep the state, close all subscriptions, and nack all cards sent to the app. This can be resuscitated by loading the state, except we also need to specify that subscriptions were closed. This is very similar to the case of importing data after a personal breach. - Throw away the state, close all subscriptions, and act as though the app never existed. This is potentially problematic since other apps on the network may not realize your app was removed. Many apps won't care about this, but the general solution would be along the lines of "notify all relevant apps when an app is nuked", similar to how a personal breach works. When an update comes in, it's possible it'll fail to compile because of an app you don't care about too much. It should be easy to mark the app as noncritical, suspending it until it compiles again. # Trying now If you're an app developer, consider putting it on a public desk somewhere (`|public %desk`) and telling people about it. If you're a user, try installing apps you hear about. For example, `|sync %home ~middev %kids` will install ~littel-wolfur's %srrs app. Because this flow hasn't been used extensively, I recommend trying this on a moon after it's finished downloading its first update from its parent.