duiker & serval: bittorrent trackers on urbit

[Mark]

~paldev and I have spent the past couple months working on this. Last weekend we released this to a couple select communities, and today we want to announce it more broadly.

https://github.com/fang-/suite/blob/master/app/serval.hoon

Serval is an application that implements the BitTorrent HTTP tracker protocol, in order to facilitate peer discovery and track up/download statistics. Unique-per-@p announce URLs let it associate statistics with Urbit identities.

https://github.com/Fang-/suite/blob/master/app/duiker.hoon

Duiker is a CLI application for sharing and discovering BitTorrent magnet links, backed by Serval. To access it, simply run the following in dojo:

|link ~paldev %duiker

Guaranteed risk-free "installation"! Mind the input lag, and remember you can ctrl-x to switch out of it. To remove it from rotation, simply |unlink instead.

(If you're on hosting, you'll notice that this doesn't really work in web dojo. Fear not, I'm presently tasked with addressing that! See also previous discussion.)
https://groups.google.com/u/1/a/urbit.org/g/dev/c/ubp3UNxkQBc

Give it a shot! People have already been sharing some things.

I'll probably continue developing these things at some point, but want to let them exist as-is for a little bit. I don't know if you can call this a post-mortem per se, but here's some scattered thoughts:

- As a command line application, this is obviously way different from chat-cli. I spent a lot of time trying to figure out nice patterns for dealing with multi-step navigation and interactions, before deciding that a single-mode implementation is easier on both developer and user... for now.

- Having to work around the fact that other content might push the app's output offscreen is a bit weird, and certainly influences decisions like the above. There's a command (.) just for reprinting the file listing that was last shown. You know, in case your chats are so active you couldn't finish reading it. ...This ended up being less important after settling on the single-mode interacting pattern.

- The /lib/shoe %table command came out of this. It's a good first stab at "generic complex drawing operation" that shoe should provide more of, but in some ways it's still lacking. The idea that this is a generic table is great, but breaks down when the CLI app expects the user to make a selection from the table, or otherwise needs to be notified of interaction with it. It doesn't allow for styling table contents either, which would've been nice for the seeder column in duiker.

https://github.com/urbit/urbit/pull/3419

- Text rendering logic in general feels a little awkward. $styx is... fine, I guess, but the codeblocks it results in feel a bit inelegant. Not sure how to do better, besides parsing bbcode into styx nouns. (^:

https://github.com/Fang-/suite/blob/f7e1e714432aa0a5f94a414d4e972a4c7908f1ba/app/duiker.hoon#L476

- The idea behind running this on ~paldev and having people |link into it (a la server-side rendering) was simplification in a number of areas. Primarily, it lets users skip past whatever the current state of software distribution is, and just use the thing straight away. This is important! It got you to  But the input lag is a bit of a buzzkill, and software distribution made decent strides over the course of development.

- I should've thought more carefully about the properties of magnet links before relying on them this hard. They're not really usable in private tracker contexts. Right now, Serval is really just there to keep track of statistics. Magnet links will do peer discovery just fine with BitTorrent's DHT! A clear next step is to implement support for full-fledged torrent files and, uh, probably make those up- & downloadable through a Serval web endpoint? Once that is in, we can start thinking seriously about group-exclusive torrent collections.

- Besides supporting torrent files, these things could probably benefit from groups integration. Having a "public tracker" experience like this is nice to get the ball rolling, but I imagine you'd be reluctant to share your rarest bytes in such a space.

Happy to answer questions about this if there are any. More importantly, I'd be curious to hear any ideas you all might have for interaction patterns involving these tools. Those would be a big help for guiding further development in the right direction.

—
~palfun-foslup
https://urbit.org

[Matilde Park]

No on-peek yet. Would be cool to scry in surface across interfaces ...

On Oct 10, 2020, at 6:01 PM, Mark <ma...@tlon.io> wrote:

~paldev and I have spent the past couple months working on this. Last weekend we released this to a couple select communities, and today we want to announce it more broadly.

https://github.com/fang-/suite/blob/master/app/serval.hoon

Serval is an application that implements the BitTorrent HTTP tracker protocol, in order to facilitate peer discovery and track up/download statistics. Unique-per-@p announce URLs let it associate statistics with Urbit identities.

https://github.com/Fang-/suite/blob/master/app/duiker.hoon

Duiker is a CLI application for sharing and discovering BitTorrent magnet links, backed by Serval. To access it, simply run the following in dojo:

|link ~paldev %duiker

Guaranteed risk-free "installation"! Mind the input lag, and remember you can ctrl-x to switch out of it. To remove it from rotation, simply |unlink instead.

(If you're on hosting, you'll notice that this doesn't really work in web dojo. Fear not, I'm presently tasked with addressing that! See also previous discussion.)
https://groups.google.com/u/1/a/urbit.org/g/dev/c/ubp3UNxkQBc

Give it a shot! People have already been sharing some things.

<PastedGraphic-2.png>

[Mark]

For those who missed them: these things are back online on new-era ~paldev. |link ~paldev %duiker as usual.

They have been updated to support real .torrent files. Magnet links remain supported as well.
I recommend using duiker from the web terminal: selecting a torrent file will give you a browser-native download prompt!

Among the changes required to make this happen is a library for parsing multipart/form-data http requests, to accept uploaded files. Might be useful for others looking to support file uploads in some way.

Full diff for this release:
https://github.com/Fang-/suite/compare/310ec71...bed5d97

—
~palfun-foslup
https://urbit.org