How to contribute?

[Michael Frister]

Hi,

I was pretty surprised some days ago when I discovered bazil. A while ago, after being frustrated with standard network filesystems and WebDAV performance over wide-area networks, I had pretty much the same idea of a filesystem that syncs metadata and loads file contents on demand.

I began implementing such a filesystem in Python, but soon switched to Go due to performance issues. I actually used bazil.org/fuse as FUSE package, and used SQLite as database (which obviously is not as fast as Bolt). I didn't get as far as implementing and using CAS, but the filesystem could store data in local files with metadata in the database.

I focused on the local part first to see how reliable a FUSE filesystem could be on OS X (that's what I'm using as my primary machine). I found two issues with bazil.org/fuse [1,2], but also got git to produce an error that probably originates in osxfuse. I'll have to look further into this, but I guess osxfuse works well enough to attempt to write a filesystem on top of it.

Long introduction, here's the main part: I'd love to help with implementing bazil. I think bazil is far more likely to succeed than my own filesystem, especially as mine was my first non-trivial Go application. I'm aware that I still have to learn Go better, but I don't mind taking a few rounds in a pull request (I guess that would happen for larger changes anyway).

How can I help?

Cheers,

Michael

[1] https://github.com/bazillion/fuse/issues/11

[2] https://github.com/bazillion/fuse/pull/12

[f...@fortes.com]

I also found Bazil a recently and wish this were available today. It solves exactly the issues I was having (been debating using git-annex & git-annex-assistant for this scenario).

Is development still active? How can folks help? Can I contribute funds?

[Tommi Virtanen]

Thank you for your interest. I've been distracted with other things for a little bit too long, and will be for a few weeks more, but let's try to get this thing back on tracks.

These are the current biggest roadblocks, and experimenting with solutions to them has been consuming the hours that would have gone into Bazil proper. These are mostly building blocks that are missing or inadequate, and thus Bazil itself has not received as much attention yet.

This list is a bit off-the-cuff, so I may be forgetting something major. Apologies for being scatterbrained right now. I've been quiet because I haven't been able to formulate this more coherently, but I realize too much silence is not a good thing, so let's try this for now. Each one of these topics alone is worth a long blog post and/or 1 hour talk, and I've been trying to avoid going too deep into any single one of them, hoping that suitable libraries show up from elsewhere -- that largely hasn't happened, yet.

ROADBLOCKS

1. A good bidirectional protocol / messaging system that will survive in the greater Internet

(Protocol, not a message queue. And please do not suggest 0mq.)

Not a request-response RPC mechanism, but one where peers can stream messages at each other freely. See my birpc experiment for inspiration, though JSON will absolutely not be good enough, and constructing a more complex dialogue of RPC calls gets frustrating, with continuation tokens or such.. https://github.com/tv42/birpc/

Low overhead per message. Crypto-safe; we'll know the peers nacl/box public key. Can be TLS, but cannot force user to understand CAs; must bootstrap from the before-mentioned known public key. Must avoid head-of-line blocking, and must do some sort of backpressure to control flow; we need to multiplex large data transfers with interactive requests without using too many TCP/IP connections.

I've been looking at layering on top of HTTP, to have the URL path differentiate between API endpoints, handle versioning, etc. Add something to prove TLS key is owned by nacl/box key, both ways. But REST really doesn't work for complex protocols. I would also like to avoid the XOR cost of WebSocket, so I've been looking at Upgrade headers and essentially using HTTP for handshake only. And WebSocket explicitly doesn't do anything about head-of-line blocking, so I'd end up treating it as a stream transport only anyway, so what's the point.

But HTTP Upgrade: will fail for anyone behind a strict proxy, and just devolves into a TCP/TLS protocol from scratch: it won't fulfill the "use URL path to differentiate" goal from earlier, as there's no good way to go *back* from the ugprade to speaking HTTP -- so it really doesn't tie in with HTTP well at all (even if proxy switched to pure-TCP passing when it saw the Upgrade:, now it won't know when we're back in HTTP land.. imagine trying to put that behind a load balancer). Maybe we don't care about proxies? Maybe the HTTP thing is just a distraction. It is fashionable..

I have a couple of half-tolerable from-scratch protocols (whether to be run inside HTTPS Upgrade:, or WebSocket, or not), but nothing good enough. One source of inspiration has been IX: http://lsub.org/ls/export/creepy.pdf

I'd really like to avoid this work, but I don't see a real contender out there. Right now I have a sketch of a protocol with "streams" in both directions, and each "stream" is a conversation that's semantically a superset of an RPC call. Whether that piggybacks on top of HTTP+Upgrade or not is still open.

Anyone really familiar with SPDY? Could it behave like a non-retarded HTTPS+WebSocket for me? Is it too tied to HTTP semantics? Are the Go libraries good enough already? It's all been too much in flux for me to keep track of.

Is the XOR cost of WebSocket even relevant if it sits inside TLS anyway? Each byte gets touched for the crypto anyway.

Anyone familiar with bootstrapping TLS into two-way trust *without* the CA dance? I wrote a proof-of-concept thing that handshakes at the beginning inside an anything-goes TLS connection, but haven't audited the it much against MITM and e.g. how to recover from accidental key disclosure. Once again, peers know each others public keys (nacl/box not TLS).

Long term, I'd love to have the large data transfers go over CurveCP+LEDBAT or something like that; yield to normal (TCP-like) traffic.

The Go code I see out there (https://github.com/inconshreveable/muxado https://github.com/alecthomas/multiplex https://github.com/docker/spdystream and so on) has so far not convinced me.

Once this is solidly in place, plugging in the file synchronization logic is much easier.

2. On top of the above, "endpoints" would basically build a little protocol within their allotted stream, e.g. a file tree sync looks like (using mscgen):

msc {

  // A pulls from B

  a, b;

  a->b [label="list(dirPath)"];

  a<-b [label="(basename, vPair, {file: manifest; dir: -})"];

  ... [label="<above repeats>"];

  a<-b [label="endOfDir"];

}

Except it might actually have multiple of *these* little dialogs multiplexed & interleaved inside one "synchronization". (The originating side decides which subtrees to recurse into, based on the vPair version vectors it sees in the parent directory listing. This proceeds pretty much concurrently across the whole tree. Yet these are all part of a single sync event, so the bookkeeping costs are shared across them all.)

So whereas with RPC you have method+request data -> response data or error, with this you get a little dialog for every "RPC call", all multiplexed over a single connection. I'm thinking this layer cares about serialization formats, where as 1. above is just shuffling streams around. This probably needs some sort of library/framework help, akin to net/rpc.

3. Help OSXFUSE release a new stable version, with a newer version of the FUSE protocol.

I want to migrate to the newer protocol on Linux, for performance, but I'm wary of splitting the codebase into multiple variants.

4. Semi-unrelated but taking up time, bazil.org/fuse Serve/Mount/fs.Init API needs to be replaced with something cleaner. Think about both Linux and OS X, and how their behavior is completely different. Don't take away too much power from pure protocol translators that don't want to use bazil.org/fuse/fs. Worry about InitResponse.MaxWrite ultimately being a question of bazil.org/fuse, not the filesystem, etc.

5. bazil.org/fuse on linux just isn't fast enough. CPU profiles are pretty evenly smeared across I recall 3 major groups, no single easy bottleneck. Use the newer protocol and vmsplice? Needs re-work of the bazil.org/fuse ReadRequest level; one []byte will be backing multiple incoming requests.

To avoid gc churn, that large []byte should be in a sync.Pool. That means it needs to stay alive long enough for all of the handlers to complete, before sync.Pool.Put; so there needs to be e.g. a refcount on the larger []byte, and then sync.Pool.Put when it drops to zero.

Also, current handlers may hold on to e.g. WriteRequest.Data even after returning; that's equivalent to C's use-after-free bug, and nasty things will happen. (Putting the sync.Pool.Put behind a const conditional will help in debugging, but that just means going back to a whole lot of gc churn.)

6. all the TODO notes littered over bazil.org/bazil source

7. all the TODO notes littered over bazil.org/fuse source

1000. Something better than protocol buffers. Fast access to fields of a message inside a read-only []byte (mmapped memory). Capnproto (as used from Go, with the current lib) is promising until the moment you actually try to access the data, and then it's even slower than gogoprotobuf. And the APIs are all semi-hideous too.

I'd be willing to trade off some of the flexibility guarantees of the protobuf design. I'm halfway-tempted to just unsafe.Pointer a struct on top of the mmapped []byte.

General things you can study, to contribute:

- Go. Duh.

- Tra, the file syncer; especially the algorithm

- FUSE and bazil.org/fuse, on Linux and OS X

I'll write up more of this and/or do talks/hangouts or something to spread the word more; I just need to deal with a couple of other things before I'm ready for that. Please ask good questions, they really help guide me into explaining things in a way that actually communicates.

[Dobrosław Żybort]

Maybe you can post some of this questions (like searching for specific package) on golang-nuts?

About serialization there is nice benchmark: https://github.com/alecthomas/go_serialization_benchmarks

Thank you for your hard work.

[Michael Frister]

Am Mittwoch, 16. Juli 2014 02:00:04 UTC+2 schrieb Tommi Virtanen:

> Thank you for your interest. I've been distracted with other things for a little bit too long, and will be for a few weeks more, but let's try to get this thing back on tracks.

Thanks your detailed description of the problems, that was the response I was originally hoping for. But then I was also distracted with other things, so I didn't ask for more.

> [...]

> 1. A good bidirectional protocol / messaging system that will survive in the greater Internet
>

> [..]

> But HTTP Upgrade: will fail for anyone behind a strict proxy, and just devolves into a TCP/TLS protocol from scratch: it won't fulfill the "use URL path to differentiate" goal from earlier, as there's no good way to go *back* from the ugprade to speaking HTTP -- so it really doesn't tie in with HTTP well at all (even if proxy switched to pure-TCP passing when it saw the Upgrade:, now it won't know when we're back in HTTP land.. imagine trying to put that behind a load balancer). Maybe we don't care about proxies? Maybe the HTTP thing is just a distraction. It is fashionable..
>

> [...]

>
> Anyone familiar with bootstrapping TLS into two-way trust *without* the CA dance? I wrote a proof-of-concept thing that handshakes at the beginning inside an anything-goes TLS connection, but haven't audited the it much against MITM and e.g. how to recover from accidental key disclosure. Once again, peers know each others public keys (nacl/box not TLS).

Using SSL seems like a good idea to me as building encrypted protocols is a hard thing to do (refferring to SSL and CurveCP I guess not building your own was your plan anyway).
I think the same about the authentication mechanism. Doing this within unauthenticated SSL seems to be easy to get wrong (looking at all the problems SSL had), so I suggest trying to rely on SSL for authentication.

If you absolutely don't want to distribute SSL certs together with NaCl public keys, one way to bootstrap the connection would be to bind a device's SSL cert to its NaCl key before starting the SSL connection.

One could do a plaintext request to the other party (e.g. via HTTP) and fetch a the hash of the public key for a (probably automatically generated) SSL cert, with the hash signed with the NaCl private key. During SSL handshake, one could then compare the received certificate's public key hash with the previously received hash. One could obviously cache these hashes for faster connection setup. Client and server (in the sense of who opens the connection) could exchange their cert hashes in a single HTTP request and response flow, so both sides could then authenticate each other for the SSL connection via certificates.

Before thinking too much about proxies when using SSL, one would have to think about how to authenticate the other end. When doing the authentication I suggested above, a proxy would not be able to look into the SSL connection (the hash wouldn't match), so the protocol used within that SSL tunnel wouldn't matter.

I think not using any authentication (or forcing a user to disable it) to increase compatibility with corporate proxies is not an option. Thus, we'd have to allow a user to explicitly allow the proxy (and no one else) to intercept the SSL connection. But then, we'd be doing the CA dance.

As far as I know, more and more applications are doing certificate pinning now, which also excludes proxies from looking into SSL traffic thus making the connection opaque for the proxies. Apple e.g. uses a custom binary protocol for their device push service that runs within an SSL connection to a custom port with a fallback to port 443. As far as I remember from reversing their protocol (I wrote a proxy for intercepting it [1]), they use some sort of certificate pinning. Google's Cloud Messaging uses a similar approach (fallback to 443) and also uses certificate pinning [2]. Google Chrome is a notable exception here, which allows user-installed CAs to override certificate pinning [3].

I think using a properly authenticated SSL connection and breaking usage for anyone who can't use non-intercepted SSl-connections is ok. But then, I'm not working for a big company that's proxying SSL traffic and hope I never will.

> [...]

> I've been looking at layering on top of HTTP, to have the URL path differentiate between API endpoints, handle versioning, etc.

> [...] constructing a more complex dialogue of RPC calls gets frustrating, with continuation tokens [...]

While I like HTTP, I'm not sure it's a good idea to try to build a stateful protocol on top of a stateless one when we don't have to. Instead, we could multiplex multiple stateful, bi-directional streams via a single SSL connection as you suggested.

If we can find an existing multiplexing protocol (preferrably with a Go implementation) that works well enough, I think worrying about a better protocol might be something we could do later. An interface providing multiple streams might be simple enough to encapsulate, so we can easily replace it later and get the RPC and file synchronization protocols implemented faster. Worrying about backwards compatibility is probably also a thing for much later. Do you agree to such an approach in general?

> [...]

I wanted to comment on more, but I ended up writing mostly about SSL encryption. I hope I didn't express my opinions too strongly for not having contributed any code to bazil itself. Just tell me what you like and don't like :)

> I'll write up more of this and/or do talks/hangouts or something to spread the word more; I just need to deal with a couple of other things before I'm ready for that. Please ask good questions, they really help guide me into explaining things in a way that actually communicates.

That sounds great, I think especially writing a proposal for the file synchronization protocol up would be a good starting point for further discussion on it. I'll probably ask more questions once I read a bit on Tra and the Creepy's IX protocol.

I'm glad you wrote all this up, no matter how well structured it was. This motivates me and reassures me that you're still interested in bazil.

If you want to do a hangout or talk somehow else, I'd be happy to.

Cheers,
Michael

[1] https://github.com/meeee/pushproxy
[2] http://books.google.de/books?ei=JcDGU5OXGYKq0QWkoIGACA&id=xpc6AwAAQBAJ&dq=google+cloud+messaging+certificate+pinning&ots=JQzZJL4nEW&q=pinned#v=snippet&q=pinned&f=false
[3] https://www.imperialviolet.org/2011/05/04/pinning.html

[Mark Abspoel]

I was quite pleased to learn about Bazil, since it's been a project I've been considering building myself for a while now. Are you guys still interested in this project?

If you absolutely don't want to distribute SSL certs together with NaCl public keys, one way to bootstrap the connection would be to bind a device's SSL cert to its NaCl key before starting the SSL connection.

One could do a plaintext request to the other party (e.g. via HTTP) and fetch a the hash of the public key for a (probably automatically generated) SSL cert, with the hash signed with the NaCl private key. During SSL handshake, one could then compare the received certificate's public key hash with the previously received hash. One could obviously cache these hashes for faster connection setup. Client and server (in the sense of who opens the connection) could exchange their cert hashes in a single HTTP request and response flow, so both sides could then authenticate each other for the SSL connection via certificates.

I think you could also just create a TLS-compatible public key and put it in a certificate together with a NaCl-signature of the key as an extra attribute, and then self-sign it. Verification would then not be through the usual CA chains (all certs are self-signed), but use the NaCl-signature to verify the certificate's public key. This way you wouldn't need a separate channel.

 

While I like HTTP, I'm not sure it's a good idea to try to build a stateful protocol on top of a stateless one when we don't have to. Instead, we could multiplex multiple stateful, bi-directional streams via a single SSL connection as you suggested.

If we can find an existing multiplexing protocol (preferrably with a Go implementation) that works well enough, I think worrying about a better protocol might be something we could do later. An interface providing multiple streams might be simple enough to encapsulate, so we can easily replace it later and get the RPC and file synchronization protocols implemented faster. Worrying about backwards compatibility is probably also a thing for much later. Do you agree to such an approach in general?

 

I agree with this. Multiplexing libraries should share similar interfaces, so anything layered on top of one can probably be easily switched to another one. I didn't really find one for Go that stands out either at the moment though. Maybe muxado or spdystream like Tommi mentioned (or https://github.com/hashicorp/yamux) could work for now, perhaps with some improvements.

As for the protocol on top: I think you'll be stuck with writing a custom protocol since it's fairly tied in to how you represent your filesystem and the synchronization primitives used. Have you written anything down about the operations the protocol would need to support?

Cheers,

Mark

[Tommi Virtanen]

On Sunday, October 26, 2014 5:41:34 AM UTC-7, Mark Abspoel wrote:

I think you could also just create a TLS-compatible public key and put it in a certificate together with a NaCl-signature of the key as an extra attribute, and then self-sign it. Verification would then not be through the usual CA chains (all certs are self-signed), but use the NaCl-signature to verify the certificate's public key. This way you wouldn't need a separate channel.

Has anyone figured out how to use the go.crypto libraries to make TLS certs with these custom fields? That's what I have not yet had the chance to figure out, and why I prototyped with the NaCl signatures in-band.

As for the protocol on top: I think you'll be stuck with writing a custom protocol since it's fairly tied in to how you represent your filesystem and the synchronization primitives used. Have you written anything down about the operations the protocol would need to support?

I have a good draft of the basic sync mechanism that has gotten unfortunately delayed due to life events. I'll get back to it, and once that building block is there I hope it's easier for others to pitch in. Right now, things are still described too vaguely for me to expect anyone else to intuit what I meant.

In the meanwhile, I'd welcome everyone to explore what the current master does as a local-only filesystem, how its persistency works (stop the bazil server daemon and bolt-mount the database to explore what's there!), etc. And if someone were to contribute heavily to OSXFUSE kext, that'd be splendid (see the open issues there).

Also, if anyone in SF/east bay wants a 1-2 hour deep dive session into what the missing parts are, I think such a sounding board for the architecture would be very helpful. Prior knowledge of distributed systems, merkle trees, version vectors, and such is sort of required; understanding (traditional or NFS-style networked) file systems is less relevant.

It's still early days, and I'm very aware of that.