Inbound HTTP APIs design proposal

[Kenton Varda]

Hi sandstorm-dev and Jas,

Here's how I'm thinking of allowing Sandstorm apps to expose HTTP APIs:

- For each Sandstorm server, there will be a designated API host, e.g. alpha.sandstorm.io might have alpha-api.sandstorm.io. This host serves HTTP APIs for all apps on the Sandstorm instance.

- All requests must have an HTTP "Authorization" header in OAuth 2 bearer token style:

  Authorization: Bearer 0cIFXMm47dDPZNSjVSAZEQqvyXo

- The bearer token actually not only proves authorization, but also identifies the specific grain (app instance) to which the request is addressed. This makes it not just a token, but a capability.

- Notice that it's thus not possible to open an API directly in a browser tab, because the browser will not send the "Authorization" header. This is by design: By ensuring that resources served from an API cannot be opened in a browser context, we close a large number of possible security issues. For instance, we ensure that no app can cause arbitrary code to execute in the API's origin, which makes it safe for all APIs to share an origin. (It's also important that API endpoints cannot be used in e.g. <object> tags, as this would allow an app to do all kinds of bad things by serving a malicious Flash file.)

- That said, XmlHTTPRequests will have no problem adding the appropriate header, so APIs can be accessed in the browser through appropriate Javascript.

- The standard way to designate an API capability in text will be to form a URL by taking the API origin, appending a '#', and then appending the capability token. If such a URL is literally opened in a browser, we can potentially present the user with an "API explorer" interface which allows them to poke at the API. The access key will be embedded in the fragment (the # part), which has the nice property that browsers will never include it in a "Referrer" header. (We should probably do this with grain URLs too.)

- APIs will always be served with "Access-Control-Allow-Origin: *" so that they can be invoked from anywhere (provide you have the right capability token).

- An app will be able to ask Sandstorm to create new API token pointing back at it. The app can then present an API URL to the user, and the user can copy/paste that URL into some app (e.g. a mobile client) that wants to connect to the API.

- If you want an API to be public, it is perfectly valid to simply publish the token. You might even embed it in a web site's Javascript to create a site that interacts with your API (e.g. this could be used to implement comments in a blogging app).

- For now, API requests will be delivered by opening another special kind of session, HackApiSession. sandstorm-http-bridge will be configurable to forward these requests to the app with a specified host header and path prefix. In the future, APIs will be exposed by implementing some "HttpApi" interface and publishing it to the Powerbox.

- Eventually, Sandstorm's frontend will support incoming OAuth so that you can initiate a request to access the user's stuff externally. The OAuth flow will in fact invoke the Powerbox. Notice that existing OAuth clients may "just work" with the Powerbox and capabilities even though they weren't designed to work with them, since Sandstorm will just return them a capability that they use like a regular OAuth access token.

Thoughts?

Unfortunately, many client apps may need minor tweaking to work with Sandstorm APIs. Many apps have hand-rolled authentication in their APIs, e.g. passing usernames and passwords, which obviously doesn't work with Sandstorm. Instead, such apps will need to learn how to accept and use a bearer token, which is probably a minor tweak in most cases. Apps that are already based on OAuth 2 will have an easier time, although until we actually have the OAuth UI implemented, these apps will need to provide some way to specify a URL and access token manually. Some may have this already, but many won't.

-Kenton

--

Sandstorm.io is crowdfunding! http://igg.me/at/sandstorm

[dc...@madmode.com]

On Friday, August 15, 2014 9:03:37 PM UTC-5, Kenton Varda wrote:
...

> - For each Sandstorm server, there will be a designated API host, e.g. alpha.sandstorm.io might have alpha-api.sandstorm.io. This host serves HTTP APIs for all apps on the Sandstorm instance.

How about alpha-api-KEYID.sandstorm.io a la YURL? If we're building a capability based platform, let's cut out the dependency on the ambient certificate infrastucture while we're at it... at least pave the way in that direction.

http://www.waterken.com/dev/YURL/

--
Dan Connolly
http://www.madmode.com/

[Kenton Varda]

On Sat, Aug 16, 2014 at 11:14 AM, <dc...@madmode.com> wrote:

How about alpha-api-KEYID.sandstorm.io a la YURL? If we're building a capability based platform, let's cut out the dependency on the ambient certificate infrastucture while we're at it... at least pave the way in that direction.

We could add support for this later as a separate change, but it almost certainly has to be optional.

Placing a key ID in the URL must be done with care, because if that key is ever compromised (think Heartbleed), you must revoke it. In the case of YURLs, that implies revoking every single capability that has been given out using that key ID. This seems unacceptably bad.

A way to make this safer is to make sure the key ID identifies an offline key which is only used to sign a certificate delegating authority to some online key. Then if the online key is compromised, you just need to generate a new key with certificate signed by the same offline key.

Of course, you also want to make sure the compromised key's certificate is revoked. But we know of no working technique to actively revoke a certificate, so the only thing you can do is make sure the expiration time on the certificate is short enough that you can ride it out. This of course means you need to sign a new certificate using your offline key on a regular basis. If your key is truly offline, this involves manual labor (and associated chances for human error).

Also, make sure to keep lots of backups of your offline key in multiple secure geographic locations. Safe deposit boxes, perhaps. Because if your only copy is lost or damaged, everything is ruined. But if any of the copies are stolen, everything is ruined too. Can the NSA coerce a bank to open a safe deposit box? Probably. Sure, you can encrypt them with a passphrase, but you'll want the bus factor for that passphrase to be more than 1.

Obviously, a typical user maintaining their own personal server cannot possibly be expected to get this right. A professional hosting provider might, but it's hard for users to verify, and most users won't understand any of this in the first place. Our hosting service will probably end up doing all of the above on principle.

Abstractly speaking, the question here is between rooting our security in cryptography (the YURL/web-of-trust approach) or human organizations (the CA approach). Crypto is tantalizing because, if done just right, it is infallible -- something humans can never achieve. And, certainly, we have seen lots of fallibility in the CA system in particular. However, that infallibility comes at the price of extreme fragility. A small mistake can ruin the whole system with no way to recover.

-Kenton

[Jasvir Nagra]

On Fri, Aug 15, 2014 at 7:03 PM, Kenton Varda <ken...@sandstorm.io> wrote:

Hi sandstorm-dev and Jas,

Here's how I'm thinking of allowing Sandstorm apps to expose HTTP APIs:

- For each Sandstorm server, there will be a designated API host, e.g. alpha.sandstorm.io might have alpha-api.sandstorm.io. This host serves HTTP APIs for all apps on the Sandstorm instance.

- All requests must have an HTTP "Authorization" header in OAuth 2 bearer token style:

  Authorization: Bearer 0cIFXMm47dDPZNSjVSAZEQqvyXo

- The bearer token actually not only proves authorization, but also identifies the specific grain (app instance) to which the request is addressed. This makes it not just a token, but a capability.

+1 on the resource = f(token) which of course avoids the common case where a token forgets to be checked.

* What is the function that you plan to use to generate the token?

* Do these bearer tokens expire?

You should also set the cache-control header as 2616 suggests. 

- Notice that it's thus not possible to open an API directly in a browser tab, because the browser will not send the "Authorization" header. This is by design: By ensuring that resources served from an API cannot be opened in a browser context, we close a large number of possible security issues. For instance, we ensure that no app can cause arbitrary code to execute in the API's origin, which makes it safe for all APIs to share an origin. (It's also important that API endpoints cannot be used in e.g. <object> tags, as this would allow an app to do all kinds of bad things by serving a malicious Flash file.)

What is the return format for these apis?  If we can control the format and specifically the first few bytes of the format, we can better defend against quirky plugins which we don't know about and rosetta flash like problems.

Non-blockers but nice if we can be paranoia and have some defense in depth - disallow null bytes, agree on a charset (UTF-8).

- That said, XmlHTTPRequests will have no problem adding the appropriate header, so APIs can be accessed in the browser through appropriate Javascript.

+1  Also if the file does not need to be displayed by the browser ever, I'd recommend applying Content-Disposition: attachment; filename="XXX" although this might interfere with the api browser below.

- The standard way to designate an API capability in text will be to form a URL by taking the API origin, appending a '#', and then appending the capability token. If such a URL is literally opened in a browser, we can potentially present the user with an "API explorer" interface which allows them to poke at the API. The access key will be embedded in the fragment (the # part), which has the nice property that browsers will never include it in a "Referrer" header. (We should probably do this with grain URLs too.)

I really like the ease of sharing you get from the url being the entire cap especially when you're debugging and building an app.  That said, by putting the bearer token in the auth header, you've eliminated a huge class of leak problems.  The balance is worth exploring.

FYI, I've intermittently grepped for fragments in access logs in my own servers and in access logs made public (say here http://ita.ee.lbl.gov/html/traces.html fr'instance).  "Every once in a while", fragments do show up in server logs even though they are not supposed to.  I have not had a chance to investigate why this happens but the reported user-agent does vary.  It might be XHR or just users using curl/wget.

 

- APIs will always be served with "Access-Control-Allow-Origin: *" so that they can be invoked from anywhere (provide you have the right capability token).

+1 I especially like that given this, your threat modeling simplifies.  You now no longer have to think about two cases - specifically what happens if one of the endpoints you ACAO restricted something to can proxy someone else's request à la confused deputy.

- An app will be able to ask Sandstorm to create new API token pointing back at it. The app can then present an API URL to the user, and the user can copy/paste that URL into some app (e.g. a mobile client) that wants to connect to the API.

In what part of the URL does the bearer token feature in this generated URL? For this exchange url, it is worth investigating whether YURLs meet your needs.

- If you want an API to be public, it is perfectly valid to simply publish the token. You might even embed it in a web site's Javascript to create a site that interacts with your API (e.g. this could be used to implement comments in a blogging app).

- For now, API requests will be delivered by opening another special kind of session, HackApiSession. sandstorm-http-bridge will be configurable to forward these requests to the app with a specified host header and path prefix. In the future, APIs will be exposed by implementing some "HttpApi" interface and publishing it to the Powerbox.

- Eventually, Sandstorm's frontend will support incoming OAuth so that you can initiate a request to access the user's stuff externally. The OAuth flow will in fact invoke the Powerbox. Notice that existing OAuth clients may "just work" with the Powerbox and capabilities even though they weren't designed to work with them, since Sandstorm will just return them a capability that they use like a regular OAuth access token.

Thoughts?

Unfortunately, many client apps may need minor tweaking to work with Sandstorm APIs. Many apps have hand-rolled authentication in their APIs, e.g. passing usernames and passwords, which obviously doesn't work with Sandstorm. Instead, such apps will need to learn how to accept and use a bearer token, which is probably a minor tweak in most cases. Apps that are already based on OAuth 2 will have an easier time, although until we actually have the OAuth UI implemented, these apps will need to provide some way to specify a URL and access token manually. Some may have this already, but many won't.

-Kenton

--

Sandstorm.io is crowdfunding! http://igg.me/at/sandstorm

--
You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kenton Varda]

On Tue, Aug 19, 2014 at 12:23 PM, Jasvir Nagra <jas...@gmail.com> wrote:

* What is the function that you plan to use to generate the token?

For now, just random bits which the front-end will look up on a table. Hmm, perhaps the table should be keyed by hash of those bits.

Later on, the token may actually be a Cap'n Proto SturdyRef, with the explicit intent that clients can decode the SturdyRef and connect to it as a Cap'n Proto capability if they so desire.

 

* Do these bearer tokens expire?

It will be possible to set an expiration when obtaining one, but in many cases expiration would be inconvenient.

Eventually, tokens will appear in the sharing UI and will be revocable there.

 

You should also set the cache-control header as 2616 suggests. 

We will certainly need to force "Vary: Authorization". Given that, can we allow the app to set Cache-Control arbitrarily? An app that explicitly intends for its APIs to be public may very well want to enable caching.

Or is the Vary header too broken? If so, we probably need to disable all caching.

 

What is the return format for these apis?  If we can control the format and specifically the first few bytes of the format, we can better defend against quirky plugins which we don't know about and rosetta flash like problems.

Unfortunately, I don't think we can control this. Apps need the ability to set arbitrary content-types and send arbitrary content. Consider an app like ownCloud which is literally a filesystem. Storing Flash videos in your ownCloud is a totally legitimate use case.

But my hope was that requiring the Authorization header totally blocked rosetta-flash-like attacks, because the browser will never send this header (or at least will never set it to an appropriate value) when fetching content for the main browser window or an <object> tag.

- That said, XmlHTTPRequests will have no problem adding the appropriate header, so APIs can be accessed in the browser through appropriate Javascript.

+1  Also if the file does not need to be displayed by the browser ever, I'd recommend applying Content-Disposition: attachment; filename="XXX" although this might interfere with the api browser below.

Good idea.

- The standard way to designate an API capability in text will be to form a URL by taking the API origin, appending a '#', and then appending the capability token. If such a URL is literally opened in a browser, we can potentially present the user with an "API explorer" interface which allows them to poke at the API. The access key will be embedded in the fragment (the # part), which has the nice property that browsers will never include it in a "Referrer" header. (We should probably do this with grain URLs too.)

I really like the ease of sharing you get from the url being the entire cap especially when you're debugging and building an app.  That said, by putting the bearer token in the auth header, you've eliminated a huge class of leak problems.  The balance is worth exploring.

FYI, I've intermittently grepped for fragments in access logs in my own servers and in access logs made public (say here http://ita.ee.lbl.gov/html/traces.html fr'instance).  "Every once in a while", fragments do show up in server logs even though they are not supposed to.  I have not had a chance to investigate why this happens but the reported user-agent does vary.  It might be XHR or just users using curl/wget.

Well, the idea is that all clients should be removing the fragment and moving the key over to the Authorization header before sending any requests, so hopefully it won't show up in any logs.

- APIs will always be served with "Access-Control-Allow-Origin: *" so that they can be invoked from anywhere (provide you have the right capability token).

+1 I especially like that given this, your threat modeling simplifies.  You now no longer have to think about two cases - specifically what happens if one of the endpoints you ACAO restricted something to can proxy someone else's request à la confused deputy.

- An app will be able to ask Sandstorm to create new API token pointing back at it. The app can then present an API URL to the user, and the user can copy/paste that URL into some app (e.g. a mobile client) that wants to connect to the API.

In what part of the URL does the bearer token feature in this generated URL? For this exchange url, it is worth investigating whether YURLs meet your needs.

The fragment, as described above.

-Kenton