The next generation of Sandcats

[Troy Farrell]

sandcats.io is the domain that Sandstorm has offered for those who chose not to install on their own domains.  The current edition of the Sandcats service is good enough, but not really manageable for the longer-term.

For that reason, I intend to create a replacement using Cloudflare DNS and Cloudflare Workers.  This means that using sandcats.io will involve Cloudflare having a small bit of information about our users.  (I will enumerate this information when I have sorted out what exactly it is.)  I'm OK with this and I suspect that most users are OK with this.  I'm posting this message to see if anyone objects to this.

Troy

[Julian Foad]

I have a general objection to using megacorp services. In general they
distort, in megacorp's favour, the landscape of everything our society
does, in particular away from user agency and interoperability and the
like, the sort of things I feel Sandstorm stands for. Even "just" using
a service creates reliance on it, familiarity and promotion of it. I'm
acutely aware the ethical stance has to be balanced against the desire
to Just Get Stuff Done. I have no particular insight into what's best
for this specific proposition.

That's all. Well, you did ask :-)

- Julian Foad
Matrix me: @julian:foad.me.uk
Follow me: @jul...@fed.foad.me.uk

[Dan Krol]

My objections would depend on the enumeration. Off the top I don't anticipate problems. I guess just give users a heads up when they install.

Though to Julian's point, this can be our motivation (somewhere in the long term roadmap) to provide a way to guide users through setting up their own domain easily enough.

--
You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CED3E97E-B275-4CED-8598-608591ADACF0%40getmailspring.com.

[Kenton Varda]

Hey Julian,

You may be interested to know that I'm the original founder of Sandstorm, and I am now the lead engineer building Cloudflare Workers (and one of the top engineers at Cloudflare in general).

I don't know if I'd say Cloudflare is a mega-corp yet. We're still like 1/100 the size of Google or Amazon. We're flattered that people have started putting us in the same league but also a little puzzled by the characterization that Cloudflare is taking over the internet when we're still so small compared to AWS. :)

I think Sandcats is inherently a centralized service, and so inherently goes against the philosophy of Sandstorm. Of course, it's not required at all -- you can put Sandstorm on any domain and DNS provider you want. But doing so requires a lot of manual systems administration work, which we were trying to free people from. So we created Sandcats as a very simple alternative. In the long run it would be cool if Sandstorm had integrations with many popular DNS providers, but building those out is a lot of work.

I originally suggested Cloudflare DNS and Workers as a good way to re-implement Sandcats because it would perform incredibly well while requiring essentially zero maintenance. Of course, I'm a bit biased. :) But the benefits would be:

1. Cloudflare DNS is hosted out of hundreds of cities worldwide. Compare to sandcats today, which is hosted out of one machine in a Google Cloud datacenter in Iowa. Sandcats sets DNS cache TTL to only 60s, so today everyone is incurring a round trip to Iowa every 60 seconds... To be honest, I'm a little surprised the performance hasn't come up as a major issue, especially for people outside the US.

2. With Workers you don't have to maintain a whole OS instance for your service. This is a big problem for Sandcats today -- it's running on a custom stack of several different software components on a Linux machine that was stood up in 2015 and hardly touched since. It's liable to stop working at any time and I'm not sure anyone could fix it if that happened. With Workers, everything other than the application code is maintained by Cloudflare and we have a strong commitment to backwards compatibility, so pretty much once the Worker code is written it should just work forever.

3. If anything goes wrong you can complain directly to me!

With all that said, when I originally proposed moving Sandcats to Workers, I was also planning to write it myself. It's now becoming clear I'll never find the time for that. :( If Troy is going to write a new Sandcats, I would encourage him to use whatever tech stack he is most comfortable with.

-Kenton

[Jacob Weisz]

I assume the impression of Cloudflare's size comes mostly from the number of websites which use Cloudflare for free (which doesn't translate to "business size" in a strong sense), combined with the occasional issue privacy folks have with Cloudflare's bot protection. (Cloudflare's CAPTCHA thing just infinitely loops for me on one my devices, and I know others have reported similar at various times, most likely for some odd combination of blocking of signals that makes it look very spammy.)

I know in my case I am drastically more comfortable with a company that does not sell ads having my data than a company that sells ads, so moving Sandcats from Google to Cloudflare seems like an unambiguous privacy and reliability gain.

But I also would imagine considering the strong ethical boundaries Sandstorm and Sandcats has operated under, we will have to have both significant advance notice and user consent for a change of venue.

--

  Jacob Weisz

  goo...@jacobweisz.com

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CAOP%3D4wgioTc9z8fV%2BwP4xxrFF2_gXQxERWnYGoOBQVNbhG7yhg%40mail.gmail.com.

[sand...@greenant.net]

Some interesting points regarding moving Sandcats.
From my understanding, Sandcats is essentially a DNS server which allows the registration of wildcard certs required for users without their own domain.

While users who adminster their own domain don't require Sandcats, I can see it remaining an important service with regards to new users and the Sandstorm/Tempest ecosystem as a whole. I also wonder whether it could help as an arbitration service for federation of servers in the future.

Therefore, I think it remains important for that part of the infrastructure to also be portable and open-source. While Kenton's offer to use Cloudflare is certainly generous, I do wonder if this locks Sandcats unnecessarily to the Cloudflare ecosystem.

I think it would make sense to have a look at the Linux image used for the current Sandcats to see if it is easy or difficult to maintain. I am guessing it's probably a DNS server connected to an LDAP server plus some scripting for new accounts, so may not be too difficlt to maintain. Hosting is less of a problem if it's a generic Linux image.

The other concern raised regarding latency on DNS queries is worth considering, raising TTL should be a simple matter. For the moment, I'm not sure that the extra DNS queries would be a significant performance bottleneck for most users. Ideally, the infrastructure could be geo-distributed, again not too difficult with the use of a generic Linux VM.


All the best,

Frank Giorlando
GreenAnt Networks
W: https://greenant.net <https://greenant.net>

>>   Matrix me: @julian:foad.me.uk <http://foad.me.uk>
>>   Follow me: @jul...@fed.foad.me.uk <mailto:jul...@fed.foad.me.uk>

>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Sandstorm Development" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to sandstorm-de...@googlegroups.com

>> <mailto:sandstorm-dev%2Bunsu...@googlegroups.com>.

>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/sandstorm-dev/CED3E97E-B275-4CED-8598-608591ADACF0%40getmailspring.com <https://groups.google.com/d/msgid/sandstorm-dev/CED3E97E-B275-4CED-8598-608591ADACF0%40getmailspring.com>.

>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Sandstorm Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to sandstorm-de...@googlegroups.com

>> <mailto:sandstorm-de...@googlegroups.com>.

>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/sandstorm-dev/CAOP%3D4wgioTc9z8fV%2BwP4xxrFF2_gXQxERWnYGoOBQVNbhG7yhg%40mail.gmail.com <https://groups.google.com/d/msgid/sandstorm-dev/CAOP%3D4wgioTc9z8fV%2BwP4xxrFF2_gXQxERWnYGoOBQVNbhG7yhg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Sandstorm Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sandstorm-de...@googlegroups.com

> <mailto:sandstorm-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sandstorm-dev/90acc05c-ba87-4fd7-b6e1-a375e67ac80a%40app.fastmail.com <https://groups.google.com/d/msgid/sandstorm-dev/90acc05c-ba87-4fd7-b6e1-a375e67ac80a%40app.fastmail.com?utm_medium=email&utm_source=footer>.

[Jacob Weisz]

I think now that workerd is open source we would have options to move even a Workers-based solution.

The source for Sandcats is here https://github.com/sandstorm-io/sandcats and it's readme has a pretty good summary of the current stack.

--
Jacob Weisz
goo...@jacobweisz.com

> an email to sandstorm-de...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sandstorm-dev/24f2dd5c-3465-43ed-ba1e-4dff196e725b%40greenant.net.

[Kenton Varda]

Keep in mind that Sandcats is a service, not software. There is some software involved in running the service, but it's relatively simple, not hard to reproduce from scratch. Moreover, there is only one Sandcats -- there is simply no reason for anyone else to be running this code, except for the official Sandcats service itself.

So I'd encourage everyone to resist the temptation to think about this the same way you think about software (like the rest of Sandstorm). The thing to optimize for here is minimal maintenance burden for whomever intends to maintain it long-term. Do not optimize for self-hosting or reusability across multiple service providers... in the worst case you just build a new sandcats from scratch if you need to, since the code is simple.

That said, the Cloudflare Workers runtime is in fact open source; code written as a Worker could in fact run on any old Linux box.

It's worth clarifying there are really two components of the service:

1. The control plane / business logic. Currently https://github.com/sandstorm-io/sandcats -- this is what is being proposed to rewrite as a Cloudflare Worker, which could run on Cloudflare or open-source workerd anywhere.

2. The DNS service infrastructure. Currently self-hosted PowerDNS + MySQL. This is proposed to be replaced with Cloudflare DNS. Cloudflare DNS is proprietary, but commodity. It would likely be trivial to replace it with any other DNS service by swapping out a few API calls.

Note there's an additional concern we need to address here which is: Is sandcats.io itself going to have control transferred to a new owner? If so, do users need to opt into that? What happens if they don't opt in before the switch-over date? If we can get this stuff built as a Worker, I might just be willing to operate it myself, which sidesteps the whole question since there's no transfer of control... just a thought. But I really don't want to operate any more Linux VMs. :)

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/24f2dd5c-3465-43ed-ba1e-4dff196e725b%40greenant.net.

[Jacob Weisz]

I also did pick up sandcats.org so we have some flexibility on the community org being able to potentially offer a new namespace to new users in the interim of figuring out the transfer of sandcats.io (which would need some lengthy time to ensure people have time to opt in or migrate away). I also imagine we would have to still block out any sandcats.io subdomains that didn't transfer so they couldn't be maliciously misused. In the short term we can use the .org for testing, Sandstorm helpfully leaves the base domain of Sandcats as a configurable option.

There'd definitely be a risk to the existing stack collapsing during the migration period, so the idea of something Kenton is also comfortable operating is definitely appealing, as is possibly community members looking at the maintenance of the existing platform.

--

  Jacob Weisz

  goo...@jacobweisz.com

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CAOP%3D4wiv0vy%3DwDwX7CvOx3SqGrADNhqLyFJkYTv88Oszg6%2Byog%40mail.gmail.com.

[Troy Farrell]

I've been studying the current implementation of Sandcats and its integration with Sandstorm.  I've been working under several assumptions:

1. The use of Cloudflare DNS is desirable for performance reasons.
2. The use of Cloudflare Workers is desirable for simplification of maintenance.
3. Because Cloudflare Workers do not support UDP, an additional server will be necessary to support the UDP ping protocol.

Open questions include:

1. Can Sandcat's mutual TLS (mTLS) be made to work with Cloudflare Workers? (I'm not convinced it can, but I will attempt to make it work.)
2. The existing Sandstorm implementation of Sandcats assumes that the REST API and the UDP pings are sent to the same host: sandcats.io.  The API would be provide by Cloudflare Workers and the UDP ping would be supported by an additional server.  Should we change the planned architecture or modify the existing Sandstorm implementation?
3. Will existing users be satisfied with having the necessary information stored at Cloudflare?

Per my current understanding, the following information would be hosted in a Cloudflare D1 database:

1. IPv4 and IPv6 addresses of Sandstorm servers which use Sandcats
2. hostnames of Sandstorm servers which use Sandcats
3. mTLS information for Sandstorm servers which use Sandcats
4. a recovery e-mail address for each hostname

This list is likely incomplete.  I suspect that additional information may be necessary, such as timestamps and additional DNS records.  (One of my hopes is to implement SPF and/or DKIM support.)

This message is an invitation to provide feedback on the above open questions regarding the future of Sandcats.

Thanks!
Troy

[Kenton Varda]

That all sounds right to me.

Cloudflare supports mTLS authentication of clients. I guess this is a feature of the "API Shield" product:

https://developers.cloudflare.com/api-shield/security/mtls/configure/

I'm not personally very familiar with this product, but the end result of using it should be that your worker receives client cert information in `request.cf.tlsClientApth`.

https://developers.cloudflare.com/workers/runtime-apis/request/#incomingrequestcfproperties

That said, the docs imply that all certificates have to be signed by a CA -- possibly a custom CA you set up. I'm not sure this works for us, though, since I think clients of the sandcats API all use self-signed certificates, and we decide whether to trust them based on key fingerprint (it must match the fingerprint stored in the database). I can maybe check with the team to see if there's a way to allow self-signed certs here... it's a valid use case.

Alternatively, we could consider changing Sandstorm to use a different authentication mechanism, like a simple bearer token in the HTTP request. Note we'd have to change not just the Sandstorm code, but also the installer, which does the initial Sandcats registration.

As you note, it seems like we can't escape making some sort of change to the Sandstorm code, in order to send UDP and HTTP to different hosts. But that in itself could be a very minimal change.

Another possible issue to consider here: The sandcats database currently contains 24,580 registered hostnames. The vast majority of these are abandoned, but the database doesn't currently track which ones are active. The UDP responder could, of course, figure out very quickly which hosts are active, and it would frankly be a good idea for us to auto-delete the records for any host that isn't active.

Relatedly, the maximum number of DNS records on Cloudflare is 3500 by default. I can probably get the limit raised for us if needed, but if you didn't have me here, you'd need an enterprise plan to get a higher limit. 3500 is probably plenty for the time being though.

Given all these challenges, to be perfectly honest, I am not sure if Cloudflare is the best hosting option for this. If it were me building it, I'd still try, just because I'm a Cloudflare employee, but I'd hesitate to recommend that you follow this route.

Perhaps it would be more reasonable after all to try modernizing the existing codebase, and seeing if you are able to set up a new instance of it that works. Then, we can move the data over from the current servers?

-Kenton

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/3fad997b-2e17-4edb-b3e0-aa40a256863fn%40googlegroups.com.

[Troy Farrell]

Thanks for taking the time to comment.  Comments inline.

On Monday, January 1, 2024 at 3:49:00 PM UTC-6 Kenton wrote:

 

That said, the docs imply that all certificates have to be signed by a CA -- possibly a custom CA you set up. I'm not sure this works for us, though, since I think clients of the sandcats API all use self-signed certificates, and we decide whether to trust them based on key fingerprint (it must match the fingerprint stored in the database). I can maybe check with the team to see if there's a way to allow self-signed certs here... it's a valid use case.

 

Yes, the custom CA was the only configuration that I saw documented.  I'm OK with trying to make this work if we satisfy the other constraints.  More below.

Alternatively, we could consider changing Sandstorm to use a different authentication mechanism, like a simple bearer token in the HTTP request. Note we'd have to change not just the Sandstorm code, but also the installer, which does the initial Sandcats registration.

I prefer trusting mTLS, but lots of other systems use bearer tokens successfully.

 

Another possible issue to consider here: The sandcats database currently contains 24,580 registered hostnames. The vast majority of these are abandoned, but the database doesn't currently track which ones are active. The UDP responder could, of course, figure out very quickly which hosts are active, and it would frankly be a good idea for us to auto-delete the records for any host that isn't active.

I don't believe that we have a policy of deleting abandoned host names.  I agree that we should delete abandoned hostnames, though I have the authentication certificate for a hostname that I haven't used in two years, which I plan to use again soonish.  Implementing this would require adding a last-seen timestamp to the database.  I don't think that is much of a barrier, either in technical or privacy terms.

 

Relatedly, the maximum number of DNS records on Cloudflare is 3500 by default. I can probably get the limit raised for us if needed, but if you didn't have me here, you'd need an enterprise plan to get a higher limit. 3500 is probably plenty for the time being though.

That's a bit of a problem.  I'm glad that you're at Cloudflare, but I believe that we should strive for sustainability where that's practical.  We're sort of reviving an octopus at this point, so I'm willing to be flexible.  If we agree to remove abandoned hostnames, we could still use Cloudflare.

 

Given all these challenges, to be perfectly honest, I am not sure if Cloudflare is the best hosting option for this. If it were me building it, I'd still try, just because I'm a Cloudflare employee, but I'd hesitate to recommend that you follow this route.

Perhaps it would be more reasonable after all to try modernizing the existing codebase, and seeing if you are able to set up a new instance of it that works. Then, we can move the data over from the current servers?

If we move the data from your control, then we probably need to sunset the existing system and give people time to move on if they aren't willing to entrust their data (limited though it may be) to a community-operated system.  I suspect we'll need to do that anyway at some point.

So perhaps we have two paths forward:

1. Semi-crazy idea: Modernize or replace the existing Sandcats code with a system that has a main server, which hosts the (current) API and builds a zonefile, which is then distributed to 3-5 DNS servers which are hosted by community members.  This would give us some geographic distribution and fault-tolerance (as well as lots of points of failure.)  The main server would then verify that the community-hosted DNS servers are serving the correct records.  If they aren't it would update the zone to remove them as authoritative for the zone.

2. Build on Cloudflare.  Delete abandoned hostnames.  Use mTLS if it can be made to work.  If not, use bearer tokens.  Kenton remains in control for now.

I'm happy to write the code for either one.  I could host a DNS server in Germany for the first option.

Happy new year!

Troy

[Jacob Weisz]

> I agree that we should delete abandoned hostnames

I think it's worth clarifying: We should still lock them out to avoid reuse, right?

--

  Jacob Weisz

  goo...@jacobweisz.com

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/ade426e5-0042-45f6-ae54-20dad3ba2989n%40googlegroups.com.

[Troy Farrell]

On Monday, January 1, 2024 at 11:49:17 PM UTC-6 Jacob Weisz wrote:

> I agree that we should delete abandoned hostnames

I think it's worth clarifying: We should still lock them out to avoid reuse, right?

Off the top of my head, Sandcats should not resolve them (i.e., not keep them in the zonefile), but should keep the information in the database so that an interested party could recover them via the recovery e-mail address or the registered self-signed certificate, should she have kept a backup.

A more interesting question is this: should Sandcats ever allow the re-registration of a previously registered hostname which has been abandoned?  I believe that the answer is yes, but with a delay of five to ten years.  Allowing re-registration of abandoned hostnames is essential for avoiding a situation where new hosts are only able to register names like appster4731.sandcats.io.  It has the downside of breaking links.  I'm OK with this because I consider the probability of those links ever working to be low if the hostname has been abandoned for such a long time already.

[Kenton Varda]

Sorry, to clarify, I meant that we should delete the DNS records for abandoned hostnames, and totally forget the IP address. We would still of course keep track of the registration in our database and would not allow someone new to claim the hostname. Removing the DNS record could happen more or less immediately after we stop receiving pings from the server. (This would also solve a separate pain point: Occasionally people set up a sandcats hostname, then decide they don't want it, but notice that it continues to resolve to their IP address, which makes them uncomfortable, so they email me to request I delete the hostname, but I don't have any good way to do that... best if shutting down your server means the hostname goes away on its own.)

I think for security reasons we shouldn't allow re-registration of old hostnames ever. I don't think that namespace crowding is a serious problem for us with only ~25k registrations. But if it did become a problem, the solution can be to just register more domains, each of which has its own namespace.

Cloudflare's 3500-host limit would of course only apply to DNS records, so if that's limited to active hosts then it's probably plenty for now.

-Kenton

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/98f98dca-8216-4ea4-add6-bce93df6c119n%40googlegroups.com.

[Troy Farrell]

I would prefer to leave DNS records in place for at least several hours, preferably a day or so, should a server stop replying to pings.  This would allow e-mail to continue to work even if connectivity is lost temporarily.  (Specifically, I have MX and SPF records, which Sandcats does not currently support, in mind.)

I would be willing to kick the discussion of re-registration of old hostnames down the road by several years.  To do that, I propose storing the most recent year that a hostname sent pings.  That seems to be a reasonable compromise of having information about the use or disuse of a hostname and maintaining privacy.  If we decide that we need to allow reuse, we'll be able to see which hostnames haven't been used.  If not, a year is not much in the way of private information.

If this is acceptable, here's what we have:

1. Servers which do not respond to pings are removed from DNS (shortly) after 24 hours.  IPv4 and IPv6 addresses could also be removed from the database at this point.

2. Servers which are responding to pings have corresponding records in Cloudflare DNS.

3. Cloudflare DNS records are managed by a Cloudflare Workers-based API, working with one or more cloud servers which handle UDP pings.
4. Database records are stored in Cloudflare D1.

5. The above is contingent on making Cloudflare Workers work with mTLS as implemented by Sandstorm.

6. Database records: IPv4 and IPv6 addresses, hostnames, mTLS information, recovery e-mail addresses and most-recent year seen.

If mTLS doesn't work with Cloudflare Workers, we could proxy requests through a different server (which would allow for compatibility with existing Sandstorm systems) or switch to a bearer-token authentication system (or both.)

I'm going to try to sort out the mTLS question now.

[Troy Farrell]

This is a status update.

I have not sorted out how to get Cloudflare Workers to request the certificate for mTLS when the CA for the Cloudflare Workers does not match the CA that signed the client certificate.  Here's a request from cURL and the request.cf object (with irrelevant properties removed) that the Cloudflare Worker sees:

$ curl --max-time 20 -X POST --data-urlencode "rawHostname=foobarbaz" --data-urlencode "email=f...@bar.com" -H "X-Sand: cats" -H "Accept: text/plain" --cert id_rsa.private_combined -v https://35b1222b.sandcats.pages.dev/mtls-test
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 188.114.96.1:443...
* Connected to 35b1222b.sandcats.pages.dev (188.114.96.1) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=sandcats.pages.dev
*  start date: Dec 23 07:42:53 2023 GMT
*  expire date: Mar 22 07:42:52 2024 GMT
*  subjectAltName: host "35b1222b.sandcats.pages.dev" matched cert's "*.sandcats.pages.dev"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: POST]
* h2 [:scheme: https]
* h2 [:authority: 35b1222b.sandcats.pages.dev]
* h2 [:path: /mtls-test]
* h2 [user-agent: curl/8.1.2]
* h2 [x-sand: cats]
* h2 [accept: text/plain]
* h2 [content-length: 41]
* h2 [content-type: application/x-www-form-urlencoded]
* Using Stream ID: 1 (easy handle 0x7f7cc7011e00)
> POST /mtls-test HTTP/2
> Host: 35b1222b.sandcats.pages.dev
> User-Agent: curl/8.1.2
> X-Sand: cats
> Accept: text/plain
> Content-Length: 41
> Content-Type: application/x-www-form-urlencoded
>
* We are completely uploaded and fine
< HTTP/2 200
< date: Thu, 04 Jan 2024 18:11:20 GMT
< content-type: text/plain;charset=UTF-8
< content-length: 1497
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=SdG2JF%2FGB5Dazq0qpC%2ByQ9qzQwYwfc1VqQrriYQT%2FlsrrI7eHA%2F0%2BT2cCGkE%2Bc5TgMkUgh0Mp5Lu6%2BKQI7iRdLdeHAGscCeFR9WtHOPffoV8gUONxGfCqFZ%2BsEk7Zp62iCw4CK%2FXYcFr6HWTUvA%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 84057182bc65b45b-HKG
< alt-svc: h3=":443"; ma=86400
<
* Connection #0 to host 35b1222b.sandcats.pages.dev left intact
{"httpProtocol":"HTTP/2","tlsCipher":"AEAD-AES256-GCM-SHA384","tlsClientAuth":{"certIssuerDNLegacy":"","certIssuerSKI":"","certSubjectDNRFC2253":"","certSubjectDNLegacy":"","certFingerprintSHA256":"","certNotBefore":"","certSKI":"","certSerial":"","certIssuerDN":"","certVerified":"NONE","certNotAfter":"","certSubjectDN":"","certPresented":"0","certRevoked":"0","certIssuerSerial":"","certIssuerDNRFC2253":"","certFingerprintSHA1":""},"tlsExportedAuthenticator":{"clientFinished":"faa51032df8a81cc50998ee61094e2461278d2c87b958daab49b768f2645b362cd4a43b3b386ee4603f7c2915de35768","clientHandshake":"4ff562d4369b021af26ca368519655f442d238b6d5af3538af980d3f1128c55cd18b0e96e8194f1d9cc1e03f07e1468c","serverHandshake":"929b7a5f611f3f214edaa945f9462bcb3fc400c0f5ad02756be54ca1b127be3af0631bce9c9aa54cea9f382e177501c5","serverFinished":"84f8c0b29c56a7d9ed437aa872adc957a2fa4eb834c9e4d68e11cdd0e15861ad758c11ff6876c4134dda882c9637ceb0"},"tlsVersion":"TLSv1.3","colo":"HKG","verifiedBotCategory":"","edgeRequestKeepAliveStatus":1,"requestPriority":"weight=16;exclusive=0;group=0;group-weight=0","pagesHostName":"35b1222b.sandcats.pages.dev","botManagement":{"corporateProxy":false,"verifiedBot":false,"jsDetection":{"passed":false},"staticResource":false,"detectionIds":{},"score":99}}

Notice that the cURL does not send the client certificate in the TLS handshake.  By way of contrast, here's a request to the current version of sandcats.io, to a known-to-exist domain, so the 400 status response is expected:

$ curl --max-time 20 --data-urlencode "rawHostname=<hostname>" --data-urlencode "email=<private>" -H "X-Sand: cats" -H "Accept: text/plain" --cert id_rsa.private_combined -v https://sandcats.io/register
*   Trying 104.197.28.173:443...
* Connected to sandcats.io (104.197.28.173) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=sandcats.io
*  start date: Dec 29 03:39:54 2023 GMT
*  expire date: Mar 28 03:39:53 2024 GMT
*  subjectAltName: host "sandcats.io" matched cert's "sandcats.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /register HTTP/1.1
> Host: sandcats.io
> User-Agent: curl/8.1.2
> X-Sand: cats
> Accept: text/plain
> Content-Length: 53
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 400 Bad Request
< Server: nginx/1.9.11
< Date: Thu, 04 Jan 2024 07:13:21 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
<
* Connection #0 to host sandcats.io left intact

Note that cURL sends the client certificate to sandcats.io.  It does not send the client certificate to the Cloudflare Worker.  I don't know if Cloudflare API Shield has a setting like the "ssl_verify client optional_no_ca setting" which sandcats is using with NGINX.

I will continue research on this in several days.

[Kenton Varda]

This all sounds fine. Yes, keeping the DNS records around for 24 hours seems reasonable.

I suppose mTLS connections could be proxied through the same server that is handling the UDP. This avoids the need to change Sandstorm to use two different hostnames for these. Traffic should be extremely low so one VM somewhere should be sufficient. Downsides are:

* That server needs to keep its HTTPS certificate updated, but that can be automated.

* The server now needs to have some credentials (to authenticate to the "back-end" CF Worker) whereas the UDP server required no special access to anything (it would just check against public DNS addresses and send a reply if they didn't match). Also, serving HTTPS traffic requires exposing a lot more attack surface to the public. So now this server needs security maintenance. But I guess it can still be kept simple enough that this should not be too bad.

-Kenton

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/d57e1ca5-b42a-428b-9afa-a59ea580ce7fn%40googlegroups.com.