can we make cockroach demo go back to --insecure by default?

[Andrei Matei]

Hey Rafa, Rohan,

A recent change you've made switched the default for cockroach demo from insecure to secure. This seems bad to me. It also seems like it was controversial (1, 2) while in the making. So I'd like to relitigate :)

When I use cockroach demo, I want to bang at the thing as easily as possible. Security is the last thing I care about. With the cluster defaulting to secure mode, I immediately run into two problems; I wasn't even trying:

1. The admin UI is asking me for a user/pass. I was at a loss for a while before I saw that the command line had printed a user for me to use.

2. I can't open another sql client to the cluster, cause I don't have the certificates. How am I supposed to get them? Couldn't figure it out - other than lsof-ing the process and snooping on the /tmp location where it put them.

The reason for this change seems to have been:
"
cockroach demo does not enable teaching about SQL privileges (all privilege checking is disabled in insecure mode)
cockroach demo does not enable teaching about roles (all role checking is disabled in insecure mode)
cockroach demo does not enable teaching about multiple users (disabled in insecure mode)
"

Then, in #46551 where Andy is complaining about it, you bring up:

1. using the demo cluster from a different shell is not something demo was designed for -- we do not document it and we never intended this to be a thing. If you want a multi-session cluster, use cockroach start-single-node.

2. there are too many features that are plainly disabled / broken in the insecure mode. That's the reason why We do not want the insecure mode, like, at all. The insecure should only be used by cockroachdb developers and not even become available in production builds shipped to users/customers.

3. Something else you should consider here is that if you have a multi-user system, or say some background apps running on the same machine as your demo shell and it's insecure, it's possible for malware to (ab)use the crdb software and cause nefarious effects on your user account.
   There's no track record of applications built to run on a personal laptop that start non-authenticated services over the network. This is just "not done" in the entire industry. FWIW this is exactly the kind of practice that makes people very critical of Zoom atm because of a similar flaw they had in their software a year or two ago. We don't want to be in the same reputation ball park.

My opinion is that the teaching about security features is a very specific thing, probably stemming from a lab we run, and for the purposes of that the instructions can very easily use --insecure=false. Or not use demo at all.

To me, the easy of use is the biggest criteria for cockroach demo. That's what it's all about!

You say that demo wasn't designed to attach a 2nd shell to it. But why not / what's there to design? It used to work just fine. And in fact that's exactly how I got here; I'm attaching second shells! You care a lot about teaching "security features", but how about teaching transaction interactions?
And it's not just shells; other applications should connect to the demo just fine. I don't see reasons not to.

You say features are broken in insecure mode. What features?

To the other point about how insecure should only be used by crdb devs, I call citation needed. Demo was just fine for everybody in insecure mode, in my opinion.

The point about non-authenticated networked services, I don't understand very well. Does the demo listen to other interfaces than localhost? Cause if so, maybe we can make localhost only by default. And does talking about authentication really make sense when the account for the UI is root/admin and the certs can in fact be found by anybody but a regular user?

I'd really like to revert this behavior... It seems like one of those decisions for which I'd bet on a 10-0 outcome in a random poll. But I'm curious what others think, hence this message.

Thanks!

- a_m

[Andrei Matei]

I forgot the other thing that brought me here: in a `demo --nodes=3` cluster, one would really expect to be able to connect to any / multiple of the nodes.

[Jordan Lewis]

I agree with you, Andrei. Raphael and I have talked about it a little. He agrees (I think) that at the bare minimum we need to make sure that the web ui is loginless with cockroach demo. Without that, I think it's a nonstarter.

I like the idea of just having people use ./cockroach demo --insecure=false if they want to check out our users system. (Which, 9 times out of 10 at least, that is not what they're doing.)

Jordan

[Andy Kimball]

I agree wholeheartedly with Andrei/Jordan. *Any* friction to using the demo command undermines its usefulness as a tool for "kicking the tires" on CRDB. It is *so* frustrating to use some new tool or service and get "access denied" errors when you're running on your local machine and just trying to try out some simple scenario. We're letting the tail wag the dog by being worried about exposing every important feature by default in the demo. That's less important than trying to get to as near zero friction as possible.

--
You received this message because you are subscribed to the Google Groups "CockroachDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroach-db...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/CAJg57DiEjwqzPOL%2BRUwwhK0jQiQy4c%2B-EpJ5aBtbRW27dq39gg%40mail.gmail.com.

[Nathan VanBenschoten]

I'll throw my vote strongly behind what seems to be the growing consensus as well. I stumbled upon this yesterday and it left a bad taste in my mouth. I tried to open the web UI and was confused to find that I needed permissions. After a bit of searching, I plugged them in to find that the login didn't even work - the credentials were accepted but I was continually redirected to the login page. I can't seem to reproduce so maybe something was fixed here recently, but that doesn't really matter. It was all just too much friction for what is meant to be the easiest way to get people playing with CRDB and that undermines the entire point of the tool.

Like Andrei said above, it's a great idea to extend cockroach demo to become a learning tool for roles/users/privs/other security concerns, but that doesn't mean that needs to be the default, especially if making it the default comes with severe downsides.

Maybe there's a middleground. Could we enable some of the features that are typically disabled in insecure mode (roles, privs, etc.) in the insecure cockroach demo mode?

To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/CACJj5QmwD6baB%3DgbXkoEqM-wpriLBR3SWxQ0q7A2_aJyoxq2vA%40mail.gmail.com.

[Amruta Ranade]

As Marc’s residual echo in security-related matters, I need to warn against enabling IAM features (roles and privileges) in the insecure mode because he believed that it gives people the false sense of security about their clusters when in reality, the clusters aren’t really secure without TLS certs stuff. 

FWIW,  I like the idea of using `./cockroach demo --insecure=false` to learn about CRDB IAM features. 

To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/CAPucRz7y1BPjw9DPzmPNX1RvcNM9d8phYrzxOTTVOMCdeOjESw%40mail.gmail.com.

[Jordan Lewis]

On Wed, Apr 1, 2020 at 9:47 PM Amruta Ranade <amr...@cockroachlabs.com> wrote:

As Marc’s residual echo in security-related matters, I need to warn against enabling IAM features (roles and privileges) in the insecure mode because he believed that it gives people the false sense of security about their clusters when in reality, the clusters aren’t really secure without TLS certs stuff. 

This is a fair point. But, in my opinion, cockroach demo shouldn't really need to be cert-enabled or especially secure, and I doubt our users expect it to be, because it's explicitly an ephemeral cluster that leaves no trace behind when you exit it. It's explicitly designed to be a demo to let users get an "as-quick-as-possible" basic CockroachDB experience, from the CLI and Web UI.

Either way, I would still be happy to only allow users and roles and so on given --insecure=false.

[Amruta Ranade]

Oh, I totally agree that cockroach demo need not have `secure` as the default mode. Our security UX is painful for even motivated users; I wouldn't blame casual users just wanting to try out CRDB to get frustrated and give up if we make them deal with certs and stuff. At the most, I like the idea of having `--insecure=false` (or `--secure=true`) for educational purposes.

[Raphael 'kena' Poss]

friends,

first of all this is a product security discussion, and not "just" a product discussion, so please cc at least ben and aaron.

then, this conversation makes it painfully clear how convenience trumps security if users get a say.

The problem with today's `cockroach demo` is that it silently creates a networked service on the user's device that any concurrent app can use without restriction.

Zoom used to do something similar "for convenience" on macOs and got pilloried by the security community until they stopped doing it.

Their reputation was severely affected when users realized what was going on.

When Apple kicked Flash out and Google locked down Android permissions, there were plenty of users complaining "my favorite app doesn't work any more".

What lesson is learned here? There were _many_ more malware and adware authors who were thwarted in making the devices dangerous to their users. Apple and Google made the right choice.

There was another conversation on Slack this week that you may have missed.

The truth is that `demo` should never have let become so large as it now is. It's not just a single node cluster with a SQL shell any more.

It has:

- an admin UI

- all the RPC endpoints

- all the workloads

- geo-distributed latency simulator

- soon, if some folk get their say, the ability to upload files (via the insecure network listener) to the user's data directory (nodelocal upload)

It's not a simple shell any more. It has become a product on its own.

As such it needs proper product direction and a set of security guidelines.

If the security decision here is "the tool makes the user's less secure for themselves" (that is the true meaning of "insecure mode"), this needs to be advertised upfront to users much more loudly than it is today.

And we'd need to start discouraging users from using it in our docs and start researching proper alternatives.

-- 
Raphael 'kena' Poss

[Raphael 'kena' Poss]

There's also an elephant in the room:

- YOU cockroachdb developers should have no issue creating a shell alias that aliases `cockroach demo` to `cockroach demo --insecure` (THIS WORKS TODAY)

- 99% of users exposed to 'cockroach demo' will only use it to enter some SQL and try out the admin UI. That user journey works PERFECTLY now.

-- 
Raphael 'kena' Poss

[Raphael 'kena' Poss]

On 02-04-2020 13:42, Raphael 'kena' Poss wrote:

- YOU cockroachdb developers should have no issue creating a shell alias that aliases `cockroach demo` to `cockroach demo --insecure` (THIS WORKS TODAY)

or better yet put COCKROACH_INSECURE=1 in your shell config.

This works already like it always has.

-- 
Raphael 'kena' Poss

[Raphael 'kena' Poss]

After my other response which was basically reactive I want to move this forward.

The situation we're facing here is that you guys need a test environment that's more powerful than what most users need, _and_ also you are very mindful of the power of the tools you have.

- For you (and me included) it's 100% ok to build a "test" shell using something called "TestServerInterface", with zero security guarantees. It's a hack, and it's OK because it's only ever used in the context of a crdb development environment.

- For our users (and me included, as a user), it is 0% ok to ship a product based off something called "TestServerInterface", with zero security guarantees. A hack is not a product, certainly not a product we want to show to people whom we ask X.000.000$ from.

Also I'll take the opportunity to point out there are things we want to put in the hands of our users which I, as crdb developer, definitely do NOT want to find in my hands; including:

- automatic license acquisition

- enabled telemetry

- automatic SQL "safe mode" enabled

- all manners of informational notices

What I think is becoming glaringly obvious here is that there are two user groups with separate user journeys.

I'd support something like this:

1. a 'cockroach demo' shell that uses TLS auth between nodes, password auth for its SQL connections (i.e. TLS-less but WITH authentication) and password auth (also without TLS) for its UI. That would be built using something else than TestServerInterface (a combination of the current `start-single-node` and `sql` code, in fact). It'd be as close to the real thing as needed for bottom-up adoption by future customers.

(Conversely, if a user would find such configuration, without TLS but WITH auth, too difficult to use, that would be a clear signal that the "real thing" is not sellable. We'd then have good evidence to move forward in making the "real thing" better and more simple to use in secure mode.)

2. a 'cockroach dev' shell that's whatever you want it to be, but would not be built in the production binaries.

-- 
Raphael 'kena' Poss

[David Taylor]

I don’t think we should split off `dev` vs `demo` — I think using things like our users is part of how we feel their pain and fix things. I think the fact we all use `—insecure` and then tell users not to is a big blindspot (and is the reason why we missed things like admin ui endpoint controls).

I think demo should be secure by default, and I think _we internally in development_ should be using secure by default as well all the time: in demo, in roachprod, in local nodes, etc.

The backlash here, and the fact we all use —insecure, is a strong signal that secure has too much friction, but to me the answer isn’t to give us something different to shield us from that friction, but to work on reducing it for everyone. In this case, if the issue is that the admin UI is requiring auth, maybe we can put a token in the printed URI that logs you in directly (e.g. printing “Admin UI URL (automatically logged in as root)” or whatever to make it clear that that is what it is doing)? Or maybe `demo` could use a secret in the process owner’s home dir, to allow signed cookies to persist across restarts so we’d stay logged in? I donno.

--
You received this message because you are subscribed to the Google Groups "CockroachDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroach-db...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/f8fdd515-8181-a084-2aaf-f833a750b8cc%40cockroachlabs.com.

[Raphael 'kena' Poss]

On 02-04-2020 16:55, David Taylor wrote:

maybe we can put a token in the printed URI that logs you in directly (e.g. printing “Admin UI URL (automatically logged in as root)”

ooh I like that! that's doable.

Also heads up that Rohan has a PR in the pipeline to do exactly that with the SQL URLs too.

--

Raphael 'kena' Poss

[Andrew Werner]

I can see both sides of the argument for the default. What I know for sure is that connecting to a secure demo cluster from a separate session is painfully difficult. I love the suggestion of generating admin URLs with tokens. Can we further improve the usability of the secure demo cluster so that it doesn't get Andrei so worked up? Here's some more suggestions:

1) Add a builtin or something like it to generate a certs dir for easy use? Right now they go in a tmp dir with a random name

2) Add guidance in the initial login about how to do that and how to get pgurls to the various nodes? Right now you have to pull it out of crdb_internal.gossip nodes or something like that.

--
You received this message because you are subscribed to the Google Groups "CockroachDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroach-db...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/90b31a56-6ae9-d13f-14e3-35d5ba8bb710%40cockroachlabs.com.

[Andrei Matei]

I feel like important points are not being addressed. cockroach demo is just listening on the localhost interface. Does this still make it a "networked service"? The password for the admin ui is well known. The client certs for connecting to it are in a /tmp dir (with permissions only for the current user).
What exactly is the threat that we're worried about? That there's already malware on the machine, but running under a different user than the one running cockroach demo (and also not root), and that malware would then use a SQL or RPC connection to a cockroach demo server to take advantage of a CRDB vulnerability into expanding the malware's privileges to the ones of the user running the demo? Is that it?

For the UI, if the password is well known, what exactly is the point of that password? I guess it's just there because we don't allow root to have no password?

Here's something that I ran into yesterday: running explain analyze (debug) <foo> prints out a URL. If you go to it without having previously logged into the cluster, you get an error. For a cockroach demo cluster, you probably have not previously logged into it.

And there's the case of connecting to other nodes in a demo --node=3 cluster. Even if I had a good way to access the client certs (or generate new ones) I don't really want to mess with certs at all. If I want to connect another app, it always takes me a long time to figure out how to even get that app to use the certs. As far as I understand the thread model here, I don't think it's worth it.

Perhaps if we'd support unix domain sockets for SQL connections (do we already?), I'd be fine with making demo just listen on domain sockets for SQL instead of localhost. Assuming your average client driver works over these sockets.

- a_m

[Andrei Matei]

I just went back through the demo and realized that I misunderstood that we are indeed already binding to localhost for the current insecure demo instructions. (So that doesn't represent a change from what we are already doing). Apologies for the confusion all.

Well, so given that it's just localhost, where do you stand on the issue?

- a_m

 

Regarding local unix sockets. I'm fairly certain we already support this in 19.2 (at least) but Raphael can validate this.

It sounds like the biggest complaint is that certificates are hard and a new user shouldn't have to fight with them. This is actively being addressed on the 20.2 roadmap where we will separate out different certificate usages so that the database can default to secure mode and behave near identically from a user standpoint to what today we have as insecure mode.

Related and to what David Taylor raised, I've heard estimates that as high as 60% of our paying on-prem customers deploy in insecure mode because the certificate user experience is so bad. This is on our radar and we will make this better!

Aaron

[Raphael 'kena' Poss]

On 02-04-2020 18:10, Andrei Matei wrote:

I just went back through the demo and realized that I misunderstood that we are indeed already binding to localhost for the current insecure demo instructions. (So that doesn't represent a change from what we are already doing). Apologies for the confusion all.

Well, so given that it's just localhost, where do you stand on the issue?

localhost + non-authenticated is what Zoom was doing and what got them in trouble.

The vulnerability is relative to other apps on the same system. They all have access to the localhost listeners.

Consider that a new user of a database is likely to enter data/values in their test database that are valuable to them: contacts, e-mail addresses, maybe passwords. If you allow concurrent apps to scrape things without friction, that data can make its way anywhere. That's the kind of risk I'm thinking about.

A small layer of authentication (even with a known password) reduces this risk. With a fixed password you're only vulnerable to malware designed specifically for cockroachdb.

The situation would be all-around better by providing the auth token in the URL, like suggested elsewhere. The SQL URLs will get their cert details embedded via https://github.com/cockroachdb/cockroach/pull/46913  thanks to Rohan.

If we combine that PR with Andrew's suggestion to place the certs in a place that more nicely named (e.g. ~/.cockroach_certs) we can get a lot of mileage this way.

-- 
Raphael 'kena' Poss

[Andrei Matei]

On Thu, Apr 2, 2020 at 12:30 PM Raphael 'kena' Poss <k...@cockroachlabs.com> wrote:

On 02-04-2020 18:10, Andrei Matei wrote:

I just went back through the demo and realized that I misunderstood that we are indeed already binding to localhost for the current insecure demo instructions. (So that doesn't represent a change from what we are already doing). Apologies for the confusion all.

Well, so given that it's just localhost, where do you stand on the issue?

localhost + non-authenticated is what Zoom was doing and what got them in trouble.

The vulnerability is relative to other apps on the same system. They all have access to the localhost listeners.

Consider that a new user of a database is likely to enter data/values in their test database that are valuable to them: contacts, e-mail addresses, maybe passwords. If you allow concurrent apps to scrape things without friction, that data can make its way anywhere. That's the kind of risk I'm thinking about.

A small layer of authentication (even with a known password) reduces this risk. With a fixed password you're only vulnerable to malware designed specifically for cockroachdb.

As opposed to what other type of malware? We're talking about the admin UI here, so what kind of generic malware that scans random ports on the machine for http servers and exploits vulnerabilities that the CRDB admin UI has is there?

You don't seem to be phased by the fact that the client certs are there for everybody to use. It's not like we're handing out certs from Consul or anything; the certs are right there on the owned machine. So can you please qualify even more than you did the risk that you're mitigating? Is it strictly about apps running on the machine but as other users? So I'm sharing my macbook with a p0wned nitwit who's logged in at the same time as I am, and they're running cockroach-specific malware and so now the secrets that I've entered into the demo get hacked?

I want to force a very explicit definition of the threat, to structure my own thinking.

[David Taylor]

I think the zoom comparison is poor and doesn’t serve as a particularly instructive example for us. What got them in trouble as a) starting a network service in a something that users didn’t expect to be a network service (a video chat client) which was b) for the purpose of circumventing an intentional security measure to prevent browser sandbox escapes and then, most importantly, c) including in that service the ability to execute arbitrary commands and even to download and execute arbitrary binaries (i.e. an RCE, or, honestly, a straight up backdoor). 

We don’t tick those boxes — yes, we’re starting a network service, but that is because CockroachDB is a network service. As long as `demo` is clear about what it is — a command to start one or more cockroachdb nodes (which are network services!) and then connect a SQL shell to them — and it does what it says on the tin, then we're not in the same boat as Zoom — we’re not sneaking around doing something other than what we said we would, and we’re not doing it for all the wrong reasons and then intentionally doing it the worst possible way.

That said, I still think we _should_ be running secure by default though, if not for the security of the user’s local machine so much as because the point of demo, and any other time we fire up nodes or clusters to try things, is to be representative of what the to expect, so the closer to the real thing it is, the better. Aside: IMO the only choice you should make at startup of a new node is if you want to provide your own certs or let the cluster configure TLS itself -- `—insecure` should be forgotten, relegated to some obscure flag for a raspberry pi which don’t have hardware AES or something and we should be use and experience the system the same way users in production do.

Now the one thing I should have stated earlier, when I said I’d prefer to fix the friction with secure rather than keep insecure or worse, a new, separate special thing, is that for 20.1, the time to fix that friction has likely gone so that option may be off the table for now. Given the choices of demo being secure with friction or staying insecure, I guess I’d pick stick staying with insecure for 20.1, and then make the change to secure on master and have all of 20.2 development for us to use it, feel the rough edges and and polish them down before pushing it on end users.

[Irfan Sharif]

(Might be totally off base here but) Out of curiosity, have we looked at what Jupyter does? https://jupyter-notebook.readthedocs.io/

Running a local jupyter notebook generates an auth-token in the URL, which I think caches something in the browser so that going forward you can simply browse to localhost:port/ and not sweat the details. They also have the same threat model we seem to be concerned about (probably worse since you can actually execute code in jupyter servers), and it's a pretty frictionless experience running a local + "demo" + secure jupyter notebook. I do believe they're using certs and what not, so perhaps we could look towards what they do for prior art.

--
You received this message because you are subscribed to the Google Groups "CockroachDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroach-db...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/d67a184e-2753-a44c-ccdf-3aacb2409937%40cockroachlabs.com.

--

Irfan Sharif, Technical Of Member Staff

Cockroach Labs, Inc.

[Aaron Blum]

Thanks Andrei,

I just went back through the demo and realized that I misunderstood that we are indeed already binding to localhost for the current insecure demo instructions. (So that doesn't represent a change from what we are already doing). Apologies for the confusion all.

Regarding local unix sockets. I'm fairly certain we already support this in 19.2 (at least) but Raphael can validate this.

It sounds like the biggest complaint is that certificates are hard and a new user shouldn't have to fight with them. This is actively being addressed on the 20.2 roadmap where we will separate out different certificate usages so that the database can default to secure mode and behave near identically from a user standpoint to what today we have as insecure mode.

Related and to what David Taylor raised, I've heard estimates that as high as 60% of our paying on-prem customers deploy in insecure mode because the certificate user experience is so bad. This is on our radar and we will make this better!

Aaron

On Thu, Apr 2, 2020 at 11:48 AM Andrei Matei <and...@cockroachlabs.com> wrote:

[Aaron Blum]

Well, so given that it's just localhost, where do you stand on the issue?

David beat me to the send button.

That said, I still think we _should_ be running secure by default though, if not for the security of the user’s local machine so much as because the point of demo, and any other time we fire up nodes or clusters to try things, is to be representative of what the to expect, so the closer to the real thing it is, the better. Aside: IMO the only choice you should make at startup of a new node is if you want to provide your own certs or let the cluster configure TLS itself -- `—insecure` should be forgotten, relegated to some obscure flag for a raspberry pi which don’t have hardware AES or something and we should be use and experience the system the same way users in production do.

I also think we should dogfood secure mode during the 20.2 development process so we can experience, identify, and FIX the user experience here. As long as we internally dodge the usability nightmare around "Secure Mode" we will continue to underinvest in its UX.

On an aside: After speaking with some of the SE's and TAMs I am awed by how hard customer enterprise security teams have worked to find ways to securely deploy CRDB - just so they can use our product. One customer designed an entirely novel certificate orchestration workflow just to be able to run CRDB after getting multiple security exceptions from their executive team. There is no other product I know of that customers would be willing to endure such a terrible security user experience from and we should not become complacent because some of ours have.

Aaron

[Aaron Blum]

Thanks for looping me in Raphael!

Ok. I can see there's been a fair bit of conversation already so I'd like to summarize what I'm seeing now to make sure I understand.

1. Cockroach Demo has been switched from defaulting to insecure mode to defaulting to secure mode. https://github.com/cockroachdb/cockroach/pull/45727
2. This change created confusion and friction for existing users (specifically the engineers here at CRL).
3. We want need the most streamlined demo experience to enable new developers to flock to our product and make it the database of choice.
4. We should not make a user less secure for running our demo than their machine was without it.

Please let me know if you disagree with my summary or characterization. :)

Now my thoughts based on what I think I've read here:

I think the most important statement in this thread is captured very well by the sentence:

Security is the last thing I care about.

Yes, I'm taking that out of context, but it's also worth noting that I'm not doing this to shame anyone here. It's a fact of system and development life that engineers and developers don't want to have to wrestle with security. It's hard, it's confusing, and generally even if you think you got it right, it usually isn't really (unless you abandon doing any work whatsoever). Moreover, there's enough engineering work to do without having to wrestle with seemingly arbitrary security constraints and steps that as a developer you are certain, don't apply to you!

I don't think I could change that mindset even if I wanted to. BUT what we can do is work together to make it so that security is part of that out of the box experience for the engineers here at CRL and the developers we will have swarming to our product so that they don't have to think about it.

So where does that put us today? I like Raphael's (1) and (2) above as they sound to me like they strike a nice balance.

Per Raphael's (1), it has been rare in my experience that I've tested a network listening application that didn't at least have me create a default login of some sort. That's not saying we have to conform to this (and it's an interesting engineering challenge to present that securely) but I would quibble with objections that setting a password for a demo app block adoption.

Per Raphael's (2), I've commonly seen across industry the use of "internal only" builds of products akin to the "dev" shell mentioned. These usually include features (or lack of features) specifically tuned for development and testing. These often include hard coded open license checks, abridged security controls, and far more verbose logging options for dev/test work.

Re: Listening Network Services

Regarding concerns about making the user environment less secure by running our demo. One of the fundamental assumptions I've heard repeatedly from our engineers is that "root" access to the database means you are also trusted as an administrator of the running system. We can discuss the merits of that assumption on another thread, but it is built into the way we currently handle and enforce permissions within CRDB.

However, if our target dev adopter is a coffee shop user, and we build the demo in such a way that it binds to an external network port without any auth while running in insecure mode, then because of our assumption about privileges AND our choice to deploy the demo in this way, we are exposing this coffee shop user to a very real risk of unauthenticated administrative control of their whole system. This is something I'm certain they would not expect, and not something they would want to have to think about when spinning up Cockroach Demo.

Per my 4. above, let's figure out how to make this a local bind for demoing regardless of whether we decide to also request login credentials for these interfaces.

All, please feel free to explicitly @ me on these things. I think our approach here towards security should be risk oriented and pragmatic. If you'd like to know more about how we can do that, please join today's Weekly Engineering meeting where I'll be sharing more of how I hope we can go about it.

Aaron

[Ben Darnell]

On Thu, Apr 2, 2020 at 10:53 AM Aaron Blum <aa...@cockroachlabs.com> wrote:

Regarding concerns about making the user environment less secure by running our demo. One of the fundamental assumptions I've heard repeatedly from our engineers is that "root" access to the database means you are also trusted as an administrator of the running system. We can discuss the merits of that assumption on another thread, but it is built into the way we currently handle and enforce permissions within CRDB.

Yikes, this is definitely not true - that's why `--external-io-dir` is off by default (we don't enable it for `demo`, do we? that would be bad. Maybe we should disallow `--external-io-dir` completely in insecure mode)

-Ben

[Raphael 'kena' Poss]

On 02-04-2020 18:59, Andrei Matei wrote:

You don't seem to be phased by the fact that the client certs are there for everybody to use. It's not like we're handing out certs from Consul or anything; the certs are right there on the owned machine. So can you please qualify even more than you did the risk that you're mitigating?

So here's the thing: file-based isolation between apps is relatively good in the unix world.

Permissions have some issues, but your OS is good at preventing access to files when the ownership/perms don't match.

That's what's used in Android and macOS: different apps run "under" different system-level principals and can't blindly access each others _files_ without some disclaimer (or permission requests) upfront.

That's not true of networking though. An app running as user X can connect to the network services of user Y on the same machine. It's the responsibility of the network service to enforce authn/authz. TCP local listeners are "free for all" and public to every app on the local machine.

(Yes I know there is some nuance here but as always - unless we are very careful at spelling out that nuance in our docs / warnings, it's not good to expose our users to an element of surprise.)

-- 
Raphael 'kena' Poss

[Amruta Ranade]

I would caution against relying on docs/warnings to communicate security nuances. We already have a significantly complex UX that users have to piece together across the doc set. Adding nuanced warnings is more of a cover-all-bases strategy than one that can guarantee customer success.

--
You received this message because you are subscribed to the Google Groups "CockroachDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroach-db...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cockroach-db/9bbbcee5-b813-1967-bfa9-4c81745ab570%40cockroachlabs.com.

[Andrei Matei]

On Thu, Apr 2, 2020 at 1:14 PM Aaron Blum <aa...@cockroachlabs.com> wrote:

Well, so given that it's just localhost, where do you stand on the issue?

David beat me to the send button.

That said, I still think we _should_ be running secure by default though, if not for the security of the user’s local machine so much as because the point of demo, and any other time we fire up nodes or clusters to try things, is to be representative of what the to expect, so the closer to the real thing it is, the better. Aside: IMO the only choice you should make at startup of a new node is if you want to provide your own certs or let the cluster configure TLS itself -- `—insecure` should be forgotten, relegated to some obscure flag for a raspberry pi which don’t have hardware AES or something and we should be use and experience the system the same way users in production do.

I also think we should dogfood secure mode during the 20.2 development process so we can experience, identify, and FIX the user experience here. As long as we internally dodge the usability nightmare around "Secure Mode" we will continue to underinvest in its UX.

So for Rafa this seems to be about security. For Aaron and David, it seems to be about dogfooding secure clusters.

My opinion on dogfooding is that the point of cockroach demo is not to be a place to dogfood production experience. As a "demo", it's supposed to put things in the users' arms in a trivial way, and also to put our best foot forward. Not in a disingenuous way, but definitely by removing all the roadblocks that can be removed. I don't think that production considerations such as certificates should be part of it (or at least not by default). I think they can only harm the purpose of cockroach demo.

As far as us, developers, being driven to improve the usability of certs, or such, by having demo use them, I think that's not going to happen. We'll just all use --insecure (or set that env var Rafa mentioned); I know I will. For people in the dedicated security team, I can see why you wouldn't want to use --insecure.

- a_m

[Andy Woods]

If we plan to improve this experience in 20.2...why not wait to make demo work like this in 20.2? 

--

Andy Woods

Senior Product Manager, Cockroach Labs

an...@cockroachlabs.com

[Raphael 'kena' Poss]

On 02-04-2020 20:51, Andrei Matei wrote:

My opinion on dogfooding is that the point of cockroach demo is not to be a place to dogfood production experience.

I think this thread has revealed there is a diversity of opinions on this particular point. Your opinion (and thus UX requirements) is valid; but the other opinion is valid too.

Let me reply to the rest of this message assuming that the default was indeed switched back to insecure in 20.1

As a "demo", it's supposed to [...] remov[e] all the roadblocks that can be removed. I don't think that production considerations such as certificates should be part of it

Agreed. At least there shouldn't be anything for the user to even think about -- even if they request the secure mode.

(That was top and foremost reason to remove TLS for the admin UI, regardless of whether secure mode is enabled or not.)

As we were discussing this thread, the following already happened:

1) the SQL URL printed out by the SQL shell when the secure mode is requested now embeds the cert details, so that you can point your concurrent SQL clients to the thing and get proper TLS auth.

2) I'm investigating how to skip certs entirely by doing the thing with the unix socket instead (all client apps support that because it's standard with pg)

As far as us, developers, being driven to improve the usability of certs, or such, by having demo use them, I think that's not going to happen.

Given that you've not been earmarked to solve the UX problem, your position on this is OK. :-)

Meanwhile there will be cert improvements throughout the 20.2 release cycle. It's important for folk to provide continuous feedback on those improvements, so that we can tweak them towards the best UX possible. Therefore we do have a collective incentive to have at least some % of our team trying it out regularly.

Finally: my opinion is swayed by David's argument that we should proceed more conservatively for the 20.1 release regardless of our security roadmap, and thus I am now moderately in favor of flipping the default for 20.1.

Just for 20.1 though -- we do need to dogfood, and there's this env var to move forward until we fix the brokenness.

-- 
Raphael 'kena' Poss