has anyone successfully implemented keywhiz outside of Square?

[Mike Atlas]

Hi guys,

Warning. Grumble-grumble whiney post ahead. 

Has anyone successfully implemented keywhiz outside of Square? Can you share your learnings, contribute to the "wiki"?

The keywhiz blog post and github page sold me hard that this is a great project, and with nearly 900 github stars, surely keywhiz must have traction and a knowledgeable userbase!

But ... there are no StackOverflow tagged posts, only 16 members on this mailing list, four threads, a starkly empty "wiki", and relatively-speaking, a sleepy issues tracker on the git repo.

After actually attempting to build the product from scratch, I was left... scratching my head repeatedly. I've hit several and more of the problems described in the 4 threads that exist here, overcame them on my own, but was still left feeling more in the dark about ... next steps in getting keywhiz up and running.

Despite being a seasoned engineer, I'm relatively green in experience w/r/t usage of modern (circa 2012ish+) Java based products, Maven, pom.xml, JCE, signed JARs, and basically everything this project presumes working knowledge to build, deploy, configure this project. I don't actively know a Java-oriented IDE inside-out like Eclipse, and prefer my command line tools anyways. Available Maven Central compiled JARs don't really help me either - what are the CLI options? Should I run things in daemons and with what kind of isolation? What kind of working knowledge should I assemble before I attempt to even touch this product? What database(s) is required locally for testing? How do I compile keywhiz.model to run it's test DB against a remote database? Where's the "config file"? Where are sample configurations? Where are some architecture diagrams?

I'm sorry for whining. I basically think that keywhiz is all fuss and no meat at the moment. I'm sad, because I want to use keywhiz, love and have a need for this kind of tool (apparently others do too) but the code and docs available are nowhere close to usable, nor is anything else well-described. I'm going to backburner this as a potential tool and hope that the friction to adopt and deploy this tool is far better in 6-12 months. 

For what it's worth, this is the same sentiment I had when I first poked at Docker in it's early days. I'm optimistic for keywhiz, and wish the team good luck. See you in 6-12 months.

-Mike

[Mike Atlas]

So - nobody has adopted or deployed keywhiz successfully yet? 

I'm not joking here, I'd like to hear back if there are others out there and how much effort it took.

Thanks,

Mike

[Justin Cummins]

There are some teams adapting keywhiz to their environments, but I’d rather not speak publicly for them. I think usually takes longer to add infrastructure than pulling in other projects like a database or library, so the project is still really recently open-sourced. We’ve tried to consider what types of features or problems others may run in to, but I am certain there are many we have not thought of.

We will continue to keep working on softening edges, adding functionality, and documenting areas of knowledge, starting with questions people bring up. As much as possible, we try to not lock people to more infrastructure decisions than necessary while also trying to provide something work-able to start from. A great example is that Keywhiz requires some sort of PKI system. PKI can be crazy complex and many organizations have something in place, all wildly different. We suggest certstrap as somewhere to start that’s as easy as we can think ok.

Thanks for writing your email. It’s really useful information on difficult spots. I’ll try to touch on the specifics you mentioned for you and posterity.

What are the CLI options?

The CLI options are best viewed by getting the usage. If invoked as keywhiz.cli --help, it will output them all. There’s a more detailed wiki page for it with some examples.

Should I run things in daemons and with what kind of isolation?

I assume you’re referring to Keywhiz server. Keywhiz server starts with an embedded jetty server and the process will not exit until shutdown. I definitely recommend service supervision and we use runit. For KeywhizFs we use monit and other monitoring to ensure mountpoints are working.

What kind of working knowledge should I assemble before I attempt to even touch this product?

This is hard for me to enumerate, but I’d say rudimentary knowledge of certificates and TLS, using the mount command and some unix basics around fstab, DevOps knowledge around service deployment and databases, and probably some modern Java.

What database(s) is required locally for testing?

Keywhiz initially used PostgreSQL. We’re adding support for MySQL on master, and very soon the default with be H2 for development & test environments. H2 is an embedded, in-memory DB so no additional setup will be needed (as opposed to installing/setting-up PostgreSQL).

How do I compile keywhiz.model to run it’s test DB against a remote database?

All the relevant settings are in keywhiz-model’s pom.xml. I suppose it would just be changing the DB URLs and information you need for authentication. I wouldn’t typically use a remote database for this. The coming changes to supporting more database types will make the relevant settings more clear and easier to change by using maven profiles.

Where’s the “config file”?

When Keywhiz server is invoked, it requires a path to a configuration file as a command-line argument. This is is a DropWizard configuration with some Keywhiz-specific settings.

Where are sample configurations?

keywhiz-development.yaml is a sample configuration. All of the secret values and keystores should be changed for a staging and production deployment.

Where are some architecture diagrams?

Here’s an overview:

+---------------+                             +----------------+                     
|KeywhizFs mount+--+                       +> |Keywhiz server 1+---+                 
+---------------+  |                       |  +----------------+   |                 
                   |                       |                       |   +------------+
+---------------+  |                       |  +----------------+   |   |R/W database|
|KeywhizFs mount+-->                       +> |Keywhiz server 2+-----> |cluster     |
+---------------+  |  +-----------------+  |  +----------------+   |   +------------+
                   +> |TCP load balancer+--+                       |                 
+---------------+  |  +-----------------+  |          .            |   +------------+
|KeywhizFs mount+-->                       |          .            |   |R/O database|
+---------------+  |                       |          .            +-> |cluster     |
                   |                       |                       |   +------------+
+---------------+  |                       |  +----------------+   |                 
|KeywhizFs mount+--+                       +> |Keywhiz server 3+---+                 
+---------------+                             +----------------+

​

--
You received this message because you are subscribed to the Google Groups "keywhiz-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keywhiz-user...@googlegroups.com.
To post to this group, send email to keywhi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/ab73333f-7d2e-4468-8492-56886138f5b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carol Bloch]

Hi Mike,

I spent the last couple weeks trying to get a prototype working and was unsuccessful.  The lack of community support and documentation made this a really frustrating process.  Thanks to Justin for the help in my quest, but I am hanging it up for now.  We may check back in 6 months to see if it matures.  

Carol

[Tony Tarlano]

Mike,

I successfully compiled Keywhiz and started the CLI and server in about 2 hours of spare time spread across two days. I have attached my *rough* notes for you. I hope they help you along.

Cheers,

Tony

[dsom...@thoughtworks.com]

I'd like to bump this thread to see if anyone has this in production or production-like. I'm doing a conference talk on secret stores and am trying to gather information on the current crop. This one seems promising, but the docs are definitely rough. I've been mining the source code and have worked some things out, but there are details I'm yet to understand particularly around authentication of API requests.

I have gotten the dev server up and running easily and have it pretty well automated via chef. I have a reference app that I am using to test against various secret stores but haven't had a tremendous amount of luck so far working with keywhiz because of the auth issues.

CLI is working just fine though.

If anyone (including Square) wants to share a bit of information on the product, I'll work the details into my talk.

Daniel

[Terry Fundak]

This is a shoutout to Tony.

Hey Tony .... Totally awesome terminal dump. I hope you a real, not a single mistake in your terminal commits OR perhaps a perfect edit of your terminal commands - input and output - OR perhaps you did it a few times in the 2 hours.... Not sure, but I'm super impressed by the clarity of what amounts to the best tutorial walk thru I've seen in sometime. It is encouraging.....

Could you add one more command to it...

>>uname -a

This would help define the working environment better....

Thanx again,

Terry

[Alan Boxberger]

Similar experience. Was asked to set keywhiz up, but have run into many problems. It's really a timesink at the moment. I think it's a while away from being generally usable/configurable. Installation instructions really need work. Recommend that developers sit down and watch the installation experience by users and amend doco appropriately. Looks like a great product though and best of luck. Hope you get some resources to take to the next level.

[Alok Menghrajani]

What specific issues have you been running into? Matt has been very
responsive on github, whenever someone files an issue he responds to
it within a few hours.

If anyone in san francisco/bay area would like to physically sit down
together and go over installation/rough edges I'll be more than happy
to dedicate some time for it.

Finally, the website is part of the keywhiz repo, so pull requests are welcome.

Alok

> --
> You received this message because you are subscribed to the Google Groups
> "keywhiz-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to keywhiz-user...@googlegroups.com.
> To post to this group, send email to keywhi...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/keywhiz-users/296161c4-7f66-4857-aeb7-617cf19e32d4%40googlegroups.com.

[Mike Atlas]

@Alan - In the the interim, Vault from Hashicorp has gained a lot of traction. No knocking on keywhiz, but Vault looks stable, mature (traction wise), and usable. Disclosure - I haven't tried Vault myself yet, other priorities have still had precedent in my work, but I probably would try Vault next rather than approach keywhiz (despite Alok's generous offer for help).

https://vaultproject.io/intro/vs/keywhiz.html (for comparison)

https://groups.google.com/forum/#!forum/vault-tool (306 members, 255 threads)

https://github.com/hashicorp/vault/issues (very active)

https://webchat.freenode.net/ (#vault-tool) looks like it has roughly 40 people in the IRC channel - although these days I'm surprised they didn't go with Gitter, which is eating away at freenode channels.

-Mike

You received this message because you are subscribed to a topic in the Google Groups "keywhiz-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/keywhiz-users/aqwkEKLnkyE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to keywhiz-user...@googlegroups.com.

To post to this group, send email to keywhi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/CAEVqHxnyV2V%2B8RKFjFA8P22SfqJEZ-FVuGj2N4fu7RZ2dX_9VQ%40mail.gmail.com.

[m...@squareup.com]

I'm sorry to hear that.  Setting up Keywhiz shouldn't be hard, but I'm sure there's things missing in the documentation, and getting some things like certificates right may be tricky.

I'll find a volunteer who has never set up Keywhiz or seen it used and get them to go through a deploy, and hopefully we can audit where our documentation has blind spots.

I'm happy to write additional documentation, and to answer any questions you may have, including working with you to get keywhiz deployed where you need it.

Feel free to contact me personally for any and all questions, open github issues for any and all problems encountered, or send a note on this list.

My email is m...@squareup.com.

[Alan Boxberger]

Hi Alok

Thanks for getting back.

I guess from my point of view (I'm a sys admin), I'd just like to have a set of instructions that if followed would install the app.

However, there was a lot of wrangling, running the commands as listed, and trying to figure out why there were failures.

I've never used maven before and several of the initial build steps failed. I spent a good deal of time working through the error output trying to figure out what was going wrong. One example was that java.time couldn't be found. Again, I'm not a java programmer, and it took me a while to figure out that that package was only available in java 8. I don't think there was a list of that kind of pre-requisite. If I can think of any other errors I'll post them (in Ireland, so not at work now).

The next step involved running java -jar commands, but the jar files weren't where they were supposed to be (server/target, I think). I looked in the jar files download but could only find sync, rather than SNAPSHOT  jars.

The database created was different to one referenced in subsequent scripts (_development v _test).

Trying to find the references to correct in the middle of a busy day just caused me to give up.

I know that a lot of the above might be more straightforward to developers and people building packages, but I'm guessing a lot of the people implementing the product would be like me and need better doco.

Honestly, I'm not complaining. I think guys like you who are making their product freely available are the soul of the open source movement and I'm very grateful that you do. Nevertheless, at this stage, I need to be doing less work to wrangle an application to work for me.

Regards, Alan

--

Al

========================
http://www.ika.ie/organ-donor-card
<div id="gk-widget-boxingburger2">Play <a href="http://gameknot.com/">online chess</a></div><script src="http://gameknot.com/widget-profile.pl?u=boxingburger2&r=1&w=1&p=1&b=0&s=0&o=1" type="text/javascript"></script>

[Alan Boxberger]

Hi Mike.

Yep, I threw Vault on after giving Keywhiz another couple of hours this morning.

It was up and running in < 10 minutes. I didn't have time to have more than a cursory play with it, so I can't say if it'll do what my boss needs it for.

Again, no criticism of the Keywhiz product, but from a implementation PoV, the install and config needs to be as smooth as Vault's, or at least not feel like a debugging exercise.

Regards, Al

[Matthew McPherrin]

Alan,

Thanks for the feedback!  This isn't the first time somebody has mentioned problems with having older Java, so we should probably try to get a better error message for that.  It's definitely a blind spot for us, since we all have a consistent dev environment here at Square.

How did you download keywhiz?  Check out from github and build?

What version did you get?  What command did you use to compile with maven?

I'm worried you've gotten the v0.7.9-sync release, which wasn't intended as a general-purpose release, but as a bit of a hack to help us do a migration:  We recently switched our production deploy to new database servers, so we had a one-off release called v0.7.9-sync that made you configure two databases, and had a command to copy between them.

It was never merged to `master` on github, so I had made the possibly incorrect assumption that it wouldn't be used, but we didn't really communicate what we were doing with that release.

As for the _development and _test databases:  The _test is used to run unit tests against and should be considered non-persistent.  The keywhizdb_development one is the DB that keywhiz is configured to run against in the example server/src/main/resources/keywhiz-development.yaml configuration.  The keywhizdb_development database should be set up when you run the "migrate" command on the jar.  I can see how that would be unclear if you were unfamiliar with our use of FlywayDB to set up the databases.

If you ran all the commands in http://square.github.io/keywhiz/#setup there should be a keywhizdb_development database set up.

I have some questions about the environment you're running in.  I can add some prerequisites and example install scripts. Linux? What distro and version? What database would you like to use if you put this in production?  Does your dev environment match your production?  Are you interested in using Docker or similar to deploy?

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/CAK296eNnga7hS0C%2B7-1ZgVDmijeJiZwfpD9OB-rYmaGT-%2BmQ_w%40mail.gmail.com.

[Alan Boxberger]

Hey Matt!

Cloned from git using

git clone https://github.com/square/keywhiz.git && cd keywhiz

then ran 
mvn package -am -pl server -P mysql

0.7.9 looks familiar, but I'll confirm when back in the office on monday.

I was trying to build on an Ubuntu 14.04 VM (xen).

mysql is db of choice (I had set up a mariadb galera cluster).

It's all dev at the moment.

As for docker, management are looking at deploying our own company's app using docker, so it would definitely be something I'd consider.

Hope this helps. If you need any more info, I can look at it again on Monday. I can probably throw a little more time at it then, but not much.

If you want me to run through an install and flag things in more detail, happy to do so.

Nearly bedtime here, so have a good weekend and talk to you guys next week.

Al

[Matthew McPherrin]

Okay, great.  I'll post explicit instructions for Ubuntu in the near future, along with Fedora, CentOS, and Arch Linux - that covers what everyone around here uses, at least.

I've got a volunteer who has never set up Keywhiz before and we're going to run through that as an exercise in the next few days, and hopefully she can point out anything we're missing in the docs and get them a little more polished up.

I think the current instructions are a little more focused on setting up a development environment, versus a staging/test/production setup that a sysadmin like yourself would be interested in.  I'm going to keep that split in mind as we go through our docs again, too.

[Mike Atlas]

Here's my plus one vote for a Docker compose project linking to database images and compiled product, isolating all the dependencies and ready to run - at least for evaluation purposes. 

The only thing that makes me worried is that the encryption export licensing JCE wrinkle may legally stop you from being able to package and redistribute in such a manner as a Docker compose project. 

You received this message because you are subscribed to a topic in the Google Groups "keywhiz-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/keywhiz-users/aqwkEKLnkyE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to keywhiz-user...@googlegroups.com.

To post to this group, send email to keywhi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/CAGKXR4suF-oFD9_ucPciedt9iyPyrx50QYVgMuH-XcCWtD_j3A%40mail.gmail.com.

[Dave Daley]

The attached file has been tested a few time and will get you to the https://YOURHOST:4444 page.

[Gyula Wéber]

Hi guys,

I wanted to try out keywhiz, but none of the available documentation worked for me, except for the development environment. 

I think it's a very cool project, and it can be very popular if someone can ease trying it out. I've started something, which generates every password / cert / keys / etc..., and now it automatically / successfully builds on aws (I've used terraform just because I don't like to click).

https://github.com/gyulaweber/keywhiz-env

Let me know if anyone is interested in making it better, I think the first thing can be using it with docker-compose / mysql, and creating some mechanism to add / remove clients from the truststore.

Kind Regards,

Gyula

[Tuomo Salo]

It's not so much that there's things missing in the documentation, it's more about the documentation being completely non-existent in its entirety. The basic concept of Keywhiz is entirely awesome, which is why I didn't give up instantly after running into problems like "Really? I need to *manually* update the JCE policies?" and "The plugin compilation crashes on RHEL7? Wow!". At this point I'm afraid I'll have to call it quits though. It's mainly because of these two reasons:

1. There is no demo/evaluation guide. When I'm evaluating products, I really do not want to spend several days debugging the installation process for a product I may or may not want to use. Instead of resolving maven errors, I'd much rather see detailed and tested instructions on how to get a demo instance and a couple of demo clients up and running from scratch. Yes, really, from scratch, since that is exactly where every single person evaluating this project starts at. Some example sentences I'd like to see in the doc are: "Go to AWS (link here) and buy yourself a virtual machine like this (instructions here)." and "Run these commands in this order to get a working demo setup."

2. There is no user guide. What should things look like when everything is working perfectly? How do I perform the basic operations like adding secrets, clients and groups? Yeah, I finally figured out that the instructions in the Dockerfile were the correct way to go, and now I have a fancy (...yeah) UI that I have absolutely no idea what to do with. Yay me.

I strongly believe that adding these two documents would significantly increase the chance of somebody someday replying to this thread with "Yeah, we are using Keywhiz, and it's totally awesome!".

  -Tuomo