What is a true way to work with vault audit logs?

[S K]

Hello, everyone!
I'm trying to get familiar with vault and one of the biggest trouble I've meet is how to use audit logs. It's completely unclear to me what can I get from it. Most of my colleagues as the answer to this question, tell me something like this: "Yes, we write audit logs, but we never read it." 

So my question is: what is a true way to work with it? How do you analyze it? How do you know if one of your tokens was compromised for example? Can you share a link to the docs, articles or video from the events where somebody who knows, share his experience?

Or maybe I didn't get right the purpose of the audit logs in the Vault?

[Becca Petrin]

Hi S K,

Great question! I'm sure you've seen the Audit Devices page here.

Since the audit logs are JSON formatted, in my opinion, one great way to consume them would be using Filebeat with ElasticStack. There are other methods, I'm simply speaking of what I'm most familiar with. This would ultimately cause Vault's logs to end up in ElasticSearch which could enable them to be viewable by members of your team like engineering and support. They can then search Vault logs easily to trace requests and see who is performing what action within Vault.

Notably, Vault does also have an ElasticSearch database secrets engine if you wanted to be particularly careful with how people could access those (and any other application's) logs (which I would recommend). This is a shameless plug since I wrote that engine. :-)

This is only part of the monitoring story. There's a separate, metrics-related guide here as well.

I hope that helps.

-Becca

[S K]

Hi, Becca! Thanks for so quick answer!

But I still don't understand, how to use it. For example, let's say, that one of the our admin-tokens was compromised and somebody created another one root-token with it. So audit log will be like this:

{
  "time": "2019-08-17T21:53:07.625384189Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:0c97855631748ce0a775e3efc79fc607b0d2f61ddeb78b15e915a5087013fb5b",
    "accessor": "hmac-sha256:c081cc37603419f02e67fb93f2f1362aa0eb37fa42635606cc51b9b7ed1ed561",
    "display_name": "root",
    "policies": [
      "root"
    ],
    "token_policies": [
      "root"
    ],
    "token_type": "service"
  },
  "request": {
    "id": "f8b0f707-7e38-1410-4173-235ff9e250b6",
    "operation": "update",
    "client_token": "hmac-sha256:0c97855631748ce0a775e3efc79fc607b0d2f61ddeb78b15e915a5087013fb5b",
    "client_token_accessor": "hmac-sha256:c081cc37603419f02e67fb93f2f1362aa0eb37fa42635606cc51b9b7ed1ed561",
    "namespace": {
      "id": "root"
    },
    "path": "auth/token/create",
    "data": {
      "display_name": "hmac-sha256:0f235cb7061e26e25b346c787a036860e247e0e32181b8adf13850812a27a9f1",
      "entity_alias": "hmac-sha256:0f235cb7061e26e25b346c787a036860e247e0e32181b8adf13850812a27a9f1",
      "explicit_max_ttl": "hmac-sha256:3cf83aa363c8f73a7e23ccd56baa8f4e1119bc15800030f663f2d07c5420ce91",
      "num_uses": "hmac-sha256:943213e389eae841e8d03f94149bc8e564973fd4c6f0eabe76061dd4355b03b0",
      "period": "hmac-sha256:3cf83aa363c8f73a7e23ccd56baa8f4e1119bc15800030f663f2d07c5420ce91",
      "renewable": true,
      "ttl": "hmac-sha256:3cf83aa363c8f73a7e23ccd56baa8f4e1119bc15800030f663f2d07c5420ce91",
      "type": "hmac-sha256:792572c378bcb0b0400ad2033078e80334dfd06d76d948866960ad9b8547ba62"
    },
    "remote_address": "127.0.0.1"
  }
}
{
  "time": "2019-08-17T21:53:07.709275872Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:0c97855631748ce0a775e3efc79fc607b0d2f61ddeb78b15e915a5087013fb5b",
    "accessor": "hmac-sha256:c081cc37603419f02e67fb93f2f1362aa0eb37fa42635606cc51b9b7ed1ed561",
    "display_name": "root",
    "policies": [
      "root"
    ],
    "token_policies": [
      "root"
    ],
    "token_type": "service"
  },
  "request": {
    "id": "f8b0f707-7e38-1410-4173-235ff9e250b6",
    "operation": "update",
    "client_token": "hmac-sha256:0c97855631748ce0a775e3efc79fc607b0d2f61ddeb78b15e915a5087013fb5b",
    "client_token_accessor": "hmac-sha256:c081cc37603419f02e67fb93f2f1362aa0eb37fa42635606cc51b9b7ed1ed561",
    "namespace": {
      "id": "root"
    },
    "path": "auth/token/create",
    "data": {
      "display_name": "hmac-sha256:0f235cb7061e26e25b346c787a036860e247e0e32181b8adf13850812a27a9f1",
      "entity_alias": "hmac-sha256:0f235cb7061e26e25b346c787a036860e247e0e32181b8adf13850812a27a9f1",
      "explicit_max_ttl": "hmac-sha256:3cf83aa363c8f73a7e23ccd56baa8f4e1119bc15800030f663f2d07c5420ce91",
      "num_uses": "hmac-sha256:943213e389eae841e8d03f94149bc8e564973fd4c6f0eabe76061dd4355b03b0",
      "period": "hmac-sha256:3cf83aa363c8f73a7e23ccd56baa8f4e1119bc15800030f663f2d07c5420ce91",
      "renewable": true,
      "ttl": "hmac-sha256:3cf83aa363c8f73a7e23ccd56baa8f4e1119bc15800030f663f2d07c5420ce91",
      "type": "hmac-sha256:792572c378bcb0b0400ad2033078e80334dfd06d76d948866960ad9b8547ba62"
    },
    "remote_address": "127.0.0.1"
  },
  "response": {
    "auth": {
      "client_token": "hmac-sha256:fdb305fbabaf0044fc6d696fb2d0ff3d96574ff4d7fab804e8d5d36b7f2ddd14",
      "accessor": "hmac-sha256:19f3a70ceea337f067c053249504fbf8e8c164304b66a8c97fad421d43b5e4af",
      "display_name": "token",
      "policies": [
        "root"
      ],
      "token_policies": [
        "root"
      ],
      "token_type": "service"
    }
  }
}

And I will get an alert from Elastic Search, that new token was created. So what? How can I get an accessor of new compromised token to revoke it? Or what should I do in this case?

Or maybe there is exist any success stories about how Vault users faced some menaces and defended thmselves from it with correctly configured audit logs, alerting etc?

среда, 21 августа 2019 г., 20:16:48 UTC+3 пользователь Becca Petrin написал:

[Tob Ter]

By default Vault hashes most values in the audit logs. If you need the token_accessor in clear text you can set hmac_accessor = false in your audit log configuration, see the documentation in Beccas link.

Personally I store most values in the audit logs in the clear, of course except really sensitive information like tokens, private keys, passwords, etc. You can configure this with the parameters audit_non_hmac_request_keys and audit_non_hmac_response_keys of the secret engine you're interested in / working with.

As a last note, if an attacker is able to get on the Vault server itself (remote_address = localhost or the IP address of any Vault server), then you're pretty much hosed anyway, since all that's stopping him from taking over full control of the server is a local privilege escalation bug in Linux, then he will start attaching debuggers to Vault, dump the memory to get ahold of all kind of keys, manipulate the audit log stream and so on.

[S K]

Hi, Tob Ter! It is cool answer, thank you so much!

Did I get you right, you do not hash the accessors, you do hash only secrets?

Also, I have one more question: do you have something like accessors catalog? Or how do you keep records of tokens? For example, you created token for some service, you got it with it's accessor and other parameters, then you use it. But one year later this service will be halted. What will you do with token?

четверг, 22 августа 2019 г., 14:23:03 UTC+3 пользователь Tob Ter написал:

[Becca Petrin]

Hi!

I also want to add that - the primary use case for the Vault audit logs is more around compliance. It's not focused around alerting.

If what you're looking for is security alerting, we don't presently have a feature to that effect. In the present state, you'd want to build a sane security dashboard based on Vault metrics. It would also be good to monitor for things regarding Vault's environment, like if there's a server in front of Vault, you'd want to monitor for unusual traffic that may indicate a DDOS or brute force attack. You'd want to monitor the environment in which Vault is deployed for attackers getting onto the instance, like Tob mentioned. Vault is one piece of a larger security strategy.

We have a Production Hardening guide that explains how to take steps to prevent many attacks, and the Vault Reference Architecture has also been created with those concerns in mind.

-Becca

[Tob Ter]

Did I get you right, you do not hash the accessors, you do hash only secrets?

Yes. Of course I'm just one user and you'll have to decide yourself what information you want to have in the audit logs, how long you store them and who can potentially access those audit logs.

 

Also, I have one more question: do you have something like accessors catalog? Or how do you keep records of tokens? For example, you created token for some service, you got it with it's accessor and other parameters, then you use it. But one year later this service will be halted. What will you do with token?

You can list all currently issued tokens (referenced through their accessors) and see their attached policies, creation time, metadata and so on:

vault list auth/token/accessors

vault write auth/token/lookup-accessor accessor=<ID>

Of course you can also revoke such a token:

vault write auth/token/revoke-accessor accessor=<ID>

Regarding the lifecycle management of tokens: Vault encourages to use dynamic, restricted, short living secrets. If you are regularly issuing orphan (standalone) tokens with a lifetime of multiple years and maybe broad access rights, then you are just replacing static passwords with static tokens. For the service in your example, you could issue a periodic token with a lifetime of like only 3 days, but which can be renewed indefinitely. The service will then have to regularily renew its token by a simple API call. When the service is shutdown, no renewal will take place anymore and the token will automatically expire 3 days later.