Securing Secrets With HashiCorp Vault and Logz.io Security Analytics

Secrets, i.e. passwords, API keys, certificates, and any other type of credential used for digital authentication, have exploded in number and type. Even small-sized organizations might have thousands of SSH keys for example. 

Secrets are also a common security weakness often exploited by attackers. A recent study found that “89% of deployment scans show companies are not using Kubernetes’ secrets resources, with secrets wired in the open.” Because secrets are used for authentication and restricting access to sensitive data they have to be handled and managed extremely carefully. 

And that’s where tools like HashiCorp’s Vault come into the picture. Vault enables users to easily manage secrets across applications and the infrastructure they are deployed on, providing secure storage, revocation, renewal, encryption, and a long list of integrations with identity providers. Vault also enables users to generate audit logs that contain information on all the requests and responses that have been made to Vault. 

Providing visibility into who is accessing what and when, these audit logs can play a key role in a SOC. Users can collect these logs and ship them into a log management solution for further analysis. To be more proactive, however, a more sophisticated solution is required that can provide threat intelligence and alert when necessary.                                                       

In this article, I’ll show how to integrate Vault with Logz.io Security Analytics. You’ll learn how to install Vault, enable audit logs, ship them to Logz.io and use the provided rules to get alerted on Vault events.

Step 1: Installing Vault

If you’ve already got Vault installed, you can skip to the end of this section to see how to enable audit devices. If you haven’t, follow these steps to install and start the Vault dev server. 

Our first step is to download and install Vault’s executable. I’m installing Vault on an Ubuntu 16.04 EC2 instance but you’ll be able to find links to all the latest versions here:

wget
https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip 

It’s probably a good idea to download the checksum for the file but for the sake of keeping this simple we’ll proceed with unzipping the package:

unzip vault_*.zip

And the output:

Archive: vault_1.2.2_linux_amd64.zip
inflating: vault 

sudo cp vault /usr/local/bin/ 

We can now use the vault command. Verify the installation with:

vault --version

You should see the help output displayed as follows:

Vault v1.2.2

Our next step is to start the Vault server. For the sake of this tutorial, we will use the Vault dev server — a built-in, pre-configured server useful for playing around with Vault locally. 

Start Vault with:

cd /usr/local/bin
./vault server -dev 

You should see a bunch of information displayed in the output, starting with: 

=> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
                 Storage: inmem
                 Version: Vault v1.2.2

After this text, you’ll see some information required to continue on with the steps below, including the unseal key and root token.

Open a new tab in your terminal and as instructed in the output, export the Vault address:

export VAULT_ADDR='https://127.0.0.1:8200'

Then, save the provided unseal token somewhere and export the root token as follows:

Your next step is to enable audit logging by creating a new Audit Device. To do this, simply enter:

./vault audit enable file file_path=/var/log/vault/vault_audit.log log_raw=true

Note that we are using the log_raw flag in the command to remove hashing and facilitate analysis in Kibana.

That’s it! Vault will start to output audit logs to the configured file path.  Open the log file and take a look at the JSON entries:

sudo vim /var/log/vault/

{
	"time": "2019-09-02T09:37:39.711616033Z",
	"type": "response",
	"auth": {
		"client_token": "s.Ubtbu3BkWLROmnZ8lGwdAXKn",
		"accessor": "noCiri5z03Mja0daCI2xRvD4",
		"display_name": "root",
		"policies": ["root"],
		"token_policies": ["root"],
		"token_type": "service"
	},
	"request": {
		"id": "faeb518b-c403-e1b3-bf68-aab293f8a07c",
		"operation": "update",
		"client_token": "s.Ubtbu3BkWLROmnZ8lGwdAXKn",
		"client_token_accessor": "noCiri5z03Mja0daCI2xRvD4",
		"namespace": {
			"id": "root"
		},
		"path": "sys/audit/file",
		"data": {
			"description": "",
			"local": false,
			"options": {
				"file_path": "/home/ubuntu/Desktop/vault_audit_full_demo.log",
				"log_raw": "true"
			},
			"type": "file"
		},
		"remote_address": "127.0.0.1"
	},
	"response": {}
}

Step 2: Shipping Audit logs to Logz.io Security Analytics

To ship Vault’s audit logs to Logz.io, we’ll use Filebeat. If you haven’t already, follow the steps below to install it.

For secure shipping to Logz.io, we’ll start with downloading the public SSL certificate:

wget 
https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt && sudo mkdir -p /etc/pki/tls/certs && sudo mv COMODORSADomainValidationSecureServerCA.crt /etc/pki/tls/certs/

Then, you’ll need to add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - 

The next step is to add the repository definition to your system:

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

All that’s left to do is to update your repositories and install Filebeat:

sudo apt-get update && sudo apt-get install filebeat 

Next, open up the Filebeat configuration file at /etc/filebeat/filebeat.yml and use the following configuration:

filebeat.inputs:
- type: log
  paths:
    - /var/log/vault/vault_audit.log
  fields:
    token: 
    logzio_type: hashi_vault
  fields_under_root: true
  json.keys_under_root: true
  encoding: utf-8
  ignore_older: 3h

filebeat.registry.path: /var/lib/filebeat

processors:
- rename:
    fields:
     - from: "agent"
       to: "beat_agent"
    ignore_missing: true
- rename:
    fields:
     - from: "log.file.path"
       to: "source"
    ignore_missing: true
- rename:
    fields:
     - from: "type"
       to: "hashi_type"
    ignore_missing: true
- rename:
    fields:
     - from: "logzio_type"
       to: "type"
    ignore_missing: true
    
output:
  logstash:
    hosts: ["listener.logz.io:5015"]
    ssl:
      certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

A few things about this configuration are worth pointing out:

• Be sure to enter your Logz.io account token in the relevant placeholder. You can find this token in the Logz.io UI.
• The processors defined here are used to both comply with ECS (Elastic Common Schema) and perform some basic parsing for the fields in the logs. 
• The output section defines the Logz.io listener as the destination for the logs. Be sure to comment out any other output destination.

Save the file and start Filebeat with:

sudo service filebeat start 

Within a minute or two, you will begin to see Vault audit logs in Logz.io:

Step 3: Analyzing Vault audit logs

Vault’s audit logs contain a long list of fields that can be used for gaining visibility into requests and responses. Before you begin analyzing the data, it’s always a good best practice to become familiar with the data set. 

In Kibana’s Discover page, open a log message and try and understand the fields constructing it:

The hashi_type field, for example, can be used to understand whether the log is a request log or a response log. Another useful field is request.operation that tells you what type of operation was performed in Vault (e.g. delete, read, list, etc.). You can add these fields to Kibana to gain better visibility:

Next, you can begin building some visualizations to help you with monitoring these logs. The sky’s the limit — Kibana allows you to slice and dice your data in any way you like. Here is a simple example of a pie chart breaking down the different Vault request operations by type:

Step 4: Monitoring Vault events

As mentioned above, to be more proactive, regular log management is not enough. In a SOC collecting millions of log messages a day from the different layers of an application, Vault’s audit logs will most likely get lost within all of this noise. 

In this last section of the tutorial, I’d like to show how you can use Logz.io Security Analytics to monitor Vault events more proactively. 

First, let’s switch over to Logz.io Security Analytics:

The Summary dashboard provides us with an overview of our environment and we can see that a series of HashiCorp Vault events have taken place. These events are recorded when a correlation rule is triggered. 

Opening the Rules page, and searching for ‘vault’, we see a list of the available rules:

Each rule contains a set of conditions that if met, create an event and send off a notification to a defined endpoint. The rules also contain some useful information that will help you with next steps. 

This Multiple failed logins attempts rule, for example, creates an event if Vault’s audit logs record failed authentication attempts:

As defined in the rule, if the conditions are met five times or more within a timeframe of 5 minutes, a notification is sent via Slack or email. You can, of course, create your own rule if you like based on any query you enter in Kibana, or enable/disable/configure the existing rules as you see fit.

Vault Overview dashboard

Logz.io Security Analytics also bundles a pre-made monitoring dashboard for getting a birds-eye view of Vault-related events. The dashboard includes details on the alerts triggered, various breakdowns of unauthorized Vault operations, malicious IPs, geographic origins of requests made to vault, and plenty more. 

To use this dashboard, simply open the Dashboard page under the Research tab and select the dashboard from the list:

Endnotes

Vault’s auditing functionality gives security teams the ability to gain visibility into a crucial element of an application’s architecture but these teams will need a complementary solution to be able to actually make use of this data for security analysis. 

Collecting, storing and analyzing Vault’s audit logs is crucial but without being able to be automatically alerted when an event takes place, these teams will most likely not even notice something is amiss until it’s too late. 

Logz.io Security Analytics provides you with easy integration with Vault as well as analysis tools for security monitoring and a set of built-in rules for being alerted as soon as your secrets are compromised. You can learn more about Logz.io Security Analytics on our website.