The pace in which new releases of the Elastic Stack are being rolled out is breathtaking. Just two weeks after Elastic Stack 6.7 was announced, versions 7.0 of Elasticsearch, Kibana, Logstash and Beats have been released full of new features and improvements.

The rapid pace of development attests to both the engineering feat of the folks at Elastic and the strength of this community. I find it hard to come up with an open source project developing at this pace.

If you’re an old ELK hand, though, you might not be as enthusiastic. As much as Elastic Stack 7 is jam packed with goodies, it also includes a long list of changes that will affect the way you currently index data in Elasticsearch, parse it with Logstash and analyze it in Kibana.

I’ll let you be the judge of the matter. Let’s get started.

Kibana

I’ve decided to break my habit of starting my review with Elasticsearch because I simply love the new look and feel that Kibana has been given in version 7. Remembering version 4.x, it’s hard to believe it’s the same product!

Going dark

Ever since version 5.x and the introduction of Timelion and Visual Builder, Kibana has made efforts to close the gap with Grafana — its open source twin for visualizing metrics. From a visual perspective at least, it seems Kibana has finally closed that gap, applying the dark theme across the board and not just in dashboards.

This is a space-level setting (Spaces were introduced in Kibana 6.5) that can be turned on easily. Users, however, will no longer be able to select between light and dark per dashboard. I think we can live with that limitation, don’t you?

Improved querying and filtering

This might put off some Kibana power users out there but one of the main functions used in Kibana — querying, is totally different in version 7.

First and foremost, autocomplete and the new Kibana querying language (KQL), initially introduced in version 6.3, is now enabled by default (you can opt to use Lucene syntax if you like).  

Kibana 7 also ships with a new filter bar that makes filtering logs a much easier experience. Your filters now have dropdown menus and the ability to be managed together in bulk.

Kibana also boasts a new date picker that is much easier to use since it doesn’t force a query reload or push the whole page down.

Canvas new features

For most Kibana users, the good old pie/line/bar charts in Kibana are rich enough to visualize their operational data. Others, perhaps using Kibana for different use cases, can use Canvas to build amazingly-rich dashboards. If you haven’t played around with Canvas before, I highly recommend you do so now.

In version 7, Canvas was upgraded with new keyboard shortcuts and an expandable expression editor.

Elasticsearch

This new version includes a long list of new features that promise to improve Elasticsearch performance and resiliency as well as it usability (JSON logging!). I won’t cover ALL these changes but will try and focus on the more important ones.

Typeless APIs

The removal of types has been a very gradual change, and in version 7.0 APIs that accept types have been deprecated to be replaced with new “typeless” APIs. Support for _default_ mapping has also been removed. Don’t worry about your existing 6.x indices, _default_ mapping will only be rejected on new indices.

Easier installation

Getting started with Elasticsearch is much faster in version 7 because Java is now built into the package. So you don’t have to set up one of the supported JDKs anymore, and if you do have your own JDK you will have to define its execution path in the JAVA_HOME environment variable before starting Elasticsearch.

If you’re a Kubernetes user, there are some helm charts available for Elasticsearch and Kibana that will help you set up these two components of the stack in a Kubernetes-native manner. Keep in mind, though, that these charts are still in beta mode.

Real memory circuit breaker

Elasticsearch nodes faced with processing too many requests or “bad” requests can result in slow performance and in extreme scenarios, even bring down the node.

In previous versions of Elasticsearch, circuit breakers helped by calculating whether a given request can be served or not by the node. If the request could not be serviced, it was rejected or subsequently retried or executed on another node depending on the behavior users set up in their code.

Since it’s difficult, if not impossible, to accurately gauge exact memory usage, this mechanism proved to be insufficient. The new circuit breaker in Elasticsearch 7 remedies this by rejecting requests based on the actual memory being used by JVM and not estimations. Requests exceeding a configured memory usage threshold (default is 95% of the heap) are automatically rejected.

Improved search performance

Thanks to new algorithmic changes, Elasticsearch can now be configured to skip the ranking process for a configurable number of documents and display top results only. By skipping this process, Elasticsearch returns results much faster. The default setting is 10,000 but users can configure a different number that suits their search use case. As noted in Elastic’s release notes, this improvement does not apply for aggregation queries.

Other improvements for the search use case are a brand new Intervals query to mainly support legal and patents searches and a new script_score query for easier and simpler generation of ranking scores.

Cluster coordination

Elasticsearch 7 ships with a new cluster coordination layer, promising to make Elasticsearch more scalable and resilient and dramatically cutting the master election time from seconds to sub-seconds. Replacing the Zen Discovery system, the new coordination layer is easier to configure and makes scaling up and down safer.

In the fine print, you’ll find that when starting a new Elasticsearch cluster you will now need to define master-eligible nodes using the cluster.initial_master_nodes setting. Without this, Elasticsearch will fail to bootstrap (once the cluster has formed, this setting is no longer required and is ignored).

One shard policy

Elasticsearch default shard per index number is now 1 as opposed to the previous setting of 5. The idea behind this change is to help users avoid a situation in which they have too many shards per index, something that can affect Elasticsearch performance. Of course, you can set a different shard number if you like.

New Java REST client

One of the main reasons why Elasticsearch is so popular is its extensive API and the large number of clients that exist for different languages. Traditionally, Java developers used a TransportClient to connect to a cluster and execute APIs but this method suffered from security and dependency issues.

The Low Level REST Client released with version 5.x aimed to solve some of these issues but was just one step on the road to the newer High Level REST Client (HLRC) made available in version 7 which supports most of the Elasticsearch search, aggregation and administrative APIs.

Soft_delete enabled by default

Newly created Elasticsearch indices now have the soft_deletes setting enabled by default which makes them eligible for replication automatically.

To those new to Elasticsearch, version 6.5 introduced Cross Cluster Replication to enable easier replication of data across multiple Elasticsearch clusters. The soft_deletes setting helps maintain a history of document changes on the leader shards in an index to enable the replication of these changes to other clusters. The setting, required for Cross Cluster Replication, is now enabled by default for new indices.

Logstash

The big news for Logstash users in this release is the new and long-awaited Java execution engine, which is now enabled by default. Replacing the old Ruby execution engine, this puppy promises better performance, reduced memory usage, lower configuration startup and reload times, and overall — an entirely faster experience.

It will be interesting to see some proper benchmarking stats but this holds great promise for large Logstash-based ingestion pipelines. Quoting from Elastic’s release notes: “Some execution optimizations can be applied at this stage and the startup and reload times for some large configurations have been reduced by as much as 20x.” Run, Logstash, run!

There is quite a long list of breaking changes made to Logstash configuration options that users would do well to get acquainted with before upgrading. Take a look here.

Beats

Complying with ECS

The Elastic Common Schema introduced a few months ago aims at making it easier for users to correlate between data sources by sticking to a uniform field format. Sure, you can parse you data but what if all the fields are formatted the same way from the start?

Starting from version 7, all the beats will now conform to one standard format. This is good news but it also means that for those of us upgrading to the new version we will have to adjust our Kibana objects since field names have changed.

New AWS Module in Metricbeat

If you’re using Metricbeat to monitor your EC2 instances, you’ll be glad to learn that version 7 includes a new AWS module that negates the requirement of installing Metricbeat per host. Instead, you can configure one Metricbeat instance to pull metrics from CloudWatch into Elasticsearch. You will need to set IAM policies and include your AWS credentials in the new module configuration.

The only metricset supported right now is EC2 but I’m pretty sure we will see additional AWS metricsets soon. Anyone say Lambda?

New security modules in Filebeat

Filebeat supports new security use cases with new modules for Zeek (aka. Bro) – an open source network security monitoring tool, and Santa – a whitelisting/blacklisting system for macOS.

New system module in Auditbeat

Auditbeat collects and reports on user activity and processes in your system. The new system module added in version 7 collects host-specific data, such as information on running processes and the operating system.

Endnotes

The stack has received some major performance boosts on the backend as well as a facelift to the UI, so I would understand if our readers felt a sudden eagerness to upgrade. For me, believe it or not, the new Logstash execution is the main reason causing me to itch.

It will be interesting to see some real benchmarking stats coming out from the community testing these changes. We will be researching the new version to prepare for our eventual upgrade.

As usual, allow me to preach caution. Most ELK users will avoid upgrading to any version with an “0” in it simply because they’d rather wait to see if there are any issues with the new version that bring on fixes and it’s usually a best practice to wait a month or two before upgrading.

Last month, AWS announced an initiative called Open Distro for Elasticsearch. Since then, there has been a lot of debate as to what this move means for the future of the ELK Stack and the community. What are the motives behind this move? Does this split actually signify a fork?  

While these questions are extremely important to answer, I’m not going to attempt to answer them here. What I do think might benefit end users at this point is an understanding of what this new stack (yes, it’s not just Elasticsearch) actually includes in comparison to Elastic’s open source and commercial repositories, and how to get started.  

A few clarifications

We will take a closer look at specific features included in the project later on, but just to make things clear — Open Distro for Elasticsearch, despite the misleading name that hopefully will be changed soon, does not include only Elasticsearch but also Kibana and some additional plugins. It does NOT include Logstash or any of the Beats. The Elasticsearch and Kibana version currently used is 6.6.2 and Kibana 6.5.4.

A second point worth pointing out, and this is probably obvious — at version 0.8.0 Open Distro for Elasticsearch is not production-ready. There are some nifty features included in the package but I think we still have some time to wait until version 1.0.0.  

Installing Open Distro for Elasticsearch

Currently, the project supports two installation methods – via RPM package or suing Docker images. Since I have Docker Desktop installed on my Mac, I opted for the latter.

You can pull and run the individual Elasticsearch and Kibana images if you like, but the easier option is using docker-compose. One important note if you’re using Docker Desktop — be sure to allocate at least 8GB of RAM to the program (Preferences → Advanced).

Create a docker-compose.yml file:

sudo mkdir open-distro-elasticsearch
cd open-distro-elasticsearch
sudo vim docker-compose.yml

Paste the following configuration:

version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:0.7.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:0.7.0
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.zen.ping.unicast.hosts=odfe-node1
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:0.7.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
    networks:
      - odfe-net

volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net:

As you can see, there are no big surprises here, with the regular ports for Elasticsearch and Kibana mapped (9200/5601) as well as an additional port for the Perf Analyzer (9600). Memory swap is disabled and JVM heap size set.

It’s time to deploy the images:

docker-compose up

Docker will take a while to download the images and run the containers, but when done, you’ll see two Elasticsearch containers and one Kibana container running:

docker ps

CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                      NAME
9873efd238ab        amazon/opendistro-for-elasticsearch:0.7.0          "/usr/local/bin/dock…"   About a minute ago   Up About a minute   9200/tcp, 9300/tcp, 9600/tcp                               odfe-node2
e3d5558670df        amazon/opendistro-for-elasticsearch-kibana:0.7.0   "/usr/local/bin/kiba…"   About a minute ago   Up About a minute   0.0.0.0:5601->5601/tcp                                     odfe-kibana
1cb902342766        amazon/opendistro-for-elasticsearch:0.7.0          "/usr/local/bin/dock…"   About a minute ago   Up About a minute   0.0.0.0:9200->9200/tcp, 0.0.0.0:9600->9600/tcp, 9300/tcp   odfe-node1

Once you’ve verified all the containers are up and running as expected, we can test our stack by querying Elasticsearch and opening Kibana in our browser.

Now, one key difference compared to what we’ve been used to with the ELK Stack is that the security plugin is enabled by default.

Opening the Elasticsearch configuration file (on Docker container, located at: /usr/share/elasticsearch/elasticsearch.yml) this difference is quickly apparent:

cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

######## Start OpenDistro for Elasticsearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
opendistro_security.ssl.transport.pemcert_filepath: esnode.pem
opendistro_security.ssl.transport.pemkey_filepath: esnode-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: esnode.pem
opendistro_security.ssl.http.pemkey_filepath: esnode-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_unsafe_democertificates: true
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3

And Kibana’s configuration (on Docker container, located at config/kibana.yml):

server.name: kibana
server.host: "0"
elasticsearch.url: https://localhost:9200
elasticsearch.ssl.verificationMode: none
elasticsearch.username: kibanaserver
elasticsearch.password: kibanaserver
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.preferred: ["Private", "Global"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]

So using the default authentication credentials, admin: admin, we will first query Elasticsearch:

curl -XGET https://localhost:9200 -u admin:admin --insecure

{
  "name" : "28j31GT",
  "cluster_name" : "odfe-cluster",
  "cluster_uuid" : "xIMhCp8zR1CVM0t3mfFYow",
  "version" : {
    "number" : "6.5.4",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "d2ef93d",
    "build_date" : "2018-12-17T21:17:40.758843Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

And then access Kibana at: http://localhost:5601:

Security plugin

Security is the most meaningful open source addition to the ELK Stack and as such deserves a followup article. Still, some points are worth mentioning.

AWS concurs that most of the security features in Open Distro for Elasticsearch are based on Search Guard code, developed together with floragunn, the developers of Search Guard.

Anyone familiar with this security plugin for ELK will find a lot of the same functionality. Role-based access, user control and mapping, audit logging, API, AD and LDAP support — these are all part and parcel of the security plugin.

As mentioned above, Open Distro for Elasticsearch will authenticate with some default credentials (admin:admin) but you should definitely change those after setup.

You can of course disable security if you like by adding the following line to your elasticsearch.yml file:

opendistro_security.disabled: true

Alerting plugin

Open Distro for Elasticsearch provides a built-in mechanism for sending an alert when an Elasticsearch index meets a threshold and conditions that you define.

On the new Alerts page in Kibana, you will find a tab where you can define where you want to send the alert to (destination). Currently, Slack and AWS Chime are the only built-in destinations.

The alerts themselves are configured on the Monitors tab. Here you define the schedule of the alert (i.e. when you want the monitor to run) and the specific query you want to run against your Elasticsearch indices. You can build the query itself or use a graph as a visual aid to building the query.

You then need to build what’s called a trigger which defines the specific condition that the monitor checks and an action to execute when an alert is triggered.

After an initial look, I would carefully say that the alerting mechanism in Open Distro for Elasticsearch is still a bit raw at the edges and immature but let’s wait and see how it evolves.

Performance Analyzer

A known pain point with handling Elasticsearch is monitoring performance. The Performance Analyzer plugin is an agent that can be installed locally and used to query performance metrics on your cluster.

First, you’ll need to download the Performance Analyzer from: https://opendistro.github.io/for-elasticsearch/downloads.html

Extract it, and access the main installation folder:

sudo unzip perf-top-0.7.0.0-MACOS.zip
cd perf-top-0.7.0.0-MACOS

In the dashboards folder, you’ll see four provided dashboards which we can use to monitor our Elasticsearch cluster: ClusterNetworkMemoryAnalysis.json, ClusterOverview.json, ClusterThreadAnalysis.json and NodeAnalysis.json. You can also create your own dashboards for monitoring other metrics.

The syntax for using the Performance Analyzer specifies the dashboard name and an endpoint for the cluster, in our case, our local cluster (remember, we opened port 9600 when deploying the cluster):

./perf-top-macos --dashboard dashboards/ClusterNetworkMemoryAnalysis.json --endpoint localhost:9600

All in all, the tool is easy to install and use but of course lacks the ability to perform analysis over time.  

Side-by-side

So is all the fuss we’ve seen in the ELK community warranted?

From a features perspective, all versions support the key functionality provided in Elasticsearch and Kibana — data indexing, searching, aggregation and analysis.

The added plugins provided in Open Distro for Elasticsearch are more or less on par with the corresponding features in the ELK Stack available under the Basic license. The only exception here is the Alerting feature, which seems to be a nice first step, if somewhat more complicated to use.

Here’s a comparison that sums up the key differences. I’m comparing Open Distro for Elasticsearch with the open source ELK Stack and the commercial version of the ELK Stack:

The biggest difference, of course, is licensing. This was the key driver behind the project to start with. The additional plugins reviewed above are all Apache 2.0 licensed and completely open sourced compared to the same features in the ELK Stack. For many users, the fact that they can freely use security, alerting, and the other plugins provided in Open Distro for Elasticsearch that I didn’t mention here — JDBC and SQL, makes all the difference.

Everything comes at a cost

I’d like to end this review with another point that I think is worth mentioning — the cost.

If it’s open source then there is no cost, right? Well, we’ve been very clear in the past on what it entails to maintain the ELK Stack at scale. Handling large data pipelines in production, deploying buffering mechanisms, monitoring your Elasticsearch cluster and maintaining it, handling upgrades — all these will come at a cost. There is little to no difference between Open Distro for Elasticsearch and Elastic’s ELK Stack when looking at the real costs of doing ELK on your own.

I think it’s great to see how more groups are contributing to the ecosystem and I believe that at the end of the day, this new development will benefit ELK users by driving new innovation and developing the stack.

Reiterating what I said in the introduction, Open Distro for Elasticsearch is not production-ready. Proceed with caution!

While Logz.io provides Kibana — the ELK Stack’s visualization tool — as part of its service, a lot of users have asked us to support Grafana. One of the leading open source visualization tools today, Grafana has some added value when compared to Kibana, especially around visualizing time-series data.

We are happy to announce that the Logz.io Grafana plugin is now available as an official data source plugin. Grafana users can now install the plugin and add Logz.io as a data source for analyzing the data stored in Logz.io. 

The plugin is basically a fork of the existing Elasticsearch plugin, with the addition of support for custom HTTP headers, required for passing a Logz.io API token. To use the integration, you will need write access to your Grafana instance and access to Logz.io API (Enterprise users only).   

Installing Grafana

Just in case you do not have a Grafana instance, here are instructions for installing the latest stable version (5.1.2) on Ubuntu/Debian. If you’re using a different OS, refer to Grafana’s great docs here (if you’re using Docker, that’s probably the easiest way to get Grafana up and running).

Download the package and install it with these three commands:

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.1.2_amd64.deb
sudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_5.1.2_amd64.deb

Next, start Grafana with:

sudo service grafana-server start

To access Grafana and make sure all is working as expected, open your browser at

http://<serverIP>:3000

Use admin/admin as the credentials to access Grafana:

As you can see, we have no data source in Grafana, so our next step is to hook into the data stored in Logz.io by adding a new data source.

Installing the Logz.io plugin

Before we can add Logz.io as a data source in Grafana, however, we need to download and install the plugin.

There are a number of ways to do this, the simplest being using the Grafana CLI to install from the command line:

grafana-cli plugins install logzio-datasource

The plugin is installed in the Grafana plugins directory at /var/lib/grafana/plugins.

Or, you can try the manual approach by downloading the plugin and copying it into the plugins folder:

git clone https://github.com/logzio/grafana-logzio-datasource.git
sudo cp -a grafana-logzio-datasource/dist/ /var/lib/grafana/plugins/logzio/

Restart Grafana with:

sudo service grafana-server restart

Adding the Logz.io data source to Grafana

Before you add Logz.io as a data source, retrieve an API token.

To do this, access Logz.io and under Cogwheel → Tools → API Tokens, copy one of your tokens or generate a new one.

Open up Grafana and click Add Data Source.

Configure the Logz.io data source as follows.

As the data source name, enter whatever name you want to give the data source. In my case, I’m going to use Metricbeat.

As the type, open the drop-down menu and select Logz.io from the list of available data sources.

In the HTTP section, use this URL – https://api.logz.io/v1/elasticsearch. You can leave the Access type as-is.

You can leave the default settings in the Auth and Advanced HTTP Settings sections untouched.

In the Custom Headers section, add the following header definition:

• Key – X-API-TOKEN
• Value – your Logz.io API token created in the steps above

Last but not least, in the Elasticsearch details section, enter an index name (e.g. metricbeat) and use @timestamp as the Time field name.

The final configuration should look something like this:

Click Save & Test to add the new Logz.io data source. If all goes as expected, you will see a green success message and the Logz.io data source will be added to Grafana.

Congrats! You’ve added Logz.io as a Grafana data source.

Analyzing your Logz.io data in Grafana

Grafana is a different beast compared to Kibana, but if you’re setting up this integration you are probably acquainted with the basics (in Grafana, visualizations are called Panels!).

I’m not going to bore you with how to build new dashboards in Grafana, but one of Grafana’s strongest features is its ecosystem of ready-made dashboards for different data sources and types. With a few adjustments, we can make use of these dashboards.

Browse for a dashboard that interests you on Grafana’s Dashboards page. In my case, I’m going to search for a dashboard for Metricbeat system metrics.

Download the dashboard’s JSON spec, and edit it to work with Logz.io’s data source:

• Change the pluigId from Elasticsearch to logzio in both the _inputs and the _requires sections.
• Change the plugin’s version in the _requires section to 1.0.0

"__inputs": [
    {
      "name": "DS_EIGHTY20-ES",
      "label": "elasticsearch",
      "description": "",
      "type": "datasource",
      "pluginId": "logzio",
      "pluginName": "Elasticsearch"
    }
  ],
  "__requires": [
    {
      "type": "datasource",
      "id": "logzio",
      "name": "Elasticsearch",
      "version": "1.0.0"
    },
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "4.2.0"
    },
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph",
      "version": ""
    },
    {
      "type": "panel",
      "id": "singlestat",
      "name": "Singlestat",
      "version": ""
    }
  ],

In Grafana’s toolbar, select + → Create → Import, and paste the updated JSON in the relevant field.

Click Load, and you will be presented with some options for defining the imported dashboard.

Click Load, and you will be presented with some options for defining the imported dashboard.

Endnotes

Kibana has made a lot of progress towards providing users with visualization tools for time-series data, first introducing Timelion and then later on adding the Time Series Visual Builder to the list of the supported visualization types.

Still, many users prefer to stick to Grafana which is designed to integrate with a variety of time series data stores such as Prometheus and Graphite. Indeed, it is not unique to find users using both Kibana and Grafana for different data types and with different data stores.

We are glad we can help our users keep to their current workflows and use the tool they prefer. As many of our users know, we are also working on tighter integration with Grafana to support time-series analytics and will be announcing some news about this soon. Stay tuned!

Whether an active member of the ELK community or just a happy user, you’ve probably heard of a recent data breach involving Elasticsearch. Indeed, not a month goes by where we don’t come across an article or research showcasing a set of sensitive information exposed on an Elasticsearch cluster.

A substantial amount of this research into vulnerable Elasticsearch instances is conducted by Bob Diachenko, a security analyst and consultant at Security Discovery. For example, a big discovery in the beginning of the year involved millions of sensitive files, including home loan applications, credit reports, bankruptcy records, and more.

I asked Bob a few questions about his work, his findings on Elasticsearch vulnerabilities and his general outlook on the future.

tl;dr – Read the manual and don’t be lazy!

What motivated you to begin researching Elasticsearch security breaches?

It all started back in 2015, when a company I was working for experienced a serious data breach. I was in charge of cleaning up the mess after the fact and communicating an “all is safe and secure now” message to our customers, this event became a turning point in my career. I decided to take a closer look into data breaches — how they occur, what technologies are especially vulnerable and why, and how best to mitigate these occurrences.

Since then, I’ve focused on taking a look at open source NoSQL databases and their weaknesses, specifically Elasticsearch, MongoDB and CouchDB, but also other smaller and less popular databases. My goal is to both try and stay ahead of malicious actors targeting these exposed databases, alert users when possible and raise awareness.

Can you explain a bit about how you conduct your research?

My methodology is pretty simple. I use a set of search engines that crawl the web for exposed IPs and ports. Similar to how Google crawls the web for websites, these tools crawl the web for connected and exposed endpoints.

Using Shodan for example, one can simply see a list of IPs of exposed databases that can be accessed easily via a browser:

I also use tools such as BinaryEdge, Censys, ZoomEye and others.

Elasticsearch has been mentioned a lot in recent reports on data breaches. Do you see this in your data?

I would estimate that out of all the data breaches I’ve identified, 60% can be traced to exposed Elasticsearch instances, 30% are MongoDB-related, and the rest are equally distributed across the other databases. So yes, Elasticsearch is most definitely the most vulnerable NoSQL database showing up in my research.

Are there any specific regions or deployment types where you see a relative large amount of vulnerable Elasticsearch instances?

This probably won’t surprise anyone but Elasticsearch instances installed either on AWS or Microsoft Azure are the most exposed, with the U.S and China leading the pack in terms of geographical regions. I suspect the popularity of AWS as a deployment method for installing Elasticsearch and the sheer amount of Azure users are what are driving these numbers.

So what makes Elasticsearch so vulnerable compared to the other databases coming up in your research?

It all goes down to the default security configurations available or provided by the open source distribution of the database. Elasticsearch specifically, does not provide a built-in authentication and authorization mechanism and requires further investment and configuration on the user’s side. Sure, there are some basic steps that can be taken to secure Elasticsearch, but users are either not aware or are simply too lazy to bother.   

How do you see this developing in the future?

The sad truth is that I don’t see the situation improving in the years ahead. As long as human beings are the ones installing Elasticsearch and making the configurations, that is. Humans are humans. A lot of developers look for the easiest way to get started and that does not involve setting up proper security for Elasticsearch.

We have to end this interview with a positive message! What best practices do you recommend for organizations using Elasticsearch?

At the end of the day, it’s all in the documentation. Of course organizations cannot force their employees to read the Elasticsearch user manuals but elaborate internal processes can be put in place to make sure ports are not open and safe configurations are used as a standard.

If you’re limited by budget, you can use simple tactics like not using default ports, binding Elasticsearch to a secure IP, or deploying a proxy like nginx in front of Elasticsearch. If you can invest some time and resources, there are a few open source options such as SearchGuard and if you have a bit of a budget you can, of course, go for a fully managed solution like Logz.io.

Security Discovery was founded by a team of cyber security researchers and offers news, best practices, consulting services and more. Members of the Security Discovery security team have identified data breaches that were covered by news outlets such as the BBC, Forbes, Financial Times, Washington Post, Engadget, TechCrunch, NYDaily News, and many more. Security Discovery reports on data breaches discovered and offers unique perspectives on how they occurred and who may have been impacted. 

Following the recent announcement of our partnership with Microsoft, Azure users can now monitor, troubleshoot, and secure their environments with a fully managed Azure-native ELK solution. However, If you want to set up the ELK Stack on Azure on your own this guide will help you get started.

The ELK Stack (Elasticsearch, Logstash & Kibana) offers Azure users with all the key ingredients required for monitoring their applications — Elasticsearch for scalable and centralized data storage, Logstash for aggregation and processing, Kibana for visualization and analysis, and Beats for collection of different types of data and forwarding it into the stack.The ELK Stack can be deployed in a variety of ways and in different environments and we’ve covered a large amount of these scenarios in previous articles in this blog. As mentioned, this article covers what is an increasingly popular workflow — installing the ELK stack on Azure. We’ll start with setting up our Azure VM and then go through the steps of installing Elasticsearch, Logstash, Kibana and Metricbeat to set up an initial data pipeline.

The Azure environment

Our first and initial step is to set up the Azure environment. In the case of this tutorial, this includes an Ubuntu 18.04 VM, with a Network Security Group configured to allow incoming traffic to Elasticsearch and Kibana from the outside. 

We’ll start by creating a new resource group called ‘elk’:

In this resource group, I’m going to deploy a newUbuntu 18.04 VM:

Please note that when setting up the VM for testing or development purposes, you can make do with the default settings provided here. But for handling real production workloads you will want to configure memory and disk size more carefully.

Once your VM is created, open the Network Security Group created with the VM and add inbound rules for allowing ssh access via port 22 and TCP traffic to ports 9200 and 5601 for Elasticsearch and Kibana:

Now that our Azure environment is ready, we can now proceed with installing the ELK Stack on our newly created VM.

Installing Elasticsearch

To install Elasticsearch we will be using DEB packages.   

First, you need to add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

For Debian, we need to then install the apt-transport-https package:

sudo apt-get update
sudo apt-get install apt-transport-https

The next step is to add the repository definition to your system:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo 
tee -a /etc/apt/sources.list.d/elastic-7.x.list

All that’s left to do is to update your repositories and install Elasticsearch:

sudo apt-get update && sudo apt-get install elasticsearch

Elasticsearch configurations are done using a configuration file (On Linux: /etc/elasticsearch/elasticsearch.yml)that allows you to configure general settings (e.g. node name), as well as network settings (e.g. host and port), where data is stored, memory, log files, and more.

sudo su
vim /etc/elasticsearch/elasticsearch.yml

Since we are installing Elasticsearch on Azure, we will bind Elasticsearch to localhost. Also, we need to define the private IP of our Azure VM (you can get this IP from the VMs page in the console) as a master-eligible node:

network.host: "localhost"
http.port:9200
cluster.initial_master_nodes: ["<PrivateIP"]

Save the file and run Elasticsearch with:

sudo service elasticsearch start

To confirm that everything is working as expected, point curl or your browser to http://localhost:9200, and you should see something like the following output (give Elasticsearch a minute to run):

{
  "name" : "elk",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "nuDp3c-JSU6tW-kqqPW68A",
  "version" : {
    "number" : "7.0.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "e4efcb5",
    "build_date" : "2019-04-29T12:56:03.145736Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Installing an Elasticsearch cluster requires a different type of setup. Read our Elasticsearch Cluster tutorial for more information on that.

Installing Logstash

Logstash requires Java 8 or Java 11 to run so we will start the process of setting up Logstash with:

sudo apt-get install default-jre

Verify Java is installed:

java -version

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Since we already defined the repository in the system, all we have to do to install Logstash is run:

sudo apt-get install logstash

Before you run Logstash, you will need to configure a data pipeline. We will get back to that once we’ve installed and started Kibana.

Installing Kibana

As before, we will use a simple apt command to install Kibana:

sudo apt-get install kibana

Open up the Kibana configuration file at: /etc/kibana/kibana.yml, and make sure you have the following configurations defined:

server.port: 5601
elasticsearch.url: "http://localhost:9200"

These specific configurations tell Kibana which Elasticsearch to connect to and which port to use.

Now, start Kibana with:

sudo service kibana start

Open up Kibana in your browser with: http://localhost:5601. You will be presented with the Kibana home page.

Installing Beats

The various data shippers belonging to the Beats family can be installed in exactly the same way as we installed the other components.

As an example, let’s install Metricbeat to send some host metrics from our Azure VM:

sudo apt-get install metricbeat

To start Metricbeat, enter:

sudo service metricbeat start

Metricbeat will begin monitoring your server and create an Elasticsearch index which you can define in Kibana. Curl Elasticsearch to make sure:

curl 'localhost:9200/_cat/indices?v'

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   metricbeat-7.0.1-2019.05.12-000001 ULSQBIigQqK2zCUL3G-4mQ   1   1       6523            0      3.8mb          3.8mb
green  open   .kibana_1                          tlx3dbPySR2CBQM58XTYng   1   0          3            0     12.2kb         12.2kb

To begin analyzing these metrics, open up the Management → Kibana → Index Patterns page in Kibana. You’ll see the newly created ‘metricbeat-*’ index already displayed:

All you have to do now is enter the index pattern, select the @timestamp field and hit the Create index pattern button. Moving over to the Discover page, you’ll see your Azure VM host metrics displayed:

Congratulations! You have set up your first ELK data pipeline! More information on using the different beats is available on our blog: Filebeat, Metricbeat, Winlogbeat, Auditbeat.

ELK on Azure…at scale

Azure offers users the flexibility and scalability they require to install and run the ELK Stack at scale. As seen in the instructions above, setting up ELK on Azure VMs is pretty simple. Of course, things get more complicated in full-blown production deployments of the stack.  

Data pipelines built for collecting, processing, storing and analyzing production environments require additional architectural components and more advanced configuration. For example, a buffering layer (e.g. Kafka) should be put in place before Logstash to ensure a data flow that is resilient and that can stand the pressure of data growth and bursts. Replicating across availability zones is recommended to ensure high availability. And the list goes on.

This requires time and resources that not every organization can afford to spend. Logz.io offers Azure users with a fully managed ELK solution for monitoring their applications, including a seamless integration with Azure and built-in dashboards for various Azure resources such as Active Directory, Application Gateway, Activity Logs and more.

Have you ever found yourself trying to reconstruct an event from the past only to come up blank because you cannot go so far back in time? If only you could bring back that missing piece of the puzzle!

In the world of IT, logs are the way machines and software record events. They help us understand when an event happened, where they happened and most importantly, why they happened. In a perfect world, engineers would have access to all the data generated by their applications regardless of the timestamp attached.

But of course, we don’t live in a perfect world. Data volumes are exploding and because retention is extremely expensive, engineers end up making painful decisions about what data to retain and for how long.

Hot vs. cold retention

Retention periods vary from organization to organization, use case to use case. You might need to retain data for an entire year for compliance and security needs, or you might make do with just a few days of retention for the purpose of troubleshooting.

Whatever retention period you opt for, once it’s over your data is lost. Unless, that is, you’ve put a system into place that archives this data and “revives” it when necessary by re-ingesting it back into the system. Most organizations will opt for a hybrid solution based on a short “hot” retention period in which data is searchable, and an extended “cold” retention period in which data is archived to a 3rd party data storage service but is not searchable.

Amazon S3 is one of the more common solutions for storing and archiving log data for extended retention periods because of its cost-efficient storage, its different storage options and integrability with other AWS services.

Logz.io Archive/Restore

Logz.io makes it easy for users to archive data to S3 and reingest it back into the system via the user interface.

This feature is accessed via the Settings icon  > Tools > Archive & restore in the top menu. Here you are required to enter the name of the S3 bucket you want to archive to and specific IAM credentials for the bucket (the IAM user must have the following S3 permissions: PutObject, ListBucket, and GetObject.)

You can test the configuration using the Test connection button to make sure Logz.io has access to the bucket.

If all is OK, you’ll see a success message at the bottom of the page and all that’s left to do now is click the Start archiving button. Archiving is enabled for the specific Logz.io account you set up archiving from, as reflected in the Archiving is on message at the top of the page.

That’s one side of the story. How do you restore this historical data for analysis? This is done on the same page, on the Restore tab.

Here, all you have to do is enter the name of the account you want to restore and the specific time frame you want to restore logs for. Currently, the user interface limits you to a maximum of one day worth of logs. If you need more than that, let the Support team know and we’ll help you out.

Hit the Restore button.

Logz.io begins the process of restoring the data from the S3 bucket, and you can view the restore actions on the Restored accounts tab:

The restore process will run in the background, and once it completes it will become Active — meaning your data is restored and can be searched in Kibana.

You’ll also receive an email notifying you that the restore was completed:

As the message states, the restored data is now “hot” for five days. Conveniently, you can click the provided link to begin analyzing the data in Kibana:

Optimizing logging costs

Data retention is one of the most important components of any monitoring solution. On the one hand, you need to be able to search your data for as long as your use case organization requires. On the other hand, your also need to remain cost-efficient. Logz.io offers users a number of ways to maintain this balance.

Using the Data Optimizer, you can aggregate data into dedicated Timeless accounts with no retention limits, allowing you to differentiate between important data that requires extended retention and data that does not. Logz.io’s Archive/Restore helps you easily ship your data into long-term storage for subsequent retrieval when required.

Archive/Restore is available for all users. As always, we’d love to get your feedback. So if you have any comments or questions, feel free to reach out to info@logz.io.

Happy archiving/restoring!

At RSA this year, we introduced a series of new enhancements to Security Analytics – our new app for helping organizations combat security threats and meet compliance requirements. We are now happy to announce the official release of one of these features — Drilldown!

Multi-dashboard investigations

Dashboards are an extremely useful tool to use for security analytics. If properly constructed, dashboards can provide us with deep insight into events taking place in our environment, pointing us to suspicious behavior and helping us with the next step in our investigation.

Often enough, we will have a series of dashboards we use to monitor our environment. A common best practice is to have one primary dashboard that provides us with a general view of the system and a series of secondary dashboards that aid us in gaining more granular insight into a specific element.

But switching back and forth between these dashboards is not very user-friendly. Not only that, I might not even know that a secondary dashboard exists unless I actually created it. Wouldn’t it be cool if we could easily move from one dashboard to another to help us drill down into specific elements displayed in the dashboard?

Building interactivity into dashboards

This is where Drilldown comes into the picture — allowing Logz.io Security Analytics users to map specific log fields to target dashboards. This enables jumping from one Kibana dashboard to another, and makes dashboards more interactive and investigation simpler!

This is done on the Drilldown Settings page (Cogweel → Tools → Drilldown settings):

All you have to do is select the source field you want to map and the target dashboard you want it to link to.

Example – IP investigation

In the dashboard below, we’re monitoring our environment for potential threats. The dashboard is comprised of a series of visualizations, each depicting these threats in a different manner.

In the Threats by attacker visualization, we’re displaying a list of attacks per IP, and as you can see, the IPs are hyperlinked:

Clicking on an IP leads us to another dashboard that helps us dive deeper into the rabbit hole and investigate that specific IP:

This linkage was achieved using Drilldown. We mapped the source_ip field to the IP Investigation dashboard.

Endnotes

The ability to jump from one dashboard to another is a hugely sought-after feature in Kibana and we are proud to provide our users with an easy and simple way to do this. Drilldown will make your dashboards more interactive, and your investigation workflows more simpler.   

Drilldown is enabled for all our Security Analytics users and we will be rolling this out for Operations users in the near future. As always, we’d love to get your feedback. So if you have any comments or questions, feel free to reach out to info@logz.io.

Amazon EC2 is the cornerstone for any Amazon-based cloud deployment. Enabling you to provision and scale compute resources with different memory, CPU, networking and storage capacity in multiple regions all around the world, EC2 is by far Amazon’s most popular and widely used service.

Monitoring EC2 is crucial for making sure your instances are available and performing as expected. Metrics such as CPU utilization, disk I/O, and network utilization, for example, should be closely tracked to establish a baseline and identify when there is a performance problem.

Conveniently, these monitoring metrics, together with other metric types, are automatically shipped to Amazon CloudWatch for analysis. While it is possible to use the AWS CLI, API or event the CloudWatch Console to view these metrics, for deeper analysis and more effective monitoring, a more robust monitoring solution is required.

In this article, I’d like to show how to ship EC2 metrics into the ELK Stack and Logz.io. The method I’m going to use is a new AWS module made available in Metricbeat version 7 (beta). While still under development, and as shown below, this module provides an extremely simple way for centrally collecting performance metrics from all your EC2 instances.

Prerequisites

I assume you already have either your own ELK Stack deployed or a Logz.io account. For more information on installing the ELK Stack, check out our ELK guide. To use the Logz.io community edition, click here.

Step 1: Creating an IAM policy

First, you need to create an IAM policy for pulling metrics from CloudWatch and listing EC2 instances. Once created, we will attach this policy to the IAM user we are using.

In the IAM Console, go to Policies, hit the Create policy button, and use the visual editor to add the following permissions to the policy:

• ec2:DescribeRegions
• ec2:DescribeInstances
• cloudwatch:GetMetricData

The resulting JSON for the policy should look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "cloudwatch:GetMetricData",
                "ec2:DescribeRegions"
            ],
            "Resource": "*"
        }
    ]
}

Once saved, attach the policy to your IAM user.

Step 2: Installing Metricbeat

Metricbeat can be downloaded and installed using a variety of different methods, but I will be using Apt to install it from Elastic’s repositories.

First, you need to add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | 
sudo apt-key add -

The next step is to add the repository definition to your system. Please note that I’m using the 7.0 beta repository since the AWS module is bundled with this version only for now:

echo "deb https://artifacts.elastic.co/packages/7.x-prerelease/apt stable 
main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x-prerelease.list

All that’s left to do is to update your repositories and install Metricbeat:

sudo apt-get update && sudo apt-get install metricbeat

Step 3: Configuring Metricbeat

Before we run Metricbeat, there are a few configurations we need to apply.

First, we need to disable the system module that is enabled by default. Otherwise, we will be seeing system metrics in Kibana collected from our host. This is not mandatory but is recommended if you want to keep a cleaner Kibana workspace.

sudo metricbeat modules disable system

Verify with:

ls /etc/metricbeat/module.d

envoyproxy.yml.disabled     kvm.yml.disabled         postgresql.yml.disabled
aerospike.yml.disabled      etcd.yml.disabled        logstash.yml.disabled   prometheus.yml.disabled
apache.yml.disabled         golang.yml.disabled      memcached.yml.disabled  rabbitmq.yml.disabled
aws.yml.disabled            graphite.yml.disabled    mongodb.yml.disabled    redis.yml.disabled
ceph.yml.disabled           haproxy.yml.disabled     mssql.yml.disabled      system.yml.disabled
couchbase.yml.disabled      http.yml.disabled        munin.yml.disabled      traefik.yml.disabled
couchdb.yml.disabled        jolokia.yml.disabled     mysql.yml.disabled      uwsgi.yml.disabled
docker.yml.disabled         kafka.yml.disabled       nats.yml.disabled       vsphere.yml.disabled
dropwizard.yml.disabled     kibana.yml.disabled      nginx.yml.disabled      windows.yml.disabled
elasticsearch.yml.disabled  kubernetes.yml.disabled  php_fpm.yml.disabled    zookeeper.yml.disabled

The next step is to configure the AWS module.

sudo vim /etc/metricbeat/module.d/aws.yml.disabled

Add your AWS IAM user credentials to the module configuration as follows:

- module: aws
  period: 300s
  metricsets:
    - "ec2"
  access_key_id: 'YourAWSAccessKey'
  secret_access_key: 'YourAWSSecretAccessKey'
  default_region: 'us-east-1'

In this example, we’re defining the user credentials directly but you can also refer to them as env variables if you have defined as such. There is also an option to use temporary credentials, and in that case you will need to add a line for the session token. Read more about these options in the documentation for the module.

The period setting defines the interval at which metrics are pulled from CloudWatch.

To enable the module, use:

sudo metricbeat modules enable aws

Shipping to ELK

To ship the EC2 metrics to your ELK Stack, simply start Metricbeat (the default Metricbeat configuration has a locally Elasticsearch instance defined as the output so if you’re shipping to a remote Elasticsearch cluster, be sure to tweak the output section before starting Metricbeat):

sudo service metricbeat start

Within a few seconds, you should see a new metricbeat-* index created in Elasticsearch:

health status index                                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1                                kzb_TxvjRqyhtwY8Qxq43A   1   0        490            2    512.8kb        512.8kb
green  open   .kibana_task_manager                     ogV-kT8qSk-HxkBN5DBWrA   1   0          2            0     30.7kb         30.7kb
yellow open   metricbeat-7.0.0-beta1-2019.03.20-000001 De3Ewlq1RkmXjetw7o6xPA   1   1       2372            0        2mb            2mb

Open Kibana, define the new index patten under Management → Kibana Index Patterns, and you will begin to see the metrics collected by Metricbeat on the Discover page:

Shipping to Logz.io

By making a few adjustments to the Metricbeat configuration file, you can ship the EC2 metrics to Logz.io for analysis and visualization.

First, you will need to download an SSL certificate to use encryption:

wget https://raw.githubusercontent.com/logzio/public-certificates/
master/COMODORSADomainValidationSecureServerCA.crt

sudo mkdir -p /etc/pki/tls/certs

sudo cp COMODORSADomainValidationSecureServerCA.crt 
/etc/pki/tls/certs/

Next, retrieve your Logz.io account token from the UI (under Settings → General).

Finally, tweak your Metricbeat configuration file as follows:

fields:
  logzio_codec: json
  token: <yourToken>
fields_under_root: true
ignore_older: 3hr
type: system_metrics

output.logstash:
  hosts: ["listener.logz.io:5015"]
  ssl.certificate_authorities: 
['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

Be sure to enter your account token in the relevant placeholder above and to comment out the Elasticsearch output.

Restart Metricbeat with:

sudo service metricbeat restart

Within a minute or two, you will see the EC2 metrics collected by Metricbeat show up in Logz.io:

Step 4: Analyzing EC2 metrics in Kibana

Once you’ve built a pipeline of EC2 metrics streaming into your ELK Stack, it’s time to reap the benefits. Kibana offers rich visualization capabilities that allow you to slice and dice data in any way you want. Below are a few examples of how you can start monitoring your EC2 instances with visualizations.

Failed status checks

CloudWatch performs different types of status checks for your EC2 instances. Metrics for these checks can be monitored to keep tabs on the availability and status of your instances. For example, we can create a simple metric visualization to give us an indication on whether any of these checks failed:

CPU utilization

Kibana’s visual builder visualization is a great tool for monitoring time series data and is improving from version to version. The example below gives us an average aggregation of the ‘aws.ec2.cpu.total.pct’ field per instance.

Network utilization

In the example below, we’re using the visual builder again to look at an average aggregation of the ‘aws.ec2.network.in.bytes’ field per instance to monitor incoming traffic. In the Panel Options tab, I’ve set the interval at ‘5m’ to correspond with the interval at which we’re collecting the metrics from CloudWatch.

We can do the same of course for outgoing network traffic:

Disk performance

In the example here, we’re monitoring disk performance of our EC2 instances. We’re showing an average of the ‘aws.ec2.diskio.read.bytes’ and the ‘aws.ec2.diskio.write.bytes’ fields, per instance:

Summing it up

The combination of CloudWatch and the ELK Stack is a great solution for monitoring your EC2 instances. Previously, Metricbeat would have been required to be installed per EC2 instance. The new AWS module negates this requirement, making the process of shipping EC2 metrics into either your own ELK or Logz.io super-simple.

Once in the ELK Stack, you can analyze these metrics to your heart’s delight, using the full power of Kibana to slice and dice the metrics and build your perfect EC2 monitoring dashboard!

This dashboard is available in ELK Apps — Logz.io’s library of premade dashboards and visualizations for different log types. To install, simply open ELK Apps and search for ‘EC2’.

Looking forward, I expect more and more metricsets being supported by this AWS module, meaning additional AWS services will be able to be monitored with Metricbeat. Stay tuned for news on these changes in this blog!

The ELK Stack (Elasticsearch, Logstash and Kibana) is the weapon of choice for many Kubernetes users looking for an easy and effective way to gain insight into their clusters, pods and containers. The “L” in “ELK” has gradually changed to an “F” reflecting the preference to use Fluentd instead of Logstash and making the “EFK Stack” a more accurate acronym for what has become the de-facto standard for Kubernetes-native logging.

While Elasticsearch, Logstash/Fluentd and Kibana can be installed using a variety of different methods, using Kubernetes is becoming more and more popular. The same reasons you’d use Kubernetes to deploy your application — for automanaging and orchestrating the underlying containers — apply here as well, and there are a lot of examples and resources online to make the process easier. 

In this article, I’ll be taking a look at one of these resources — a relatively new app available in GKE’s app marketplace called, rather succinctly, “Elastic GKE Logging”. As the name implies, this app is a turnkey solution for deploying a fully functioning logging solution for Kubernetes comprised of Elasticsearch, Kibana and Fluentd.

As described on the GitHub page, the app deploys an Elasticsearch StatefulSet for the Elasticsearch cluster with a configurable number of replicas, a stateless Kibana deployment and a fluentd DaemonSet that includes predefined configurations for shipping different logs.  

Source: Elastic GKE App repo on GitHub.

Step 1: Preparing your GKE environment

Before we can proceed with deploying the GKE Logging app from the marketplace, there are some basic steps to take to prepare the ground.

If you’re new to Google Cloud Platform (GCP) and haven’t created a project yet, this is the right time to do so.  Projects are the basis for creating, enabling, and using GCP services, including GKE’s services.

Once you’ve created your GCP project, you will need to create a Kubernetes cluster on GKE. Just follow the steps as outlined in this article. When creating your cluster, I recommend you choose a node size with 4, 8, or 16 CPUs and a cluster size of at least 2. Of course, you can always start with a small setup and then scale up later.

To connect to my GKE cluster with kubectl, I use GCP’s CloudShell service. You can, of course, do the same from your own local terminal in which case you will need to download and set up gcloud.

Next, I find it a good best practice to use dedicated namespaces to separate between different deployments in my Kubernetes cluster. In this case, I’m going to create a new namespace for all our logging infrastructure:

Create a new object file:

sudo vim kube-logging-ns.yaml

Paste the following namespace configuration:

kind: Namespace
apiVersion: v1
metadata:
  name: kube-logging

Save the file, and create the namespace with:

kubectl create -f kube-logging-ns.yaml

Confirm that the namespace was created:

kubectl get namespaces

NAME           STATUS    AGE
default        Active    1d
kube-logging   Active    1m
kube-public    Active    1d
kube-system    Active    1d

Step 2: Deploying the EFK Stack

We’re now ready to deploy our EFK-based logging solution using the Elastic GKE Logging app.

In the GCP console, open the marketplace and search for “Elastic GKE Logging”. You will see only one choice — select it to see more information on the app:

Hit the Configure button. You’ll now be presented with a short list of configuration options to customize the deployment process:

• Cluster – In case you have multiple Kubernetes clusters, select the relevant one for deploying the EFK Stack. In my case, I only have on running in my project so it’s selected by default.
• Namespace – For your namespace, be sure to select the kube-logging namespace you created in the previous step.
• App instance name – You can play around with the name for the deployment or simply go with the provided name.
• Elasticsearch replicas – The default number of replicas for the Elasticsearch ReplicaSet is 2 but for larger environments and production loads I would change this to a higher number.
• Fluentd Service Account – You can leave the default selection for the fluentd service account.

Click the Deploy button when ready.

GKE will deploy all the components in the app within the namespace and the cluster you defined and within a few minutes will present you with a summary of your deployment:

Step 3: Reviewing the deployment

Before we take a look at our Kubernetes logs, let’s take a look at some of the objects created as part of the deployment.

First, the deployment itself:

kubectl get deployment --namespace kube-logging

We can take a closer look at the deployment configuration with:

kubectl describe deployment elastic-gke-logging-kibana --namespace kube-logging

Name:                   elastic-gke-logging-kibana
Namespace:              kube-logging
CreationTimestamp:      Sun, 24 Mar 2019 13:10:09 +0200
Labels:                 app.kubernetes.io/component=kibana-server
                        app.kubernetes.io/name=elastic-gke-logging
Annotations:            deployment.kubernetes.io/revision=2
                        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apps/v1beta2","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"kibana-server","app.kubernetes.i...
Selector:               app.kubernetes.io/component=kibana-server,app.kubernetes.io/name=elastic-gke-logging
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app.kubernetes.io/component=kibana-server
           app.kubernetes.io/name=elastic-gke-logging
  Containers:
   kibana:
    Image:      gcr.io/cloud-marketplace/google/elastic-gke-logging/kibana@sha256:2d7675a1cc23800ab8040e28cd12f9ccdaceace4c0a2dae4a7802ed6be8964d7
    Port:       5601/TCP
    Host Port:  0/TCP
    Liveness:   http-get http://:5601/api/status delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:5601/api/status delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ELASTICSEARCH_URL:    http://elastic-gke-logging-elasticsearch-svc:9200
      KIBANA_DEFAULTAPPID:  discover
    Mounts:                 <none>
  Volumes:                  <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   elastic-gke-logging-kibana-579db88676 (1/1 replicas created)
Events:          <none>

Diving a bit deeper, we can see that the deployment consists of 3 Elasticsearch pods, a Kibana pod and 3 Fluentd pods deployed as part of the DaemonSet:

kubectl get pods --namespace kube-logging

NAME                                          READY     STATUS      RESTARTS   AGE

elastic-gke-logging-elasticsearch-0           1/1       Running     0          1h
elastic-gke-logging-elasticsearch-1           1/1       Running     0          1h
elastic-gke-logging-elasticsearch-2           1/1       Running     0          1h
elastic-gke-logging-fluentd-es-6hxst          1/1       Running     4          1h
elastic-gke-logging-fluentd-es-sfskc          1/1       Running     5          1h
elastic-gke-logging-fluentd-es-zcl49          1/1       Running     0          1h
elastic-gke-logging-kibana-579db88676-tdqst   1/1       Running     0          1h

The provided configurations applied to the deployment can be viewed with:

kubectl get configmap --namespace kube-logging

NAME                                    DATA      AGE
elastic-gke-logging-configmap           2         1h
elastic-gke-logging-deployer-config     8         1h
elastic-gke-logging-fluentd-es-config   4         1h
elastic-gke-logging-kibana-configmap    6         1h

A closer look at the fluentd ConfigMap reveals the provided log aggregation and processing applied as part of the deployment. Here’s an abbreviated version of this configuration:

kubectl describe configmap elastic-gke-logging-fluentd-es-config --namespace kube-logging

Name:         elastic-gke-logging-fluentd-es-config
Namespace:    kube-logging
Labels:       app.kubernetes.io/component=fluentd-es-logging
              app.kubernetes.io/name=elastic-gke-logging
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"containers.input.conf":"\u003csource\u003e\n  @id fluentd-containers.log\n  @type tail\n  path /var/log/containers/*.log\n ...

Data
====
containers.input.conf:
----
<source>
  @id fluentd-containers.log
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  tag raw.kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

Step 4: Accessing Kibana

The app deploys two ClusterIP type services, one for Elasticsearch and one for Kibana. Elasticsearch is mapped to ports 9200/9300 and Kibana to port 5601:

kubectl get svc --namespace kube-logging

NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
elastic-gke-logging-elasticsearch-svc   ClusterIP   10.39.243.235   <none>        9200/TCP,9300/TCP   1h
elastic-gke-logging-kibana-svc          ClusterIP   10.39.254.15    <none>        
5601/TCP            1h

As seen above, the Kibana service does not have an external IP so we need to expose Kibana to be able to access it. To do this from your local terminal, use:

kubectl port-forward --namespace kube-logging svc/elastic-gke-logging-kibana-svc 5601

You will then be able to access Kibana at http://localhost:5601:

Your next step is of course to begin analyzing your Kubernetes logs. I’m not going to specify the different ways in which Kibana allows you to query and visualize the data, take a look at this article for more information. Suffice to say that within a few minutes, and without too much work on your part, you’ll be able to gain operational visibility into your cluster.

Endnotes

The Elastic GKE Logging app is a nice way to set up Elasticsearch, Kibana and Fluentd on Kubernetes. For production environments, however, you will want to explore more advanced Kubernetes deployment methods.

Separating nodes in different roles for example, with master, data and ingest nodes is a common design pattern that will result in better performance (I experienced some sluggish performance even with a basic configuration). Exposing both Elasticsearch and Kibana using ingress is also a better way of allowing access from the outside to the cluster. You will also need to look into ways of cleaning up your Elasticsearch clusters, with Curator being one way to go.

But again — if you’re just getting started with Kubernetes and want to set up a quick development EFK Stack, this is a great way to go about it.

In the midst of all the turmoil and debate around Open Distro for Elasticsearch, Elastic continues to produce, and last week announced both a new major release of the Elastic Stack — version 6.7 (and also the first release candidate for 7.0!). 

So what exactly was released in version 6.7.

As usual, I’ve put together a brief overview of the main features introduced. One change I’ve applied this time is adding a comment for each feature detailing what license it falls under. I’ve encountered increasing confusion over the issue of licensing so hopefully this will help.

Elasticsearch

Elasticsearch 6.7 includes a few new features but the big news in this version is the graduation of a lot of major features that were released in beta mode in previous versions and that are now GA.

Index Lifecycle Management

A beta in version 6.6, Index Lifecycle Management is a super-useful feature that allows you to manage the lifecycle of your Elasticsearch indices more easily. Using API or the new dedicated page in Kibana, you can set rules that define the different phases that your indices go through — hot, warm, cold and deletion. In version 6.7 the ability to manage frozen indices was added (for long term and memory-efficient storage).  

Index Lifecycle Management is available under the Basic license.

Cross-Cluster Replication

A beta in version 6.5, this now-GA feature in Elasticsearch offers cross-cluster data replication for replicating data across multiple datacenters and across multiple regions. This feature followed the steps of other minor updates to Elasticsearch, specifically soft deletes and sequence numbers, and gives users a much easier way to load data into multiple clusters across data centers.

In version 6.7, the ability to replicate existing indices with soft deletes was added, as well as management and monitoring features in Kibana for gaining insight into the replication process.

Cross-Cluster Replication is only available for paid subscriptions.

SQL

A lot of Elasticsearch users were excited to hear about the new SQL capabilities announced way back in Elasticsearch 6.3. The ability to execute SQL queries on data indexed in Elasticsearch had been on the wishlist of many users, and in version 6.5, additional SQL functions were added as well as the ability to query across indices. All of this goodness, as well as the accompanying JDBC and ODBC drivers, is now GA. Additional SQL statements and functions such as the ability to sort groups by aggregates, were also added in this release.

The SQL interface is available under the Basic license. JDBC and ODBC clients are only available for paid subscriptions.

Elasticsearch index management

A series of improvements have been made to managing Elasticsearch indices in the Index Management UI in Kibana. Tags have been added to the index name to be able to differentiate between the different indices (Frozen, Follower, Rollup). It’s also easier now to freeze and unfreeze indices from the same UI in Kibana.

Index management is available under the Basic license.

Upgrading to 7.0

Version 6.7 is the last major version before 7.0 and as such, includes some new features to help users migrate to 7.0 more easily:

• The Upgrade Assistant in Kibana now allows users to leave the page when performing a reindex operation.
• Users using the API to upgrade will be pleased to know that the Deprecation Info and Upgrade Assistant APIs were enhanced.
• Reindexing data from remote clusters is easier, with the added ability to apply custom SSL parameters and added support for reindexing from IPv6 URLs.

The upgrade assistant UI and API are available under the Basic license.

Kibana

Similar to Elasticsearch, a lot of the features announced in version 6.7 are beta features maturing to GA status. Still, there are some pretty interesting new capabilities included as well.

Maps

This brand new page in Kibana is going to take geospatial analysis in Kibana to an entirely new level. Released in beta mode, Maps supports multiple layers and data sources, the mapping of individual geo points and shapes, global searching for ad-hoc analysis, customization of elements, and more.

Maps is available under the Basic license.

Source: Elastic.

Uptime

This is another brand new page in Kibana, allowing you to centrally monitor and gauge the status of your applications using a dedicated UI. The data monitored on this page, such as response times and errors, is forwarded into Elasticsearch with Heartbeat, another shipper belonging to the beats family, that can be installed either within your network or outside it — all you have to do is enter the endpoint URLs you’d like it to ping. To understand how to deploy Heartbeat, check out this article.

Uptime is available under the Basic license.

Logs

Logs was announced as a beta feature in version 6.5 and gives you the option to view your logs in a live “console-like” view. The changes made in version 6.7 allow you to configure default index and field names viewed on the page from within Kibana as opposed to configuring Kibana’s .yml file. An additional view can be accessed per log message, detailing all the fields for the selected log message and helping you gain more insight into the event.

Logs is available under the Basic license.

Infrastructure

Another beta feature going GA, the Infrastructure page in Kibana helps you gain visibility into the different components constructing your infrastructure, such as hosts and containers. You can select an element and drill further to view not only metrics but also relevant log data.

Infrastructure is available under the Basic license.

Canvas

What I call the “Adobe Photoshop” of the world of machine data analytics — Canvas — is now GA. I had the pleasure of covering the technology preview a while back, and am super excited to see how this project has progressed and finally matured.

Canvas is available under the Basic license.

Beats

Not a lot of news for beats lovers in this release as I expect most of the new goodies will be packaged in version 7.0.

Functionbeat

Functionbeat — a serverless beat that can be deployed on AWS Lambda to ship logs from AWS CloudWatch to an Elasticsearch instance of your choice — is now GA. For triggering the function, you can use either CloudWatch, SQS events, and from version 6.7 — Kinesis streams.

New datasets in Auditbeat

The system module in Auditbeat was improved and supports new datasets and data enhancements, such as a login dataset that collects login information, a package dataset that collects information on installed DEB/RPM and Homebrew packages, and the addition of a new entity_id field to datasets.

Endnotes

What about Logstash? The only news here is that there is no news. It seems that the long-awaited Java execution engine (better performance, reduced memory usage) is still in the works and hopefully will go GA in version 7.0.

As always, be careful before upgrading. Some of the features listed are still in beta so keep that in mind before upgrading. Read the breaking changes and release notes carefully as well as the licensing information.

The ELK Stack is a great open-source stack for log aggregation and analytics. It stands for Elasticsearch (a NoSQL database and search server), Logstash (a log shipping and parsing service), and Kibana (a web interface that connects users with the Elasticsearch database and enables visualization and search options for system operation users). With a large open-source community, ELK has become quite popular, and it is a pleasure to work with.

In this article, we will guide you through the simple installation process for installing the ELK Stack on Amazon Web Services.

The following instructions will lead you through the steps involved in creating a working sandbox environment. Due to the fact that a production setup is more comprehensive, we decided to elaborate on how each component configuration should be changed to prepare for use in a production environment.

We’ll start by describing the environment, then we’ll walk through how each component is installed, and finish by configuring our sandbox server to send its system logs to Logstash and view them via Kibana.

The AWS Environment

We ran this tutorial on a single AWS Ubuntu 16.04 instance on an m4.large instance using its local storage. We started an EC2 instance in the public subnet of a VPC, and then we set up the security group (firewall) to enable access from anywhere using SSH and TCP 5601 (Kibana). Finally, we added a new elastic IP address and associated it with our running instance in order to connect to the internet.

Production tip: A production installation needs at least three EC2 instances — one per component, each with an attached EBS SSD volume.

Installing Elasticsearch

Elasticsearch is a widely used database and a search server, and it’s the main component of the ELK setup.

Elasticsearch’s benefits include:

• Easy installation and use
• A powerful internal search technology (Lucene)
• A RESTful web interface
• The ability to work with data in schema-free JSON documents (noSQL)
• Open source

There are various ways to install Elasticsearch but we will be using DEB packages.   

To begin the process of installing Elasticsearch, add the following repository key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Install the apt-transport-https package:

sudo apt-get install apt-transport-https

Add the following Elasticsearch list to the key:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

To install a version of Elasticsearch that contains only features licensed under Apache 2.0, use:

echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Update your system and install Elasticsearch with:

sudo apt-get update 
sudo apt-get install elasticsearch

Open the Elasticsearch configuration file at: /etc/elasticsearch/elasticsearch.yml, and apply the following configurations:

network.host: "localhost"
http.port: 9200
cluster.initial_master_nodes: ["<PrivateIP"]

Start the Elasticsearch service:

sudo service elasticsearch start

Verify the installation by cURLing:

sudo curl http://localhost:9200

If the output is similar to this, then you will know that Elasticsearch is running properly:

{
 "name" : "ip-172-31-49-60",
 "cluster_name" : "elasticsearch",
 "cluster_uuid" : "yP0uMKA6QmCsXQon-rxawQ",
 "version" : {
   "number" : "7.0.0",
   "build_flavor" : "default",
   "build_type" : "deb",
   "build_hash" : "b7e28a7",
   "build_date" : "2019-04-05T22:55:32.697037Z",
   "build_snapshot" : false,
   "lucene_version" : "8.0.0",
   "minimum_wire_compatibility_version" : "6.7.0",
   "minimum_index_compatibility_version" : "6.0.0-beta1"
 },
 "tagline" : "You Know, for Search"
}

To make the service start on boot run:

sudo update-rc.d elasticsearch defaults 95 10

Production tip: DO NOT open any other ports, like 9200, to the world! There are many bots that search for 9200 and execute groovy scripts to overtake machines. DO NOT bind Elasticsearch to a public IP.

Installing Logstash

Logstash is an open-source tool that collects, parses, and stores logs for future use and makes rapid log analysis possible. Logstash is useful for both aggregating logs from multiple sources, like a cluster of Docker instances, and parsing them from text lines into a structured format such as JSON. In the ELK Stack, Logstash uses Elasticsearch to store and index logs.

Installing Java

Logstash requires the installation of Java 8 or Java 11:

sudo apt-get install default-jre

Verify that Java is installed:

java -version

If the output of the previous command is similar to this, then you’ll know that you’re heading in the right direction:

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Install Logstash with:

sudo apt-get install logstash

Example pipeline: Collect Apache Access Logs with Logstash

For the purpose of this tutorial, we’ve prepared some sample data containing Apache access logs that is refreshed daily. You can download the data here: https://logz.io/sample-data

Create a Logstash configuration file:

sudo vim /etc/logstash/conf.d/apache-01.conf

Enter the following configuration:

input {
  file {
    path => "/home/ubuntu/apache-daily-access.log"
  start_position => "beginning"
  sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  geoip {
    source => "clientip"
  }
}

output {
  elasticsearch { 
  hosts => ["localhost:9200"] 
  }
}

This file is telling Logstash to collect the local /home/ubuntu/apache-daily-access.log file and send it to Elasticsearch for indexing.

The input section specifies which files to collect (path) and what format to expect. The filter section is telling Logstash how to process the data using the grok, date and geoip filters. The output section defines where Logstash is to ship the data to – in this case, a local Elasticsearch.

In this example, we are using localhost for the Elasticsearch hostname. In a real production setup, however, the Elasticsearch hostname would be different because Logstash and Elasticsearch should be hosted on different machines.

Production tip: Running Logstash and Elasticsearch is a very common pitfall of the ELK stack and often causes servers to fail in production. You can read some more tip on how to install ELK in production.

Finally, start Logstash to read the configuration:

sudo service logstash start

To make sure the data is being indexed, use:

sudo curl -XGET 'localhost:9200/_cat/indices?v&pretty'

You should see your new Logstash index created:

health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2019.04.16-000001 rfA5aGYBTP6j27opDwD8VA   1   1          4168            0       230b           230b

You can set up your own ELK stack using this guide or try out our simple ELK as a Service solution.

Installing Kibana

Kibana is an open-source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line, and scatter plots; pie charts; and maps on top of large volumes of data.

Among other uses, Kibana makes working with logs super easy and even fun, and its graphical web interface lets beginners execute powerful log searches.

To install Kibana, use this command:

sudo apt-get install kibana

Open the Kibana configuration file and enter the following configurations:

sudo vim /etc/kibana/kibana.yml

server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"

Start Kibana:

sudo service kibana start

Test:

Point your browser to ‘http://YOUR_ELASTIC_IP:5601’ after Kibana is started (this may take a few minutes).

You should see a page similar to this:

Your next step in Kibana is to define an Elasticsearch index pattern.

What does an “index pattern” mean, and why do we have to configure it? Logstash creates a new Elasticsearch index (database) every day. The names of the indices look like this: logstash-YYYY.MM.DD — for example, “logstash-2019.04.16” for the index we created above on April 16, 2019.

Kibana works on top of these Elasticsearch indices, so it needs to know which one you want to use. Go to Management -> Kibana Index Patterns. Kibana automatically identifies the Logstash index, so all you have to do is define it with ‘logstash-*:

In the next step, we will select the @timestamp timestamp field, and then click the “Create index pattern” button to define the pattern in Kibana.

Production tip: In this tutorial, we are accessing Kibana directly through its application server on port 5601, but in a production environment you might want to put a reverse proxy server, like Nginx, in front of it.

To see your logs, go to the Discover page in Kibana:

As you can see, creating a whole pipeline of log shipping, storing, and viewing is not such a tough task. In the past, storing, and analyzing logs was an arcane art that required the manipulation of huge, unstructured text files. But the future looks much brighter and simpler.

Load balancers play a key component in any cloud-based deployment. By distributing incoming traffic across backend servers or services, load balancers help improve responsiveness and increase the availability of your applications.

Monitoring load balancers is important for analyzing traffic patterns and troubleshooting performance and availability issues. In this article we’ll take a look at Application Gateway — Azure’s load balancer and how to monitor it using Logz.io’s Azure integration.

What is Azure Application Gateway?

To quote the official docs, “Azure Application Gateway is a web traffic load balancer that enables you to manage traffic to your web applications.” Application Gateway is deployed between the client and the applications and services deployed in the backend, and supports routing based on IPs/ports (layer 4 load balancing) and on the request content (layer 7 load balancing).

Source: Azure Docs

Azure’s Application Gateway also supports autoscaling, SSL termination, redirections and some additional features we’re used to seeing in cloud-based load balancing services.

Types of Application Gateway logs

To monitor and troubleshoot Azure’s Application Gateway, there are a number of log types made available for use. These logs need to be turned on, shipped to a storage endpoint (a storage account, an Event Hub or Azure Monitor Logs), and then extracted for analysis:

Access logs

Access logs contain important information on the traffic routed by Application Gateway, including the caller’s IP, requested URL, response latency, return code, and bytes in and out.

Performance logs

Performance logs help you monitor the performance of your Application Gateway instances, providing information on the total requests served, throughput in bytes, total requests served, failed request count, and healthy and unhealthy back-end instance count.

Activity logs

Activity logs will give you visibility into any action performed within your Azure subscription, including those related to Application Gateway. I recommend reading this article to learn how to collect and analyze these logs.  

Collecting and shipping Azure Application Gateway Logs

Now that we understand what we’re monitoring, it’s time to establish a logging pipeline. As mentioned above, we’ll be using the Logz.io Azure integration that is based on Event Hub and Azure Functions. To read more about this integration, take a look at this article.

Deploying the integration template

The easiest way to setup the integration is to use the Deploy to Azure button displayed in the first step of the repo’s readme:

The Custom Deployment page opens up in the Azure portal:

Verify the following entries:

• Resource group: Select the same resource group in which Application Gateway is deployed.
• Location: Verify that the same region used for Application Gateway is entered.
• Logzio Logs Host: Enter the URL of the Logz.io listener. If you’re not sure what this URL is, check your login URL – if it’s app.logz.io, use listener.logz.io (this is the default setting). If it’s app-eu.logz.io, use listener-eu.logz.io.
• Logzio Metrics Host: Enter the URL of the Logz.io listener. If you’re not sure what this URL is, check your login URL – if it’s app.logz.io, use listener.logz.io (this is the default setting). If it’s app-eu.logz.io, use listener-eu.logz.io.
• Logzio Logs Token: Enter the token of the Logz.io account you want to ship Azure logs to. You can find this token on the account page in the Logz.io UI.
• Logzio Metrics Token: Enter a token for the Logz.io account you want to use for shipping Azure metrics to. You can use the same account used for Azure logs.

Agree to the terms at the bottom of the page, and click Purchase.

Azure will then deploy the template. This may take a while as there is a long list of resources to be deployed, but after a minute or two, you will see the Deployment succeeded message at the top of the portal.

Turning on Azure Application Gateway logs

Now that we have all the building blocks in place for streaming the data into Logz.io, our next step is to turn on the logs and make sure they are shipped into the event hub we just created as part of the integration with Logz.io.

In the Azure portal, open up your Application Gateway and click Diagnostic settings from the menu:

Click the Turn on diagnostics link. The Diagnostic settings page opens up and we now need to do two things. First, select the Stream to an event hub option and configure the event hub to ship logs to. Second, click the log types we want to collect and ship.

Save the settings.

Azure will apply the diagnostic settings and within a minute or two you will start to see Application Gateway logs in Logz.io.

Analyzing Azure Application Gateway Logs

As mentioned above, there is a wealth of information available in the different log types collected from an Application Gateway instance.

Before you begin analyzing the data, a good best practice is to try and understand exactly what you’re looking at, or in other words — the fields the logs contain. The docs contain a full description of each field and examples of the logs in JSON format, so that’s a good resource to use for this purpose.

Since all the logs are shipped into the system together, being able to easily search per log type can be helpful. For example, if I only want to look at access logs, I can use the operationName field to look only at access logs:

Save the search for future use.

Adding some fields by selecting them from the list of available fields on the left helps us get more visibility into the data. For example, in the example below I added the httpMethod, httpStatus and requestUri fields:

Using different types of queries, you can then search for specific events.

For example, say you want to look for traffic resulting in a 400 and above response:

operationName:"ApplicationGatewayAccess"  AND properties.httpStatus>= 
400

Kibana supports rich querying options that will help you dive deeper into the rabbit hole. To learn about the different query types, read this post.

Visualizing Azure Application Gateway Logs

Of course, Kibana is well known for its visualization capabilities and once you’ve gained a better understanding of the data collected in Application Gateway logs, you can start building visualizations. Again, there is a wide variety of options to play around with and I’ll provide you with some examples here.

Total requests

Metric visualizations in Kibana are great for displaying single metrics on a key metric. In this example, we’re showing the total amount of requests handled by our Application Gateway instance:

Incoming vs. outgoing traffic

To monitor the sent vs. received amount of bytes, we can use a line chart visualization. We’ll use two average aggregations of the relevant fields as the Y axis and a date histogram as the X axis:

Top URIs

Using data tables is a great way to display the top requested URIs handled by Application Gateway:

Once you have all your visualizations ready, you can combine them into one comprehensive dashboard that gives you a nice overview of your Application Gateway instances:

Again, these are just examples. You can slice and dice the different logs and metrics collected from Azure using Kibana’s rich analysis capabilities. The dashboard here is available, if you like, in ELK Apps — Logz.io’s library of pre-made Kibana objects.

Endnotes

Azure Application Gateway logs contains a lot of valuable information on how load balancers are behaving and the traffic they are routing. You can monitor latencies, anomalies in traffic load, amount of failed requests and plenty more. The challenge is how to aggregate the data efficiently and analyze it effectively.

The integration with Logz.io is simple to implement and will provide you with the analysis tools to help you make sure traffic is being handled in an optimized manner and give you the tools to perform root cause analysis should an issue occur.

Elasticsearch is the living heart of what is today’s the most popular log analytics platform — the ELK Stack (Elasticsearch, Logstash and Kibana). The role played by Elasticsearch is so central that it has become synonymous with the name of the stack itself. Used primarily for search and log analysis, Elasticsearch is today one of the most popular database systems available today.

This guide provides new users with the knowledge and tools needed to get started with Elasticsearch, and includes installation instructions, and initial indexing and data handling instructions.

What is Elasticsearch?

Initially released in 2010, Elasticsearch is a modern search and analytics engine which is based on Apache Lucene. Completely open source and built with Java, Elasticsearch is categorized as a NoSQL database. That means it stores data in an unstructured way and that you cannot use SQL to query it.

Unlike most NoSQL databases, though, Elasticsearch has a strong focus on search capabilities and features — so much so, in fact, that the easiest way to get data from Elasticsearch is to search for it using its extensive REST API.

In the context of data analysis, Elasticsearch is used together with the other components in the ELK Stack, Logstash and Kibana, and plays the role of data indexing and storage.

Installing Elasticsearch

The requirements for Elasticsearch are simple: Java 8 (specific version recommended: Oracle JDK version 1.8.0_131). Take a look at this Logstash tutorial to ensure that you are set. Also, you will want to make sure your operating system is on the Elastic support matrix, otherwise you might run up against strange and unpredictable issues. Once that is done, you can start by installing Elasticsearch.

Elasticsearch can be downloaded as a standalone distribution or installed using the apt and yum repositories. We will be installing Elasticsearch on an Ubuntu 16.04 machine running on AWS EC2 using apt.

First, you need to add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

For Debian, we need to then install the apt-transport-https package:

sudo apt-get install apt-transport-https

The next step is to add the repository definition to your system:

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

All that’s left to do is to update your repositories and install Elasticsearch:

sudo apt-get update
sudo apt-get install elasticsearch

Configuring Elasticsearch

Elasticsearch configurations are done using a configuration file that is located in different locations depending on your operating system. In this file, you can configure general settings (e.g. node name), as well as network settings (e.g. host and port), where data is stored, memory, log files, and more.

For development and testing purposes, the default settings will suffice yet it is recommended you do some research into what settings can be manually defined before going into production.

For example, and especially if installing Elasticsearch on the cloud, it is a good best practice to bind Elasticsearch to either a private IP or localhost:

sudo vim /etc/elasticsearch/elasticsearch.yml

network.host: "localhost"
http.port:9200

Running Elasticsearch

Elasticsearch will not run automatically after installation and you will need to manually start it. How you run Elasticsearch will depend on your specific system. On most Linux and Unix-based systems you can use this command:

sudo service elasticsearch start

And that’s it! To confirm that everything is working fine, simply point curl or your browser to http://localhost:9200, and you should see something like the following output:

{
  "name" : "33QdmXw",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "mTkBe_AlSZGbX-vDIe_vZQ",
  "version" : {
    "number" : "6.1.2",
    "build_hash" : "5b1fea5",
    "build_date" : "2018-01-10T02:35:59.208Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

To debug the process of running Elasticsearch, use the Elasticsearch log files located (on Deb) in /var/log/elasticsearch/.

Creating an Index in Elasticsearch

The process of adding data to Elasticsearch is called “indexing.” This is because when you feed data into Elasticsearch, the data is placed into Apache Lucene indexes. This makes sense because Elasticsearch uses the Lucene indexes to store and retrieve its data. Although you do not need to know a lot about Lucene, it does help to know how it works when you start getting serious with Elasticsearch.

Elasticsearch behaves like a REST API, so you can use either the POST or the PUT method to add data to it. You use PUT when you know the or want to specify the ID of the data item, or POST if you want Elasticsearch to generate an ID for the data item:

curl -XPOST 'localhost:9200/logs/my_app' -H 'Content-Type: application/json' -d' 
{
	"timestamp": "2018-01-24 12:34:56",
	"message": "User logged in",
	"user_id": 4,
	"admin": false
}
'

curl -X PUT 'localhost:9200/app/users/4' -H 'Content-Type: application/json' -d '
{
  "id": 4, 
  "username": "john", 
  "last_login": "2018-01-25 12:34:56"
}
'

And the response:

{"_index":"logs","_type":"my_app","_id":"ZsWdJ2EBir6MIbMWSMyF","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

{"_index":"app","_type":"users","_id":"4","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

The data for the document is sent as a JSON object. You might be wondering how we can index data without defining the structure of the data. Well, with Elasticsearch, like with most other NoSQL databases, there is no need to define the structure of the data beforehand. To ensure optimal performance, though, you can define mappings for data types. More on this later.

If you are using any of the Beats shippers (e.g. Filebeat or Metricbeat), or Logstash — the indices are automatically created.

To see a list of your Elasticsearch indices, use:

curl -XGET 'localhost:9200/_cat/indices?v&pretty'

health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2018.01.23 y_-PguqyQ02qOqKiO6mkfA   5   1      17279            0      9.9mb          9.9mb
yellow open   app                 GhzBirb-TKSUFLCZTCy-xg   5   1          1            0      5.2kb          5.2kb
yellow open   .kibana             Vne6TTWgTVeAHCSgSboa7Q   1   1          2            0      8.8kb          8.8kb
yellow open   logs                T9E6EdbMSxa8S_B7SDabTA   5   1          1            0      5.7kb          5.7kb

The list in this case includes the indices we created above, a Kibana index and an index created by a Logstash pipeline.

Elasticsearch Querying

Once you have your data indexed into Elasticsearch, you can start searching and analyzing it. The simplest query you can do is to fetch a single item.

Once again, because Elasticsearch is a REST API, we use GET:

curl -XGET 'localhost:9200/app/users/4?pretty'

And the response:

{
  "_index" : "app",
  "_type" : "users",
  "_id" : "4",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "id" : 4,
    "username" : "john",
    "last_login" : "2018-01-25 12:34:56"
  }
}

The fields starting with an underscore are all meta fields of the result. The _source object is the original document that was indexed.

We also use GET to do searches by calling the _search endpoint:

curl -XGET 'localhost:9200/_search?q=logged'

{"took":173,"timed_out":false,"_shards":{"total":16,"successful":16,"skipped":0,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"logs","_type":"my_app","_id":"ZsWdJ2EBir6MIbMWSMyF","_score":0.2876821,"_source":
{
    "timestamp": "2018-01-24 12:34:56",
    "message": "User logged in",
    "user_id": 4,
    "admin": false
}
}]}}

The result contains a number of extra fields that describe both the search and the result. Here’s a quick rundown:

• took: The time in milliseconds the search took
• timed_out: If the search timed out
• shards: The number of Lucene shards searched, and their success and failure rates
• hits: The actual results, along with meta information for the results

The search we did above is known as a URI Search, and is the simplest way to query Elasticsearch. By providing only a word, all of the fields of all the documents are searched for that word. You can build more specific searches by using Lucene queries:

• username:johnb – Looks for documents where the username field is equal to “johnb”
• john* – Looks for documents that contain terms that start with john and is followed by zero or more characters such as “john,” “johnb,” and “johnson”
• john? – Looks for documents that contain terms that start with john followed by only one character. Matches “johnb” and “johns” but not “john.”

There are many other ways to search including the use of boolean logic, the boosting of terms, the use of fuzzy and proximity searches, and the use of regular expressions.

Elasticsearch Query DSL

URI searches are just the beginning. Elasticsearch also provides a request body search with a Query DSL for more advanced searches. There is a wide array of options available in these kinds of searches, and you can mix and match different options to get the results that you require. Some of the options include geo queries, “more like this” queries, and scripted queries.

The DSL also makes a distinction between a filtering and a query context for query clauses. Clauses used as filters test documents in a boolean fashion: Does the document match the filter, “yes” or “no”? Filters are also generally faster than queries, but queries, can also calculate a score based on how closely a document matches the query. This is used to determine the ordering and inclusion of documents:

curl -XGET 'localhost:9200/logs/_search?pretty' -H 'Content-Type: application/json' -d'
{
  "query": {
    "match_phrase": {
      "message": "User logged in"
    }
  }
}
'

And the result:

{
  "took" : 28,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.8630463,
    "hits" : [
      {
        "_index" : "logs",
        "_type" : "my_app",
        "_id" : "ZsWdJ2EBir6MIbMWSMyF",
        "_score" : 0.8630463,
        "_source" : {
          "timestamp" : "2018-01-24 12:34:56",
          "message" : "User logged in",
          "user_id" : 4,
          "admin" : false
        }
      }
    ]
  }
}
'

Removing Elasticsearch Data

Deleting items from Elasticsearch is just as easy as entering data into Elasticsearch. The HTTP method to use this time is — surprise, surprise! — DELETE:

curl -XDELETE 'localhost:9200/app/users/4?pretty'

{
  "_index" : "app",
  "_type" : "users",
  "_id" : "4",
  "_version" : 2,
  "result" : "deleted",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 1,
  "_primary_term" : 1
}

To delete an index, use:

curl -XDELETE 'localhost:9200/logs?pretty'

To delete all indices (use with extreme caution) use:

curl -XDELETE 'localhost:9200/_all?pretty'

The response in both cases should be:

{

 "acknowledged" : true

}

What’s Next?

This tutorial helps beginners to get started with Elasticsearch and as such provides just the basic steps of CRUD operations in Elasticsearch. Elasticsearch is a search engine, and as such features an immense depth to its search features.

Recent versions have introduced some incredible new Elasticsearch features and also some significant changes to the underlying data structure.

It’s always a good idea to explore the official Elasticsearch documentation as well as our Logstash tutorial and Kibana tutorial.

Logs are unpredictable.

Following a production incident, and precisely when you need them the most, logs can suddenly surge and overwhelm your logging infrastructure. To protect Logstash and Elasticsearch against such data bursts, users deploy buffering mechanisms to act as message brokers.

Apache Kafka is the most common broker solution deployed together the ELK Stack. Usually, Kafka is deployed between the shipper and the indexer, acting as an entrypoint for the data being collected:

In this article, I’ll show how to deploy all the components required to set up a resilient data pipeline with the ELK Stack and Kafka:

• Filebeat – collects logs and forwards them to a Kafka topic.
• Kafka – brokers the data flow and queues it.
• Logstash – aggregates the data from the Kafka topic, processes it and ships to Elasticsearch.
• Elasticsearch – indexes the data.
• Kibana – for analyzing the data.

My environment

To perform the steps below, I set up a single Ubuntu 16.04 machine on AWS EC2 using local storage. In real-life scenarios you will probably have all these components running on separate machines.

I started the instance in the public subnet of a VPC and then set up a security group to enable access from anywhere using SSH and TCP 5601 (for Kibana). Finally, I added a new elastic IP address and associated it with the running instance.

The example logs used for the tutorial are Apache access logs.

Step 1: Installing Elasticsearch

We will start with installing the main component in the stack — Elasticsearch. Since version 7.x, Elasticsearch is bundled with Java so we can jump right ahead with adding Elastic’s signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

For installing Elasticsearch on Debian, we also need to install the apt-transport-https package:

sudo apt-get update
sudo apt-get install apt-transport-https

Our next step is to add the repository definition to our system:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo 
tee -a /etc/apt/sources.list.d/elastic-7.x.list

All that’s left to do is to update your repositories and install Elasticsearch:

sudo apt-get update && sudo apt-get install elasticsearch

Before we bootstrap Elasticsearch, we need to apply some basic configurations using the Elasticsearch configuration file at: /etc/elasticsearch/elasticsearch.yml:

sudo su
vim /etc/elasticsearch/elasticsearch.yml

Since we are installing Elasticsearch on AWS, we will bind Elasticsearch to localhost. Also, we need to define the private IP of our EC2 instance as a master-eligible node:

network.host: "localhost"
http.port:9200
cluster.initial_master_nodes: ["<InstancePrivateIP"]

Save the file and run Elasticsearch with:

sudo service elasticsearch start

To confirm that everything is working as expected, point curl to: http://localhost:9200, and you should see something like the following output (give Elasticsearch a minute or two before you start to worry about not seeing any response):

{
  "name" : "ip-172-31-49-60",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "yP0uMKA6QmCsXQon-rxawQ",
  "version" : {
    "number" : "7.0.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "b7e28a7",
    "build_date" : "2019-04-05T22:55:32.697037Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Step 2: Installing Logstash

Next up, the “L” in ELK — Logstash. Logstash will requires us to install Java 8 which is fine because this is also a requirement for running Kafka:

sudo apt-get install default-jre

Verify java is installed:

java -version

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Since we already defined the repository in the system, all we have to do to install Logstash is run:

sudo apt-get install logstash

Next, we will configure a Logstash pipeline that pulls our logs from a Kafka topic, process these logs and ships them on to Elasticsearch for indexing.

Let’s create a new config file:

sudo vim /etc/logstash/conf.d/apache.conf

Paste the following configurations:

input { 
  kafka { 
    bootstrap_servers => "localhost:9092" 
    topics => "apache" 
    } 
}

filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  geoip {
      source => "clientip"
    }
}

output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
}

As you can see — we’re using the Logstash Kafka input plugin to define the Kafka host and the topic we want Logstash to pull from. We’re applying some filtering to the logs and we’re shipping the data to our local Elasticsearch instance.

Save the file.

Step 3: Installing Kibana

Let’s move on to the next component in the ELK Stack — Kibana. As before, we will use a simple apt command to install Kibana:

sudo apt-get install kibana

We will then open up the Kibana configuration file at: /etc/kibana/kibana.yml, and make sure we have the correct configurations defined:

server.port: 5601
elasticsearch.url: "http://localhost:9200"

These specific configurations tell Kibana which Elasticsearch to connect to and which port to use.

Now, we can start Kibana with:

sudo service kibana start

Open up Kibana in your browser with: http://localhost:5601. You will be presented with the Kibana home page.

Step 4: Installing Filebeat

As mentioned above, we will be using Filebeat to collect the log files and forward them to Kafka.

To install Filebeat, we will use:

sudo apt-get install filebeat

Let’s open the Filebeat configuration file at:

/etc/filebeat/filebeat.yml

sudo vim /etc/filebeat/filebeat.yml

Enter the following configurations:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/access.log

output.kafka:
  codec.format:
    string: '%{[@timestamp]} %{[message]}'
  hosts: ["localhost:9092"]
  topic: apache
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

In the input section, we are telling Filebeat what logs to collect — apache access logs. In the output section, we are telling Filebeat to forward the data to our local Kafka server and the relevant topic (to be installed in the next step).

Note the use of the codec.format directive — this is to make sure the message and timestamp fields are extracted correctly. Otherwise, the lines are sent in JSON to Kafka.

Save the file.

Step 4: Installing Kafka

Our last and final installation involves setting up Apache Kafka — our message broker.

Kafka uses ZooKeeper for maintaining configuration information and synchronization so we’ll need to install ZooKeeper before setting up Kafka:

sudo apt-get install zookeeperd

Next, let’s download and extract Kafka:

wget http://apache.mivzakim.net/kafka/2.2.0/kafka_2.12-2.2.0.tgz
tar -xvzf kafka_2.12-2.2.0.tgz
sudo cp -r kafka_2.12-2.2.0 /opt/kafka

We are now ready to run kafka, which we will do with this script:

sudo /opt/kafka/bin/kafka-server-start.sh 
/opt/kafka/config/server.properties

You should begin to see some INFO messages in your console:

[2019-04-22 11:48:16,489] INFO Registered 
kafka:type=kafka.Log4jController MBean 
(kafka.utils.Log4jControllerRegistration$)
[2019-04-22 11:48:18,589] INFO starting (kafka.server.KafkaServer)

Next, we’re going to create a topic for our Apache logs:

bin/kafka-topics.sh --create --zookeeper localhost:2181 
--replication-factor 1 --partitions 1 --topic apache

Created topic apache.

We are all set to start the pipeline.

Step 5: Starting the data pipeline

Now that we have all the pieces in place, it’s time to start all the components in charge of running the data pipeline.

First, we’ll start Filebeat:

sudo service filebeat start

Then, Logstash:

sudo service logstash start

It will take a few minutes for the pipeline to start streaming logs. To see Kafka in action, enter the following command in a separate tab:

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 
localhost:9092 --topic apache --from-beginning

If your Apache web server is indeed serving requests, you should begin to see the messages being forwarded to the Kafka topic by Filebeat in your console:

2019-04-23T13:50:01.559Z 89.138.90.236 - - [23/Apr/2019:13:50:00 +0000] 
"GET /mysite.html HTTP/1.1" 200 426 "-" "Mozilla/5.0 
(Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/73.0.3683.86 Safari/537.36"
2019-04-23T13:51:36.581Z 89.138.90.236 - - 
[23/Apr/2019:13:51:34 +0000] "GET /mysite.html HTTP/1.1" 200 427 "-" 
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"

To make sure Logstash is aggregating the data and shipping it into Elasticsearch, use:

curl -X GET "localhost:9200/_cat/indices?v"

If all is working as expected, you should see a logstash-* index listed:

health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_task_manager       zmMH6yy8Q6yg2jJHxq3MFA   1   0          2            0     45.4kb         45.4kb
yellow open   logstash-2019.04.23-000001 rBx5r_gIS3W2dTxHzGJVvQ   1   1          9            0     69.4kb         69.4kb
green  open   .kibana_1                  rv5f8uHnQTCGe8YrcKAwlQ   1   0          5            0     29.9kb         29.9kb

If you don’t see this index, it’s time to do some debugging I’m afraid. Check out this blog post for some tips on debugging Logstash.

All we have to do now is define the index pattern in Kibana to begin analysis. This is done under Management → Kibana Index Patterns.

Kibana will identify the index, so simply define it in the relevant field and continue on to the next step of selecting the timestamp field:

Once you create the index pattern, you’ll see a list of all the parsed and mapped fields:

Open the Discover page to begin analyzing your data!

Summing it up

Resilient data pipelines are a must in any production-grade ELK deployments. Logs are crucial for piecing together events and they are needed most in emergencies. We cannot afford to have our logging infrastructure fail at the exact point in time when it is most required. Kafka, and similar brokers, play a huge part in buffering the data flow so Logstash and Elasticsearch don’t cave under the pressure of a sudden burst.

The example above is a basic setup of course. Production deployments will include multiple Kafka instances, a much larger amount of data and much more complicated pipelines. This will involve a considerable amount of engineering time and resources, something to take into consideration. The instructions here will help you understand how to get started though. I also recommend taking a look at our article explaining how to log Kafka itself.

Good luck!

Distributed tracing has become a de-facto standard for monitoring distributed architectures, helping engineers to pinpoint errors and identify performance bottlenecks. Zipkin is one of the popular open source “tracers” available in the market, and I’m now happy to inform our users that we’ve recently introduced a new integration that allows users to easily ship trace data collected by Zipkin to Logz.io! 

A few words on Zipkin

Zipkin is a Java-based app used for distributed tracing and identifying latency issues. Unique identifiers are automatically attached to requests which are then passed downstream through the different waypoints, or services. Zipkin collects this data and allows users to analyze it in a UI.

The Zipkin UI provides users with some basic analysis options for monitoring and troubleshooting requests, such as a dependency diagram to view the execution flow, filtering and sorting the traces collected by Zipkin per application, the length of trace and timestamp. Using Logz.io, however, opens up a whole lot of other options, such as full querying, additional visualization options, alerting, and plenty more.

Let’s take a look at how to set up the integration and enjoy some of this goodness.

Setting up Zipkin and Logz.io

First, you will need a Logz.io API token to set up the integration (Note: API is only available for Enterprise users). This can be done by clicking the cogwheel in the top-right corner of the Logz.io UI and selecting Tools → API tokens:

If you haven’t already downloaded Zipkin, do so with:

wget -O zipkin.jar 
'https://search.maven.org/remote_content?g=io.zipkin.java&a=zipkin-server
&v=LATEST&c=exec'

Next, download the Logz.io extension into the same directory you downloaded the zipkin.jar (look for the most recent version here: https://github.com/logzio/zipkin-logzio/releases):

wget -O logzio-zipkin.jar 
https://github.com/logzio/zipkin-
logzio/releases/download/v0.0.1/zipkin-logzio-0.0.1.jar

What’s left to do now is run Zipkin with the following parameters:

• STORAGE_TYPE – the storage type you want Zipkin to use, in our case: logzio
• LOGZIO_ACCOUNT_TOKEN –  the Logz.io account token. Can be retrieved by clicking the cogwheel in the top-right corner of the Logz.io UI and selecting Settings → General.
• LOGZIO_API_TOKEN – the Logz.io API token you created in the previous step.

For information on additional run options, please refer to the integration’s repo.

The complete run command should look something like this (be sure to change the name of the .jar file if necessary and replace the token placeholders):

STORAGE_TYPE=logzio \
LOGZIO_ACCOUNT_TOKEN=<yourLogzioAccountToken> \
LOGZIO_API_TOKEN=<yourLogzioAPIToken> \
java -Dloader.path='logzio-zipkin.jar,logzio-zipkin.jar!lib' -Dspring.profiles.active=logzio -cp zipkin.jar org.springframework.boot.loader.PropertiesLauncher

To verify Zipkin is installed and running as expected, open it at http://localhost:9411: Shipping

Shipping trace data to Logz.io

As Zipkin users probably already know, most applications can be traced using built-in and community instrumentations. For the sake of demonstrating how to hook up Zipkin with Logz.io, though, I’ll use an instrumentation library for Java applications called Brave to track http communication between two Java servlet services.

To download the source code, use:

git clone https://github.com/openzipkin/brave-webmvc-example.git

In this example, there are two services that send tracing info to Zipkin — a backend service and a frontend service.

The start the front end service:

cd /brave-webmvc-example/webmvc4
mvn jetty:run -Pfrontend

And now the backend service:

cd /brave-webmvc-example/webmvc4
mvn jetty:run -Pbackend

Both services should report the same success message in the run output:

[INFO] Started Jetty Server

It’s time to send out some requests. The front end service can be accessed via http://localhost:8081 and calls the backend service and displays a timestamp.

Within a few seconds you’ll see the trace data appear in Logz.io:

Analyzing traces in Logz.io

Again, the benefit of using Logz.io and Kibana to analyze the trace data tracked by Zipkin is the analysis capabilities. There is a wide variety of search types to use to query the data, including free text searches, field-level searches and REGEX-based queries.  

Average service duration

Using an average aggregation of the duration field together with a grouping of the service name, you can create a graph with Kibana’s visual builder that shows the average time for your different services.

Trace list

You can create a data table visualization that gives you a nice breakdown of all the traces collected from your application by Zipkin and stored in Logz.io.

To do this you would need to use a sum metric aggregation of the ‘duration’ field and a number of basic bucket aggregations as depicted below.

You can then add these up into one dashboard that provides a nice overview of your trace data:

Endnotes

Logs, metrics and traces are considered as the three data types used to reach observability of a system and hooking up Logz.io with Zipkin allows you to unify them under one platform.

Yes, at version 0.0.1, this is an initial release. But since we are heavy Zipkin users ourselves, we believe in the benefits of consolidated tooling and wanted to share the news as soon as possible. We’d love to get your feedback, so please feel free to fork, PR and submit issues.

Happy tracing!

Kibana is the visualization layer of the ELK Stack — the world’s most popular log analysis platform which is comprised of Elasticsearch, Logstash, and Kibana.

This tutorial will guide you through some of the basic steps for getting started with Kibana — installing Kibana, defining your first index pattern, and running searches. Examples are provided throughout, as well as tips and best practices.

Installing Kibana

Presuming you already have Elasticsearch installed and configured, we will start with installing Kibana. If you want to find out more about installing Elasticsearch, check out this Elasticsearch tutorial.

Depending on your operating system and your environment, there are various ways of installing Kibana. We will be installing Kibana on an Ubuntu 16.04 machine running on AWS EC2 on which Elasticsearch and Logstash are already installed.

Start by downloading and installing the Elastic public signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key 
add -

Add the repository definition:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo 
tee -a /etc/apt/sources.list.d/elastic-7.x.list

It’s worth noting that there is another package containing only features available under the Apache 2.0 license.  To install this package, use:

echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | 
sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

All that’s left to do is to update your repositories and install Kibana:

sudo apt-get update
sudo apt-get install kibana

Open up the Kibana configuration file at: /etc/kibana/kibana.yml, and make sure you have the following configurations defined:

server.port: 5601

elasticsearch.url: "http://localhost:9200"

These specific configurations tell Kibana which Elasticsearch to connect to and which port to use.

Now, start Kibana with:

sudo service kibana start

Open up Kibana in your browser with: http://<yourServerIP>:5601. You will be presented with the Kibana home page.

Defining an index pattern

Your next step is to define a new index pattern, or in other words, tell Kibana what Elasticsearch index to analyze. To do that you will of course need to have data indexed.

For the purpose of this tutorial, we’ve prepared some sample data containing Apache access logs that is refreshed daily. You can download the data here: https://logz.io/sample-data

Next, we will use Logstash to collect, parse and ship this data into Elasticsearch. If you haven’t installed Logstash yet, or are not familiar with how to use it, check out this Logstash tutorial.

Create a new Logstash configuration file at: /etc/logstash/conf.d/apache-01.conf:

sudo vim /etc/logstash/conf.d/apache-01.conf

Enter the following Logstash configuration (change the path to the file you downloaded accordingly):

input {
  file {
    path => "/home/ubuntu/apache-daily-access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
    geoip {
      source => "clientip"
    }
}

output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
}

Start Logstash with:

sudo service logstash start

If all goes well, a new index will be created in Elasticsearch, the pattern of which can now be defined in Kibana.

In Kibana, go to Management → Kibana Index Patterns, and Kibana will automatically identify the new “logstash-*” index pattern.  

Define it as “logstash-*”, and in the next step select @timestamp as your Time Filter field.

Hit Create index pattern, and you are ready to analyze the data. Go to the Discover tab in Kibana to take a look at the data (look at today’s data instead of the default last 15 mins).

Using Kibana in Logz.io

If you’re using Logz.io, simply use this cURL command to upload the sample log data. Logz.io listeners will parse the data using automatic parsing so there’s no need to configure Logstash (the token can be found on the Settings page in the Logz.io UI, and the type of the file is apache_access):

curl -T <Full path to file> 
http://listener.logz.io:8021/file_upload/<Token>/apache_access

Kibana Searching

Kibana querying is an art unto itself, and there are various methods for performing searches on your data. This section will describe some of the most common search methods as well as some tips and best practices that should be memorized for optimized user experience.

KQL and Lucene

Up until version 6.2, the only way to query in Kibana was using Lucene syntax. Starting in version 6.2, another query language was introduced called Kuery, or as it’s been called now — KQL (Kibana Querying Language) to improve the searching experience.

Since version 7.0, KQL is the default language for querying in Kibana but you can revert to Lucene if you like. For the basic example below, there will be little difference in the search results.

Free-Text Search

Free text search works within all fields — including the _source field, which includes all the other fields. If no specific field is indicated in the search, the search will be done on all of the fields that are being analyzed.

In the search field at the top of the Discover page, run these searches and examine the result ( set the time parameter on the top right of the dashboard to the past month to capture more data):

• category
• Category
• categ
• cat*
• categ?ry
• “category”
• category\/health
• “category/health”
• Chrome
• chorm*

Tips and gotchas

1. Text searches are not case sensitive. This means that category and CaTeGory will return the same results. When you put the text within double quotes (“”), you are looking for an exact match, which means that the exact string must match what is inside the double quotes. This is why [category\/health] and [“category/health”] will return different results
2. Kibana wildcard searches – you can use the wildcard symbols [*] or [?] in searches. [*] means any number of characters, and [?] means only one character

Field-Level Searches

Another common search in Kibana is field-level queries, sued for searching for data inside specific fields. To use this type of search that, you need to use the following format:

<fieldname>:search

As before, run the following searches to see what you get (some will purposely return no results):

• name:chrome
• name:Chrome
• name:Chr*
• response:200
• bytes:65
• bytes:[65 TO *]
• bytes:[65 TO 99]
• bytes:{65 TO 99}
• _exists_:name

Tips and gotchas

1. Field-level searches depend on the type of field. (Logz.io users – all fields are not analyzed by default, which means that searches are case-sensitive and cannot use wildcard searches. The reason we save all of the fields as “not analyzed” is to save space in the index because the data is also duplicated in an analyzed field called _source)
2. You can search a range within a field. If you use [], this means that the results are inclusive. If you use {}, this means that the results are exclusive.
3. Using the _exists_ prefix for a field will search the documents to see if the field exists
4. When using a range, you need to follow a very strict format and use capital letters TO to specify the range

Logical Statements

You can use logical statements in searches in these ways:

• USA AND Firefox
• USA OR Firefox
• (USA AND Firefox) OR Windows
• -USA
• !USA
• +USA
• NOT USA

Tips and gotchas

1. You need to make sure that you use the proper format such as capital letters to define logical terms like AND or OR
2. You can use parentheses to define complex, logical statements
3. You can use -,! and NOT to define negative terms

Kibana special characters

All special characters need to be properly escaped. The following is a list of all available special characters:

+ – && || ! ( ) { } [ ] ^ ” ~ * ? : \

Proximity searches

Proximity searches are an advanced feature of Kibana that takes advantage of the Lucene query language.

[categovi~2] means a search for all the terms that are within two changes from [categovi]. This means that all category will be matched.

Tips and gotchas

Proximity searches use a lot of system resources and often trigger internal circuit breakers in Elasticsearch. If you try something such as [catefujt~10], it is likely not to return any results due to the amount of memory used to perform this specific search.

Kibana Autocomplete

To help improve the search experience in Kibana, the autocomplete feature suggests search syntax as you enter your query. As you type, relevant fields are displayed and you can complete the query with just a few clicks. This speeds up the whole process and makes Kibana querying a whole lot simpler.

Kibana Filtering

To assist users in searches, Kibana includes a filtering dialog that allows easier filtering of the data displayed in the main view.

To use the dialog, simply click the Add a filter + button under the search box and begin experimenting with the conditionals. Filters can be pinned to the Discover page, named using custom labels, enabled/disabled and inverted.

Power users can also enter Elasticsearch queries using the Query DSL.

Summary

This tutorial covered the basics of setting up and using Kibana and provided the steps for setting up a test environment only. Tracking Apache access logs in production, for example, is better done using Filebeat and the supplied Apache module. The sample data provided can, of course, be replaced with other types of data, as you see fit. Kibana also provides sets of sample data to play around with, including flight data and web logs.

If you have any suggestions on what else should be included in the first part of this Kibana tutorial, please let me know in the comments below. The next Kibana tutorial will cover visualizations and dashboards.

Enjoy!

As you may very well know, Kibana currently has almost 20 different visualization types to choose from. This gives you a wide array of options to slice and dice your logs and metrics, and yet there are some cases where you might want to go beyond what is provided in these different visualizations and develop your own kind of visualization.

In the past, extending Kibana with customized visualizations meant building a Kibana plugin, but since version 6.2, users can accomplish the same goal more easily and from within Kibana using Vega and Vega-Lite — an open source, and relatively easy-to-use, JSON-based declarative languages. In this article, I’m going to go show some basic examples of how you can use these frameworks to extend Kibana’s visualization capabilities.

Vega and Vega-Lite

Quoting the official docs, Vega is a “visualization grammar, a declarative language for creating, saving, and sharing interactive visualization designs.” Vega allows developers to define the exact visual appearance and interactive behavior of a visualization. Among the supported designs are scales, map projections, data loading and transformation, and more.

Vega-Lite is a lighter version of Vega, providing users with a “concise JSON syntax for rapidly generating visualizations to support analysis”. Compared to Vega, Vega-Lite is simpler to use, helps automate some of the commands and uses shorter specifications. Some visualizations, however, cannot be created with Vega-Lite and we’ll show an example below.

Setting up the environment

For the purpose of this article, we deployed Elasticsearch and Kibana 7.1 on an Ubuntu 18.04 EC2 instance. The dataset used for the examples are the web sample logs available for use in Kibana. To use this data, simply gop to Kibana’s homepage and click the relevant link to install sample data.

Example 1: Creating a custom bar visualization

Like any programming language, Vega and Vega-Lite have a precise syntax to follow. I warmly recommend to those exploring these frameworks to check out the documentation.

For the sake of understanding the basics though, I’ll provide a very simple Vega-Lite configuration that visualizes the number of requests being sent to our Apache server over time. Yes, I know, the same can easily be done with the provided bar/area/line charts in Kibana but the purpose here is to illustrate the basics of Vega-Lite declarations, so please bare with me.  

As seen in the example below, the JSON syntax contains the following specifications:

• $schema – URL to Vega-Lite JSON schema
• title – a title of the visualization
• data – an Elasticsearch query defining the dataset used for the visualization (e.g. the Elasticsearch index, timestamp, etc.)
• mark –  the visual element type we want to use (e.g. circle, point, rule)
• encoding –  instructs the mark on what data to use and how

Example (full):

{
  $schema: https://vega.github.io/schema/vega-lite/v2.json
  title: Requests
  data: {
    url: {
      %context%: true
      %timefield%: @timestamp
      index: kibana_sample_data_logs
      body: {
        aggs: {
          time_buckets: {
            date_histogram: {
              field: @timestamp
              interval: {%autointerval%: true}
              extended_bounds: {
                min: {%timefilter%: "min"}
                max: {%timefilter%: "max"}
              }
              min_doc_count: 0
            }
          }
        }
        size: 0
      }
    }
    format: {property: "aggregations.time_buckets.buckets"}
  }
  mark: bar
  encoding: {
    x: {
      field: key
      type: temporal
      axis: {title: "Time"}
    }
    y: {
      field: doc_count
      type: quantitative
      axis: {title: "No. of requests"}
    }
  }
}

Vega-Lite of course can be used for much more than this simple chart. You can actually transform data, play with different layers of data, and plenty more.

Example 2: Creating a Sankey chart visualization

Our next example is a bit more advanced — a Sankey chart that displays the flow of requests from their geographic source to their geographic destination.  

Sankey charts are great for visualizing the flow of data and consists of three main elements: nodes, links, and instructions. Nodes are the visualization elements at both ends of the flow — in the case of our Apache logs — the source and destination countries for requests. Links are the elements that connect the nodes, helping us visualize the data flow from source to destination. Instructions are the commands, or the lines of JSON, which help us transform the Sankey Chart according to our needs.

In the example below we are using Vega since Vega-Lite does not support the creation of Sankey Charts just yet. Again, I can’t go into all the details of the configuration but will outline some of the main high-level components:

• $schema – URL to Vega JSON schema
• data – the dataset used for the visualization
• transform – various processing applied to the data stream (e.g. filter, formula, linkpath)
• scales – calculates the stack positioning in the chart
• marks – connects the lines between the stacks
• axes – visualizes spatial scale mappings with ticks
• signals – highlights the traffic to/from the source/destination

Example (shortened):

The full JSON configuration of the Sankey chart displayed below is simply too long to share in this article. You can take a look at it here.

Note the Vega schema used here, as well as the two sources based on the geo.src and geo.dest fields in our index.

Schema/data declaration

{
  "$schema": "https://vega.github.io/schema/vega/v3.0.json",
  "data": [
    {
      "name": "rawData",
      "url": {
        "%context%": true,
        "%timefield%": "timestamp",
        "index": "kibana_sample_data_logs",
        "body": {
          "size": 0,
          "aggs": {
            "table": {
              "composite": {
                "size": 10000,
                "sources": [
                  {
                    "stk1": {
                      "terms": {
                        "field": "geo.src"
                      }
                    }
                  },
                  {
                    "stk2": {
                      "terms": {
                        "field": "geo.dest"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      },
….                  
Axes
....
"axes": [
    {
      "orient": "bottom",
      "scale": "x",
      "encode": {
        "labels": {
          "update": {
            "text": {
              "scale": "stackNames",
              "field": "value"
            }
          }
        }
      }
    },
    {"orient": "left", "scale": "y"}
  ],
….

Marks

….

"marks": [
    {
      "type": "path",
      "name": "edgeMark",
      "from": {"data": "edges"},
      "clip": true,
      "encode": {
        "update": {
          "stroke": [
            {
              "test": "groupSelector && groupSelector.stack=='stk1'",
              "scale": "color",
              "field": "stk2"
            },
            {
              "scale": "color",
              "field": "stk1"
            }
          ],
          "strokeWidth": {
            "field": "strokeWidth"
          },
          "path": {"field": "path"},
          "strokeOpacity": {
            "signal": "!groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 0.9 : 0.3"
          },
          "zindex": {
            "signal": "!groupSelector && (groupHover.stk1 == datum.stk1 || groupHover.stk2 == datum.stk2) ? 1 : 0"
          },
          "tooltip": {
            "signal": "datum.stk1 + ' → ' + datum.stk2 + '\t' + format(datum.size, ',.0f') + '   (' + format(datum.percentage, '.1%') + ')'"
          }
        },
        "hover": {
          "strokeOpacity": {"value": 1}
        }
      }
    },
….

The final outcome of this JSON when we run it should look something like this:

Endnotes

Kibana is a fantastic tool for visualizing your logs and metrics and offers a wide array of different visualization types to select from. If you’re a Kibana newbie, the provided visualizations will most likely suffice.

But more advanced users might be interested in exploring the two frameworks reviewed above as they extend Kibana with further options, opening up a world of visualization goodness with scatter plots, violin plots, sunbursts and a whole lot more.

Allow me to end with some small pointers to take into consideration before you dive into the world of Vega and Vega-Lite for creating your beautiful custom visualizations:

• Vega and Vega-Lite visualizations in Kibana are still defined as experimental.
• You will need to have a good understanding of ELK, especially Elasticsearch and Kibana.
• Basic JSON skills are definitely a plus as it will make configuring the visualization much simpler.

If you’re still unsure whether it’s worth the effort, check out the available examples.

Enjoy!

The ELK Stack, which traditionally consisted of three main components — Elasticsearch, Logstash and Kibana, has long departed from this composition and can now also be used in conjunction with a fourth element called “Beats” — a family of log shippers for different use cases. It is this departure that has led to the stack being renamed as the Elastic Stack.

This tutorial provides a guide for those just getting acquainted with the stack, and provides information for getting started working with the different beats — Filebeat, Packetbeat, Metricbeat, Auditbeat, Heartbeat, Winlogbeat, and Functionbeat.

A brief history lesson

In centralized logging, a data pipeline consists of three main stages: aggregation, processing, and storage. In the ELK Stack, the first two stages were traditionally the responsibility of Logstash, the stack’s work horse.

Executing these tasks comes at a cost. Due to inherent issues related to how Logstash was designed, performance issues became a frequent occurrence, especially with complicated pipelines that require a large amount of processing.

The idea of outsourcing some of Logstash’s responsibilities also came into being, especially the task of data extraction, to other tools. As I describe in this article, the idea first manifested itself in Lumberjack and later in the Logstash forwarder. Eventually, and a few cycles of development later, a new and improved protocol was introduced that became the backbone of what is now called the “Beats” family.

What are Beats?

Beats are a collection of lightweight (resource efficient, no dependencies, small) and open source log shippers that act as agents installed on the different servers in your infrastructure for collecting logs or metrics.

These can be log files (Filebeat), network data (Packetbeat), server metrics (Metricbeat), or any other type of data that can be collected by the growing number of Beats being developed by both Elastic and the community. Once collected, the data is sent either directly into Elasticsearch or to Logstash for additional processing.

Beats are built on top of a Go framework called libbeat — a library for data forwarding which means that new beats are being developed and contributed by the community all the time.

Filebeat

Filebeat, as its name implies, is used for collecting and shipping log files, and is also the most commonly used beat. One of the facts that make Filebeat so efficient is the way it handles backpressure — so if Logstash is busy, Filebeat slows down it’s read rate and picks up the beat once the slowdown is over.

Filebeat can be installed on almost any operating system, including as a Docker container, and also comes with internal modules for specific platforms such as Apache, MySQL, Docker and more, containing default configurations and Kibana objects for these platforms.

Packetbeat

A network packet analyzer, Packetbeat was the first beat introduced. Packetbeat captures network traffic between servers, and as such can be used for application and performance monitoring.

Packetbeat can be installed on the server being monitored or on its own dedicated server.  Packetbeat tracks the network traffic, decodes the protocols, and records data for each transaction. The protocols supported by Packetbeat include: DNS, HTTP, ICMP, Redis, MySQL, MongoDB, Cassandra, and many more.

Read more about how to use Packetbeat here.

Metricbeat

Metricbeat is an extremely popular beat that collects and reports various system-level metrics for various systems and platforms. Metricbeat also supports internal modules for collecting statistics from specific platforms.

You can configure the frequency in which Metricbeat collects the metrics and what specific metrics to collect using these modules and sub-settings called metricsets.

Read more about how to use Metricbeat here.

Heartbeat

Heartbeat is meant for “uptime monitoring.” In essence, what Heartbeat does is probe services to check if they are reachable or not — it’s useful, for example, to verify that the service uptime complies with your SLA. All you need to do is supply Heartbeat with a list of URLs and uptime metrics to send to your stack, either directly to Elasticsearch or to Logstash for enrichment before indexing.

Read more about how to use Heartbeat here.

Auditbeat

Auditbeat can be used for auditing user and process activity on your Linux servers. Similar to other traditional system auditing tools (systemd, auditd), Auditbeat can be used to identify security breaches — file changes, configuration changes, malicious behavior, etc.  

Read more about how to use Auditbeat here.

Winlogbeat

Winlogbeat will only interest Windows sysadmins or engineers as it is a beat designed specifically for collecting Windows Event logs. It can be used to analyze security events, updates installed, and so forth.

Read more about how to use Winlogbeat here.

Functionbeat

Functionbeat is defined as a “serverless” shipper that can be deployed as a function to collect and ship data into the ELK Stack. Designed for monitoring cloud environments, Functionbeat is currently tailored for Amazon setups and can be deployed as an Amazon Lambda function to collect data from Amazon CloudWatch, Kinesis and SQS.

Community beats

There is also a long and growing list of beats developed by the community. These, of course, are not supported in any official way, but are worth exploring if relevant for your use case. For example, a beat called pubsubbeat allows users to ship data from Google Cloud Platform’s Pub/Sub service into the ELK Stack.

The list of community beats includes beats for AWS, Apache, Cassandra, Docker, GitHub and many more.

Installing Beats

Installing the various beats is super simple. Most likely, you’ve already set up one of the other components in the ELK Stack, so you should be familiar with the commands.

Just in case, here is a full example of installing a beat from scratch on an Ubuntu 16.04 host.

First, download and install the Elasticsearch signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Then, save the repository definition:

echo "deb https://artifacts.elastic.co/packages/6.x-prerelease/apt 
stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

To install your beat, use:

sudo apt-get update 
sudo apt-get install <beatname>

Configuring Beats

Being based on the same underlying architecture, Beats follow the same structure and configuration rules.

Configuration files are usually located in the same directory —  for Linux, this location is the /etc/<beatname> directory. For Filebeat, this would be /etc/filebeat/filebeat.yml, for Metricbeat, /etc/metricbeat/metricbeat.yml. And so forth.

Generally speaking, and again — the exact structure may differ depending on the beat in question — the configuration file will include two main sections: one defines what data to collect and how to handle it, the other where to send the data to.  

Beats configuration files are based on the YAML format with a dictionary containing a group of key-value pairs, but they can contain lists and strings, and various other data types.

Most of the beats also include files with complete configuration examples, useful for learning the different configuration settings that can be used. Use it as a reference.

Filebeat configuration example

Here is an example of a very basic Filebeat configuration:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
  fields: 
    app_id: service-a
    env: dev

output.logstash:
  hosts: ["localhost:5044"]

The prospector is where you define what log file to track — its path, its type, encoding type and any other processing steps you would like to apply. You can define multiple prospectors per Filebeat or multiple paths per prospector.  You can add custom fields to each prospector, useful for tagging and identify data streams.

The output section informs Filebeat where to send the data to — in the example above we are defining a Logstash instance, but you can also define Elasticsearch as an output destination if you do not require additional processing.

Metricbeat configuration example

Here is an example of a Metricbeat configuration:

metricbeat.modules:
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
fields:
  env: dev

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

I am using the Docker module, defining what specific statistics I want to be shipped (called metricsets), adding a custom env field and shipping directly into Elasticsearch.

Configuration best practices

Each beat contains its own unique configuration file and configuration settings, and therefore requires its own set of instructions. Still, there are some common configuration best practices that can be outlined here to provide a solid general understanding.

• Some Beats, such as Filebeat, include full example configuration files (e.g, /etc/filebeat/filebeat.full.yml). These files include long lists all the available configuration options.
• YAML files are extremely sensitive. DO NOT use tabs when indenting your lines — only spaces. YAML configuration files for Beats are mostly built the same way, using two spaces for indentation.
• Use a text editor (I use Sublime) to edit the file.
• Sections are separated by a long line of # (hash) symbols — as much as this sounds funny — don’t mistake these for commented out configuration lines.
• The ‘-’ (dash) character is used for defining new elements — be sure to preserve their indentations and the hierarchies between sub-constructs.

Additional information and tips are available in the Musings in YAML article.

If you are using Logz.io, we have made it easy to configure a Filebeat YAML with a dedicated wizard. Even if you are not a Logz.io user, it can come in handy. Go through the wizard and simply remove the fields specific to Logz.io.

Beats modules

As mentioned above, Filebeat and Metricbeat both support what is a growing number of modules — built-in configurations and settings for specific platforms and systems. So instead of configuring these two beats from scratch, using these modules will help you hit the ground running with pre-configured settings.

Modules only take care of the “what” and the “how”, not the “where to” – you will still need to configure the output section.

Each module comes with a set of files containing module-specific configurations, field definitions, and sample Kibana visualizations. Modules have default configurations which will most likely be fine in most use cases. You can see these default settings in the /etc/<beatname>/module.d directory (DEB).

There are various ways of enabling these modules, the simplest being to use the modules enable command:

sudo <beatname> modules enable apache

Or, you can enable modules from within the Filebeat or Metricbeat configuration file. For example:

filebeat.modules:

 - module: apache

To see a list of the enabled modules, use:

sudo <beatname> modules list

Your will see a list of enabled and disables modules:

Enabled:
apache
system

Disabled:
aerospike
aws
ceph
...

You will need to restart Filebeat of course for changes to be applied. 

Filebeat modules: Apache, Auditd, Cisco, Coredns, Elasticsearch, Envoyproxy, HAProxy, Icinga, IIS, Iptables, Kafka, Kibana, Logstash, MongoDB, MySQL, Nats, NetFlow, Nginx, Osquery, Palo Alto Networks, PostgreSQL, RabbitMQ, Redis, Santa, Suricata, System, Traefik, Zeek (Bro).

Metricbeat modules: Aerospike, Apache, AWS, Ceph, Couchbase, Docker, Dropwizard, Elasticsearch, Envoyproxy, Etcd, Golang, Graphite, HAProxy, HTTP, Jolokia, Kafka, Kibana, Kubernetes, kvm, Logstash, Memcached, MongoDB, mssql, Munin, MySQL, Nats, Nginx, PHP_FPM, PostgreSQL, Prometheus, RabbitMQ, Redis, System, traefik, uwsgi, vSphere, Windows, Zookeeper.

What’s next?

Once you’ve configured your beat, you are ready to begin enjoying the wealth of information they collect.

To start your beat, use:

sudo service <beatname> start

Within seconds, a new index will be created in Elasticsearch which you can define in Kibana to begin analysis which is the simplest way to make sure your beat is working correctly.

Endnotes

Beats by no means replace Logstash. Users often get confused as to how the two interact. Beats act as lightweight log shippers for specific use cases, while Logstash is responsible for the heavy lifting, acting as an aggregator and processor. You will most likely use them together, especially when using Filebeat for collecting and processing log files.

Beats are extremely versatile and easy to use, and the addition of modules makes them, even more, feature rich. In this tutorial, I provided only the tip of the iceberg — the different beats contain specific usage instructions and configurations which are worth exploring.

Happy YAMLing, and keep the beat going guys!  

Pulling in data exposed via API is not one of the most common use cases for ELK Stack users but it is definitely one I’ve come across in the past. Developers wrapping their database services with REST API, for example, might be interested in analyzing this data for business intelligence purposes.

Whatever the reason, the ELK Stack offers some easy ways to integrate with this API. One of these methods is the Logstash HTTP poller input plugin. In the example below, I’ll be pulling in some weather data exposed via public API using this plugin.

To follow these steps and build your own pipeline for HTTP API analysis in ELK you’ll need to set up the following:

• Your own ELK Stack or a Logz.io account. To install ELK, take a look at our ELK guide. To create a Logz.io account, simply go to https://logz.io/freetrial and sign up for free.
• Access to a weather API service. For this article, I’m using OpenWeatherMap, which provides free access to current weather stats via API but you can hook up to any API using the same methodology.

The dataset

As mentioned above, I’m collecting current weather data using OpenWeatherMap API. The dataset includes statistics on existing weather conditions all over the world collected from 40,000 weather stations.

The data itself is available in JSON, XML or HTML but the default format is JSON, which suits our use case perfectly because we’re using Elasticsearch to index the data. Calling the data is done in a variety of ways — you can call by providing a city name (the method we’ll use here), city ID, geographic coordinates, or ZIP code. You can also call for multiple cities within specific coordinates.

The example below is calling for weather stats within a bounding box that includes the longtitude and latitude coordinates:

http://api.openweathermap.org/data/2.5/box/city?bbox=12,32,15,37,10

The data returned by the API provides us with all the stats we need for gauging current weather conditions, including temperature, humidity, atmospheric pressure, wind speed and direction, and more.

Here’s an example:

{"coord":
{"lon":145.77,"lat":-16.92},
"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],
"base":"cmc stations",
"main":{"temp":293.25,"pressure":1019,"humidity":83,"temp_min":289.82,"temp_max":295.37},
"wind":{"speed":5.1,"deg":150},
"clouds":{"all":75},
"rain":{"3h":3},
"dt":1435658272,
"sys":{"type":1,"id":8166,"message":0.0166,"country":"AU","sunrise":1435610796,"sunset":1435650870},
"id":2172797,
"name":"Cairns",
"Cod":200}

Configuring Logstash

The method for importing data returned via API into the ELK Stack is by using the Logstash http_poller input plugin, which can aggregate, decode and ship data from a defined endpoint URL.  

If you haven’t installed Logstash yet as prescribed above, do so now. Then, create a new Logstash pipeline configuration file:

sudo vim /etc/logstash/conf.d/weather.conf

Logstash input

The input section defines the http_poller input plugin — the URL endpoint to poll, a request timeout, a CRON-based schedule (every 5 minutes) and the codec to use (JSON). The metadata_target setting is optional, and adds some fields to the responses on the performance of the poller.

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "5m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

The API we’re inserting here contains the following query parameters after the call URL:

• q=London,uk – asks to return weather data for London
• APPID= – OpenWeatherMap API key
• units=metrics – converting the units format to celsius

http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7
341764f682c2242e744c4f167b0&units=metric

Logstash filter

Since the API responses are in JSON, there is little processing or parsing that is required from Logstash. And so we can leave the filter section empty for now.

filter {}

Logstash output

I think this last section is pretty self-explanatory, right?

output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
}

Starting Logstash, you will see a new Elasticsearch index created with the weather data. Define the new index pattern in Kibana to begin analysis:

Shipping to Logz.io

To ship the data to Logz.io, a few adjustments need to be made to the Logstash configuration file. A mutate filter containing the Logz.io account token needs to be added, and we need to change the output to point to Logz.io listeners instead of Elasticsearch.

The amended configuration file looks like this (be sure to enter your token in the designated position:

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/find?lat=42.3&lon=-71.05&cnt=20&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "5m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

filter {
  mutate {
    add_field => { "token" => "logzioAccountToken" }
  }
}
 
output {
   tcp {
    host => "listener.logz.io"
    port => 5050
    codec => json_lines
  }
}

Restart Logstash.

You should see the weather data load in Logz.io automatically within a minute or two:

Forget your favorite weather app, hook up your ELK with a weather API and use the dashboard below to stay up to date!

Monitoring an Azure environment can be a challenging task for even the most experienced and skilled team. Applications deployed on Azure are built on top of an architecture that is distributed and extremely dynamic.

But all is not doom and gloom. Azure users have a variety of tools they can use to overcome the different challenges involved in monitoring their stack, helping them gain insight into the different components of their apps and troubleshoot issues when they occur.

In this guide, I’d like to provide readers with some information they can use when considering what path to take on their Azure monitoring journey. I will outline everything from what there is to monitor to start with, to the challenges involved and the two main solutions available to overcome them — Azure Monitor and the ELK Stack.

Anatomy of an Azure environment

Every application is designed differently and is comprised of different building blocks. Still, there are some common architectural guidelines when it comes to deploying modern web applications on the cloud. Whether on AWS, Azure or Google Cloud Platform, the same key ingredients are all there.

Before we understand what we are monitoring, it would help to understand what a classic Azure deployment looks like. Modern applications that must comply with strict SLAs cannot make do with the traditional web tier and data tier architecture. These two tiers are just part of a much more complex architecture as shown in diagram 1 below.

In the example shown here, our application is replicated across two Azure regions with the Azure Traffic Manager routing incoming requests to these regions. We’ve got Azure DNS for name resolution. We’re using Active Directory for securing access to the app and user identity control and management. We’re using Application Gateway as a load balancer. And so forth.

Diagram 1: Classic Azure environment

Azure full-stack monitoring

To be able to ensure all these components are performing as expected, Azure users need to tap into all the different layers an Azure environment is comprised of by collecting and analyzing the different data they generate (see diagram 2).

Starting with tenant-related data such as Azure Active Directory audit logs, through Azure Subscription data such as Activity Logs, diagnostic logs from various Azure resources, and application and operating system level data — all of this data provides Azure users with a comprehensive view of their environment, allowing them to monitor production performance and troubleshoot issues should they occur.

Diagram 2: Azure multi-tiered monitoring

Azure monitoring data

To help gain visibility into these different components, Azure users can use a variety of different data sources and types made available by Azure. This data can be collected by users and stored in Azure storage accounts for analysis with Azure Monitor or 3rd party monitoring tools. Below is a brief overview of the main data types falling under this category.

Azure Activity Logs

Azure activity logs allow users to monitor who did what and when for any write operations (PUT, POST, DELETE) executed for Azure resources in a specific Azure subscription and to understand the status of the operation and other relevant properties.

Azure Diagnostic Logs/Metrics

Azure diagnostic logs provide users with insight into the operation of a specific Azure resource and can contain both logs and metrics. Diagnostic logs will differ in format and content from one another.

Azure Storage Logs/Metrics

Azure storage logs and metrics provide users on storage accounts to enable them to trace requests, analyze usage trends and diagnose issues.

Compute Logs

Azure VMs (Windows/Linux) also generate logs that can be collected and stored on Azure storage accounts. This data provides users with operational insight into how the servers running their applications and services are performing.

Azure key monitoring requirements

If only it was that easy. Cloud architectures pose a long list of challenges that must be overcome to be able to achieve that comprehensive view we described above.

From their very nature, cloud environments are highly distributed, meaning that at any given time there are multiple sources generating data. Not only that, these environments are highly dynamic so if your services are containerized, for example, you must think of a way to counter the transient nature of containers to be able to persist data. Add to this the sheer amount of data even a small to medium sized Azure environment generates and the picture gets even more complicated.

To overcome these challenges, the solution Azure users rely on for monitoring and troubleshooting their applications must provide them with the following basic capabilities:

• Data aggregation – users need to be able to easily access and integrate with all the data sources across their Azure environment, collect the generated data, whether logs or metrics, and store this data in a centralized data store.
• Data ingestion – data pipelines are responsible for handling large amounts of data and in some cases can come under a substantial amount of pressure that results in failing components. Data shippers, and the storage backends they ship the data to, must be designed to be resilient enough to handle large and fluctuant volumes of data.    
• Data storage – the data collected from the different Azure data sources must be stored in a centralized data store that can provide the scale required to support data growth and data bursts. The last thing you want to face when diagnosing a production issue is a failing datastore because you reached capacity.
• Data processing – collecting and storing data is great, but if you cannot process this data then analyzing it will be more challenging. For example, an Azure monitoring solution needs to be able to process fields containing IPs and enhance them with geographical information. Querying data and visualizing it is easier if data is parsed and processed accurately.
• Data analysis – end users must be able to effectively analyze Azure monitoring data. Processing will help but also analysis tools. A basic requirement is the ability to search or query the data. Being able to slice and dice the data with visualizations and dashboards is another key requirement. Ideally, users should be able to use advanced analysis capabilities such as anomaly detection and machine learning to find the needle in the haystack.
• Alerting – to be aware when issues with a potential business impact are taking place in their Azure environment, users must have the ability to create alerts that notify them in real-time. This enables users to be more proactive and stay on top of events as they occur.

Azure monitoring solutions

To monitor their Azure environment, there are a number of solutions users can deploy. In this section, we will examine two of these solutions — Microsoft Azure Monitor and the ELK Stack. Both solutions help users overcome the key challenges outlined above but both are also very different than each other.

Let’s take a closer look.

Azure Monitor

Simply put, Azure Monitor is an Azure-native, all-in-one, logging and monitoring solution for Azure environments.

As an Azure service, Azure Monitor offers scalability, high availability, and security. From a functionality perspective, Azure Monitor provides built-in support for Azure resources, alerts, APIs for 3rd party integrations and advanced analytics capabilities powered by machine learning.

The Azure Monitor service incorporates two components that used to be offered separately in the Operations Management Suite (OMS) — Log Analytics and Application Insights. Taken together, Azure Monitor is an extremely robust solution that can provide end-to-end visibility into an Azure environment.

Because it is an Azure-native solution, the different types of logs and metrics that can be used for monitoring are collected almost automatically. Activity Logs, for example, show up in the Azure Monitor without the need to build a dedicated data pipeline.

There are a few challenges that are worth mentioning as well.

While Azure Monitor provides some pre-built dashboards, specifically for metrics, users might find themselves somewhat limited as to how much they can customize the data displayed (a list of limitations is available here).

Being tied to Azure means that integrating Azure Monitor with external data sources is not straightforward. If your apps are deployed on a multi-cloud architecture for example, you will find it difficult to centralize your monitoring and logging pipelines.

And of course, Azure Monitor is a proprietary solution which incurs an actual cost in $ (based on the volume of data being collected) and which also requires skills and expertise to master.

The ELK Stack

In the open source corner, and as the second most popular monitoring solution for Azure, is the ELK Stack (Elasticsearch, Logstash & Kibana). As the world’s most popular open source log analysis platform, the ELK Stack is a natural choice for monitoring and troubleshooting any cloud-based application.

This is not the place to explain the popularity of ELK but suffice to say it provides users with an open source platform that answers almost all of the requirements we outlined above. Elasticsearch acts as a scalable data store, Logstash aggregates and processes the data and Kibana is used to query and visualize the data (the fourth element in the stack, beats, are used to actually collect and forward different types of data from data sources.)

In an Azure environment, this is what an ELK-based monitoring architecture would most likely look like:

Diagram 3: Azure monitoring with ELK

The ELK Stack is a preferred option for Azure users also because it’s open source, which means there is a large community backing it and driving new features and that it helps companies avoid vendor lock-in. Because ELK is so popular, training and onboarding new members of the team is usually simpler.

And of course, it’s free.

Costs, though, come in many shapes and colors. The ELK Stack can be relatively easy to deploy and get started with but can become quite an architectural headache as your data pipelines become more complicated and the environment itself becomes more complex. Data volumes grow. And as great as the open source ELK is, it requires some additional legwork to make it production-grade.

Production checklist

Whether deployed on the cloud or in your own data center, if you’re planning on using the ELK Stack for monitoring Azure, here is a list of some key requirements that must be taken into consideration:

• Resiliency – because you cannot afford to lose data, data pipelines must be resilient. Buffering and queuing components such as Apache Kafka and RabbitMQ need to be included in the architecture.
• Scalability & high availability – to support data growth and data bursts, Elasticsearch clusters must be designed for scale. Replication across regions, or segments in your datacenter, must be considered for high availability.
• Security – your data must be  secured and so adding authentication and encryption to the architecture are a minimum requirement.
• Processing – As mentioned above, the more accurately data is parsed the easier analysis is down the road so be sure to apply the right processing in Logstash or using Elasticsearch ingest nodes.
• Retention – Depending on your needs, you need to plan for long term retention. For troubleshooting, for example, a few days might suffice, but for other use cases you’ll need to make sure you have enough resources for longer retention and automatic processes to remove old indices and for archiving and backups.
• Alerting – to be more proactive in how you monitor Azure, you will need to build an alerting mechanism on top of Elasticsearch and Kibana.
• Monitoring – last but not least – to be sure your ELK Stack is performing as required, think about monitoring whatever you can, including ingestion rates, lags in Kafka, Elasticsearch indices, Logstash performance, etc.

Suggested ELK architecture

Based on this checklist, what would an ELK production deployment look like? In diagram 4 below is a suggested architecture for deploying ELK on Azure. This might not suit everyone’s environment and will most likely vary from use case sto use case, but it complies with our list of key requirements.

We have Kafka acting as a buffer in front of Logstash for data resiliency. We‘re using Elasticsearch clusters with 3 master nodes and 2 data nodes, each. We’re using Elasticsearch Curator to delete old indices and are archiving to Azure blob. We’ve installed ElastAlert for alerting and are deploying nginx as a proxy server for authentication. And we’ve replicated this architecture in two Azure regions for high availability.

Diagram 4: ELK on Azure architecture

Logz.io for Azure

At Microsoft Build 2019, Microsoft and Logz.io announced a joint partnership that enables Azure users to leverage Logz.io’s fully managed ELK solution to monitor their environment.

The solution provides users with Azure-native monitoring features, including seamless integration with Azure resources, built in dashboards for different Azure resources and availability across 6 Azure regions.

What this means for Azure users, first and foremost, is that instead of spending time monitoring their ELK Stack, they can focus on what matters most to their business — building, deploying and monitoring their apps.

Endnotes

Monitoring applications on Azure is an engineering challenge that requires a very specific type of solution. Ideally, this solution needs to be easily deployed, it needs to integrate natively with Azure data sources, it needs to be able to scale automatically, and it needs to provide users with the tools to analyze all the data collected.

The ELK Stack can help you overcome these core challenges and meet these basic requirements but requires some work to deploy at scale. Handling large ELK deployments is not impossible — a lot of teams around the world have succeeded.

But at the end of the day it’s a decision every organization needs to take. Is the organization willing to invest time and resources in monitoring their monitoring stack or would they rather focus on monitoring what matters most to them and their business — their apps?

To summarize the Azure monitoring solutions reviewed above, here is a high-level comparison comparing some of the key features and functionality:

Additional resources

Here is a list of some of the resources available to learn more about how to use ELK and Logz.io to monitor Azure:

• Installing ELK on Azure
• Monitoring Azure with Logz.io
• Monitoring Activity Logs with Logz.io
• Monitoring Application Gateway with Logz.io
• Logz.io and Azure