In the midst of all the turmoil and debate around Open Distro for Elasticsearch, Elastic continues to produce, and last week announced both a new major release of the Elastic Stack — version 6.7 (and also the first release candidate for 7.0!).
So what exactly was released in version 6.7.
As usual, I’ve put together a brief overview of the main features introduced. One change I’ve applied this time is adding a comment for each feature detailing what license it falls under. I’ve encountered increasing confusion over the issue of licensing so hopefully this will help.
Elasticsearch
Elasticsearch 6.7 includes a few new features but the big news in this version is the graduation of a lot of major features that were released in beta mode in previous versions and that are now GA.
Index Lifecycle Management
A beta in version 6.6, Index Lifecycle Management is a super-useful feature that allows you to manage the lifecycle of your Elasticsearch indices more easily. Using API or the new dedicated page in Kibana, you can set rules that define the different phases that your indices go through — hot, warm, cold and deletion. In version 6.7 the ability to manage frozen indices was added (for long term and memory-efficient storage).
Index Lifecycle Management is available under the Basic license.
Cross-Cluster Replication
A beta in version 6.5, this now-GA feature in Elasticsearch offers cross-cluster data replication for replicating data across multiple datacenters and across multiple regions. This feature followed the steps of other minor updates to Elasticsearch, specifically soft deletes and sequence numbers, and gives users a much easier way to load data into multiple clusters across data centers.
In version 6.7, the ability to replicate existing indices with soft deletes was added, as well as management and monitoring features in Kibana for gaining insight into the replication process.
Cross-Cluster Replication is only available for paid subscriptions.
SQL
A lot of Elasticsearch users were excited to hear about the new SQL capabilities announced way back in Elasticsearch 6.3. The ability to execute SQL queries on data indexed in Elasticsearch had been on the wishlist of many users, and in version 6.5, additional SQL functions were added as well as the ability to query across indices. All of this goodness, as well as the accompanying JDBC and ODBC drivers, is now GA. Additional SQL statements and functions such as the ability to sort groups by aggregates, were also added in this release.
The SQL interface is available under the Basic license. JDBC and ODBC clients are only available for paid subscriptions.
Elasticsearch index management
A series of improvements have been made to managing Elasticsearch indices in the Index Management UI in Kibana. Tags have been added to the index name to be able to differentiate between the different indices (Frozen, Follower, Rollup). It’s also easier now to freeze and unfreeze indices from the same UI in Kibana.
Index management is available under the Basic license.
Upgrading to 7.0
Version 6.7 is the last major version before 7.0 and as such, includes some new features to help users migrate to 7.0 more easily:
- The Upgrade Assistant in Kibana now allows users to leave the page when performing a reindex operation.
- Users using the API to upgrade will be pleased to know that the Deprecation Info and Upgrade Assistant APIs were enhanced.
- Reindexing data from remote clusters is easier, with the added ability to apply custom SSL parameters and added support for reindexing from IPv6 URLs.
The upgrade assistant UI and API are available under the Basic license.
Kibana
Similar to Elasticsearch, a lot of the features announced in version 6.7 are beta features maturing to GA status. Still, there are some pretty interesting new capabilities included as well.
Maps
This brand new page in Kibana is going to take geospatial analysis in Kibana to an entirely new level. Released in beta mode, Maps supports multiple layers and data sources, the mapping of individual geo points and shapes, global searching for ad-hoc analysis, customization of elements, and more.
Maps is available under the Basic license.
Source: Elastic.
Uptime
This is another brand new page in Kibana, allowing you to centrally monitor and gauge the status of your applications using a dedicated UI. The data monitored on this page, such as response times and errors, is forwarded into Elasticsearch with Heartbeat, another shipper belonging to the beats family, that can be installed either within your network or outside it — all you have to do is enter the endpoint URLs you’d like it to ping. To understand how to deploy Heartbeat, check out this article.
Uptime is available under the Basic license.
Logs
Logs was announced as a beta feature in version 6.5 and gives you the option to view your logs in a live “console-like” view. The changes made in version 6.7 allow you to configure default index and field names viewed on the page from within Kibana as opposed to configuring Kibana’s .yml file. An additional view can be accessed per log message, detailing all the fields for the selected log message and helping you gain more insight into the event.
Logs is available under the Basic license.
Infrastructure
Another beta feature going GA, the Infrastructure page in Kibana helps you gain visibility into the different components constructing your infrastructure, such as hosts and containers. You can select an element and drill further to view not only metrics but also relevant log data.
Infrastructure is available under the Basic license.
Canvas
What I call the “Adobe Photoshop” of the world of machine data analytics — Canvas — is now GA. I had the pleasure of covering the technology preview a while back, and am super excited to see how this project has progressed and finally matured.
Canvas is available under the Basic license.
Beats
Not a lot of news for beats lovers in this release as I expect most of the new goodies will be packaged in version 7.0.
Functionbeat
Functionbeat — a serverless beat that can be deployed on AWS Lambda to ship logs from AWS CloudWatch to an Elasticsearch instance of your choice — is now GA. For triggering the function, you can use either CloudWatch, SQS events, and from version 6.7 — Kinesis streams.
New datasets in Auditbeat
The system module in Auditbeat was improved and supports new datasets and data enhancements, such as a login dataset that collects login information, a package dataset that collects information on installed DEB/RPM and Homebrew packages, and the addition of a new entity_id field to datasets.
Endnotes
What about Logstash? The only news here is that there is no news. It seems that the long-awaited Java execution engine (better performance, reduced memory usage) is still in the works and hopefully will go GA in version 7.0.
As always, be careful before upgrading. Some of the features listed are still in beta so keep that in mind before upgrading. Read the breaking changes and release notes carefully as well as the licensing information.