Release Notes v1.4
Fluent Bit is a Fast and Lightweight Data Processor and Forwarder for Linux, BSD and OSX. We are proud to announce the availability of Fluent Bit v1.4.
For people upgrading from previous versions you must read the Upgrading Notes section of our documentation:
Fluent Bit v1.4 is the next major release and include several improvements:
Keep calm, no changes, just some pending adjustments in our banner and messaging.
Fluent Bit has always been a Fluentd sub-project and since the projects joined the Cloud Native Computing Foundation (CNCF) we had pending the copyright message changes. When you start Fluent Bit now you will see the following banner:
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
As a CNCF sub-project, all contributions belongs to
The Fluent Bit Authors :)
Fluent Bit project adoption keeps growing, as well the number of companies contributing back to it. As part of this grow now we are more maintainers!, here is the list of current and active project maintainers:
For monitoring purposes overall metrics are good but knowing the
current state of our data ingestion can be very helpful for performance configuration and general troubleshooting.
This new version comes with a feature to dump the internals state of the data ingestion, this can be triggered anytime using the Unix signal
kill -CONT `pidof fluent-bit`
For more details about it output and description of each value refer to the official documentation here:
Plugins Logging API
Our internal logging API has been extended and now every plugin instance can set their preferred logging level through the
Log_Level property. Usage:
We have implemented a native support for AWS Signv4 protocol, that means that now we can sign HTTP requests for Amazon services. This is one of the foundation for our next implementations of Amazon Services plugins (planned for v1.5 release).
Keep reading to learn more about our extended support for AWS Elasticsearch Service.
Storage Layer & Buffering
Our Storage layer based on Chunk I/O library has been improved and upgraded to it latest version v1.0.3. The new version comes with performance improvements and overall optimizations. The number of required system calls has been reduced and disabled some CRC32 checksum calculation when not needed.
In addition, some improvements where done on Fluent Bit logic to decide to which Chunk we should write data every time, this helped to reduce the number of Chunks in the file system and reduce the load generated by the storage layer.
New KeepAlive Support
When flushing data through an output plugin that requires network I/O, the default behavior is to perform a new TCP connection, this make sense for cases where the flush time is long, but for a streaming data fashion or for cases where we need to reduce the number of continuous TLS handshakes we have implemented optional KeepAlive support.
KeepAlive mode aims to be used by Output plugins configuration, when enabled, any TCP/TLS connection can be reused. From a configuration perspective we now expose two new properties:
||Enable or disable Keepalive mode
||Set an expiration time for unused KeepAlive connections. If a KeepAlive connection is not used after
KeepAlive_Timeout seconds, it will be closed.
TLS Handshake Timeout
When connecting to a third party service using TLS, there are cases where the TLS handshake is never completed by the remote peer. To avoid situations on waiting for a TCP disconnection, now we handle our own timeout to validate that if a TLS handshake has not finished within the first
5 seconds (default value), just drop the connection and retry later.
Config Maps: Configuration Helper
DISCLAIMER: Fluent Bit Config Maps are NOT related to Kubernetes ConfigMaps
As the project grows we need better ways to alert about configuration errors and better ways to offer help.
This version introduces the concept of
Config Maps which is a mechanism on where Plugins now can register expected configuration properties to the engine, specifying data type and further descriptions. If a given option by the user is not available, the plugin won’t start and the user will get the proper error.
In addition, the new helper from the command line allow us to retrieve all configuration properties available with further details. Take the following example where with the simple command
fluent-bit -F kubernetes -h we can retrieve a full list of options:
kubernetes filter plugin
Filter to append Kubernetes metadata
buffer_size buffer size to receive response from API server
> default: 32K, type: size
tls.debug set TLS debug level: 0 (no debug), 1 (error), 2
(state change), 3 (info) and 4 (verbose)
> default: 0, type: integer
tls.verify enable or disable verification of TLS peer
> default: true, type: boolean
tls.vhost set optional TLS virtual host
> type: string
merge_log merge 'log' key content as individual keys
> default: false, type: boolean
merge_parser specify a 'parser' name to parse the 'log' key
> type: string
merge_log_key set the 'key' name where the content of 'key' will
be placed. Only used if the option 'merge_log' is
> type: string
merge_log_trim remove ending '\n' or '\r' characters from the log
> default: true, type: boolean
Plugins & Connectors
Fluent Bit community continues contributing improvements and extensions, the following is a summary of the new plugins that are part of v1.4 release:
The new AWS Metadata Filter allows to enrich logs with AWS Metadata. For more details about it usage please refer to the documentation here:
We now expose a new filter called
Expect that aims to be used as a mechanism to test the content of your records after one or multiple modifications. This filter aims to simplify unit testing for the end user and is flexible enough to validate the existence of keys, subkeys and expected values.
Tags are what makes routing possible. Tags are set in the configuration of the Input definitions where the records are generated, but there are certain scenarios where might be useful to modify the Tag in the pipeline so we can perform more advanced and flexible routing.
The rewrite_tag filter, allows to re-emit a record under a new Tag. Once a record has been re-emitted, the original record can be preserved or discarded.
We have added a new option to set the TLS virtual host name through the
Our users have demanded a native integration with one of the famous relational databases: PostgreSQL. On this new version of Fluent Bit, we have introduced a native output plugin for PostgreSQL Server >= 9.4 that takes advantage of the native JSON type.
Output: Elasticsearch + AWS
We have implemented native AWS Signv4 protocol so our Elasticsearch output plugin now is compatible with the Amazon Elasticsearch Service.
For more details about it configuration of credentials and AWS mode, please refer to the following documentation section:
The Datadog connector has been extended and now it supports HTTP proxies through the configuration property
proxy. For more details refer to the plugin documentation:
On every release, there are many people involved doing contributions on different areas like bug reporting, troubleshooting, documentation and coding, without these contributions from the community, the project won’t be the same and won’t be in the good shape that it is now. So THANK YOU! to everyone who takes part of this journey!
We want to hear about you, our community is growing and you can be part of it!, you can contact us at: