Knowledge Base

Release Notes v1.4

Fluent Bit is a Fast and Lightweight Data Processor and Forwarder for Linux, BSD and OSX. We are proud to announce the availability of Fluent Bit v1.4.

For people upgrading from previous versions you must read the Upgrading Notes section of our documentation:

https://docs.fluentbit.io/manual/installation/upgrade_notes


Changes

Fluent Bit v1.4 is the next major release and include several improvements:


Keep calm, no changes, just some pending adjustments in our banner and messaging.

Fluent Bit has always been a Fluentd sub-project and since the projects joined the Cloud Native Computing Foundation (CNCF) we had pending the copyright message changes. When you start Fluent Bit now you will see the following banner:

* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

As a CNCF sub-project, all contributions belongs to The Fluent Bit Authors :)


Maintainers

Fluent Bit project adoption keeps growing, as well the number of companies contributing back to it. As part of this grow now we are more maintainers!, here is the list of current and active project maintainers:

Maintainer Name Components Company
Eduardo Silva All Arm Treasure Data
Masoud Koleini Stream Processor Arm
Fujimoto Seiji Windows Platform Clear Code
Wesley Pettit Amazon Plugins (AWS) Amazon Web Services
Cedric Lamoriniere Datadog Output Plugin Datadog
Jonathan Gonzalez V. PostgreSQL Output Plugin 2ndQuadrant


Core

Dump Internals

For monitoring purposes overall metrics are good but knowing the current state of our data ingestion can be very helpful for performance configuration and general troubleshooting.

This new version comes with a feature to dump the internals state of the data ingestion, this can be triggered anytime using the Unix signal CONT, e.g:

kill -CONT `pidof fluent-bit`

For more details about it output and description of each value refer to the official documentation here:

https://docs.fluentbit.io/manual/administration/dump-internals-signal


Plugins Logging API

Our internal logging API has been extended and now every plugin instance can set their preferred logging level through the Log_Level property. Usage:

[OUTPUT]
    Name      http
    Match     *
    Log_Level debug
    Host      192.168.3.4
    Port      9090


Signv4

We have implemented a native support for AWS Signv4 protocol, that means that now we can sign HTTP requests for Amazon services. This is one of the foundation for our next implementations of Amazon Services plugins (planned for v1.5 release).

Keep reading to learn more about our extended support for AWS Elasticsearch Service.


Storage Layer & Buffering

Our Storage layer based on Chunk I/O library has been improved and upgraded to it latest version v1.0.3. The new version comes with performance improvements and overall optimizations. The number of required system calls has been reduced and disabled some CRC32 checksum calculation when not needed.

In addition, some improvements where done on Fluent Bit logic to decide to which Chunk we should write data every time, this helped to reduce the number of Chunks in the file system and reduce the load generated by the storage layer.


Network I/O

New KeepAlive Support

When flushing data through an output plugin that requires network I/O, the default behavior is to perform a new TCP connection, this make sense for cases where the flush time is long, but for a streaming data fashion or for cases where we need to reduce the number of continuous TLS handshakes we have implemented optional KeepAlive support.

KeepAlive mode aims to be used by Output plugins configuration, when enabled, any TCP/TLS connection can be reused. From a configuration perspective we now expose two new properties:

Property Description Default
KeepAlive Enable or disable Keepalive mode Off
KeepAlive_Timeout Set an expiration time for unused KeepAlive connections. If a KeepAlive connection is not used after KeepAlive_Timeout seconds, it will be closed. 10


Configuration example:

[OUTPUT]
    Name              http
    Match             *
    KeepAlive         on
    KeepAlive_Timeout 10
    Format            json_lines


TLS Handshake Timeout

When connecting to a third party service using TLS, there are cases where the TLS handshake is never completed by the remote peer. To avoid situations on waiting for a TCP disconnection, now we handle our own timeout to validate that if a TLS handshake has not finished within the first 5 seconds (default value), just drop the connection and retry later.


Configuration example:

[OUTPUT]
    Name                  http
    Match                 *
    tls                   on
    tls.verify            on
    tls.handshake_timeout 5


Config Maps: Configuration Helper

DISCLAIMER: Fluent Bit Config Maps are NOT related to Kubernetes ConfigMaps

As the project grows we need better ways to alert about configuration errors and better ways to offer help.

This version introduces the concept of Config Maps which is a mechanism on where Plugins now can register expected configuration properties to the engine, specifying data type and further descriptions. If a given option by the user is not available, the plugin won’t start and the user will get the proper error.

In addition, the new helper from the command line allow us to retrieve all configuration properties available with further details. Take the following example where with the simple command fluent-bit -F kubernetes -h we can retrieve a full list of options:

HELP
kubernetes filter plugin

DESCRIPTION
Filter to append Kubernetes metadata

OPTIONS
buffer_size                  buffer size to receive response from API server
                             > default: 32K, type: size

tls.debug                    set TLS debug level: 0 (no debug), 1 (error), 2
                             (state change), 3 (info) and 4 (verbose)
                             > default: 0, type: integer

tls.verify                   enable or disable verification of TLS peer
                             certificate
                             > default: true, type: boolean

tls.vhost                    set optional TLS virtual host
                             > type: string

merge_log                    merge 'log' key content as individual keys
                             > default: false, type: boolean

merge_parser                 specify a 'parser' name to parse the 'log' key
                             content
                             > type: string

merge_log_key                set the 'key' name where the content of 'key' will
                             be placed. Only used if the option 'merge_log' is
                             enabled
                             > type: string

merge_log_trim               remove ending '\n' or '\r' characters from the log
                             content
                             > default: true, type: boolean

....


Plugins & Connectors

Fluent Bit community continues contributing improvements and extensions, the following is a summary of the new plugins that are part of v1.4 release:


Filter: AWS

The new AWS Metadata Filter allows to enrich logs with AWS Metadata. For more details about it usage please refer to the documentation here:

https://docs.fluentbit.io/manual/pipeline/filters/aws-metadata


Filter: Expect

We now expose a new filter called Expect that aims to be used as a mechanism to test the content of your records after one or multiple modifications. This filter aims to simplify unit testing for the end user and is flexible enough to validate the existence of keys, subkeys and expected values.


Filter: Rewrite_Tag

Tags are what makes routing possible. Tags are set in the configuration of the Input definitions where the records are generated, but there are certain scenarios where might be useful to modify the Tag in the pipeline so we can perform more advanced and flexible routing.

The rewrite_tag filter, allows to re-emit a record under a new Tag. Once a record has been re-emitted, the original record can be preserved or discarded.


Filter: Kubernetes

We have added a new option to set the TLS virtual host name through the tls.vhost option.


Output: PostgreSQL

Our users have demanded a native integration with one of the famous relational databases: PostgreSQL. On this new version of Fluent Bit, we have introduced a native output plugin for PostgreSQL Server >= 9.4 that takes advantage of the native JSON type.

Configuration example:

[OUTPUT]
    Name          pgsql
    Match         *
    Host          172.17.0.2
    Port          5432
    Database      fluentbit
    Table         fluentbit
    User          postgres
    Password      YourCrazySecurePassword
    Timestamp_Key timestamp


Output: Elasticsearch + AWS

We have implemented native AWS Signv4 protocol so our Elasticsearch output plugin now is compatible with the Amazon Elasticsearch Service.

For more details about it configuration of credentials and AWS mode, please refer to the following documentation section:

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#fluent-bit-amazon-elasticsearch


Output: Datadog

The Datadog connector has been extended and now it supports HTTP proxies through the configuration property proxy. For more details refer to the plugin documentation:

https://docs.fluentbit.io/manual/pipeline/outputs/datadog

Contributors

On every release, there are many people involved doing contributions on different areas like bug reporting, troubleshooting, documentation and coding, without these contributions from the community, the project won’t be the same and won’t be in the good shape that it is now. So THANK YOU! to everyone who takes part of this journey!

Join us!

We want to hear about you, our community is growing and you can be part of it!, you can contact us at: