v1.2.0

KNOWLEDGE BASE

Release Notes v1.2.0

Fluent Bit is a Fast and Lightweight Data Forwarder for Linux, BSD and OSX. We are proud to announce the availability of Fluent Bit v1.2.0.

For people upgrading from previous versions you must read the Upgrading Notes section of our documentation:

https://docs.fluentbit.io/manual/installation/upgrade_notes

Changes

Fluent Bit v1.2.0 is the next major release and include several improvements:

Routing: optional routable records

By default all records in Fluent Bit are routable , meaning: after ingestion and filtering, they are routed to the output destination based on the matching rules applied to the original record tags.

Starting from this version, you can create records that are not routable. The use case is for records that aims to target the Stream Processor but do not longer need to be routed to some output destination, e.g:

[INPUT]
    Name     tail
    Path     /path/to/my/*.log
    Tag      logs.*
    Routable false

JSON Encoding & UTF-8

The following version fixes all the problems with JSON encoding/decoding and UTF-8 handling, for short: if you are parsing JSON records you no longer need the parser decoders. We have fixed all related problems with encoding. Discussion thread of the original issue and fixes are described in the following ticket:

https://github.com/fluent/fluent-bit/issues/1278

Stream Processor

Recently we introduced the new Stream Processing capabilities, for short, an additional engine that allows to perform SQL queries and create new streams of data based on conditions and windows of time. On this new v1.2 release the Stream Processor is getting extended with the new following capabilities:

Sub-key selection and conditionals support

It’s pretty common that records contains nested maps or sub-keys. Now we provide the ability to use sub-keys to perform conditionals and keys selection. Consider the following record:

1
2
3
4
5
6
7
8
9

{
  "key1": 123,
  "key2": 456,
  "key3": {
    "sub1": {
      "sub2": 789
    }
  }
}

Now you can perform queries like:

SELECT key3['sub1']['sub2'] FROM STREAM:test WHERE key3['sub1']['sub2'] = 789;

New @record functions

On conditionals we have introduced the new @record functions:

IS NULL, IS NOT NULL

We currently support different data types such as strings, integers, floats, maps and null. In Fluent Bit, a null value is totally valid and is not related to the absense of a value as in normal databases. To compare if an existing key in the record have a null value or not, we have introduced IS NULL and IS NOT NULL statements, e.g:

SELECT * FROM STREAM:test WHERE key3['sub1'] IS NOT NULL;

Note that to check if a key exists or not in the record you have to use @record.contains(k) function

Architecture: ARMv8 (aarch64) fixes

The following version implements proper fixes for our co-routines backend in ARMv8 architecture environments. Note that the fix is only required if you are compiling with GCC, Clang compiled versions do not face any issue.

Windows Support (beta)

Continuing with our Windows suport, we have introduced a new Windows Event Log plugin:

https://docs.fluentbit.io/manual/input/winlog

Development: new support for external (Dynamic C) plugins!

Extending Fluent Bit through it pluggable architecture is straightforward, but sometimes we need more flexibility to incorporate plugins that are not part of the code base. The new version incorporate support for external C plugins. To get started, refer to the following helper repository:

https://github.com/fluent/fluent-bit-plugin

List of general changes

Core

• ci: improve GitLab CI/CD Pipeline (#1372)
• io: always set file descriptor number on event structure
• utils: fix integer size to avoid overflow on 32 bits (#1328)
• parser: decoder: json: do not unescape and skip empty spaces (#1278)
• scheduler: fix backoff jitter calculation (#798 #670 #649)
• conf: parsers: docker: comment out decoders, not longer required on k8s
• regex: remove onigmo data types from regex context
• input: new ‘routable’ option for input plugins
• input: chunk: safe chunk creation validating file up/down
• config: add new storage option ‘storage.max_chunks_up’
• core: allow use of relative paths in configuration files on Windows
• compat: add realpath(3) emulation for Windows
• compat: introduce FLB_DIRCHAR
• upstream: always do a reset of connection event structure
• oauth2: fix to try ipv6 when getting upstream connection
• unescape: utf-8: do not skip byte if character cannot be encoded (#1278)
• lib: msgpack: upgrade from 3.1.1 to 3.2.0

Stream Processor

• new ‘routable’ property for created streams
• add @record.time() function for conditionals
• add support for sub-key selection
• parser: support keys that starts with an underscore
• add record-specific functions (@record.contains) (#1369)
• sp: add support for checking if a key value is NULL or not (#1356)

Storage

• set default storage.max_chunks_up to 128
• initialize using new API content loader

Plugins

• Storage_Backlog (Input)
  • Always try to put chunk up, if it fails just continue
  • Handle chunks down/up
• Winlog (Input)
  • Add a new plugin for windows event log
• Tail (Input)
  • Fix packaging and corruption on multiline mode (#1368)
• MQTT (Input)
  • Fix incorrect mqtt length handling (#1391)
• Kubernetes (Filter)
  • Do not unescape and fix handling of a merged map (#1278)
  • New ‘merge_parser’ option
  • Fix unescaping and pack on all cases (#1278)
• Stackdriver (Output)
  • Fix to enable ipv6 when connect to stackdriver logging
  • Add const in functions signatures and limit casts
• Forward (Output)
  • Implement ‘require_ack_response’ (#1321)

Join us

We want to hear about you, our community is growing and you can be part of it!, you can contact us at:

• Slack: http://slack.fluentd.org
• Mailing list: https://groups.google.com/forum/#!forum/fluent-bit
• Github: http://github.com/fluent/fluent-bit
• IRC: irc.freenode.net #fluentbit
• Twitter: @fluentbit