Fluent Bit is a Fast and Lightweight Data Processor and Forwarder for Linux, BSD and OSX. We are proud to announce the availability of Fluent Bit v1.7.
For people upgrading from previous versions you must read the Upgrading Notes section of our documentation:
Fluent Bit v1.7 is the next major release!, here you get the exciting news:
Most of users are very pleased with the high performance of Fluent Bit at a low cost of CPU and Memory. Our event-driven architecture and optimized handling of I/O through co-routines and buffering techniques is more than sufficient for most of cases. But running at high scale a single-thread program might face some limits since we were using a single CPU, and after working hard with our Cloud Providers partners like Google and Amazon, now we are pleased to introduce our new multi-thread architecture support: 5x times faster throughput!.
Running in multiple threads allows us to scale to multiple CPU and scale data processing and delivery at higher levels. This new mechanism can be configured at start time and can be enabled optionally per output connector (plugin). Now output connectors can run in multi thread mode using the new option
[OUTPUT] name http host 192.168.3.4 port 443 tls on format json_lines workers 4
The example above enable
4 workers for the connector, so every data delivery procedure will run independently in a separate thread, further connections are balanced in a round-robin fashion.
The multithread implementation is lock-free at runtime. One tip: if you are curious about the improvement, just try setting
workers 1 in your configuration, that option will force the engine to spawn a dedicated thread for data delivery, but note that this thread also can handle hundred of I/O requirements.
Since day zero Fluent Bit supported TLS/SSL through the mbedTLS library. The library is very lightweight and gets the job done. Doing more performance research we found that replacing mbedTLS with OpenSSL will bring even more performance improvements due to it optimized crypto routines. Starting from Fluent Bit v1.7 we provide an abstracted layer for TLS/SSL handling. Now OpenSSL is the preferred option for performance and ecosystem compatibility.
The are two major enhancements on this version, now we provide a new configuration property called
offset_key which optionally allows to append a key with the file offset to each record, and the known option
ignore_older when enabled, now allows to skip unmodified files at start time and also stop monitoring files that have not modified in the interval of time set. This last property behavior change helps to reduce significantly the number of open file descriptors.
We are including a new beta plugin that allows to receive data over HTTP protocol. This input plugin is fully functional and we expect extending it capabilities over the 1.7.x release cycle.
Enrich your records with geoip2 data!. If you have records that contains IP addressed and need a country reference, this is the filter for you.
We have introduced a new native Websocket output plugin.
There are tons of other imrpovements and fixes on this major release, here is a list of the most relevant ones:
On every release, there are many people involved doing contributions on different areas like bug reporting, troubleshooting, documentation and coding, without these contributions from the community, the project won’t be the same and won’t be in the good shape that it is now. So THANK YOU! to everyone who takes part of this journey!
We want to hear about you, our community is growing and you can be part of it!, you can contact us at: