v2.0.0

We provides the means for the collection, organization and computerized retrieval of knowledgeand Lightweight Data Forwarder for Linux, BSD, macOS and Windows.

October 25, 2022

KNOWLEDGE BASE

Release Notes v2.0.0

Fluent Bit is a Fast and Lightweight Data Processor and Forwarder for Linux, BSD and OSX. We are proud to announce the availability of Fluent Bit v2.0.0.

For people upgrading from previous versions you must read the Upgrading Notes section of our documentation:

https://docs.fluentbit.io/manual/installation/upgrade_notes

Introduction

Fluent Bit v2.0 is the start of the new stable series of the project. During the last months our primary focus has been around extending support for Metrics, Traces and improving performance, among many others.

News

Fluent Bit v2.0.0 is the stable release!, new changes on this version:


  • Logs, Metrics, and Traces
    • Metrics
    • OpenTelemetry
  • Performance
  • Full OpenSSL support / deprecate mbedTLS
  • TLS for input plugins
  • Plugin Changes
    • Loki
    • Splunk + Metrics
  • Developers Experience
  • TAP
  • Internal Metrics
  • New Storage Metrics
  • Memory Ring Buffer
  • Metrics exposed
  • Deprecating TD Agent Bit (packages)

Logs, Metrics, and Traces

Fluent Bit has always been agnostic of the data that its processes and moves around; one of the major use cases has been around Log Processing. Recently we introduced functionality to integrate Metrics and now with Fluent Bit v2, we are formally announcing the support for Traces.

As a vendor-agnostic solution, Fluent Bit natively integrates with protocols and other ecosystems, so having Metrics and Traces support means that we fully integrate with systems like Prometheus and OpenTelemetry.


Metrics

As part of the metrics collectors (inputs) and outputs supported for Metrics, allow us to collect and deliver metrics in a smooth way, a common example of these components are:

Input Description
node_exporter_metrics A plugin that implements a subset of host metrics collection like the known external tool Prometheus Node Exporter.
nginx_metrics Scrape Nginx metrics endpoints. It supports the OSS and Nginx+ enterprise editions.
windows_exporter_metrics Collect system metrics from a Windows system, for it support CPU collection.
fluentbit_metrics Collect internal Fluent Bit metrics and ingest them into the pipeline.

Output Description
prometheus_exporter Expose metrics in an HTTP endpoint in Prometheus text format. This mechanism is commonly used when you want to scrape metrics available by Fluent Bit by other a third part solution like Prometheus Server.
prometheus_remote_write Deliver metrics to a remote endpoint by using the Prometheus Remote Write protocol.
splunk New support for Splunk Metrics (HEC)
Influxdb Send metrics to InfluxDB database

One of the biggest addition on this are is the support for OpenTelemetry metrics, in the input and output side, more details below.

OpenTelemetry

OpenTelemetry is a framework, spec and protocol to unify the aspects of Telemetry collection and delivery. One of the major use cases in production is for Traces; Metrics and Logs support has been recently added.

In Fluent Bit, we are announcing full support for OpenTelemetry, where now we can receive and send telemetry data by using OpenTelemetry protocol:

OpenTelemetry input plugin

This new plugin, can receive OpenTelemetry Logs, Metrics and Traces, all of them are supported through this new implementation that allow easily integration with applications instrumented with OpenTelemetry SDKs.

OpenTelemetry output plugin

This new plugin allows to deliver any Log, Metric or Trace collected to a remote endpoint that supports OpenTelemetry, it could be any vendor platform or the OpenTelemetry collector.

Performance

New threaded mechanism allows input plugins to run in a separate thread which helps to desaturate the main pipeline and move certain loads to a different CPU core. This feature must be enabled manually in the configuration of each input plugin by adding the threaded property, e.g:

[INPUT]
    name     tail
    path     /var/log/containers/*.log
    tag      kube.*
    threaded on

Full OpenSSL support / deprecate mbedTLS

With this new version, all the network transport layer that needs security enabled will use OpenSSL instead of mbedTLS: networking & crypto functions. From now on, mbedTLS is officially deprecated.

All interfaces and plugins that were using mbedTLS has been ported to use the new crypto and networking layer we have implemented.


TLS for input plugins

Using Fluent Bit as an aggregator is a common use case, but one component was missing: TLS support for ingestion (Transport Layer Security), which for many users it was a big blocker; the same problem has been reported by many Fluentd users looking forward to migrating to Fluent Bit.

In this new version, we are announcing full-native TLS capabilities for input plugins.


Plugins Changes

Loki (output)

In previous versions, the Loki project did not support out-of-order records, for performance reasons, Fluent Bit delivers messages in parallel which could cause exceptions on the transmission since some messages might not arrive in the order expected by Loki; to solve this problem, we used to force our Loki connector to just send one Chunk at a time, so the order was preserved.

Since newer versions of Loki no longer have this restriction, we are removing the restriction too from our side and allowing concurrency again, which is a boost in performance. No changes are needed.

Note: this improvement might be a breaking change if you are using an older version of Loki. If you are using **Loki >= 2.4 ** you are good to go.

To learn more about this Loki restriction and the enhancements, you can refer to the following blog post from Grafana team:

https://grafana.com/blog/2021/12/03/new-feature-in-loki-2.4-no-more-ordering-constraint/

Splunk + Metrics (output)

The current Splunk output connector now supports the delivery of Metrics. No changes in your configuration are needed, if you send metrics records to a Splunk output plugin, the data will be converted and delivered as expected by the Splunk HEC endpoint.


Developers Experience

There are many cases where a user would like to extend the functionality that Fluent Bit provides, usually for specific business reasons.

For customizations, we currently support plugins that can be written in C, filters in Lua, or output plugins in Golang. But understanding our user’s needs and the restrictions of this limited functionality, we are going beyond that.

Fluent Bit v2 comes with support to integrate more plugin types with Golang and WebAssembly. The following table describes the supported languages by plugin type.

Language Input / Source Filter Output / Destination
C Yes Yes Yes
Golang Yes —- Yes
WebAssembly Yes Yes —-

TAP

One of the common questions from our users has been “how can I see the data flowing through a pipeline?”, usually, this comes up when there is a need to perform troubleshooting without stopping the agent.

TAP is a very advanced functionality that allows inspecting the data that flows through a pipeline, it can be enabled on runtime by just using a simple HTTP call.

More details about this functionality can be seen here in the official documentation:

Internal Metrics

Historically, a common way to extract internal metrics from Fluent Bit has been by enabling the built-in HTTP service which exposes the following endpoints in the REST API:

Endpoint Description
/api/v1/metrics JSON metrics for records and byte sizes associated.
/api/v1/metrics/prometheus Same metrics as /api/v1/metrics but formatted in Prometheus Text format.
/api/v1/storage JSON metrics for the storage component which expose metrics for Chunks.

One of the restrictions of this interface, is that those metrics are not part of the pipeline, which means, you can only access them remotely. But what about if you want to send your metrics to a destination like Prometheus, InfluxDB or other ?.

During the release cycle of the previous series of Fluent Bit, we introduced a new mechanism to collect and process internal metrics through a new input plugin plugin called fluentbit_metrics. This new mechanism allows to expose or send the metrics to a destination in many ways like:


  • Prometheus Exporter
  • Prometheus Remote Write
  • InfluxDB
  • OpenTelemetry
  • Standard output interface (stdout)

New Storage Metrics

As mentioned, the Storage metrics originally were exposed in a different endpoint which forced the user to scrape another endpoint and also faced the restriction that storage metrics were only exposed in JSON.

Starting from Fluent Bit v2, fluentbit_metrics exposes seven new metrics for the Storage layer:

Metrics name Description
fluentbit_input_storage_overlimit It takes a value of 1 if the input plugin instance is over the limit imposed by a mem_buf_limit configuration. Otherwise is just set to 0.
fluentbit_input_storage_memory_bytes Ttotal number of bytes used by the chunks which are up in memory.
fluentbit_input_storage_chunks Total number of chunks currently associated to the input plugin instance. This includes chunks in an ‘up’ and ‘down’ state.
fluentbit_input_storage_chunks_up Total number of chunks that are ‘up’ in memory generated by the input plugin instance.
fluentbit_input_storage_chunks_down Total number of chunks that are in a ‘down’ state. When the input plugin has enabled the filesystem storage type, not all chunks are loaded ‘up’ in memory, some of them will be on disk only and not loaded in memory until a later time. This metric represents the number of chunks on that state.
fluentbit_input_storage_chunks_busy Total number of chunks that are in a busy state. If a chunk is being used by an output plugin, likely under a flush operation, the chunk is marked as busy.
fluentbit_input_storage_chunks_busy_bytes Total number of bytes being used by chunks in a busy state.

Memory Ring buffer

The memory-ring-buffer mechanism is an optional buffering strategy for input plugins that relies only in memory. When this is enabled, the chunks management behaves as a ring buffer of a fixed size were setting up mem_buf_limit is mandatory to specify that limit.

When the chunks are enqueued, and mem_buf_limit is reached, instead of pausing the input plugin as normally would happen with the default memory strategy, this new mechanism removes the oldest Chunks from the queue until to make sure there is enough space for the new incoming data.

The configuration of this new strategy is as follows:

   [INPUT]
       name          tail
       path          /var/log/*
       # new configuration options for memory ring buffer
       storage.type  memrb
       mem_buf_limit 20M

NOTE: by default Fluent Bit tries to enqueue Chunks of size around 2M, this is an internal size that cannot be changed, but there are scenarios where the Chunk size can be greater than that. So mem_buf_limit should be no less than 20M.

Metrics exposed

In addition to the ring-buffer functionality, this PR implements an extension of metrics to count the number of dropped bytes and dropped chunks, e.g:

  # HELP fluentbit_input_memrb_dropped_chunks Number of memrb dropped chunks.
  # TYPE fluentbit_input_memrb_dropped_chunks counter
  fluentbit_input_memrb_dropped_chunks{name="dummy.0"} 22
  # HELP fluentbit_input_memrb_dropped_bytes Number of memrb dropped bytes.
  # TYPE fluentbit_input_memrb_dropped_bytes counter
  fluentbit_input_memrb_dropped_bytes{name="dummy.0"} 572

The metrics above are an example of how they are retrieved when exposed through the Prometheus Exporter output plugin. The following configuration was used:

[INPUT]
    name          dummy
    # new configuration options for memory ring buffer
    storage.type  memrb
    mem_buf_limit 20M

[INPUT]
    name        fluentbit_metrics
    tag         metrics

[OUTPUT]
    name        prometheus_exporter
    match       metrics

[OUTPUT]
    name        http
    match       dummy*
    host        127.0.0.1
    port        9999

Deprecating td-agent-bit

In the early days, Fluent Bit packages were called td-agent-bit, since it was the distribution of Fluent Bit by Treasure Data, the original sponsor of this project, but that’s sponsorship is no longer active.

TD Agent Bit was widely adopted by the community and during the v1.9 release cycle, we informed the community that this package name will be deprecated, now with v2, we are finalizing that transition.

From now on, all Fluent Bit packages and container images follow the project name fluent-bit as it should be.

We encourage users to move to Fluent Bit v2 and adopt the new package name. Note that repositories keep being the same.

Contributors

On every release, there are many people involved doing contributions on different areas like bug reporting, troubleshooting, documentation and coding, without these contributions from the community, the project won’t be the same and won’t be in the good shape that it is now. So THANK YOU! to everyone who takes part of this journey!

Join us

We want to hear about you, our community is growing and you can be part of it!, you can contact us at:

Latest Version

New release on Nov 27, 2024

Fluent Bit v3.2.2 is out!

Check out the Release Notes, read the Updated Documentation or jump directly to the Downloads Section.

We are part of a wide community, no vendor lock-in.