From 85ad3df02a7b87dec29cb11f112edabf75c332fb Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Mon, 17 Aug 2026 12:59:38 +0200 Subject: [PATCH] docs: pipeline: outputs: file: document enable_strftime option Document the strftime placeholder support added to the file output plugin in v5.1.1 (fluent-bit 239681499). - Add enable_strftime to the configuration parameters table - Note strftime expansion in the path, file, fallback_file, and max_dynamic_files descriptions - List '%' alongside '$' as a dynamic destination trigger, and note that fallback_file is required for placeholders as well - Add a "strftime placeholders" section covering UTC event timestamp formatting, the literal-'%' rationale for the disabled default, expansion order when combined with record accessors, and the max_dynamic_files cardinality warning - State that dynamic destinations apply to log records only, with metrics always going to the fallback destination - Add a time-based destination example in YAML and classic formats - Enable_strftime with a strftime path marks the destination dynamic, so max_dynamic_files applies and on_limit_reached defaults to error. Without it the example contradicted its own prose about unresolvable destinations falling back to unrouted.log. - Document that literal percent characters must be escaped as %% when enable_strftime is true, and that fallback_file is required for any path or file containing a percent, including values made up only of escaped %% sequences. - Correct the static prefix rule, which the plugin enforces for record accessors only. A path may begin with a strftime placeholder, so note that a value such as %Y/%m/%d resolves relative to the Fluent Bit working directory and recommend a static prefix. Note this update for code change without docs PR. Signed-off-by: Eric D. Schabell --- pipeline/outputs/file.md | 81 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 74 insertions(+), 7 deletions(-) diff --git a/pipeline/outputs/file.md b/pipeline/outputs/file.md index 4e17d6361..e57f00050 100644 --- a/pipeline/outputs/file.md +++ b/pipeline/outputs/file.md @@ -10,15 +10,16 @@ The _File_ output plugin lets you write the data received through the input plug | Key | Description | Default | | :--- | :--- | :--- | -| `fallback_file` | Static filename used when a `fallback` action is applied. Required whenever `path` or `file` uses a record accessor. See [Dynamic destinations](#dynamic-destinations). | _none_ | +| `enable_strftime` | Enable [`strftime`](#strftime-placeholders) placeholders in `path` and `file`. Disabled by default so that literal percent (`%`) characters in existing filenames are preserved. | `false` | +| `fallback_file` | Static filename used when a `fallback` action is applied. Required whenever `path` or `file` uses a record accessor or a `strftime` placeholder. See [Dynamic destinations](#dynamic-destinations). | _none_ | | `fallback_path` | Static directory path used together with `fallback_file` when a `fallback` action is applied. | _none_ | -| `file` | Set filename to store the records. If not set, the filename will be the `tag` associated with the records. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) expressions. | _none_ | +| `file` | Set filename to store the records. If not set, the filename will be the `tag` associated with the records. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) expressions. When `enable_strftime` is `true`, `strftime` placeholders are expanded using the UTC event timestamp. | _none_ | | `format` | The [format](#format) of the file content. | _none_ | -| `max_dynamic_files` | Maximum number of distinct destinations that record accessor expressions can resolve to. Set to `0` for unlimited. | `1024` | +| `max_dynamic_files` | Maximum number of distinct destinations that record accessor expressions and `strftime` placeholders can resolve to. Set to `0` for unlimited. | `1024` | | `mkdir` | Recursively create output directory if it doesn't exist. Permissions set to `0755`. | `false` | | `on_limit_reached` | Action to take for a record whose destination would exceed `max_dynamic_files`. Accepted values: `error`, `drop`, `fallback`. | `error` | | `on_missing_field` | Action to take for a record whose record accessor field is missing, or resolves to an unsafe value. Accepted values: `error`, `drop`, `fallback`. | `error` | -| `path` | Directory path to store files. If not set, Fluent Bit will write the files in its own working directory. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) expressions, which must follow a static prefix. | _none_ | +| `path` | Directory path to store files. If not set, Fluent Bit will write the files in its own working directory. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) expressions, which must follow a static prefix. When `enable_strftime` is `true`, `strftime` placeholders are expanded using the UTC event timestamp. | _none_ | | `rotate` | Enable size-based [log rotation](#log-rotation). When enabled, files that exceed `rotate_max_size` are rotated and optionally compressed. | `false` | | `rotate_gzip` | Compress rotated files using gzip. Only applies when `rotate` is enabled. | `true` | | `rotate_max_files` | Maximum number of rotated files to retain per output file. Oldest files are deleted first. Must be `1` or greater. Only applies when `rotate` is enabled. | `7` | @@ -183,17 +184,41 @@ pipeline: Dynamic destinations are available in Fluent Bit version 5.1 and greater. -The `path` and `file` parameters accept [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) expressions, which lets one output instance write records to different files based on the content of each record. Fluent Bit treats a destination as dynamic when `path` or `file` contains a `$` character. +The `path` and `file` parameters accept [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) expressions, which lets one output instance write records to different files based on the content of each record. They also accept [`strftime` placeholders](#strftime-placeholders), which route records by the time at which each event occurred. Fluent Bit treats a destination as dynamic when `path` or `file` contains a `$` character, or when `enable_strftime` is `true` and `path` or `file` contains a percent (`%`) character. Two rules apply when you configure a dynamic destination: -- A dynamic `path` must begin with a static prefix. Fluent Bit rejects a `path` that starts with a record accessor, so that generated paths always stay under a directory you chose. -- You must set `fallback_file`. Fluent Bit fails to start if a record accessor is used without it, because there would be nowhere to write records whose destination can't be resolved. +- A `path` that uses a record accessor must begin with a static prefix. Fluent Bit rejects a `path` that starts with a record accessor, so that generated paths always stay under a directory you chose. This check doesn't apply to `strftime` placeholders. See [`strftime` placeholders](#strftime-placeholders). +- You must set `fallback_file`. Fluent Bit fails to start if a record accessor or a `strftime` placeholder is used without it, because there would be nowhere to write records whose destination can't be resolved. Fluent Bit rejects a resolved destination as unsafe when the filename is empty, is `.` or `..`, or contains a path separator or one of the characters `:`, `*`, `?`, `"`, `<`, `>`, or `|`. Path components of `.` and `..` are also rejected. This keeps a record value from redirecting output outside of the configured directory. Generated directories are only created if you also enable `mkdir`. +Dynamic destinations apply to log records only. Metrics records are always written to `fallback_path` and `fallback_file`. + +### `strftime` placeholders + +Support for `strftime` placeholders is available in Fluent Bit version 5.1.1 and greater. + +Set `enable_strftime` to `true` to expand [`strftime`](https://man7.org/linux/man-pages/man3/strftime.3.html) placeholders such as `%Y`, `%m`, and `%d` in `path` and `file`. Fluent Bit formats each placeholder using the timestamp of the event being written, expressed in UTC. Because the value comes from the event itself and not from the clock at flush time, records that arrive late are still written to the file for the period in which they occurred. + +This parameter is disabled by default. When `enable_strftime` is `false`, a percent character in `path` or `file` is written literally, which preserves existing filenames that contain one. + +When `enable_strftime` is `true`, escape every literal percent character in `path` and `file` as `%%`. A `%` that isn't escaped is consumed as the start of a `strftime` conversion specifier and is replaced with a time value. + +Fluent Bit treats any `path` or `file` containing a percent character as a dynamic destination when `enable_strftime` is `true`, so `fallback_file` is required even when the value contains only escaped `%%` sequences and always resolves to the same destination. + +You can combine placeholders with record accessors in the same value. Fluent Bit expands the `strftime` placeholders first, then resolves the record accessors against the result: + +```text +path: /var/log/fluent-bit/%Y/%m/%d/$namespace +``` + +Most rules described in [Dynamic destinations](#dynamic-destinations) apply to placeholders the same way they apply to record accessors. `fallback_file` is required, resolved destinations are rejected when unsafe, and each distinct destination counts toward `max_dynamic_files`. Because a new destination is created for every time period you include, choose the smallest unit your retention needs. An hourly layout such as `%Y%m%d%H` reaches the default `max_dynamic_files` limit of `1024` after about 42 days of continuous operation. + +The static prefix requirement is the exception. Fluent Bit enforces it for record accessors only, so a `path` can begin with a placeholder. A `path` of `%Y/%m/%d` resolves to a relative directory such as `2026/08/17`, located inside the Fluent Bit working directory. Begin the value with a static prefix, as in `/var/log/fluent-bit/%Y/%m/%d`, to keep output under a directory you chose. + ### Actions for unresolved destinations Two settings control what happens to a record whose destination Fluent Bit can't use. `on_missing_field` applies when the record accessor field is missing or resolves to an unsafe value, and `on_limit_reached` applies when writing to a new destination would exceed `max_dynamic_files`. Both accept the same actions: @@ -248,6 +273,48 @@ pipeline: {% endtab %} {% endtabs %} +### Time-based destination example + +The following configuration writes each record to a daily directory derived from the UTC event timestamp. Records whose destination can't be resolved go to `/var/log/fluent-bit/unrouted.log`: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + outputs: + - name: file + match: '*' + path: /var/log/fluent-bit/%Y/%m/%d + file: app.log + enable_strftime: true + mkdir: true + on_missing_field: fallback + on_limit_reached: fallback + fallback_path: /var/log/fluent-bit + fallback_file: unrouted.log +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[OUTPUT] + Name file + Match * + Path /var/log/fluent-bit/%Y/%m/%d + File app.log + Enable_Strftime true + Mkdir true + On_Missing_Field fallback + On_Limit_Reached fallback + Fallback_Path /var/log/fluent-bit + Fallback_File unrouted.log +``` + +{% endtab %} +{% endtabs %} + ## Get started You can run the plugin from the command line or through the configuration file.