Add option to build with ruby-like capture group names

[bdentino]

Although fluent-bit advertises support for ruby regular expressions, there are subtle differences because onigmo is built with different compiler flags when packaged with ruby. Specifically, capture group names can include special characters in a ruby regex but this does not work in fluent-bit (for example ^(?<time>[^ ]+) (?<crio.stream>stdout|stderr) (?<crio.tags.multiline>[FP]?):?(?<crio.tags.other>([^ ]+))? (?<log>.*)$)

I've added a build option to enable support for this in fluent-bit so that regexes like the above which work in ruby and debugging tools like rubular will not silently fail in fluent-bit parsers.

This is my first contribution so I'm not exactly sure what the maintainers would like to see in terms of testing, documentation, etc for a change like this. Any guidance would be appreciated.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Attached local packaging test output showing all targets (including any new ones) build.

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[patrick-stephens]

Thanks for this, just some niggles from me for a build/release perspective.

Needs to be provided to package builds otherwise will not be used.
Also DCO and commit messages need to follow contribution guidelines.

Please run the packaging test locally as well to confirm it builds for all targets afterwards and we need to verify that with the ok-package-test label on the PR once it is reviewed completely.

[patrick-stephens]

This also needs to flow into the package builds and there I would say it needs to be configurable via an ARG:

• ARG with default:
  

  fluent-bit/packaging/distros/ubuntu/Dockerfile

  Line 147 in 2b77938

  ARG FLB_CHUNK_TRACE=On

• CMake config:
  

  fluent-bit/packaging/distros/ubuntu/Dockerfile

  Line 160 in 2b77938

  -DFLB_CHUNK_TRACE="${FLB_CHUNK_TRACE}" \

Once done we also need to ensure this builds for all targets - see the template above to run a local test first then we can label with ok-package-test.

[patrick-stephens]

Note we will need a docs PR I think to update the existing caveat (I think I added it actually when I found this issue previously): https://docs.fluentbit.io/manual/pipeline/parsers/regular-expression

A common pitfall is that you cannot use characters other than alphabets, numbers and underscore in group names. For example, a group name like (?.*) will cause an error due to containing an invalid character (-).

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[edsiper]

@cosmo0920 is this something that is safe to add from a functionality perspective that don't break existing users ?

[cosmo0920]

@cosmo0920 is this something that is safe to add from a functionality perspective that don't break existing users ?

This doesn't break existing users. Just operates permissive behavior for non-ASCII class of characters for capture strings.

With the above PR descriptions:

^(?<time>[^ ]+) (?<crio.stream>stdout|stderr) (?<crio.tags.multiline>[FP]?):?(?<crio.tags.other>([^ ]+))? (?<log>.*)$ is rejected by default because capture strings has . which is one of the symbols inside the captures.

Instead, to enable these symbols, we need to provide RUBY or newly provided compiler flags as like above patch:

@@ -267,6 +268,10 @@ if(ONIGMO_SHARED_LIB)
     RUNTIME DESTINATION ${ONIGMO_INSTALL_BINDIR})
 endif()
 
+if(ONIGMO_RUBY_REGEX_NAMES)
+  add_definitions(-DRUBY_CODE_NAME)
+endif()
+
 # Test Source
 
 if(ONIGMO_CTESTS)
diff --git a/lib/onigmo/regparse.c b/lib/onigmo/regparse.c
index 49b7e9c841d..adb2fb341a6 100644
--- a/lib/onigmo/regparse.c
+++ b/lib/onigmo/regparse.c
@@ -2506,7 +2506,7 @@ get_name_end_code_point(OnigCodePoint start)
 }
 
 #ifdef USE_NAMED_GROUP
-# ifdef RUBY
+# if defined(RUBY) || defined(RUBY_CODE_NAME)
 #  define ONIGENC_IS_CODE_NAME(enc, c)  TRUE
 # else
 #  define ONIGENC_IS_CODE_NAME(enc, c)  ONIGENC_IS_CODE_WORD(enc, c)

The above patch defines the new compiler flag which is RUBY_CODE_NAME but personally using RUBY as a compiler flag would be enough for this case.

[patrick-stephens]

This is such a time sink when you forget the hidden rules so be good to merge!