Skip to content

Escaping special characters in multiple appenders - #315

Open
FreeAndNil wants to merge 11 commits into
masterfrom
Feature/315-local-syslog-newlines
Open

Escaping special characters in multiple appenders#315
FreeAndNil wants to merge 11 commits into
masterfrom
Feature/315-local-syslog-newlines

Conversation

@FreeAndNil

@FreeAndNil FreeAndNil commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Follow-ups from the second security scan, stacked on #314. One rule, applied to
every sink that had got it wrong: content a sink cannot carry is escaped visibly,
never silently deleted, and never allowed to cost the record or the event.

  • LocalSyslogAppender escapes newlines, which reached syslog(3) unchanged and
    let content forge a second record. NewLineHandling mirrors the remote appender,
    and SyslogNewLineHandling moves out of RemoteSyslogAppender so both can use
    it. Measured: glibc does no escaping of its own, contrary to the old comment.
  • EventLogAppender and OutputDebugStringAppender escape NUL, which ended the
    stored record with no error. Measured on Windows: a 45 character message with a
    NUL at 23 stored as its 23 character prefix.
  • RemoteSyslogAppender escapes characters outside RFC 3164 rather than deleting
    them. "Schönwetter 你好" reached the collector as "Schnwetter ".
  • TelnetAppender and SmtpPickupDirAppender escape unpaired surrogates. Both
    used a writer whose encoding throws: the first disconnected every connected
    client, the second destroyed the whole buffered batch and left a truncated mail.
  • EventLogAppender's size limit is computed rather than guessed. The old constant
    sat above the point where the service silently discards the record, so log4net
    truncated events into oblivion.

Behaviour changes: syslog exception traces are now one escaped record instead of
several lines, Keep restores the old behaviour; RemoteSyslogAppender.SyslogNewLineHandling is no longer nested,
configuration binds by value name and is unaffected.

FreeAndNil added a commit that referenced this pull request Sep 1, 2026
LocalSyslogAppender needs the same option, so nesting it in one of the two
appenders no longer fits.

Breaking for code naming RemoteSyslogAppender.SyslogNewLineHandling. Configuration
binds the value by name and is unaffected.
FreeAndNil added a commit that referenced this pull request Sep 1, 2026
A newline in logged content ends the record for a syslog daemon that writes the
message through to a line oriented log, so content could forge a second entry that
looks authentic.

The code claimed syslog(3) escapes control characters itself. It does not: glibc
formats the buffer and hands it over, and the escaping seen on a mainstream Linux
is the daemon's. Measured with LOG_PERROR, an embedded newline comes out as two
lines.

NewLineHandling mirrors the option RemoteSyslogAppender has had all along, which
already escaped by default. Keep restores the previous behaviour. The remote
appender's habit of dropping non-ASCII is deliberately not copied.
FreeAndNil added a commit that referenced this pull request Sep 1, 2026
OutputDebugStringW takes a null terminated string, so a NUL in logged content
ended the record there and dropped whatever the layout rendered after it.

The escape LocalSyslogAppender already had is now shared by both, since
EventLogAppender is the same shape and will want it too. Its tests moved onto the
shared helper with it.

The appender level test only runs on Windows: Append refuses to run elsewhere.
FreeAndNil added a commit that referenced this pull request Sep 1, 2026
ReportEventW takes a null terminated string, so a NUL in logged content ended the
stored record there and dropped whatever the layout rendered after it. WriteEntry
raises nothing, so the record simply stored short and no ErrorHandler call fired.

Measured on Windows 11 build 26200: a 45 character message with a NUL at 23 stored
as its 23 character prefix.

Escaping happens before the size limit is applied, since it doubles each NUL, and
PrepareEventText exists so that ordering can be tested without an event log.
FreeAndNil added a commit that referenced this pull request Sep 1, 2026


RFC 3164 allows only visible ASCII and space in the message, and everything else
fell through the loop unwritten. "Schoenwetter <CJK>" reached the collector as
"Schnwetter ", and a tab vanished from between its neighbours, with no marker and
no error.

Such characters are now written as a \uXXXX escape, which stays inside the allowed
range. Encoding still cannot make the message body non-ASCII, which the appender
page now says.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
Does.Not.Contain is culture sensitive, and a culture sensitive comparison treats
NUL as ignorable: it reports a match in a string that contains none. Both new
escape tests therefore failed on Windows against correctly escaped output.

ContainsConstraint has no comparison knob at all, so these use
Contains.Substring(x).Using(StringComparison.Ordinal), negated with the ! operator
Constraint defines. The EventLog test asserts the whole value instead, which is
ordinal and pins the length too.

This is the shape f018 reports in StringMatchFilter, which is still open.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
LocalSyslogAppender needs the same option, so nesting it in one of the two
appenders no longer fits.

Breaking for code naming RemoteSyslogAppender.SyslogNewLineHandling. Configuration
binds the value by name and is unaffected.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
A newline in logged content ends the record for a syslog daemon that writes the
message through to a line oriented log, so content could forge a second entry that
looks authentic.

The code claimed syslog(3) escapes control characters itself. It does not: glibc
formats the buffer and hands it over, and the escaping seen on a mainstream Linux
is the daemon's. Measured with LOG_PERROR, an embedded newline comes out as two
lines.

NewLineHandling mirrors the option RemoteSyslogAppender has had all along, which
already escaped by default. Keep restores the previous behaviour. The remote
appender's habit of dropping non-ASCII is deliberately not copied.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
OutputDebugStringW takes a null terminated string, so a NUL in logged content
ended the record there and dropped whatever the layout rendered after it.

The escape LocalSyslogAppender already had is now shared by both, since
EventLogAppender is the same shape and will want it too. Its tests moved onto the
shared helper with it.

The appender level test only runs on Windows: Append refuses to run elsewhere.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
ReportEventW takes a null terminated string, so a NUL in logged content ended the
stored record there and dropped whatever the layout rendered after it. WriteEntry
raises nothing, so the record simply stored short and no ErrorHandler call fired.

Measured on Windows 11 build 26200: a 45 character message with a NUL at 23 stored
as its 23 character prefix.

Escaping happens before the size limit is applied, since it doubles each NUL, and
PrepareEventText exists so that ordering can be tested without an event log.
@FreeAndNil
FreeAndNil force-pushed the Feature/315-local-syslog-newlines branch from e722104 to 6aca022 Compare September 2, 2026 04:32
FreeAndNil added a commit that referenced this pull request Sep 2, 2026


RFC 3164 allows only visible ASCII and space in the message, and everything else
fell through the loop unwritten. "Schoenwetter <CJK>" reached the collector as
"Schnwetter ", and a tab vanished from between its neighbours, with no marker and
no error.

Such characters are now written as a \uXXXX escape, which stays inside the allowed
range. Encoding still cannot make the message body non-ASCII, which the appender
page now says.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
Does.Not.Contain is culture sensitive, and a culture sensitive comparison treats
NUL as ignorable: it reports a match in a string that contains none. Both new
escape tests therefore failed on Windows against correctly escaped output.

ContainsConstraint has no comparison knob at all, so these use
Contains.Substring(x).Using(StringComparison.Ordinal), negated with the ! operator
Constraint defines. The EventLog test asserts the whole value instead, which is
ordinal and pins the length too.

This is the shape f018 reports in StringMatchFilter, which is still open.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
LocalSyslogAppender needs the same option, so nesting it in one of the two
appenders no longer fits.

Breaking for code naming RemoteSyslogAppender.SyslogNewLineHandling. Configuration
binds the value by name and is unaffected.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
A newline in logged content ends the record for a syslog daemon that writes the
message through to a line oriented log, so content could forge a second entry that
looks authentic.

The code claimed syslog(3) escapes control characters itself. It does not: glibc
formats the buffer and hands it over, and the escaping seen on a mainstream Linux
is the daemon's. Measured with LOG_PERROR, an embedded newline comes out as two
lines.

NewLineHandling mirrors the option RemoteSyslogAppender has had all along, which
already escaped by default. Keep restores the previous behaviour. The remote
appender's habit of dropping non-ASCII is deliberately not copied.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
OutputDebugStringW takes a null terminated string, so a NUL in logged content
ended the record there and dropped whatever the layout rendered after it.

The escape LocalSyslogAppender already had is now shared by both, since
EventLogAppender is the same shape and will want it too. Its tests moved onto the
shared helper with it.

The appender level test only runs on Windows: Append refuses to run elsewhere.
@FreeAndNil
FreeAndNil force-pushed the Feature/315-local-syslog-newlines branch from 6aca022 to 3e5e016 Compare September 2, 2026 19:20
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
ReportEventW takes a null terminated string, so a NUL in logged content ended the
stored record there and dropped whatever the layout rendered after it. WriteEntry
raises nothing, so the record simply stored short and no ErrorHandler call fired.

Measured on Windows 11 build 26200: a 45 character message with a NUL at 23 stored
as its 23 character prefix.

Escaping happens before the size limit is applied, since it doubles each NUL, and
PrepareEventText exists so that ordering can be tested without an event log.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026


RFC 3164 allows only visible ASCII and space in the message, and everything else
fell through the loop unwritten. "Schoenwetter <CJK>" reached the collector as
"Schnwetter ", and a tab vanished from between its neighbours, with no marker and
no error.

Such characters are now written as a \uXXXX escape, which stays inside the allowed
range. Encoding still cannot make the message body non-ASCII, which the appender
page now says.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
Does.Not.Contain is culture sensitive, and a culture sensitive comparison treats
NUL as ignorable: it reports a match in a string that contains none. Both new
escape tests therefore failed on Windows against correctly escaped output.

ContainsConstraint has no comparison knob at all, so these use
Contains.Substring(x).Using(StringComparison.Ordinal), negated with the ! operator
Constraint defines. The EventLog test asserts the whole value instead, which is
ordinal and pins the length too.

This is the shape f018 reports in StringMatchFilter, which is still open.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
LocalSyslogAppender needs the same option, so nesting it in one of the two
appenders no longer fits.

Breaking for code naming RemoteSyslogAppender.SyslogNewLineHandling. Configuration
binds the value by name and is unaffected.
@FreeAndNil
FreeAndNil force-pushed the Feature/315-local-syslog-newlines branch from 3e5e016 to 65c603f Compare September 2, 2026 20:44
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
A newline in logged content ends the record for a syslog daemon that writes the
message through to a line oriented log, so content could forge a second entry that
looks authentic.

The code claimed syslog(3) escapes control characters itself. It does not: glibc
formats the buffer and hands it over, and the escaping seen on a mainstream Linux
is the daemon's. Measured with LOG_PERROR, an embedded newline comes out as two
lines.

NewLineHandling mirrors the option RemoteSyslogAppender has had all along, which
already escaped by default. Keep restores the previous behaviour. The remote
appender's habit of dropping non-ASCII is deliberately not copied.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
OutputDebugStringW takes a null terminated string, so a NUL in logged content
ended the record there and dropped whatever the layout rendered after it.

The escape LocalSyslogAppender already had is now shared by both, since
EventLogAppender is the same shape and will want it too. Its tests moved onto the
shared helper with it.

The appender level test only runs on Windows: Append refuses to run elsewhere.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
ReportEventW takes a null terminated string, so a NUL in logged content ended the
stored record there and dropped whatever the layout rendered after it. WriteEntry
raises nothing, so the record simply stored short and no ErrorHandler call fired.

Measured on Windows 11 build 26200: a 45 character message with a NUL at 23 stored
as its 23 character prefix.

Escaping happens before the size limit is applied, since it doubles each NUL, and
PrepareEventText exists so that ordering can be tested without an event log.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026


RFC 3164 allows only visible ASCII and space in the message, and everything else
fell through the loop unwritten. "Schoenwetter <CJK>" reached the collector as
"Schnwetter ", and a tab vanished from between its neighbours, with no marker and
no error.

Such characters are now written as a \uXXXX escape, which stays inside the allowed
range. Encoding still cannot make the message body non-ASCII, which the appender
page now says.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
Does.Not.Contain is culture sensitive, and a culture sensitive comparison treats
NUL as ignorable: it reports a match in a string that contains none. Both new
escape tests therefore failed on Windows against correctly escaped output.

ContainsConstraint has no comparison knob at all, so these use
Contains.Substring(x).Using(StringComparison.Ordinal), negated with the ! operator
Constraint defines. The EventLog test asserts the whole value instead, which is
ordinal and pins the length too.

This is the shape f018 reports in StringMatchFilter, which is still open.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
The limit is a whole record budget, and the log name, the source and the machine
name are spent from it one character for one. The fixed 31837 sat above the real
ceiling, so log4net truncated to a size the service then discarded: the event was
lost whole rather than shortened, with no exception and no record.

Measured on Windows 11 build 26200 over five source name lengths and two log names
with no residual: stored while message + logName + applicationName stays within
31736. One character more and nothing is stored.

ApplicationName defaults to the app domain name, so the consumer's assembly name
came out of the budget invisibly. A 1024 margin is held back because crossing the
line is not one lost message: the write still consumes log space, and a log given
about thirty of them was later found reporting a negative record count.

Truncation is now reported through the error handler. There is no channel where
the service records a dropped write, so that is the only signal available.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
LocalSyslogAppender needs the same option, so nesting it in one of the two
appenders no longer fits.

Breaking for code naming RemoteSyslogAppender.SyslogNewLineHandling. Configuration
binds the value by name and is unaffected.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
A newline in logged content ends the record for a syslog daemon that writes the
message through to a line oriented log, so content could forge a second entry that
looks authentic.

The code claimed syslog(3) escapes control characters itself. It does not: glibc
formats the buffer and hands it over, and the escaping seen on a mainstream Linux
is the daemon's. Measured with LOG_PERROR, an embedded newline comes out as two
lines.

NewLineHandling mirrors the option RemoteSyslogAppender has had all along, which
already escaped by default. Keep restores the previous behaviour. The remote
appender's habit of dropping non-ASCII is deliberately not copied.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
OutputDebugStringW takes a null terminated string, so a NUL in logged content
ended the record there and dropped whatever the layout rendered after it.

The escape LocalSyslogAppender already had is now shared by both, since
EventLogAppender is the same shape and will want it too. Its tests moved onto the
shared helper with it.

The appender level test only runs on Windows: Append refuses to run elsewhere.
@FreeAndNil
FreeAndNil force-pushed the Feature/315-local-syslog-newlines branch from 65c603f to fc122b1 Compare September 2, 2026 21:32
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
ReportEventW takes a null terminated string, so a NUL in logged content ended the
stored record there and dropped whatever the layout rendered after it. WriteEntry
raises nothing, so the record simply stored short and no ErrorHandler call fired.

Measured on Windows 11 build 26200: a 45 character message with a NUL at 23 stored
as its 23 character prefix.

Escaping happens before the size limit is applied, since it doubles each NUL, and
PrepareEventText exists so that ordering can be tested without an event log.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026


RFC 3164 allows only visible ASCII and space in the message, and everything else
fell through the loop unwritten. "Schoenwetter <CJK>" reached the collector as
"Schnwetter ", and a tab vanished from between its neighbours, with no marker and
no error.

Such characters are now written as a \uXXXX escape, which stays inside the allowed
range. Encoding still cannot make the message body non-ASCII, which the appender
page now says.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
Does.Not.Contain is culture sensitive, and a culture sensitive comparison treats
NUL as ignorable: it reports a match in a string that contains none. Both new
escape tests therefore failed on Windows against correctly escaped output.

ContainsConstraint has no comparison knob at all, so these use
Contains.Substring(x).Using(StringComparison.Ordinal), negated with the ! operator
Constraint defines. The EventLog test asserts the whole value instead, which is
ordinal and pins the length too.

This is the shape f018 reports in StringMatchFilter, which is still open.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
The limit is a whole record budget, and the log name, the source and the machine
name are spent from it one character for one. The fixed 31837 sat above the real
ceiling, so log4net truncated to a size the service then discarded: the event was
lost whole rather than shortened, with no exception and no record.

Measured on Windows 11 build 26200 over five source name lengths and two log names
with no residual: stored while message + logName + applicationName stays within
31736. One character more and nothing is stored.

ApplicationName defaults to the app domain name, so the consumer's assembly name
came out of the budget invisibly. A 1024 margin is held back because crossing the
line is not one lost message: the write still consumes log space, and a log given
about thirty of them was later found reporting a negative record count.

Truncation is now reported through the error handler. There is no channel where
the service records a dropped write, so that is the only signal available.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
The helper held only the NUL escape. It now also escapes unpaired surrogates,
which f013 needs and f011 will, so the name no longer fitted.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
The default writer encoding threw on an unpaired surrogate, and Send reads a throw
as a client that hung up, so one event reached nobody and disconnected everybody.
Escaped as \uXXXX now; the non-throwing encoding stays as belt and braces.
FreeAndNil added a commit that referenced this pull request Sep 2, 2026
File.CreateText throws on an unpaired surrogate, which abandoned the whole buffered
batch and left a truncated mail for the pickup service to send. Reverting the fix
leaves the test with a file that exists and is empty.

Writing under the final name stays as it was, with a note why.
@FreeAndNil FreeAndNil added this to the 3.5.0 milestone Sep 2, 2026
LocalSyslogAppender needs the same option, so nesting it in one of the two
appenders no longer fits.

Breaking for code naming RemoteSyslogAppender.SyslogNewLineHandling. Configuration
binds the value by name and is unaffected.
A newline in logged content ends the record for a syslog daemon that writes the
message through to a line oriented log, so content could forge a second entry that
looks authentic.

The code claimed syslog(3) escapes control characters itself. It does not: glibc
formats the buffer and hands it over, and the escaping seen on a mainstream Linux
is the daemon's. Measured with LOG_PERROR, an embedded newline comes out as two
lines.

NewLineHandling mirrors the option RemoteSyslogAppender has had all along, which
already escaped by default. Keep restores the previous behaviour. The remote
appender's habit of dropping non-ASCII is deliberately not copied.
OutputDebugStringW takes a null terminated string, so a NUL in logged content
ended the record there and dropped whatever the layout rendered after it.

The escape LocalSyslogAppender already had is now shared by both, since
EventLogAppender is the same shape and will want it too. Its tests moved onto the
shared helper with it.

The appender level test only runs on Windows: Append refuses to run elsewhere.
ReportEventW takes a null terminated string, so a NUL in logged content ended the
stored record there and dropped whatever the layout rendered after it. WriteEntry
raises nothing, so the record simply stored short and no ErrorHandler call fired.

Measured on Windows 11 build 26200: a 45 character message with a NUL at 23 stored
as its 23 character prefix.

Escaping happens before the size limit is applied, since it doubles each NUL, and
PrepareEventText exists so that ordering can be tested without an event log.


RFC 3164 allows only visible ASCII and space in the message, and everything else
fell through the loop unwritten. "Schoenwetter <CJK>" reached the collector as
"Schnwetter ", and a tab vanished from between its neighbours, with no marker and
no error.

Such characters are now written as a \uXXXX escape, which stays inside the allowed
range. Encoding still cannot make the message body non-ASCII, which the appender
page now says.
Does.Not.Contain is culture sensitive, and a culture sensitive comparison treats
NUL as ignorable: it reports a match in a string that contains none. Both new
escape tests therefore failed on Windows against correctly escaped output.

ContainsConstraint has no comparison knob at all, so these use
Contains.Substring(x).Using(StringComparison.Ordinal), negated with the ! operator
Constraint defines. The EventLog test asserts the whole value instead, which is
ordinal and pins the length too.

This is the shape f018 reports in StringMatchFilter, which is still open.
The limit is a whole record budget, and the log name, the source and the machine
name are spent from it one character for one. The fixed 31837 sat above the real
ceiling, so log4net truncated to a size the service then discarded: the event was
lost whole rather than shortened, with no exception and no record.

Measured on Windows 11 build 26200 over five source name lengths and two log names
with no residual: stored while message + logName + applicationName stays within
31736. One character more and nothing is stored.

ApplicationName defaults to the app domain name, so the consumer's assembly name
came out of the budget invisibly. A 1024 margin is held back because crossing the
line is not one lost message: the write still consumes log space, and a log given
about thirty of them was later found reporting a negative record count.

Truncation is now reported through the error handler. There is no channel where
the service records a dropped write, so that is the only signal available.
The helper held only the NUL escape. It now also escapes unpaired surrogates,
which f013 needs and f011 will, so the name no longer fitted.
The default writer encoding threw on an unpaired surrogate, and Send reads a throw
as a client that hung up, so one event reached nobody and disconnected everybody.
Escaped as \uXXXX now; the non-throwing encoding stays as belt and braces.
Four appenders have needed the same two escapes. New ones belong in ContentEscape,
and escaping comes before any length limit, not after.
File.CreateText throws on an unpaired surrogate, which abandoned the whole buffered
batch and left a truncated mail for the pickup service to send. Reverting the fix
leaves the test with a file that exists and is empty.

Writing under the final name stays as it was, with a note why.
@FreeAndNil
FreeAndNil force-pushed the Feature/315-local-syslog-newlines branch from ef97324 to 4d2e10f Compare September 3, 2026 09:53
@FreeAndNil
FreeAndNil marked this pull request as ready for review September 3, 2026 09:54
@FreeAndNil FreeAndNil changed the title Feature/315 local syslog newlines Escaping special chacters in multiple appenders Sep 3, 2026
@FreeAndNil FreeAndNil changed the title Escaping special chacters in multiple appenders Escaping special characters in multiple appenders Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant