Skip to content

in_tail: restrict file_cache_advise eviction to the consumed range #12322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
baizhenyu wants to merge 1 commit into
base: master
Choose a base branch
from
Open

in_tail: restrict file_cache_advise eviction to the consumed range #12322

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 39 additions & 3 deletions plugins/in_tail/tail_file.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1907,6 +1907,11 @@ int flb_tail_file_chunk(struct flb_tail_file *file)
char *tmp;
int ret;
int lines;
#ifdef __linux__
int64_t advise_end;
long page_size;
int advise_ret;
#endif
struct flb_tail_config *ctx;

/* Check if we the engine issued a pause */
Expand Down Expand Up @@ -2001,9 +2006,40 @@ int flb_tail_file_chunk(struct flb_tail_file *file)

#ifdef __linux__
if (ctx->file_cache_advise) {
if (posix_fadvise(file->fd, 0, 0, POSIX_FADV_DONTNEED) == -1) {
flb_errno();
flb_plg_error(ctx->ins, "error during posix_fadvise");
/*
* Advise only the page-aligned byte range that has already been
* consumed (up to file->offset).
*
* Advising the whole file would also hit the last page, which the
* writer is still appending to: POSIX_FADV_DONTNEED starts
* writeback of dirty pages, so the same tail page would be flushed
* to disk over and over (write amplification). It would also evict
* pages that other readers of the file (e.g. log rotation
* compressors) have not consumed yet, turning their cache hits
* into physical reads.
*
* The advice is repeated over the whole consumed range on every
* cycle on purpose: the kernel cannot drop pages that are still
* dirty when advised, it only starts their writeback, so they can
* only be evicted by a later call once they are clean. Consumed
* pages are never re-dirtied (the writer only appends), so each
* page is still written back at most once, and re-advising a
* mostly evicted range is cheap.
*/
page_size = sysconf(_SC_PAGESIZE);
if (page_size > 0) {
advise_end = file->offset & ~((int64_t) page_size - 1);
if (advise_end > 0) {
advise_ret = posix_fadvise(file->fd, 0, (off_t) advise_end,
POSIX_FADV_DONTNEED);
if (advise_ret != 0) {
/* posix_fadvise() returns the error number, it does
* not set errno */
flb_plg_error(ctx->ins,
"posix_fadvise error=%i file=%s",
advise_ret, file->name);
}
}
}
}
#endif
Expand Down
Loading