FLUME-3325 fixed TaildirSource oom issue#279
Conversation
when tailing a lot of files where some file is writing frequently but the others are idle can cause the idleInodes List increasing infinitely, change it to set(list backend) can fix this problem.
This reverts commit 80325c6.
when tailing a lot of files where some file is writing frequently but the others are idle can cause the idleInodes List increasing infinitely, change it to set(list backend) can fix this problem.
|
Hi @oldbelvey, Thank you for this PR! The change looks good, but can you please explain why the change resolves the issue described in the title? |
Hello bessbd, the oom problem is caused by repeatedly add same value into the list. changing list to set just preventing from adding repeated values. |
bessbd
left a comment
bessbd
left a comment
There was a problem hiding this comment.
LGTM, +1
Thank you for this change, @oldbelvey
|
@oldbelvey : can you please include some directions to prove this change resolves the issue? |
|
@bessbd I don't think it's easy to reproduce. Tho OOM happens at our production environment, where one flume agent monitored more than fifteen app's log. And as it is described at the JIRA, with one file frequently write while they others are idle, the oom will happen right after the Arraylist boom. |
2 participants
when tailing a lot of files where some file is writing frequently but the others are idle can cause the idleInodes List increasing infinitely, change it to set(list backend) can fix this problem.