Problem with eio_clean_thr process

[ssaba-a]

Hi,

I compiled and installed master version in kernel 4.14.34.
When writing sequentially into the disk, the cache device gets full and cannot evict data from the cache.
eio_clean_thr sticks in D state and the eviction (flushing/cleaning) process cannot continue even after stopping IO (sequential write).
This is the stack trace of the process:

eio_clean_thr D 0 28023 2 0x80000080
Dec 7 12:08:46 srv01 kernel: Call Trace:
Dec 7 12:08:46 srv01 kernel: ? __schedule+0x1ad/0x6a0
Dec 7 12:08:46 srv01 kernel: schedule+0x32/0x80
Dec 7 12:08:46 srv01 kernel: rwsem_down_write_failed+0x1fe/0x380
Dec 7 12:08:46 srv01 kernel: call_rwsem_down_write_failed+0x13/0x20
Dec 7 12:08:46 srv01 kernel: down_write+0x29/0x40
Dec 7 12:08:46 srv01 kernel: eio_clean_set+0x14c/0x9f0 [enhanceio]
Dec 7 12:08:46 srv01 kernel: ? del_timer_sync+0x35/0x40
Dec 7 12:08:46 srv01 kernel: ? call_timer_fn+0x130/0x130
Dec 7 12:08:46 srv01 kernel: eio_clean_thread_proc+0x1bc/0x360 [enhanceio]
Dec 7 12:08:46 srv01 kernel: ? __schedule+0x1b5/0x6a0
Dec 7 12:08:46 srv01 kernel: kthread+0xfc/0x130
Dec 7 12:08:46 srv01 kernel: ? eio_clean_all+0xd0/0xd0 [enhanceio]
Dec 7 12:08:46 srv01 kernel: ? __kthread_parkme+0x70/0x70
Dec 7 12:08:46 srv01 kernel: ret_from_fork+0x35/0x40

Thanks,
Saba

[ssaba-a]

Please note that when eio is locked IO can be issued directly in both source and cache devices.

[ssaba-a]

I just compiled and installed master version in kernel 4.14.158. I still face the same bug in this kernel.
Here is the stack trace:
enhanceio_lru: Initialized 32639 sets in LRU
Dec 7 14:48:09 srv01 kernel: sysrq: SysRq : Show Blocked State
Dec 7 14:48:09 srv01 kernel: task PC stack pid father
Dec 7 14:48:09 srv01 kernel: eio_clean_threa D 0 3640 2 0x80000080
Dec 7 14:48:09 srv01 kernel: Call Trace:
Dec 7 14:48:09 srv01 kernel: ? schedule+0x1b0/0x6b0
Dec 7 14:48:09 srv01 kernel: ? switch_to_asm+0x41/0x70
Dec 7 14:48:09 srv01 kernel: ? switch_to_asm+0x35/0x70
Dec 7 14:48:09 srv01 kernel: schedule+0x32/0x80
Dec 7 14:48:09 srv01 kernel: rwsem_down_write_failed+0x206/0x380
Dec 7 14:48:09 srv01 kernel: ? switch_to_asm+0x41/0x70
Dec 7 14:48:09 srv01 kernel: ? switch_to_asm+0x35/0x70
Dec 7 14:48:09 srv01 kernel: call_rwsem_down_write_failed+0x13/0x20
Dec 7 14:48:09 srv01 kernel: down_write+0x29/0x40
Dec 7 14:48:09 srv01 kernel: eio_clean_set+0x14c/0x980 [enhanceio]
Dec 7 14:48:09 srv01 kernel: ? del_timer_sync+0x35/0x40
Dec 7 14:48:09 srv01 kernel: ? call_timer_fn+0x140/0x140
Dec 7 14:48:09 srv01 kernel: eio_clean_thread_proc+0x1bc/0x360 [enhanceio]
Dec 7 14:48:09 srv01 kernel: ? schedule+0x1b8/0x6b0
Dec 7 14:48:09 srv01 kernel: kthread+0xff/0x140
Dec 7 14:48:09 srv01 kernel: ? eio_clean_all+0xd0/0xd0 [enhanceio]
Dec 7 14:48:09 srv01 kernel: ? __kthread_parkme+0x90/0x90
Dec 7 14:48:09 srv01 kernel: ret_from_fork+0x35/0x40

[ssaba-a]

Master version flushes cache without any problem in kernel 3.10.0. But it takes too long to fill the cache (compared to kernel 4.14) with the same sequential write workload.

[ssaba-a]

I tested the master version on kernel 5.4.2 and the flush process works correctly.