Hi gVisor Team,
I’ve been investigating a performance issue in gVisor that seems related to file-system cleanup during container exit.
The main bottleneck appears to be a noticeable stall in the Gofer process during the execution worker’s cleanup phase. From the attached gVisor logs, we occasionally observe ~2-second gaps between the `sock read failed, closing connection: EOF entries.`
We also ran experiments with different directory structures. Mounting a folder with 1,000 subdirectories × 1 small file each results in normal cleanup performance, whereas 10 subdirectories × 100 files each (same total file count) sometimes triggers significant slowdown during Gofer cleanup.
Could someone from the team help answer the following questions?
1. Is this proportional, blocking slowdown during Gofer’s cleanup on process exit expected when a mount contains a large amount of file state, and could the directory/file structure also contribute to this behavior?
2. What are the recommended best practices for handling this volume of file state? Are there configurations or flags that allow early cleanup of stale files (e.g., any file flag)?
Thank you for your time and expertise.
Best,
Zoey
Slower gVisor log:
D1029 22:12:13.522557 1 task_exit.go:215] [ 1: 2] Transitioning from exit state TaskExitNone to TaskExitInitiated
D1029 22:12:13.525479 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528093 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528146 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528186 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528233 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528273 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528316 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528364 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528484 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.528498 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:13.533563 1 sampler.go:191] Time: Adjusting syscall overhead down to 671
D1029 22:12:15.534241 1 sampler.go:191] Time: Adjusting syscall overhead down to 875
D1029 22:12:15.934951 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:15.934988 1 connection.go:127] sock read failed, closing connection: EOF
:15.935030 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:15.935083 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:12:15.935220 1 connection.go:127] sock read failed, closing connection: EOF
I1029 22:12:15.935212 1 loader.go:1289] Gofer socket disconnected, killing container ...
Normal gVisor log:
D1029 22:20:04.135268 1 task_exit.go:215] [ 1: 2] Transitioning from exit state TaskExitNone to TaskExitInitiated
D1029 22:20:04.138439 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.138460 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.138515 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.138537 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.138567 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.138654 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.138660 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.138710 1 connection.go:127] sock read failed, closing connection: EOF
on: EOF
D1029 22:20:04.138797 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.188498 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.188534 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.190730 1 connection.go:127] sock read failed, closing connection: EOF
D1029 22:20:04.190764 1 connection.go:127] sock read failed, closing connection: EOF
I1029 22:20:04.190882 1 loader.go:1289] Gofer socket disconnected, killing container ...
