dirty pages not being written out

[Robert Riches]

Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
writing out dirty blocks to disk?

Yesterday, the contents of a number of files were lost when I
crashed the machine in a failed experiment with SCSI rescan to
remove a hot-unplugged ESATA disk. (Rebuilding the software
RAID10s is another story...) Today, recompiling WINE left 860MB
of dirty blocks needing to be written to disk, according to
/proc/meminfo. After the compilation finished, I waited over 90
seconds, with the system essentially completely idle, and the
amount of dirty data did not go down. Then, when I did a 'sync'
(as a non-root user), immediately everything flushed out to disk
in about 10-15 seconds time and the amount of dirty data went to
zero. (I had "vmstat 3" and xcpustate running, along with a
periodic manual "grep Dirty /proc/meminfo".)

There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
which accoring to this page (and others I have read)

http://www.westnet.com/~gsmith/content/linux-pdflush.htm

means that the kernel's helper processes _should_ get busy and
start to write out any dirty blocks at _MOST_ 30 seconds after
said blocks went dirty. Even approx. 90 seconds after the
compilation job had finished, nothing was getting written. Based
on yesterday's experience, nothing would have been written even
many minutes later.

What gives? Why are my kernel and its helper processes being so
lazy?

Thanks.

--
Robert Riches
spamt...@jacob21819.net
(Yes, that is one of my email addresses.)

[The Natural Philosopher]

I'm no expert BUT I thought pages weren't flushed at all until the
buffers were needed..or at least until the system was pretty much idle.

http://www.makelinux.net/books/lkd2/ch15lev1sec4

seems to have some detail.

[Roger Blake]

On 2012-01-30, Robert Riches <spamt...@jacob21819.net> wrote:
> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
> writing out dirty blocks to disk?

man sync

--
-----------------------------------------------------------------------------
Roger Blake (Change "invalid" to "com" for email. Google Groups killfiled.)

"Climate policy has almost nothing to do anymore with environmental
protection... the next world climate summit in Cancun is actually
an economy summit during which the distribution of the world's
resources will be negotiated." -- Ottmar Edenhofer, IPCC
-----------------------------------------------------------------------------

[Robert Riches]

Yes, that page has info similar to what I had already studied.
These are what is in /proc/sys/vm/dirty*:

/proc/sys/vm/dirty_background_bytes:0
/proc/sys/vm/dirty_background_ratio:10
/proc/sys/vm/dirty_bytes:0
/proc/sys/vm/dirty_expire_centisecs:3000
/proc/sys/vm/dirty_ratio:20
/proc/sys/vm/dirty_writeback_centisecs:500

According to the descriptions on that page and elsewhere, the
worker threads should wake up ever 5 seconds and flush out
anything older than 30 second old. The system had been idle
(>90% idle CPU with 12 logical CPUs for the worker threads to
use, disks essentially completely idle) for over 90 seconds, and
_nothing_ was getting flushed to disk.

On other Linux systems, I have seen writeback happen very well
right around 30 seconds after the pages went dirty. Manually
running sync ever minute or so should _NOT_ be necessary.

[The Natural Philosopher]

Maybe the drivers were buffering everything...not the kernel.
I had that problem with a RAID once. you could even sync, but it was no
guarantee the data was on the disks. The RAID sub sytem reported 'all
written' and that was that. Except it wasn't. it was in the RAID system
buffers.

[Richard Kettlewell]

Robert Riches <spamt...@jacob21819.net> writes:
> Why is my kernel (Mageia 1's 2.6.38.8-server-10.mga) _NOT_
> writing out dirty blocks to disk?
>
> Yesterday, the contents of a number of files were lost when I
> crashed the machine in a failed experiment with SCSI rescan to
> remove a hot-unplugged ESATA disk. (Rebuilding the software
> RAID10s is another story...) Today, recompiling WINE left 860MB
> of dirty blocks needing to be written to disk, according to
> /proc/meminfo. After the compilation finished, I waited over 90
> seconds, with the system essentially completely idle, and the
> amount of dirty data did not go down. Then, when I did a 'sync'
> (as a non-root user), immediately everything flushed out to disk
> in about 10-15 seconds time and the amount of dirty data went to
> zero. (I had "vmstat 3" and xcpustate running, along with a
> periodic manual "grep Dirty /proc/meminfo".)
>
> There is a value of 3000 in /proc/sys/vm/dirty_expire_centisecs,
> which accoring to this page (and others I have read)
>
> http://www.westnet.com/~gsmith/content/linux-pdflush.htm

pdflush no longer exists, and dirty_expire_centisecs is no longer used.
Which doesn't answer your question as such but does explain why the old
documentation isn't helping.

As a starting point for the new logic:

http://kernelnewbies.org/Linux_2_6_32#head-72c3f91947738f1ea52f9ed21a89876730418a61

--
http://www.greenend.org.uk/rjk/

[Robert Riches]

On 2012-01-31, The Natural Philosopher <t...@invalid.invalid> wrote:
>...

>
> Maybe the drivers were buffering everything...not the kernel.
> I had that problem with a RAID once. you could even sync, but it was no
> guarantee the data was on the disks. The RAID sub sytem reported 'all
> written' and that was that. Except it wasn't. it was in the RAID system
> buffers.

The 'Dirty' line in /proc/meminfo said the pages were still in
dirty state. A manual sync did clear them out.

[Robert Riches]

Thanks. I hadn't heard solid reporting of the demise of pdflush,
but I had noticed some changes in the names of the kernel worker
or helper processes (but wrote it off as a result of going to my
first SMP machine). I bookmarked that page for study.