Strange Error in Lusca-Head

[Nyamul Hassan]

Hi,

I just compiled LuscaHead as of 0700 hrs GMT today, and I was surprised how
fast it was in loading up my "cache_dir". It loaded 4 x 20G COSS and 4 x
105G AUFS (in 4 x 160G SATA) in less than 7 mins.

However, after it loaded, I saw the following error message as soon as all
the store dirs were loaded

WARNING: Disk space over limit: 1425104352 KB > 536870912 KB

And, after the "storeLateRelease" statement, I get the following message
almost every 20 - 30 seconds:

squidaio_queue_request: WARNING - Disk I/O overloading
squidaio_queue_request: Queue Length: current=652, high=723, low=361,
duration=20

What does this indicate? Can you help me please?

Regards
HASSAN

[Nyamul Hassan]

Strangely, I have another server configured similarly which is running with
LuscaHead as of 0800 GMT yesterday, and that one does not show the same
error.

Regards
HASSAN

[Adrian Chadd]

This means it hasn't configured all of the cache dirs. How weird?

2009/8/11 Nyamul Hassan <mnha...@usa.net>:

[Nyamul Hassan]

Yes, you are right. When I check the "Store Directory Stats" in "CacheMgr",
I see "negative" values for "store swap size", and "current capacity".

What could be the problem?

Regards
HASSAN



----- Original Message -----

[Adrian Chadd]

I'm not certain. Can you copy/paste the store directory stats output?
It is probably something to do with integer overflow/underflow which I
can investigate and fix.



Adrian

2009/8/11 Nyamul Hassan <mnha...@usa.net>:

[Nyamul Hassan]

Here is what I see in CacheMgr:

Store Directory Statistics:
Store Entries : 6680429
Maximum Swap Size : 536870912 KB
Current Store Swap Size: -1721490088 KB
Current Capacity : -2147483648% used, 421% free

Store Directory #0 (coss): /cachestore/cache1/coss
FS Block Size 2048 Bytes

Maximum Size: 20971520 KB
Current Size: 16671334 KB
Percent Used: 79.50%
Current load metric: 0 / 1000
Number of object collisions: 0
Flags:
Pending Relocations: 0
Current Stripe: 20479
Stripe: 0, lockcount: 0, numobjects 34, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,MEMONLY
Stripe: 20479, lockcount: 0, numobjects 0, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,DISK
Pending Relocations:
Accepted object sizes: 0 - 131072 bytes

Store Directory #1 (coss): /cachestore/cache2/coss
FS Block Size 2048 Bytes

Maximum Size: 20971520 KB
Current Size: 15873790 KB
Percent Used: 75.69%
Current load metric: 0 / 1000
Number of object collisions: 0
Flags: SELECTED
Pending Relocations: 0
Current Stripe: 0
Stripe: 0, lockcount: 0, numobjects 83, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,DISK
Stripe: 0, lockcount: 0, numobjects 11, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,MEMONLY
Pending Relocations:
Accepted object sizes: 0 - 131072 bytes

Store Directory #2 (coss): /cachestore/cache3/coss
FS Block Size 2048 Bytes

Maximum Size: 20971520 KB
Current Size: 16643822 KB
Percent Used: 79.36%
Current load metric: 0 / 1000
Number of object collisions: 0
Flags:
Pending Relocations: 0
Current Stripe: 20479
Stripe: 0, lockcount: 0, numobjects 125, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,MEMONLY
Stripe: 20479, lockcount: 0, numobjects 3, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,DISK
Pending Relocations:
Accepted object sizes: 0 - 131072 bytes

Store Directory #3 (coss): /cachestore/cache4/coss
FS Block Size 2048 Bytes

Maximum Size: 20971520 KB
Current Size: 16218990 KB
Percent Used: 77.34%
Current load metric: 0 / 1000
Number of object collisions: 0
Flags:
Pending Relocations: 0
Current Stripe: 20479
Stripe: 0, lockcount: 0, numobjects 15, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,MEMONLY
Stripe: 20479, lockcount: 0, numobjects 2, flags:
NOTFULL,NOTWRITING,NOTWRITTEN,DISK
Pending Relocations:
Accepted object sizes: 0 - 131072 bytes

Store Directory #4 (aufs): /cachestore/cache1
FS Block Size 4096 Bytes
First level subdirectories: 128
Second level subdirectories: 256
Maximum Size: 113246208 KB
Current Size: 107549348 KB
Percent Used: 94.97%
Current load metric: 100 / 1000
Filemap bits in use: 11115 of 16777216 (0%)
Filesystem Space in use: 72950292/153837436 KB (47%)
Filesystem Inodes in use: 95102/19546112 (0%)
Flags:
Accepted object sizes: 131073 - (unlimited) bytes
Removal policy: heap

Store Directory #5 (aufs): /cachestore/cache2
FS Block Size 4096 Bytes
First level subdirectories: 128
Second level subdirectories: 256
Maximum Size: 113246208 KB
Current Size: -882356384 KB
Percent Used: -779.15%
Current load metric: 100 / 1000
Filemap bits in use: 71190 of 16777216 (0%)
Filesystem Space in use: 133661920/153837436 KB (87%)
Filesystem Inodes in use: 154332/19546112 (1%)
Flags:
Accepted object sizes: 131073 - (unlimited) bytes
Removal policy: heap

Store Directory #6 (aufs): /cachestore/cache3
FS Block Size 4096 Bytes
First level subdirectories: 128
Second level subdirectories: 256
Maximum Size: 113246208 KB
Current Size: -1119642680 KB
Percent Used: -988.68%
Current load metric: 100 / 1000
Filemap bits in use: 71571 of 16777216 (0%)
Filesystem Space in use: 133099224/153837436 KB (87%)
Filesystem Inodes in use: 154891/19546112 (1%)
Flags:
Accepted object sizes: 131073 - (unlimited) bytes
Removal policy: heap

Store Directory #7 (aufs): /cachestore/cache4
FS Block Size 4096 Bytes
First level subdirectories: 128
Second level subdirectories: 256
Maximum Size: 113246208 KB
Current Size: 107551692 KB
Percent Used: 94.97%
Current load metric: 100 / 1000
Filemap bits in use: 118951 of 524288 (23%)
Filesystem Space in use: 133186312/153837436 KB (87%)
Filesystem Inodes in use: 155278/19546112 (1%)
Flags:
Accepted object sizes: 131073 - (unlimited) bytes
Removal policy: heap