Please test lusca-head as much as possible!

[Adrian Chadd]

Hi everyone!

I'm about to start on some work to finally replace the horrible HTTP
request/reply processing code and the HTTP request/reply/header
parser. I've spent quite a bit of time tidying things up and
reorganising code in preparation for this so it won't be as big a task
as it would've been in the past. It is still going to be quite risky,
so I'd like to make certain the rest of the codebase is as bug-free
and production-ready as possible.

I've had a few emails in private and to the list about weird stuff
that they're seeing, from rebuild failures to crashes. Please create
issues through the google code project
(http://code.google.com/p/lusca-cache/issues/list) so I can track what
kind of broken behaviour you are all seeing.

I'm hoping that the result from this work will be both cleaner, faster
-and- more memory efficient code (shock, horror!) which should result
in Lusca handling higher request rates than any previous and current
Squid version.

So please, please, please, test and report as much as you can!

Adrian

[Adrian Chadd]

I'm going to assume the lack of responses means that Lusca is working
100% for everyone with no issues, and that noone has tried it and
found any issues preventing them from deploying it.

Sigh. :)

Adrian

2009/9/12 Adrian Chadd <adrian...@gmail.com>:

[Odhiambo Washington]

On Tue, Sep 15, 2009 at 7:38 AM, Adrian Chadd <adrian...@gmail.com> wrote:

I'm going to assume the lack of responses means that Lusca is working
100% for everyone with no issues, and that noone has tried it and
found any issues preventing them from deploying it.

Hello Adrian,

I have been using -head since you posted, but mine is a small setup. Is there anything in particular I should look for and report on? I've not noticed any problems at all:)

 

--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254733744121/+254722743223
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
"If you have nothing good to say about someone, just shut up!."
              -- Lucky Dube

[Adrian Chadd]

Just say "its working fine for me" :)



adrian

2009/9/15 Odhiambo Washington <odhi...@gmail.com>:

[Odhiambo Washington]

On Tue, Sep 15, 2009 at 8:42 AM, Adrian Chadd <adrian...@gmail.com> wrote:

Just say "its working fine for me" :)

In that case, it is fine, since noone has complained to me. I have no complaints of myself. I use aufs, if that matters.
 

[Faysal Banna]

adrian my friend
i use LUCA_HEAD since day0
i am faceing an issue where memory is always growing untill it starts getting from the swap memory unless i manually issue sysctl -w vm.drop_caches=3
besides when i restart squid i get tlv_unpack stuff i posted before

other than this its working fine
yet sometimes it just takes hell lots of time to reply the squidclient request

much regards

--
============================
        Faysal Banna
Meteorological Services
Rafic Harriri International Airport
     Beirut - Lebanon
   Mob: +961-3-258043
=============================

[Nyamul Hassan]

I've been running the head from Aug/11/2009.  This is an extensive setup with 12,000+ req / sec since the last restart 9 days ago.  However, this head did crash once on Sep 6, since it has been put in production on Aug/11.

The only complain that I have is that a lot of the counters in the CacheMgr show "negative" values.  Perhaps some integers have to be played with.  Am I out of date?  If yes, then I can download and retry.

Regards

HASSAN

--
Nyamul Hassan
=======================
IS Pros Limited
Zeenat Bhaban (5th Floor)
41/1 Kazi Nazrul Islam Avenue
Kawranbazaar, Dhaka
Tel:  +880 2 8116422, 8125967

[Adrian Chadd]

Tell me which counters are showing up negative under that load and
I'll turn them into 64 bit counters.

12,000 requests a second? In production? Seriously? :)

Thanks,


adrian

2009/9/15 Nyamul Hassan <mnha...@usa.net>:

[Adrian Chadd]

If you're using COSS then you'll occasionally get those tlv_unpack
errors at startup. There's some random COSS bugs that I haven't yet
put my finger on. I'll sort it out at some point.

I've no idea why your system is growing in memory use the way it is.
:( It shouldn't be going into swap purely because of the VM/buffer
cache growth.

2009/9/15 Faysal Banna <degr...@gmail.com>:

[Faysal Banna]

am running Lusca with 30Mbit/sec and around 10,000 client....
had no problem except those i told you about
in squidclient i see 16000 + requests per minute
much regards

[Faysal Banna]

Adrian my friend
this is what happens
it keeps on growing untill it starts getting from swap ... or i manually do sync && sysctl -w vm.drop_caches=3
anyhow other than this its running smooth and fine with something around 10000+ client and 300~400 req/sec

[Adrian Chadd]

Good to hear, thanks!


2009/9/15 Faysal Banna <degr...@gmail.com>:

[Faysal Banna]

ah yes
i remember
my issue was with lusca T-Proxy setup and iptables/ebtables bridge setup
after a short while the system becomes unstable with kernel panic on kernel 2.6.29

there was a ticket by pokemon before about this
i need to verify the setup of T-Proxy as i didn't deploy it yet for two main issues
1st is the multiroute issue
2nd and most important was the kernel panic in bridge setup

if those you may have them solved then we would be gratefull if you post notes of simple setup for them

and in ur basic opinion
settin Tproxy in bridge vs gateway ... which one is better !???

[pokemon]

I agreed with faysal still T-proxy we can't use in bridge mode.
another issue i was found with user experience. The cache sites
results is very fast but HTTP-MISS in peek hours are very bad. 2 days
ago i putted 70 MB load on single machine. the clients are complaining
about slow performance even client not appropriate speed http download
on single session. after off peek hours when the input was 35 MB
everything was normal. i checked dns, it was perfect.

On Sep 15, 1:40 pm, Faysal Banna <degre...@gmail.com> wrote:
> ah yes
> i remember
> my issue was with lusca T-Proxy setup and iptables/ebtables bridge setup
> after a short while the system becomes unstable with kernel panic on kernel
> 2.6.29
>
> there was a ticket by pokemon before about this
> i need to verify the setup of T-Proxy as i didn't deploy it yet for two main
> issues
> 1st is the multiroute issue
> 2nd and most important was the kernel panic in bridge setup
>
> if those you may have them solved then we would be gratefull if you post
> notes of simple setup for them
>
> and in ur basic opinion
> settin Tproxy in bridge vs gateway ... which one is better !???
>

> On Tue, Sep 15, 2009 at 10:32 AM, Adrian Chadd <adrian.ch...@gmail.com>wrote:
>
>
>
>
>
>
>
> > Good to hear, thanks!
>

> > 2009/9/15 Faysal Banna <degre...@gmail.com>:

>
> > > Adrian my friend
> > > this is what happens
> > > it keeps on growing untill it starts getting from swap ... or i manually
> > do
> > > sync && sysctl -w vm.drop_caches=3
> > > anyhow other than this its running smooth and fine with something around
> > > 10000+ client and 300~400 req/sec
>

> > > On Tue, Sep 15, 2009 at 10:22 AM, Adrian Chadd <adrian.ch...@gmail.com>

> > > wrote:
>
> > >> If you're using COSS then you'll occasionally get those tlv_unpack
> > >> errors at startup. There's some random COSS bugs that I haven't yet
> > >> put my finger on. I'll sort it out at some point.
>
> > >> I've no idea why your system is growing in memory use the way it is.
> > >> :( It shouldn't be going into swap purely because of the VM/buffer
> > >> cache growth.
>

> > >> 2009/9/15 Faysal Banna <degre...@gmail.com>:

>
> > >> > adrian my friend
> > >> > i use LUCA_HEAD since day0
> > >> > i am faceing an issue where memory is always growing untill it starts
> > >> > getting from the swap memory unless i manually issue sysctl -w
> > >> > vm.drop_caches=3
> > >> > besides when i restart squid i get tlv_unpack stuff i posted before
>
> > >> > other than this its working fine
> > >> > yet sometimes it just takes hell lots of time to reply the squidclient
> > >> > request
>
> > >> > much regards
> > >> > On Tue, Sep 15, 2009 at 8:47 AM, Odhiambo Washington

> > >> > <odhia...@gmail.com>

> > >> > wrote:
>
> > >> >> On Tue, Sep 15, 2009 at 8:42 AM, Adrian Chadd <

> > adrian.ch...@gmail.com>

> =============================- Hide quoted text -
>
> - Show quoted text -

[Nyamul Hassan]

Sorry for the confusion on "per second".  It is 12000+ req per min.  Around 4000 users, doing around 60 Mbps download during peak hours.  The only bottle neck that I do see now is on the IOWait, which goes to almost 100% during peak load.

I don't visit every single page on CacheMGR all the time.  The most frequent for me are "General Runtime Info", and "Directory Stats".  I've removed all other entries apart from the negative ones that caught my eye just now:

General Runtime Information
=> Connection information for squid:
=> => Select loop called: -1797068016 times, -0.451 ms avg
=> Cache information for squid:
=> => Storage Swap size:    -812472178 KB
=> => Mean Object Size:    -149.12 KB
=> Memory usage for squid via mallinfo():
=> => Total space in arena:  -1616572 KB
=> => Ordinary blocks:       -1817937 KB  51662 blks
=> => Total in use:          -1740229 KB 113%
=> => Total free:            201364 KB -12%
=> => Total size:            -1538864 KB
Store Directory Stats
=> Maximum Swap Size      : -791908552 KB
=> Current Store Swap Size: -812513440 KB

Are these normal?  They don't seem to be there with Squid 2.7STABLE6

Do you recommend I upgrade from my Aug/8 head to most latest one?

[Adrian Chadd]

G'day,

Yes, I'd suggest upgrading to the very latest lusca-head and test
that. Make sure you do a full reconfigure, make, make install.

IOwait - are you using any COSS directories? Or are they all AUFS?



adrian

2009/9/16 Nyamul Hassan <mnha...@usa.net>:

[Nyamul Hassan]

I've got 4 physical disks to do the store, each of them having 1 COSS and 1 AUFS.

I'm going to upgrade right now and let you know how it's going.

Regards
HASSAN