Hanging profile - load balanced servers

[WillW]

I have a profile that consistently hangs during processing, always
when processing the first log file of the second of two load-balanced
servers. (It's set up to run the logs serially - all of the "A" server
logs first followed by the "B" server logs.) We're running Urchin on
Win XP and the log files are from IIS.

I had a similar problem in Urchin 5. We discovered that there were
monitoring tools hitting the site frequently, creating excessively
long sessions. By filtering out the IP addresses of those tools we
were able to fix the Urchin 5 problem.

But in Urchin 6 the problem is back and the filters don't fix it this
time. I used Microsoft's LogParser tool to look at example log files
and identify any other high-volume IP addresses, but it identified
only those we were already filtering.

I am processing the same log files successfully in other profiles, but
those all have additional filters so that the profiles are limited to
certain content areas of the site. Those profiles have smaller data
sets.

Any help would be appreciated.

Will

[WillW]

[Kirk]

Hi Will,

What are the hardware specs of the Urchin machine?

Kirk

[WillW]

Intel quad-core, 3GB RAM, 250GB SATA, Win XP Pro. It's a new Dell
Optiplex that we purchased just to run Urchin.

[Kirk]

Well that rules out hardware limitations. How often is this profile
processing and what is the log source set to? (are you using date
substitution so all logs aren't grabbed?). It may be an issue of
large sessions remaining open, but that doesn't sound right. How
large are the log files? Is there anything else different between
this and the other profiles that are working fine besides filters?

[Actual Metrics]

What visitor tracking method are you using? If IP+UA, try setting
"keep raw tracking data" to off in the storage/db tab of the profile.
Also...don't process more than a month of logs at a time.

--Mike
www.actualmetrics.com

On Nov 12, 6:20 am, WillW <w...@weidmanconsulting.com> wrote:

[WillW]

We are using IP+User Agent tracking. I have the log filters set to
process just the first month's data. Each daily log file (from each
of the A and B load-balanced servers) is about 200MB. I tried turning
the "keep raw tracking data" setting off, deleting all the profile
data, and running it again. But it still stalls on the first log file
of from the second load-balanced server. The content filters are the
only difference between this profile and the ones that run OK.

Will

[DennisO.]

Hello, Will,

What is the number of unique visitors that you have in the logs (bulk
number is Ok). What is your DB row limit value set in Urchin? Do you
process logs in memory or using cache?

Thank you,
DennisO.

[WillW]

The site gets about 20k visits per day. On the database tab, the
settings are:

DB Memory - limit w/ cache
DB table limit - 500k
Raw Tracking - off
Log tracking - on
DB backups - all on, keeping 2 backups
Archive DB - on, after 12 months

[Bozidar]

Try changing the amount of memory Urchin uses in the Settings link on
the left site under the DB Setting tab, increase to 1GB and see if it
makes a difference.

Bozidar

[WillW]

The way I was finally able to "solve" this problem was to process the
log files in one-month increments. For each month I first configured
the profile to look only at the "A" load-balanced server. When it
finished processing I edited the profile to add the "B" server logs
in, and ran it again. I was able to successfully process several
months of data this way. Day forward, the profile is configured to
process both the "A" and "B" servers in one run, which it process OK
because it can handle a daily amount of data without hanging.

Will