no graph using custom log format

[Baraut Romain]

Hi guys I come to ask help,

The thing is I created special log format:

PrimaryPositions: "16,17,11,8,10,4"
SecondaryPositions: -
PrimaryKey: -
SecondaryKey: -
PrimaryContent: HIT
SecondaryContent: -
CommentKey: #
FieldSeparator1: \s
FieldSeparator2: \t
QuotesEscapeSep: YES
BracketsEscapeSep: YES
MergeSuccessiveSep: NO
CleanWhiteSpace: NO
StatusRequired: YES
CustomDateFormat: "%y/%m/%d"
"CustomTimeFormat: "%H:%M:%S"
TimeZoneOffset: 0


I use these fields with only a few lines just to test:

#Fields: custom_date custom_time sc_bytes cs_uristem sc_status c_ip

2009-11-27 00:00:00 15548 http://image01. lala.com/bonprixbilder/
bundles/32er/195x271/var1/vv/x2/vvx2065x02.jpg 200 77.0.98.69
2009-11-27 00:00:00 118 http://image01. lala.com/bonprixbilder/bundles/
variante_mittel/var2/bj/b2/bjb200x07.jpg 304 93.48.37.99
2009-11-27 00:00:00 2418 http://image01. lala.com/bonprixbilder/
artikelklein/7er/28x39/var1/92/2/922410.jpg 200 79.226.135.28
2009-11-27 00:00:00 118 http://image01. lala.com/bonprixbilder/bundles/
variante_mittel/var1/bj/b2/bjb200x07.jpg 304 93.48.37.99
2009-11-27 00:00:00 18112 http://image01. lala.com/bonprixbilder/
bundles/32er/195x271/var4/vv/68/vv680x14.jpg 200 82.113.106.113
2009-11-27 00:00:00 118 http://image01. lala.com/bonprixbilder/bundles/
variante_mittel/var3/bj/b2/bjb200x07.jpg 304 93.48.37.99
2009-11-27 00:00:00 551 http://image01. lala.com/bonprixbilder/
bookmarks/logos/blinzo-16x16.gif 200 77.1.153.111
2009-11-27 00:00:00 19129 http://image01. lala.com/bonprixbilder/
bundles/32er/195x271/var1/vv/x2/vvx2200x04.jpg 200 77.0.98.69
2009-11-27 00:00:00 400 http://image01. lala.com/bonprixbilder/
bookmarks/logos/blunzo-16x16.gif 200 77.1.153.111
2009-11-27 00:00:00 17517 http://image01. lala.com/bonprixbilder/
bundles/artikel_klein/var1/yy/33/yy333x05.jpg 200 77.25.150.181


But, it doesn't work, can you help me ?


Thx by advance;

Regards.

[Kirk - Actual Metrics]

First thing I would check is that your visitor tracking method is set
to IP+User Agent. Then, you will need some sort of user agent field
recorded in the log.

Kirk
www.urchintools.com
www.urchinexperts.com
www.actualmetrics.com

On Dec 14, 5:15 am, Baraut Romain <baric...@gmail.com> wrote:
> Hi guys I come to ask help,
>
> The thing is I created special log format:
>
> PrimaryPositions:    "16,17,11,8,10,4"
> SecondaryPositions:  -
> PrimaryKey:          -
> SecondaryKey:        -
> PrimaryContent:      HIT
> SecondaryContent:    -
> CommentKey:          #
> FieldSeparator1:     \s
> FieldSeparator2:     \t
> QuotesEscapeSep:     YES
> BracketsEscapeSep:   YES
> MergeSuccessiveSep:  NO
> CleanWhiteSpace:     NO
> StatusRequired:      YES
> CustomDateFormat:    "%y/%m/%d"
>  "CustomTimeFormat:  "%H:%M:%S"
> TimeZoneOffset:      0
>
> I use these fields with only a few lines just to test:
>
> #Fields: custom_date custom_time sc_bytes cs_uristem sc_status c_ip
>

> 2009-11-27 00:00:00 15548http://image01. lala.com/bonprixbilder/
> bundles/32er/195x271/var1/vv/x2/vvx2065x02.jpg 200 77.0.98.69
> 2009-11-27 00:00:00 118http://image01. lala.com/bonprixbilder/bundles/
> variante_mittel/var2/bj/b2/bjb200x07.jpg 304 93.48.37.99
> 2009-11-27 00:00:00 2418http://image01. lala.com/bonprixbilder/
> artikelklein/7er/28x39/var1/92/2/922410.jpg 200 79.226.135.28
> 2009-11-27 00:00:00 118http://image01. lala.com/bonprixbilder/bundles/
> variante_mittel/var1/bj/b2/bjb200x07.jpg 304 93.48.37.99
> 2009-11-27 00:00:00 18112http://image01. lala.com/bonprixbilder/
> bundles/32er/195x271/var4/vv/68/vv680x14.jpg 200 82.113.106.113
> 2009-11-27 00:00:00 118http://image01. lala.com/bonprixbilder/bundles/
> variante_mittel/var3/bj/b2/bjb200x07.jpg 304 93.48.37.99
> 2009-11-27 00:00:00 551http://image01. lala.com/bonprixbilder/
> bookmarks/logos/blinzo-16x16.gif 200 77.1.153.111
> 2009-11-27 00:00:00 19129http://image01. lala.com/bonprixbilder/
> bundles/32er/195x271/var1/vv/x2/vvx2200x04.jpg 200 77.0.98.69
> 2009-11-27 00:00:00 400http://image01. lala.com/bonprixbilder/
> bookmarks/logos/blunzo-16x16.gif 200 77.1.153.111
> 2009-11-27 00:00:00 17517http://image01. lala.com/bonprixbilder/

[Baraut Romain]

Plz any help ? I stuck ...

Thx by advance,

regards,

Romain

On 14 déc, 19:34, Kirk - Actual Metrics <kirk.mora...@gmail.com>
wrote:

[Jeremy Aube]

Hi Baraut,

It looks like your custom date format is off. I would use something
like this instead:

PrimaryPositions: "16,17,11,8,10,4"
SecondaryPositions: -
PrimaryKey: -
SecondaryKey: -
PrimaryContent: HIT
SecondaryContent: -
CommentKey: #
FieldSeparator1: \s
FieldSeparator2: \t
QuotesEscapeSep: YES
BracketsEscapeSep: YES
MergeSuccessiveSep: NO
CleanWhiteSpace: NO
StatusRequired: YES

CustomDateFormat: "%Y-%m-%d"

CustomTimeFormat: "%H:%M:%S"
TimeZoneOffset: 0

A capital Y is needed since the year includes the century, and you
should use dashes to separate the variables since the dates in your
log file use dashes.

Jeremy Aube
http://www.roirevolution.com/urchin

On Dec 14, 7:15 am, Baraut Romain <baric...@gmail.com> wrote:
> Hi guys I come to ask help,
>
> The thing is I created special log format:
>
> PrimaryPositions:    "16,17,11,8,10,4"
> SecondaryPositions:  -
> PrimaryKey:          -
> SecondaryKey:        -
> PrimaryContent:      HIT
> SecondaryContent:    -
> CommentKey:          #
> FieldSeparator1:     \s
> FieldSeparator2:     \t
> QuotesEscapeSep:     YES
> BracketsEscapeSep:   YES
> MergeSuccessiveSep:  NO
> CleanWhiteSpace:     NO
> StatusRequired:      YES
> CustomDateFormat:    "%y/%m/%d"
>  "CustomTimeFormat:  "%H:%M:%S"
> TimeZoneOffset:      0
>
> I use these fields with only a few lines just to test:
>
> #Fields: custom_date custom_time sc_bytes cs_uristem sc_status c_ip
>

> 2009-11-27 00:00:00 15548http://image01. lala.com/bonprixbilder/
> bundles/32er/195x271/var1/vv/x2/vvx2065x02.jpg 200 77.0.98.69
> 2009-11-27 00:00:00 118http://image01. lala.com/bonprixbilder/bundles/
> variante_mittel/var2/bj/b2/bjb200x07.jpg 304 93.48.37.99
> 2009-11-27 00:00:00 2418http://image01. lala.com/bonprixbilder/
> artikelklein/7er/28x39/var1/92/2/922410.jpg 200 79.226.135.28
> 2009-11-27 00:00:00 118http://image01. lala.com/bonprixbilder/bundles/
> variante_mittel/var1/bj/b2/bjb200x07.jpg 304 93.48.37.99
> 2009-11-27 00:00:00 18112http://image01. lala.com/bonprixbilder/
> bundles/32er/195x271/var4/vv/68/vv680x14.jpg 200 82.113.106.113
> 2009-11-27 00:00:00 118http://image01. lala.com/bonprixbilder/bundles/
> variante_mittel/var3/bj/b2/bjb200x07.jpg 304 93.48.37.99
> 2009-11-27 00:00:00 551http://image01. lala.com/bonprixbilder/
> bookmarks/logos/blinzo-16x16.gif 200 77.1.153.111
> 2009-11-27 00:00:00 19129http://image01. lala.com/bonprixbilder/
> bundles/32er/195x271/var1/vv/x2/vvx2200x04.jpg 200 77.0.98.69
> 2009-11-27 00:00:00 400http://image01. lala.com/bonprixbilder/
> bookmarks/logos/blunzo-16x16.gif 200 77.1.153.111
> 2009-11-27 00:00:00 17517http://image01. lala.com/bonprixbilder/

[Baraut Romain]

OMG !
2 days looking for the problem :'(

Thx Jeremy !

On 18 déc, 13:54, Jeremy Aube <ja...@roirevolution.com> wrote:
> Hi Baraut,
>
> It looks like your custom date format is off. I would use something
> like this instead:
>
> PrimaryPositions:    "16,17,11,8,10,4"
> SecondaryPositions:  -
> PrimaryKey:          -
> SecondaryKey:        -
> PrimaryContent:      HIT
> SecondaryContent:    -
> CommentKey:          #
> FieldSeparator1:     \s
> FieldSeparator2:     \t
> QuotesEscapeSep:     YES
> BracketsEscapeSep:   YES
> MergeSuccessiveSep:  NO
> CleanWhiteSpace:     NO
> StatusRequired:      YES
> CustomDateFormat:    "%Y-%m-%d"
> CustomTimeFormat:  "%H:%M:%S"
> TimeZoneOffset:      0
>
> A capital Y is needed since the year includes the century, and you
> should use dashes to separate the variables since the dates in your
> log file use dashes.
>

> Jeremy Aubehttp://www.roirevolution.com/urchin

[Baraut Romain]

Hello,

An additional question ;)

When I test with a 30 lines log file test, I lunch:/usr/local/urchin/
bin/urchin -D -p test > test.txt

I obtain a lot of information in the output showing me that urchin
took care og this log:

[11:58:09] Logfile: /home/rbaraut/test2.log
data lines: 0 (0%) Hit: image01.bonprix.de:
80;2009-12-02;00:00:00;0;24257;http://image01.bonprix.de/bonprixbilder/
bundles/32er/195x271/var1/vv/x2/vvx2146x04.jpg;200;-;
88.67.151.162;TCP_HIT
viewtime 1259712000
yearmonth 200912
day 0
hour 16
time_offset -28800
c_ip "88.67.151.162"
cs_uriquery "http://image01blabla.de/bonprixbilder/
bundles/32er/195x271/var1/vv/x2/vvx2146x04.jpg"
sc_status "200"
sc_bytes "24257"
custom_date "2009-12-02"
custom_time "00:00:00"
request_query "http://image01blabla/bonprixbilder/
bundles/32er/195x271/var1/vv/x2/vvx2146x04.jpg"
domain_primary "net"
domain_complete "arcor-ip.net"
utm_campaign "(direct)"
utm_medium "(none)"
utm_source "(direct)"
log_source_name "test"
client_ipaddress "88.67.151.162"
client_hostname "arcor-ip.net"
geo_country "Germany"
geo_region "Baden-Württemberg"
geo_city "Goppingen"
geo_latitude "487031"
geo_longitude "96539"
geo_organization "arcor ag"
geo_connection_speed "Dialup"
bytes 24257
nonpages 1
DADB (26,100000): ST(1477556 1477556 1477556 0 0), Ifield 4, Day 0,
Value 24257
STDB (1477564): test
DADB (36,2): ST(1477564 0 0 0 0), Ifield 4, Day 0, Value 24257

Still, when I test with the real log file (28.3 giga) the output of my
test.txt is null:

[11:58:09] Logfile: /mnt/filer-genet-1/b/o/boreus.de/ftp/logs/
logcdn4.20091216.log
data lines: 0 (0%) (skipping 25165823160 bytes)
data lines: 0 (0%) ^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H0
(100%)
data hits: 0
data proc: 0.00 B in less than 1 second
data range: 0 - 0

[11:58:09] Post processing data for 200912
sessions: 0
^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H0 (100%)

[11:58:09] Backing up database files for 200912: /usr/local/urchin/
data/reports/%28NONE%29/test/200912-backupv6-20091221115809.zip

[11:58:12] Removing outdated backup for 200912: /usr/local/urchin/
data/reports/%28NONE%29/test/200912-backupv6-20091218145709.zip

------------------------------------------------------
Urchin 6.602 (linux2.6_kernel) finishing: 20091221 11:58:12
------------------------------------------------------

Can you tell me what's happen ? why urchin take my test.log and not my
real one.

Regards,

[Jeremy Aube]

It may be that you've already processed your logfile previously, so
Urchin is skipping the data to prevent duplicate data from showing up
in your profile. When you process the logfile, are you making sure to
delete all data in the profile first or starting with a brand new
profile?

Jeremy Aube
http://www.roirevolution.com/urchin

[Baraut Romain]

No, don't delete my already processed data ;)
you have reason :( Thx for this answer.

What would you advice regarding custom log fields ?
I mean our CD Networks give us log (the same log previously shown),
what kind of fields I would need to put in place to have an
efficicient stats reporting ?

Why this question ?
Because the actual fields on which we are working on are the
followings and I had to say that reporting is pretty poor since I have
information only in "domains and users" section:

image01.blalbvlald.de:80= unknown fields... setted on 0 in urchin
custom log
2009-12-16;= date
21:51:24 = time
0; = unknown field (but always at 0, so no need...setted on 0 in
urchin custom log)
7673= sc_bytes
http://image01.bonprix.de/bonprixbilder/bundles/30er/klein/var3/vv/x8/vvx853x01.jpg=
cs_uriquery
200= sc_status
- = unknown (but never known so no need, setted on 0 in urchin custom
log)
85.179.123.27 = cs_ip
TCP_HIT = unknown fields....(setted on 0 in urchini custom log)

Thx for your very good advices.
Regards,

Romain

On 21 déc, 13:16, Jeremy Aube <ja...@roirevolution.com> wrote:
> It may be that you've already processed your logfile previously, so
> Urchin is skipping the data to prevent duplicate data from showing up
> in your profile. When you process the logfile, are you making sure to
> delete all data in the profile first or starting with a brand new
> profile?
>

> Jeremy Aubehttp://www.roirevolution.com/urchin

[Jeremy Aube]

Hi Romain,

Urchin requires the following fields:

date
time
c-ip
cs-uri-stem
sc-status
sc-bytes

If you want better reporting, however, you should try to include all
of the following fields:

date
time
c-ip
cs-username
cs-method
cs-uri-stem
cs-uri-query
sc-status
sc-bytes
cs[User-Agent]
cs[Referer]

For best reporting, include cs[Cookie] as well and configure the UTM
method:

https://secure.urchin.com/helpwiki/en/UTM_Quick-Install_%28IIS%29

Jeremy Aube
http://www.roirevolution.com/urchin

On Dec 21, 8:32 am, Baraut Romain <baric...@gmail.com> wrote:
> No, don't delete my already processed data ;)
> you have reason :( Thx for this answer.
>
> What would you advice regarding custom log fields ?
> I mean our CD Networks give us log (the same log previously shown),
> what kind of fields I would need to put in place to have an
> efficicient stats reporting ?
>
> Why this question ?
> Because the actual fields on which we are working on are the
> followings and I had to say that reporting is pretty poor since I have
> information only in "domains and users" section:
>
> image01.blalbvlald.de:80= unknown fields... setted on 0 in urchin
> custom log
> 2009-12-16;= date
> 21:51:24 = time
> 0; =  unknown field (but always at 0, so no need...setted on 0 in
> urchin custom log)

> 7673=  sc_byteshttp://image01.bonprix.de/bonprixbilder/bundles/30er/klein/var3/vv/x8...

[Baraut Romain]

Thx a lot for your advices,

Very appreciated !

Regards,