ReportMagic and ProFTPD on Centos5

[cultavix]

Hi all,

I have got ReportMagic running perfectly via Analog and the ProFTPD
logs. The problem is this;

I want to be able to view what users are logging in, what they are
downloading and for how long. I believe this all relates to the log
file format. Does anyone have any experience with ReportMagic and
ProFTPD ?

[Jeremy Wadsack]

You can get the users who logged in, out of your basic report by including the User report (USERS ON in analog.cfg). 

You can tell what files everyone is downloading using the Request report (REQUEST ON in analog.cfg), but that should be on by default.

Unfortunately, Analog does not do multivariate (or cross-reference) reports directly, so you can't get this out-of-the box without some extra work. 

Once you have that you can find out what each user has downloaded by running a new report for that user (USERINCLUDE xxxx in analog.cfg). See the include command for details. It probably makes sense to only run this for your top N users.

You can't precisely tell how long someone was on the site under HTTP (which is what Analog was designed for) so it doesn't do time-on-site reports. However, you could use the Processing Time report (PROCTIME ON in analog.cfg) to tell how long the files take to download, in a historgram. Of course, you could also use the USERINCLUDE method to get this for individual users if you wish.

--
Jeremy Wadsack

[cultavix]

Thank you for your quick reply, I am very thankful. I will try these
things out and see how it goes.

As of now, I can get a near perfect report of xferlog from ProFTPD but
the problem is, that log only collects information of the transfers,
not the users....

I think that the only log to contain the users and everything else I
want is the "Extendedlog" that Proftpd can generate.

Which log should I be analyzing?

Cheers!

James

On Sep 22, 5:21 pm, Jeremy Wadsack <jeremy.wads...@gmail.com> wrote:
> You can get the users who logged in, out of your basic report by including
> the User report (USERS ON in analog.cfg).
>
> You can tell what files everyone is downloading using the Request report
> (REQUEST ON in analog.cfg), but that should be on by default.
>
> Unfortunately, Analog does not do multivariate (or cross-reference) reports

> directly <http://www.analog.cx/docs/faq.html#faq128>, so you can't get this

> out-of-the box without some extra work.
>
> Once you have that you can find out what each user has downloaded by running
> a new report for that user (USERINCLUDE xxxx in analog.cfg). See the include

> command <http://www.analog.cx/docs/include.html> for details. It probably

> makes sense to only run this for your top N users.
>
> You can't precisely tell how long someone was on the site under

> HTTP<http://www.analog.cx/docs/webworks.html>(which is what Analog was

> designed for) so it doesn't do time-on-site
> reports. However, you could use the Processing Time report (PROCTIME ON in
> analog.cfg) to tell how long the files take to download, in a historgram. Of
> course, you could also use the USERINCLUDE method to get this for individual
> users if you wish.
>
> --
> Jeremy Wadsack
>

[Jeremy Wadsack]

James - 

I really don't know much about ProFTPD logs. A quick look at the docs, though, suggests that the ExtendedLog is what you want to use. It looks like you could configure it to have all the fields you want and then can analyze that.

I don't think you'll need to create both logs, just one or the other, and Extended sounds right. If you do have both logs and they don't all have the information you want, Analog can sometimes read both sets of logs and still build reports. You just add a LOGFORMAT command for each file before the LOGFILE command that loads it.

--
Jeremy Wadsack

[cultavix]

Hi Jeremy,

Thank you once again for your quick reply. There is not much help out
there and if there is, I cant find it!! :P

Anyhow, me and my work-mate were working on this today and we cannot
seem to get the correct output from analog that we need for
reportmagic to work. We simply want the name of the usernames(local
userid), his IP address/hostname, the date/time, the files that were
dl/upl and a few other things. We can only get it working by simply
using the defaults, which do not include the local userid. So if we
could somehow add on to the default configuration the userid at least,
that would be good enough for now.

I can supply you with my conf file if you would like, here is a sneak
peek;
This is from our analog.cfg;
#LOGFORMAT (%j %M %d %h:%n:%j %Y %t %S %b %r %j %j %j %j %u %j %j %j)
LOGFILE /var/log/proftpd/extendedlog.log

This is from our proftpd.conf;
LogFormat test-string "%h %l %u %t \"%r\" %s %b"
ExtendedLog /var/log/proftpd/extendedlog.log ALL test-string

Any help is apreciated...

Thank you very much!!!!

- James (and jacob)

On Sep 23, 5:36 pm, Jeremy Wadsack <jeremy.wads...@gmail.com> wrote:
> James -
>
> I really don't know much about ProFTPD logs. A quick look at the

> docs<http://www.proftpd.org/docs/howto/Logging.html>,

> though, suggests that the ExtendedLog is what you want to use. It looks like
> you could configure it to have all the fields you want and then can analyze
> that.
>
> I don't think you'll need to create both logs, just one or the other, and
> Extended sounds right. If you do have both logs and they don't all have the
> information you want, Analog can sometimes read both sets of logs and still

> build reports. You just add a LOGFORMAT command for each file *before* the

> LOGFILE command that loads it.
>
> --
> Jeremy Wadsack
>

[Jeremy Wadsack]

Looking at the ProFTPd docs and your logformat below, I don't think you need %l in there — nobody supports ident anymore. If you want to know what they downloaded you may find %F (or %f) more useful than %r. Adding %T might also be useful if you want to know how long downloads are taking.

The first step is getting the information in your log files that you want to be there.

I'm guessing that this doesn't produce W3C Extended Log files (that are self-documenting), so you'll have to provide a LOGFORMAT command to Analog. I don't know how ProFTPd writes out time or request details, but if you're having trouble getting Analog to parse the file, feel free to post a line and I can give it a shot.

Also, note that the analog-help list may have more people with experience with ProFTPd, if you want to try the request over there until you get something that works in Analog.

Once Analog is working, it sounds like you have Report Magic mostly figured out.

--
Jeremy Wadsack