BSDstats v3.0 - The Security Rewrite
Marc G. Fournier
Over the past few days, I've been working with Paul Schmehl and Matthew
Seaman to come up with a more "security sensitive" version of BSDstats ...
one that reduces the amount of "sensitive information" stored in the
database down to ... zero. No IPs, no hostnames ...
This new version also reduces the number of 'network fetches' down to 4
for the first run, and 3 for subsequent runs, so it runs a bit faster, and
talks across the network less.
And, finally, this one has its own domain for check in server ...
None of the pre-v3.x clients can talk to the v3.x server, since the DB
format has totally changed, so everyone needs to grab the latest version
and run it so that we can re-sync the database properly ...
>From now forward, the stats will be viewable from:
----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scr...@hub.org MSN . scr...@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"
John Nielsen
> Over the past few days, I've been working with Paul Schmehl and Matthew
> Seaman to come up with a more "security sensitive" version of BSDstats ...
> one that reduces the amount of "sensitive information" stored in the
> database down to ... zero. No IPs, no hostnames ...
>
> This new version also reduces the number of 'network fetches' down to 4
> for the first run, and 3 for subsequent runs, so it runs a bit faster, and
> talks across the network less.
>
> And, finally, this one has its own domain for check in server ...
>
> None of the pre-v3.x clients can talk to the v3.x server, since the DB
> format has totally changed, so everyone needs to grab the latest version
> and run it so that we can re-sync the database properly ...
>
> >From now forward, the stats will be viewable from:
>
> http://www.bsdstats.org
This is great!
Is the 15-minute first-time waiting period enforced on the server side?
Obviously there's nothing to stop an administrator from editing the script
locally..
Thanks again for all your efforts.
JN
Marc G. Fournier
> This is great!
>
> Is the 15-minute first-time waiting period enforced on the server side?
> Obviously there's nothing to stop an administrator from editing the script
> locally..
It is enforced on the server side ... in fact, one person just reported to
me that they killed the script and re-ran it, and the stats went through
... they didn't, the server side will reject the submission until the
first minute time has elapsed ... I've got some ideas on how to better
clean that up on the client side ...
Scott Sipe
On Aug 14, 2006, at 4:28 PM, Marc G. Fournier wrote:
> On Mon, 14 Aug 2006, John Nielsen wrote:
>
>> This is great!
>>
>> Is the 15-minute first-time waiting period enforced on the server
>> side?
>> Obviously there's nothing to stop an administrator from editing
>> the script
>> locally..
>
> It is enforced on the server side ... in fact, one person just
> reported to me that they killed the script and re-ran it, and the
> stats went through ... they didn't, the server side will reject the
> submission until the first minute time has elapsed ... I've got
> some ideas on how to better clean that up on the client side ...
>
This just happened to me as well--I installed the port, added lines
to periodic.conf and manually ran:
/usr/local/etc/periodic/monthly/300.statistics
output as follows:
# /usr/local/etc/periodic/monthly/300.statistics
chown: /var/db/bsdstats: No such file or directory
To protect against abuse, the initial challenge/response phase
contains a 15 minute pause. Please be patient while this time
limit elapses
^C
# /usr/local/etc/periodic/monthly/300.statistics
Posting monthly OS statistics to bsdstats.org
Posting monthly device statistics to bsdstats.org
Posting monthly CPU statistics to bsdstats.org
I cancelled it to see why it showed the chown error.
Scott
Marc G. Fournier
chown error fixed in CVS now ... and if you run the script *now*, it will
be past the 15 min mark and will submit properly ...
Jonathan Horne
> Over the past few days, I've been working with Paul Schmehl and Matthew
> Seaman to come up with a more "security sensitive" version of BSDstats ...
> one that reduces the amount of "sensitive information" stored in the
> database down to ... zero. No IPs, no hostnames ...
>
> This new version also reduces the number of 'network fetches' down to 4
> for the first run, and 3 for subsequent runs, so it runs a bit faster, and
> talks across the network less.
>
> And, finally, this one has its own domain for check in server ...
>
> None of the pre-v3.x clients can talk to the v3.x server, since the DB
> format has totally changed, so everyone needs to grab the latest version
> and run it so that we can re-sync the database properly ...
>
> >From now forward, the stats will be viewable from:
>
> http://www.bsdstats.org
>
> ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> Email . scr...@hub.org MSN . scr...@hub.org
> Yahoo . yscrappy Skype: hub.org ICQ . 7615664
mark,
i noticed the percentages columns, they definatly make the information all
that more fascinating. would it be possible to segregate the far right
column by release, and show what percentages of the 6.1s are stable, p3, p2,
release? etc etc? that column as it sits is still good information (so im
not suggesting dumping it), but i would like to see another section to break
out the versions, by percentages.
my 2 cents, :)
jonathan
Igor Robul
> None of the pre-v3.x clients can talk to the v3.x server, since the DB
> format has totally changed, so everyone needs to grab the latest version
> and run it so that we can re-sync the database properly ...
v2.0 worked fine
John Nielsen
> On Mon, Aug 14, 2006 at 10:19:05AM -0300, Marc G. Fournier wrote:
> > format has totally changed, so everyone needs to grab the latest version
> > and run it so that we can re-sync the database properly ...
>
> v2.0 worked fine
It doesn't "build" at all (although it does create a "work" directory to keep
track of its progress). I always set WRKDIRPREFIX and didn't have any trouble
upgrading. Are you sure there's not something else going on?
JN
Igor Robul
> On Tuesday 15 August 2006 08:12, Igor Robul wrote:
> > On Mon, Aug 14, 2006 at 10:19:05AM -0300, Marc G. Fournier wrote:
> > > format has totally changed, so everyone needs to grab the latest version
> > > and run it so that we can re-sync the database properly ...
> >
> > v2.0 worked fine
>
> It doesn't "build" at all (although it does create a "work" directory to keep
> track of its progress). I always set WRKDIRPREFIX and didn't have any trouble
> upgrading. Are you sure there's not something else going on?
strange ... when I had done "make install" today it didnt complainned ...
so maybe I really did something strange :-)
Marc G. Fournier
> i noticed the percentages columns, they definatly make the information
> all that more fascinating. would it be possible to segregate the far
> right column by release, and show what percentages of the 6.1s are
> stable, p3, p2, release? etc etc? that column as it sits is still good
> information (so im not suggesting dumping it), but i would like to see
> another section to break out the versions, by percentages.
Took me a couple of reads to understand what you were asking here ... yes,
should be possible, but will have to do it later (ie. on TODO list) ...
Oliver Fromme
> Over the past few days, I've been working with Paul Schmehl and Matthew
> Seaman to come up with a more "security sensitive" version of BSDstats ...
> one that reduces the amount of "sensitive information" stored in the
> database down to ... zero. No IPs, no hostnames ...
That's very cool. I've installed it on some of my machines.
Unfortunately, I haven't been able to use it on all of them,
for reasons outlined below.
I've got a few suggestions and ideas ...
(1) When run for the first time, you get an error message:
: not found
That's because a few bogus spaces after the backslash in
the line containing the chmod command. Those trailing
spaces should be removed. I suppose I don't need to send
a PR for that. :-)
(2) Some people aborted the inital "sleep 900" (because
of the above-mentioned error message, or other reasons),
then restarted the script. In this case there is no sleep,
and the submission _seems_ to be successful (no negative
feedback), but it isn't.
One way to improve the situation would be to check the
mtime on the /var/db/bsdstats file. If it's younger than
900 seconds, a sleep is required. For example, something
like this piece of shell code (untested):
FILETIME=$( stat -f %m $id_token_file )
NOW=$( date +%s )
if [ $(( $NOW - 900 )) -le $FILETIME ]; then
SLEEPTIME=$(( 900 - ($NOW - $FILETIME) ))
echo "Token key is younger than 15 minutes!"
echo "Sleeping $SLEEPTIME seconds, please wait."
sleep $SLEEPTIME
fi
(3) Some sites require the use of a proxy for HTTP access.
Such sites usually have an entry in /etc/make.conf, so the
ports can fetch their distfiles:
FETCH_ENV= FTP_PROXY=http://proxy.my.site:3128 \
HTTP_PROXY=http://proxy.my.site:3128
The bsdstats script could easily pick up that entry and set
the environment variables appropriatly. This line at the
beginning of the script should be sufficient:
export $( make -V FETCH_ENV 2>/dev/null )
(4) Some sites have a proxy that requires authentication.
It is possible to include the password in the FETCH_ENV
entry in /etc/make.conf, but it's usually not a good idea
to do that, because you shouldn't write passwords to files
that are world-readable.
That problem could be solved in different ways. One way
would be a periodic.conf setting that instructs the script
not to try to submit the data, but instead just print a
reminder to the admin that he should run the monthly script
manually (or print that reminder automatically when the
submission fails because the proxy denies access).
When the admin runs the script manually (which could be
detected by "test -t 0", i.e. stdin is a terminal), it
could ask for the HTTP proxy password and then set the
HTTP_PROXY_AUTH variable appropriately (see fetch(3)).
(5) Some machines might not be able to access the web at
all. For example, I'm right now working on a farm of 35
machines which don't have internet access, not even via
a proxy. I can connect to them via ssh/scp (port 22) from
a management machine, and that management machine only has
web access via a proxy.
It would be nice to be able to request token keys on behalf
of those 35 servers from the management machine, transfer
them to the servers, run the data gathering script on the
servers (putting it into a file instead of submitting it
directyl), copy the results to the management machine and
finally submit them from there. That's pretty complicated,
but I'm afraid I haven't gotten a better idea so far. :-(
(6) All of the statistics on the web page are sorted by
percentages. It would be nice to be able to click on a
column header and have the table sorted by that value.
That would be especially useful for the release statistics
and the country statistics.
(If the PHP sources and a database export were publicly
available, I would have taken a shot at implementing it.)
(7) In order to make the bsdstats project really useful,
it is very important to have as many FreeBSD people as
possible install it. Currently, only very few people will
notice the port and bother to install it. Therefore I
suggest to put bsdstats into the base system (it's only a
small script after all, no bloat), and add a small switch
to sysinstall which asks users whether they want to enable
it, creating appropriate periodic.conf entry for them, and
maybe even automatically running it when booting the newly
installed system for the first time.
Maybe it should be proposed and discussed in the arch@
mailing list.
Best regards
Oliver
--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.
"File names are infinite in length, where infinity is set to 255 characters."
-- Peter Collinson, "The Unix File System"
Marc G. Fournier
Thanks for your comments and suggestions ... I am currently in the middle
of a campground working on a laptop (missing my desktop dearly, since this
laptop is my wife's Windows box) ... when I get back online properly
beginning of Sept, I will work on the suggestions though ... thx ...
----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scr...@hub.org MSN . scr...@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
Marc G. Fournier
> (1) When run for the first time, you get an error message:
> : not found
> That's because a few bogus spaces after the backslash in
> the line containing the chmod command. Those trailing
> spaces should be removed. I suppose I don't need to send
> a PR for that. :-)
'k, that one is/was fixed ...
> (2) Some people aborted the inital "sleep 900" (because
> of the above-mentioned error message, or other reasons),
> then restarted the script. In this case there is no sleep,
> and the submission _seems_ to be successful (no negative
> feedback), but it isn't.
>
> One way to improve the situation would be to check the
> mtime on the /var/db/bsdstats file. If it's younger than
> 900 seconds, a sleep is required. For example, something
> like this piece of shell code (untested):
>
> FILETIME=$( stat -f %m $id_token_file )
> NOW=$( date +%s )
> if [ $(( $NOW - 900 )) -le $FILETIME ]; then
> SLEEPTIME=$(( 900 - ($NOW - $FILETIME) ))
> echo "Token key is younger than 15 minutes!"
> echo "Sleeping $SLEEPTIME seconds, please wait."
> sleep $SLEEPTIME
> fi
Code tested, and committed ... thanks ...
> (3) Some sites require the use of a proxy for HTTP access.
> Such sites usually have an entry in /etc/make.conf, so the
> ports can fetch their distfiles:
>
> FETCH_ENV= FTP_PROXY=http://proxy.my.site:3128 \
> HTTP_PROXY=http://proxy.my.site:3128
>
> The bsdstats script could easily pick up that entry and set
> the environment variables appropriatly. This line at the
> beginning of the script should be sufficient:
>
> export $( make -V FETCH_ENV 2>/dev/null )
'k, that one could be a problem, since adding that line produces:
# /usr/local/etc/periodic/monthly/300.statistics
MACHTYPE
SSH_CLIENT
USER
MAIL
SHLVL
VENDOR
HOME
SSH_TTY
PAGER
GROUP
LOGNAME
BLOCKSIZE
TERM
PATH
REMOTEHOST
HOST
SHELL
OSTYPE
PWD
SSH_CONNECTION
FTP_PASSIVE_MODE
HOSTTYPE
EDITOR
Token key is younger than 15 minutes!
Sleeping 361 seconds, please wait.
Is there a better way of doing it that it would be silent?
> (4) Some sites have a proxy that requires authentication.
> It is possible to include the password in the FETCH_ENV
> entry in /etc/make.conf, but it's usually not a good idea
> to do that, because you shouldn't write passwords to files
> that are world-readable.
>
> That problem could be solved in different ways. One way
> would be a periodic.conf setting that instructs the script
> not to try to submit the data, but instead just print a
> reminder to the admin that he should run the monthly script
> manually (or print that reminder automatically when the
> submission fails because the proxy denies access).
> When the admin runs the script manually (which could be
> detected by "test -t 0", i.e. stdin is a terminal), it
> could ask for the HTTP proxy password and then set the
> HTTP_PROXY_AUTH variable appropriately (see fetch(3)).
'k, this one has been brought up, and was something that I am hoping to
address once I get back online properly this week ...
> (5) Some machines might not be able to access the web at
> all. For example, I'm right now working on a farm of 35
> machines which don't have internet access, not even via
> a proxy. I can connect to them via ssh/scp (port 22) from
> a management machine, and that management machine only has
> web access via a proxy.
>
> It would be nice to be able to request token keys on behalf
> of those 35 servers from the management machine, transfer
> them to the servers, run the data gathering script on the
> servers (putting it into a file instead of submitting it
> directyl), copy the results to the management machine and
> finally submit them from there. That's pretty complicated,
> but I'm afraid I haven't gotten a better idea so far. :-(
Actually, this is one that we have discussed, and believe we have a
solution for already, I just have to sit down and code this one ...
And I think it might actually act as a way of dealing with (4) as well ...
We're goin ot have a 'use_email' setting ... what will happen is as
follows:
Initial Install / Run:
Email is sent to root containing IDTOKEN= as generated by host, root
forwards that to r...@bsdstats.org, r...@bsdstats.org sends back KEY= value
to be put in /var/db/bsdstats (manual cut-n-paste) ... script is re-run a
second time, submits report values to root, root forwards that to
r...@bsdstats.org ...
Subsequent Runs:
Email is sent to root containing report values, root forwards that to
r...@bsdstats.org
Now, in theory, root could have a filter on it that 'if subject = bsdstats
report, auto-forward to r...@bsdstats.org', but that would be totally up to
the admin whether they wanted to do each report manually or not ...
> (6) All of the statistics on the web page are sorted by percentages.
> It would be nice to be able to click on a column header and have the
> table sorted by that value. That would be especially useful for the
> release statistics and the country statistics.
That is all major work in progress ... my first goal was to get data into
the database, and reporting in place (and improve the script based on
feedback like above) ... improving the stats is goal two :)
> (If the PHP sources and a database export were publicly
> available, I would have taken a shot at implementing it.)
Antony is working on site design and such, but if you are interested in
helping out with building reports, and have time, we'd be happy for the
extra help :)
Marc G. Fournier
> Marc, thanks for all your hard work on these issues.
>
> One small change needs to be made. The pkg-message file reads, at its end:
> o view current statistics, go to:
> http://bsdstats.hub.org
>
> That needs to be changed to http://www.bsdstst.org/
>
> This patch will fix it.
>
> --- pkg-message.orig Mon Aug 14 10:02:29 2006
> +++ pkg-message Mon Aug 14 10:02:51 2006
> @@ -10,6 +10,6 @@
> /usr/local/etc/periodic/monthly/300.statistics
>
> To view current statistics, go to:
> - http://bsdstats.hub.org
> + http://www.bsdstats.org
> ********************
Fixed, thanks ...