More frequent database updates?
SilverDK
Before i found wview i was using a windows program running inside a
VMware virtual machine to process the data from my Vantage Pro 2
station..
In that program it updated the database every minute instead of every
5 mins, and i kinda miss that functionality..
I was wondering if it's possible to set wview to add information to
the database at other frequencies than 5 mins (as a minimum) ??
I thank you all in advance.
Del
favorite file transfer tool) to a web site you can write to, then look
at (and modify) wviewftp.conf.
If you are referring to how often data is pushed to CWOP, it looks
like there was a recent change:
http://www.wxqa.com/faq.html (see point #1, and the request to send
no more frequently than every 10 minutes)
I noticed some errors today sending data to CWOP: "CWOP-error: 32:
failed to write to rotate.aprs.net:14580!" I have logcheck running
and at worst case I saw this message three times in an hourly report.
Now I'm seeing it about once per hour. I haven't dug into it, but
there is no new news on the CWOP page indicating any problems with
host "rotate".
The new recomendations suggest that the sending software a) try to
send every ten minutes, based on the last digit of the sender's CWOP
ID, and b) if your CWOP ID ends in "0" or "5", to send every 9 or 11
minutes. I'm CW9500: does wview follow these new rules?? I'm running
3.4.1.
-Del
Mark S. Teel
Mark
SilverDK
Vantage Pro Console..
I know that it can be set to lower than 5 mins at min, because the
windows program i used could set it at 1 min intervals..
Also i've noticed, after playing a bit around with the source, that
you can change the vpconfig program to set the console to archive
every 1 min, but then the rest of wview doesn't seem to work. :(
Is it possible that we in the future can support lower than 5 min
archive intervals ?
On Jan 22, 2:40 am, Del <Uncl...@gmail.com> wrote:
> If you are referring to how often wview pushes data (via FTP or your
> favorite file transfer tool) to a web site you can write to, then look
> at (and modify) wviewftp.conf.
>
> If you are referring to how often data is pushed to CWOP, it looks
> like there was a recent change:
>
Del
interval (and offset?) is configured. Would you offer a pointer?
-Del
On Jan 21, 9:35 pm, "Mark S. Teel" <mteel2...@gmail.com> wrote:
> Nope! wview (and I) were unaware of this. I'll look into it...
>
> Mark
>
> Del wrote:
> > If you are referring to how often wview pushes data (via FTP or your
> > favorite file transfer tool) to a web site you can write to, then look
> > at (and modify) wviewftp.conf.
>
> > If you are referring to how often data is pushed to CWOP, it looks
> > like there was a recent change:
>
Mark Teel
WEATHERFORYOU_INTERVAL...
Unless you want to change your archive interval (and throw away your WLK
files), you are stuck ;)
Mark
Del CW9500
mode? They say, that rapid fire PWS data can come in as often as
every 2.5 seconds. Yet CWOP doesn't wanna' hear from us more often
than every 9 to 11 minutes.
Now POLL_INTERVAL in my wview.conf is the default 15sec. I don't see
any references in the .conf files I have to any other "_INTERVAL"
names besides GENERATE_INTERVAL (1sec) and PUSH_INTERVAL (1min). Are
the others currently hardcoded in wview? Or can they be expressed in
a .conf and simply default as you mention below? And is
"ARCHIVE_INTERVAL" the same as "POLL_INTERVAL"? (I'm lazy on that
last question - I could look it up I'm sure...) It seems to me that
WUNDERGROUND is getting notified more often than CWOP, so it doesn't
follow that CWOP_INTERVAL == WUNDERGROUND_INTERVAL.
Thanks. This is such a cool tool. If I wasn't in the middle of my
own BIg Project I'd be a lot less lazy about RTFC.
-Del
On Jan 22, 2:31 pm, Mark Teel <mteel2...@gmail.com> wrote:
> ARCHIVE_INTERVAL = CWOP_INTERVAL = WUNDERGROUND_INTERVAL =
> WEATHERFORYOU_INTERVAL...
>
> Unless you want to change your archive interval (and throw away your WLK
> files), you are stuck ;)
>
> Mark
>
> Del wrote:
> > In the meantime, I seem to have overlooked where the push-to-CWOP
> > interval (and offset?) is configured. Would you offer a pointer?
>
> > -Del
>
> > On Jan 21, 9:35 pm, "Mark S. Teel" <mteel2...@gmail.com> wrote:
> >> Nope! wview (and I) were unaware of this. I'll look into it...
>
> >> Mark
>
> >> Del wrote:
> >>> If you are referring to how often wview pushes data (via FTP or your
> >>> favorite file transfer tool) to a web site you can write to, then look
> >>> at (and modify) wviewftp.conf.
> >>> If you are referring to how often data is pushed to CWOP, it looks
> >>> like there was a recent change:
Mark S. Teel
Bottom line: wview updates those services once every archive interval - period. They are not configurable. CWOP may or may not have to be changed. Don't get me started on "Rapid Fire"...
Mark
gsrabbit
junk. If CWOP doesn't want to hear from a station more often than
once/interval, they can dump it to null. Admittedly, they only upload
the data to NOAA 4X/hr.
Frankly, it would help them with redundancy in the data, especially
given the collision/retransmit nature of ethernet.
I find it hard to believe that their pipe is overloaded with input.
But, I'm not willing to sweat it. If CWOP is being picayune, I don't
have to send them data.
Keep up the great work Mark!
All the best,
Peter
On Jan 22, 11:31 am, Mark Teel <mteel2...@gmail.com> wrote:
> ARCHIVE_INTERVAL = CWOP_INTERVAL = WUNDERGROUND_INTERVAL =
> WEATHERFORYOU_INTERVAL...
>
> Unless you want to change your archive interval (and throw away your WLK
> files), you are stuck ;)
>
> Mark
>
> Del wrote:
> > In the meantime, I seem to have overlooked where the push-to-CWOP
> > interval (and offset?) is configured. Would you offer a pointer?
>
> > -Del
>
> > On Jan 21, 9:35 pm, "Mark S. Teel" <mteel2...@gmail.com> wrote:
> >> Nope! wview (and I) were unaware of this. I'll look into it...
>
> >> Mark
>
> >> Del wrote:
> >>> If you are referring to how often wview pushes data (via FTP or your
> >>> favorite file transfer tool) to a web site you can write to, then look
> >>> at (and modify) wviewftp.conf.
> >>> If you are referring to how often data is pushed to CWOP, it looks
> >>> like there was a recent change:
Del CW9500
wview's ability to meet that policy is a challenge, at least one of
these needs to happen:
1) wview stops sending "too frequent data"
2) open the issue up that led to the policy change and see what it
would take to reconsider it
3) ignore it all and hope that there are few-enough wview users that
we don't adversely impact CWOP's servers
Both 1 and 2 require someone (Mark?) do something. Of course, any of
us with time, patience, and a little programming expertise can "fix"
our local versions; it's a matter of whether or not Mark wants the
changes back. One place to discuss this might be here:
http://www.weather-watch.com/smf/index.php/topic,29185.0.html
#3 doesn't smack of being a "good citizen", but it certainly is a
simple solution for now.
In the meantime, my logs are seeing more "connect to CWOP" failures,
e.g.:
This email is sent by logcheck. If you wish to no-longer receive it,
you can either deinstall the logcheck package or modify its
configuration file (/etc/logcheck/logcheck.conf).
System Events
=-=-=-=-=-=-=
Feb 1 13:15:08 weathuh wvcwopd[9334]: <3593832954> : CWOP-error: 32:
The "32" error is the most common; the "104" sneaks in every so
often. Both appear to be radlib error codes that are not themselves
turned into human-readable messages. I haven't dug further.
When I have the time, I will look at wview source to see what it will
take to follow the CWOP recommendation. It seems the hard part will
be coalescing several intervals of data into a single "every 10
minute" packet, with the second hard part being handing of stations
ending in "0" or "5" (of which I happen to be one) sending every "9 or
11 minutes". A simplest-case solution would have wview sending the
data it has collected that is closest in time to the "proper" time and
just sending that.
-Del
pendragon
I'm new to wview, but I have a couple of years of archive interval = 1
min that I would like to extend when I convert over. So I have the
same issue as SilverDK. If I patch only vpconfig.c, htmlgend never
works. I did a quick scan of the sources and found that glchart.h
limits the maximum graph points to 1024, which isn't enough for one
day at one minute per day. Once that was changed, all seems to work. I
haven't done a detailed scrub yet, though. In case anyone is
interested, here's the patch I generated against 3.5.0:
--- htmlgenerator/glchart.h.orig 2008-02-15 00:38:25.000000000
-0700
+++ htmlgenerator/glchart.h 2008-02-15 00:38:49.000000000 -0700
@@ -41,7 +41,7 @@
#define GLC_CONTENTS_OFFSET 4
-#define MAX_GRAPH_POINTS 1024
+#define MAX_GRAPH_POINTS 2048
#define GLC_VALUE_NONE -10000.0
--- stations/VantagePro/vpconfig/vpconfig.c.orig 2007-10-29
08:25:57.000000000 -0600
+++ stations/VantagePro/vpconfig/vpconfig.c 2008-02-14
21:37:30.000000000 -0700
@@ -122,12 +122,13 @@
static int setInterval (int value)
{
- if (value != 5 &&
+ if (value != 1 &&
+ value != 5 &&
value != 10 &&
value != 15 &&
value != 30)
{
- printf ("archive interval must be one of: 5, 10, 15, 30\n");
+ printf ("archive interval must be one of: 1, 5, 10, 15,
30\n");
return ERROR;
}
--- wviewconfig/wviewconfig.sh.orig 2008-02-12 20:23:41.000000000
-0700
+++ wviewconfig/wviewconfig.sh 2008-02-14 21:43:17.000000000 -0700
@@ -459,7 +459,7 @@
echo "# DO NOT CHANGE this once you have archive records
stored in /var/wview/archive!" >> $WVIEW_CONF_DIR/wview.conf
echo "# This effects many things (CWOP/Wunderground update
interval, etc.) so choose carefully..." >> $WVIEW_CONF_DIR/wview.conf
- echo "# Weather data archive interval (minutes, one of 5, 10,
15, 30):" >> $WVIEW_CONF_DIR/wview.conf
+ echo "# Weather data archive interval (minutes, one of 1, 5,
10, 15, 30):" >> $WVIEW_CONF_DIR/wview.conf
echo "STATION_ARCHIVE_INTERVAL=$STATION_ARCHIVE_INTERVAL" >>
$WVIEW_CONF_DIR/wview.conf
echo "" >> $WVIEW_CONF_DIR/wview.conf
fi
@@ -758,7 +758,7 @@
echo "--Note-- choose it carefully and be prepared to
stay with your choice!"
echo "--Note-- If unsure what all this affects, CTRL-C to
exit this script and do"
echo "--Note-- a search for 'archive interval' in the
wview User Manual..."
- echo "Weather data archive interval (minutes, one of 5, 10,
15, 30)?"
+ echo "Weather data archive interval (minutes, one of 1, 5,
10, 15, 30)?"
echo -n "($STATION_ARCHIVE_INTERVAL): "
read INVAL
if [ "" != "$INVAL" ]; then
pendragon
corrupted by another array that needs to be expanded. Please add this
to the previous patch:
--- htmlgenerator/images.c.orig 2008-02-15 02:34:42.000000000 -0700
+++ htmlgenerator/images.c 2008-02-15 02:35:20.000000000 -0700
@@ -51,7 +51,7 @@
/* ... (local) memory declarations
*/
-static char *labels[MONTHLY_NUM_VALUES];
+static char *labels[MAX_DAILY_NUM_VALUES];
static char *monthLabels[12] =
{
--- htmlgenerator/images-user.c.orig 2008-02-15 02:34:56.000000000
-0700
+++ htmlgenerator/images-user.c 2008-02-15 02:35:38.000000000 -0700
@@ -48,7 +48,7 @@
/* ... (local) memory declarations
*/
-static char *labels[MONTHLY_NUM_VALUES];
+static char *labels[MAX_DAILY_NUM_VALUES];
static char *monthLabels[12] =
{
chrisale
Hey Mark... is there any way at all that we could bribe you into
rolling this into a new release? It would seriously be like the holy
grail for me...
I send data to the University of Victoria... the person in charge of
the program is a contributor to the International Panel on Climate
Change, and they demand 1 minute data! I currently have a bit of a
hack satisfying them... this would sure help out!
Chris
Mark S. Teel
There is more to this I believe, I don't want to hear all of the
follow-on complaints and bug reports, and it is a silly requirement. One
minute of weather data does not constitute an "archive" record. Have
any of you done the math to realize how much storage that would
require? And I don't think the history graphs would be very nice
either. Way too many unintended consequences.
Sorry, hack to your heart's content, but leave me out of it.
Mark
Chris Alemany
If I went to 1 minute archive that would increase the records to 1,186,125 and around 110MB of storage. That means it would take me 18000 years to fill up one 1TB drive today that I can buy for $250. Plus my web and mysql host gives me unlimited space! (for $5/mo!)
Is storage really an issue Mark?
There are quite a few ways to optimize the mysql structure as it is now...
INT takes 4 bytes per entry...
Winddir colums could use SMALLINT and cut down to 2 bytes per entry
ArcInt could use TINYInt... cut down to 1 byte per entry.
Could also use TIMESTAMP instead of DATETIME... 4 bytes instead of 8, and arguably easier to deal with time on both ends since we're all on *Nix platforms.
If the default is still 5 minute inteval archives for wview, with great big bold letters saying "change to 1 minute at own risk", what's the harm?
Chris
Gerry Creager
"autocorrelation" depending on the frequency we're talking about.
We've had quite a few discussions within CWOP (more the professional
level than the general volunteers, although a LOT of them are
knowledgable enough to well participate) and have not reached much
conclusion save that 1 minute intervals are likely of little scientific
utility. I'll have to check on whether IPCC is looking at 1 minute
data: If so, and they have a lot of sites doing so, almost certainly
they have a larger time-synchronization problem than they realize.
Getting cheap clocks synchronized is not easy. Even for GPS it takes at
least 7 satellites and a couple of billion dollars of otherwise hidden
costs.
I agree that space for one station is hardly the issue but space for 4k
stations, plus data organization, catalog service for discovery, quality
checking, recovery, etc., all are considerations. I tend, these days,
to buy disk in 16-32TB chunks, and then I have to organize whatever I
put out there, so I sorta live these things all day long.
Oh, and while we're at it, translating the data to put it into another
format is usually verboten. OR, at least, bad practice: Save it in its
original form in a real archive setting and use a translation for
transient/ephemeral activities. When you do that, storage space
requirements can as much as double.
I'm not sure I agree with all of Mark's arguments, but I do support his
conclusion at this point, although having something that archived at a
truly user-definable rate would be interesting for some short-term
projects...
gerry
> <mailto:m...@taom.com>> wrote:
> >
> >> Whoops. When I looked at the generated graph labels, they were
> >> corrupted by another array that needs to be expanded. Please add
> this
> >> to the previous patch:
> >>
> >> --- htmlgenerator/images.c.orig 2008-02-15 02:34:42.000000000 -0700
> >> +++ htmlgenerator/images.c 2008-02-15 02:35:20.000000000 -0700
> >> @@ -51,7 +51,7 @@
> >>
> >> /* ... (local) memory declarations
> >> */
> >> -static char *labels[MONTHLY_NUM_VALUES];
> >> +static char *labels[MAX_DAILY_NUM_VALUES];
> >>
> >> static char *monthLabels[12] =
> >> {
> >> --- htmlgenerator/images-user.c.orig 2008-02-15
> 02:34:56.000000000
> >> -0700
> >> +++ htmlgenerator/images-user.c 2008-02-15 02:35:38.000000000 -0700
> >> @@ -48,7 +48,7 @@
> >>
> >> /* ... (local) memory declarations
> >> */
> >> -static char *labels[MONTHLY_NUM_VALUES];
> >> +static char *labels[MAX_DAILY_NUM_VALUES];
> >>
> >> static char *monthLabels[12] =
> >> {
> >>
> >> On Feb 15, 1:09 am, pendragon <m...@taom.com
> <mailto:m...@taom.com>> wrote:
> >>
> >>
> >>> I'm new to wview, but I have a couple of years of archive
> interval = 1
> >>> min that I would like to extend when I convert over. So I have the
> >>> same issue as SilverDK. If I patch only vpconfig.c, htmlgend never
> >>> works. I did a quick scan of the sources and found that glchart.h
> >>> limits the maximum graph points to 1024, which isn't enough for one
> >>> day at one minute per day. Once that was changed, all seems to
> work. I
> >>> haven't done a detailed scrub yet, though. In case anyone is
> >>> interested, here's the patch I generated against 3.5.0:
> >>>
> > >
> >
>
>
>
>
> >
--
Gerry Creager -- gerry....@tamu.edu
Texas Mesonet -- AATLT, Texas A&M University
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
Mark S. Teel
wview does archive at a user-definable rate, just not an idiotic rate.
Why not archive every second or millisecond or nanosecond? Sounds sexy,
right?
My fundamental argument with this is based on common sense and logic.
It makes no more sense to me to use a millisecond archive interval than
a minute. Weather data is quite "constant" when viewed with a precision
of 5 minutes, much less one minute or 1 millisecond. As stated before,
with the exception of wind, weather data "rate of change" is generally
near zero even at 5 minute intervals. They are even more boring at one
minute intervals. Do we just want to archive data more often to say we
can? And the transient nature of wind fluctuations does not generally
contain interesting data - in the last minute it fluctuated from 5-15
mph, but averaged 7-8 just like it did for the last five hours... Get a
sense of my argument yet?
The number of rational numbers in the interval [0,1] is countably
infinite yet the number of irrational numbers in that same interval is
uncountably infinite. Shouldn't it therefore drive you crazy that we
can't make infinitely many archive observations in the next second or
minute?
Common sense: why make 1 minute archive observations when it will
benefit us roughly 0.05% of the time and otherwise waste disk space (and
my time continuing to make this argument)? Most folks aren't going to
buy terrabyte storage, keep it backed up, etc. It's silly for this
purpose with no demonstrable gain or improvement to our ability to
forecast or advance the human race. More folks are running from a thumb
drive with 4 GB of storage space. Now that is the right scale for
weather data for one station.
I am rather dubious that CWOP, WeatherUnderground or any other data
mining service really has a handle on storage requirements, network
issues, appropriate protocols, etc. Look no further than the recent
CWOP meltdown (with temporary solutions implemented) or the idiocy of
WeatherUnderground's "Rapid Fire" service (and their immediate network
issues since) for examples of poor judgement in higher places. Does
anyone feel better or have our forecast models improved because hundreds
of misguided folks can shotgun data to WU every five seconds or whatever
the ridiculous interval is? No, I didn't think so. But again, it
sounded sexy, right? Who cares if so much internet bandwidth is eaten up
with the same weather data that was sent 5 seconds ago, let's just make
the pipe bigger, right? Hell, I guess the US government should be
subsidizing all this anyway since we are entitled to as much excess and
waste as we can muster - dare not judge if it is right or wrong - just
shut up and give more taxes next year so the secularists can divvy it up
as they see fit.
I feel better now. Please let the conservative old Texan be - he is
pretty nice if you don't poke him too much ;)
Mark
Gerry Creager
Texan. However, this is much more in the spirit of a debate than an
argument to me. Please take no offense.
The CWOP service should have been on its own servers, using a different
protocol, long ago but was in fact adapted from APRS, its internet
service and transport/spec, because APRS had some users with weather
stations who were willing to volunteer their data. I don't think anyone
expected CWOP to grow to the size it did... either as fast as it did or,
for that matter, ever. That's success, and with precious little
advertising. I'm now trying to deal with the aftermath of that. What
precipitated this harkens back to some well-intentioned plans to host
CWOP services on a 2nd tier of the APRS servers, which were at that time
subject to some hardware and connectivity failures. Moving the CWOP
services to the Core APRS servers worked fine until a spike of almost
1000 new CWOP stations after Christmas 2007 highlighted protocol
problems: The server software rather faithfully follows the spec
requirements, and therefore introduces sorting and filtering overhead
that is compounded by operation in the limitations of a Java VM
environment. Solution in the long run? Change the spec, transport
protocol and server operation. Short-term? Separate the services so
that CWOP and APRS can continue down their own development paths. The
reporting precision and accuracy for CWOP need to come into the same
level of consideration laid out by WMO and NWS guidelines, and that's
never gonna happen under the APRS spec without adding the info as a
comment... say, in METAR format.
WUnderground can handle both the update rates and storage with little
problem. Their servers are not using the same protocol or software and
don't have to conform to the spec CWOP has to live with. They can
accept (and do) much faster updates. I'm working to get the servers
ready to handle faster updates for CWOP, but there the story compounds
again.
NOAA GSD, the group that runs the MADIS program, has been doing MADIS as
a GSD initiative at the Director's discretion for some time. It's a
successful program and slated to go "operational" over the next several
years. In fact, it's so successful that some folks want to completely
rewrite the applications that comprise it and start from scratch,
without the scientists who set it up in the first place. GSD is not
supporting this position. That's a whole 'nother issue. The MADIS
folks currently accept CWOP data at 15 min intervals and perform their
processing magic involving QC and de-duplication. That's
processing-intensive and the rate the elect to accept the data is based
on what they can process. They've got money for more CPU horsepower and
we're hoping to get them ready to accept data at 5 min intervals.
There's little chance they'll want it at 1 min intervals in the near
future. There _is_ a chance they won't be able to handle 5 min
intervals and we'll have to drop back. It's a hardware constraint.
Considering the number of stations they DO process, and that CWOP is one
of a number of networks contributing data into them, they do a pretty
credible job.
Do weather models improve because of high frequency data? Some, such as
the RUC, probably do. Others, eg., NAM and GFS, likely do not. ECMWF
incorporates the MADIS data at the 15 min rate. It's arguably the best
skilled model out there right now. Will more frequent updates help
models? Past 5 min rates, it's my official opinion the answer is "NO! NO
WAY!". And, yes, I'm shouting. That said...
Higher frequency data in real time, and not necessarily subject to the
MADIS QC, could have a significant effect on operational forecasting.
The oft-cited example/poster-child for this is the 6 MAR 2004 water taxi
event in the Baltimore Inner Harbor, when a gust-front associated with a
cold front caused a boat to capsize. NWS, relying in ASOS equipment and
WSR-88D, had little indication that the gust-front was forming. AWS...
and CWOP... had much more indication because of their increased density
across the area. AWS leveraged this into a contract to provide data to
NWS. CWOP data, already available to NWS by those few WFOs who used
MADIS, gained more visibility, as did MADIS itself. Lots more WFOs are
using MADIS and your data now, as a result of that increased visibility.
Another issue with data sample rate for CWOP goes, once again, back to
the way the original data were reported: On a shared radio channel with
a lot of other users. If you had your weather station reporting too
often you were a bad user, because you weren't sharing. Therein lies
another part of the problem: We transfered an artificial restriction
onto a communications method (internet) from another more restrictive
one (RF) that we didn't need, and called it normative.
Now, a lot of Mark's points are well taken. I'm not going to quibble
with much. Instantaneous wind measurements are problemmatic because so
many folks fail to regularly and sufficiently service their anemometers,
and after awhile, they get sticky, require more wind to turn them or
start 'em turning. And few enough of us use the expensive sonic
anemometers (I do because I'm using hardware from work, but I couldn't
afford it otherwise!). Similarly, most temperature changes are
relatively slow and reflect environmental changes such as the overall
thermal mass of the sheld they're in. Barometric pressure is often
measured inside the house, generally OK, since most houses are poorly
sealed. Even the good ones. Humidity sensors, especially the
inexpensive ones, tend to be very slow to move at higher humidity
readings because you're talking about measuring capacitance in a cheap
ceramic electronic lumped sum part. Ceramic holds water well. Lots
more to this story but you get the idea.
Several of us have debated the correct sampling rate of the atmosphere
for a surface site and in fact, the rate varies with the phenomenon
being sampled but that's sorta impractical for most folks.
Realistically, one wants to sample the whole site every time one
samples. And then do it again. My personal feeling is that 5 minute
rates are pretty close to the optimum and that I don't have time to take
on the research needed to determine what the correct sampling rate is,
or, for that matter if it's related to auto-correlation as opposed to
Shannon/Nyquist. That may end up as a graduate thesis in the future.
That said, I commend the interested reader to follow
http://www.weatherchaos.umd.edu/papers/RDM.pdf for more information.
In some of the work I do, it's occasionally nice to be able to obtain
the data and write it to archive at seemingly stupid (or was that
stupifying) rates. These, however, are few and far between as my two
competing interests are mesoscale events where 5 min updates are plenty,
and tropical cyclone prediction enhancement, where 15 minute surface
data are more than adequate. That said, if we can ever nail down the
optimax frequency of surface weather data collection, it might be nice
to have that as a max setting, or it might not: Perhaps, for the
purposes of this software, Mark's absolutely right. For that matter, he
IS right as he's the coder and maintainer. Source is available: If you
want to tweak, do so at your own risk/leisure.
Let's wander toward the arena of data storage and management. I handle
a couple of fairly large datasets. My WRF model results, excluding the
nearly 1600 images for analysis, comprise about 32GB of data every 6
hours. That's a lot but it's one data subset per big batch, so it's
manageable. I also handle data for tropical cyclone predictions of
ocean circulation, wind-waves, surge and inundation, formulated in
ensemble format with members numbering from 11 to 57 dependign on storm
and setting. That's a more formidable problem in data handling. I also
handle data from buoys, ships, and NOAA PORTS and CMANs, as well as
tidal and water level gauges. Lots of small observations. Best handled
by a well-designed relational (and spatially aware) database. I'm
charged with maintaining the holdings, knowing where they are, how to
get to them, and with populating a catalog service that enables data
discovery. With a little organization, well, it just works. Lots of
spinning disk (although not by some business standards: I'm at less than
100 terabytes still). So handling the data isn't a huge problem.
Did I miss anything? I'm off to a Little League game.
gerry
--
Chris Alemany
Gerry Creager
things out.
gerry
--
Chris Alemany
Just have one other thing to add... that strikes me about the debate on the usefulness of 1min data.
I suppose I see it more as something users want to see.... like, would we all be happy if our Vantage consoles only displayed updated data every five minute?
Of course not... we want to see that gust of wind, or sudden downpour out our window and go running to the console to see how big it really was.
The same applies to an Internet site. I have many people who just leave alberniweather.ca on their Explorer in a seperate window for hours at a time. (at least according to my stats). Especially when the weather is notable.
Do they need updates every 15 seconds like a console? No... bandwidth is always a concern.. but I already have my website updating every minute with new data... bandwidth is not a problem for the traffic I get... why not allow the database to support that as well so that you can see those spikes of windchill in the graph, or sudden downpour. Quite often this year I've seen low windchills displayed on the webpage... but the graph does not reflect it at all, simply because it doesn't have the same resolution.
....
OK, I'm done. :)
Chris
buzzkill
talking about 2 separate issues. And the reason I noticed it was that
Mark always referred to the "updates" as "archives". And I would have
to agree that from an archival perspective 5 minutes is more than
adequate. But to Chris' point regarding web page and graph data it also
makes sense. But since the graphs are generated from the archive data,
I do not know how Mark would reconcile updating graphs every minute, but
only archiving data every 5.
I know that when I was running Weather Display on Windows, I very much
liked the interface of Weather Display Live. I also understand the
genesis of wview though, and its compact, efficient, and speedy graph
generation. Frankly, I am so freakin happy to have a stable linux
weather app for my VP2 that I can just sit on the fence with this one.
I would be fine continuing towards making the dials and graphs look
nicer. I know that it would require more CPU, but some of us have the
CPU to use.
Chris Alemany wrote:
> Hey yall,
>
> Just have one other thing to add... that strikes me about the debate
> on the usefulness of 1min data.
>
> I suppose I see it more as something users want to see.... like, would
> we all be happy if our Vantage consoles only displayed updated data
> every five minute?
>
> Of course not... we want to see that gust of wind, or sudden downpour
> out our window and go running to the console to see how big it really was.
>
> The same applies to an Internet site. I have many people who just
> leave alberniweather.ca <http://alberniweather.ca> on their Explorer
Mark S. Teel
Wind, etc.) are updated as often as your generation interval (mine is 1
minute). Only the historical graphs are based on archive data. Each
daily (24 hour) graph has 24 x 12 (assuming 5 minute archive interval) =
288 points.
Now, I don't know how many times I have to explain this, but that gust
of wind or that downpour *WILL* show up on your website *IF* you have
your generation (not archive) interval set to 1 minute. It doesn't show
up in the historical graphs (get it, historical) but does show up in
tabular data and the *CURRENT CONDITIONS* dials (and buckets for that
matter). Did that get through this time? Again, every *1 minute*
CURRENT CONDITIONS are updated and every *5 minutes* HISTORICAL data are
updated for your website.
I believe it provides a balance of *near* real time for current
conditions and a configurable archive (ARCHIVE) interval for historical
data. If running outside because of an "in progress" event is the
argument for 1 minute archive interval, then the argument is lost.
Current conditions give you that "feel good" responsiveness and no one
gives a damn about what minute it happened in this 5 minute interval
when mining the data 6 days/weeks/months/years from now.
One more time: IN TERMS OF GRAPHICS FOR YOUR WEBSITE, ONLY THE
HISTORICAL GRAPHS ARE AFFECTED BY ARCHIVE INTERVAL.
Mark