Predictor

[Daniel Richman]

Hi everyone,

TL;DR: the predictor now works, and works very quickly at that!
http://predict.habhub.org (the old address redirects). While I've been
quite careful to make sure it behaves exactly the same as the old
predictor, it's a mix of old and relatively new software so should
perhaps be regarded as beta.
(It's also maybe worth reminding you that no guarantee is given for
the accuracy of any data included or produced by this program, use it
at your own risk.)


(not TL;DR - What follows is a brief tale of the problems of prediction:)

As you've no doubt aware, the predictor has been a little bit broken
over the last week.
This was, as almost every problem with the predictor is, due to the
NOAA wind data servers. (Though, we are of course very grateful for
them providing wind data for free :-).)

The predictor used to use their OPeNDAP servers. It's basically a Java
(and therefore slow) web application; we would ask it "can I have wind
data in this latitude/longitude range please". We've had some minor
issues with them (e.g., "blah is not an available dataset" due to its
domain resolving to two IP addresses with a very low TTL), but it was
generally OK. However, over the last week they've become too slow for
the predictor to get any wind data at all.

It turns out you can just wget the the entire forecast off a NOAA
HTTP/FTP server - so that's what we're doing now. Every 6 hours a
daemon downloads ~6-7GB - the entire forecast - and decompresses it
for the predictor to use. As a result, predictions complete
near-instantly; sometimes taking a second or two to bring the data
from disk into RAM, or if you're lucky and it's cached, about 20
milliseconds.

I've upgraded the predictor and spacenearus' live predictions. I will
upgrade the hourly predictor soon (though am going on holiday
shortly...), however, the 8GB download may be prohibitive to people
running their own hourly predictors. Luckily, since we now have
worldwide data on the habhub servers, we can start running hourly
predictions for anyone that wants them ourselves. I'll email when this
is ready.

It's worth noting that if you want a copy of the dataset yourself, it
might be better if we mirror it for you. Several benefits:

- the habhub.org servers (perhaps owing to us having gigabit, maybe
due to less load) are faster than the NOAA HTTP server
- only 25% of the 8GB download is actually useful, so we can mirror a
compressed 1.8G file.
- reduce load on the NOAA servers, because we're nice people

I'll have to double check that this is legal first. Let us know
(#habhub FreeNode, ukhas...@googlegroups.com) if you're interested.

This upgrade was actually part of larger CUSpaceflight work on
rewriting the predictor (replacing bits one at a time), which was
going to happen later; replacement of the wind downloading bit was
brought forward since the current one wasn't working.

Thanks,
Daniel

[Chris Atherton]

Thank you Daniel ( and everyone else ) for all the hard work!

Kind Regards,
Chris Atherton

> --
> You received this message because you are subscribed to the Google Groups "UKHAS" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ukhas+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Arko]

You guys rock! That loaded instantly!

Cheers,
Arko

[David Akerman]

Yeah this is pretty spectacular, and I can only guess at the concentrated burst of cleverness that went it making it happen.  So thanks from me also.

Dave

[Matt]

Thanks guys,

It's a great tool, and I'll feel far happier launching with it rather than "blind" next week.

Nice one

Matt

[Frits PE2G]

Many thanks Daniel, for your work on this! I use the predictor mostly for
landing prediction of meteorological sondes. And I know many other met
sonde chasers do as well.

Regards,
Frits PE2G

Op Fri, 12 Jul 2013 19:09:34 +0200 schreef Daniel Richman
<ma...@danielrichman.co.uk>:

> TL;DR: the predictor now works, and works very quickly at that!
> http://predict.habhub.org

--
| PE2G | JO32HI | pe...@xs4all.nl | http://home.deds.nl/~pe2g |

[Wolfbl]

Am Freitag, 12. Juli 2013 19:09:34 UTC+2 schrieb Daniel Richman:

 - only 25% of the 8GB download is actually useful, so we can mirror a
compressed 1.8G file.

If you only use some variables and levels from the entire forecast you may

use a partial http transfer as described on

http://www.nomads.ncep.noaa.gov/txt_descriptions/fast_downloading_grib.shtm

best regards

Wolfbl 

l

[Andreas Richter]

Hi Daniel,

the behaviour of the noaa-servers is a little bit strange. Last week I tried to install the landing-predictor 2.5 on my server and it worked for 1 day. Then downloading of the wind-data gets very slowly and completely stops. I tried a solution with www.ncep.noaa.gov:9090 - and this worked for me since yesterday. Now this server is broken, too. But there must be another working server - because of http://www.projecthorus.org/predict/ works ("normally" slow...)

There is a possibility to get fast access http://www.nomads.ncep.noaa.gov/txt_descriptions/fast_downloading_grib.shtml ...

Can you tell me if there is a new version of the predict.py in the next future on github or is it possible to get the reworked file(s) in other places?

Best regards - 73 - df8oe (Andreas)

[Daniel Richman]

Hi Wolfbl,

I was aware of and investigated this. The variables and levels are spread throughout the entire file, and the overhead of using HTTP Range: to get little chunks outweighs the benefits. The overall download time was approximately the same either way.

Thanks,
Daniel

[Daniel Richman]

Hi Andreas,

I can't offer any advice on the NOMADs (port 9090) servers beyond what I've said already I'm afraid since I neither run them nor use them any more.
With regards to v2.5 - our downloads have been consistently approx 1hr, occasionally 1h30. Are you sure this is what you installed (check the about section)?

Daniel

--

[Andreas Richter]

Hi Daniel,

I think I haven't written it exactly so that you have misunderstood. This is he version I installed:

https://github.com/jonsowman/cusf-standalone-predictor

It gets gfs-data via dods-server port 9090. Now there is a possibility to get faster access to the gfs-data in this way:

http://www.nomads.ncep.noaa.gov/txt_descriptions/fast_downloading_grib.shtml

I think it is much faster than the java-port9090-method.

The way of downloading 8GB 4 times a day sounds not very attractive - and the bandwidth of the noaa-servers is weak if it takes one hour to get the file. The solution of downloading the complete file and then mirroring the wind-data we need is an interesting choice - it reduces datastram for the noaa-servers and may be your servers are not so busy. But 1.8GB are much data - in spite of beeing reduced. I don't know if it is possible to change data-acquisition in jonsowmans peoject to the "fast-access-method". And may be, the newer models appear earlier there as on dods-servers??!!

Your site is "fast like a lightning" - I have just time to move my mousepointer out of the window ;)

Best regards

Andreas

[Andreas Richter]

Maybe you can use filtered grib2 access... I have had a look at the 9090/dods - reliability and the http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_hd.pl simultaneously. The data from 9090-Server very often get stuck - but during these periods access to the grib-files was possible and fast.

Andreas

[Daniel Richman]

Hi Andreas,

Answers to your email are in-line:

On 21 July 2013 16:33, Andreas Richter <df8...@gmail.com> wrote:
> Hi Daniel,
>
> I think I haven't written it exactly so that you have misunderstood. This is
> he version I installed:
>
> https://github.com/jonsowman/cusf-standalone-predictor

From there, you will have got the code for v2. The code for v2.5 was
cooked up very quickly in response to the sudden degradation of the
DODS servers. It's in a branch (juryrig_tawhiri) on my fork of
cusf-standalone-predictor, but not really ready to be deployed at the
moment. While I have written the code to generate the mirror of the
dataset, I have not yet written code to automatically download and
unpack it: since we're offering both normal predictions and hourly
predictions via habhub.org to everyone, I haven't noticed much
interest in doing so.

> It gets gfs-data via dods-server port 9090. Now there is a possibility to
> get faster access to the gfs-data in this way:
> http://www.nomads.ncep.noaa.gov/txt_descriptions/fast_downloading_grib.shtml
> I think it is much faster than the java-port9090-method.

The server that mirrors those files is indeed far faster - it is the
server we are using.
With regards to using HTTP Range to select bits of the file, please
see my response to Wolfbl
https://groups.google.com/d/msg/ukhas/dgngiSs9Tak/0B52uWPf7HYJ

> The way of downloading 8GB 4 times a day sounds not very attractive - and
> the bandwidth of the noaa-servers is weak if it takes one hour to get the
> file.

The main limitation is that it takes 1 hour for the NOAA to push all
the data to their mirrors; the limiting factor is not the download
speed itself.

> The solution of downloading the complete file and then mirroring the
> wind-data we need is an interesting choice - it reduces datastram for the
> noaa-servers and may be your servers are not so busy. But 1.8GB are much
> data - in spite of beeing reduced. I don't know if it is possible to change
> data-acquisition in jonsowmans peoject to the "fast-access-method". And may
> be, the newer models appear earlier there as on dods-servers??!!

I'm afraid I don't have the free time to implement a different
download method - having made http://predict.habhub.org work, I
consider the downloading problem solved.
The complete datasets appear to end up on both servers at roughly the
same time, though this is only anecdotal evidence and I haven't
investigated properly.

> Your site is "fast like a lightning" - I have just time to move my
> mousepointer out of the window ;)
>
> Best regards
> Andreas

Thanks,
Daniel