Block web crawlers on your Tomcat

[rsignell]

Folks,

We found out the hard way that web crawlers can screw up Tomcat. We
had a catalog served by TDS that would intermittantly fail, and we
discovered eventually that it failed when Google crawled the Tomcat
location. After blocking all crawlers to the tomcat area, we haven't
had any more problems. Doing a little googling showed that others
have had this problem and came up with the same solution. So make a
"robots.txt" file and put it in your tomcat/webapps/ROOT directory.
Something like this:

$ cd /usr/local/tomcat/webapps/ROOT
$ cat robots.txt
User-agent: *
Disallow: /


-Rich

[wen long]

Hi Rich,

  Thanks! I'm using tomcat as well.

Wen

[Richard Signell]

Wen,

Did you install the THREDDS Data Server to serve your forecast results?

There is a very cool feature of the TDS called "Forecast Model Run
Collection" which makes serving useful products from forecast model
results very easy.

-Rich

--
Dr. Richard P. Signell (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

[wen long]

Rich,

   Yes, I had THREDDS + ncWMS + netcdf Java etc on my laptop,, still experimenting it, but in general it works.

   I also have MapServer etc to show model results stored in GIS shape files in an old fashioned way.  Not sure what is the state of art in marrying GIS shapefile systems with netCDF, given both are large communities.

   I'll check out "Forecast Model Run Collection" when I get a chance, sounds very good stuff.

   I also see your njTBX on sourceforge,,, https://sourceforge.net/projects/njtbx/   http://njtbx.sourceforge.net/ . I love it and have been using it for taking Ruoying He's MABGOM results from THREDDS server at Rutgers to force MACROMS (Delawayre Bay and Chesapeake Bay all together) model .

   BTW, how do I enroll in the group using my work email wen...@hpl.umces.edu? I don't often visit gmail .

Wen