Geo::WebService::Elevation::USGS

[Mike]

If there is anybody on Strawberry Perl, would
you please go to CPAN and install Geo::WebService::Elevation::USGS

Then run this program:

#!/usr/bin/perl

use strict;
use warnings;
use Geo::WebService::Elevation::USGS;

my $latd = '38.72360';
my $longd = '-90.76983';

my $getelev = Geo::WebService::Elevation::USGS->new();

my $elev = $getelev->elevation($latd, $longd)->{Elevation};

print "All done.   Elevation = $elev ft.\n\n";

__END__


I expect you will get this error:
500 Can't connect to ned.usgs.gov:443 (Bad address) at XXXX.pl line 12.


If instead you get the elevation, please let me know.


Then go to USGS.pm, take it off Read Only, and change line 133 from
use constant USGS_URL => 'https://ned.usgs.gov/epqs/pqs.php';
to
use constant USGS_URL => 'https://nationalmap.gov/epqs/pqs.php';

Then run the program and I expect you will get this error:
500 Can't connect to nationalmap.gov:443 (Bad address) at XXXX.pl line 12.

If you get either to run correctly, see if you have
Geo::WebService in your environment variables.


Thank you.


Mike

[Uri Guttman]

On 8/26/19 8:00 PM, Mike wrote:
>
>
>
> use constant USGS_URL => 'https://ned.usgs.gov/epqs/pqs.php';
> to
> use constant USGS_URL => 'https://nationalmap.gov/epqs/pqs.php';
>

have you tried to just telnet to those hosts?

 telnet nationalmap.gov:443
telnet: could not resolve nationalmap.gov:443/telnet: Name or service
not known
telnet ned.usgs.gov:443
telnet: could not resolve ned.usgs.gov:443/telnet: Name or service not known


so there is something wrong with the urls and not the code (or the code
has bad urls).

and just getting the web pages is also failing

GET https://ned.usgs.gov/epqs/pqs.php
<error>General failure: Invalid Coordinates</error>

GET https://nationalmap.gov/epqs/pqs.php
<error>General failure: Invalid Coordinates</error>

i dunno the module but it seems that you may not be passing in any
coordinates or something else is needed.

uri

[Mike]

Thank you all for the responses.

I now think this has nothing to do with
Geo::WebService::Elevation::USGS.  See
the script below for other web links that
LWP::Simple cannot reach and parse.  I have
not figured out why, but the problem is not
with USGS.pm,.

#!/usr/bin/perl
use strict;
use warnings;

use LWP::Simple;

chdir 'C:/Copy';

# Works

#my $test = get('http://osgeo-org.1560.x6.nabble.com/') or die 'Unable
to get page'; # Works
#my $test = get('https://www.chron.com/neighborhood/bayarea/') or die
'Unable to get page'; # Works
#my $test = get('http://history.house.gov/Map/Mapping-Congress/') or die
'Unable to get page'; # Works
#my $test = get('http://hint.fm/wind/') or die 'Unable to get page'; # Works
#my $test = get('http://antwrp.gsfc.nasa.gov/apod/astropix.html') or die
'Unable to get page'; # Works


# Does not work

my $test = get('https://nationalmap.gov/epqs/') or die 'Unable to get
page'; # Does not work
#my $test = get('http://www.wrh.noaa.gov/zoa/mwmap3.php?map=usa') or die
'Unable to get page'; # Does not work
#my $test = get('http://www.glorecords.blm.gov/') or die 'Unable to get
page'; # Does not work
#my $test = get('https://glorecords.blm.gov/default.aspx') or die
'Unable to get page'; # Does not work


print "\nAll done.\n\n\$test = $test\n\n";


__END__


There is stuff posted about this problem.  I have
been through using UserAgent, declaring the browser,
environment variables, and other things and it
still doesn't work.  Good thing I don't need this
or it would be driving me crazy.  I will probably
move on to other things now.



use strict;
use warnings;
use LWP::UserAgent;

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;


my $url = "https://nationalmap.gov/epqs/pqs.php";

my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0, } );


$ua->env_proxy;
$ua->agent("Mozilla/5.0 (Windows)");
my $response = $ua->get($url);

if ($response->is_success) {
     print $response->content;
} else {
   die $response->status_line;
}

print "All done.\n\n";

__END__


Mike

[Mike]

Thanks for the response.

Yeah, I don't understand why
https://nationalmap.gov/epqs/pqs.php
won't open in a browser, but
https://nationalmap.gov/epqs/
will.  I imagine this may have something
to do with redirects, but who knows.


When I put
https://nationalmap.gov/epqs/pqs.php
into a web browser it says:
General failure: Invalid Coordinates


When I put
http://ned.usgs.gov/epqs/pqs.php
into a web browser it says the same exact thing.


When I go to
https://nationalmap.gov/epqs/
and put -90.76983 for X
and put 38.72360 for Y
and hit "Get Elevation"
it gives:

<USGS_Elevation_Point_Query_Service>
  <Elevation_Query x="-90.76983" y="38.72360">
    <Data_Source>3DEP 1/3 arc-second</Data_Source>
    <Elevation>583.54</Elevation>
    <Units>Feet</Units>
  </Elevation_Query>
</USGS_Elevation_Point_Query_Service>



When I go to:
https://nationalmap.gov/epqs/pqs.php?x=-90.76983&y=38.72360&units=feet&output=xml
it gives:

<USGS_Elevation_Point_Query_Service>
  <Elevation_Query x="-90.76983" y="38.72360">
    <Data_Source>3DEP 1/3 arc-second</Data_Source>
    <Elevation>583.54</Elevation>
    <Units>Feet</Units>
  </Elevation_Query>
</USGS_Elevation_Point_Query_Service>

So that works.

Mike



On 8/26/2019 10:26 PM, Uri Guttman wrote:
>

[Mike]

Thank you.  That helps me a lot to know that it works for you.

Your code, exactly as you sent it, still does not work for me.
It gives:
500 Can't connect to nationalmap.gov:443 (Bad address) at trash.pl line 19.

I'm going to try to assign an environment variable to my
system.  I doubt that will fix it, but I am trying just
about everything.

I still can't figure out why it works on some URL's,
and not others.  It's probably going to take me a
long time to figure this out.


Mike



On 8/28/2019 6:26 PM, $Bill wrote:
> This seems fine on my Win10:
>
> use strict;
> use warnings;
> use LWP::UserAgent;
>
> my $url;
> if (0) {    # set to 1 or 0 to flop urls
>     $url = 'https://nationalmap.gov/epqs';
> } else {
>     $url =
> 'https://nationalmap.gov/epqs/pqs.php?y=34.0&x=-118.0&units=Feet&output=xml';
> }
> my $ua = LWP::UserAgent->new(); #  ssl_opts => { verify_hostname => 0,
> } );

> $ua->agent("Mozilla/5.0 (Windows)");
> my $response = $ua->get($url);
> if ($response->is_success) {
>     print $response->content;
> } else {
>     die $response->status_line;
> }
> print "All done.\n\n";
>
> __END__
>

> Dumps the page or elevation depending on url:
>
> perl test.pl
> <?xml version="1.0" encoding="utf-8"
> ?><USGS_Elevation_Point_Query_Service><Elevation_Query x="-118.0"
> y="34.0"><Data_Source>3DEP 1/3
> arc-second</Data_Source><Elevation>916.71</Elevation><Units>Feet</Units></Elevation_Query></USGS_Elevation_Point_Query_Service>All
> done.
>
>
>

[Olivier]

Mike <te...@mflan.com> writes:

> Thanks for the response.
>
> Yeah, I don't understand why
> https://nationalmap.gov/epqs/pqs.php
> won't open in a browser, but

In fact, it does open in a browser and the result is "General failure:
Invalid Coordinates".

This URL runs the PHP script pqs.php, this script needs input
coordinates, if there is no coordinates supplied, this script returns
the message about invalid coordinates.

This message is not a message from your browser, it is not a message
from the web server, it is a message from pqs.php. Your browser and the
web server are just fine.

And frankly speaking, what result are you expecting? pqs.php is a script
that returns some data depending on the input data you give. You give no
input, you get no output.

> https://nationalmap.gov/epqs/
> will.  I imagine this may have something
> to do with redirects, but who knows.

This URL is missing the filename part (after the last /), in that case,
the web server uses a default file (depending on the configuration of
the web server) and the default file is presenting a page that asks for
coordinates.

So you are asking 2 differents URL, that corresponds to two different
pages, each page giving the correct result.

> When I put
> https://nationalmap.gov/epqs/pqs.php
> into a web browser it says:
> General failure: Invalid Coordinates
>
>
> When I put
> http://ned.usgs.gov/epqs/pqs.php
> into a web browser it says the same exact thing.

And it is expected that the URL with http and with https give the same
correct result.

>
> When I go to
> https://nationalmap.gov/epqs/
> and put -90.76983 for X
> and put 38.72360 for Y
> and hit "Get Elevation"
> it gives:
>
> <USGS_Elevation_Point_Query_Service>
>   <Elevation_Query x="-90.76983" y="38.72360">
>     <Data_Source>3DEP 1/3 arc-second</Data_Source>
>     <Elevation>583.54</Elevation>
>     <Units>Feet</Units>
>   </Elevation_Query>
> </USGS_Elevation_Point_Query_Service>
>
>
>
> When I go to:
> https://nationalmap.gov/epqs/pqs.php?x=-90.76983&y=38.72360&units=feet&output=xml
> it gives:
>
> <USGS_Elevation_Point_Query_Service>
>   <Elevation_Query x="-90.76983" y="38.72360">
>     <Data_Source>3DEP 1/3 arc-second</Data_Source>
>     <Elevation>583.54</Elevation>
>     <Units>Feet</Units>
>   </Elevation_Query>
> </USGS_Elevation_Point_Query_Service>
>
> So that works.

It works because you are providing correct input.

Best regards,

Olivier

>
> Mike
>
>
>
> On 8/26/2019 10:26 PM, Uri Guttman wrote:
>>
>> have you tried to just telnet to those hosts?
>>
>>  telnet nationalmap.gov:443
>> telnet: could not resolve nationalmap.gov:443/telnet: Name or service
>> not known
>> telnet ned.usgs.gov:443
>> telnet: could not resolve ned.usgs.gov:443/telnet: Name or service not
>> known
>>
>>
>> so there is something wrong with the urls and not the code (or the
>> code has bad urls).
>>
>> and just getting the web pages is also failing
>>
>> GET https://ned.usgs.gov/epqs/pqs.php
>> <error>General failure: Invalid Coordinates</error>
>>
>> GET https://nationalmap.gov/epqs/pqs.php
>> <error>General failure: Invalid Coordinates</error>
>>
>> i dunno the module but it seems that you may not be passing in any
>> coordinates or something else is needed.
>>
>> uri
>>

--

[Mike]

Thanks for the response.

I agree with everything you say.
I just want my LWP::Simple to access every
web link that my browser can access.


I now know that some of you can run the
code below and get all 4 of the websites
I have listed to return the website info.
All 4 of those web links gives:
"Unable to get page at httpfetch.pl line 21."

I can see I am not alone in having this problem.
Others have had the same problem with only a few
web links.


#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;


# Does not work

my $test = get('https://nationalmap.gov/epqs/pqs.php') or die 'Unable to

get page'; # Does not work
#my $test = get('http://www.wrh.noaa.gov/zoa/mwmap3.php?map=usa') or die
'Unable to get page'; # Does not work
#my $test = get('http://www.glorecords.blm.gov/') or die 'Unable to get
page'; # Does not work
#my $test = get('https://glorecords.blm.gov/default.aspx') or die
'Unable to get page'; # Does not work


print "\nAll done.\n\n\$test = $test\n\n";


__END__

[Olivier]

Mike <te...@mflan.com> writes:

> Thanks for the response.
>

> I agree with everything you say.
> I just want my LWP::Simple to access every
> web link that my browser can access.
>
>
> I now know that some of you can run the
> code below and get all 4 of the websites
> I have listed to return the website info.
> All 4 of those web links gives:
> "Unable to get page at httpfetch.pl line 21."
>
> I can see I am not alone in having this problem.
> Others have had the same problem with only a few
> web links.

I don't see why you have a 'die at line 21' while your script has way
less than 21 lines. Also, the script comes all mangled by the mail, so
one cannot say if the error is yours or the mailer.

Make your script simple first, so you can confirm your environment is
working:

#!/usr/bin/perl
use LWP::Simple;
$test=get('https://google.com');
print $test;

When that is working, start changing things one at a time and check at
each stage that it is still working.

You may also explore the documentation of LWP::Simple to get an error
message that makes sense, because what you get so far is only 'it does
not work', one cannot make a diagnostic with that only.

Good luck,

Olivier

--

[Olivier]

--
And another question... Are you using any proxy in your broswer? If you
do, how did you imnplemented that proxy in LWP::Simple?

Olivier

[Mike]

In my Firefox it is set at:
"Use system proxy settings"
I turned it off briefly, but didn't really see
any changes.

I do not implement a proxy in LWP::Simple.

I don't think I use a system Proxy:
http://www.mflan.com/temp/proxy.jpg


Mike

On 8/29/2019 5:52 AM, Olivier wrote:

And another question... Are you using any proxy in your broswer? If you
do, how did you implemented that proxy in LWP::Simple?

Olivier


Olivier <o...@banyan.cs.ait.ac.th> writes:

[Mike]

I sure appreciate your help.

I have done quite a few things listed on the internet
with both Simple and UserAgent.

We see that many of the failing sites are govt
sites.  One guy can access all my Not Working sites,
but he says that glorecords and nationalmap.gov react
very slow, even when UserAgent uses the FireFox
identity.

I know I can use Mechanize and really dive into this.
And I might do that someday.  But for the moment I'm
thinking this is unlikely to get fixed in the next week.
Good thing I don't need this.


Mike

[Mike]

I was thinking it might give the fill in form
that you get when you go here:
https://nationalmap.gov/epqs/

I think you are right.  It is acting as intended
when visited with a browser.


Mike

On 8/28/2019 10:00 PM, Olivier wrote:

[Olivier]

Mike <te...@mflan.com> writes:

> I sure appreciate your help.
>
> I have done quite a few things listed on the internet
> with both Simple and UserAgent.

Sorry Mike, but I have hard time to get a clear picture of what you are
saying.

Did you manage to use LWP::Simple to load google.com?

This is really the first question, the first step to build the
diagnostic.

So far, I cannot even decide whether the version of LWP::Simple
installed on your machine is properly working or not. So I cannot decide
whether it is a network problem or a software problem.

Best regards,

Olivier

--

[Mike]

Yes.  Both of these work fine:

my $test = get('http://google.com/') or die 'Unable to get page'; # Works
my $test = get('https://google.com/') or die 'Unable to get page'; # Works

On 8/29/2019 9:33 PM, Olivier wrote:
> Mike <te...@mflan.com> writes:
>

> Sorry Mike, but I have hard time to get a clear picture of what you are
> saying.
>
> Did you manage to use LWP::Simple to load google.com?

Yes.  Both of these work:

my $test = get('http://google.com/') or die 'Unable to get page'; # Works
my $test = get('https://google.com/') or die 'Unable to get page'; # Works

[Olivier]

Mike <te...@mflan.com> writes:

> Yes.  Both of these work fine:
>
> my $test = get('http://google.com/') or die 'Unable to get page'; # Works
> my $test = get('https://google.com/') or die 'Unable to get page'; # Works

Good, do we know two things:
- your LWP::Simple is correctly installed
- you have no proxy

Next step is to see what is the result of https://nationalmap.gov/epqs/

So you should try with (note the / at the end):

my $test = get('https://nationalmap.gov/epqs/') or die...

Problem is that it is going through redirections and beacuase it is
encrypted, it is almost impossible to debug by hand.

Olivier

>
> On 8/29/2019 9:33 PM, Olivier wrote:
>> Mike <te...@mflan.com> writes:
>>
>> Sorry Mike, but I have hard time to get a clear picture of what you are
>> saying.
>>
>> Did you manage to use LWP::Simple to load google.com?
>
> Yes.  Both of these work:
>
> my $test = get('http://google.com/') or die 'Unable to get page'; # Works
> my $test = get('https://google.com/') or die 'Unable to get page'; # Works
>
>> This is really the first question, the first step to build the
>> diagnostic.
>>
>> So far, I cannot even decide whether the version of LWP::Simple
>> installed on your machine is properly working or not. So I cannot decide
>> whether it is a network problem or a software problem.
>>
>> Best regards,
>>
>> Olivier
>>
>
>
>

--

[Mike]

This one gives:
Unable to get page at httpfetch.pl line 35.

I wonder if all of these are redirects:


# Does not work

my $test = get('https://nationalmap.gov/epqs/') or die 'Unable to get

page'; # Does not work

#my $test = get('https://nationalmap.gov/epqs/pqs.php') or die 'Unable

to get page'; # Does not work
#my $test = get('http://www.wrh.noaa.gov/zoa/mwmap3.php?map=usa') or die
'Unable to get page'; # Does not work
#my $test = get('http://www.glorecords.blm.gov/') or die 'Unable to get
page'; # Does not work

#my $test = get('https://glorecords.blm.gov/default.aspx') or die
'Unable to get page'; # Does not work


Mike