Issue 63 in ioostech: Inconsistent representation of URLs in templates and GetCapabilites implementations

[ioos...@googlecode.com]

Status: New
Owner: dpsnowde...@gmail.com
CC: alexande...@noaa.gov, sh...@axiomalaska.com, wilcox.k...@gmail.com,
emilioma...@gmail.com
Labels: Type-Defect Priority-Medium

New issue 63 by dpsnowde...@gmail.com: Inconsistent representation of URLs
in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

This issue was submitted to Derrick by Bob Simons. He raises a good point
about our clarity in writing the documentation. It is old, so we should
check the current versions of 1) templates, 2) ncSOS RC8, 3) 52N v0.7.5 and
alter the templates to be consistent.

This will sound like a minor, petty, difference, but it is a good example
of an anomaly that forces the client software to deal with different types
of servers differently (or at least have code to deal with different
possibilities): In the GetCapabilities response from the 52N server
http://sensorweb.demo.52north.org/52nSOSv3.2.1/sos
for every ows:Operation/ows:DCP/ows:HTTP,
every ows:Get URL includes a ? at the end
every ows:Post URL doesn't have a ? at the end.
I don't know if this is correct. But it is reasonable.

However, the IOOS GetCapabilities template at
https://code.google.com/p/ioostech/source/browse/trunk/templates/Milestone1.0/SOS-GetCapabilities.xml
just has SOS_SERVER for both all Get URLs and Post URLs.

Similarly, the GetCapabilities document from the NDBC SOS server at
http://sdf.ndbc.noaa.gov/sos/server.php?request=GetCapabilities&service=SOS
does not include a ? at the end of the ows:Get URLs.

Similarly, the GetCapabilities document from the NOS SOS server at
http://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS?service=SOS&request=GetCapabilities
also does not include a ? at the end of the ows:Get URLs.


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[ioos...@googlecode.com]

Comment #1 on issue 63 by emilioma...@gmail.com: Inconsistent

representation of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

I completely agree with that our template XML's and representative
endpoints (demo ncSOS and 52N) should be as consistent as humanly possible.

In the template GetCapabilities, the SOS_SERVER entries should be replaced
with a fake but correct-looking URL's.

I don't know if the 52N approach is best, regarding the use of "?". Does
OGC have anything to say about this, for SOS GetCapabilities, and more
generally for other OGC GetCapabilities responses (since this is a "ows"
element)? Is there a common practice used in, say, GeoServer GC's and other
OGC W*S implementations? One place to ask may be the Python OWSLib list, to
see what's most common, if OGC doesn't have a documented recommendation.

[ioos...@googlecode.com]

Comment #2 on issue 63 by dpsnowde...@gmail.com: Inconsistent

representation of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

I did a brief google based survey of some example servers and more often
than not I saw the URL punctuated with a ?. Can't say whether it is in the
spec or not yet.

[ioos...@googlecode.com]

Comment #3 on issue 63 by alexande...@noaa.gov: Inconsistent representation

of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

Section 3 'Syntax Components' of the RFC-6874 'Uniform Resource Identifier
(URI): Generic Syntax' reads:

The generic URI syntax consists of a hierarchical sequence of components
referred to as the scheme, authority, path, query, and fragment.

URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

Then, Section 3.4. 'Query' explains further:

The query component is indicated by the first question mark ("?")
character and terminated by a number sign ("#") character or by the end of
the URI.

So, my interpretation of these rules is that "?" is a part of a query, and
as such should not be left behind if the query itself is omitted. Leaving
"?" alone without variables may be harmless (e.g. server may refresh the
page or do just nothing) but I believe that it is a bad practice.

Alex Birger
Systems Engineer

Skjei Telecom
7777 Leesburg Pike, Suite 315N
Falls Church, VA 22043
Direct: +1 703 917 9889
Main: +1 703 917 4077
Fax:���� +1 703 917 0098
e-mail: alex....@skjeitelecom.com
My availability: http://www.timebridge.com/mytime/alexanderbirger

CONFIDENTIALITY NOTICE: For intended Skjei Telecom recipients only.

[Derrick Snowden]

Just to be clear Alex, you're advocating for

Sos-serverurl?

And not

Sos-serverurl

Inside the capabilities XML.

Derrcik

--
--
You received this message because you are subscribed to the Google
Groups "ioostech_dev" group.
To post to this group, send email to iooste...@googlegroups.com
To unsubscribe from this group, send email to
ioostech_dev...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ioostech_dev?hl=en


---
You received this message because you are subscribed to the Google Groups
"ioostech_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to ioostech_dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
--
You received this message because you are subscribed to the Google
Groups "ioostech_dev" group.
To post to this group, send email to iooste...@googlegroups.com
To unsubscribe from this group, send email to
ioostech_dev...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ioostech_dev?hl=en


---
You received this message because you are subscribed to the Google Groups "ioostech_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ioostech_dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Alexander Birger]

Just the opposite:

 

I believe that there should not be “?” at the end of the URL unless it is followed by a real query. For example, “http://server/sos?service=SOS&request=GetCapabilities” is a good URL, while “http://server/sos?” is a bad one.

 

Alex Birger
Systems Engineer

Skjei Telecom

7777 Leesburg Pike, Suite 315N
Falls Church, VA 22043

Direct:   +1 703 917 9889

Main:    +1 703 917 4077

Fax:       +1 703 917 0098
e-mail: alex....@skjeitelecom.com

My availability: http://www.timebridge.com/mytime/alexanderbirger

 

CONFIDENTIALITY NOTICE: For intended Skjei Telecom recipients only.

 

[David Foster]

I back this 100%.

[ioos...@googlecode.com]

Comment #4 on issue 63 by sh...@axiomalaska.com: Inconsistent

representation of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

From OGC 06-121r9 OGC Web Services Common Standard [1] (from which SOS
derives), section 11.2, page 82:

A URL prefix is defined as a string including, in order, the scheme
("http" or "https"),
Internet Protocol hostname or numeric address, optional port number,
path, mandatory
question mark '?', and optional string comprising one or more
server-specific parameters
ending in a mandatory ampersand '&'. Thus, an HTTP GET URL prefix always
ends in
either a question mark '?' or an ampersand '&'.

[1] http://portal.opengeospatial.org/files/?artifact_id=38867

[ioos...@googlecode.com]

Comment #5 on issue 63 by alexande...@noaa.gov: Inconsistent representation

of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

That's right, but... In the very same section it also says that "The '&' is
a separator between name/value pairs, and is therefore optional after the
last pair in the request string", which implies that "?" is also optional as
it is the same kind of separator (Table 35 on page 76). Moreover, in the
Table 34 the structure of the URL is presented as

http://host[:port]/path[?{name[=value]&}],

where "[ ]" denotes 0 or 1 occurrence of an optional part, and "{}" denotes
0 or more occurrences. That example places "?" in the optional query part,
which is in fact exactly the same as in the RFC 2616 that OGC spec refers
to. The RFC 2616 in section 3.2.2, page 18 provides the following structure
of the HTTP URL:

http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]]

That syntactic record shows "?" as a part of an optional query rather than a
mandatory part of a URL.

Then, the RFC 2616 refers in turn to RFC 2396, which was replaced by RFC
3986 (sorry, I used a wrong RFC number in my previous comment), for syntax
details explanation. There (section 3.3, page 21) is a statement similar to
the one in the OGC spec:

The path is terminated by the first question mark ("?")
or number sign ("#") character, or by the end of the URI.

However, the very next section 3.4, page 22, defines a query component as

The query component is indicated by the first question
mark ("?") character and terminated by a number sign ("#") character
or by the end of the URI.

In addition, all RFCs and the OGC spec identify "?" as a general separator
along with "&" and "#".

Summing all that up, I believe that the statement in the OGC spec "HTTP GET

URL prefix always ends in either a question mark '?' or an ampersand '&'"

does not mean that "?" is a part of a prefix, and the URL must end up with
it even if query is missing. I believe that it rather means that "?"
separates prefix from query, and as such should be dropped altogether with
the query if the query is omitted.

Alex Birger
Systems Engineer

Skjei Telecom
7777 Leesburg Pike, Suite 315N
Falls Church, VA 22043
Direct: +1 703 917 9889
Main: +1 703 917 4077
Fax:���� +1 703 917 0098
e-mail: alex....@skjeitelecom.com
My availability: http://www.timebridge.com/mytime/alexanderbirger

CONFIDENTIALITY NOTICE: For intended Skjei Telecom recipients only.

[ioos...@googlecode.com]

Comment #6 on issue 63 by sh...@axiomalaska.com: Inconsistent

representation of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

I'm sorry but I don't think there's any ambiguity here:

"An Online Resource URL intended for HTTP GET requests is in fact only a
URL prefix"

"A URL prefix is defined as a string including...mandatory question
mark '?'"

"an HTTP GET URL prefix always ends in either a question mark '?' or an
ampersand '&'"

It's true that the URL pattern in Table 39 indicates that the question mark
is optional, but this is probably because they copied the pattern from
another spec.

There are multiple instances of clear, deliberate language stating that the
question mark is mandatory for GET URLs. I don't feel like there's any
argument to be made unless there's something in the SOS spec itself to the
contrary.

[Dan Ramage]

I'm debugging and testing the PyOOS library against an ncSOS server and have noticed that some of the server errors are a bit vague.

For instance the current version of the ncSOS server doesn't recognize eventTime as a valid parameter, the exception returned is:

<ows:Exception exceptionCode="NoApplicableCode">

<ows:ExceptionText>Unable to correctly create response for request: java.lang.StringIndexOutOfBoundsException: String index out of range: -1

</ows:ExceptionText></ows:Exception>

When possible, it would be helpful to pass along information detailing what piece of the request caused the issue, something like:

<ows:Exception exceptionCode="NoApplicableCode">

<ows:ExceptionText>Unable to correctly create response for request: java.lang.StringIndexOutOfBoundsException:String value: eventTime unknown

</ows:ExceptionText></ows:Exception>

Another example is passing in an observedProperty value that the server might not understand. The ncSOS server response is:

<ows:Exception exceptionCode="NoApplicableCode">

<ows:ExceptionText>Internal Error in creating output for GetObservation request - java.lang.NullPointerException

</ows:ExceptionText></ows:Exception>

Now I sussed out that the issue was the observedProperty by trial and error, but it would have been nice if the exception told me something like:

<ows:Exception exceptionCode="NoApplicableCode">

<ows:ExceptionText>Internal Error in creating output for GetObservation request - java.lang.NullPointerException. observedProperty: urn:ioos:sensor:org.secoora:carocoops.cap2.buoy:water_temperature is invalid

</ows:ExceptionText></ows:Exception>

I'm not picking on ncSOS in particular, it's just what I am currently testing against.

Dan

[Mike Garcia - NOAA Affiliate]

Shouldn't the exception code be InvalidParameterValue in this case with the locator attribute specifying the bad parameter?  Something like this?

<ows:Exception exceptionCode="InvalidParameterValue" locator="observedProperty">

<ows:ExceptionText>urn:ioos:sensor:org.secoora:carocoops.cap2.buoy:water_temperature</ows:ExceptionText>

</ows:Exception>

From Table 25 in OGC 06-121r3.

--

Mike Garcia

Senior Programmer Analyst

NDBC/TSC Data Systems

www.ndbc.noaa.gov

[Mohamed Chaouchi - NOAA Federal]

Hello,

If the error occurs because of an invalid parameter, the user should know about it. And the error message should not display any other extra information.

When an exception occurs, the application must be able to handle it internally, and display a very generic message back to the user. But, the application behind the scene should log the true cause of the error and probably send an email to the developer.

The main reason to implement the above is that stack traces or more revealing error messages can lead to the whole system be exploited.

Thank you,
Mohamed

[Dan Ramage]

Mike,

That type of exception is along the lines of what I was thinking. 

Dan

[Dan Ramage]

Testing against the 52 N test server, the error codes are inline with those. 

Adding an invalid parameter results in:

<ows:Exception exceptionCode="OptionNotSupported">

<ows:ExceptionText>The optional parameter 'eventime' is not supported by this service!

</ows:ExceptionText></ows:Exception>

Requesting an invalid observedProperty results in:

<ows:Exception exceptionCode="InvalidParameterValue" locator="observedProperty">

<ows:ExceptionText>The value 'Fsea_water_temperature' of the parameter 'observedProperty' is invalid</ows:ExceptionText>

</ows:Exception>

Dan

From: Mike Garcia - NOAA Affiliate <mike....@noaa.gov>
Reply-To: <iooste...@googlegroups.com>
Date: Fri, 24 Jan 2014 12:08:14 -0600
To: ioostech_dev <iooste...@googlegroups.com>
Subject: Re: [ioostech_dev] SOS Server Errors

[Kyle Wilcox]

This is most likely a NcSOS bug and an issue should be filed on GitHub.

The NcSOS tests that check for valid parameters are here:

https://github.com/asascience-open/ncSOS/blob/master/src/test/java/com/asascience/ncsos/GOAdditionalPlatformTests.java

There are no tests for validity or observedProperty or eventTime.  Pull requests are always accepted!

[Alexander Birger]

Could it be just a typo? The parameter name should be ‘eventtime’ rather than ‘eventime’ that I can see in the exception report.

 

Alex Birger
Systems Engineer

Skjei Telecom

7777 Leesburg Pike, Suite 315N
Falls Church, VA 22043

Direct:   +1 703 917 9889

Main:    +1 703 917 4077

Fax:       +1 703 917 0098
e-mail: alex....@skjeitelecom.com

My availability: http://www.timebridge.com/mytime/alexanderbirger

 

CONFIDENTIALITY NOTICE: For intended Skjei Telecom recipients only.

 

[Dan Ramage]

Alex,

That was intentional to see what the error record would be. Indeed it should be eventtime. I previously had issues running against an ncSOS server that returned an error message that was most likely returning the error/exception the server had thrown.

Dan

[Alexander Birger]

I second that: it should not be of any concern of an SOS client what happened behind the curtains of the ncSOS implementation. Therefore, only exception reports that are required by OGC standards must be sent out. Extended exception reports might be allowed for testing but must be suppressed for operations.

 

Alex Birger
Systems Engineer

Skjei Telecom

7777 Leesburg Pike, Suite 315N
Falls Church, VA 22043

Direct:   +1 703 917 9889

Main:    +1 703 917 4077

Fax:       +1 703 917 0098
e-mail: alex....@skjeitelecom.com

My availability: http://www.timebridge.com/mytime/alexanderbirger

 

CONFIDENTIALITY NOTICE: For intended Skjei Telecom recipients only.

 

From: iooste...@googlegroups.com [mailto:iooste...@googlegroups.com] On Behalf Of Mohamed Chaouchi - NOAA Federal

Sent: Friday, January 24, 2014 1:10 PM
To: iooste...@googlegroups.com

[ioos...@googlecode.com]

Comment #7 on issue 63 by sh...@axiomalaska.com: Inconsistent

representation of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

Propose to close with following actions:

1. Add trailing ? to ows:Get URLs in IOOS GetCap template
2. Replace SOS_SERVER with fake but correct looking URL in IOOS GetCap
template
3. Add ows:Get trailing ? to list of IOOS SOS rules

[ioos...@googlecode.com]

Comment #8 on issue 63 by emilioma...@gmail.com: Inconsistent

representation of URLs in templates and GetCapabilites implementations
http://code.google.com/p/ioostech/issues/detail?id=63

+1 to Shane's 3 proposals.
#3 may be tricky b/c the documentation of IOOS SOS rules is in a state of
flux, sort of. The official, up-to-date documentation is the monolithic
Word/pdf document, but the more compartmentalized wiki page is still far
easier to digest and edit, plus it's there. So, to minimize confusion both
the wiki page and the official document should be updated.

[Alexander Birger]

Shane,

If I am not mistaken, I believe that I've made #3 change in the WSDD. I will
double-check.

Alex Birger
Systems Engineer

Skjei Telecom
7777 Leesburg Pike, Suite 315N
Falls Church, VA 22043
Direct: +1 703 917 9889
Main: +1 703 917 4077
Fax:     +1 703 917 0098
e-mail: alex....@skjeitelecom.com
My availability: http://www.timebridge.com/mytime/alexanderbirger

CONFIDENTIALITY NOTICE: For intended Skjei Telecom recipients only.

-----Original Message-----
From: iooste...@googlegroups.com [mailto:iooste...@googlegroups.com]

On Behalf Of ioos...@googlecode.com
Sent: Wednesday, April 23, 2014 1:47 PM
To: iooste...@googlegroups.com

--
--

You received this message because you are subscribed to the Google Groups
"ioostech_dev" group.
To post to this group, send email to iooste...@googlegroups.com To
unsubscribe from this group, send email to
ioostech_dev...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ioostech_dev?hl=en

---
You received this message because you are subscribed to the Google Groups
"ioostech_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to ioostech_dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.