FDSN web service specification changes proposed by IRIS
Chad Trabant
Hello WG III members,
As presented at the WG meeting in Gothenburg, Sweden, below is a list of FDSN web service additions proposed by the IRIS DMC. With the exception of the POST submission method to fdsnws-station, all of these changes are in production at the IRIS DMC.
To provide further details and clarify how the changes would alter the specification, a draft version is attached with additions highlighted in red.
We proposed that these additions to the 1.0 specification become version 1.1 of the specification.
regards,
Chad
Proposed changes:
* Designate optional versus required parameters for each service interface. Effectively defining a minimum level of functionality that clients can expect. The WADL return from each service interface specifies which of the optional parameters are supported.
* Allow fdsnws-station to accept arbitrary selections via the POST method. Support for this method is required for all 1.1 implementations of the service. The request is to be formatted similarly to the POST selection method supported by fdsnws-dataselect, where parameter key-value pairs are followed by a block of windowed channel requests.
* Add optional 'format' parameter to all three interfaces. The default value for the fdsnws-station and fdsnws-event services is 'xml' (returning FDSN StationXML and QuakeML respectively), an additional value of 'text' is defined as returning results in a simple ASCII format (illustrated in the specification). The default value for fdsnws-dataselect is 'miniseed'. Future expansion of the accepted values for this parameter are anticipated to return the results in alternate formats.
* Add optional 'nodata' parameter to all three interfaces accepting values of '204' and '404' to control the HTTP status code returned when a successful request matched no data. The default value is '204'.
* Add optional 'matchtimeseries' parameter to fdsnws-station to limit the results to matching data availability times.
* Clarify definition of starttime and endtime parameters to explicitly include intersection with metadata epochs.
* Clarify definition of minimum and maximum latitude and longitude to be inclusive by adding "or equal to".
Except the POST method addition to fdsnws-station, all other additions are optional. Service implementations should return a WADL which appropriately advertises the optional features supported.
Jeremy Fee
I have a few comments after reading the updated specification:
- The "Required" column in the specification is a little ambiguous. Either
1) the parameter is required to be supported, which I suspect is the
intent, or 2) the parameter is required for each request.
- The event "text" format looks a lot like CSV, why not CSV?
And a couple general comments, not specific to the latest changes:
- The POST method circumvents any HTTP caching. I would suggest this be
optional, because the requests could be broken into smaller GET requests
that are cacheable.
- HTTP status 413 (Request Entity Too Large) refers to the _request body_
being too large, rather than the _response content_ being too large. It's
possible an application firewall inserted into a network between the
provider and client would generate this response even if the query would
otherwise be acceptable by the provider; particularly with POST requests.
One alternative would be 400 Bad Request when exceeding search limit, and
specified "limit" and "count" methods so an agent could determine the
result size was too large.
Thanks,
Jeremy
> _______________________________________________
> fdsn-wg3-products mailing list
> fdsn-wg3...@iris.washington.edu
> http://www.iris.washington.edu/mailman/listinfo/fdsn-wg3-products
>
>
Chad Trabant
Hi Jeremy,
Thanks for the feedback.
1) Regarding ambiguity in the "Required" column of the parameter table. Good point, a sentence can be added after each service-specific table to clarify that Required refers to support and not each request. Alternatively, the column can be called "Support" and the entries can be "Required" and "Optional". Any suggestions or preferences? Earlier in the spec is language strongly imply that this designation refers to support, but it is not very close to the table.
2) Regarding the text output in CSV. The 'text' output (event or station) can include commas in certain fields, comma-separated values are then mis-parsed. Schemes like quoted fields to contain or escape embedded commas can be a pain to parse, where as vertical bars are easily parsed in shell scripts, etc.
The general comments:
3) Regarding GET and HTTP caching. There are a number of use cases where a huge number of GET requests becomes a real limitation due to the overhead in the requests (thousands of requests) that the POST request capability resolves. In our experience with services so far there are very few requests that are exactly the same and therefore very limited use of HTTP caching at a level that is unaware of the data characteristics and therefore cannot subset data or recognize when to subset.
4) Regarding HTTP status 413 usage. I agree that the HTTP spec states this is only for the request entity, for FDSN service it should technically only be used to communicate the passing of a limit on the size of a POST body that a service interface will accept.
HTTP status 400 does not seem like a good fit either, it specifically means "The request could not be understood by the server due to malformed syntax.", which is not the case. At least a 413 has something to do with size, the message returned with the error can clarify that it refers to the size of the response.
There seems to be no HTTP status for "Response entity is too large". Are there any other suggestions for how to communicate this condition?
Chad
Wolfgang Lenhardt
I have forwarded your message to Ms.Jia and Mr. Horn who will deal
with this matter.
With kind regards, Wolf
At 23:37 19.09.2013, Fee, Jeremy wrote:
>Hello,
>
>I have a few comments after reading the updated specification:
>
>- The "Required" column in the specification is a little ambiguous.
>Â Either 1) the parameter is required to be supported, which I
>suspect is the intent, or 2) the parameter is required for each request.
>
>- The event "text" format looks a lot like CSV, why not CSV?
>
>
>And a couple general comments, not specific to the latest changes:
>
>- The POST method circumvents any HTTP caching. Â I would suggest
>this be optional, because the requests could be broken into smaller
>GET requests that are cacheable.
>
>- HTTP status 413 (Request Entity Too Large) refers to the _request
>body_ being too large, rather than the _response content_ being too
>large. Â It's possible an application firewall inserted into a
>network between the provider and client would generate this response
>even if the query would otherwise be acceptable by the provider;
>particularly with POST requests.
>
>One alternative would be 400 Bad Request when exceeding search
>limit, and specified "limit" and "count" methods so an agent could
>determine the result size was too large.
>
>
>Thanks,
>
>Jeremy
>
>
>
>On Wed, Sep 18, 2013 at 10:39 PM, Chad Trabant
><<mailto:ch...@iris.washington.edu>ch...@iris.washington.edu> wrote:
>
>Hello WG III members,
>
>As presented at the WG meeting in Gothenburg, Sweden, below is a
>list of FDSN web service additions proposed by the IRIS DMC. Â With
>the exception of the POST submission method to fdsnws-station, all
>of these changes are in production at the IRIS DMC.
>
>To provide further details and clarify how the changes would alter
>the specification, a draft version is attached with additions
>highlighted in red.
>
>We proposed that these additions to the 1.0 specification become
>version 1.1 of the specification.
>
>regards,
>Chad
>
>
>Proposed changes:
>
>* Designate optional versus required parameters for each service
>interface. Â Effectively defining a minimum level of functionality
>that clients can expect. Â The WADL return from each service
>interface specifies which of the optional parameters are supported.
>
>* Allow fdsnws-station to accept arbitrary selections via the POST
>method. Â Support for this method is required for all 1.1
>implementations of the service. Â The request is to be formatted
>similarly to the POST selection method supported by
>fdsnws-dataselect, where parameter key-value pairs are followed by a
>block of windowed channel requests.
>
>* Add optional 'format' parameter to all three interfaces. Â The
>default value for the fdsnws-station and fdsnws-event services is
>'xml' (returning FDSN StationXML and QuakeML respectively), an
>additional value of 'text' is defined as returning results in a
>simple ASCII format (illustrated in the specification). Â The
>default value for fdsnws-dataselect is 'miniseed'. Â Future
>expansion of the accepted values for this parameter are anticipated
>to return the results in alternate formats.
>
>* Add optional 'nodata' parameter to all three interfaces accepting
>values of '204' and '404' to control the HTTP status code returned
>when a successful request matched no data. Â The default value is '204'.
>
>* Add optional 'matchtimeseries' parameter to fdsnws-station to
>limit the results to matching data availability times.
>
>* Clarify definition of starttime and endtime parameters to
>explicitly include intersection with metadata epochs.
>
>* Clarify definition of minimum and maximum latitude and longitude
>to be inclusive by adding "or equal to".
>
>Except the POST method addition to fdsnws-station, all other
>additions are optional. Â Service implementations should return a
>WADL which appropriately advertises the optional features supported.
>
>
>
>
>
>
>
>
>
>_______________________________________________
>fdsn-wg3-products mailing list
><mailto:fdsn-wg3...@iris.washington.edu>fdsn-wg3...@iris.washington.edu
>http://www.iris.washington.edu/mailman/listinfo/fdsn-wg3-products
>
>
>_______________________________________________
>fdsn-wg3-products mailing list
>fdsn-wg3...@iris.washington.edu
>http://www.iris.washington.edu/mailman/listinfo/fdsn-wg3-products
Univ.-Doz.Dr. Wolfgang A. Lenhardt
Abteilung Geophysik/ Department of Geophysics
Bereich Daten, Methoden, Modelle/Division Data, Methods, Modeling
Zentralanstalt für Meteorologie und Geodynamik
1190 Wien, Hohe Warte 38
Tel.: +43 1 36026 2501
Mobile: +43 664 520 1492
Fax: +43 1 368 66 21
E-Mail: wolfgang...@zamg.ac.at
www.zamg.ac.at
Join us on facebook: http://www.facebook.com/zamg.at
P Please consider the environment before printing this email.
John Clinton
thanks for proposing these changes. We at the SED are happy to see these adopted in v1.1.
John
Hello WG III members,
regards,
Chad
Proposed changes:
_______________________________________________
fdsn-wg3-products mailing list
fdsn-wg3...@iris.washington.edu<mailto:fdsn-wg3...@iris.washington.edu>
http://www.iris.washington.edu/mailman/listinfo/fdsn-wg3-products
<FDSN-WS-Specifications-1.1draft1.pdf>
Marcelo Belentani de Bianchi
Thanks for the draft for the 1.1 version of the specification. Even that I
was not at the meeting in Sweden I hope I can contribute with some
comments for the document, more specifically on the ASCII format
definitions proposed. Beyond that I think that in general they are nice
additions to the specifications and my comments follows:
1) The StartTime/EndTime fields changes meaning depending on the value of
the level="value" parameter. We should add a notice for that behavior.
When level=network they are network operation times. When the
"level=station" is in effect station operation times.
2) I miss the restricted flag on the ASCII format. Should we add a column
on all levels to indicate that the network/station/channel is
open/restricted/some like in the XML version?
3) I could not understand what would be the contents for the "instrument"
field before looking into one of the given examples (it is not straight
forward from its name). I would suggest to rename it to
InstrumentDescription (or just description) ? Moreover I don't oppose
myself on having this field there, but I don't see any real usage, do you
have already something in mind or it is just an informative field? I
would still suggest that would be more useful to have a URI/URL where the
user could fetch the response for the channel inside instrument field than
the description itself. But for now, this is just one idea.
With my best regards,
Marcelo Bianchi
Em Mon, 23 Sep 2013 11:19:49 -0300, Clinton John <jcli...@sed.ethz.ch>
escreveu:
--
Dr. Marcelo Bianchi
Centro de Sismologia da USP
Instituto de Astronomia, Geofísica e Ciências Atmosféricas (IAG/USP)
Rua do matão, 1226 São Paulo/SP 05508-090 Brasil
Phone: +55 11 3091-4743 Cel: +55 15 99134-8532
http://moho.iag.usp.br/ http://www.iag.usp.br/
Chad Trabant
Dear Marcelo,
Thank you for your comments.
1) True, it is only implied that the time criteria are tested against different values depending on the level. This is not limited to the text output but also applies to the XML. I have no issue with adding a description of expected results. By the way, the IRIS DMC's implementation of the service always applies the time criteria to channel times to simplify the coding. The assumption is that channels "fit" within stations that "fit" within network time ranges. The consequence is that we cannot report networks or stations that do not have channels, so far this has not been a problem as we do not have any such metadata.
2) We have to strike the right balance between too many fields in the text output versus keeping it small and useful. In my opinion it is OK to leave restricted status out of the text format. A caller can still limit the response to include restricted stations or not using the includrestricted parameter, this fulfills the main use case of restricted status.
3) I have no problem changing the header name in the comment field. It is informative, but also designed to match the Instrument field of a SAC header. In the IRIS implementation this actually maps to the Network::Station::Channel::Sensor::Type field in StationXML which is the "instrument identifier" field of Blockette 52 in SEED. I propose we stick with something explicit such as "Sensor" or "InstrumentDescription" instead of anything generic, otherwise the field will not be very useful. I also do not think a URL is needed, a user already knows the URL to get more information, from the same service it requested the text from.
At IRIS we have used this text format for a couple of years now and found it to be very useful, taking the place of the XML format for many needs. Regarding changes to this format I propose we first try this text format and discover where it is lacking, in the future we can define other text output formats are needed. There are many use cases and no one simple set of metadata will solve them all.
best regards & obrigato,
Chad
Jeremy Fee
I'll reply inline:
On Thu, Sep 19, 2013 at 5:59 PM, Chad Trabant <ch...@iris.washington.edu>wrote:
>
> Hi Jeremy,
>
> Thanks for the feedback.
>
> 1) Regarding ambiguity in the "Required" column of the parameter table.
> Good point, a sentence can be added after each service-specific table to
> clarify that Required refers to support and not each request.
> Alternatively, the column can be called "Support" and the entries can be
> "Required" and "Optional". Any suggestions or preferences? Earlier in the
> spec is language strongly imply that this designation refers to support,
> but it is not very close to the table.
>
Support, with Required and Optional, would work, or 2 sections in each
table to separate required supported parameters from optionally supported
parameters.
2) Regarding the text output in CSV. The 'text' output (event or station)
> can include commas in certain fields, comma-separated values are then
> mis-parsed. Schemes like quoted fields to contain or escape embedded
> commas can be a pain to parse, where as vertical bars are easily parsed in
> shell scripts, etc.
>
I agree quoted CSV is tricky in shell scripts, although most languages have
parsers available that handle this.
It would also be useful for each of the columns in the format to be
specified, for example what is the difference between eventid and
contributorid?
The general comments:
>
> 3) Regarding GET and HTTP caching. There are a number of use cases where
> a huge number of GET requests becomes a real limitation due to the overhead
> in the requests (thousands of requests) that the POST request capability
> resolves. In our experience with services so far there are very few
> requests that are exactly the same and therefore very limited use of HTTP
> caching at a level that is unaware of the data characteristics and
> therefore cannot subset data or recognize when to subset.
>
Agreed that many users will submit unique requests. However, web
applications that consume a web service can make requests consistent across
many users, allowing regular caching to be effective and keeping resources
available for unique requests.
I'm reluctant to see POST method support _required_ for something that
would otherwise still be possible using GET.
Maybe station web services are used differently from event web services.
> 4) Regarding HTTP status 413 usage. I agree that the HTTP spec states
> this is only for the request entity, for FDSN service it should technically
> only be used to communicate the passing of a limit on the size of a POST
> body that a service interface will accept.
>
> HTTP status 400 does not seem like a good fit either, it specifically
> means "The request could not be understood by the server due to malformed
> syntax.", which is not the case. At least a 413 has something to do with
> size, the message returned with the error can clarify that it refers to the
> size of the response.
>
> There seems to be no HTTP status for "Response entity is too large". Are
> there any other suggestions for how to communicate this condition?
>
After re-reading, I agree that 400 is more appropriate for syntax errors.
Maybe 403 Forbidden: "the server understood the request, but is refusing to
fulfill it." Text describing how it exceeds the server limit would explain
why the request is forbidden.
Thanks,
Jeremy
Marcelo Belentani de Bianchi
Em Wed, 25 Sep 2013 23:44:36 -0300, Chad Trabant
<ch...@iris.washington.edu> escreveu:
> Dear Marcelo,
>
> Thank you for your comments.
>
> 1) True, it is only implied that the time criteria are tested against
> different values depending on the level. This is not limited to the
> text output but also applies to the XML. I have no issue with adding a
> description of expected results. By the way, the IRIS DMC's
> implementation of the service always applies the time criteria to
> channel times to simplify the coding. The assumption is that channels
> "fit" within stations that "fit" within network time ranges. The
> consequence is that we cannot report networks or stations that do not
> have channels, so far this has not been a problem as we do not have any
> such metadata.
That would be very nice to see better described so people could implement a
coherent service.
> 2) We have to strike the right balance between too many fields in the
> text output versus keeping it small and useful. In my opinion it is OK
> to leave restricted status out of the text format. A caller can still
> limit the response to include restricted stations or not using the
> includrestricted parameter, this fulfills the main use case of
> restricted status.
This I also agree, it is necessary to have this balance always. It is true
that most of the use cases should be covered by the flags of the service
itself
independent of the output format.
> 3) I have no problem changing the header name in the comment field. It
> is informative, but also designed to match the Instrument field of a SAC
> header. In the IRIS implementation this actually maps to the
> Network::Station::Channel::Sensor::Type field in StationXML which is the
> "instrument identifier" field of Blockette 52 in SEED. I propose we
> stick with something explicit such as "Sensor" or
> "InstrumentDescription" instead of anything generic, otherwise the field
> will not be very useful. I also do not think a URL is needed, a user
> already knows the URL to get more information, from the same service it
> requested the text from.
>
> At IRIS we have used this text format for a couple of years now and
> found it to be very useful, taking the place of the XML format for many
> needs. Regarding changes to this format I propose we first try this
> text format and discover where it is lacking, in the future we can
> define other text output formats are needed. There are many use cases
> and no one simple set of metadata will solve them all.
Ok. I would just suggest that the description of the fields to be added
or a just a note specifically for this column to avoid confusion
similar to the startime, endtime issue raised at point 1.
With my best regards,
Abraços,
Marcelo
Chad Trabant
Hi Jeremy,
My reply is inline below.
>
> Thanks for the feedback.
>
> 1) Regarding ambiguity in the "Required" column of the parameter table. Good point, a sentence can be added after each service-specific table to clarify that Required refers to support and not each request. Alternatively, the column can be called "Support" and the entries can be "Required" and "Optional". Any suggestions or preferences? Earlier in the spec is language strongly imply that this designation refers to support, but it is not very close to the table.
>
> Support, with Required and Optional, would work, or 2 sections in each table to separate required supported parameters from optionally supported parameters.
I'll work on adding the former.
> 2) Regarding the text output in CSV. The 'text' output (event or station) can include commas in certain fields, comma-separated values are then mis-parsed. Schemes like quoted fields to contain or escape embedded commas can be a pain to parse, where as vertical bars are easily parsed in shell scripts, etc.
>
> I agree quoted CSV is tricky in shell scripts, although most languages have parsers available that handle this.
>
> It would also be useful for each of the columns in the format to be specified, for example what is the difference between eventid and contributorid?
I will add brief descriptions of the columns for the text where they are ambiguous, many of them are obvious and map directly back to XML elements of the same data.
> The general comments:
>
> 3) Regarding GET and HTTP caching. There are a number of use cases where a huge number of GET requests becomes a real limitation due to the overhead in the requests (thousands of requests) that the POST request capability resolves. In our experience with services so far there are very few requests that are exactly the same and therefore very limited use of HTTP caching at a level that is unaware of the data characteristics and therefore cannot subset data or recognize when to subset.
>
> Agreed that many users will submit unique requests. However, web applications that consume a web service can make requests consistent across many users, allowing regular caching to be effective and keeping resources available for unique requests.
>
> I'm reluctant to see POST method support _required_ for something that would otherwise still be possible using GET.
>
> Maybe station web services are used differently from event web services.
Yes, much differently. The station service is, in many ways, the metadata equivalent of the dataselect service for time series where the POST method already exists and is required. This change brings these two, closely related, services more in line with each other.
>
> 4) Regarding HTTP status 413 usage. I agree that the HTTP spec states this is only for the request entity, for FDSN service it should technically only be used to communicate the passing of a limit on the size of a POST body that a service interface will accept.
>
> HTTP status 400 does not seem like a good fit either, it specifically means "The request could not be understood by the server due to malformed syntax.", which is not the case. At least a 413 has something to do with size, the message returned with the error can clarify that it refers to the size of the response.
>
> There seems to be no HTTP status for "Response entity is too large". Are there any other suggestions for how to communicate this condition?
>
> After re-reading, I agree that 400 is more appropriate for syntax errors.
>
> Maybe 403 Forbidden: "the server understood the request, but is refusing to fulfill it." Text describing how it exceeds the server limit would explain why the request is forbidden.
The dataselect service can support authenticated access to grant access to restricted data sets, in this scenario a 403 is commonly used to mean the credentials do no grant access to the selected data. Using a 403 for "too much data" would confuse the situation involving authenticated access.
Unless there are others that object to using a 413 in this way and there is another code identified that does not conflict with other meanings I suggest we keep it the way it was in version 1.0.
thanks,
Chad
Jeremy Fee
One additional suggestion, about the "format" parameter. Could that be
implemented via a pseudo file extension:
/fdsnws/event/1/query.txt
/fdsnws/event/1/query.xml
It would address some issues users report when they try to save a file and
browsers save it as "query".
It is also suggested by these standards, which we try to adhere to for our
own services:
https://github.com/WhiteHouse/api-standards
This is also somewhat supported by the "application.wadl" method. We added
a "application.json" method for our own service:
http://comcat.cr.usgs.gov/fdsnws/event/1/application.json
We can just implement this as an extension for our own service, but thought
you might want to consider it for the standard.
Thanks,
Jeremy
Chad Trabant
Hi Jeremy,
That's an interesting idea. Another option would be to allow output format selection based on the HTTP Accept header included in the request.
I suggest we consider these topics for the next version of the specification. In the mean time, we can learn from your usage of them.
Chad