A request for server developers

[Roberto De Almeida]

Hi, guys!

In 2006 I wrote an implementation of an OPeNDAP client in Javascript called jsdap (https://code.google.com/p/jsdap/). At the time Javascript was still a toy language and the XML HTTP Request (XHR) was unable of handling binary data, but I managed to hack a full client that worked in all major browsers (including IE by injecting vbscript!). And while it was written more as a proof-of-concept the client is actually used in some data portals like http://www.ifremer.fr/oceanotronPortal/. (A Node.js OPeNDAP server was also added 3 years ago.)

Fast forward 7 years and we now have a lot of new technologies on the table: a new XHR object with support for binary transfers, typed arrays and WebGL. I've been playing again with using Javascript as an OPeNDAP client, in particular to display real time information from OPeNDAP servers. I have set up a small OPeNDAP server on one of my VPS streaming the system load information:

  http://vps.dealmeida.net:5000/.dds

  http://vps.dealmeida.net:5000/.das

This is an infinite dataset (try "curl http://vps.dealmeida.net:5000/.asc"), and it will keep streaming the data at one record per second until the connection is broken. Keep in mind that this is a regular OPeNDAP Sequence, and nothing was changed in the specification to make this work. Nevertheless, I'm not aware of OPeNDAP clients that can access the stream other than the development version of Pydap.

On another machine I have a widget displaying the information on a real time graph: http://dealmeida.net/opendap-streaming/

You can see how everything was implemented on this Mercurial repository. The data is displayed by fetching the .dods response and parsing it. We still need a few hacks to do this, but only because the data is being streamed (Mozilla handles it nice; Chrome cannot stream binary data, so it still fetches it as string). Handling regular OPeNDAP datasets should be pretty straightforward with the new XHR, and I plan to rewrite jsdap as soon as I have some free time.

Now to my request: the only reason that the demo works -- having a page in one host displaying data from an OPeNDAP server on another -- is because I enabled CORS on Pydap. By default, now all DODS, DAS and DDS responses from Pydap have the following additional headers:

  Access-Control-Allow-Origin: *

  Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type

These headers (the first one, actually) allow the responses to be accessed through XHR from any host. As far as I know there is no downside in doing this. Even if your server use cookies for authenticating access to certain datasets the cookies will not be sent unless the Access-Control-Allow-Credentials header is set (and set to true), which would allow other sites to "steal" your data and download it by impersonating a logged user.

My request is that all OPeNDAP servers enable CORS from any host by default today, at least in the DODS, DAS and DDS responses; and if not by default, at least as an option. This way, by the time Javascript matures enough so that its performance on the browser becomes comparable to desktop applications we can start building rich web applications that use all the data available through OPeNDAP.

Some resources

About CORS: http://en.wikipedia.org/wiki/Cross-origin_resource_sharing / https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS

Security concerns: https://code.google.com/p/html5security/wiki/CrossOriginRequestSecurity

Thank you,

Rob

--

Roberto De Almeida, PhD

http://dealmeida.net/
:wq

[Dennis Heimbigner]

Perhaps more concretely, the thredds server
currently supports access controls such as
passwords and client-side keys. How would
CORS affect those?

=Dennis Heimbigner
Unidata

> repository<http://code.dealmeida.net/opendap-streaming/src/356dde80f6e55603c2ab7e581244015663504fda?at=demo>.

> The
> data is displayed by fetching the .dods response and parsing it. We still
> need a few hacks to do this, but only because the data is being streamed
> (Mozilla handles it nice; Chrome cannot stream binary data, so it still
> fetches it as string). Handling regular OPeNDAP datasets should be pretty
> straightforward with the new XHR, and I plan to rewrite jsdap as soon as I
> have some free time.
>

> *Now to my request:* the only reason that the demo works -- having a page

> in one host displaying data from an OPeNDAP server on another -- is because

> I enabled<http://code.dealmeida.net/pydap/commits/4c2d38b5822ba8f5f61e83bcb23230a2ca7e5da1>
> CORS <http://en.wikipedia.org/wiki/Cross-origin_resource_sharing> on Pydap.

> By default, now all DODS, DAS and DDS responses from Pydap have the
> following additional headers:
>
> Access-Control-Allow-Origin: *
> Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type
>
> These headers (the first one, actually) allow the responses to be accessed
> through XHR from any host. As far as I know there is no downside in doing
> this. Even if your server use cookies for authenticating access to certain

> datasets the cookies *will not* be sent unless

[Dennis Heimbigner]

I need a more detailed explanation of exactly
why this is needed. Unidata previously rejected
a request to add a cross-domain file to its server
and this seem too similar to me to be comfortable with it.
So, specifically, if we do not include these headers,
I would like to see an example of a request
and show what things fail.

=Dennis Heimbigner
Unidata

Roberto De Almeida wrote:

> repository<http://code.dealmeida.net/opendap-streaming/src/356dde80f6e55603c2ab7e581244015663504fda?at=demo>.

> The
> data is displayed by fetching the .dods response and parsing it. We still
> need a few hacks to do this, but only because the data is being streamed
> (Mozilla handles it nice; Chrome cannot stream binary data, so it still
> fetches it as string). Handling regular OPeNDAP datasets should be pretty
> straightforward with the new XHR, and I plan to rewrite jsdap as soon as I
> have some free time.
>

> *Now to my request:* the only reason that the demo works -- having a page

> in one host displaying data from an OPeNDAP server on another -- is because

> I enabled<http://code.dealmeida.net/pydap/commits/4c2d38b5822ba8f5f61e83bcb23230a2ca7e5da1>
> CORS <http://en.wikipedia.org/wiki/Cross-origin_resource_sharing> on Pydap.

> By default, now all DODS, DAS and DDS responses from Pydap have the
> following additional headers:
>
> Access-Control-Allow-Origin: *
> Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type
>
> These headers (the first one, actually) allow the responses to be accessed
> through XHR from any host. As far as I know there is no downside in doing
> this. Even if your server use cookies for authenticating access to certain

> datasets the cookies *will not* be sent unless

[Tom Kunicki]

I've created a maven project to create a CORS enable thredds war:

https://github.com/tkunicki-usgs/thredds-cors

CORS is useful for working around single origin issues in browser apps.  A CORS enabled server essentially tells the browser that it's ok to let code loaded from a different server utilize resources from the CORS enabled server.

Tom Kunicki

Center for Integrated Data Analytics

U.S. Geological Survey

8505 Research Way

Middleton, WI  53562

_______________________________________________
thredds mailing list
thr...@unidata.ucar.edu
For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/

[Dennis Heimbigner]

There are some things that worry me.

1. From http://en.wikipedia.org/wiki/Cross-origin_resource_sharing:
> To allow access from all domains, a server can send the following
> response header:
> Access-Control-Allow-Origin: *
> However, this might not be appropriate for situations in
> which security is a concern.

2. This page:
https://code.google.com/p/html5security/wiki/CrossOriginRequestSecurity
illustrates a number of security issues.

3. from https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS
> Important note: when responding to a credentialed request,
> server must specify a domain, and cannot use wild carding.
> The above example would fail if the header was wildcarded as:
> Access-Control-Allow-Origin: *.

I think this needs a lot more thinking out.

=Dennis Heimbigner
Unidata


Jon Blower wrote:
> Hi all,
>
> Regarding the CORS issue - as far as I know, Roberto is right and there is no issue for the server in enabling this. I
considered the same question for our ncWMS server software, although I haven't looked into it in detail. My tentative
conclusion is that this is a matter for deployers not developers. CORS can be enabled using a servlet filter that is installed
separately from THREDDS (e.g. [1]), so data providers can make a decision whether or not to enable this, and software providers
don't have to make the decision for them.
>
> Just my thoughts. It would of course be possible to bundle such a filter with THREDDS but if so, my inclination would be to
turn it off by default and publish a document about the implications of turning it on (i.e. a document written by someone who
knows more about this than I do!)
>
> Cheers,
> Jon
>
> [1] http://software.dzhuvinov.com/cors-filter-installation.html

[tom cook]

We had a similar discussion a few months back due to a request from a
3rd party developer who wanted to use Flash with our TDS. It used the
crossdomain.xml, which the differences are explained in the
code.google.com link Dennis sent. There was a similar concern about
the analog to the "access-control-allow-origin: *" setting. I've been
running our server with it, and haven't noticed anything out of the
ordinary. I do explicitly block IP addresses from certain regions with
tomcat already, but haven't seen any nefarious activity. (Of course
now that I said it aloud, I will be slammed by rogue requests and DoS
attacks).

[Roberto De Almeida]

That was fast, Tom! How does this work, does it add an option for turning on CORS on THREDDS?

As I was saying to Dennis, what I'm suggesting here is: Javascript is becoming a major player as a programming language, and if we want our data to be widely accessed and consumed by the new generation of applications in ways that we cannot anticipate, we should add these headers. If security is important we can explain to users how to configure authentication, how to setup SSL certificates, and how to turn off CORS (or restrict the hosts instead of using a wildcard).

I think this should be on by default (like I did in Pydap) because otherwise no one will turn it on, and all the data currently available via OPeNDAP will be inaccessible to Javascript, because this is not a policy that can be changed on the browser. But if we start doing it now, providers will eventually upgrade their servers, and maybe next year we'll have WebGL applications that can display data from any DAP server on the browser in 3D, and analyse it using asm.js.

--Rob

--
You received this message because you are subscribed to the Google Groups "pydap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydap+un...@googlegroups.com.
To post to this group, send email to py...@googlegroups.com.
Visit this group at http://groups.google.com/group/pydap?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dennis Heimbigner]

Turning this on by default is not a good idea.
As I indicated elsewhere, if data leaks, then unidata/opendap/...
are responsible and that makes us look really bad.

=Dennis Heimbigner
Unidata

Roberto De Almeida wrote:
> That was fast, Tom! How does this work, does it add an option for turning
> on CORS on THREDDS?
>
> As I was saying to Dennis, what I'm suggesting here is: Javascript is
> becoming a major player as a programming language, and if we want our data
> to be widely accessed and consumed by the new generation of applications in
> ways that we cannot anticipate, we should add these headers. If security is
> important we can explain to users how to configure authentication, how to
> setup SSL certificates, and how to turn off CORS (or restrict the hosts
> instead of using a wildcard).
>
> I think this should be on by default (like I did in Pydap) because
> otherwise no one will turn it on, and all the data currently available via
> OPeNDAP will be inaccessible to Javascript, because this is not a policy
> that can be changed on the browser. But if we start doing it now, providers
> will eventually upgrade their servers, and maybe next year we'll have WebGL
> applications that can display data from any DAP server on the browser in
> 3D, and analyse it using

> asm.js<http://ejohn.org/blog/asmjs-javascript-compile-target/>
> .

[Roy Mendelssohn - NOAA Federal]

I don't know enough to add anything technical to the discussion, but I know most of us these days operate in an environment of increased security, and anything that generally opens up holes is frowned upon ( we have started restricting our http ,ethos to GET, POST and HEAD for example).

I agree that more discussion of the security implications is needed, and the fact that someone has implemented it and hasn't seen anything nefarious won't past muster on security scans.

-roy

> You received this message because you are subscribed to the Google Groups "OPeNDAP Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to opendap-tech...@opendap.org.
> To post to this group, send email to openda...@opendap.org.
> Visit this group at http://groups.google.com/a/opendap.org/group/opendap-tech/?hl=en.
> For more options, visit https://groups.google.com/a/opendap.org/groups/opt_out.
>
>

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Men...@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

[Tom Kunicki]

Hi Rob,

The project uses a maven war overlay, With this, wars listed as dependencies are exploded and the contents of the project injected and then the war file is recreated. This project lists a jar file with a CORS filter implementation and then uses an XSLT to merge the contents of the web.xml files needed to configure thredds and the CORS filter. So it's just a project that injects things into the thredds war. We use this technique to customize and version control our thredds deployments (inject IOSPs, ncSOS, etc...). The CORS filter configuration is in src/main/webapps/WEB-INF/web.xml

I guess the point in putting up this project is to show how quickly it could be done if needed. As Jon mentioned earlier you can also enable CORS for an entire servlet container.

CORS support doesn't require any development on Unidata's side. I don't think it's desirable for the behavior to be on by default, this is something that the user deploying thredds should be enabling explicitly if desired.

Tom

[Dennis Heimbigner]

IMO, I think the CORS model is exactly backwards
for our purposes. What is really needed is a simple
way to tell the browser to trust specified domains
and to accept (possibly transitively) any cross-domain
references that come from a trusted site.

=Dennis Heimbigner
Unidata

[Lynnes, Christopher S. (GSFC-6102)]

On Apr 24, 2013, at 8:56 PM, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:

> I don't know enough to add anything technical to the discussion, but I know most of us these days operate in an environment of increased security, and anything that generally opens up holes is frowned upon ( we have started restricting our http ,ethos to GET, POST and HEAD for example).
>
> I agree that more discussion of the security implications is needed, and the fact that someone has implemented it and hasn't seen anything nefarious won't past muster on security scans.

To reinforce Roy's comments, from the perspective of a U.S. Govt. data provider, if the security implications are not crystal-clear, we have to treat it as a potential increase in risk. We already have an uphill battle in getting any OPeNDAP server approved at all, security-wise. As a result, for our purposes, we're not comfortable with server distributions that default to completely open CORS.

I'm generally sympathetic to the desire to enable this kind of functionality, but the history of computing is rife with cases where really cool functionality clashed with the potential security implications. I also recognize that not all data providers operate within our security constraints, so we will be interested in watching this unfold, with an emphasis on watching.

BUt note that absence of evidence doesn't mean it's safe, as Roy points out. The burden of proof for us is on proving from first principles that there is no increase in vulnerability due to open CORS. That is, given a malicious (inadvertent or otherwise) site, and malicious JS code using it for a cross-site scripting, will it be crystal clear that our CORS policy cannot be remotely associated with such an exploit? In the govt., any kind of guilt by association is almost as bad as actual guilt.

--
Dr. Christopher Lynnes, NASA/GSFC, ph: 301-614-5185

[Benno Blumenthal]

Note that it is particularly important, but we did what Rob requested a year or two ago, opening http://iridl.ldeo.columbia.edu/ to cross-browser access.  My understanding is that it means that a browser can use javascript from one site to access data on our site -- it does not change the security of the data, which is still dependent on the browser.  Other sites running our software will have the headers set as well.   The example that Dennis sent seems to be about using the Origin header to restrict access -- that is not real security, there must be lots of exploits around that.  I would like to better understand cookie-leakage, but a apriori cookies are supposed to be confined to the domain that set them.

Dr. M. Benno Blumenthal          be...@iri.columbia.edu
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000   (845) 680-4450

[Gerry Creager - NOAA Affiliate]

Benno,

You make a couple of good points about our missions of data sharing and interoperability, and the fact that we shouldn't fear a security audit. However, some of us DO live in an environment where this discussion has to occur, and I disagree that the conversation's been hijacked. 

I don't see the ORIGIN "security" example as particularly pertinent: As you point out, it's not really security, it's noise. However, that noise is the first place my IT Security Officer is likely to find when I ask about something like this... or the second if this conversation is already indexed on Google. 

I don't live in fear of the security audits, in fact, I welcome them, as an academic exercise in beating the system where rote methods are the anticipated outcome. But in able to beat the audit, we, especially in Federal service (or, as in my case, supporting it) have to completely understand the implications of a change like this. And, my first sense, upon reading Rob's request was, "No way: It opens too much." I may change my mind upon further reflection and study, but not yet

gerry

On Thu, Apr 25, 2013 at 9:08 AM, Benno Blumenthal <be...@iri.columbia.edu> wrote:

Sorry for the following bluntness, but this conversation has been hijacked.

As a community, we are in the business of cross-platform, cross-site sharing of data.  As Robert points out, the cross-origin blocking in javascript prevents that.  Note that cross-origin blocking is a half-hearted attempt to lock a site's data to a site's apps, not a fundemental security flaw.  In fact, javascript allows cross-site code imports, while blocking cross-site data and cross-site formatting (css) -- not a coherent security policy.

There are security implications in everything we do, there are people who figure these things out, really that is what a security audit is for.  Ignorant fear of an audit is not a legitimate argument, one can manage a site that way, but please don't impose it on the rest of us -- I already have a finance dept that behaves that way.

Bottom line is we need to make sure our data security is consistent with cross-origin data access -- which it probably is already.

--

Dr. M. Benno Blumenthal          be...@iri.columbia.edu
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000   (845) 680-4450

_______________________________________________
thredds mailing list
thr...@unidata.ucar.edu
For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/

--

Gerry Creager

NSSL/CIMMS

405.325.6371

++++++++++++++++++++++

“Big whorls have little whorls,

That feed on their velocity; 

And little whorls have lesser whorls, 

And so on to viscosity.” 

Lewis Fry Richardson (1881-1953)

[Benno Blumenthal]

Sorry for the following bluntness, but this conversation has been hijacked.

As a community, we are in the business of cross-platform, cross-site sharing of data.  As Robert points out, the cross-origin blocking in javascript prevents that.  Note that cross-origin blocking is a half-hearted attempt to lock a site's data to a site's apps, not a fundemental security flaw.  In fact, javascript allows cross-site code imports, while blocking cross-site data and cross-site formatting (css) -- not a coherent security policy.

There are security implications in everything we do, there are people who figure these things out, really that is what a security audit is for.  Ignorant fear of an audit is not a legitimate argument, one can manage a site that way, but please don't impose it on the rest of us -- I already have a finance dept that behaves that way.

Bottom line is we need to make sure our data security is consistent with cross-origin data access -- which it probably is already.

On Thu, Apr 25, 2013 at 7:54 AM, Lynnes, Christopher S. (GSFC-6102) <christophe...@nasa.gov> wrote:

--

[Roy Mendelssohn - NOAA Federal]

Hi Benno:

I am glad your environment is such that the concerns you state trump other concerns. I do not. We have to battle just to do the amount we do. Even more, if something we do does cause a security breach, then it makes it infinitely harder to do anything like it again (ask any one in the government what happened after the Hyrax problem a few years back, and how it affected all OpeNDAP servers in the Government, not just Hyrax).

Rob made an interesting and legitimate request. My first reaction was that request also had potential security issues, but as I stated originally, I am not certain I totally understand these potential issues, nor if there are ways to mitigate these issues. I do not see why asking people to comment if these concerns are legitimate or if there are ways to mitigate these issues hijacks the discussion. In fact, it does the opposite, by hopefully giving me enough information to be able to implement what Rob has requested in a way that will also pass muster with my IT folks.

Just look at Dennis''s response also. Clearly many people have reservations about this. If there are good solutions to these problems, other than just ignoring them, or if there is a reasonable literature that these are not issues, please point me to them. so that I have better information.

Thanks,

-Roy

[Benno Blumenthal]

So if we can persuade you of the value of Cross-Origin access, and me of the value of the discussion despite detours, we will be all set.

Seems like we need a security person to formulate an accurate statement of risk to counterbalance what one might find on the web...

Also, I have not been able to get cross-origin access to work in IE8, and I am not sure about more recent versions of IE.  So we need Microsoft to get its act together enough to upgrade users out of IE8, or fail utterly.

[Roberto De Almeida]

On Thu, Apr 25, 2013 at 7:46 AM, Benno Blumenthal <be...@iri.columbia.edu> wrote:

So if we can persuade you of the value of Cross-Origin access, and me of the value of the discussion despite detours, we will be all set.

Until something else appears, CORS will be required if we want the data to be fully accessible. i agree that making it on for all hosts by default is maybe not the best solution, but it definitely should be supported one way of another. 

 

Seems like we need a security person to formulate an accurate statement of risk to counterbalance what one might find on the web...

I found some more information here: http://enable-cors.org/. The site includes a list of APIs that supports CORS, including Amazon S3, Google, Github, Dropbox and Facebook: http://enable-cors.org/resources.html. And their message is "If you serve public content, please consider using CORS to open it up for universal JavaScript/browser access."

 

Also, I have not been able to get cross-origin access to work in IE8, and I am not sure about more recent versions of IE.  So we need Microsoft to get its act together enough to upgrade users out of IE8, or fail utterly.

IE 10 has native support, but in IE 8 & 9 you need to use XDomainRequest instead; here's how I do it:

  http://code.dealmeida.net/opendap-streaming/src/9a589c780e1dcdd14a9d8468ace27ca0a4f5027d/static/js/dap.js?at=default#cl-72

 

Although I haven't tested the code on IE yet.

--Rob

--