olfs returns 400 for some requests

[Golpayegani, Navid (GSFC-6190)]

Hi,

  Some of our users are reporting that some requests to our opendap are returning a 400 to them. For example this link returns a 400 error:

https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:359]

 

We consistently get a 400 error on that link but if you remove the XDim parameter the link works. Additionally, the response seems to be slightly different. For example with curl we simply receive a 400 HTTP code with no data while with Chrome browser we get  a 400 code along with message:

Error {

    code = 400;

    message = "libdap exception building response: error_code = 1005: Failed to get values as ascii: Constraint expression parse error: syntax error";

}

 

Since this is a live system with a lot of requests it’s hard to associate error messages from tomcat with a particular request but I think when we make this request we receive this error message from tomcat:

 

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | 24-May-2018 21:55:17.313 INFO [http-nio-8080-exec-4] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    |  Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    |  java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:479)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:687)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

OPeNDAP_olfs.1.q7u7ndf72w94@dockr002    | at java.lang.Thread.run(Thread.java:748)

 

I partially suspect that the log is unrelated to this error.

 

We’re running olfs version 1.17.0, libdap and bes version 3.19.1-1.

 

Any help?

 

Navid

[Nathan Potter]

Hi Navid,

I looked at this and noticed a couple of things:

1) Any use of the square bracket notation in the query string causes the "400 BAD REQUEST” error, regardless of what is requested:

curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:1359]
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0]
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?Solar_Zenith_Mean[0]

curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dmr?Solar_Zenith_Mean[0]
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dds?Solar_Zenith_Mean[0]
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dods?Solar_Zenith_Mean[0]
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dap?Solar_Zenith_Mean[0]

2) Dropping the square brackets entirely makes everything cool:

curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?Solar_Zenith_Mean

curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dmr?Solar_Zenith_Mean
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dds?Solar_Zenith_Mean
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dods?Solar_Zenith_Mean
curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.dap?Solar_Zenith_Mean

3) And replacing the square brackets with their equivalent HTTP escape value works too:

curl -i -g https://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim%5B0%5D

Conclusions:

At this point my thinking is that since you are the only person reporting this issue with Hyrax-1.14.0 (which is pretty widely deployed) I am thinking that there must be some kind of security layer in your cloud deployment that is getting rankled but the square brackets in the constraint. I think this especially because none headers returned with the 400 error:

HTTP/1.1 400
Server: openresty
Date: Thu, 24 May 2018 22:40:53 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: tisSession=a0a52f7f-9e77-4c3e-82b1-5faa85f0c0cb; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains

Are generated from Hyrax. In other words, I don’t think the request is getting to Hyrax.

The headers make me think that your deployment is in the OpenResty cloud, (possibly using NGINX), which is an environment with which I have zero experience.

Can you tell me about the environment you are running in?

Possibilities:

- Some upstream component (NGINX?) is returning the error or munging the URL
- There are new security rules in place that we have not yet encountered that may require that we adjust the Data Request Form to do the escaping on the way out.

I don’t think the stuff in your error log is noise. I think that’s probably related.

= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317

[Golpayegani, Navid (GSFC-6190)]

Hi Nathan,
Thanks for the quick reply. Sorry I should have included that information as well. We're running opendap in a docker environment. And you're right we've been running 1.14.0 in docker for months now without any issues. For an unknown reason this started last week.

We are running a 3 container setup with 1 container being bes and one container being olfs. The third container is nginx (openresty) acting as a load balancer which proxies requests to olfs if the endpoint is /opendap. The opendap endpoint in nginx is very simple:

location /opendap{
proxy_pass http://opendap/opendap;
}

The olfs container is setup to expose the internal port 8080 as an external port 3001. I tested if it's our nginx server by directly connecting to the container and bypassing nginx. I still get the same result:
# curl -v -g 'http://dockr002:3001/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:1359]'
* About to connect() to dockr002 port 3001 (#0)
* Trying 192.168.55.72...
* Connected to dockr002 (192.168.55.72) port 3001 (#0)
> GET /opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:1359] HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dockr002:3001
> Accept: */*
>
< HTTP/1.1 400
< Date: Fri, 25 May 2018 14:42:45 GMT
< Connection: close
<
* Closing connection 0

Also to make sure it's not the docker firewall rules causing problems I connected directly into the docker container and ran the command directly against the tomcat running inside the docker container and the problem is still there:
$ curl -v -g 'http://localhost:8080/opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:1359]'
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /opendap/allData/6/MOD08_D3/2016/122/MOD08_D3.A2016122.006.2016123095613.hdf.ascii?XDim[0:1:1359] HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 400
< Date: Fri, 25 May 2018 14:44:16 GMT
< Connection: close
<
* Curl_http_done: called premature == 0
* Closing connection 0

And you're right as soon as I remove the brackets or escape them the error seems to go away.

Navid

[Nathan Potter]

Hi Navid,

So can you tell me more about the docker containers?

What Tomcat version?

What OS for the OLFS container?

What OS for the BES container?

Thanks,

N

[Golpayegani, Navid (GSFC-6190)]

The olfs container is based off of the tomcat:8-jre8 image. The exact tomcat version is 8.5.31.0. The base OS is Debian 9.4.

The bes container is based off of centos:7 image. The bes image is identical to https://github.com/OPENDAP/hyrax-docker/blob/master/hyrax-1.14.0/besd/Dockerfile

Navid

[Nathan Potter]

Hi Navid,

I understand the problem. I can reproduce it on my dev system, and I think I have a work around for you until the next release arrives.

I have opened a ticket in our JIRA system for this: https://opendap.atlassian.net/browse/HYRAX-767


The Work Around:

In the Tomcat configuration file: $CATALINA_HOME/conf/server.xml add the following attribute to your Connector element:

relaxedQueryChars="&lt;&gt;[\]{|}"

So you end up with something like:

<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
relaxedQueryChars="&lt;&gt;[\]{|}"
/>

Obviously you may have a very different Connector instance, but just poke that one attribute in there and you should be good to go.

Here’s the relevant Tomcat doc: https://tomcat.apache.org/tomcat-8.5-doc/config/http.html

I think the fix will make it into our next release. We are in discussion about a date for that, but have not set one yet.

I hope this gets you up and running!


Sincerely.,

Nathan

[Golpayegani, Navid (GSFC-6190)]

Hi Nathan,
Thank you. That was indeed the problem and it fixed it.

Just out of curiosity. Are you aware if there's any reason why that would suddenly stop working for us or why nobody else has encountered this before? Our server.xml file is version controlled and I just verified it was last modified 4 months ago so it's a bit of a mystery why it would suddenly break.

Thanks again for debugging the problem for us,

Navid

[Nathan Potter]

Hi Navid,

My thought is that you might have a look at the Dockerfile used to build the OLFS container. Near/At the top there will typically be a “FROM” statement like this:

FROM unidata/tomcat-docker:8

or

FROM tomcat:8.5-jre8

Every time the docker container is built, the FROM statement is evaluated.

If the underlying reference image has been updated it is pulled into the build.

Since both of the above examples reference a sort of “latest” release image as opposed to a specific version like this:

FROM tomcat:8.5.31-jre8

This means that as Apache rolls out new code under Tomcat-8.x (first example) or Tomcat-8.5.x (2nd example) their containers get updated and so do the dependent containers.

This is a great plan for security patches, which is exactly what we are encountering here. But, the sudden behavioral changes can be disconcerting to say the least!

I think in general there has been a big push to be more strict about URL encodings as these unescaped characters have been used to force arbitrary code injection into poorly implemented servers. Apparently the front-line step to mitigate this kind of attack is to utilize correct URL encoding in requests and to reject requests that are not correctly encoded.

https://security.elarlang.eu/request-uri-query-string-and-url-encoding.html
https://perishablepress.com/stop-using-unsafe-characters-in-urls/

I would be interested to know if the Dockerfile in question references a specific Tomcat version or some variant of “latest” in the FROM line.


Does all that make sense to you?



Sincerely,

Nathan

[gaj...@gmail.com]

Hi Nathan,

I've just come across the same problem with thredds-docker (5.0-beta5) and opendap subsettng.  I'll pass the solution to them.

Setting relaxedQueryChars is a pragmatic solution for now.

Do you think it is a sustainable solution or will opendap need to change?

Note, I could also urlencode the opendap query before giving it to my client (libcurl or web browser in testing) and any client library access could potentially be modified to use this solution, but clients passing [X:Y] is likely to hang around for a long time and may impact opendap uptake.

Gareth

[Nathan Potter]

Hi Gareth,

The encoding of the URL is the requesting clients responsibility. I agree that setting relaxedQueryChars is only a band-aid to a larger issue. In the latest Hyrax release I have modified all of the Data Request Form pages (which are basically simple DAP request client applications) so that that when people click the buttons at the top of the form to request data the URLs are correctly encoded before the request is sent.

I have been monitoring a thread from TDS support that shows that they have been trashing pretty hard to sort this out for the netcdf-java library (which is a DAP client). I suspect (but have not seen emails to the effect) that they are also addressing the netcdf-c library (also a DAP client).

Additionally, some browsers will automatically encode for you when type unencoded URLs into the address bar (that's good!), others will not (that's not so good...)

As far as curl goes you can have it do the encoding for you by using the --data-urlencode parameter.

See: https://ec.haxx.se/http-post.html

Here is a curl command file from our regression tests that asks for a DAP2 dataset with an array constraint:

url = http://localhost:8080/opendap/data/hdf4/S2000415.HDF.gz.dods
--data-urlencode "NSCAT_Rev_20_Num_Beam_34[0:10:457][0:4:23]"
-G
-s

And here is one that does the DAP4 business:

url = http://localhost:8080/opendap/data/hdf4/S2000415.HDF.gz.dap
--data-urlencode "dap4.ce=NSCAT_Rev_20_Num_Beam_34[0:10:457][0:4:23]"
-G
-s

Note that each query parameter needs it's own --data-urlencode statement. So when you do this you must decompose the query into each of the statements between the "&" and place each them in a --data-urlencode param.

Does that all make sense?


Sincerely,

Nathan

[Gareth Williams]

All makes beautiful sense. I am glad to hear you are well on top of this.

Some of my tests are run with ansible. It also has inbuilt capability to urlencode, but you have to explicitly use it (as a jinja2 filter).

Gareth