stardog returns no results for queries with a large numbe of default and name graph uri restrictions

[Ric Roberts]

Hi.

We're using Sesame's "SparqlRepository" client to make requests to a stardog database over http. This posts to the stardog query endpoint with a 'x-www-form-encoded' body containing content like:

queryLn=SPARQL&query=<query>&infer=false&default-graph-uri=<default-graph>&named-graph-uri=<name-graph>

We've found that with the query below, when using 1019 named and default graph restrictions, stardog returns a full set of correct results, but when there are 1020, stardog returns no results (and doesn't report an error).

SELECT * 

WHERE { 

  GRAPH ?g {

    ?s ?p ?o .

  }

}

Some of our queries use a lot of named graph and default graph restrictions (thousands).

This problem seems to also occur with any query uses a graph clause.

Ric.

[Michael Grove]

What I suspect is happening is due to Netty's QueryStringDecoder [1], which is used to decode the incoming HTTP request.  By default, it limits the maximum number of parameters (1024) in the query and is their workaround for protection against the HashDOS attack.  Once you go over the threshold, it just ignores the rest of the parameters.  This explains why there's a deterministic limit where the query does and does not work and why there's no error on the server.  The results are incomplete because the active graph constructed from the request is incomplete.

The threshold is configurable in the decoder, so we can expose this and you can ratchet it up to whatever is appropriate for your use case.  That's probably safe in your case since the queries are not coming from the wild, and instead are coming from a known good source which is unlikely to try and utilize the vulnerability.

I created a ticket for this (#2392).  We'll work on this for the next release.

Cheers,

Mike

[1] http://netty.io/4.0/api/io/netty/handler/codec/http/QueryStringDecoder.html

 

Ric.

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Ric Roberts]

I was just about to add an update to this report to say that even with fewer named and default graphs (but still more than 1024 params in total), then it gives some unusual behaviour (ignoring some named graphs). So your analysis would make sense. Thanks for this - a configurable limit would be great!

Cheers!

Ric

[Ric Roberts]

Hi - I've noticed that this is marked as fixed in the 3.1.3 release. Is the parameter limit configurable, or just unbounded now? 

Cheers,

RIc.

[Michael Grove]

On Wed, Jul 15, 2015 at 3:00 PM, Ric Roberts <r...@swirrl.com> wrote:

Hi - I've noticed that this is marked as fixed in the 3.1.3 release. Is the parameter limit configurable, or just unbounded now? 

It's configurable.  The default is still 1024, so no change from the previous behavior.  But you can specify the property `http.max.request.parameters` in your `stardog.properties` file and provide any value smaller than `Integer.MAX_VALUE` that you need.

If starting the server programmatically, you can set it via `ServerBuilder#set` using `HttpServerOptions#MAX_REQUEST_PARAMETERS`

Cheers,

Mike

[Ric Roberts]

Great - thanks. I'll try this out.