Configure the maximum length of a SPARQL query allowed in an HTTP GET request

[Yuanzhe Yang]

Hi there,

We are using Stardog to serve our data as an endpoint. Usually queries are sent to the endpoint through http get requests. However, during tests we discover that when the length of a query is more than 2380 bytes, the request will get stuck into an infinite waiting for the response until we terminate it manually. We tried the same query from the web interface of Stardog and the consequence is the same. Referring to the official documentation, we found such a configurable property:

"http.max.get.query.length: The length in characters of the longest SPARQL query that will be serialized via HTTP GET; queries longer are serialized via POST"

We assume it is a way to increase this limit. So we created a "stardog.properties" file with content "http.max.get.query.length = 4000", and restarted the server. But there is no difference, even if the length of the query is only 2381 bytes. We also tried to add other properties so we are sure that this file is successfully loaded.

Has anyone else encountered such type of problems before? Any suggestion or experience are highly appreciated. Thank you in advance.

Regards,

Yang

[Mike Grove]

If you issue the query against an empty database, does it hang?  What happens if you use the SNARL protocol?

If you can provide the query, and the data if possible, so that we can reproduce, that would be helpful.

Cheers,

Mike

 

Regards,

Yang

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Yuanzhe Yang]

Hi Mike,

Thank you very much for your reply.

I have tried a similar query on an empty database, but it still hangs.

I tried to test your suggestion with SNARL, but Gradle can't find the dependency "com.complexible.stardog.protocols.snarl:client:2.1.3". I also tried to find it from maven repository but I didn't make it.

The queries we use are ordinary queries with long filter expressions, you can try the following query on an empty database. I think it will prove my point:

SELECT (count(*) as ?total)   

WHERE { ?s ?p ?o. 

      FILTER ( ?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test"  ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test" ||?s = "test"||?s = "test"||?s = "test"||?s = "test")      

      }

When we have the default prefixes rdf, rdfs, xsd and owl, the aforementioned query will hang. But if you delete one ||?s="test" from the filter expression or just remove one prefix, you can receive the response quickly. That's why I assume the length of this query causes such type of problems. Moreover, when the query is hang, I also cannot see anything from the query management.

If you cannot reproduce the problem, you may try to make the query longer and see what will happen. Otherwise I will send you our data and see if it is our fault.

Thank you very much for your help in advance. Have a nice day.

Regards,

Yang

在 2015年1月2日星期五UTC+1下午1时46分37秒，Michael Grove写道：

[Mike Grove]

On Sun, Jan 4, 2015 at 1:34 PM, Yuanzhe Yang <yyz...@gmail.com> wrote:

Hi Mike,

Thank you very much for your reply.

I have tried a similar query on an empty database, but it still hangs.

Are you doing this via the web console?

 

I tried to test your suggestion with SNARL, but Gradle can't find the dependency "com.complexible.stardog.protocols.snarl:client:2.1.3". I also tried to find it from maven repository but I didn't make it.

You can just use the CLI which uses SNARL by default to test.

Cheers,

Mike

[Yuanzhe Yang]

Hi Mike,

Thank you very much.

In our application we use curl to query the database. When we discovered this problem, we also tried it from the web console provided by stardog, and the consequence is the same.

Today we tried your suggestion, it works when using the CLI!

So what should we do now?

Regards,

Yang

在 2015年1月5日星期一UTC+1下午3时12分39秒，Michael Grove写道：

[Mike Grove]

On Tue, Jan 6, 2015 at 6:18 AM, Yuanzhe Yang <yyz...@gmail.com> wrote:

Hi Mike,

Thank you very much.

In our application we use curl to query the database. When we discovered this problem, we also tried it from the web console provided by stardog, and the consequence is the same.

Today we tried your suggestion, it works when using the CLI! 

So what should we do now?

Make sure you're setting the max length correctly, it's a *client* parameter, so you must set it when you open the connection; you can pass the desired value for the parameter, which is in HttpOptions, to your ConnectionConfiguration.

Cheers,

Mike

[Evren Sirin]

If you are using curl just send the query as a post request. As Mike
said, the mentioned option is used only by the Stardog client to
decide which query method to use and would have no effect when you use
curl as the client.

Best,
Evren

[Yuanzhe Yang]

Hi Mike,

Thank you very much for your reply.

I am sorry that I didn't get your suggestion.

I don't know how to set the max length. According to the manual of Stardog, I need to add an expression in the "stardog.properties", which is what I have done already. Now the content of this file is 

http.max.get.query.length = 8000

In addition to this, I cannot find any place to configure any parameter.

Regards,

Yang

在 2015年1月6日星期二UTC+1下午2时29分27秒，Michael Grove写道：

[Mike Grove]

On Tue, Jan 6, 2015 at 2:42 PM, Yuanzhe Yang <yyz...@gmail.com> wrote:

Hi Mike,

Thank you very much for your reply.

I am sorry that I didn't get your suggestion.

I don't know how to set the max length. According to the manual of Stardog, I need to add an expression in the "stardog.properties", which is what I have done already.

That's a typo in the docs.

 

Now the content of this file is 

http.max.get.query.length = 8000

In addition to this, I cannot find any place to configure any parameter.

As I said, the option is part of HttpOptions, and you can set any value you want, like 8000, by setting that option on the ConnectionConfiguration you're using to create your connection.

[Yuanzhe Yang]

Hi Mike,

OK, I see. Let's me try. Thank you very much.

Regards,

Yang

在 2015年1月6日星期二UTC+1下午8时45分11秒，Michael Grove写道：

[Yuanzhe Yang]

Hi Evren,

Thank you for your kind help. So far I can only use post requests for lengthy queries. But it doesn't work for get requests. I am still looking for a way to use get. I will try Mike's suggestion first.

Regards,

Yang

在 2015年1月6日星期二UTC+1下午2时49分38秒，Evren Sirin写道：

[Mike Grove]

On Tue, Jan 6, 2015 at 2:53 PM, Yuanzhe Yang <yyz...@gmail.com> wrote:

Hi Evren,

Thank you for your kind help. So far I can only use post requests for lengthy queries. But it doesn't work for get requests. I am still looking for a way to use get. I will try Mike's suggestion first.

To clarify, you *must* use POST for requests for URLs beyond a certain length; there is a URI length limit imposed by the server, which is fairly normal.  The client parameter in HttpOptions is for telling the client what that limit is.

[Martynas Jusevičius]

I had the same problem on a different store. You should just use POST
if your queries are long. Here's the relevant spec:
http://www.w3.org/TR/2013/REC-sparql11-protocol-20130321/#query-operation

Martynas

> To unsubscribe from this group and stop receiving emails from it, send an
> email to stardog+u...@clarkparsia.com.

[Yuanzhe Yang]

Hi Mike,

Yes, I am aware of it. However my queries are more or less around 3kb so it's still far from the boundary. In addition, we also have some other side effects when we use post. That's why I am still looking for a way to use get. So thank you very much for your explaination.

Regards,

Yang

在 2015年1月6日星期二UTC+1下午8时58分15秒，Michael Grove写道：

[Yuanzhe Yang]

Hi Martynas,

Thank you very much for your kind reminder. I am aware of it but I thought 3kb is still far from that boundary.

Regards,

Yang

在 2015年1月7日星期三 UTC+1上午2:26:59，Martynas Jusevičius写道：

[Mike Grove]

On Thu, Jan 8, 2015 at 4:03 PM, Yuanzhe Yang <yyz...@gmail.com> wrote:

Hi Martynas,

Thank you very much for your kind reminder. I am aware of it but I thought 3kb is still far from that boundary.

I'm not sure what boundary you're referring to, but the server will not accept GET requests where the URL larger than 2k.  For requests larger than that, you should use POST.

[Yuanzhe Yang]

Hi Mike,

OK I see. Maybe it's my mistake. Because I have impression that most of web servers supports at least 8kb URL. I will think about it carefully. Thank you very much for all your replies and suggestions. Have a nice day.

Regards,

Yang

在 2015年1月9日星期五 UTC+1上午12:41:02，Michael Grove写道：