sparql update limit ?

[Samuel Morello]

Hi,


I am trying to do a simple insert like this but i think i have limit problem.
i use the sparql http server and do a query like this :


prefix frm: <http://example.com/test/>
INSERT {
GRAPH <http://mygraph.com> {
?s frm:categoryURI <http://testuri.com/e>
}
} WHERE {
GRAPH <http://mygraph.com> {
?s frm:category "3"
}
}



under 1 100 000 triples everything works perfectly but once above this query has no effect.
The configuration of 4store is with --enable-dedup-insert
I did try to had &soft-limit=-1 at the uri level and in the body of the request but with no success.
Any hint ?

Thanks,

Samuel

[Samuel Morello]

Seems that my problem is not linked to the amount of triples.
But i do progressive imports
i can do this insert until a certain time.
It seems that when there is too much data i can not do it any more ?
Strange...

Samuel

[Steve Harris]

Hm, no that's an interesting one.

Are all the triples of the form ?s frm:category "3"? If not, how many match that?

I suspect you're the first person to try an UPDATE WHERE with that many consequences!

If you're running from HEAD (you should be got Update support, it's better in head than release), can you try uncommenting DEBUG_MERGE in debug.h, and rebuilding?

That should spit out debug that will tell us what's going on.

It probably is related to the soft limit - I would guess that there WHERE part has around 1000 results.

- Steve

> --
> You received this message because you are subscribed to the Google Groups "4store-support" group.
> To post to this group, send email to 4store-...@googlegroups.com.
> To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/4store-support?hl=en.
>

--
Steve Harris, CTO
Garlik, a part of Experian
1-3 Halford Road, Richmond, TW10 6AW, UK
+44 20 8439 8203 http://www.garlik.com/
Registered in England and Wales 653331 VAT # 887 1335 93
Registered office: Landmark House, Experian Way, Nottingham, Notts, NG80 1ZZ

[Samuel Morello]

I use the last version now of store now ;-)

For the WHERE part, using it with a select return around 40000 triples for some of the inserts

I'll reinstall with debug support and come back with the logs

Thanks,

Samuel

> > To post to this group, send email to 4store-...@googlegroups.com (mailto:4store-...@googlegroups.com).
> > To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com (mailto:4store-suppor...@googlegroups.com).

> > For more options, visit this group at http://groups.google.com/group/4store-support?hl=en.
>
>
>
> --
> Steve Harris, CTO
> Garlik, a part of Experian
> 1-3 Halford Road, Richmond, TW10 6AW, UK
> +44 20 8439 8203 http://www.garlik.com/
> Registered in England and Wales 653331 VAT # 887 1335 93
> Registered office: Landmark House, Experian Way, Nottingham, Notts, NG80 1ZZ
>

> --
> You received this message because you are subscribed to the Google Groups "4store-support" group.

> To post to this group, send email to 4store-...@googlegroups.com (mailto:4store-...@googlegroups.com).
> To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com (mailto:4store-suppor...@googlegroups.com).

[Samuel Morello]

When it is around 10000 results every things seems to work perfectly

[Steve Harris]

Interesting - how many segments?

- Steve

> To post to this group, send email to 4store-...@googlegroups.com.
> To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com.

[Samuel Morello]

I don't know how to check the number of segments ?
This query don't work in this case

Here is the output with debug :



ouva:frm ouvasam$ 4s-update frm "PREFIX frm: <http://ec.europa.eu/open-data/plants/forestry/> INSERT { GRAPH frm:import { ?s frm:categoryURI <http://test.com/z2> } } WHERE { ?s a frm:ApprovForest ; frm:category '3' . } "
ADD triple(variable(s), uri<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>, uri<http://ec.europa.eu/open-data/plants/forestry/ApprovForest>) to B1
ADD triple(variable(s), uri<http://ec.europa.eu/open-data/plants/forestry/category>, string("3")) to B1
Merge B1 to B0

After compact:
B0, join NONE, parent B0
P0 triple(variable(s), uri<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>, uri<http://ec.europa.eu/open-data/plants/forestry/ApprovForest>)
P1 triple(variable(s), uri<http://ec.europa.eu/open-data/plants/forestry/category>, string("3"))

Processing B0, parent is B0
execute 0/2: triple(variable(s), uri<http://ec.europa.eu/open-data/plants/forestry/category>, string("3"))
triple(variable(s), uri<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>, uri<http://ec.europa.eu/open-data/plants/forestry/ApprovForest>) DISTINCT
nnnnr (_,?,_[f777ee569472ae88 e95e3998e1fb6613],_[392ea3b704e1c578 f6a1a7550a503fe3]) -> 25532
sort took 0.005575 seconds
uniq took 0.000678s (25532->25532 rows)
s
row p-nb A00 D-1
0 c00014a98351c46e
1 c000d2d9b0b0b948
2 c0014b195049819c
3 c001b39de3b816c9
4 c002606f3bc69b43
5 c004119cceb41f47
6 c0045354a589bd0a
7 c004b7685f37abf0
8 c004d4fa361c2c4f
9 c004f1b61564f60c
10 c0052b2d6be484ca
11 c0059b57ac5324c1
12 c005b348e0b858b3
13 c00641b82369b0c0
14 c006b3c77628b5f1
15 c006dc8355301fb1
16 c0086c2f5daa01bd
17 c0092b4059ba377b
18 c00ab6df0e878ca0
19 c00c95244e5314d5
20 c00c9c9df5c827c3
21 c00cdb2933df5170
...
25530 fffdfb0e1fba484e
25531 ffffbca717f94ba0
append all, result:
s
row p-nb A00 D-1
0 c00014a98351c46e
1 c000d2d9b0b0b948
2 c0014b195049819c
3 c001b39de3b816c9
4 c002606f3bc69b43
5 c004119cceb41f47
6 c0045354a589bd0a
7 c004b7685f37abf0
8 c004d4fa361c2c4f
9 c004f1b61564f60c
10 c0052b2d6be484ca
11 c0059b57ac5324c1
12 c005b348e0b858b3
13 c00641b82369b0c0
14 c006b3c77628b5f1
15 c006dc8355301fb1
16 c0086c2f5daa01bd
17 c0092b4059ba377b
18 c00ab6df0e878ca0
19 c00c95244e5314d5
20 c00c9c9df5c827c3
21 c00cdb2933df5170
...
25530 fffdfb0e1fba484e
25531 ffffbca717f94ba0
25532 bindings (25532)
Processing B1, parent is B0
skipping B1, no bindings
skipping B1, no bindings

[Steve Harris]

Thanks for the debug trace, all looks fine.

The default for segments is 2, so I guess you have 2, but it's nothing that obvious. This is running on a single machine is it?

Can you check in /var/log/messages (if it's a Linux box, maybe /var/log/system.log if it's not) to see if there's any errors in there? Seem a bit odd that it would suddenly stop working as it got bigger.

Cheers,
Steve

> To post to this group, send email to 4store-...@googlegroups.com.
> To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com.

[Samuel Morello]

Hi,

I tried today without success to split the inserts using LIMIT OFFSET
It work as always when there a little set of data but not when too many

I also check the logs but nothing inside

I don't know if there is another solution ?
Thanks,

Samuel

[Steve Harris]

I don't think so. Someone that understands the guts of the system needs to reproduce it, and find what the bug is.

> To post to this group, send email to 4store-...@googlegroups.com.
> To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com.

[Samuel Morello]

Something may be interesting ?

At first i tried to import all the data in a unique graph and then do insert
This does not work when i have too much data

Using 4s-import to import data, create a graph by filename
With this, my inserts work perfectly ???

Thanks,

Samuel

[Steve Harris]

Aaaah, that is interesting.

That must mean you ended up with a different INSERT syntax?

The one before had an explicit GRAPH in it.

- Steve

> To post to this group, send email to 4store-...@googlegroups.com.
> To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com.

[Samuel Morello]

Every tests i've done are inconsistents...

For now when i have the full dataset imported in multiple graph using 4s-import

If i do an insert like this


INSERT {
?s <http://test.com/test> <http://test.com/zzz>
} WHERE {
?s <http://ec.europa.eu/open-data/plants/forestry/category> '3'
}


I have no results if i do this :

SELECT ?s ?o
where {
?s ?o <http://test.com/zzz>
}

but if i do this :

SELECT
?s ?p ?o
where {
GRAPH <default:> {
?s ?p ?o
}
}


i see all the inserted triples ...

I am completely lost

[Samuel Morello]

For info if i do

SELECT
?s ?p
where {
graph <default:> {
?s ?p <http://test.com/zzz>
}
} LIMIT 100


this query return nothing

but

SELECT
?s ?p ?o
where {

graph <default:> {
?s ?p ?o
}
} LIMIT 100



return triples like this :
<result>
<binding name="s"><uri>http://ec.europa.eu/open-data/plants/forestry/approvforest/8960</uri></binding>
<binding name="p"><uri>http://test.com/test</uri></binding>
<binding name="o"><uri>http://test.com/zzz</uri></binding>
</result>

[Samuel Morello]

I can now confirm that there is something with the graph.
If i insert all the data in a specific graph then i have the following :



if i do a select like this :

SELECT
?s ?p ?o
WHERE {
GRAPH <http://test.com/test> {
?s ?p ?o
}
}

I get all the triple like this one (on 3 lines ) :

http://ec.europa.eu/open-data/plants/forestry/approvforest/12609
http://ec.europa.eu/open-data/plants/forestry/purposeURI
http://ec.europa.eu/open-data/plants/forestry/datadictionary/purposes#other-specific-purpose

If i do this :

SELECT
?p ?o
WHERE {
GRAPH <http://test.com/test> {
<http://ec.europa.eu/open-data/plants/forestry/approvforest/12609> ?p ?o
}
}

i get the follwoing result :
1.
http://ec.europa.eu/open-data/plants/forestry/categoryURI
http://ec.europa.eu/open-data/plants/forestry/datadictionary/categories#source-identified
2.
http://ec.europa.eu/open-data/plants/forestry/materialtypeURI
http://ec.europa.eu/open-data/plants/forestry/datadictionary/material-types#seed-source

These triples have also been inserted and visible with a normal select but i have nothing with what i found on the first request (i have material-type and categories but no purposes)

If i reduce the number of file to be import i have all the triples as expected but if i put more files i have sometimes only one (e.g categories) and sometimes nothing.

But all the triples are inserted correctly if i check with the first query

SELECT
?s ?p ?o
WHERE {
GRAPH <http://test.com/test> {
?s ?p ?o
}
}



I did try to go with the 1.1.4 version but same results.

Thanks,

Samuel

[Samuel Morello]

I did it another way :

I first SELECT the resources like this

SELECT ?af
WHERE {
?af a frm:ApprovForest ;
frm:category '1'
}


then i create a turtle document that i send to 4store
In this way everything works perfectly, and that is good for me ;-)

Many thanks,

Samuel

[Steve Harris]

Aha!

That's quite informative.

It means that the data was added some some indexes, and not others.

The most likely cause is a crash during the INSERT - are you sure there's nothing in /var/log/messages when you run the big INSERT?

Cheers,
Steve

> To post to this group, send email to 4store-...@googlegroups.com.
> To unsubscribe from this group, send email to 4store-suppor...@googlegroups.com.