Help with performance
Mark
We are looking at the performance of our app using Empire. I can send
you details but let me tell you what we have observed and you may have
a simple solution.
I have a hosted version of Sesame and am using Empire in an
application on an adjacent machine. We are seeing that with a search
query that returns 400 results the performance is quite slow. By slow
I mean it is taking a minute for the Empire JPA logic to return
results to our application. Sesame is responding in 200ms. Doing a
TCP dump we observe that Empire is first doing a sparql request to get
a list of identifiers and then N sparql requests to download the
details of each resultant object.
Is this the normal behaviour or is it likely that we have
misconfigured our models' annotations?
Regards Mark
Mike Grove
No, that's the designed behavior. While doing that many requests is
not ideal, it's not straightforward to write a query that will pull
down everything you need to create all of those objects in one go.
This is especially true given how many optional's you'd need to allow
for properties to not be present on any given individual, as well as
using unions to pull in information about child objects who are
fetched eagerly -- which may require more optionals, more unions, etc.
So you can see how difficult a query that becomes when trying to do
it all in one shot.
I had a prototype awhile ago (pre eager fetching) that did it all in
one shot and the performance was surprisingly bad given the reduced IO
overhead of doing only a single query. The database just got killed
by how difficult the query is to evaluate. For beans w/ no child
objects an all the properties were non-optional, you did get better
performance; but that's mostly because you don't have optionals in the
query, and you don't have unions to pull in child objects.
This was also pre-SPARQL 1.1. Getting the child objects w/ sub
queries would be a better approach, and databases are generally faster
now than they were two years ago when that code was originally
written. Though I suspect the best implementation for this is
somewhere in the middle of loading the top level objects in one go, or
in chunks, and then doing separate queries for eagerly fetched child
objects.
I have a lot on my plate right now with the Stardog release, but I'd
generally be interested in helping with any modifications to this
code. I'll happily accept a patch the addresses the issue, but I can
try and help where I can with the work as well.
Cheers,
Mike
>
>
> Regards Mark
Mark
Many thanks for the reply. It is useful to know we are not making a
mistake before looking further into the issue.
I am going to have to find a solution as it is not going to be
uncommon for us to make a request returning 400-1000 items. We don't
have child objects but having a lazy child annotation might be
interesting to look into. Then the developer could optimize the
lookup type depending on the application and the complexity of the
models. If you like, we could provide an implementation. I'd rather
do it in a way that we could give to you. Then, we don't have to work
on a customised version of Empire moving forwards.
Regards Mark
On Feb 17, 12:57 pm, Mike Grove <m...@clarkparsia.com> wrote:
Mike Grove
> Hi,
>
> Many thanks for the reply. It is useful to know we are not making a
> mistake before looking further into the issue.
>
> I am going to have to find a solution as it is not going to be
> uncommon for us to make a request returning 400-1000 items. We don't
> have child objects but having a lazy child annotation might be
> interesting to look into.
You can use any of the JPA annotations OneToOne, ManyToOne, OneToMany,
ManyToMany alongside RdfProperty to specify a fetch type of eager or
lazy.
Then the developer could optimize the
> lookup type depending on the application and the complexity of the
> models.
Yes, it might be nice if the algorithm to put together the beans could
do something smart wrt those annotations to try and optimize for what
the user is doing.
Perhaps also adding a @Required annotation might be useful, the query
generator could know that it should not use optionals for those BGP's
which would help make the query eval faster.
>If you like, we could provide an implementation. I'd rather
> do it in a way that we could give to you. Then, we don't have to work
> on a customised version of Empire moving forwards.
>
That sounds great. I could not find the old prototype I did for this,
but if you have any questions about implementation or other vagaries
of Empire, just give a shout.
Cheers,
Mike