Optimizing a query - can I do this from my code?

[Gordon Watts]

Hi,

  As I split a QueryModel apart by the Concat’s, some redundancies occur. For example:

 

from Int32 r in {from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select [r].run} select ([r] * 2) => Count()

 

  If I were to feed a query that looked like this to re-linq, I’d see something more along the lines of (it combines several Select statements):

 

from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select (([r].run + 1) * 2) => Count().

 

  Which is a lot more concise. Is there some way I can call the optimizer in my code? The way the more complex sequence is generated is understandable: when Concat statements are in the query, the “from Int32 r in…” makes total sense. But after their removal, that outter from is no longer needed. Being able to optimize it away would make my code cleaner, and testing easier (too).

 

  Many thanks!

 

Cheers,
Gordon

 

[Michael Ketting]

Hi Gordon!
Hmm...Yeah, I can totally see that :) That should also be reproable from the ConcatQuerySplitterTests in your github project, right?
Best regards, Michael

[Gordon Watts]

Hi,

  Ha! Ok, so it isn’t available at this moment? No worries.

 

  And yes, look for the method calls “CheckQuery” that are commented out. Those have the issue. I’d put this in the ConcatQuerySplitter, actually, so the QM’s that it returns for scanning were as optimized as the ones that were handed to me by re-linq in the first place.

 

  At the moment this is not a show stopper for me: so far the (simple) queries I’ve done with the Concat operator have not caused issues being translated to C++. I’m slowly building up more and more complex queries, testing out the system, and I will let you know if something like this becomes a real issue.

 

Cheers,
Gordon

--
You received this message because you are subscribed to the Google Groups "re-motion Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to re-motion-use...@googlegroups.com.
To post to this group, send email to re-moti...@googlegroups.com.
Visit this group at https://groups.google.com/group/re-motion-users.
For more options, visit https://groups.google.com/d/optout.

[Michael Ketting]

Hi Gordon!

Okay, here's the solution:
In ConcatSplitterQueryVisitor.VisitMainFromClause(), after you do qm.MainFromClause.FromExpression = qSub.WrapSQE(),  you can do the optimization if the FromExpression contains a SubQueryExpression with a QueryModel  by accepting the a SubQueryFromClauseFlattener on the QueryModel.

So, yes, we do actually already have it in the framework. Nut no worries, it was good and well burried and I didn't get much milage out of it before, which is why I missed it, too, on the first run :) There are a couple of constraints, though, which you need to check for before actually calling the optimizer (see the xml-comments). But I did manage to get yourQMWithMixedSelecAndConcats test green by calling the flattener.

You could even get rid of your WrapSQE() method, but then you will have to call the optimizer on each of the QueryModels after you are done with the visitor of split-operation itself, i.e. at the end of ConcatSplitterQueryVisitor.Split(), before you return the new QueryModels. Otherwise, there are infinite recursions in there somewhere.

Before:

from Int32 r in {from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select [r].run} select ([r] * 2) => Count()
from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select [r].run => Count()
from Int32 r in {from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select [r].run} select ([r] * 2) => Count()

After:
from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select ([r].run * 2) => Count()
from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select [r].run => Count()
from ntup r in value(LINQToTTreeLib.Tests.QMExtractorQueriable`1[LINQToTTreeLib.Tests.ntup]) select ([r].run * 2) => Count()

Best regards, Michael

To unsubscribe from this group and stop receiving emails from it, send an email to re-motion-users+unsubscribe@googlegroups.com.
To post to this group, send email to re-motion-users@googlegroups.com.

[Gordon Watts]

Wow! Thanks!! I will take a look at this later on today. I’ll post back with result. And thanks for spending some of your weekend on this. 😊

 

And I’m amazed you were able to check out my code and just have it “work”. There are some pretty complex dependencies in there! Go for continuous build integration!

 

And when you get a chance, the QueryModel.Clone guy is a bit of a show stopper for me. 😊 I’m happy to do more debugging now that I’ve gotten relinq to build and all tests to pass (btw, would you accept a small pull request with a readme that contains a section on getting your machine ready to build with it?). However, I’m afraid I don’t actually know what that query model should look like.

To unsubscribe from this group and stop receiving emails from it, send an email to re-motion-use...@googlegroups.com.
To post to this group, send email to re-moti...@googlegroups.com.

[Gordon Watts]

Hi,

  Ok – I put this in. Works like a charm, thanks!!

 

  I removed the WrapSQE and replaced it with flatten’s as you suggested. Anything that makes the code shorter and less complex. 😊

 

  Again, thanks so much for spending time on this!

 

Cheers,
Gordon.

[Michael Ketting]

You're welcome! 

Regarding the check-out. Umm...actually I had to drop the TTreeParserCPP.Tests since there were some C++ header files the system didn't find and just replaced the newly missing TTreeParserCPPTests.CreateTrees methods with stubbed URIs since they're not actually needed. Same goes for the setup/tear-down in the QueryVisitorTest. Basically, I just removed everything that didn't immediately work and concentrated on the linq-stuff. Fortunately, I'm quite used to debugging code where a dependency is missing/hard-to-setup but not actually needed for the current scenario.

On Clone: Ah, okay. I just went through the inbox stack :) Will need get to this a bit later, I'm thinking a day or two...

Pull request: Yeah, sure, there's just a CLA you need to sign and e-mail us: https://relinq.codeplex.com/wikipage?title=Contributing%20to%20re-linq
I've also just updated the docs myself in develop with the latest version we're using in our eco-system but I'm certianly interested what is missing/was missing before the update this morning. One does tend to develop a blind eye to the not-so-obvious stuff in the own documentation ;)

Best regards, Michael

On Monday, February 29, 2016 at 8:08:47 AM UTC+1, Gordon Watts wrote:

Hi,

  Ok – I put this in. Works like a charm, thanks!!

[Gordon Watts]

Hi,

  Ah right… That is a problem with this system – since I build against a C++ library, I have to use the version of the compiler that the C++ library was built against. That is currently an “older” one, so that has to be installed. So if you don’t have that installed on your system, that can cause some major pain. I’m glad after that it built (a lot of tests would have probably failed, however, if you ran-all).

 

  Ok – I will take a look at the updates!

 

  Thanks so much for looking at the Clone issue. That turns out to rear its head in all but the most simple queries when using the Concat operator. It is actually present w/out the Concat operator (as you can see in the small test case I wrote), however, Clone is just never called on those normal queires in re-linq, so the bug is not seen.

 

  It could also be that I’m using Clone in a way that it shouldn’t be. In which case I’d definitely appreciate advice on how to split up QueryModels when doing the Concat splitting!

 

  Thanks in advance to looking at these!

 

Cheers,
Gordon.

 

From: Michael Ketting
Sent: Sunday, February 28, 2016 11:26 PM
To: re-motion Users
Cc: michael...@rubicon.eu

To unsubscribe from this group and stop receiving emails from it, send an email to re-motion-use...@googlegroups.com.
To post to this group, send email to re-moti...@googlegroups.com.