Execute Select within Select in-memory

[Nathan Baulch]

I'd really like to support nested Select operators in my LINQ provider, which consumes a REST based web service that doesn't support sub-queries. Unfortunately re-linq transforms nested Select method calls into SubQueryExpression which cannot be compiled and executed in-memory.

For example, I'd like to do the following:

    client.Query<Account>()
        .Select(a => new
            {
                a.AccountName,
                a.Status,
                Contacts = a.Contacts.Select(c => c.LastName).ToList()
            })
        .ToList();

My query executor already knows how to extract the relevant metadata from the select expression to ensure the appropriate data is requested from the server. All I need now is a way to project the data from the server by passing it to the original un-modified select expression. Unfortunately it looks like this expression is discarded when the SubQueryExpression is substituted. I could move the Select after the query is materialized (ToList), but then the query will select everything from the server which is not ideal.

I've investigated three possible solutions:

1. Prevent SubQueryFindingExpressionTreeVisitor from making the substitution in the first place. This appears to be impossible because ExpressionTreeParser has no extensibility points and SubQueryFindingExpressionTreeVisitor creates it's own private QueryParser.
2. Capture the substituted Select method call and associate it with the SubQueryExpression in a dictionary somewhere so I can substitute it back just prior to compiling and executing the projection.
3. Ignore the original expression and evaluate the sub-query's parsed QueryModel in-memory. This seems like the cleanest option but looks like a lot of work and probably executes a lot slower than the original expression would.

So far I haven't had much luck with any of these approaches. Any advice?

[Fabian Schmied]

Hi Nathan,

I don't think there are currently good ways to implement your #1 and
#2. For #3, I don't know if it would really make the query less
efficient - compiling an expression is a very similar effort as
interpreting it (once), I think.

There is a fourth approach; when you traverse the Select expression
and you encounter the subquery, translate it back into an expression
by using the ReverseResolvingExpressionTreeVisitor and manually
building up a Select method call:
- Apply the visitor to the Selector ([c].LastName) and the subquery's
query source ([c]) to get a LambdaExpression that you can use as a
selector for Select method (c => c.LastName).
- Use the MainFromClause's FromExpression as the Select method's
source parameter and build a MethodCallExpression with the
Enumerable.Select method. You get [a].Contacts.Select(c =>
c.LastName).
- Then, apply the visitor again to that whole expression, using the
outer query's query source as an input parameter ([a]). You get a =>
a.Contacts.Select(c => c.LastName), which you can compile and execute
against an Account object.

However, I think that in your case, the simplest way would indeed be
to completely disable subquery detection. I'll wait for Michael to
confirm this, but I guess he would accept a patch that adds an
overridable "ProcessSubQueries" method to ExpressionTreeParser.

re-linq was originally designed for providers that would also
translate such subqueries into a query to some external system.
However, if more people need a way to get back the original expression
from a QueryModel, we could probably build this into re-linq. Again -
after some more discussion about how to tackle this -, I guess, we'd
accept patches :)

Best regards,
Fabian

> --
> You received this message because you are subscribed to the Google Groups
> "re-motion Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to re-motion-use...@googlegroups.com.
> To post to this group, send email to re-moti...@googlegroups.com.
> Visit this group at http://groups.google.com/group/re-motion-users.
> For more options, visit https://groups.google.com/d/optout.

[Nathan Baulch]

Thanks Fabian, I did look at ReverseResolvingExpressionTreeVisitor originally, but it started getting complicated fast once I considered all the different things a consumer might do in a select clause expression.

For example:

    client.Query<Account>()
        .Select(a => new

            {
                a.AccountName,
                a.Status,
                Contacts = a.Contacts.Where(c => c.Status == "Active")
                                     .Select(c => c.LastName)
                                     .Distinct()
                                     .ToList()
            })
        .ToList();

(You can assume that the consumers of my LINQ provider understand the client-side filtering/aggregation implications of the above code).

So I would also need to process the BodyClauses and all ResultOperators on the sub-query QueryModel rather than assuming that all sub-queries are simple select operations.

In other words, if I wanted a complete general purpose solution I'd have to build the mirror image of QueryParser which turns a fully parsed QueryModel back into LINQ expressions. And all that just to recreate the expression that was discarded by the parser in the first place.

So yeah, the I idea that I could disable sub-queries altogether is a much more attractive option at this stage!

[Michael Ketting]

Hello Nathan!

In regards to the patch, please see the contribution info page at https://relinq.codeplex.com/wikipage?title=Contributing%20to%20re-linq for the formalities.

I think Fabian's idea regarding the template method in ExpressionTreeVisitor should work. The actual contribution and discussion of the patch would then be done on the dev-list at https://groups.google.com/forum/?pli=1#!forum/re-motion-dev

Best regards, Michael