re-linq front-end vs. back-end (was: "NH-2254 Options" on nhibernate-development)

[Fabian Schmied]

Cross-posting from <http://groups.google.com/group/nhibernate-
development/browse_thread/thread/70bc90d98e159290> to follow up on re-
linq.

Frans:
> But kidding aside, I see the advantage re-linq brings, no question
> about that, I just wonder (and still do) where the complex linq stuff is
> solved: in re-linq or does a user of re-linq have to solve these probs?

This depends. As Stefan has pointed out, re-linq has two sides, the
front-end and multiple back-ends. re-linq contains one back-end, which
produces SQL and is currently under development. NHibernate is another
backend which constructs HQL ASTs. There are other backends as well.

The front-end performs the following simplifications automatically:
- Partial evaluation
- Subquery detection
- Query source resolution (and transparent identifier elimination)
- Plus a few minor issues (such as VB equality comparison detection)

It then represents the LINQ query using a meta-model consisting of
"clauses" and "result operators". Clauses represent the standard Where/
OrderBy/Select/From/Join query operators, result operators represent
other query operators such as Count, Take, or (also) GroupBy. This is,
of course extensible, new clauses and result operators can be added,
new query operators detected.

Consider: orders.SelectMany (o => o.OrderItems.Where (oi => oi.Product
== "pen" + "s"), (o, oi) => new { o = o, oi = oi }).Select (trans =>
trains.o).Count()

The front-end represents this as:
(from o in orders
from oi in (
from gen in [o].OrderItems
where [gen].Product == "pens"
select [gen]
)
select [o]).Count()

([o] and [gen] are QuerySourceReferenceExpressions)

The front-end does not perform any SQL-specific simplifications, e.g.
flattening of subqueries. It just provides a "normalized" view of the
LINQ query.

(Such simplifications could be performed on the front-end's query
model of couse, but when I started designing the SQL backend, I've
found it more convenient to build a domain-specific meta-model of the
SQL statement being generated and perform the transformations in that
realm.)

There are lots of things you need to do when writing a SQL backend
using re-linq, GroupBy resolution (pushing aggregations into the right
place, for example), Group joins, DefaultIfEmpty, subquery
optimization, and other things. These are, however, SQL-specific, and
therefore need to be part of the SQL back-end. Some of them, I think,
might not necessarily be as hard to do in an HQL backend as in a SQL
backend.

Yes, a re-linq backend will still have to do a lot of work. But at
least it can start in a cleaner place.

[misery query snipped]

> the above misery query has several complex problems combined which
> make it problematic to work with. What would be great if a pre-processor
> would solve these, so transformation is easier (to sql): that elements are
> at the right spot for discovery for transformation, so the provider doesn't
> have to hunt down sources for particular properties, can work with scopes
> easily so subtree references are not crossing scopes for alias assignment
> etc.

Some of this will be easier using re-linq, yes. Getting it into valid
SQL might still be a challenge; and transforming it into _optimum_ SQL
even more so :)

Frans: If you plan to take a look at re-linq (both front-end and SQL
back-end), I would be really interested in your feedback. Please post
it to <http://groups.google.com/group/re-motion-dev> or e-mail me
privately; feedback by anyone having mastered that many LINQ hurdles
will be much appreciated.

Best regards,
Fabian

On Jul 29, 1:26 pm, "Wenig, Stefan" <stefan.we...@rubicon.eu> wrote:
> > oh I agree. Similar to the thing we saw last week with the
> > skip/take/count stuff.
>
> Which was a problem reported for the old contrib provider. A coworker just pointed that out, in the heat of the discussion nobody seemed to notice ;-)
>
> > LoC measured in ndepend (so true LoC) or from sourcefiles? My
> > linq
> > provider has in ndepend 6500 LoC. In sourcecode, it's a multiple of
> > that,
> > but I'm very verbose haha :D
>
> Source files, counting comments and license headers too. Grain of salt recommended.
>
> > But kidding aside, I see the advantage re-linq brings, no
> > question
> > about that, I just wonder (and still do) where the complex linq stuff
> > is
> > solved: in re-linq or does a user of re-linq have to solve these probs?
>
> > I'll have a look at re-linq to see what kind of trees it produces
> > with some of the 'pain' linq queries like: (adventure works)
>
> (insulting query removed)
>
> In the frontend, I'm pretty sure we handle that. You'll get a nice query model with neatly separated subqueries.
>
> As for the backend: With a quick look I can't see anything that shouldn't work. (Fabian may.) But we'd have to try, this query was built to destroy LINQ providers after all ;-)
>
> I suggest you take that question to re-moti...@googlegroups.com. I don't want to hijack this list here.
>
> > it still pains me to see this one fail in my linq provider (among
> > several other 'headache' queries), but then again, it's not a common
> > query
> > ;) (it's a reproduction of a problem with a real-life query, so it
> > doesn't
> > make sense, but illustrates a couple of nasty issues)
>
> I can't see these issues, just an annoying level of subquery nesting, which re-linq should handle gracefully. But maybe I'm missing something. Ask the man!
> (The frontend will not resolve the DefaultIfEmtpy to a left join though! I think I discussed that with Fabian once, but my memory fails me.)
>
> > If re-linq can solve that, it would indeed be rather 'easy' as
> > in:
> > way easier than creating it using the 'warren' method with expression
> > objects which are converted into other expression objects etc. which
> > leads
> > to painful conversions.
>
> That's what we're trying!