ReLinq steps and concepts

[Jordi Cabré]

I've been digging a lot with re-linq for 2 weeks. 

I've took a look on several RELinq sample projects (Linq2HQLv2, LinqToHqlv1-SqlBackend, ReLinq-ProjectionSupport), and another ones like Lucene.Linq.Net.

Moreover I've read some documentation from re-motion.org.

As far I've been able to figure out, there is a main line to follow in order to provide a Linq provider implementation using re-linq:

1. Get QueryModel, received on each QueryModelVisitorBase implementation.
2. Transform it: A QueryModelVisitor goes through the Linq Expression Tree and transform a LinqExpression to Domain Specific Expression.
3. Translate it: A QueryModelVisitor which I believe it's for getting the Domain Specific Expression Tree transformed previouly and create Domain Specific objects ready to perform over the destionation Backend (SQL: SqlCommand, Lucene: Query...).
4. Perform DSL objects in order to get results.
5. Projection: Project results.

Could you refine these steps?

However, there are conceps which I've been detecting along these projects I've not quite to understand.

• What's QuerySourceMapping for? Do I always write this implementation?
  

// Now replace references like the "i" in "select i" that refers to the "i" in "from i in items"

var mapping = new QuerySourceMapping();

mapping.AddMapping(queryModel.MainFromClause, currentItemExpression);

queryModel.TransformExpressions(e => ReferenceReplacingExpressionVisitor.ReplaceClauseReferences(e, mapping, true)); 

• What's a ResultHandler for?

I'm not expecting for a detailed explication, only a single short brief would be enough.

[Michael Ketting]

Hello Jordi!

>Could you refine these steps?

I'm not really sure what specific question you have about these steps, but I'll try to give you some baselines for transforming the querymodel:
* For steps 1-3, it basically depends on whether your domain specific needs already closely match the QueryModel or if you need to do more stuff. For instance, when we did the NHibernate sample based on generating an HQL string, it was very easy to just stick wit hthe querymodel. On the other hand, our SqlBackend uses a separate model which is created using QueryModelVisitor because we needed more freedom to express tables, joins, etc and once we are done with creating the SQL model, we no longer need the query model.
* Step 4: I'm assuming this refers to executing the operation specified by the linq query, e.g. executing the linq query against the database.
* Step 5: I'm assuming this refers to materializing the query result into in-memory datastructures, typically those represented by the outermost selectexpression.

QuerySourceMapping:

>Do I always write this implementation?

Not really. I guess it depends on what you are trying to do. For instance, in our SqlBackend we don't have to use the QuerySourceMapping by ourselfs at all, everything is already taken case of by the tools provided in the Remotion.Linq library (ie. the re-linq frontend). You do need to use it when you intend to copy/fork the QueryModel instance for some reason or other.

ResultHandler:
ResultHandlers are used to represent everything that deals with how the result data is structured and/or aggregated. They're a bit of a catch-all for everything that deals with the query's result. When writing your own linq provider, you only have to answer two questions: 1) which operators do I wish to support and thus have to handle, and 2) do I have custom needs that will also be represented as a result operator, but I believe that step 2) should be a true second step after the basic infrastrucutre is up and running.

Does that help any? Also, there is the big question of your specific usecase, i.e. do you have a specific backend in mind (sql-based, document based, something else entirely) because this really affects the approaches you will have to take to get the most out of re-linq.

Best regards, Michael

[Jordi Cabré]

Hello Michael.

I'm going to explain how my system behaves. 

I've developed a WebAPI. This WebAPI has a method exposing a DSL. Yes, I've created a language in order to query over my domain objects. In short, this DSL is a query language for providing my objetcs according provided expressions.

So:

• I've my domain objects.
• I've a WebAPI client implemented for asking for: client.search("query sentence").
• I need to provide custom methods on expressions (like Boost in Lucene...).

My goal is translate Linq to this "query sentence" and send it.

I've took as reference Lucene Linq Provider.

As far I've been able to figure out, I need:

1. Modify Linq for simplication purpouses, using IExpressionTransformer (SimplifyCompareTransformer.cs).
2. According to your last comment, I need to decide whether 
• Transform Linq Expression Tree (provided on the query model) and substitute Linq expressions by another ones a bit more similar to my language, transforming last Linq Expressions by my DSL expressions. According to that:
• I need to create my custom Expressions in order to be able to transform Linq Expressions to my DSL Expressions and then get a DSL Expression Tree (LuceneQueryFieldExpression.cs).
• I need to create a custom ExpressionVisitor in order to go through the new DSL expression Tree.
• I need to transform Linq Expressions to DSL Expression using a TransformerExpressionVisitor : ExpressionVisitor.
• Directly translate Linq Expression Tree to a string sentence (I don't prefer this approach... I'd like to first create a semantic model (DSL Expression Tree) and then translate it.
3. I need to create new ExpressionNodes in order to be able to analize custom extension methods (QueryStatisticsCallbackExpressionNode.cs)
4. Translate new DSL Expression Tree to a semantic model (or directly to a string) using a custom create DSL ExpressionVisitor.
   
5. Perform this semantic model over my WebAPI.
6. Project results. I've not been able to figure out how it works. My WeAPI will provide a result according whether Linq is asking for entities or for fields (so I previously need to analize queryModel.SelectClause).

I'm good? I really need some help. 

[Michael Ketting]

Hi Jordi!

Thanks for the details. I can't go into the depth of your #2 right now but I certainly understand much better what you're trying to do. From what I'm reading, your DSL is different enough from the original Linq statement that it warrents the the custom model instead of just To-String-ifying the QueryModel as we did in the NHibernate sample. Of course, if you are already able to create everything you need from the QueryModel, there's probably no need to re-write this, except possibly because you don't like the code and/or maintainability.

With SQL, we went with the custom model because we had several additonal concepts that would need to be applied so it was easier to just start with a new model (SqlStatement). I haven't looked at the Lucene Provider in depth, so I can't speak about the approach they took but in caase you haven't already done this, you definitely should take stock of your DSL language feature (e.g. joins, subqueries, aggregate clauses, etc) and how much optimization you will want to be able to do.

Step 6, the projection. The easiest part would be to just do entities first. That way, you don't have to do any instantiation etc and just return the result from the WebApi call. Scalar results (total count, etc) should also be pretty easy as the IQueryExecutor also offeres ExecuteScalar. The difficult part are what we call custom projections. Here, you construct an anonymous type when you actually write your linq statement and you need to instantiate this type from your result data. Unfortunately, there's not quick and easy way to do this, you will instead have to things like we do in https://github.com/re-motion/Relinq-SqlBackend/blob/develop/Core/SqlGeneration/SqlGeneratingOuterSelectExpressionVisitor.cs#L112 in VisitNew. There, we create an expression that can instantiate the target type and later on, we wrap it in a lamda, compile it and execute it. As I mentioned, it's certainly doable with only a hint of black magic, but I wouldn't recommend it for the first iteration.

Best regards, Michael

[Michael Ketting]

Hi Jordi!

Okay, I've now had a bit more peace and quiet and yes, your steps look good to me. I also realized you are leaning towards the custom model approach so my previous resposne was a tad redundant :) Should have noticed that sooner :o

Best regards, Michael