A little stuck implementing my own custom query operator (similar to WHERE) using ReLinq

[Sam Fold]

Hi there,

I've been using ReLinq to generate a SQL-like language for querying document databases (N1QL) and I want to create a new Linq extension method similar to where.

In usage it would look something like this:

var query =
    QueryFactory.Queryable<Contact>(mockBucket.Object)
        .WhereMissing(e => e.Age)
        .OrderBy(e => e.Age)
        .Select(e => new { age = e.Age, name = e.FirstName });

Where I am stuck is on how to register this new node type with Relinq. There seems to be almost no documentation concerning this. 

I found this tutorial: https://www.re-motion.org/blogs/mix/2010/10/28/re-linq-extensibility-custom-query-operators/

But what I'm trying to implement doesn't seem to be a result operator. This isn't acting on the result of the query like Count or Take. This new operator would get translated into some N1QL that looked like "WHERE e.Age IS MISSING" This essentially means the property of this NoSQL document isn't present (i.e. not the same meaning as IS NULL).

I'm guessing I have to implement "ExtensionExpression" as a base class, but after that I'm unsure of the next step. 

Anyone have any experience with this?

Sam

[Michael Ketting]

Hi Sam!

I believe you're looking for these two posts here:
https://www.re-motion.org/blogs/mix/2011/01/27/re-linq-a-lot-of-new-customizability
https://www.re-motion.org/blogs/mix/2011/04/29/re-linq-customizability-explained/

I can see where you are going with the WhereMissing() method and I'll leave to it Fabian to explain how to introduce new concepts into re-linq. What I would recommend though, would be to just use the Where()-method accepting a predicate. And then you write custom predicates. We did something similiar to incorporate fulltext-queries into our SQL backend:
https://github.com/re-motion/Relinq-SqlBackend/tree/master/Core/SqlPreparation/MethodCallTransformers/ContainsFulltextMethodCallTransformer.cs
(There's also a ton of other string-based operators in he MethodCallTransformer namespace)

The query would then look like this:

var query =
    QueryFactory.Queryable<Contact

>(mockBucket.Object)
        .Where(e => e.Age.IsMissing())
        .OrderBy(e => e.Age)
        .Select(e => new { age = e.Age, name = e.FirstName });

The only downside of this is that the IsMissing()-extension method would for practical purposes be types to object, so it would pop up everywhere.

Best regards, Michael

[Fabian Schmied]

Hi,

Since Micheal asked me to do so, I'll explain what options come to my mind.

1 - Instead of ".WhereMissing(a => a.Age)", have your users write ."Where(e => e.Age.IsMissing())".

2 - Add a method called "WhereMissing" and apply the "MethodCallExpressionTransformerAttribute" to it.

3 - Implement a custom "node type parser" that analyzes the "WhereMissing" method and generates a corresponding clause in the QueryModel.

#1, which is what Michael suggested, also has the advantage of being combinable. I.e., you can say ".Where(e => e.Age.IsMissing() || e.Age == null)", which is not possible with ".WhereMissing".

#2 means that the user writes ".WhereMissing(e => e.Age)", and your transformer changes this to be ".Where(e => SomeUtilityClass.IsMissing(e.Age))". You can then easily detect that utility method in your backend. This has the advantage of providing the syntax you need with minimal complexity in extending re-linq. See https://www.re-motion.org/jira/browse/RM-5302 for details about this approach.

(#2b: You could theoretically implement the transformation from ".WhereMissing(e => e.Age)" to ".Where(e => SomeUtilityClass.IsMissing(e.Age))" in a few other ways as well, but I think they are all more complicated than the attribute,)

#3 means you derive a class from "MethodCallExpressionNodeBase", similar to how "WhereExpressionNode" is implemented. That class overrides "ApplyNodeSpecificSemantics" to add your own clause to the QueryModel. However, this is the most complex way of extending re-linq and it's meant mainly for scenarios where you want to add a new kind of clause (e.g., a "WhereMissingClause"). In your case, you can just as easily fall back to the "WhereClause", so I'd rather use #2 instead of this approach. If you still want to give it a try, have a look at https://www.re-motion.org/blogs/mix/2011/04/29/re-linq-customizability-explained/ (last section - "Custom Expression Nodes").

Best regards,

Fabian

--
You received this message because you are subscribed to the Google Groups "re-motion Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to re-motion-use...@googlegroups.com.
To post to this group, send email to re-moti...@googlegroups.com.
Visit this group at http://groups.google.com/group/re-motion-users.
For more options, visit https://groups.google.com/d/optout.

[Sam Fold]

Hi guys,

So, in the interim I managed to get a .WhereMissing node working, using Fabians #3 from above. It took quite a while and I had to dig through Relinq's source to grok most of what was going on, but I think it's working as expected now. However, I may have not done the implementation in the "approved" fashion. 

public static IQueryParser CreateQueryParser()
{
    var customNodeTypeRegistry = new MethodInfoBasedNodeTypeRegistry();
 
    customNodeTypeRegistry.Register(WhereMissingExpressionNode.SupportedMethods, typeof(WhereMissingExpressionNode));
 
    var nodeTypeProvider = ExpressionTreeParser.CreateDefaultNodeTypeProvider();
    
    nodeTypeProvider.InnerProviders.Add(customNodeTypeRegistry);
 
    var transformerRegistry = ExpressionTransformerRegistry.CreateDefault();
 
    var processor = ExpressionTreeParser.CreateDefaultProcessor(transformerRegistry);
 
    var expressionTreeParser = new ExpressionTreeParser(nodeTypeProvider, processor);
    
    var queryParser = new QueryParser(expressionTreeParser);
 
    return queryParser;
}

Here is my "WhereMissingClause"

public class WhereMissingClause : IBodyClause
{
    private Expression _predicate;
 
    /// <summary>
    /// Initializes a new instance of the <see cref="WhereMissingClause"/> class.
    /// </summary>
    /// <param name="predicate">The predicate used to filter data items.</param>
    public WhereMissingClause (Expression predicate)
    {
        _predicate = predicate;
    }
 
    /// <summary>
    /// Gets the predicate, the expression representing the where condition by which the data items are filtered
    /// </summary>
    public Expression Predicate
    {
        get { return _predicate; }
        set { _predicate = value; }
    }
 
    /// <summary>
    /// Accepts the specified visitor 
    /// </summary>
    /// <param name="visitor">The visitor to accept.</param>
    /// <param name="queryModel">The query model in whose context this clause is visited.</param>
    /// <param name="index">The index of this clause in the <paramref name="queryModel"/>'s <see cref="QueryModel.BodyClauses"/> collection.</param>
    public virtual void Accept (IQueryModelVisitor visitor, QueryModel queryModel, int index)
    {
        var visotorx = visitor as N1QlQueryModelVisitor;
        if (visotorx != null) visotorx.VisitWhereMissingClause(this, queryModel, index);
    }
 
    /// <summary>
    /// Transforms all the expressions in this clause and its child objects via the given <paramref name="transformation"/> delegate.
    /// </summary>
    /// <param name="transformation">The transformation object. This delegate is called for each <see cref="Expression"/> within this
    /// clause, and those expressions will be replaced with what the delegate returns.</param>
    public void TransformExpressions (Func<Expression, Expression> transformation)
    {
        Predicate = transformation (Predicate);
    }
 
    /// <summary>
    /// Clones this clause.
    /// </summary>
    /// <param name="cloneContext">The clones of all query source clauses are registered with this <see cref="CloneContext"/>.</param>
    /// <returns></returns>
    public virtual WhereMissingClause Clone (CloneContext cloneContext)
    {
        var clone = new WhereMissingClause (Predicate);
        return clone;
    }
 
    IBodyClause IBodyClause.Clone (CloneContext cloneContext)
    {
        return Clone (cloneContext);
    }
 
    public override string ToString ()
    {
        return "WHERE MISSING " + FormattingExpressionTreeVisitor.Format (Predicate);
    }
}

Here is the custom expression node:

public class WhereMissingExpressionNode : MethodCallExpressionNodeBase
{
    public static readonly MethodInfo[] SupportedMethods = new[]
    {
        GetSupportedMethod (() => QueryExtensions.WhereMissing<object, object> (null, o => null)),
        GetSupportedMethod (() => QueryExtensions.WhereMissing<object, int> (null, o => 10))
    };
 
    private readonly ResolvedExpressionCache<Expression> _cachedPredicate;
 
    public WhereMissingExpressionNode(MethodCallExpressionParseInfo parseInfo, LambdaExpression predicate)
        : base(parseInfo)
    {
        if (predicate.Parameters.Count != 1)
            throw new ArgumentException("Predicate must have exactly one parameter.", "predicate");
 
        Predicate = predicate;
        _cachedPredicate = new ResolvedExpressionCache<Expression>(this);
    }
 
    public LambdaExpression Predicate { get; private set; }
 
    public Expression GetResolvedPredicate(ClauseGenerationContext clauseGenerationContext)
    {
        var expression = _cachedPredicate.GetOrCreate(r => r.GetResolvedExpression(Predicate.Body, Predicate.Parameters[0], clauseGenerationContext));
 
        return expression;
    }
 
    public override Expression Resolve(ParameterExpression inputParameter, Expression expressionToBeResolved, ClauseGenerationContext clauseGenerationContext)
    {
        return Source.Resolve(inputParameter, expressionToBeResolved, clauseGenerationContext);
    }
 
    protected override QueryModel ApplyNodeSpecificSemantics(QueryModel queryModel, ClauseGenerationContext clauseGenerationContext)
    {
        queryModel.BodyClauses.Add(new WhereMissingClause(GetResolvedPredicate(clauseGenerationContext)));
       
        return queryModel;
    }
}

And in the querymodelvisitor I just have this addition method:

public void VisitWhereMissingClause(WhereMissingClause whereClause, QueryModel queryModel, int index)
{
    var expression = GetN1QlExpression(whereClause.Predicate);
    _queryPartsAggregator.AddWhereMissingPart(String.Concat(expression, " IS MISSING"));                        
}

The result of these additions are that if I run the solution on this Linq query:

var query =
    QueryFactory.Queryable<Contact>(mockBucket.Object)

        .Where(e => e.Email == "some...@gmail.com")

        .WhereMissing(e => e.Age)
        .OrderBy(e => e.Age)
        .Select(e => new { age = e.Age, name = e.FirstName });

I get this result: SELECT e.age, e.name FROM default as e WHERE (e.Email = 'some...@gmail.com') AND e.Age IS MISSING ORDER BY e.Age ASC

Which would be a correct result. Can you see any obvious miss-steps? Forgive my relative naivety with this framework, I literally only started working with it 4 days ago and there's obviously a fair degree of complexity here!

[Fabian Schmied]

Hi Sam,

At first glance, you seem to have gotten everything right, congratulations! :)

However, I'd still recommend refactoring towards my suggestion #2, as it's probably easier to maintain in the long run.

Also, ask yourself if you really don't need to allow combining "IsMissing" with other predicates using "OR" (e.g., e => IsMissing(e.Age) || e.Age <= DateTime.Today). That's only possible using an approach similar to Michael's suggestion (#1).

Best regards,

Fabian