Collection Property Any method -> node must be a reducible node

497 views
Skip to first unread message

Jordi Cabré

unread,
Mar 15, 2016, 8:48:05 AM3/15/16
to re-motion Users
I've been able to transform slight Linq AST like:

       this.backend.DigitalInputs.Where(di => di.Matter.EndsWith(string.Empty));

I've achieved that overriding VisitMember, VisitQuerySourceReference and VisitMethodCall implemented along two Visitors.

Now, I'm trying to do that:

        this.backend.DigitalInputs.Where(di => di.Properties.Any(p => p.Key.StartsWith(string.Empty)) && di.Matter.EndsWith(string.Empty));

As you can see I'm trying to query over Properties List property of each query entity.
Basiclly, I don't know how to go on, what to do, which Visit* methods override...

As far I've been able to test, I've used the same slight linq provider implementation I've implemented up to now. When I test the last Linq sentence, it raises me a System.ArgumentException with message "node must be a reducible node".
It raises when I'm trying to transform Linq Expression AST to my DSL Expression AST:

        public override void VisitWhereClause(WhereClause whereClause, QueryModel queryModel, int index)
       
{
           
foreach (var visitor in whereSelectClauseVisitors)
           
{
                whereClause
.TransformExpressions(visitor.Visit);   <<<<<
           
}

           
base.VisitWhereClause(whereClause, queryModel, index);
       
}


When exception comes up visitor variable is a Visitor that overrides simply VisitMethodCall method:

    internal class MethodCallToQueryPredicateExpressionTreeVisitor : ExpressionVisitor
   
{
       
protected override Expression VisitMethodCall(MethodCallExpression expression)
       
{
           
var queryField = expression.Object as LQLFieldExpression;

           
if (queryField == null)
               
return base.VisitMethodCall(expression);

           
if (expression.Method.Name == "StartsWith")
           
{
               
return new LQLPredicateExpression(queryField, expression.Arguments[0], SemanticModel.Schema.PredicateOperationsTypeEnum.LIKE);
           
}
           
if (expression.Method.Name == "EndsWith")
           
{
               
return new LQLPredicateExpression(queryField, expression.Arguments[0], SemanticModel.Schema.PredicateOperationsTypeEnum.LIKE);
           
}
           
if (expression.Method.Name == "Contains")
           
{
               
return new LQLPredicateExpression(queryField, expression.Arguments[0], SemanticModel.Schema.PredicateOperationsTypeEnum.LIKE);
           
}

           
return base.VisitMethodCall(expression);
       
}
   
}

I've took a breakpoint in this method, however it's never reached.
What do I need to analize/transform previously?

Thanks.

Michael Ketting

unread,
Mar 16, 2016, 5:55:34 PM3/16/16
to re-motion Users
Hi Jordi!

Okay, what you have in the second query example is a SubQueryExpression nested inside the WhereClause and RelinqExpressionVisitor exposes a dedicated VisitSubQuery method for this. Unless I'm missing something from the sample, you will need to derive from RelinqExpressionVisitor instead of ExpressionVisitor and handle the re-linq specific nodes, too. You can check out https://github.com/re-motion/Relinq-SqlBackend/blob/develop/Core/SqlPreparation/SqlPreparationExpressionVisitor.cs for how we do this in the SqlBackend but the gist is that we build the substatement as it would be done stand alone except that it contains a reference to the outside context and when we generate the SQL, we resolve those dependencies to correct identifiers. From what I understand of your DSL usecase, you're going to have a similiar requirement.

Now, for the ArgumentException, that sounds like it is thrown from the SubQueryExpression (hard to be 100% sure without the stacktrace) and there probably should be a better exception for this (I've created an issue: https://www.re-motion.org/jira/browse/RMLNQ-97, thanks for pointing this out). Anyhow, if that's the cause, then using the RelinqExpressionVisitor as a base class will fix this.

I can't tell you right now if you actually need the TransformExpressions in your QueryModelVisitor, all I can tell you is that we don't need to do that in the SqlBackend but that's just a baseline.

Your MethodCallToQueryPredicateExpressionTreeVisitor looks good to me, it basically does the same thing we do in the SqlPreparationExpressionVisitor.

Best regards, Michael

Jordi Cabré

unread,
Mar 18, 2016, 9:01:21 AM3/18/16
to re-motion Users
Thanks a lot Michael. I really appreciate your comments!

I've implemented a dummy SubQueryTreeVisitor:

    internal class SubQueryTreeVisitor : RelinqExpressionVisitor
    {

        protected override Expression VisitSubQuery(SubQueryExpression expression)
        {
            return base.VisitSubQuery(expression);
        }

    }

I've set a breakpoint inside VisitSubQuery, and it's reached. Perfect!

    this.backend.DigitalInputs.Where(di => di.Properties.Any(=> p.Key.Equals("p1") && p.Value.StartsWith(string.Empty)) && di.Matter.EndsWith(string.Empty));

Transformation process is fairly straightforward. It needs to change any MemberExpression by a new DSLFieldExpression (this is already implemented on QuerySourceReferencePropertyTransformingTreeVisitor class).
Example:

    di.Matter.EndsWith(string.Empty) -> [MemberExpression ([di].Matter)] is translated as [DSLFieldExpression (name: "matter", type: field)].EndsWith(string.Empty)

There is a little trouble: As far as you could figure out seeing the Linq sentence, di.Properties might seem a relational ship between di entities and its Properties entities.
However, it's not quite true.

In C#, di.Properties is a list of Property object. Each Property object has a Key and a Value.
It's like this due to my WebAPI backend is schemaless (ElasticSearch), so I send each schema column as a Property(Key, Value).

So, each di has a list of Property with Key and Value.
My DSL is using WebAPI concepts. So, there is no a di entity related with several property entities.

A sample of last Linq sentence translated to my DSL:

    search digitalinputs query property.p1 eq "" and field.matter "+"

So, I would need to translate Any() expression by a DSLFieldExpression (name: "p1", type: property).

When VisitSubQuery is reached, SubQueryExpression expression.QueryModel parameter is:

   
    from IndexableProperty p in [di].Properties where [p].Key.Equals("p1") && [p].Value.StartsWith(string.Empty) select [p] => Any()


As far I'm able to figure out, I'd need to pick a VisitMember() up for [di].Properties and:

  1. Analyze [p] is from "Properties" property.
  2. Translate it to a DSLExpression (name: ?, type: property).
  3. Analyze [p].Key and fill expression up with (name: "p1", type: property).
I've already commented before a QuerySourceReferencePropertyTransformingTreeVisitor visitor exist in order to change MemberExpression by a DSLFieldExpression. How to perform it again on the subquery?

Code (QuerySourceReferencePropertyTransformingTreeVisitor.cs):

    /// <summary>
    /// Replaces MemberExpression instances like [QuerySourceReferenceExpression].PropertyName with <c ref="LuceneQueryFieldExpression"/>
    /// </summary>
    internal class QuerySourceReferencePropertyTransformingTreeVisitor : RelinqExpressionVisitor
    {
        private MemberExpression parent;
        private DSLFieldExpression queryField;

        protected override Expression VisitMember(MemberExpression expression)
        {
            this.parent = expression;

            var result = base.VisitMember(expression);

            return this.queryField ?? result;
        }

        protected override Expression VisitQuerySourceReference(QuerySourceReferenceExpression expression)
        {
            var propertyInfo = parent.Member as PropertyInfo;

            var propertyType = propertyInfo.PropertyType;
            if (propertyType.IsEnum)
            {
                propertyType = Enum.GetUnderlyingType(propertyType);
            }

            queryField = new LQLFieldExpression(null, null, propertyInfo.Name, propertyType);
            return base.VisitQuerySourceReference(expression);
        }
    }




Michael Ketting

unread,
Mar 20, 2016, 4:45:00 PM3/20/16
to re-motion Users
Hi Jordi!

Thank you for explaining the details of your use case. I've thought about your problem and from what I now understand, you actually don't care that you are getting a subquery. Am I correct that you would prefer to just write
this.backend.DigitalInputs.Where(di => di.p1.StartsWith(string.Empty)) && di.Matter.EndsWith(string.Empty));

Of course, that's not possible in c# without using dynamic types but you could write
this.backend.DigitalInputs.Where(di => di.Properties["p1"].StartsWith(string.Empty)) && di.Matter.EndsWith(string.Empty));
Here, properties would not actually be a List<Property> but simply a PropertyBag wrapper object implementing an indexer.

Or, you could go shorter and implement the indexer directly on you domain basetype:
this.backend.DigitalInputs.Where(di => di["p1"].StartsWith(string.Empty)) && di.Matter.EndsWith(string.Empty));

If that syntax is possible for you, you can significantly simplify the linq tree generated, i.e. you would eliminate the subquery from problem domain. From what I understand of your target syntax, my suggestion should match your needs exactly.

For your question on the subquery: Basically, in your case you want to get a simple expression from the subquery which represents your predicate. I'm guessing the Any() result operator must always be there for your syntax to work, so you can start by checking that the from clause, the result operators and the select clause always match your constaints. Then, you can go an extract the body clause from the QueryModel. This is where your property access and the condition is located. This expression you can then transform into a DslFieldExpression as the source of your value. I think thats what your QuerySourceReferencePropertyTransformingTreeVisitor is meant to do, but I'm not 100% sure about that. Either way, you can just apply the correct visitor to the where expression located in the QueryModel's body expression list.

If you happen to be stuck with using subqueries and my answer isn't sufficient, I'm, sure we can drill deeper, but I would highly recommend taking a closer look at the above suggestions as an easier way to accomplish your use case.

Best regards, Michael

Jordi Cabré

unread,
Mar 21, 2016, 10:53:07 AM3/21/16
to re-motion Users
Hello Michael.

Your approaches are welcome and, from now on, I'm going to try to provide this kind of property access, I'm sure.
However, it's important for me provide queries like:

this.backend.DigitalInputs.Where(di => di.Properties.Any(p => p.Key.StartsWith("x") && p.Value.Equals("v1"))); --> any [di] with a property starting with "x" and value is equals to "v1"
Translation DSL: search digitalinputs query property.name "x+" && "v1";

this.backend.DigitalInputs.Where(di => di.Properties.Any(p => p.Value.EndsWith("v1"))); --> any [di] with a whichever property with value ending with "v1"
Translation DSL: search digitalinputs query property "+v1";

this.backend.DigitalInputs.Where(di => di.Properties.All(p => p.Value.EndsWith("v1"))); --> [di] with every property value equals to "v1"
Translation DSL: search digitalinputs query property "+v1";

So, I mean the name of the "key" property has not always to be provided on query sentence.

I'm glad to say, I've been able to replace each [p].Key and [p].Value with a custom DSLFieldExpression.
So, after transformation I 've a subquery like this:

where (DSLField(field:key1, type:property).Equals("key1") AndAlso LivingField(field:key1, type:property).Equals("v1"))

However, it's not usefull this "where" expression as a subquery. I would like this transformed subquery move up building a raw query.

So, 

    this.backend.DigitalInputs.Where(di => di.Properties.Any(p => p.Key.Equals("key1") && p.Value.Equals("v1")) && di.Matter.EndsWith(string.Empty));

it's translated like:

    this.backend.DigitalInputs.Where(di => di.Properties.Any(p => [DSLField].Equals("key1") && [DSLField].Equals("v1")) && di.Matter.EndsWith(string.Empty));



I would like to get:

    this.backend.DigitalInputs.Where(di => [DSLField(field:key1, type:property)].Equals("key1") && [DSLField(field:key1, type:property)].Equals("v1") && [DSLField(field:matter, type:field)].EndsWith(string.Empty));


Is it possible to do that? So, move a subquery up to main query (to main where clause)?

Best regards.

Michael Ketting

unread,
Mar 21, 2016, 4:47:37 PM3/21/16
to re-motion Users
Hi Jordi!

Ah, yes, I see. In this case you do need to extract the subquery and transform it back into your desired Where clause. To do that you would need to pull out the body clause the way I suggested in my previous mail. This is certainly doable (regarding your last question). Can I give you a specific answer to those bits?

There might also another way, provided that you only need a couple of the query operators (e.g. Any, All, possibly Concat and Union, maybe Count): You could just define instance methods for the Properties collection and handle Any and All, etc directly. That way, you would also prevent the subquery from happening in the first place. (And yes, this is still just an extrapolation from what I now know, but I do like to offer suggestions on how the problem domain might be redefined in attempt to circumvent the difficult bits. I hope you don't mind :) )

class PropertyBag
{
  public bool Any (Func<Property, bool> predicate);
  // other relevant query methods
}

That way, you're not starting a subquery, you get the information that you are interested in Any property matching the predicate, and you get the predicate itself. And the query in C# still looks the same as the one you wrote.

The downside is that you have to re-implement the basic linq extension methods on your own, which is not a bad thing necessarily, but I would only recommend it when you don't have to interact with general purpose query factories. For instance, this approach wouldn't work if you try to integrate with a generic OData provider or something of the sort. Then again, I'm assuming that you wouldn't be able to write the query you like with a generic OData provider in the first place.

The upside is that you could also direclty express what makes sense, for instance do you wish to expose a Take/Skip concept for the Properties collection? Sum? Join? GroupBy? OrderyBy? So, by not implementing IEnumerable<T> and IQueryable<T> but just the relevant methods, you could accomplish just that.

Best regards, Michael

Jordi Cabré

unread,
Mar 29, 2016, 4:20:38 AM3/29/16
to re-motion Users
Hi Michael.

I've to tell you I've been to get my goal. I've been able to transform the Linq tree to my DSL Tree like:

    this.backend.DigitalInputs.Where(di => di.Properties.Any(p => p.Key.Equals("key1") && p.Value.Equals("v1")) && di.Matter.EndsWith(string.Empty));
is transformed to:
   
this.backend.DigitalInputs.Where(di => [DSLPredicate[EQ](DSLField(field:key1, type:property) - ("v1"))] && [DSLPredicate[LIKE](DSLField(field:matter, type:field) - (""))];
and translated to:
   search digitaInputs query property
.key1 eq "v1" && field.matter like "+"

So, everything is well now.

By now, I don't want to provide Take/Skip, Sum, Join, GroupBy, OrderyBy concepts for the Properties collection? 
Nevertheless I first would like to provide two features:

  • Query over whichever facet (field (di.Matter) or property (di.Properties)): Do I need to extend an IClause? How am I able to visit it?
       this.backend.DigitalInputs.Where(di.WhicheverFacet<string>(f => f.Value.StartsWith(string.Empty)));
       
[DSLPredicate[LIKE](DSLField(field:*, type: none))("")];
  • Get OrderBy functions on main Queryable.
        this.backend.DigitalInputs.Where(di.WhicheverFacet<string>(f => f.Value.StartsWith(string.Empty))).OrderBy(di => di.Matter);

Could you briefly point out me how to get those features?

Michael Ketting

unread,
Mar 30, 2016, 2:19:21 AM3/30/16
to re-motion Users
Hello Jordi!

Great to hear your property queries are working :)

OrderBy: OrderBy statements are represented by OrderyBy clauses, which can be found in the BodyClauses collection of the QueryModel and the VisitOrderByClause method of the QueryModelVisitor. It should be pretty straight forward from there on out.

WhicheverFacet: No, you should be able to solve this without a custom clause. It's more (or quite likely, exactly) like the Any and All result operators:
* You define your own WhicheverFacetNode, derived from ResultOperatorExpressionNodeBase.
* You register the WhicheverFacetNode in the NodeTypeProvider via the MethodInfoBasedNodeTypeRegistry.
* You define a WhicheverFacetNodeResultOperator derived from ValueFromSequenceResultOperatorBase
* You handle the result operator when visiting the QueryModel in the VisitResultOperator method.

Here's how we do this to extend Linq with eager-fetching support:
https://svn.re-motion.org/svn/Remotion/trunk/Remotion/Data/DomainObjects/Queries/LinqProviderComponentFactory.cs

Best regards, Michael
Reply all
Reply to author
Forward
0 new messages