Using re-linq to parse expression trees containing DynamicExpression

[Alex Norcliffe]

Hi there, I'm trying to use ReLinq to parse an expression tree that
contains some DynamicExpression nodes (DynamicExpression being a new
type in .NET 4).

I'm running into an issue in PartialEvaluatingExpressionTreeVisitor.

For an expression tree that happily works against an
IQueryable<dynamic> of a couple of ExpandoObjects, using ReLinq to
analyse the same expression causes the following error:

System.InvalidOperationException : variable 'x' of type
'System.Object' referenced from scope '', but it is not defined

Noticing the stack trace points towards
PartialEvaluatingExpressionTreeVisitor, I decided to test that
directly. Here's one I knocked up which just makes a regular Lambda
referring to a dynamic member getter:

[Test]
public void TestRelinqWithDynamic()
{
ParameterExpression parameterExpression =
Expression.Parameter(typeof(object), "x");

var returnMember =
Expression.Dynamic(
Binder.GetMember(
CSharpBinderFlags.InvokeSimpleName,
"colour",
typeof(object),
new[]
{ CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) }),
typeof(object), parameterExpression);

var body = Expression.MakeBinary(ExpressionType.Equal,
returnMember, Expression.Constant("orange"));

var lambda = Expression.Lambda(body, parameterExpression);

var casted = (Expression<Func<dynamic, bool>>)lambda;

dynamic aDynamic = new ExpandoObject();
aDynamic.colour = "orange";

var testItems = new dynamic[] { aDynamic, aDynamic,
aDynamic }.AsQueryable();
var checkExpression = testItems.Where(casted).ToList();

Assert.That(checkExpression.Count, Is.EqualTo(3));

var info =
EvaluatableTreeFindingExpressionTreeVisitor.Analyze(casted);
var visited =
PartialEvaluatingExpressionTreeVisitor.EvaluateIndependentSubtrees(casted);
}

I get the exception on the last line, whereas by then the expression
"casted" has already worked just fine on the "testItems" collection.

As always happens, I hit this just after telling everyone it would be
"easy" to do what I needed to do with Relinq in just a few days :P


The stack trace I get is below


at
System.Linq.Expressions.Compiler.VariableBinder.Reference(ParameterExpression
node, VariableStorageKind storage)
at
System.Linq.Expressions.Compiler.VariableBinder.VisitParameter(ParameterExpression
node)
at
System.Linq.Expressions.ParameterExpression.Accept(ExpressionVisitor
visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at
System.Linq.Expressions.ExpressionVisitor.VisitArguments(IArgumentProvider
nodes)
at
System.Linq.Expressions.ExpressionVisitor.VisitDynamic(DynamicExpression
node)
at System.Linq.Expressions.DynamicExpression.Accept(ExpressionVisitor
visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at
System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression
node)
at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor
visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at
System.Linq.Expressions.Compiler.VariableBinder.VisitUnary(UnaryExpression
node)
at System.Linq.Expressions.UnaryExpression.Accept(ExpressionVisitor
visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at
System.Linq.Expressions.ExpressionVisitor.Visit(ReadOnlyCollection`1
nodes)
at
System.Linq.Expressions.Compiler.VariableBinder.VisitLambda<T>(Expression`1
node)
at System.Linq.Expressions.Expression`1.Accept(ExpressionVisitor
visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at
System.Linq.Expressions.Compiler.LambdaCompiler.Compile(LambdaExpression
lambda, DebugInfoGenerator debugInfoGenerator)
at
Remotion.Linq.Parsing.ExpressionTreeVisitors.PartialEvaluatingExpressionTreeVisitor.EvaluateSubtree(Expression
subtree) in c:\build\Remotion\working\Relinq\Core\Parsing
\ExpressionTreeVisitors\PartialEvaluatingExpressionTreeVisitor.cs:
line 110
at
Remotion.Linq.Parsing.ExpressionTreeVisitors.PartialEvaluatingExpressionTreeVisitor.VisitExpression(Expression
expression) in c:\build\Remotion\working\Relinq\Core\Parsing
\ExpressionTreeVisitors\PartialEvaluatingExpressionTreeVisitor.cs:
line 76
at
Remotion.Linq.Parsing.ExpressionTreeVisitor.VisitLambdaExpression(LambdaExpression
expression) in c:\build\Remotion\working\Relinq\Core\Parsing
\ExpressionTreeVisitor.cs: line 278
at
Remotion.Linq.Parsing.ExpressionTreeVisitor.VisitExpression(Expression
expression) in c:\build\Remotion\working\Relinq\Core\Parsing
\ExpressionTreeVisitor.cs: line 114
at
Remotion.Linq.Parsing.ExpressionTreeVisitors.PartialEvaluatingExpressionTreeVisitor.VisitExpression(Expression
expression) in c:\build\Remotion\working\Relinq\Core\Parsing
\ExpressionTreeVisitors\PartialEvaluatingExpressionTreeVisitor.cs:
line 83
at
Remotion.Linq.Parsing.ExpressionTreeVisitors.PartialEvaluatingExpressionTreeVisitor.EvaluateIndependentSubtrees(Expression
expressionTree) in c:\build\Remotion\working\Relinq\Core\Parsing
\ExpressionTreeVisitors\PartialEvaluatingExpressionTreeVisitor.cs:
line 48

[Fabian Schmied]

Hi Alex,

Thanks for your detailed report (with repro!). The reason for the
exception is that re-linq's partial evaluation (written for .NET 3.5)
doesn't correctly deal with .NET 4's additional expression types. For
details, see "https://www.re-motion.org/jira/browse/RM-4465".

I'll implement a fix to be included in re-linq build 1.13.128 (due
Friday). You can probably get the fix sooner if needed, by checking
out and compiling re-linq's source code from the trunk.

Note, that re-linq will not (for now) be able to partially evaluate
any dynamic expressions. For example, in the following query, re-linq
will not inline x.Member's value as a constant if x.Member is a
dynamic expression (whereas it will inline it if x.Member is an
ordinary expression):

from item in source
where item == x.Member
select item

Regards,
Fabian

> --
> You received this message because you are subscribed to the Google Groups "re-motion Users" group.
> To post to this group, send email to re-moti...@googlegroups.com.
> To unsubscribe from this group, send email to re-motion-use...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/re-motion-users?hl=en.
>
>

[Alex Norcliffe]

Great response, thanks!

In the meantime, after sending the mail I went with an interim solution which is working OK. I'll explain a little in case there's any feedback.

The only reason I even need to analyse a tree referring to dynamic members is because I'm enabling Linq support for our CMS (Umbraco 5). Our primary model is a key-value bag not dissimilar from XElements but we're using dynamics to allow the following to be equivalent in templates:

var staticResult = Model.Attribute<string>("title")

var dynamicResult = DynamicModel.title

As you can appreciate, in a linq query, attribute bags get hairy quickly. So to keep the bar low, we allow for two additional styles:

QueryAll().Where(x => x.title == "news") // where x is dynamic

QueryAll().Where("title == \"news\"") // the string is parsed by the infamous ExpressionParser modified to support dynamics if it can't find the member 'title' on 'object'

in addition to QueryAll().Where(x => x.Attribute<string>("title") == "news")

All of the above need to parse down to the same expression tree, ultimately, before being passed to our data layer, Umbraco Hive. To avoid data provider authors needing to have a direct dependency on Relinq, or more specifically to avoid them implementing an entire Linq provider, I'm using Relinq to rewrite the expression into our own QueryDescription which has a very simple tree of custom FieldPredicateExpressions, together with info about the return type etc. (even simpler than the QueryModel in Relinq - it's a targetted subset of Linq that we need to support)

Anyway, long story short I've gone with referencing a dumb static method in the expression tree wherever a member was "not found" on object:

public static class DynamicMemberMetadata
    {
        public static object GetMember(string name, object instance = null)
        {
            throw new UnreachableException();
        }
 
        public static readonly MethodInfo GetMemberMethod = ExpressionHelper.GetMethodInfo(() => GetMember(null, null));

}

In the expression when visiting Relinq's QueryModel, I now see expressions like this: (GetMember("colour", null) == Convert("orange")) which is plenty enough "metadata" for me to rewrite into "field 'colour', 'equals', string 'orange'"

I had to make one mod, which was to re-implement PartialEvaluatingblahblah and prevent it from compiling expressions if it matches the above, since it's the metadata expressed in it that I actually want, but other than that it's working OK so far.

It's pretty much identical to what DynamicExpression gives me, afaik.

Anyways, feedback would be grand if any of that made sense!

Cheers,

Alex

Alex Norcliffe

Lead Architect, Umbraco 5

@alex_norcliffe on Twitter, blogging at boxbinary.com

Summer 2011 - We're driving an ambulance to Mongolia for charity, please sponsor us with cash, books, anything you can! More details here boxbinary.com/2011/06/drive-ambulance-london-mongolia-raise-money-go-help-charity/

[Fabian Schmied]

Hi Alex,

I've just committed the fix I talked about earlier. It avoids the
InvalidOperationException by making re-linq's partial evaluator just
ignore DynamicExpressions (and other .NET 4 expressions).

About your own workaround: If I understand you correctly, your
workaround circumvents the problem by just replacing the dynamic
expression with a "dummy" method call prior to partial evaluation,
right?

That's a good idea, but, as you said yourself, you then need to adapt
partial evaluation to ignore your metadata method. I've got an even
better idea: You could replace the DynamicExpression with a custom
metadata expression (i.e., not a method call that represents your
metadata, but an expression node that represents your metadata), which
re-linq will then know how to handle (and it will never partially
evaluate it). That way you could avoid patching the partial evaluator.

Here's how it should work:

public class DynamicMemberExpression : ExtensionExpression
{
private string _name;
private Expression _instance;

public DynamicMemberExpression (string name, Expression instance = null)
{
_name = name;
_instance = instance;
}

protected override Expression VisitChildren (ExpressionTreeVisitor visitor)
{
var newInstance = visitor.VisitExpression (_instance);

if (newInstance !=_instance)
return new DynamicMemberExpression (_name, newInstance);
else
return this;
}

public override Expression Accept (ExpressionTreeVisitor visitor)
{
var specificVisitor = visitor as IDynamicMemberExpressionVisitor;
if (specificVisitor != null)
return specificVisitor.VisitDynamicMemberExpression (this);
else
return base.Accept (visitor);
}
}

public interface IDynamicMemberExpressionVisitor
{
Expression VisitDynamicMemberExpression (DynamicMemberExpression
dynamicMemberExpression);
}

If you're using a custom expression visitor to translate the
expressions within the QueryModel into your own QueryDescription, just
implement IDynamicMemberExpressionVisitor on it.

Of course, if you don't mind the patching, you can just stick to your
existing solution :)

One more word about how you replace the DynamicExpression with your
method call (or, maybe, my DynamicMemberExpression): re-linq sports a
feature called "expression transformers" that would be nicely suited
to perform that task. Maybe you're already using it; if not, take a
look at this blog post:
"https://www.re-motion.org/blogs/mix/2011/04/29/re-linq-customizability-explained/".

Cheers,
Fabian

[Alex Norcliffe]

Thanks Fabian, that's a great idea. Yep you're correct, my method call is just a dummy one.

The expression I'm rewriting to does have custom extension expressions:

FieldPredicateExpression containing a FieldSelectorExpression (denoting the field name) and FieldValueExpression (denoting a ValuePredicateType (Equals etc.) and the Value)

I also have SchemaPredicateExpression which does something similar to express our notion of "content type" in an expression too, e.g. the name of the content type.

However I tried punting an expression tree with those in it to Relinq and I ended up with a stackoverflow after I'd allowed the nodes to be reduced, I wasn't really aware if the path I was taking had promise so I just backtracked and went with the workaround. I'll take a look at your example and the latest drop when it's built!

Cheers,

Alex Norcliffe

Lead Architect, Umbraco 5

@alex_norcliffe on Twitter, blogging at boxbinary.com

[Fabian Schmied]

Hi Alex,

Great, let me know if you need any help with my sample.

Regards,
Fabian

On Wed, Nov 2, 2011 at 12:28 PM, Alex Norcliffe