Building dynamic expression tree for Raven query

[Stif Labarque]

Hi all

I would like to build a dynamic expression tree as a where clause for my RavenDb query. The reason is that we want to be able to create where clauses for different fields without having to repeat code.

So far I've managed to build an expression tree for on specific field (see FilterLiteral) and I get the correct results. However, as soon as I want to make this dynamic (so that I could apply it to other fields) I get complaints from Lucene:

Cannot parse '( AND) AND': Encountered " <AND> "AND "" at line 1, column 2.....

Here's a simplified example of what I'm trying to do (in my real code I build the expression tree by nested foreaching over collections of search terms):

If I call FilterLiteral everything works like a charm, however calling FilterDynamic (which should yield the exact same result) causes the Lucene error mentioned above. Apparently Lucene has problems translating the InvocationExpression generated by calling the dynamic Func<LogMessage, string, bool> filterMethod.

What could I be doing wrong?

public class LogMessages_Index : AbstractIndexCreationTask<LogMessage>

{

public LogMessages_Index()

{

Map = logMessages => logMessages.Select(

lm => new

{

lm.ApplicationName,

lm.TimeStamp

});

Analyzers.Add(lm => lm.ApplicationName, typeof(StandardAnalyzer).AssemblyQualifiedName);

}

}

public class Query

{

private IDocumentStore documentStore;

public Query(IDocumentStore documentStore)

{

this.documentStore = documentStore;

}

public LogMessage[] FilterDynamic()

{

using (IDocumentSession session = documentStore.OpenSession())

{

IRavenQueryable<LogMessage> query = session.Query<LogMessage, LogMessages_BasicIndex>();

query = BuildQuery(query, (message, s) => message.ApplicationName.StartsWith(s));

List<LogMessage> logMessages = query.OrderByDescending(lm => lm.TimeStamp).ToList();

Debug.WriteLine(string.Format("Found {0} matching logmessages", logMessages.Count));

return logMessages.ToArray();

}

}

private static IRavenQueryable<LogMessage> BuildQuery(IRavenQueryable<LogMessage> query, Func<LogMessage, string, bool> filterMethod)

{

Expression<Func<LogMessage, bool>> subWhereClause = null;

Expression<Func<LogMessage, bool>> expression11 = lm => filterMethod(lm, "SDS");

subWhereClause = expression11;

Expression<Func<LogMessage, bool>> expression12 = lm => filterMethod(lm, "Que");

subWhereClause = Expression.Lambda<Func<LogMessage, bool>>(Expression<Func<LogMessage, bool>>.AndAlso(subWhereClause.Body, expression12.Body), subWhereClause.Parameters);

Expression<Func<LogMessage, bool>> expression13 = lm => filterMethod(lm, "Serv");

subWhereClause = Expression.Lambda<Func<LogMessage, bool>>(Expression<Func<LogMessage, bool>>.AndAlso(subWhereClause.Body, expression13.Body), subWhereClause.Parameters);

query = query.Where(subWhereClause);

return query;

}

public LogMessage[] FilterLiteral()

{

using (IDocumentSession session = documentStore.OpenSession())

{

IRavenQueryable<LogMessage> query = session.Query<LogMessage, LogMessages_BasicIndex>();

query = BuildLiteralQuery(query);

List<LogMessage> logMessages = query.OrderByDescending(lm => lm.TimeStamp).ToList();

Debug.WriteLine(string.Format("Found {0} matching logmessages", logMessages.Count));

return logMessages.ToArray();

}

}

private static IRavenQueryable<LogMessage> BuildLiteralQuery(IRavenQueryable<LogMessage> query)

{

Expression<Func<LogMessage, bool>> subWhereClause = null;

Expression<Func<LogMessage, bool>> expression11 = lm => lm.ApplicationName.StartsWith("SDS");

subWhereClause = expression11;

Expression<Func<LogMessage, bool>> expression12 = lm => lm.ApplicationName.StartsWith("Quer");

subWhereClause = Expression.Lambda<Func<LogMessage, bool>>(Expression<Func<LogMessage, bool>>.AndAlso(subWhereClause.Body, expression12.Body), subWhereClause.Parameters);

Expression<Func<LogMessage, bool>> expression13 = lm => lm.ApplicationName.StartsWith("Serv");

subWhereClause = Expression.Lambda<Func<LogMessage, bool>>(Expression<Func<LogMessage, bool>>.AndAlso(subWhereClause.Body, expression13.Body), subWhereClause.Parameters);

query = query.Where(subWhereClause);

return query;

}

}

[Kijana Woodard]

What is the user experience like?

Are they picking fields to query on and providing a single value? Mulitple values?

[Stif Labarque]

We want our users to be able to specify multiple search terms for the fields they choose to query on.

So in this example for instance I put a full text index on LogMessage.ApplicationName. I want users to be able look for LogMessages coming from an application called "SDS Query Service". But the user only knows that he's looking for some application name that contains words starting with "SD" AND "Quer" AND "Serv" (I know it's a bit of stupid example, but there are use cases where we actually need this logic)

[Kijana Woodard]

Have you looked at Facets to present the user with "application names"? They might be helpful for that case.

I would think with the the right analyzer and _maybe_ using the Lucene syntax, you should be able to achieve what you want without expression trees.

Is the scenario "find a particular app by partial name and page through the log messages"?

[Oren Eini (Ayende Rahien)]

Why do that?

Please use the session.Advanced.LuceneQuery(), which allows you to handle things dynamically. 

[Stif Labarque]

Thanks! I was trying to avoid Lucene queries because the downside of using session.Advanced.LuceneQuery() is that you lose the ability of using LINQ. Building a full Lucene query string is a bit more cumbersome than being able to use IRavenQuerable and LINQ.

[Troy]

I find LuceneQuery to be very nice.. it has lambda expressions for most things. The power to build complex queries is unbeatable for situations like these. I went down building custom predicates and found it was so much easier with LuceneQuery.

Perhaps I am not understanding what you are doing when you say building a Lucene query string...

[Chris Marisic]

Agreed. Once you get to a custom predicate, it's time to abandon the linq api and work with the lucene one.

[Matt Johnson]

Lucene Query is best in this case, yes. But since raven's
session.Query() exposes an IQueryable, you *could* perform other
operation on it than just pure LINQ. For example, there's a library
called System.Linq.Dynamic that although unofficial, was developed by
Microsoft. Scott Gu has a blog post about it
http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx
And you can still find it on NuGet at https://www.nuget.org/packages/System.Linq.Dynamic

I was able to wire this to Raven quite easily, and use it for simple
string expressions around filtering and sorting.

Still, it's probably best to use LuceneQuery, but I just wanted to
point out the possibilities.