Factory to convert String cqquery to Query?

[psia...@gmail.com]

Nigall,

I am migrating 1.3.2 to 2.0.1, 

Is it possible to execute query as String? e.g. 

String queryString = "lessThan(SalesOrder.CREATED_DATE,${1})";

and some util that auto converts this string queryString'  to Query object?

e.g. 

Query q = SomeFactory.toQuery(queryString);

[psia...@gmail.com]

Nevermind... found it !

com.googlecode.cqengine.query.parser.common.QueryParser<SalesOrder> qq = new CQNativeParser<>(SalesOrder.class);

Query q = qq.parse("lessThan(SalesOrder.ID,2344");

[Niall Gallagher]

Hi psianil,

The official story is that CQEngine 2.0 does not have a string-based query parser.

However, as you have seen there is an incomplete query parser. BUT it wasn't finished in time for the CQEngine 2.0 release, so for now it's completely unsupported, and is not able to parse all queries. It might be finished in a later version of CQEngine.

There has been some discussion on this list before about this parser. IIRC the discussion was about using a library like antlr or something. So maybe I would use that approach.

If you have any experience in this area I'd appreciate patches!

Niall

--
-- You received this message because you are subscribed to the "cqengine-discuss" group.
http://groups.google.com/group/cqengine-discuss
---
You received this message because you are subscribed to the Google Groups "cqengine-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cqengine-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[psia...@gmail.com]

Niall,

Yes fount that... I guess i have to stick to reflection and codegen based architecture to generate object 'Query' from String.

btw, i had adopted cqengine since over 3+ years. Its working great!

Thanks,

Anil

[Niall Gallagher]

Hi Anil,

Okay cool I'm glad you have another solution in the meantime.

I think one of the next things I'll look at, is finishing that parser :)

Cool yeah you were an early adopter for sure! 3 years is such a long time in software. Thanks for the long time support!!!

Best regards,

Niall

[psia...@gmail.com]

Hi Niall,

Look into this dynamic class generation stuff... https://github.com/robaustin123/blog/blob/master/src/main/java/JavaDynamicClassCreation.java

Here is how i am implementing it. It does not need to use Antlr....

package com.generator;

import static java.util.Collections.singletonList;

import static javax.tools.JavaFileObject.Kind.SOURCE;

import java.io.ByteArrayOutputStream;

import java.io.IOException;

import java.io.OutputStream;

import java.lang.reflect.Field;

import java.net.URI;

import java.net.URISyntaxException;

import javax.tools.FileObject;

import javax.tools.ForwardingJavaFileManager;

import javax.tools.JavaFileManager;

import javax.tools.JavaFileObject;

import javax.tools.SimpleJavaFileObject;

import javax.tools.ToolProvider;

import sun.misc.Unsafe;

public class DynamicClassCreationTest {

public static final void main(String... args) throws ClassNotFoundException, URISyntaxException, NoSuchFieldException, InstantiationException, IllegalAccessException {

simpleTest();

String dept = "Marketing";

cqengineQuery("and(and(lessThan(SalesOrder.SHIP_DATE, today), not(has(SalesOrder.SHIP_DATE))), equal(SalesOrder.DEPT, \"" + dept + "\"));");

}

static void cqengineQuery(String queryString) throws ClassNotFoundException, IllegalAccessException, InstantiationException, NoSuchFieldException, URISyntaxException {

final String className = "Query1";

final String path = "gp.core.dynamic.query";

final StringBuilder source = new StringBuilder();

source.append("package " + path + ";");

source.append("import static com.googlecode.cqengine.query.QueryFactory.*;");

source.append("import com.model.SalesOrder;");

source.append("import java.util.Date;");

source.append("import com.googlecode.cqengine.query.Query;");

//

source.append("public class " + className + " {\n");

source.append(" public Query getQuery() {\n");

source.append(" Date today = new Date();");

source.append("     return " + queryString);

source.append(" }\n");

source.append("}\n");

final Object o = stringToQuery(source, className, path);

System.out.println(o);

}

static void simpleTest() throws ClassNotFoundException, IllegalAccessException, InstantiationException, NoSuchFieldException, URISyntaxException {

final String className = "HelloWorld";

final String path = "com.bounded.buffer";

final StringBuilder source = new StringBuilder();

source.append("package " + path + ";");

source.append("public class " + className + " {\n");

source.append(" public String toString() {\n");

source.append("     return \"HelloWorld - Java Dynamic Class Creation was written by Rob Austin\";");

source.append(" }\n");

source.append("}\n");

final Object o = stringToQuery(source, className, path);

System.out.println(o);

}

static Object stringToQuery(final StringBuilder source, final String className, final String path) throws ClassNotFoundException, IllegalAccessException, InstantiationException,

URISyntaxException, NoSuchFieldException {

final String fullClassName = path.replace('.', '/') + "/" + className;

// A byte array output stream containing the bytes that would be written to the .class file

final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

final SimpleJavaFileObject simpleJavaFileObject = new SimpleJavaFileObject(URI.create(fullClassName + ".java"), SOURCE) {

@Override

public CharSequence getCharContent(boolean ignoreEncodingErrors) {

return source;

}

@Override

public OutputStream openOutputStream() throws IOException {

return byteArrayOutputStream;

}

};

@SuppressWarnings("unchecked")

final JavaFileManager javaFileManager = new ForwardingJavaFileManager(ToolProvider.getSystemJavaCompiler().getStandardFileManager(null, null, null)) {

@Override

public JavaFileObject getJavaFileForOutput(Location location, String className, JavaFileObject.Kind kind, FileObject sibling) throws IOException {

return simpleJavaFileObject;

}

};

ToolProvider.getSystemJavaCompiler().getTask(null, javaFileManager, null, null, null, singletonList(simpleJavaFileObject)).call();

final byte[] bytes = byteArrayOutputStream.toByteArray();

// use the unsafe class to load in the class bytes

final Field f = Unsafe.class.getDeclaredField("theUnsafe");

f.setAccessible(true);

final Unsafe unsafe = (Unsafe) f.get(null);

@SuppressWarnings("deprecation")

final Class aClass = unsafe.defineClass(fullClassName, bytes, 0, bytes.length);

final Object o = aClass.newInstance();

return o;

[Niall]

Hi Anil,

Sorry for my delay replying! I have nearly finished the support for string-based queries now.

Two string-based query languages are supported actually, and I used Antlr 4 to implement it: 

• SQL -- regular SQL queries which CQEngine will parse and run on the collection.
• CQN (CQEngine Native) -- string queries which use the same Java syntax that CQEngine currently uses for its programmatic queries, except as strings.
  

Here is an example of an SQL query on a collection (source here):

    public static void main(String[] args) {
        SQLParser<Car> parser = SQLParser.forPojoWithAttributes(Car.class, createAttributes(Car.class));
        IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>();
        cars.addAll(CarFactory.createCollectionOfCars(10));

        ResultSet<Car> results = parser.retrieve(cars, "SELECT * FROM cars WHERE (" +
                                        "(manufacturer = 'Ford' OR manufacturer = 'Honda') " +
                                        "AND price <= 5000.0 " +
                                        "AND color NOT IN ('GREEN', 'WHITE')) " +
                                        "ORDER BY manufacturer DESC, price ASC");

        for (Car car : results) {
            System.out.println(car); // Prints: Honda Accord, Ford Fusion, Ford Focus
        }
    }

Above, the names of the "columns" in the SQL query, are actually fields in the Java objects. The example calls AttributeBytecodeGenerator.createAttributes() to generate attributes for fields in the objects automatically, and registers them with the SQL parser.

Here something similar (source here), but written in CQN syntax instead:

    public static void main(String[] args) {
        CQNParser<Car> parser = CQNParser.forPojoWithAttributes(Car.class, createAttributes(Car.class));
        IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>();
        cars.addAll(CarFactory.createCollectionOfCars(10));

        ResultSet<Car> results = parser.retrieve(cars,
                                        "and(" +
                                            "or(equal(\"manufacturer\", \"Ford\"), equal(\"manufacturer\", \"Honda\")), " +
                                            "lessThanOrEqualTo(\"price\", 5000.0), " +
                                            "not(in(\"color\", GREEN, WHITE))" +
                                        ")");

        for (Car car : results) {
            System.out.println(car); // Prints: Ford Focus, Ford Fusion, Honda Accord
        }
    }

The Antlr 4 grammars for SQL and CQN can be used as a reference for writing string queries which CQEngine will be able to parse.

The dialect of SQL dialect used, is a subset of regular SQL. The CQN language supports some additional types of queries which SQL does not support such as matchesRegex() and isContainedIn().

Right now, the SQL support is finished. There are a few minor things to finish for CQN though. These languages will be supported/released in CQEngine 2.1, in the next couple of weeks.

Niall

[PSI A]

Hi Niall,

It was great work!, I did attempted Antlr expression to build parser, but it always ran into complex graph traversing algorithm, which were failing in cases overlooked and performance too. Did you checked some benchmarks and possible test scenarios as well so that i can drop my dynamic class generation model and incorporate this new feature? .... just wanted to get confidence from you :)

Thanks,
Anil

[Niall]

Hi Anil,

The query parser translates SQL or CQN into regular CQEngine Query objects, so once the query is parsed, it will then be evaluated in exactly the same way and with the same performance as CQEngine queries which were constructed programmatically.

Parsing should typically be sub-millisecond.

I have roughly benchmarked it yes. On my current 2.8GHz machine, parsing a string SQL query of average complexity into a CQEngine query, takes 47793 nanoseconds (0.05 milliseconds).

The parsers have full test coverage. For example, see unit tests for the SQL parser here.

The next version of CQEngine will be released any day now! :)

I know I have said that for several weeks now. But as you can see there is some unrelated work going into trunk as well, which I need included for my work as well.

At this point you would probably be better off to wait for the next official release (i.e. don't bother building from trunk). I will be releasing a new version before the end of this month.

HTH,

Niall

[Sam Bishop]

On Thu, Feb 4, 2016 at 1:59 PM, Sri <yalaman...@gmail.com> wrote:

Niall,

I am trying to execute the test CQNQueryDemo.java from your samples and its giving me the below error. I have included the below jars in my classpath. Can you please advise whether I am missing any thing here?

cqengine-2.1.1-all.jar

equalsverifier-1.7.2.jar

guava-testlib-18.0.jar

junit-dataprovider-1.9.3.jar

mockito-core-1.9.5.jar

javassist-3.18.2-GA.jar

Did you forget to include the error?  I didn't see it in your email.

[Sri]

Hi Niall,

I am trying to test your example CQNQueryDemo.java and getting the below error. Can you please advise? All these jars are in the classpath.  cqengine-2.1.1-all.jar;equalsverifier-1.7.2.jar; guava-testlib-18.0.jar; junit-dataprovider-1.9.3.jar; mockito-core-1.9.5.jar ; javassist-3.18.2-GA.jar

Exception in thread "main" java.lang.IllegalStateException: Failed to create attribute for field: final int com.googlecode.cqengine.testutil.Car.carId

at com.googlecode.cqengine.codegen.AttributeBytecodeGenerator.createAttributes(AttributeBytecodeGenerator.java:98)

at com.googlecode.cqengine.examples.codegen.GenerateAttributeByteCode.main(GenerateAttributeByteCode.java:42)

Caused by: java.lang.ExceptionInInitializerError

at com.googlecode.cqengine.codegen.AttributeBytecodeGenerator.generateSimpleAttribute(AttributeBytecodeGenerator.java:372)

at com.googlecode.cqengine.codegen.AttributeBytecodeGenerator.generateSimpleAttributeForField(AttributeBytecodeGenerator.java:124)

at com.googlecode.cqengine.codegen.AttributeBytecodeGenerator.createAttributes(AttributeBytecodeGenerator.java:73)

... 1 more

Caused by: java.lang.RuntimeException: cannot initialize ClassPool

at com.googlecode.cqengine.lib.javassist.ClassPool.<clinit>(ClassPool.java:97)

... 4 more

Caused by: java.lang.NoSuchMethodException: java.lang.ClassLoader.com.googlecode.cqengine.lib.defineClass(java.lang.String, [B, int, int)

at java.lang.Class.getDeclaredMethod(Class.java:1956)

at com.googlecode.cqengine.lib.javassist.ClassPool$1.run(ClassPool.java:80)

at java.security.AccessController.doPrivileged(Native Method)

at com.googlecode.cqengine.lib.javassist.ClassPool.<clinit>(ClassPool.java:77)

... 4 more

Thanks,Sri

[Niall Gallagher]

Looks like you are using the cqengine shaded jar whose name ends with "-all.jar". That is a fat-jar build of cqengine which already includes all dependencies.

Either: use the normal cqengine jar and make sure you include all of cqengine's dependency jars on your classpath too OR continue to use that shaded jar but don't include any cqengine dependency jars on your classpath.

Right now I guess you have duplicate classes on your classpath.

Or better yet, just use maven. Managing jars by hand is error prone and so "2004" :)

Sent from my Android