LIMIT clauses in Lingual Cascading

[Michael Peterson]

I've been able to execute limit clauses in the lingual shell and those work.  Things like:

    select t.BAR from MDP.MYTABLE t LIMIT 1

But when I try it in Lingual (using Cascading flows), it fails.  Am I doing something wrong or are LIMIT clauses not allowed?  Is there another way to get this to work?  I don't need this often, but I have a particular use case where it would be very helpful to get this working.  I am able to do it in Hive, for example.

Here's the code:

    String sql = "select t.BAR from \"MDP\".\"MYTABLE\" t LIMIT 1";
   
    final String[] mytabCols = new String[] {"FOO","BAR","BAZ","QUX"};
    final Type[] mytabTypes  = new Type[] {String.class, String.class, String.class, String.class};

    Fields mytabFields = new Fields(mytabCols, mytabTypes);
    SQLTypedTextDelimited mytabScheme = new SQLTypedTextDelimited(mytabFields, "|", "\"",
        false,  /*header*/
        true,   /*strict*/
        true);  /*safe*/

    Tap<?,?,?> mytabTap  = new FileTap(mytabScheme, "mdp/mytable", SinkMode.KEEP);
    Tap<?,?,?> resultsTap = new FileTap(new TextDelimited(false, ",", "\""), "mdp/myoutput", SinkMode.REPLACE);

    SQLPlanner sqlPlanner = new SQLPlanner().setSql(sql);
    FlowDef flowDef = FlowDef.flowDef()
        .setName("myquery")
        .addSource("MDP.MYTABLE", mytabTap)
        .addSink("myoutput", resultsTap)
        .addAssemblyPlanner(sqlPlanner);

    Flow<?> flow = new LocalFlowConnector().connect(flowDef);
    flow.complete();

(this example uses local mode, but I get the same error in Hadoop mode)

and here's the error:

2014-02-27 10:57:34,137 INFO  [main] property.AppProps (AppProps.java:getAppID(162)) - using app.id: AF620CE2915A453CAC512E0CE2FC1422
2014-02-27 10:57:34,231 INFO  [main] tap.TapSchema (TapSchema.java:addTapTableFor(125)) - adding table on schema: MDP, table: MYTABLE, fields: 'FOO', 'BAR', 'BAZ', 'QUX' | String, String, String, String, identifier: mdp/mytable
2014-02-27 10:57:34,233 INFO  [main] tap.TapSchema (TapSchema.java:addTapTableFor(125)) - adding table on schema: null, table: myoutput, fields: UNKNOWN, identifier: mdp/myoutput
Exception in thread "main" cascading.flow.planner.PlannerException: could not build flow from assembly: [net.hydromatic.linq4j.EnumerableDefaults$12 cannot be cast to cascading.lingual.optiq.enumerable.CascadingFlowRunnerEnumerable]
    at cascading.flow.planner.FlowPlanner.handleExceptionDuringPlanning(FlowPlanner.java:576)
    at cascading.flow.local.planner.LocalPlanner.buildFlow(LocalPlanner.java:108)
    at cascading.flow.local.planner.LocalPlanner.buildFlow(LocalPlanner.java:40)
    at cascading.flow.FlowConnector.connect(FlowConnector.java:459)
    at com.truven.di.Stage2Joins.limit(Stage2Joins.java:69)
    at com.truven.di.Stage2Joins.main(Stage2Joins.java:37)
Caused by: java.lang.ClassCastException: net.hydromatic.linq4j.EnumerableDefaults$12 cannot be cast to cascading.lingual.optiq.enumerable.CascadingFlowRunnerEnumerable
    at cascading.lingual.flow.SQLPlanner.resolveTails(SQLPlanner.java:113)
    at cascading.flow.planner.FlowPlanner.resolveAssemblyPlanners(FlowPlanner.java:150)
    at cascading.flow.planner.FlowPlanner.resolveTails(FlowPlanner.java:137)
    at cascading.flow.local.planner.LocalPlanner.buildFlow(LocalPlanner.java:81)
    ... 4 more

[Michael Peterson]

I just tried it using the JDBC driver and the SQL parses, but the LIMIT clause is completely ignored.  It still outputs all the rows, not the number I specified in the LIMIT clause.

So I see three different behaviors with LIMIT clauses:

1. works in lingual shell (uses catalog)
2. parsers with JDBC driver but LIMIT clause is ignored (uses catalog)
3. doesn't parse with Lingual / Cascading (does not use catalog)

-Michael

[Chris K Wensel]

we will need to look deeper. this could be an Optiq issue where the limit isn't getting pushed down or ignored. Or just a bug in our limit code. this is when abstractions starts to suck.

ckw

--
You received this message because you are subscribed to the Google Groups "Lingual User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lingual-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Chris K Wensel

ch...@concurrentinc.com

http://concurrentinc.com

[Joe Posner]

Dug a bit deeper into this. Chirs' guess about this being an issue
with the Lingual and Optiq interactions.

I'll work on a fix for this, but we don't have an ETA for it.

[Joe Posner]

Accidentally delete a phase when I hit "send"

Should have read:
Chirs' guess about it being Lingual and Optiq interactions was correct.

I'll work on a fix.

[Michael Peterson]

OK.  Thanks.  Should I file a issue or do you have that covered?

You received this message because you are subscribed to a topic in the Google Groups "Lingual User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lingual-user/JDlqJAUTNtM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lingual-user...@googlegroups.com.

[Joe Posner]

It's filed as https://github.com/Cascading/lingual/issues/19