Stack overflow error in findRunnableTaskClusterRoots

[Pushkar Khadilkar]

Hi,

We are trying to run a simplified q15_top_supplier.hive query on Hyracks (through Hivesterix).

Rather than submitting the queries as it is, we created one large query (effectively replacing temp table references in from clauses with INSERT OVERWRITES for those tables).

Upon submitting the query to cluster, we found that it throws stack overflow exception in findRunnableTaskClusterRoots method.

We verified that the DAG submitted to cluster is same as what we expect from original query.

I have attached:

1. q15_simpliefied.hive      : q15 hive query with certain operators removed. (The  modified query is syntactically correct).
2. q15_simplified_plan.txt  : Plan for simplified q15 query.
3. q15_jobspec.txt              : JobSpecification object toString()
   
4. q15_operator_DAG.pdf     : Operator DAG submitted to Hyracks (inferred from jobspec details) (also pasted below)

Is the DAG incorrect ? Upon debugging on server, we found that findRunnableTaskClusterRoots method takes a candidate parameter. For one of the calls, this parameter had itself as its dependency. It is not clear to me how this dependency was created from DAG. Does Hyracks use any other information apart from DAG to construct dependency information ?

Any help in this regard is highly appreciated.

Thanks,
Pushkar Khadilkar

[Pushkar Khadilkar]

Hi,

The exception stack trace was:

Exception in thread "Thread-1" java.lang.StackOverflowError
    at java.util.ArrayList$Itr.<init>(ArrayList.java:780)
    at java.util.ArrayList$Itr.<init>(ArrayList.java:780)
    at java.util.ArrayList.iterator(ArrayList.java:774)
    at edu.uci.ics.hyracks.control.cc.scheduler.JobScheduler.findRunnableTaskClusterRoots(JobScheduler.java:103)
    at edu.uci.ics.hyracks.control.cc.scheduler.JobScheduler.findRunnableTaskClusterRoots(JobScheduler.java:120)
    at edu.uci.ics.hyracks.control.cc.scheduler.JobScheduler.findRunnableTaskClusterRoots(JobScheduler.java:106)
    at edu.uci.ics.hyracks.control.cc.scheduler.JobScheduler.findRunnableTaskClusterRoots(JobScheduler.java:106)
    at edu.uci.ics.hyracks.control.cc.scheduler.JobScheduler.findRunnableTaskClusterRoots(JobScheduler.java:106)
    at edu.uci.ics.hyracks.control.cc.scheduler.JobScheduler.findRunnableTaskClusterRoots(JobScheduler.java:106)
    at edu.uci.ics.hyracks.control.cc.scheduler.JobScheduler.findRunnableTaskClusterRoots(JobScheduler.java:106)
   .... many more findRunnableTaskClusterRoots calls from same location

Thanks,
Pushkar Khadilkar

[Vinayak Borkar]

This stack overflow means that the Task Cluster Graph contains a cycle.
Can you provide a printout of the Task Cluster graph for this job?

The system will print it when you increase the logging level.

Thanks,
Vinayak

> 1. */q15_simpliefied.hive/* : q15 hive query with certain

> operators removed. (The modified query is syntactically correct).

> 2. *q15_simplified_plan.txt* : Plan for simplified q15 query.
> 3. *q15_jobspec.txt* : JobSpecification object toString()
> 4. *q15_operator_DAG.pdf* : Operator DAG submitted to Hyracks

> (inferred from jobspec details) (also pasted below)
>
> Is the DAG incorrect ? Upon debugging on server, we found that
> findRunnableTaskClusterRoots method takes a candidate parameter. For
> one of the calls, this parameter had itself as its dependency. It is
> not clear to me how this dependency was created from DAG. Does
> Hyracks use any other information apart from DAG to construct
> dependency information ?
>
>

> <https://lh6.googleusercontent.com/-HHXTNV4iHW8/UtuqecLhrdI/AAAAAAAAAKA/fBJw7XeGM94/s1600/q15_Operator_DAG.jpg>

>
>
> Any help in this regard is highly appreciated.
>
> Thanks,
> Pushkar Khadilkar
>

> --
> You received this message because you are subscribed to the Google
> Groups "hyracks-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hyracks-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Pushkar Khadilkar]

Hi,

Thanks for your reply.

I tried changing the log level. It did not work. I am not sure I am changing the right logging file.

Cluster.properties has option for logging as

-Djava.util.logging.config.file=logging.properties

I could not find logging file at that location. I copied one logging.properties to that location (because all other logging.properties were in test folders) with appropriate settings.

If you could mention the class/object name, I could also watch its value during debugging.

Thanks,

Pushkar Khadilkar