Almost there: 2 more queries to tackle! :-)

[Bart Vandewoestyne]

After having some difficulties in getting the benchmark up and running, I'm almost there :-)

I started my benchmark run yesterday with the command:

bart@sandy-quad-1:~/Big-Bench$ ./scripts/bigBench runBenchmark -m 4 -f 1 -s 2

and it seems like everything went fine, except for query 9 and 20.  The showErrors module gives me the following errors:

https://dl.dropboxusercontent.com/u/32340538/Big-Bench/showErrors.txt

My full logs are available from https://dl.dropboxusercontent.com/u/32340538/Big-Bench/logs-20140918-222156.zip

Does anybody have an idea why query 9 and 20 are still failing?

As soon as I get these fixed, I will do a benchmark run with scale factor 500.

Regards,
Bart

[Bart Vandewoestyne]

On Friday, September 19, 2014 10:48:24 AM UTC+2, Bart Vandewoestyne wrote:

[...]

Does anybody have an idea why query 9 and 20 are still failing?

I've investigated query 9, and my educated guess is that it is suffering the map join problem described in section 'optimization for joins' in  hiveSettings.sql.

To overcome that problem, I tried

set hive.mapjoin.smalltable.filesize=1;

to avoid that the common join is converted to a map join, but that did not work.

Any other suggestions?

[Bart Vandewoestyne]

On Friday, September 19, 2014 2:11:39 PM UTC+2, Bart Vandewoestyne wrote:

I've investigated query 9, and my educated guess is that it is suffering the map join problem described in section 'optimization for joins' in  hiveSettings.sql.

To overcome that problem, I tried

set hive.mapjoin.smalltable.filesize=1;

to avoid that the common join is converted to a map join, but that did not work.

I find it strange that this does not work.  The documentation online at https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties writes:

hive.smalltable.filesize
hive.mapjoin.smalltable.filesize

• Default Value: 25000000
• Added In: Hive 0.7.0 with HIVE-1642: hive.smalltable.filesize (replaced by hive.mapjoin.smalltable.filesize in Hive 0.8.1)
• Added In: Hive 0.8.1 with HIVE-2499: hive.mapjoin.smalltable.filesize

The threshold (in bytes) for the input file size of the small tables; if the file size is smaller than this threshold, it will try to convert the common join into map join.

but when setting hive.mapjoin.smalltable.filesize to 0 or 1, i still get

2014-09-19 02:15:23    Starting to launch local task to process map join;    maximum memory = 257949696
2014-09-19 02:15:25    Dump the side-table into file: file:/tmp/bart/hive_2014-09-19_14-14-45_643_6217887407085482903-1/-local-10006/HashTable-Stage-5/MapJoin-mapfile21--.hashtable
2014-09-19 02:15:25    Upload 1 File to: file:/tmp/bart/hive_2014-09-19_14-14-45_643_6217887407085482903-1/-local-10006/HashTable-Stage-5/MapJoin-mapfile21--.hashtable
2014-09-19 02:15:25    Dump the side-table into file: file:/tmp/bart/hive_2014-09-19_14-14-45_643_6217887407085482903-1/-local-10006/HashTable-Stage-5/MapJoin-mapfile11--.hashtable
2014-09-19 02:15:25    Upload 1 File to: file:/tmp/bart/hive_2014-09-19_14-14-45_643_6217887407085482903-1/-local-10006/HashTable-Stage-5/MapJoin-mapfile11--.hashtable
2014-09-19 02:15:25    Processing rows:    200000    Hashtable size:    199999    Memory usage:    118769408    percentage:    0.46
2014-09-19 02:15:26    Processing rows:    300000    Hashtable size:    299999    Memory usage:    151032040    percentage:    0.586
Execution failed with exit status: 3
Obtaining error information

Task failed!
Task ID:
  Stage-15

so it seems like the common join is still being converted to a map join...

Suggestions on how to solve this problem are still welcome.

[Bart Vandewoestyne]

On Friday, September 19, 2014 2:19:42 PM UTC+2, Bart Vandewoestyne wrote:

[...]

so it seems like the common join is still being converted to a map join...

Suggestions on how to solve this problem are still welcome.

OK.  It looks like I've solved the problem for query 9.  I created the file  queries/q09/hiveLocalSettings.sql with the content

set hive.auto.convert.join=false;

which seems to solve the problem.  The query runs fine now and gives me an integer as a result.

Now let's have a look at query 20 :-)

[Bart Vandewoestyne]

On Friday, September 19, 2014 10:48:24 AM UTC+2, Bart Vandewoestyne wrote:

Does anybody have an idea why query 9 and 20 are still failing?

As for query 20, i see the following exception being thrown in 'Step 3/6: Generating sparse vectors':

Error: java.lang.NumberFormatException: For input string: "\N"
    at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1241)
    at java.lang.Double.valueOf(Double.java:504)
    at org.apache.mahout.clustering.conversion.InputMapper.map(InputMapper.java:48)
    at org.apache.mahout.clustering.conversion.InputMapper.map(InputMapper.java:34)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

After some googling, I think this has to do with the fact that Hive is using the string constant "\N" to encode NULL values, while during the parsing of that 'value', a floating point value is expected.
It will probably require me some more googling to find out how to solve this.  If anybody in the meanwhile has an answer, feel free to let me know!

Regards,
Bart

[Michael Frank]

Hi Bart,

q09:
======================
Please see the following link for a fix:
https://groups.google.com/d/msg/big-bench/wghb2-GC7Y8/BfU4h2LGpVkJ
The preferred solution is solution 1) "assign more memory to the local! Hadoop JVM client"
As the other two will badly cripple your clusters performance.
I would suggest local memory sizes >= 1.5GB (we use 1.5GB and it works for us)

q20:
==========================
I have located and fixed the cause of this issue. (Update in github)
But i wonder why it surfaced now. Previous big-bench versions did not show this behavior. Maybe because of an updated and more correct database or a more recent hive version.
To retest this issue re-run q20 with:

./scripts/bigBench -q 20 runQuery

best regards,
Michael

[Bart Vandewoestyne]

On Monday, September 22, 2014 7:49:27 PM UTC+2, Michael Frank wrote:

q20:
==========================
I have located and fixed the cause of this issue. (Update in github)
But i wonder why it surfaced now. Previous big-bench versions did not show this behavior. Maybe because of an updated and more correct database or a more recent hive version.
To retest this issue re-run q20 with:

./scripts/bigBench -q 20 runQuery

I just wanted to inform you that q20 now runs fine.  I'm using the following hive version:

bart@sandy-quad-1:~$ hive --version
Hive 0.12.0-cdh5.1.2
Subversion git://ubuntu64-12-04-mk1/var/lib/jenkins/workspace/generic-package-ubuntu64-12-04/CDH5.1.2-Packaging-Hive-2014-08-25_19-47-00/hive-0.12.0+cdh5.1.2+375-1.cdh5.1.2.p0.2~precise/src -r 8e266e052e423af592871e2dfe09d54c03f6a0e8
Compiled by jenkins on Mon Aug 25 19:53:31 PDT 2014
From source with checksum 6cc3c71c3f8c2dfbf547fde9f3b23985

Kind regards,
Bart