DistCache behavior

[Alex Cozzi]

I am trying to establish whether scoobi uses correctly the distributed cache. The problem is that a single file /tmp/scoobi-b_xxx/scoobi-20130621-235229-RunSteps$-37a11197-eda2-4b26-aa39-f5c4ac7ba850/dist-objs/scoobi.mappers-scoobi-20130621-235229-RunSteps$-37a11197-eda2-4b26-aa39-f5c4ac7ba850(Mscr-1) was opened 90231 times by scoobi, and I suspect that each mapper is supposed to use the local cache but failed.

Looking at the code:

https://github.com/NICTA/scoobi/blob/master/src/main/scala/com/nicta/scoobi/impl/util/DistCache.scala?source=cc#L80

I notice that the DistCache will silently fall back to use the remote cache when the local cache fails. Do you think it would be a good idea to log in the stderr when the local cache lookup fails? It will help to track down the problem.

Alex

[Eric Torreborre]

Hi Alex,

My knowledge of the internals of Hadoop is not great so I'll appreciate any guidance you will give me on how to improve cache usage.

For now I have added a "scoobi.DistCache" WARNING when there are no local cache files (the new cdh4/cdh3 0.8.0-SNAPSHOT versions should be published in 30 minutes or so).

If you don't have time to investigate this issue can you please open a bug, I'll look into it (when I come back from holidays because I'm off next week).

Thanks,

Eric.

[Eric Torreborre]

Hi Alex,

I'm back from holidays but didn't have time unfortunately to have a look at this issue (I'm mostly working on #282 which is non-trivial). Did you find anything interesting?

Thanks,

Eric.

[Alex Cozzi]

Hi Eric,

thank you for adding the logging. So far we did not see the DistCache log being reported in our job on the cluster, but we are still looking into it.

I will keep you posted.

Alex