Distributed Cache

[SimonSays]

Hello everybody,

I'm trying to use the distributed cache an copying files into it works
fine - but reading files seems more complicated: when I distribute a
file stored in the HDFS with the path /user/hadoop/testfile.csv,
hadoop generates following path in the cache-folder: [.../mapred/local/
taskTracker/archive] /user/hadoop/testfile.csv/testfile.csv.
So it creates a new folder with the filename that contains the
original testfile. Because of that, it is impossible to acces the file
- when I use DistributedCache.getLocalCacheFiles(), I'll get the path
"/user/hadoop/testfile.csv" wich leads to the folder on the local file
system but not to the file itself (wich is stored inside the folder,
having the same name).
Has anyone a solution for acessing files in the distributed cache?


Thank you,
Simon

[Erich Nachbar]

Had the same problem. Just check for null and use a different call
instead:

Path[] files = DistributedCache.getLocalCacheFiles
(hfp.getJobConf());
// if we run locall getLocalCacheFiles doesn't work - need to
use local versions. Don't ask me why...
if (files == null) {
URI[] uris = DistributedCache.getCacheFiles(hfp.getJobConf
());
files = new Path[uris.length];
int i = 0;
for(URI uri : uris) {
files[i++] = new Path(uri.getPath());

[SimonSays]

it works, thank you!