batch ingest -- how to set "outDir" or "tmpSegmentDir"? (no space left on device)

[Wayne Adams]

I'm trying to do batch ingest on 0.5.59, and there's one remaining default directory I don't know how to set.  I run out of space on /tmp after reduce completes.  The logs refer to "outDir", which I believe (am not sure) comes from injected property "tmpSegmentDir".  Is this correct, and either way, where exactly do I set this property?  Thanks!  -- Wayne

[Fangjin Yang]

Hi Wayne,

I believe you are seeing those logs in the IndexMerger? If not, can you let me know which file you see the outDir logs?

Also, are you running the HadoopDruidIndexerMain or using the indexing service? The only place that I know of that refers to tmpSegmentDir is in the YeOldePlumberSchool, which is called as part of the index task in druid 0.5.x. If you do have an indexing service running in 0.5.x, you can set the base working directory with "druid.indexer.baseDir". The default value is "tmp".

If you are running the standalone druid hadoop indexer, the outDir is the default temporary-file directory.

Let me know if this helps.

-- FJ

[Wayne Adams]

Hi Fangjin:

  Thanks, yes, that is where we are seeing the log messages.  I'm using HadoopDruidIndexerMain.  This evening we found the source of the issue, meaning the call to File.createTempFile(...), which I suppose kinda fixes the location in /tmp.  It sounds like we can avoid this issue by using the indexing service instead, and setting the "druid.indexer.baseDir" property, then?

Thanks again -- Wayne

[Eric Tschetter]

Wayne, you can also set the tmp dir that Java chooses to use.

-Djava.io.tmpdir=/place/to/use/instead

Is the magical incantation to include on the command line.

> --
> You received this message because you are subscribed to the Google Groups
> "Druid Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to druid-developm...@googlegroups.com.
> To post to this group, send email to druid-de...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/druid-development/5b94fbfa-9594-4fdb-a9b7-025ec21d2e47%40googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.

[Wayne Adams]

Thanks, Eric!  -- Wayne

[Wayne Adams]

Hi -- me again. 

I wasn't having very much success with the old-style HadoopDruidIndexerMain on 0.5, so I'm now on latest 0.6 and also switched to using the Indexing Service.  I'm now getting the same "no space left on device" issue, but I think the issue this time is that although I can start the overlord pointing to a different tmp directory, it appears that the task that it creates to do the ingest does not inherit that property.  So now I'm back to crashing as soon as /tmp fills up.  Is there any way around this?  Is anyone loading non-trivial-sized files into Druid with either of these ingest engines?  I'm trying to load a 17-GB file.  Thanks --

Wayne

[Eric Tschetter]

Wayne,

Yes, files of that size have been loaded. Have you tried setting
-Djava.io.tmpdir=/place/to/use/instead on the process that is having
issues?

It would be helpful if you could provide the configs for the process
that is having issues along with some log lines. Also, if you have
irc, it might be easier to jump in the #druid-dev channel on freenode
and talk through what's happening.

--Eric

> https://groups.google.com/d/msgid/druid-development/b887fcae-91c7-41d1-bf42-5f0cd9a8b6c3%40googlegroups.com.

[Gian Merlino]

You should be able to set druid.indexer.fork.property.java.io.tmpdir=/hey/what on your middle manager.

[Wayne Adams]

Thanks, Eric.

I started the overlord process with -Djava.io.tmpdir=/data1/druid/druid-tmp, where /data1 is a large partition, but everything seemed to go to /tmp.  Is it actually the overlord process that is writing to /tmp, or a spawned process?  I haven't checked source for 0.6 yet but if it's using File.createTempFile I would expect it would use my property.

A colleague suggested I mount a large partition on /tmp, which I just did (I'm running on EC2 so I can afford to be a little impulsive).  re: the config files, I might have to get with my client to do some cleanup before I could show them.

As far as logs are concerned, here's a bit of cleaned-up log.  First time I ran, I got the usual "no space left on device".  This last time I got a slightly different issue, when /tmp filled:

2014-01-08 04:02:11,965 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_hadoop_xxxxxx_2014-01-08T04:02:11.958Z
2014-01-08 04:02:11,973 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution
java.io.IOException: Could not create directories: /tmp/persistent/index_hadoop_xxxxxx_2014-01-08T04:02:11.958Z/a1d71070-5fce-4cb1-a92b-550ff11944d9
        at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:129)
        at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:115)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Thanks!  -- Wayne

[Eric Tschetter]

Ah, yes, the overlord is spawning a process that actually does the
stuff. You can set properties that will be passed down to the spawned
process as gian said.

Can you share what kinds of files have filled up the disk? I'm
wondering if Hadoop is creating the files or if Druid is and
unfortunately the exception you provided doesn't quite have enough
information to tell me.

Btw, another option that I've done in the past on AWS is to make /tmp
a symlink to /mnt/tmp

--Eric

> https://groups.google.com/d/msgid/druid-development/24d86d23-42aa-4908-8365-3dde8acae761%40googlegroups.com.

[Wayne Adams]

Thanks, Gian!  I'll try that!  -- Wayne

[Wayne Adams]

Thanks, Eric. 

  Unfortunately I already cleared out the /tmp directory when I was experimenting with symlinking it (which failed for me, btw -- I just get a lot of error messages saying particular directories could not be created under /tmp.  that too is a little lost to history as I've now mounted /tmp on a large partition).

  After reconfiguring /tmp and explicitly launching a MiddleManager with forktmpdir, I'm going to try again and see how it goes; will let you know either way.  Thanks (and to Gian) for your help!

-- Wayne