incremental persist failed -- Unable to create temporary file

[Cody Ray]

I'm not sure what's causing this error.

2016-02-01T20:36:46,105 ERROR [feedster-incremental-persist] io.druid.segment.realtime.plumber.RealtimePlumber - dataSource[feedster] -- incremental persist failed: {class=io.druid.segment.realtime.plumber.RealtimePlumber, interval=2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z, count=2}

2016-02-01T20:36:46,105 INFO [feedster-incremental-persist] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-02-01T20:36:46.105Z","service":"realtime","host":"pvlchi6ddruid1.peak6.net:8084","severity":"component-failure","description":"dataSource[feedster] -- incremental persist failed","data":{"class":"io.druid.segment.realtime.plumber.RealtimePlumber","interval":"2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z","count":2}}]

Exception in thread "plumber_persist_2" java.lang.RuntimeException: java.io.IOException: Unable to create temporary file, /tmp/filePeon6851090635294403955max_avg_exch_delay/us_little.header

Caused by: java.io.IOException: Unable to create temporary file, /tmp/filePeon6851090635294403955max_avg_exch_delay/us_little.header

The realtime node was run with

$ nohup java -server -Xms2048m -Xmx2048m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=/home/cray/druid-0.8.2/p6-stats.spec -classpath /home/cray/druid-0.8.2/config/_common:/home/cray/druid-0.8.2/config/realtime:/home/cray/druid-0.8.2/lib/* io.druid.cli.Main server realtime &

Doesn't look to be any directory permission issues:

cray@pvlchi6ddruid1:~/druid-0.8.2$ ls -la /tmp/

total 38668

drwxrwxrwt 43 root      root         20480 Feb  1 14:40 .

So I'm not sure what the problem is. Any ideas?

-Cody

[Himanshu]

Hi,

Can you give full stacktrace of the exception?

--Himanshu

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/ee55d421-47db-4962-81a1-81cc98b1b829%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cody Ray]

Sure - here's some additional context from the logs.

2016-02-01T20:16:44,842 INFO [nextgen-stats-incremental-persist] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/nextgen-stats/2016-02-01T00:00:00.000Z_2016-02-02T00:00:00.000Z/6/v8-tmp] completed dim conversions in 174 millis.

2016-02-01T20:16:44,873 ERROR [publisher-incremental-persist] io.druid.segment.realtime.plumber.RealtimePlumber - dataSource[publisher] -- incremental persist failed: {class=io.druid.segment.realtime.plumber.RealtimePlumber, interval=2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z, count=0}

2016-02-01T20:16:44,893 INFO [feedster-incremental-persist] io.druid.segment.IndexMerger - outDir[/tmp/realtime/basePersist/feedster/2016-02-01T00:00:00.000Z_2016-02-02T00:00:00.000Z/0/v8-tmp] completed dim conversions in 93 millis.

2016-02-01T20:16:44,904 INFO [publisher-incremental-persist] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-02-01T20:16:44.889Z","service":"realtime","host":"pvlchi6ddruid1.peak6.net:8084","severity":"component-failure","description":"dataSource[publisher] -- incremental persist failed","data":{"class":"io.druid.segment.realtime.plumber.RealtimePlumber","interval":"2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z","count":0}}]

Exception in thread "plumber_persist_0" java.lang.RuntimeException: java.io.IOException: Unable to create temporary file, /tmp/filePeon5255254471997458323gap/s_little.header

        at com.google.common.base.Throwables.propagate(Throwables.java:160)

        at io.druid.segment.realtime.plumber.RealtimePlumber.persistHydrant(RealtimePlumber.java:994)

        at io.druid.segment.realtime.plumber.RealtimePlumber$3.doRun(RealtimePlumber.java:405)

        at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:40)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Unable to create temporary file, /tmp/filePeon5255254471997458323gap/s_little.header

        at java.io.File$TempDirectory.generateFile(File.java:1898)

        at java.io.File.createTempFile(File.java:1987)

        at java.io.File.createTempFile(File.java:2047)

        at io.druid.segment.data.TmpFileIOPeon.makeOutputStream(TmpFileIOPeon.java:42)

        at io.druid.segment.data.GenericIndexedWriter.open(GenericIndexedWriter.java:63)

        at io.druid.segment.data.CompressedLongsSupplierSerializer.open(CompressedLongsSupplierSerializer.java:73)

        at io.druid.segment.LongMetricColumnSerializer.open(LongMetricColumnSerializer.java:57)

        at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:735)

        at io.druid.segment.IndexMerger.merge(IndexMerger.java:351)

        at io.druid.segment.IndexMerger.persist(IndexMerger.java:196)

        at io.druid.segment.IndexMerger.persist(IndexMerger.java:161)

        at io.druid.segment.IndexMerger.persist(IndexMerger.java:138)

        at io.druid.segment.realtime.plumber.RealtimePlumber.persistHydrant(RealtimePlumber.java:972)

        ... 5 more

2016-02-01T20:16:44,940 ERROR [feedster-incremental-persist] io.druid.segment.realtime.plumber.RealtimePlumber - dataSource[feedster] -- incremental persist failed: {class=io.druid.segment.realtime.plumber.RealtimePlumber, interval=2016-02-01T00:00:00.000Z/2016-02-02T00:00:00.000Z, count=0}

-Cody

[Gian Merlino]

Is "max_avg_exch_delay/us" one of your column names (with the slash '/' in it)? If so then based on the error you're seeing, column names with slashes in them probably don't work. If this is a metric you could potentially work around it by using the "name" of the aggregator to rename it to something else.

Gian

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/9fe17e2b-2c17-4203-88af-8803f441ff76%40googlegroups.com.

[Gian Merlino]

https://github.com/druid-io/druid/issues/2370

Gian

[Cody Ray]

You win, thanks.