KeeperException$NoNodeException when adding a window/trigger

[Jochen Rau]

Hi,

We started to set up a little POC with Onyx. So far, we are pretty excited.

Now on to the next hurdle ;-) When I run the job below, without windows and triggers, the sequence is correctly processed. When I add them in I get

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /onyx/fbddb348-3dc5-411f-84dc-b3ccd6ad632d/ledgers/available/readonly
    code: -101
    path: "/onyx/fbddb348-3dc5-411f-84dc-b3ccd6ad632d/ledgers/available/readonly"

I looked into ZooKeeper: the node doesn't exist and adding the node manually didn't help. It might just be a simple config error.

Best
Jochen

{:catalog [{:onyx/name :output.core.async/boxoffice,
            :onyx/plugin :onyx.plugin.core-async/output,
            :onyx/type :output,
            :onyx/medium :core.async,
            :onyx/max-peers 1,
            :onyx/batch-size 10,
            :onyx/doc "Writes segments to a core.async channel"}
           {:onyx/plugin :onyx.plugin.seq/input,
            :onyx/medium :seq,
            :seq/elements-per-segment 10,
            :onyx/type :input,
            :onyx/name :source.external/boxoffice,
            :onyx/max-peers 1,
            :seq/checkpoint? true,
            :onyx/doc "Reads segments from seq",
            :onyx/batch-size 10}
           {:onyx/name :aggregator,
            :onyx/fn :clojure.core/identity,
            :onyx/type :function,
            :onyx/batch-size 10}
           {:onyx/name :filter,
            :onyx/fn :com.example.myapp/date-filter,
            :onyx/type :function,
            :onyx/batch-size 10}],
 :windows [{:window/id :sum-cumulativegross,
            :window/task :aggregator,
            :window/type :global,
            :window/aggregation [:onyx.windowing.aggregation/sum
                                 :cumulativegross],
            :window/window-key :rownum}],
 :triggers [{:trigger/window-id :sum-cumulativegross,
             :trigger/refinement :accumulating,
             :trigger/on :segment,
             :trigger/threshold [5 :elements],
             :trigger/sync :com.example.myapp/dump-window!}],
 :workflow [[:source.external/boxoffice :filter]
            [:filter :aggregator]
            [:aggregator :output.core.async/boxoffice]],
 :lifecycles [{:lifecycle/task :output.core.async/boxoffice,
               :lifecycle/calls :com.example.myapp/out-calls,
               :core.async/id #uuid "ea5a846c-a120-4b32-8148-ccf07a9deb7c"}
              {:lifecycle/task :output.core.async/boxoffice,
               :lifecycle/calls :onyx.plugin.core-async/writer-calls}
              {:lifecycle/task :source.external/boxoffice,
               :core.async/id #uuid "6e4b3fc8-f1fc-4a3a-aed9-73917388b2d0",
               :sql/creds {},
               :sql/table "boxoffice",
               :sql/columns (),
               :lifecycle/calls :com.example.myapp/sql-in-calls}
              {:lifecycle/task :source.external/boxoffice,
               :lifecycle/calls :onyx.plugin.seq/reader-calls}],
 :task-scheduler :onyx.task-scheduler/balanced}

[Mike Drogalis]

Hi Jochen :)

You need to start BookKeeper on the peers. The easiest way to do this is to add the following

to your peer configuration, which starts an embedded BookKeeper for you:

https://github.com/onyx-platform/onyx/blob/dc79cea3618ebc9c74d00030d5649a8a2c977460/resources/test-config.edn#L5-L8

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+...@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/b5ff1dd6-f559-4be8-800f-805dd81e5bc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Drogalis]

Too quick on the draw. :) That should say environment configuration, not peer configuration.

[Jochen Rau]

Hi Mike,

Too quick on the draw. :) That should say environment configuration, not peer configuration.

Thanks! Indeed, that was quick (but not too quick) :-)

[Jochen Rau]

Hi Mike,

Here is the next hurdle. The configuration is now:

{:zookeeper/server? true,
 :onyx.bookkeeper/server? true,
 :onyx.bookkeeper/base-ledger-dir "/Users/jocrau/dev/onyx/bookkeeper_ledger",
 :zookeeper.server/port 2188,
 :onyx.log/config {},
 :onyx.bookkeeper/local-quorum-ports [3196 3197 3198],
 :onyx.bookkeeper/base-journal-dir "/Users/jocrau/dev/onyx/bookkeeper_journal",
 :onyx/id #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c",
 :zookeeper/address "127.0.0.1:2188",
 :onyx.bookkeeper/delete-server-data? true,
 :onyx.bookkeeper/local-quorum? true}
{:onyx/id #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c",
 :onyx.log/config {},
 :onyx.messaging.aeron/embedded-driver? true,
 :onyx.messaging/allow-short-circuit? false,
 :onyx.messaging/peer-port 40200,
 :zookeeper/address "127.0.0.1:2188",
 :onyx.messaging/bind-addr "localhost",
 :onyx.messaging/impl :aeron,
 :onyx.peer/job-scheduler :onyx.job-scheduler/greedy}

Which produces the following exception on startup:

org.apache.bookkeeper.bookie.LedgerDirsManager$NoWritableLedgerDirException: All ledger directories are non writable
                                                 clojure.lang.ExceptionInfo: Error in component :bookkeeper in system onyx.system.OnyxDevelopmentEnv calling #'com.stuartsierra.component/start
     component: #onyx.state.bookkeeper.BookieServers{:env-config {:zookeeper/server? true, :onyx.bookkeeper/server? true, :onyx.bookkeeper/base-ledger-dir "/Users/jocrau/dev/onyx/bookkeeper_ledger", :zookeeper.server/port 2188, :onyx.log/config {}, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :onyx.bookkeeper/base-journal-dir "/Users/jocrau/dev/onyx/bookkeeper_journal", :onyx/id #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c", :zookeeper/address "127.0.0.1:2188", :onyx.bookkeeper/delete-server-data? true, ...}, :log #onyx.log.zookeeper.ZooKeeper{:config {:zookeeper/server? true, :onyx.bookkeeper/server? true, :onyx.bookkeeper/base-ledger-dir "/Users/jocrau/dev/onyx/bookkeeper_ledger", :zookeeper.server/port 2188, :onyx.log/config {}, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :onyx.bookkeeper/base-journal-dir "/Users/jocrau/dev/onyx/bookkeeper_journal", :onyx/id #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c", :zookeeper/address "127.0.0.1:2188", :onyx.bookkeeper/delete-server-data? true, ...}, :monitoring #onyx.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config #onyx.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config {}}, :server #object[org.apache.curator.test.TestingServer 0x5c9b8e44 "org.apache.curator.test.TestingServer@5c9b8e44"], :conn #object[org.apache.curator.framework.imps.CuratorFrameworkImpl 0x449e96f3 "org.apache.curator.framework.imps.CuratorFrameworkImpl@449e96f3"], :prefix #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c"}}
      function: #<Var@4a44f107:
                  #object[com.stuartsierra.component$fn__6293$G__6286__6298 0x7eb218b "com.stuartsierra.component$fn__6293$G__6286__6298@7eb218b"]>
        reason: :com.stuartsierra.component/component-function-threw-exception
        system: #onyx.system.OnyxDevelopmentEnv{:monitoring #onyx.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config #onyx.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config {}}, :bookkeeper #onyx.state.bookkeeper.BookieServers{:env-config {:zookeeper/server? true, :onyx.bookkeeper/server? true, :onyx.bookkeeper/base-ledger-dir "/Users/jocrau/dev/onyx/bookkeeper_ledger", :zookeeper.server/port 2188, :onyx.log/config {}, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :onyx.bookkeeper/base-journal-dir "/Users/jocrau/dev/onyx/bookkeeper_journal", :onyx/id #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c", :zookeeper/address "127.0.0.1:2188", :onyx.bookkeeper/delete-server-data? true, ...}, :log nil}, :log #onyx.log.zookeeper.ZooKeeper{:config {:zookeeper/server? true, :onyx.bookkeeper/server? true, :onyx.bookkeeper/base-ledger-dir "/Users/jocrau/dev/onyx/bookkeeper_ledger", :zookeeper.server/port 2188, :onyx.log/config {}, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :onyx.bookkeeper/base-journal-dir "/Users/jocrau/dev/onyx/bookkeeper_journal", :onyx/id #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c", :zookeeper/address "127.0.0.1:2188", :onyx.bookkeeper/delete-server-data? true, ...}, :monitoring #onyx.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config #onyx.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config {}}, :server #object[org.apache.curator.test.TestingServer 0x5c9b8e44 "org.apache.curator.test.TestingServer@5c9b8e44"], :conn #object[org.apache.curator.framework.imps.CuratorFrameworkImpl 0x449e96f3 "org.apache.curator.framework.imps.CuratorFrameworkImpl@449e96f3"], :prefix #uuid "a57936c4-b75a-47c0-b8ce-c7331666132c"}}
    system-key: :bookkeeper

The :onyx.bookkeeper/base-*-dirs exist and seem to have the correct permissions. The content of bookkeeper_journal/a57936c4-b75a-47c0-b8ce-c7331666132c_3196/current/VERSION is

4
bookieHost: "192.168.3.244:3196"
journalDir: "/Users/jocrau/dev/onyx/bookkeeper_journal/a57936c4-b75a-47c0-b8ce-c7331666132c_3196"
ledgerDirs: "1\t/Users/jocrau/dev/onyx/bookkeeper_ledger/a57936c4-b75a-47c0-b8ce-c7331666132c_3196"

which looks a bit suspicious, because the ledgerDirs are prepended with "1\t".

Any idea what could cause this?

Jochen

[Lucas Bradstreet]

Hi Jochen,

I've seen an error that looks like this when I didn't have much disk
space available on my machine. I believe it's preallocating space for
each bookie server, and it's doing it three times (because you're
running three servers locally in local-quorum mode).

If this is the issue you're hitting, I may look into whether we can
reduce the preallocation size. I believe I've had it pop up when I had
several GB in free space.

Thanks,

Lucas

> --
> You received this message because you are subscribed to the Google Groups
> "Onyx" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to onyx-user+...@googlegroups.com.
> To post to this group, send email to onyx...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/onyx-user/0c3ab8a9-3cfa-4e4f-b3d6-8cd8d783b5e3%40googlegroups.com.

[Jochen Rau]

Hi Lucas,

On Thursday, December 17, 2015 at 12:15:12 PM UTC-5, Lucas Bradstreet wrote:

I've seen an error that looks like this when I didn't have much disk
space available on my machine. I believe it's preallocating space for
each bookie server, and it's doing it three times (because you're
running three servers locally in local-quorum mode).

If this is the issue you're hitting, I may look into whether we can
reduce the preallocation size. I believe I've had it pop up when I had
several GB in free space.

I had 2.9GB of free disk space (of 250GB) when this error popped up the first time. I deleted all the Country albums from my my iTunes ;-). The problem still persisted with 12.9GB of free disk space. Then I deleted also my Western albums and I am now at 25GB. Onyx starts up again and the data flows. 90% memory usage seems to be the threshold for warnings (and not 95%) as in the following log messages.

>>> 2015-12-17 16:21:43,412 [read] WARN  DiskChecker - Space left on device /tmp/bookkeeper_ledger/29f216c5-fcfd-442c-977a-cb75173e7293_3196/current : 24990728192, Used space fraction: 0.9018799 < WarnThreshold 0.95.
>>> 2015-12-17 16:21:43,412 [read] WARN  LedgerDirsManager - Ledger directory /tmp/bookkeeper_ledger/29f216c5-fcfd-442c-977a-cb75173e7293_3196/current is almost full.

I think Bookie doesn't preallocate disk space but merely checks the disk space and refuses to start up > 95% and logs a warning > 90%. It might be helpful to have an option to tweak this and/or add a note to the docs.

Thanks!

Jochen

[Mike Drogalis]

Tracking this in issue 449, will definitely put out a configuration for this, and tone down the default. Thanks for the report!

--
You received this message because you are subscribed to the Google Groups "Onyx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onyx-user+...@googlegroups.com.
To post to this group, send email to onyx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/onyx-user/bd0e2186-c713-4ce9-b699-09e0d0e3f1d3%40googlegroups.com.