Async worker died! ... clojure.lang.PersistentVector cannot be cast to class java.lang.String
dgdesi...@gmail.com
Hi All!
I'm new to the crawler world and Java so please set me straight.
I've been successful injecting my target URLs but when I run the crawler local my async worker dies...
Googling has not helped me in this case.
Below is the print out of the situation.
Please let me know what other information I should include.
Any idea what's causing this?
...
storm jar target/dev1-0.1.jar org.apache.storm.flux.Flux --local --sleep 86400000 es-crawler.flux
Running: java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/apache-storm-1.2.3 -Dstorm.log.dir=/opt/apache-storm-1.2.3/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/apache-storm-1.2.3/*:/opt/apache-storm-1.2.3/lib/*:/opt/apache-storm-1.2.3/extlib/*:target/dev1-0.1.jar:/opt/apache-storm-1.2.3/conf:/opt/apache-storm-1.2.3/bin -Dstorm.jar=target/dev1-0.1.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.flux.Flux --local --sleep 86400000 es-crawler.flux
███████╗██╗ ██╗ ██╗██╗ ██╗
██╔════╝██║ ██║ ██║╚██╗██╔╝
█████╗ ██║ ██║ ██║ ╚███╔╝
██╔══╝ ██║ ██║ ██║ ██╔██╗
██║ ███████╗╚██████╔╝██╔╝ ██╗
╚═╝ ╚══════╝ ╚═════╝ ╚═╝ ╚═╝
+- Apache Storm -+
+- data FLow User eXperience -+
Version: 1.2.3
Parsing file: /home/ubuntu/crawl/dev1/es-crawler.flux
809 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
814 [main] INFO o.a.s.f.p.FluxParser - Not performing property substitution.
814 [main] INFO o.a.s.f.p.FluxParser - Not performing environment variable substitution.
886 [main] INFO o.a.s.f.p.FluxParser - Loading includes from resource: /crawler-default.yaml
891 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
895 [main] INFO o.a.s.f.p.FluxParser - Not performing property substitution.
895 [main] INFO o.a.s.f.p.FluxParser - Not performing environment variable substitution.
fetcher.queue.mode: byHost
fetcher.threads.per.queue: 1
fetcher.threads.number: 10
fetcher.max.urls.in.queues: -1
fetcher.max.queue.size: -1
fetcher.timeout.queue: -1
fetcher.max.crawl.delay: 30
fetcher.max.crawl.delay.force: false
fetcher.server.delay.force: false
fetcher.metrics.time.bucket.secs: 10
fetcher.max.throttle.sleep: -1
partition.url.mode: byHost
urlbuffer.class: com.digitalpebble.stormcrawler.persistence.SimpleURLBuffer
metadata.persist: [_redirTo, error.cause, error.source, isSitemap, isFeed]
metadata.track.path: true
metadata.track.depth: true
http.agent.name: Anonymous Coward
http.agent.version: '1.0'
http.agent.description: built with StormCrawler 1.15
http.agent.url: http://someorganization.com/
http.agent.email: som...@someorganization.com
http.accept.language: en-us,en-gb,en;q=0.7,*;q=0.3
http.accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
http.content.limit: -1
http.store.headers: false
http.timeout: 10000
http.content.partial.as.trimmed: false
http.robots.403.allow: true
robots.noFollow.strict: true
robots.cache.spec: maximumSize=10000,expireAfterWrite=6h
robots.error.cache.spec: maximumSize=10000,expireAfterWrite=1h
protocols: http,https,file
http.protocol.implementation: com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol
https.protocol.implementation: com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol
file.protocol.implementation: com.digitalpebble.stormcrawler.protocol.file.FileProtocol
selenium.implicitlyWait: 0
selenium.pageLoadTimeout: -1
selenium.setScriptTimeout: 0
selenium.instances.num: 1
selenium.capabilities: {takesScreenshot: false, loadImages: false, javascriptEnabled: true}
selenium.delegated.protocol: com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol
jsoup.treat.non.html.as.error: true
parser.emitOutlinks: true
parser.emitOutlinks.max.per.page: -1
track.anchors: true
detect.mimetype: true
detect.charset.maxlength: 10000
sitemap.filter.hours.since.modified: -1
sitemap.schedule.delay: -1
sitemap.discovery: false
scheduler.class: com.digitalpebble.stormcrawler.persistence.DefaultScheduler
fetchInterval.default: 1440
fetchInterval.fetch.error: 120
fetchInterval.error: -1
max.fetch.errors: 3
status.updater.use.cache: true
status.updater.cache.spec: maximumSize=10000,expireAfterAccess=1h
status.updater.unit.round.date: SECOND
indexer.url.fieldname: url
indexer.text.fieldname: content
indexer.text.maxlength: -1
indexer.canonical.name: canonical
indexer.md.mapping: [parse.title=title, parse.keywords=keywords, parse.description=description]
includes: null
name: null
spouts: []
streams: []
topologySource: null
929 [main] INFO o.a.s.f.p.FluxParser - Loading includes from file: crawler-conf.yaml
929 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
933 [main] INFO o.a.s.f.p.FluxParser - Not performing property substitution.
933 [main] INFO o.a.s.f.p.FluxParser - Not performing environment variable substitution.
Configuration (interpreted):
!!org.apache.storm.flux.model.TopologyDef
bolts: []
components: []
config:
topology.workers: 1
topology.message.timeout.secs: 300
topology.max.spout.pending: 100
topology.debug: false
fetcher.threads.number: 50
topology.worker.childopts: -Xmx2g -Djava.net.preferIPv4Stack=true
topology.kryo.register: [com.digitalpebble.stormcrawler.Metadata]
metadata.persist: [_redirTo, error.cause, error.source, isSitemap, isFeed]
http.agent.name: Google-Read-Aloud
http.agent.version: '0.1'
http.agent.description: ''
http.agent.url: https://support.google.com/webmasters/answer/1061943
http.agent.email: ''
http.content.limit: 65536
parsefilters.config.file: parsefilters.json
urlfilters.config.file: urlfilters.json
fetchInterval.default: 1440
fetchInterval.fetch.error: 120
fetchInterval.error: -1
textextractor.include.pattern: ['DIV[id="maincontent"]', 'DIV[itemprop="articleBody"]',
ARTICLE]
textextractor.exclude.tags: [STYLE, SCRIPT]
indexer.url.fieldname: url
indexer.text.fieldname: content
indexer.canonical.name: canonical
indexer.md.mapping: [parse.title=title, parse.keywords=keywords, parse.description=description,
domain=domain]
topology.metrics.consumer.register:
- {class: org.apache.storm.metric.LoggingMetricsConsumer, parallelism.hint: 1}
includes: null
name: null
spouts: []
streams: []
topologySource: null
940 [main] INFO o.a.s.f.p.FluxParser - Loading includes from file: es-conf.yaml
941 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
944 [main] INFO o.a.s.f.p.FluxParser - Not performing property substitution.
944 [main] INFO o.a.s.f.p.FluxParser - Not performing environment variable substitution.
Configuration (interpreted):
!!org.apache.storm.flux.model.TopologyDef
bolts: []
components: []
config:
es.indexer.addresses: localhost
es.indexer.index.name: content
es.indexer.create: false
es.indexer.bulkActions: 100
es.indexer.flushInterval: 2s
es.indexer.concurrentRequests: 1
es.indexer.settings: {cluster.name: elasticsearch}
es.metrics.addresses: http://localhost:9200
es.metrics.index.name: metrics
es.metrics.settings: {cluster.name: elasticsearch}
es.status.addresses: http://localhost:9200
es.status.index.name: status
es.status.routing: true
es.status.routing.fieldname: key
es.status.bulkActions: 500
es.status.flushInterval: 5s
es.status.concurrentRequests: 1
es.status.settings: {cluster.name: elasticsearch}
spout.ttl.purgatory: 30
spout.min.delay.queries: 2000
spout.reset.fetchdate.after: 120
es.status.max.buckets: 50
es.status.max.urls.per.bucket: 2
es.status.bucket.field: key
es.status.bucket.sort.field: [nextFetchDate, url]
es.status.global.sort.field: nextFetchDate
es.status.max.start.offset: 500
es.status.sample: false
es.status.query.timeout: -1
es.status.recentDate.increase: -1
es.status.recentDate.min.gap: -1
topology.metrics.consumer.register:
- {class: com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer,
parallelism.hint: 1}
includes: null
name: null
spouts: []
streams: []
topologySource: null
955 [main] INFO o.a.s.f.FluxBuilder - Detected DSL topology...
1399 [main] WARN o.a.s.u.Utils - STORM-VERSION new 1.2.3 old null
---------- TOPOLOGY DETAILS ----------
Topology Name: crawler
--------------- SPOUTS ---------------
spout [10] (com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout)
---------------- BOLTS ---------------
partitioner [1] (com.digitalpebble.stormcrawler.bolt.URLPartitionerBolt)
fetcher [1] (com.digitalpebble.stormcrawler.bolt.FetcherBolt)
sitemap [1] (com.digitalpebble.stormcrawler.bolt.SiteMapParserBolt)
parse [1] (com.digitalpebble.stormcrawler.bolt.JSoupParserBolt)
index [1] (com.digitalpebble.stormcrawler.elasticsearch.bolt.IndexerBolt)
status [1] (com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt)
status_metrics [1] (com.digitalpebble.stormcrawler.elasticsearch.metrics.StatusMetricsBolt)
--------------- STREAMS ---------------
spout --SHUFFLE--> partitioner
spout --SHUFFLE--> status_metrics
partitioner --FIELDS--> fetcher
fetcher --LOCAL_OR_SHUFFLE--> sitemap
sitemap --LOCAL_OR_SHUFFLE--> parse
parse --LOCAL_OR_SHUFFLE--> index
fetcher --FIELDS--> status
sitemap --FIELDS--> status
parse --FIELDS--> status
index --FIELDS--> status
--------------------------------------
1406 [main] INFO o.a.s.f.Flux - Running in local mode...
4037 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
4038 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:host.name=ip-172-31-22-101.us-west-2.compute.internal
4038 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:java.version=11.0.4
4038 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:java.vendor=Ubuntu
4038 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64
4039 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:java.class.path=/opt/apache-storm-1.2.3/*:/opt/apache-storm-1.2.3/lib/storm-rename-hack-1.2.3.jar:/opt/apache-storm-1.2.3/lib/disruptor-3.3.11.jar:/opt/apache-storm-1.2.3/lib/servlet-api-2.5.jar:/opt/apache-storm-1.2.3/lib/metrics-graphite-3.1.0.jar:/opt/apache-storm-1.2.3/lib/clojure-1.7.0.jar:/opt/apache-storm-1.2.3/lib/objenesis-2.1.jar:/opt/apache-storm-1.2.3/lib/asm-5.0.3.jar:/opt/apache-storm-1.2.3/lib/kryo-3.0.3.jar:/opt/apache-storm-1.2.3/lib/log4j-core-2.8.2.jar:/opt/apache-storm-1.2.3/lib/slf4j-api-1.7.21.jar:/opt/apache-storm-1.2.3/lib/minlog-1.3.0.jar:/opt/apache-storm-1.2.3/lib/reflectasm-1.10.1.jar:/opt/apache-storm-1.2.3/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-1.2.3/lib/ring-cors-0.1.5.jar:/opt/apache-storm-1.2.3/lib/storm-core-1.2.3.jar:/opt/apache-storm-1.2.3/lib/metrics-core-3.1.0.jar:/opt/apache-storm-1.2.3/lib/log4j-api-2.8.2.jar:/opt/apache-storm-1.2.3/lib/log4j-slf4j-impl-2.8.2.jar:/opt/apache-storm-1.2.3/extlib/*:target/dev1-0.1.jar:/opt/apache-storm-1.2.3/conf:/opt/apache-storm-1.2.3/bin
4039 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
4039 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:java.io.tmpdir=/tmp
4040 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:java.compiler=<NA>
4040 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:os.name=Linux
4040 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:os.arch=amd64
4040 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:os.version=4.15.0-1054-aws
4041 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:user.name=ubuntu
4041 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:user.home=/home/ubuntu
4041 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Server environment:user.dir=/home/ubuntu/crawl/dev1
4052 [main] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /tmp/bb22e646-cf52-47c9-8269-f32dc8116be1/version-2 snapdir /tmp/bb22e646-cf52-47c9-8269-f32dc8116be1/version-2
4068 [main] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2000
4070 [main] INFO o.a.s.zookeeper - Starting inprocess zookeeper at port 2000 and dir /tmp/bb22e646-cf52-47c9-8269-f32dc8116be1
4158 [main] INFO o.a.s.d.nimbus - Starting Nimbus with conf {topology.builtin.metrics.bucket.size.secs=60, nimbus.childopts=-Xmx1024m, ui.filter.params=null, storm.cluster.mode=local, storm.messaging.netty.client_worker_threads=1, logviewer.max.per.worker.logs.size.mb=2048, supervisor.run.worker.as.user=false, topology.max.task.parallelism=null, topology.priority=29, zmq.threads=1, storm.group.mapping.service=org.apache.storm.security.auth.ShellBasedGroupsMapping, transactional.zookeeper.root=/transactional, topology.sleep.spout.wait.strategy.time.ms=1, ui.pagination=20, scheduler.display.resource=false, topology.max.replication.wait.time.sec=60, drpc.invocations.port=3773, supervisor.localizer.cache.target.size.mb=10240, topology.multilang.serializer=org.apache.storm.multilang.JsonSerializer, storm.messaging.netty.server_worker_threads=1, nimbus.blobstore.class=org.apache.storm.blobstore.LocalFsBlobStore, resource.aware.scheduler.eviction.strategy=org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy, topology.max.error.report.per.interval=5, storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin, zmq.hwm=0, storm.group.mapping.service.params=null, worker.profiler.enabled=false, storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal, supervisor.worker.shutdown.sleep.secs=3, pacemaker.host=localhost, storm.zookeeper.retry.times=5, ui.actions.enabled=true, zmq.linger.millis=0, supervisor.enable=true, topology.stats.sample.rate=0.05, storm.nimbus.zookeeper.acls.fixup=true, storm.messaging.netty.min_wait_ms=100, worker.log.level.reset.poll.secs=30, storm.zookeeper.port=2000, supervisor.heartbeat.frequency.secs=5, topology.enable.message.timeouts=true, supervisor.cpu.capacity=400.0, drpc.worker.threads=64, supervisor.blobstore.download.thread.count=5, task.backpressure.poll.secs=30, drpc.queue.size=128, topology.backpressure.enable=false, supervisor.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.blobstore.inputstream.buffer.size.bytes=65536, topology.shellbolt.max.pending=100, drpc.https.keystore.password=*****, nimbus.code.sync.freq.secs=120, logviewer.port=8000, topology.scheduler.strategy=org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy, topology.executor.send.buffer.size=1024, resource.aware.scheduler.priority.strategy=org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy, pacemaker.auth.method=NONE, storm.daemon.metrics.reporter.plugins=["org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter"], topology.worker.logwriter.childopts=-Xmx64m, topology.spout.wait.strategy=org.apache.storm.spout.SleepSpoutWaitStrategy, ui.host=0.0.0.0, storm.nimbus.retry.interval.millis=2000, nimbus.inbox.jar.expiration.secs=3600, dev.zookeeper.path=/tmp/dev-storm-zookeeper, topology.acker.executors=null, topology.fall.back.on.java.serialization=true, topology.eventlogger.executors=0, supervisor.localizer.cleanup.interval.ms=600000, storm.zookeeper.servers=["localhost"], nimbus.thrift.threads=64, logviewer.cleanup.age.mins=10080, topology.worker.childopts=null, topology.classpath=null, supervisor.monitor.frequency.secs=3, nimbus.credential.renewers.freq.secs=600, backpressure.znode.timeout.secs=30, topology.skip.missing.kryo.registrations=true, drpc.authorizer.acl.filename=drpc-auth-acl.yaml, pacemaker.kerberos.users=[], storm.group.mapping.service.cache.duration.secs=120, blobstore.dir=/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e, topology.testing.always.try.serialize=false, nimbus.monitor.freq.secs=10, storm.health.check.timeout.ms=5000, supervisor.supervisors=[], topology.tasks=null, topology.bolts.outgoing.overflow.buffer.enable=false, storm.messaging.netty.socket.backlog=500, topology.workers=1, pacemaker.base.threads=10, storm.local.dir=/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e, backpressure.znode.update.freq.secs=15, worker.childopts=-Xmx%HEAP-MEM%m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump, storm.auth.simple-white-list.users=[], topology.disruptor.batch.timeout.millis=1, topology.message.timeout.secs=30, topology.state.synchronization.timeout.secs=60, topology.tuple.serializer=org.apache.storm.serialization.types.ListDelegateSerializer, supervisor.supervisors.commands=[], nimbus.blobstore.expiration.secs=600, logviewer.childopts=-Xmx128m, storm.nimbus.zookeeper.acls.check=true, topology.environment=null, topology.debug=false, topology.disruptor.batch.size=100, storm.disable.symlinks=false, storm.messaging.netty.max_retries=300, ui.childopts=-Xmx768m, storm.network.topography.plugin=org.apache.storm.networktopography.DefaultRackDNSToSwitchMapping, storm.zookeeper.session.timeout=20000, drpc.childopts=-Xmx768m, drpc.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.connection.timeout=15000, storm.zookeeper.auth.user=null, storm.meta.serialization.delegate=org.apache.storm.serialization.GzipThriftSerializationDelegate, topology.max.spout.pending=null, storm.codedistributor.class=org.apache.storm.codedistributor.LocalFileSystemCodeDistributor, nimbus.supervisor.timeout.secs=60, nimbus.task.timeout.secs=30, drpc.port=3772, pacemaker.max.threads=50, storm.zookeeper.retry.intervalceiling.millis=30000, nimbus.thrift.port=6627, storm.auth.simple-acl.admins=[], topology.component.cpu.pcore.percent=10.0, supervisor.memory.capacity.mb=3072.0, storm.nimbus.retry.times=5, supervisor.worker.start.timeout.secs=120, storm.zookeeper.retry.interval=1000, logs.users=null, storm.cluster.metrics.consumer.publish.interval.secs=60, worker.profiler.command=flight.bash, transactional.zookeeper.port=null, drpc.max_buffer_size=1048576, pacemaker.thread.timeout=10, task.credentials.poll.secs=30, blobstore.superuser=ubuntu, drpc.https.keystore.type=JKS, topology.worker.receiver.thread.count=1, topology.state.checkpoint.interval.ms=1000, supervisor.slots.ports=[6700 6701 6702 6703], topology.transfer.buffer.size=1024, storm.health.check.dir=healthchecks, topology.worker.shared.thread.pool.size=4, drpc.authorizer.acl.strict=false, nimbus.file.copy.expiration.secs=600, worker.profiler.childopts=-XX:+UnlockCommercialFeatures -XX:+FlightRecorder, topology.executor.receive.buffer.size=1024, backpressure.disruptor.low.watermark=0.4, nimbus.task.launch.secs=120, storm.local.mode.zmq=false, storm.messaging.netty.buffer_size=5242880, storm.cluster.state.store=org.apache.storm.cluster_state.zookeeper_state_factory, worker.heartbeat.frequency.secs=1, storm.log4j2.conf.dir=log4j2, ui.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.root=/storm, topology.tick.tuple.freq.secs=null, drpc.https.port=-1, storm.workers.artifacts.dir=workers-artifacts, supervisor.blobstore.download.max_retries=3, task.refresh.poll.secs=10, storm.exhibitor.port=8080, task.heartbeat.frequency.secs=3, pacemaker.port=6699, storm.messaging.netty.max_wait_ms=1000, topology.component.resources.offheap.memory.mb=0.0, drpc.http.port=3774, topology.error.throttle.interval.secs=10, storm.messaging.transport=org.apache.storm.messaging.netty.Context, topology.disable.loadaware.messaging=false, storm.messaging.netty.authentication=false, topology.component.resources.onheap.memory.mb=128.0, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, worker.gc.childopts=, nimbus.topology.validator=org.apache.storm.nimbus.DefaultTopologyValidator, nimbus.seeds=["localhost"], nimbus.queue.size=100000, nimbus.cleanup.inbox.freq.secs=600, storm.blobstore.replication.factor=3, worker.heap.memory.mb=768, logviewer.max.sum.worker.logs.size.mb=4096, pacemaker.childopts=-Xmx1024m, ui.users=null, transactional.zookeeper.servers=null, supervisor.worker.timeout.secs=30, storm.zookeeper.auth.password=null, storm.blobstore.acl.validation.enabled=false, client.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.thrift.socket.timeout.ms=600000, supervisor.childopts=-Xmx256m, topology.worker.max.heap.size.mb=768.0, ui.http.x-frame-options=DENY, backpressure.disruptor.high.watermark=0.9, ui.filter=null, ui.header.buffer.bytes=4096, topology.min.replication.count=1, topology.disruptor.wait.timeout.millis=1000, storm.nimbus.retry.intervalceiling.millis=60000, topology.trident.batch.emit.interval.millis=50, storm.auth.simple-acl.users=[], drpc.invocations.threads=64, java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib, ui.port=8080, storm.exhibitor.poll.uripath=/exhibitor/v1/cluster/list, storm.messaging.netty.transfer.batch.size=262144, logviewer.appender.name=A1, nimbus.thrift.max_buffer_size=1048576, storm.auth.simple-acl.users.commands=[], drpc.request.timeout.secs=600}
4165 [main] INFO o.a.s.n.NimbusInfo - Nimbus figures out its name to ip-172-31-22-101.us-west-2.compute.internal
4212 [main] INFO o.a.s.s.o.a.c.u.Compatibility - Running in ZooKeeper 3.4.x compatibility mode
4238 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4249 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
4250 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:host.name=ip-172-31-22-101.us-west-2.compute.internal
4250 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:java.version=11.0.4
4250 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:java.vendor=Ubuntu
4250 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64
4251 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:java.class.path=/opt/apache-storm-1.2.3/*:/opt/apache-storm-1.2.3/lib/storm-rename-hack-1.2.3.jar:/opt/apache-storm-1.2.3/lib/disruptor-3.3.11.jar:/opt/apache-storm-1.2.3/lib/servlet-api-2.5.jar:/opt/apache-storm-1.2.3/lib/metrics-graphite-3.1.0.jar:/opt/apache-storm-1.2.3/lib/clojure-1.7.0.jar:/opt/apache-storm-1.2.3/lib/objenesis-2.1.jar:/opt/apache-storm-1.2.3/lib/asm-5.0.3.jar:/opt/apache-storm-1.2.3/lib/kryo-3.0.3.jar:/opt/apache-storm-1.2.3/lib/log4j-core-2.8.2.jar:/opt/apache-storm-1.2.3/lib/slf4j-api-1.7.21.jar:/opt/apache-storm-1.2.3/lib/minlog-1.3.0.jar:/opt/apache-storm-1.2.3/lib/reflectasm-1.10.1.jar:/opt/apache-storm-1.2.3/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-1.2.3/lib/ring-cors-0.1.5.jar:/opt/apache-storm-1.2.3/lib/storm-core-1.2.3.jar:/opt/apache-storm-1.2.3/lib/metrics-core-3.1.0.jar:/opt/apache-storm-1.2.3/lib/log4j-api-2.8.2.jar:/opt/apache-storm-1.2.3/lib/log4j-slf4j-impl-2.8.2.jar:/opt/apache-storm-1.2.3/extlib/*:target/dev1-0.1.jar:/opt/apache-storm-1.2.3/conf:/opt/apache-storm-1.2.3/bin
4251 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
4251 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:java.io.tmpdir=/tmp
4251 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:java.compiler=<NA>
4252 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:os.name=Linux
4252 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:os.arch=amd64
4253 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:os.version=4.15.0-1054-aws
4253 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:user.name=ubuntu
4253 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:user.home=/home/ubuntu
4254 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Client environment:user.dir=/home/ubuntu/crawl/dev1
4255 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@796eeafb
4267 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4275 [main] INFO o.a.s.b.FileBlobStoreImpl - Creating new blob store based in /tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e/blobs
4281 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4287 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@196519dd
4289 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4289 [main] INFO o.a.s.d.nimbus - Using default scheduler
4289 [main] INFO o.a.s.n.NimbusInfo - Nimbus figures out its name to ip-172-31-22-101.us-west-2.compute.internal
4309 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4311 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4311 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55310
4312 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4311 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4318 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55312
4320 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55312
4322 [SyncThread:0] INFO o.a.s.s.o.a.z.s.p.FileTxnLog - Creating new log file: log.1
4322 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55310
4345 [main] INFO o.a.s.n.NimbusInfo - Nimbus figures out its name to ip-172-31-22-101.us-west-2.compute.internal
4347 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20000, negotiated timeout = 20000
4347 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20000 with negotiated timeout 20000 for client /127.0.0.1:55312
4358 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4359 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20001 with negotiated timeout 20000 for client /127.0.0.1:55310
4359 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20001, negotiated timeout = 20000
4360 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4362 [main-EventThread] INFO o.a.s.zookeeper - Zookeeper state update: :connected:none
4374 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4375 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@2778f3d1
4376 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4382 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4383 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4383 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55314
4383 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55314
4384 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20002 with negotiated timeout 20000 for client /127.0.0.1:55314
4385 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20002, negotiated timeout = 20000
4385 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4386 [main-EventThread] INFO o.a.s.zookeeper - Zookeeper state update: :connected:none
4410 [Curator-Framework-0] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - backgroundOperationsLoop exiting
4412 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Processed session termination for sessionid: 0x100040091f20002
4414 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxn - Closed socket connection for client /127.0.0.1:55314 which had sessionid 0x100040091f20002
4415 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Session: 0x100040091f20002 closed
4415 [main-EventThread] INFO o.a.s.s.o.a.z.ClientCnxn - EventThread shut down for session: 0x100040091f20002
4417 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4418 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@40c858fc
4423 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4425 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4426 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4426 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55316
4427 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4427 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55316
4428 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@6cf7de4e
4432 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4435 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4436 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4434 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20003, negotiated timeout = 20000
4436 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4434 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20003 with negotiated timeout 20000 for client /127.0.0.1:55316
4436 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55318
4437 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55318
4440 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20004 with negotiated timeout 20000 for client /127.0.0.1:55318
4440 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20004, negotiated timeout = 20000
4442 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4490 [main] INFO o.a.s.zookeeper - Queued up for leader lock.
4506 [main] INFO o.a.s.d.m.MetricsUtils - Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
4511 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f20000 type:create cxid:0x1 zxid:0x12 txntype:-1 reqpath:n/a Error Path:/storm/leader-lock Error:KeeperErrorCode = NoNode for /storm/leader-lock
4512 [main] INFO o.a.s.d.m.r.JmxPreparableReporter - Preparing...
4514 [Curator-Framework-0] WARN o.a.s.s.o.a.c.u.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
4532 [main-EventThread] INFO o.a.s.z.Zookeeper - active-topology-blobs [] local-topology-blobs [] diff-topology-blobs []
4533 [main-EventThread] INFO o.a.s.z.Zookeeper - active-topology-dependencies [] local-blobs [] diff-topology-dependencies []
4533 [main-EventThread] INFO o.a.s.z.Zookeeper - Accepting leadership, all active topologies and corresponding dependencies found locally.
4543 [main] INFO o.a.s.d.common - Started statistics report plugin...
4557 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4557 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@74de1e9d
4562 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4564 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4564 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55320
4564 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55320
4564 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4566 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20005 with negotiated timeout 20000 for client /127.0.0.1:55320
4566 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20005, negotiated timeout = 20000
4566 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4567 [main-EventThread] INFO o.a.s.zookeeper - Zookeeper state update: :connected:none
4569 [Curator-Framework-0] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - backgroundOperationsLoop exiting
4569 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Processed session termination for sessionid: 0x100040091f20005
4571 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxn - Closed socket connection for client /127.0.0.1:55320 which had sessionid 0x100040091f20005
4571 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Session: 0x100040091f20005 closed
4572 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4571 [main-EventThread] INFO o.a.s.s.o.a.z.ClientCnxn - EventThread shut down for session: 0x100040091f20005
4578 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@191d69a
4579 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4579 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4580 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@23043ba
4580 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4585 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4585 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4585 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55322
4585 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4586 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4586 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55324
4586 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55322
4586 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55324
4588 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20006 with negotiated timeout 20000 for client /127.0.0.1:55322
4588 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20006, negotiated timeout = 20000
4589 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4590 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20007 with negotiated timeout 20000 for client /127.0.0.1:55324
4590 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20007, negotiated timeout = 20000
4590 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4591 [main-EventThread] INFO o.a.s.zookeeper - Zookeeper state update: :connected:none
4592 [Curator-Framework-0] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - backgroundOperationsLoop exiting
4593 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Processed session termination for sessionid: 0x100040091f20007
4594 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxn - Closed socket connection for client /127.0.0.1:55324 which had sessionid 0x100040091f20007
4594 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Session: 0x100040091f20007 closed
4595 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4599 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@1b47e09c
4599 [main-EventThread] INFO o.a.s.s.o.a.z.ClientCnxn - EventThread shut down for session: 0x100040091f20007
4611 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4611 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55326
4611 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4612 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55326
4612 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4613 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20008 with negotiated timeout 20000 for client /127.0.0.1:55326
4614 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20008, negotiated timeout = 20000
4614 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4642 [main] INFO o.a.s.z.Zookeeper - Staring ZK Curator
4642 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4643 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@188ccb6c
4648 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4651 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4652 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4652 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55328
4653 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55328
4654 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f20009 with negotiated timeout 20000 for client /127.0.0.1:55328
4655 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f20009, negotiated timeout = 20000
4655 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4658 [Curator-Framework-0] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - backgroundOperationsLoop exiting
4659 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Processed session termination for sessionid: 0x100040091f20009
4660 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Session: 0x100040091f20009 closed
4660 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxn - Closed socket connection for client /127.0.0.1:55328 which had sessionid 0x100040091f20009
4663 [main] INFO o.a.s.z.Zookeeper - Staring ZK Curator
4663 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4660 [main-EventThread] INFO o.a.s.s.o.a.z.ClientCnxn - EventThread shut down for session: 0x100040091f20009
4665 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@79753f20
4670 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4671 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4672 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4672 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55330
4673 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55330
4674 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f2000a with negotiated timeout 20000 for client /127.0.0.1:55330
4675 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f2000a, negotiated timeout = 20000
4677 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4686 [main] INFO o.a.s.l.Localizer - Reconstruct localized resource: /tmp/ddf53adf-8313-4830-8786-4bd66e78b4fe/supervisor/usercache
4686 [main] WARN o.a.s.l.Localizer - No left over resources found for any user during reconstructing of local resources at: /tmp/ddf53adf-8313-4830-8786-4bd66e78b4fe/supervisor/usercache
4691 [main] INFO o.a.s.d.s.Supervisor - Starting Supervisor with conf {topology.builtin.metrics.bucket.size.secs=60, nimbus.childopts=-Xmx1024m, ui.filter.params=null, storm.cluster.mode=local, storm.messaging.netty.client_worker_threads=1, logviewer.max.per.worker.logs.size.mb=2048, supervisor.run.worker.as.user=false, topology.max.task.parallelism=null, topology.priority=29, zmq.threads=1, storm.group.mapping.service=org.apache.storm.security.auth.ShellBasedGroupsMapping, transactional.zookeeper.root=/transactional, topology.sleep.spout.wait.strategy.time.ms=1, ui.pagination=20, scheduler.display.resource=false, topology.max.replication.wait.time.sec=60, drpc.invocations.port=3773, supervisor.localizer.cache.target.size.mb=10240, topology.multilang.serializer=org.apache.storm.multilang.JsonSerializer, storm.messaging.netty.server_worker_threads=1, nimbus.blobstore.class=org.apache.storm.blobstore.LocalFsBlobStore, resource.aware.scheduler.eviction.strategy=org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy, topology.max.error.report.per.interval=5, storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin, zmq.hwm=0, storm.group.mapping.service.params=null, worker.profiler.enabled=false, storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal, supervisor.worker.shutdown.sleep.secs=3, pacemaker.host=localhost, storm.zookeeper.retry.times=5, ui.actions.enabled=true, zmq.linger.millis=0, supervisor.enable=true, topology.stats.sample.rate=0.05, storm.nimbus.zookeeper.acls.fixup=true, storm.messaging.netty.min_wait_ms=100, worker.log.level.reset.poll.secs=30, storm.zookeeper.port=2000, supervisor.heartbeat.frequency.secs=5, topology.enable.message.timeouts=true, supervisor.cpu.capacity=400.0, drpc.worker.threads=64, supervisor.blobstore.download.thread.count=5, task.backpressure.poll.secs=30, drpc.queue.size=128, topology.backpressure.enable=false, supervisor.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.blobstore.inputstream.buffer.size.bytes=65536, topology.shellbolt.max.pending=100, drpc.https.keystore.password=*****, nimbus.code.sync.freq.secs=120, logviewer.port=8000, topology.scheduler.strategy=org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy, topology.executor.send.buffer.size=1024, resource.aware.scheduler.priority.strategy=org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy, pacemaker.auth.method=NONE, storm.daemon.metrics.reporter.plugins=[org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter], topology.worker.logwriter.childopts=-Xmx64m, topology.spout.wait.strategy=org.apache.storm.spout.SleepSpoutWaitStrategy, ui.host=0.0.0.0, storm.nimbus.retry.interval.millis=2000, nimbus.inbox.jar.expiration.secs=3600, dev.zookeeper.path=/tmp/dev-storm-zookeeper, topology.acker.executors=null, topology.fall.back.on.java.serialization=true, topology.eventlogger.executors=0, supervisor.localizer.cleanup.interval.ms=600000, storm.zookeeper.servers=[localhost], nimbus.thrift.threads=64, logviewer.cleanup.age.mins=10080, topology.worker.childopts=null, topology.classpath=null, supervisor.monitor.frequency.secs=3, nimbus.credential.renewers.freq.secs=600, backpressure.znode.timeout.secs=30, topology.skip.missing.kryo.registrations=true, drpc.authorizer.acl.filename=drpc-auth-acl.yaml, pacemaker.kerberos.users=[], storm.group.mapping.service.cache.duration.secs=120, blobstore.dir=/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e, topology.testing.always.try.serialize=false, nimbus.monitor.freq.secs=10, storm.health.check.timeout.ms=5000, supervisor.supervisors=[], topology.tasks=null, topology.bolts.outgoing.overflow.buffer.enable=false, storm.messaging.netty.socket.backlog=500, topology.workers=1, pacemaker.base.threads=10, storm.local.dir=/tmp/ddf53adf-8313-4830-8786-4bd66e78b4fe, backpressure.znode.update.freq.secs=15, worker.childopts=-Xmx%HEAP-MEM%m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump, storm.auth.simple-white-list.users=[], topology.disruptor.batch.timeout.millis=1, topology.message.timeout.secs=30, topology.state.synchronization.timeout.secs=60, topology.tuple.serializer=org.apache.storm.serialization.types.ListDelegateSerializer, supervisor.supervisors.commands=[], nimbus.blobstore.expiration.secs=600, logviewer.childopts=-Xmx128m, storm.nimbus.zookeeper.acls.check=true, topology.environment=null, topology.debug=false, topology.disruptor.batch.size=100, storm.disable.symlinks=false, storm.messaging.netty.max_retries=300, ui.childopts=-Xmx768m, storm.network.topography.plugin=org.apache.storm.networktopography.DefaultRackDNSToSwitchMapping, storm.zookeeper.session.timeout=20000, drpc.childopts=-Xmx768m, drpc.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.connection.timeout=15000, storm.zookeeper.auth.user=null, storm.meta.serialization.delegate=org.apache.storm.serialization.GzipThriftSerializationDelegate, topology.max.spout.pending=null, storm.codedistributor.class=org.apache.storm.codedistributor.LocalFileSystemCodeDistributor, nimbus.supervisor.timeout.secs=60, nimbus.task.timeout.secs=30, drpc.port=3772, pacemaker.max.threads=50, storm.zookeeper.retry.intervalceiling.millis=30000, nimbus.thrift.port=6627, storm.auth.simple-acl.admins=[], topology.component.cpu.pcore.percent=10.0, supervisor.memory.capacity.mb=3072.0, storm.nimbus.retry.times=5, supervisor.worker.start.timeout.secs=120, storm.zookeeper.retry.interval=1000, logs.users=null, storm.cluster.metrics.consumer.publish.interval.secs=60, worker.profiler.command=flight.bash, transactional.zookeeper.port=null, drpc.max_buffer_size=1048576, pacemaker.thread.timeout=10, task.credentials.poll.secs=30, blobstore.superuser=ubuntu, drpc.https.keystore.type=JKS, topology.worker.receiver.thread.count=1, topology.state.checkpoint.interval.ms=1000, supervisor.slots.ports=[1024, 1025, 1026], topology.transfer.buffer.size=1024, storm.health.check.dir=healthchecks, topology.worker.shared.thread.pool.size=4, drpc.authorizer.acl.strict=false, nimbus.file.copy.expiration.secs=600, worker.profiler.childopts=-XX:+UnlockCommercialFeatures -XX:+FlightRecorder, topology.executor.receive.buffer.size=1024, backpressure.disruptor.low.watermark=0.4, nimbus.task.launch.secs=120, storm.local.mode.zmq=false, storm.messaging.netty.buffer_size=5242880, storm.cluster.state.store=org.apache.storm.cluster_state.zookeeper_state_factory, worker.heartbeat.frequency.secs=1, storm.log4j2.conf.dir=log4j2, ui.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.root=/storm, topology.tick.tuple.freq.secs=null, drpc.https.port=-1, storm.workers.artifacts.dir=workers-artifacts, supervisor.blobstore.download.max_retries=3, task.refresh.poll.secs=10, storm.exhibitor.port=8080, task.heartbeat.frequency.secs=3, pacemaker.port=6699, storm.messaging.netty.max_wait_ms=1000, topology.component.resources.offheap.memory.mb=0.0, drpc.http.port=3774, topology.error.throttle.interval.secs=10, storm.messaging.transport=org.apache.storm.messaging.netty.Context, topology.disable.loadaware.messaging=false, storm.messaging.netty.authentication=false, topology.component.resources.onheap.memory.mb=128.0, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, worker.gc.childopts=, nimbus.topology.validator=org.apache.storm.nimbus.DefaultTopologyValidator, nimbus.seeds=[localhost], nimbus.queue.size=100000, nimbus.cleanup.inbox.freq.secs=600, storm.blobstore.replication.factor=3, worker.heap.memory.mb=768, logviewer.max.sum.worker.logs.size.mb=4096, pacemaker.childopts=-Xmx1024m, ui.users=null, transactional.zookeeper.servers=null, supervisor.worker.timeout.secs=30, storm.zookeeper.auth.password=null, storm.blobstore.acl.validation.enabled=false, client.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.thrift.socket.timeout.ms=600000, supervisor.childopts=-Xmx256m, topology.worker.max.heap.size.mb=768.0, ui.http.x-frame-options=DENY, backpressure.disruptor.high.watermark=0.9, ui.filter=null, ui.header.buffer.bytes=4096, topology.min.replication.count=1, topology.disruptor.wait.timeout.millis=1000, storm.nimbus.retry.intervalceiling.millis=60000, topology.trident.batch.emit.interval.millis=50, storm.auth.simple-acl.users=[], drpc.invocations.threads=64, java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib, ui.port=8080, storm.exhibitor.poll.uripath=/exhibitor/v1/cluster/list, storm.messaging.netty.transfer.batch.size=262144, logviewer.appender.name=A1, nimbus.thrift.max_buffer_size=1048576, storm.auth.simple-acl.users.commands=[], drpc.request.timeout.secs=600}
4704 [main] WARN o.a.s.d.s.Slot - SLOT ip-172-31-22-101.us-west-2.compute.internal:1024 Starting in state EMPTY - assignment null
4704 [main] WARN o.a.s.d.s.Slot - SLOT ip-172-31-22-101.us-west-2.compute.internal:1025 Starting in state EMPTY - assignment null
4705 [main] WARN o.a.s.d.s.Slot - SLOT ip-172-31-22-101.us-west-2.compute.internal:1026 Starting in state EMPTY - assignment null
4705 [main] INFO o.a.s.l.AsyncLocalizer - Cleaning up unused topologies in /tmp/ddf53adf-8313-4830-8786-4bd66e78b4fe/supervisor/stormdist
4707 [main] INFO o.a.s.d.s.Supervisor - Starting supervisor with id ab823f98-11b3-4021-b8e7-f9da7ebc12f3 at host ip-172-31-22-101.us-west-2.compute.internal.
4708 [main] INFO o.a.s.z.Zookeeper - Staring ZK Curator
4709 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4709 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@405a296e
4711 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4712 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4712 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4712 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55332
4712 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55332
4714 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f2000b with negotiated timeout 20000 for client /127.0.0.1:55332
4714 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f2000b, negotiated timeout = 20000
4715 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4718 [Curator-Framework-0] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - backgroundOperationsLoop exiting
4718 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Processed session termination for sessionid: 0x100040091f2000b
4719 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxn - Closed socket connection for client /127.0.0.1:55332 which had sessionid 0x100040091f2000b
4719 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Session: 0x100040091f2000b closed
4720 [main-EventThread] INFO o.a.s.s.o.a.z.ClientCnxn - EventThread shut down for session: 0x100040091f2000b
4721 [main] INFO o.a.s.z.Zookeeper - Staring ZK Curator
4721 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
4723 [main] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@12f85dc8
4732 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
4732 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
4732 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55334
4732 [main] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
4732 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55334
4734 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f2000c with negotiated timeout 20000 for client /127.0.0.1:55334
4734 [main-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f2000c, negotiated timeout = 20000
4735 [main-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
4741 [main] INFO o.a.s.l.Localizer - Reconstruct localized resource: /tmp/a84faee2-1f19-46b6-affa-3c9d4bb2d61a/supervisor/usercache
4741 [main] WARN o.a.s.l.Localizer - No left over resources found for any user during reconstructing of local resources at: /tmp/a84faee2-1f19-46b6-affa-3c9d4bb2d61a/supervisor/usercache
4743 [main] INFO o.a.s.d.s.Supervisor - Starting Supervisor with conf {topology.builtin.metrics.bucket.size.secs=60, nimbus.childopts=-Xmx1024m, ui.filter.params=null, storm.cluster.mode=local, storm.messaging.netty.client_worker_threads=1, logviewer.max.per.worker.logs.size.mb=2048, supervisor.run.worker.as.user=false, topology.max.task.parallelism=null, topology.priority=29, zmq.threads=1, storm.group.mapping.service=org.apache.storm.security.auth.ShellBasedGroupsMapping, transactional.zookeeper.root=/transactional, topology.sleep.spout.wait.strategy.time.ms=1, ui.pagination=20, scheduler.display.resource=false, topology.max.replication.wait.time.sec=60, drpc.invocations.port=3773, supervisor.localizer.cache.target.size.mb=10240, topology.multilang.serializer=org.apache.storm.multilang.JsonSerializer, storm.messaging.netty.server_worker_threads=1, nimbus.blobstore.class=org.apache.storm.blobstore.LocalFsBlobStore, resource.aware.scheduler.eviction.strategy=org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy, topology.max.error.report.per.interval=5, storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin, zmq.hwm=0, storm.group.mapping.service.params=null, worker.profiler.enabled=false, storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal, supervisor.worker.shutdown.sleep.secs=3, pacemaker.host=localhost, storm.zookeeper.retry.times=5, ui.actions.enabled=true, zmq.linger.millis=0, supervisor.enable=true, topology.stats.sample.rate=0.05, storm.nimbus.zookeeper.acls.fixup=true, storm.messaging.netty.min_wait_ms=100, worker.log.level.reset.poll.secs=30, storm.zookeeper.port=2000, supervisor.heartbeat.frequency.secs=5, topology.enable.message.timeouts=true, supervisor.cpu.capacity=400.0, drpc.worker.threads=64, supervisor.blobstore.download.thread.count=5, task.backpressure.poll.secs=30, drpc.queue.size=128, topology.backpressure.enable=false, supervisor.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.blobstore.inputstream.buffer.size.bytes=65536, topology.shellbolt.max.pending=100, drpc.https.keystore.password=*****, nimbus.code.sync.freq.secs=120, logviewer.port=8000, topology.scheduler.strategy=org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy, topology.executor.send.buffer.size=1024, resource.aware.scheduler.priority.strategy=org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy, pacemaker.auth.method=NONE, storm.daemon.metrics.reporter.plugins=[org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter], topology.worker.logwriter.childopts=-Xmx64m, topology.spout.wait.strategy=org.apache.storm.spout.SleepSpoutWaitStrategy, ui.host=0.0.0.0, storm.nimbus.retry.interval.millis=2000, nimbus.inbox.jar.expiration.secs=3600, dev.zookeeper.path=/tmp/dev-storm-zookeeper, topology.acker.executors=null, topology.fall.back.on.java.serialization=true, topology.eventlogger.executors=0, supervisor.localizer.cleanup.interval.ms=600000, storm.zookeeper.servers=[localhost], nimbus.thrift.threads=64, logviewer.cleanup.age.mins=10080, topology.worker.childopts=null, topology.classpath=null, supervisor.monitor.frequency.secs=3, nimbus.credential.renewers.freq.secs=600, backpressure.znode.timeout.secs=30, topology.skip.missing.kryo.registrations=true, drpc.authorizer.acl.filename=drpc-auth-acl.yaml, pacemaker.kerberos.users=[], storm.group.mapping.service.cache.duration.secs=120, blobstore.dir=/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e, topology.testing.always.try.serialize=false, nimbus.monitor.freq.secs=10, storm.health.check.timeout.ms=5000, supervisor.supervisors=[], topology.tasks=null, topology.bolts.outgoing.overflow.buffer.enable=false, storm.messaging.netty.socket.backlog=500, topology.workers=1, pacemaker.base.threads=10, storm.local.dir=/tmp/a84faee2-1f19-46b6-affa-3c9d4bb2d61a, backpressure.znode.update.freq.secs=15, worker.childopts=-Xmx%HEAP-MEM%m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump, storm.auth.simple-white-list.users=[], topology.disruptor.batch.timeout.millis=1, topology.message.timeout.secs=30, topology.state.synchronization.timeout.secs=60, topology.tuple.serializer=org.apache.storm.serialization.types.ListDelegateSerializer, supervisor.supervisors.commands=[], nimbus.blobstore.expiration.secs=600, logviewer.childopts=-Xmx128m, storm.nimbus.zookeeper.acls.check=true, topology.environment=null, topology.debug=false, topology.disruptor.batch.size=100, storm.disable.symlinks=false, storm.messaging.netty.max_retries=300, ui.childopts=-Xmx768m, storm.network.topography.plugin=org.apache.storm.networktopography.DefaultRackDNSToSwitchMapping, storm.zookeeper.session.timeout=20000, drpc.childopts=-Xmx768m, drpc.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.connection.timeout=15000, storm.zookeeper.auth.user=null, storm.meta.serialization.delegate=org.apache.storm.serialization.GzipThriftSerializationDelegate, topology.max.spout.pending=null, storm.codedistributor.class=org.apache.storm.codedistributor.LocalFileSystemCodeDistributor, nimbus.supervisor.timeout.secs=60, nimbus.task.timeout.secs=30, drpc.port=3772, pacemaker.max.threads=50, storm.zookeeper.retry.intervalceiling.millis=30000, nimbus.thrift.port=6627, storm.auth.simple-acl.admins=[], topology.component.cpu.pcore.percent=10.0, supervisor.memory.capacity.mb=3072.0, storm.nimbus.retry.times=5, supervisor.worker.start.timeout.secs=120, storm.zookeeper.retry.interval=1000, logs.users=null, storm.cluster.metrics.consumer.publish.interval.secs=60, worker.profiler.command=flight.bash, transactional.zookeeper.port=null, drpc.max_buffer_size=1048576, pacemaker.thread.timeout=10, task.credentials.poll.secs=30, blobstore.superuser=ubuntu, drpc.https.keystore.type=JKS, topology.worker.receiver.thread.count=1, topology.state.checkpoint.interval.ms=1000, supervisor.slots.ports=[1027, 1028, 1029], topology.transfer.buffer.size=1024, storm.health.check.dir=healthchecks, topology.worker.shared.thread.pool.size=4, drpc.authorizer.acl.strict=false, nimbus.file.copy.expiration.secs=600, worker.profiler.childopts=-XX:+UnlockCommercialFeatures -XX:+FlightRecorder, topology.executor.receive.buffer.size=1024, backpressure.disruptor.low.watermark=0.4, nimbus.task.launch.secs=120, storm.local.mode.zmq=false, storm.messaging.netty.buffer_size=5242880, storm.cluster.state.store=org.apache.storm.cluster_state.zookeeper_state_factory, worker.heartbeat.frequency.secs=1, storm.log4j2.conf.dir=log4j2, ui.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.root=/storm, topology.tick.tuple.freq.secs=null, drpc.https.port=-1, storm.workers.artifacts.dir=workers-artifacts, supervisor.blobstore.download.max_retries=3, task.refresh.poll.secs=10, storm.exhibitor.port=8080, task.heartbeat.frequency.secs=3, pacemaker.port=6699, storm.messaging.netty.max_wait_ms=1000, topology.component.resources.offheap.memory.mb=0.0, drpc.http.port=3774, topology.error.throttle.interval.secs=10, storm.messaging.transport=org.apache.storm.messaging.netty.Context, topology.disable.loadaware.messaging=false, storm.messaging.netty.authentication=false, topology.component.resources.onheap.memory.mb=128.0, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, worker.gc.childopts=, nimbus.topology.validator=org.apache.storm.nimbus.DefaultTopologyValidator, nimbus.seeds=[localhost], nimbus.queue.size=100000, nimbus.cleanup.inbox.freq.secs=600, storm.blobstore.replication.factor=3, worker.heap.memory.mb=768, logviewer.max.sum.worker.logs.size.mb=4096, pacemaker.childopts=-Xmx1024m, ui.users=null, transactional.zookeeper.servers=null, supervisor.worker.timeout.secs=30, storm.zookeeper.auth.password=null, storm.blobstore.acl.validation.enabled=false, client.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.thrift.socket.timeout.ms=600000, supervisor.childopts=-Xmx256m, topology.worker.max.heap.size.mb=768.0, ui.http.x-frame-options=DENY, backpressure.disruptor.high.watermark=0.9, ui.filter=null, ui.header.buffer.bytes=4096, topology.min.replication.count=1, topology.disruptor.wait.timeout.millis=1000, storm.nimbus.retry.intervalceiling.millis=60000, topology.trident.batch.emit.interval.millis=50, storm.auth.simple-acl.users=[], drpc.invocations.threads=64, java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib, ui.port=8080, storm.exhibitor.poll.uripath=/exhibitor/v1/cluster/list, storm.messaging.netty.transfer.batch.size=262144, logviewer.appender.name=A1, nimbus.thrift.max_buffer_size=1048576, storm.auth.simple-acl.users.commands=[], drpc.request.timeout.secs=600}
4750 [main] WARN o.a.s.d.s.Slot - SLOT ip-172-31-22-101.us-west-2.compute.internal:1027 Starting in state EMPTY - assignment null
4750 [main] WARN o.a.s.d.s.Slot - SLOT ip-172-31-22-101.us-west-2.compute.internal:1028 Starting in state EMPTY - assignment null
4750 [main] WARN o.a.s.d.s.Slot - SLOT ip-172-31-22-101.us-west-2.compute.internal:1029 Starting in state EMPTY - assignment null
4750 [main] INFO o.a.s.l.AsyncLocalizer - Cleaning up unused topologies in /tmp/a84faee2-1f19-46b6-affa-3c9d4bb2d61a/supervisor/stormdist
4751 [main] INFO o.a.s.d.s.Supervisor - Starting supervisor with id d9f039bd-9a40-4a5b-ae52-b2638fbedaf8 at host ip-172-31-22-101.us-west-2.compute.internal.
4759 [main] WARN o.a.s.u.Utils - STORM-VERSION new 1.2.3 old 1.2.3
4850 [main] INFO o.a.s.d.nimbus - Received topology submission for crawler (storm-1.2.3 JDK-11.0.4) with conf {parser.emitOutlinks.max.per.page=-1, es.metrics.index.name=metrics, topology.max.task.parallelism=null, status.updater.cache.spec=maximumSize=10000,expireAfterAccess=1h, fetcher.queue.mode=byHost, http.agent.email=, status.updater.use.cache=true, partition.url.mode=byHost, status.updater.unit.round.date=SECOND, max.fetch.errors=3, selenium.implicitlyWait=0, robots.error.cache.spec=maximumSize=10000,expireAfterWrite=1h, detect.charset.maxlength=10000, es.status.concurrentRequests=1, es.status.routing.fieldname=key, selenium.pageLoadTimeout=-1, indexer.url.fieldname=url, es.indexer.addresses=localhost, fetchInterval.fetch.error=120, spout.reset.fetchdate.after=120, fetcher.max.urls.in.queues=-1, fetchInterval.default=1440, es.status.recentDate.min.gap=-1, http.agent.version=0.1, urlbuffer.class=com.digitalpebble.stormcrawler.persistence.SimpleURLBuffer, es.status.max.start.offset=500, http.agent.description=, es.status.addresses=http://localhost:9200, topology.submitter.principal=, topology.acker.executors=null, topology.eventlogger.executors=0, http.content.partial.as.trimmed=false, topology.worker.childopts=-Xmx2g -Djava.net.preferIPv4Stack=true, file.protocol.implementation=com.digitalpebble.stormcrawler.protocol.file.FileProtocol, http.timeout=10000, es.status.bucket.sort.field=["nextFetchDate" "url"], fetcher.timeout.queue=-1, http.agent.name=Google-Read-Aloud, topology.workers=1, jsoup.treat.non.html.as.error=true, topology.message.timeout.secs=300, topology.debug=false, sitemap.discovery=false, spout.min.delay.queries=2000, sitemap.filter.hours.since.modified=-1, textextractor.include.pattern=["DIV[id=\"maincontent\"]" "DIV[itemprop=\"articleBody\"]" "ARTICLE"], topology.max.spout.pending=100, fetcher.threads.per.queue=1, protocols=http,https,file, fetcher.server.delay.force=false, storm.zookeeper.superACL=null, es.indexer.settings={"cluster.name" "elasticsearch"}, indexer.text.fieldname=content, http.accept.language=en-us,en-gb,en;q=0.7,*;q=0.3, es.status.sample=false, http.robots.403.allow=true, http.store.headers=false, fetcher.server.delay=1.0, http.agent.url=https://support.google.com/webmasters/answer/1061943, fetcher.metrics.time.bucket.secs=10, es.status.index.name=status, spout.ttl.purgatory=30, detect.mimetype=true, fetcher.max.crawl.delay.force=false, sitemap.schedule.delay=-1, metadata.track.depth=true, es.status.settings={"cluster.name" "elasticsearch"}, es.status.global.sort.field=nextFetchDate, es.indexer.create=false, topology.users=clojure.lang.LazySeq@1, track.anchors=true, topology.submitter.user=ubuntu, metadata.track.path=true, http.content.limit=65536, http.accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, topology.metrics.consumer.register=[{"class" "com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer", "parallelism.hint" 1}], fetcher.max.crawl.delay=30, es.status.bulkActions=500, es.indexer.flushInterval=2s, es.status.flushInterval=5s, selenium.delegated.protocol=com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol, es.indexer.bulkActions=100, topology.kryo.register={"com.digitalpebble.stormcrawler.Metadata" nil}, urlfilters.config.file=urlfilters.json, es.status.query.timeout=-1, metadata.persist=["_redirTo" "error.cause" "error.source" "isSitemap" "isFeed"], parser.emitOutlinks=true, textextractor.exclude.tags=["STYLE" "SCRIPT"], indexer.text.maxlength=-1, fetcher.threads.number=50, es.metrics.settings={"cluster.name" "elasticsearch"}, selenium.instances.num=1, selenium.capabilities={"loadImages" false, "takesScreenshot" false, "javascriptEnabled" true}, scheduler.class=com.digitalpebble.stormcrawler.persistence.DefaultScheduler, robots.cache.spec=maximumSize=10000,expireAfterWrite=6h, es.metrics.addresses=http://localhost:9200, es.indexer.concurrentRequests=1, robots.noFollow.strict=true, indexer.md.mapping=["parse.title=title" "parse.keywords=keywords" "parse.description=description" "domain=domain"], es.status.recentDate.increase=-1, https.protocol.implementation=com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol, es.status.max.urls.per.bucket=2, es.indexer.index.name=content, fetcher.max.queue.size=-1, selenium.setScriptTimeout=0, indexer.canonical.name=canonical, es.status.bucket.field=key, es.status.routing=true, topology.kryo.decorators=clojure.lang.LazySeq@1, es.status.max.buckets=50, storm.id=crawler-1-1574269039, topology.name=crawler, fetcher.max.throttle.sleep=-1, fetcher.server.min.delay=0.0, parsefilters.config.file=parsefilters.json, http.protocol.implementation=com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol, fetchInterval.error=-1}
4856 [main] INFO o.a.s.d.nimbus - uploadedJar
4866 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f20000 type:create cxid:0x9 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/storm/blobstoremaxkeysequencenumber Error:KeeperErrorCode = NoNode for /storm/blobstoremaxkeysequencenumber
4874 [main] INFO o.a.s.cluster - setup-path/blobstore/crawler-1-1574269039-stormconf.ser/ip-172-31-22-101.us-west-2.compute.internal:6627-1
4919 [main] INFO o.a.s.cluster - setup-path/blobstore/crawler-1-1574269039-stormcode.ser/ip-172-31-22-101.us-west-2.compute.internal:6627-1
4947 [main] INFO o.a.s.d.nimbus - desired replication count 1 achieved, current-replication-count for conf key = 1, current-replication-count for code key = 1, current-replication-count for jar key = 1
5004 [main] INFO o.a.s.d.nimbus - Activating crawler: crawler-1-1574269039
5502 [timer] INFO o.a.s.s.EvenScheduler - Available slots: (["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1025] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1026] ["d9f039bd-9a40-4a5b-ae52-b2638fbedaf8" 1027] ["d9f039bd-9a40-4a5b-ae52-b2638fbedaf8" 1028] ["d9f039bd-9a40-4a5b-ae52-b2638fbedaf8" 1029])
5535 [timer] INFO o.a.s.d.nimbus - Setting new assignment for topology id crawler-1-1574269039: #org.apache.storm.daemon.common.Assignment{:master-code-dir "/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e", :node->host {"ab823f98-11b3-4021-b8e7-f9da7ebc12f3" "ip-172-31-22-101.us-west-2.compute.internal"}, :executor->node+port {[8 8] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [12 12] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [2 2] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [7 7] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [3 3] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [1 1] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [18 18] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [6 6] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [9 9] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [11 11] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [16 16] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [13 13] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [19 19] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [5 5] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [10 10] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [14 14] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [4 4] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [15 15] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024], [17 17] ["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024]}, :executor->start-time-secs {[8 8] 1574269040, [12 12] 1574269040, [2 2] 1574269040, [7 7] 1574269040, [3 3] 1574269040, [1 1] 1574269040, [18 18] 1574269040, [6 6] 1574269040, [9 9] 1574269040, [11 11] 1574269040, [16 16] 1574269040, [13 13] 1574269040, [19 19] 1574269040, [5 5] 1574269040, [10 10] 1574269040, [14 14] 1574269040, [4 4] 1574269040, [15 15] 1574269040, [17 17] 1574269040}, :worker->resources {["ab823f98-11b3-4021-b8e7-f9da7ebc12f3" 1024] [0.0 0.0 0.0]}, :owner "ubuntu"}
5712 [SLOT_1024] INFO o.a.s.d.s.Slot - STATE EMPTY msInState: 1008 -> WAITING_FOR_BASIC_LOCALIZATION msInState: 0
5712 [Async Localizer] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
5725 [Async Localizer] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@4a59bca0
5726 [Async Localizer-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
5727 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55336
5727 [Async Localizer-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
5728 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55336
5728 [Async Localizer] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
5728 [Async Localizer] INFO o.a.s.b.FileBlobStoreImpl - Creating new blob store based in /tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e/blobs
5729 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f2000d with negotiated timeout 20000 for client /127.0.0.1:55336
5730 [Async Localizer-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f2000d, negotiated timeout = 20000
5730 [Async Localizer-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
5736 [Curator-Framework-0] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - backgroundOperationsLoop exiting
5740 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Processed session termination for sessionid: 0x100040091f2000d
5741 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxn - Closed socket connection for client /127.0.0.1:55336 which had sessionid 0x100040091f2000d
5741 [Async Localizer] INFO o.a.s.s.o.a.z.ZooKeeper - Session: 0x100040091f2000d closed
5741 [Async Localizer-EventThread] INFO o.a.s.s.o.a.z.ClientCnxn - EventThread shut down for session: 0x100040091f2000d
5790 [Async Localizer] INFO o.a.s.l.AsyncLocalizer - Extracting resources from jar at target/dev1-0.1.jar to /tmp/ddf53adf-8313-4830-8786-4bd66e78b4fe/supervisor/tmp/9bbef7a4-772d-4a8e-8c37-b14acb2718ee/
5804 [SLOT_1024] INFO o.a.s.d.s.Slot - STATE WAITING_FOR_BASIC_LOCALIZATION msInState: 92 -> WAITING_FOR_BLOB_LOCALIZATION msInState: 0
5811 [SLOT_1024] INFO o.a.s.d.s.Container - Setting up ab823f98-11b3-4021-b8e7-f9da7ebc12f3:f5d45862-968e-43f7-97e8-669e3dae7d11
5816 [SLOT_1024] INFO o.a.s.d.s.Container - GET worker-user for f5d45862-968e-43f7-97e8-669e3dae7d11
5826 [SLOT_1024] INFO o.a.s.d.s.Container - SET worker-user f5d45862-968e-43f7-97e8-669e3dae7d11 ubuntu
5832 [SLOT_1024] INFO o.a.s.d.worker - Launching worker for crawler-1-1574269039 on ab823f98-11b3-4021-b8e7-f9da7ebc12f3:1024 with id f5d45862-968e-43f7-97e8-669e3dae7d11 and conf {topology.builtin.metrics.bucket.size.secs=60, nimbus.childopts=-Xmx1024m, ui.filter.params=null, storm.cluster.mode=local, storm.messaging.netty.client_worker_threads=1, logviewer.max.per.worker.logs.size.mb=2048, supervisor.run.worker.as.user=false, topology.max.task.parallelism=null, topology.priority=29, zmq.threads=1, storm.group.mapping.service=org.apache.storm.security.auth.ShellBasedGroupsMapping, transactional.zookeeper.root=/transactional, topology.sleep.spout.wait.strategy.time.ms=1, ui.pagination=20, scheduler.display.resource=false, topology.max.replication.wait.time.sec=60, drpc.invocations.port=3773, supervisor.localizer.cache.target.size.mb=10240, topology.multilang.serializer=org.apache.storm.multilang.JsonSerializer, storm.messaging.netty.server_worker_threads=1, nimbus.blobstore.class=org.apache.storm.blobstore.LocalFsBlobStore, resource.aware.scheduler.eviction.strategy=org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy, topology.max.error.report.per.interval=5, storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin, zmq.hwm=0, storm.group.mapping.service.params=null, worker.profiler.enabled=false, storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal, supervisor.worker.shutdown.sleep.secs=3, pacemaker.host=localhost, storm.zookeeper.retry.times=5, ui.actions.enabled=true, zmq.linger.millis=0, supervisor.enable=true, topology.stats.sample.rate=0.05, storm.nimbus.zookeeper.acls.fixup=true, storm.messaging.netty.min_wait_ms=100, worker.log.level.reset.poll.secs=30, storm.zookeeper.port=2000, supervisor.heartbeat.frequency.secs=5, topology.enable.message.timeouts=true, supervisor.cpu.capacity=400.0, drpc.worker.threads=64, supervisor.blobstore.download.thread.count=5, task.backpressure.poll.secs=30, drpc.queue.size=128, topology.backpressure.enable=false, supervisor.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.blobstore.inputstream.buffer.size.bytes=65536, topology.shellbolt.max.pending=100, drpc.https.keystore.password=*****, nimbus.code.sync.freq.secs=120, logviewer.port=8000, topology.scheduler.strategy=org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy, topology.executor.send.buffer.size=1024, resource.aware.scheduler.priority.strategy=org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy, pacemaker.auth.method=NONE, storm.daemon.metrics.reporter.plugins=["org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter"], topology.worker.logwriter.childopts=-Xmx64m, topology.spout.wait.strategy=org.apache.storm.spout.SleepSpoutWaitStrategy, ui.host=0.0.0.0, storm.nimbus.retry.interval.millis=2000, nimbus.inbox.jar.expiration.secs=3600, dev.zookeeper.path=/tmp/dev-storm-zookeeper, topology.acker.executors=null, topology.fall.back.on.java.serialization=true, topology.eventlogger.executors=0, supervisor.localizer.cleanup.interval.ms=600000, storm.zookeeper.servers=["localhost"], nimbus.thrift.threads=64, logviewer.cleanup.age.mins=10080, topology.worker.childopts=null, topology.classpath=null, supervisor.monitor.frequency.secs=3, nimbus.credential.renewers.freq.secs=600, backpressure.znode.timeout.secs=30, topology.skip.missing.kryo.registrations=true, drpc.authorizer.acl.filename=drpc-auth-acl.yaml, pacemaker.kerberos.users=[], storm.group.mapping.service.cache.duration.secs=120, blobstore.dir=/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e, topology.testing.always.try.serialize=false, nimbus.monitor.freq.secs=10, storm.health.check.timeout.ms=5000, supervisor.supervisors=[], topology.tasks=null, topology.bolts.outgoing.overflow.buffer.enable=false, storm.messaging.netty.socket.backlog=500, topology.workers=1, pacemaker.base.threads=10, storm.local.dir=/tmp/ddf53adf-8313-4830-8786-4bd66e78b4fe, backpressure.znode.update.freq.secs=15, worker.childopts=-Xmx%HEAP-MEM%m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump, storm.auth.simple-white-list.users=[], topology.disruptor.batch.timeout.millis=1, topology.message.timeout.secs=30, topology.state.synchronization.timeout.secs=60, topology.tuple.serializer=org.apache.storm.serialization.types.ListDelegateSerializer, supervisor.supervisors.commands=[], nimbus.blobstore.expiration.secs=600, logviewer.childopts=-Xmx128m, storm.nimbus.zookeeper.acls.check=true, topology.environment=null, topology.debug=false, topology.disruptor.batch.size=100, storm.disable.symlinks=false, storm.messaging.netty.max_retries=300, ui.childopts=-Xmx768m, storm.network.topography.plugin=org.apache.storm.networktopography.DefaultRackDNSToSwitchMapping, storm.zookeeper.session.timeout=20000, drpc.childopts=-Xmx768m, drpc.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.connection.timeout=15000, storm.zookeeper.auth.user=null, storm.meta.serialization.delegate=org.apache.storm.serialization.GzipThriftSerializationDelegate, topology.max.spout.pending=null, storm.codedistributor.class=org.apache.storm.codedistributor.LocalFileSystemCodeDistributor, nimbus.supervisor.timeout.secs=60, nimbus.task.timeout.secs=30, drpc.port=3772, pacemaker.max.threads=50, storm.zookeeper.retry.intervalceiling.millis=30000, nimbus.thrift.port=6627, storm.auth.simple-acl.admins=[], topology.component.cpu.pcore.percent=10.0, supervisor.memory.capacity.mb=3072.0, storm.nimbus.retry.times=5, supervisor.worker.start.timeout.secs=120, storm.zookeeper.retry.interval=1000, logs.users=null, storm.cluster.metrics.consumer.publish.interval.secs=60, worker.profiler.command=flight.bash, transactional.zookeeper.port=null, drpc.max_buffer_size=1048576, pacemaker.thread.timeout=10, task.credentials.poll.secs=30, blobstore.superuser=ubuntu, drpc.https.keystore.type=JKS, topology.worker.receiver.thread.count=1, topology.state.checkpoint.interval.ms=1000, supervisor.slots.ports=clojure.lang.LazySeq@ff880, topology.transfer.buffer.size=1024, storm.health.check.dir=healthchecks, topology.worker.shared.thread.pool.size=4, drpc.authorizer.acl.strict=false, nimbus.file.copy.expiration.secs=600, worker.profiler.childopts=-XX:+UnlockCommercialFeatures -XX:+FlightRecorder, topology.executor.receive.buffer.size=1024, backpressure.disruptor.low.watermark=0.4, nimbus.task.launch.secs=120, storm.local.mode.zmq=false, storm.messaging.netty.buffer_size=5242880, storm.cluster.state.store=org.apache.storm.cluster_state.zookeeper_state_factory, worker.heartbeat.frequency.secs=1, storm.log4j2.conf.dir=log4j2, ui.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.root=/storm, topology.tick.tuple.freq.secs=null, drpc.https.port=-1, storm.workers.artifacts.dir=workers-artifacts, supervisor.blobstore.download.max_retries=3, task.refresh.poll.secs=10, storm.exhibitor.port=8080, task.heartbeat.frequency.secs=3, pacemaker.port=6699, storm.messaging.netty.max_wait_ms=1000, topology.component.resources.offheap.memory.mb=0.0, drpc.http.port=3774, topology.error.throttle.interval.secs=10, storm.messaging.transport=org.apache.storm.messaging.netty.Context, topology.disable.loadaware.messaging=false, storm.messaging.netty.authentication=false, topology.component.resources.onheap.memory.mb=128.0, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, worker.gc.childopts=, nimbus.topology.validator=org.apache.storm.nimbus.DefaultTopologyValidator, nimbus.seeds=["localhost"], nimbus.queue.size=100000, nimbus.cleanup.inbox.freq.secs=600, storm.blobstore.replication.factor=3, worker.heap.memory.mb=768, logviewer.max.sum.worker.logs.size.mb=4096, pacemaker.childopts=-Xmx1024m, ui.users=null, transactional.zookeeper.servers=null, supervisor.worker.timeout.secs=30, storm.zookeeper.auth.password=null, storm.blobstore.acl.validation.enabled=false, client.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.thrift.socket.timeout.ms=600000, supervisor.childopts=-Xmx256m, topology.worker.max.heap.size.mb=768.0, ui.http.x-frame-options=DENY, backpressure.disruptor.high.watermark=0.9, ui.filter=null, ui.header.buffer.bytes=4096, topology.min.replication.count=1, topology.disruptor.wait.timeout.millis=1000, storm.nimbus.retry.intervalceiling.millis=60000, topology.trident.batch.emit.interval.millis=50, storm.auth.simple-acl.users=[], drpc.invocations.threads=64, java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib, ui.port=8080, storm.exhibitor.poll.uripath=/exhibitor/v1/cluster/list, storm.messaging.netty.transfer.batch.size=262144, logviewer.appender.name=A1, nimbus.thrift.max_buffer_size=1048576, storm.auth.simple-acl.users.commands=[], drpc.request.timeout.secs=600}
5834 [SLOT_1024] INFO o.a.s.m.StormMetricRegistry - Starting metrics reporters...
5839 [SLOT_1024] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
5840 [SLOT_1024] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@79eb66d
5848 [SLOT_1024] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
5852 [SLOT_1024-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
5852 [SLOT_1024-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
5852 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55338
5853 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55338
5854 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f2000e with negotiated timeout 20000 for client /127.0.0.1:55338
5855 [SLOT_1024-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f2000e, negotiated timeout = 20000
5855 [SLOT_1024-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
5856 [SLOT_1024-EventThread] INFO o.a.s.zookeeper - Zookeeper state update: :connected:none
5859 [Curator-Framework-0] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - backgroundOperationsLoop exiting
5860 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Processed session termination for sessionid: 0x100040091f2000e
5861 [SLOT_1024] INFO o.a.s.s.o.a.z.ZooKeeper - Session: 0x100040091f2000e closed
5861 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxn - Closed socket connection for client /127.0.0.1:55338 which had sessionid 0x100040091f2000e
5862 [SLOT_1024] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Starting
5862 [SLOT_1024-EventThread] INFO o.a.s.s.o.a.z.ClientCnxn - EventThread shut down for session: 0x100040091f2000e
5863 [SLOT_1024] INFO o.a.s.s.o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@2ded3962
5869 [SLOT_1024] INFO o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl - Default schema
5873 [SLOT_1024-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
5873 [SLOT_1024-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2000, initiating session
5873 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:55340
5873 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:55340
5875 [SyncThread:0] INFO o.a.s.s.o.a.z.s.ZooKeeperServer - Established session 0x100040091f2000f with negotiated timeout 20000 for client /127.0.0.1:55340
5876 [SLOT_1024-SendThread(localhost:2000)] INFO o.a.s.s.o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2000, sessionid = 0x100040091f2000f, negotiated timeout = 20000
5876 [SLOT_1024-EventThread] INFO o.a.s.s.o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
5882 [SLOT_1024] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
5885 [SLOT_1024] INFO o.a.s.d.worker - Reading Assignments.
6022 [SLOT_1024] INFO o.a.s.d.worker - Registering IConnectionCallbacks for ab823f98-11b3-4021-b8e7-f9da7ebc12f3:1024
6074 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[8 8]
6091 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[8 8]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/opt/apache-storm-1.2.3/lib/kryo-3.0.3.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
6224 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[8 8]
6235 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[12 12]
6236 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[12 12]
6250 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[12 12]
6257 [SLOT_1024] INFO o.a.s.d.executor - Loading executor __metricscom.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[2 2]
6264 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks __metricscom.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[2 2]
6269 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor __metricscom.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:[2 2]
6275 [SLOT_1024] INFO o.a.s.d.executor - Loading executor sitemap:[7 7]
6276 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks sitemap:[7 7]
6279 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor sitemap:[7 7]
6285 [SLOT_1024] INFO o.a.s.d.executor - Loading executor fetcher:[3 3]
6285 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks fetcher:[3 3]
6292 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor fetcher:[3 3]
6299 [SLOT_1024] INFO o.a.s.d.executor - Loading executor __acker:[1 1]
6300 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks __acker:[1 1]
6302 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor __acker:[1 1]
6308 [SLOT_1024] INFO o.a.s.d.executor - Loading executor status:[18 18]
6309 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks status:[18 18]
6312 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor status:[18 18]
6319 [SLOT_1024] INFO o.a.s.d.executor - Loading executor partitioner:[6 6]
6320 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks partitioner:[6 6]
6331 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor partitioner:[6 6]
6337 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[9 9]
6338 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[9 9]
6347 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[9 9]
6354 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[11 11]
6356 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[11 11]
6366 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[11 11]
6373 [SLOT_1024] INFO o.a.s.d.executor - Loading executor __system:[-1 -1]
6373 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks __system:[-1 -1]
6379 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor __system:[-1 -1]
6389 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[16 16]
6391 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[16 16]
6399 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[16 16]
6407 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[13 13]
6410 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[13 13]
6413 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[13 13]
6423 [SLOT_1024] INFO o.a.s.d.executor - Loading executor status_metrics:[19 19]
6423 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks status_metrics:[19 19]
6430 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor status_metrics:[19 19]
6438 [SLOT_1024] INFO o.a.s.d.executor - Loading executor parse:[5 5]
6520 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks parse:[5 5]
6526 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor parse:[5 5]
6536 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[10 10]
6537 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[10 10]
6546 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[10 10]
6553 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[14 14]
6554 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[14 14]
6562 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[14 14]
6568 [SLOT_1024] INFO o.a.s.d.executor - Loading executor index:[4 4]
6600 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks index:[4 4]
6606 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor index:[4 4]
6613 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[15 15]
6614 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[15 15]
6618 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[15 15]
6625 [SLOT_1024] INFO o.a.s.d.executor - Loading executor spout:[17 17]
6626 [SLOT_1024] INFO o.a.s.d.executor - Loaded executor tasks spout:[17 17]
6630 [SLOT_1024] INFO o.a.s.d.executor - Finished loading executor spout:[17 17]
6640 [SLOT_1024] INFO o.a.s.d.worker - Started with log levels: {"" #object[org.apache.logging.log4j.Level 0x2bcfe0cd "INFO"], "org.apache.zookeeper" #object[org.apache.logging.log4j.Level 0x29b0533e "WARN"]}
6648 [SLOT_1024] INFO o.a.s.d.worker - Worker has topology config {topology.builtin.metrics.bucket.size.secs=60, nimbus.childopts=-Xmx1024m, ui.filter.params=null, storm.cluster.mode=local, parser.emitOutlinks.max.per.page=-1, storm.messaging.netty.client_worker_threads=1, logviewer.max.per.worker.logs.size.mb=2048, supervisor.run.worker.as.user=false, es.metrics.index.name=metrics, topology.max.task.parallelism=null, status.updater.cache.spec=maximumSize=10000,expireAfterAccess=1h, topology.priority=29, zmq.threads=1, storm.group.mapping.service=org.apache.storm.security.auth.ShellBasedGroupsMapping, transactional.zookeeper.root=/transactional, fetcher.queue.mode=byHost, http.agent.email=, topology.sleep.spout.wait.strategy.time.ms=1, ui.pagination=20, status.updater.use.cache=true, scheduler.display.resource=false, topology.max.replication.wait.time.sec=60, drpc.invocations.port=3773, supervisor.localizer.cache.target.size.mb=10240, topology.multilang.serializer=org.apache.storm.multilang.JsonSerializer, storm.messaging.netty.server_worker_threads=1, nimbus.blobstore.class=org.apache.storm.blobstore.LocalFsBlobStore, resource.aware.scheduler.eviction.strategy=org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy, partition.url.mode=byHost, status.updater.unit.round.date=SECOND, max.fetch.errors=3, topology.max.error.report.per.interval=5, storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin, zmq.hwm=0, storm.group.mapping.service.params=null, worker.profiler.enabled=false, selenium.implicitlyWait=0, robots.error.cache.spec=maximumSize=10000,expireAfterWrite=1h, detect.charset.maxlength=10000, storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal, supervisor.worker.shutdown.sleep.secs=3, pacemaker.host=localhost, storm.zookeeper.retry.times=5, ui.actions.enabled=true, es.status.concurrentRequests=1, es.status.routing.fieldname=key, zmq.linger.millis=0, supervisor.enable=true, topology.stats.sample.rate=0.05, selenium.pageLoadTimeout=-1, indexer.url.fieldname=url, es.indexer.addresses=localhost, storm.nimbus.zookeeper.acls.fixup=true, storm.messaging.netty.min_wait_ms=100, fetchInterval.fetch.error=120, worker.log.level.reset.poll.secs=30, storm.zookeeper.port=2000, supervisor.heartbeat.frequency.secs=5, spout.reset.fetchdate.after=120, topology.enable.message.timeouts=true, supervisor.cpu.capacity=400.0, fetcher.max.urls.in.queues=-1, fetchInterval.default=1440, drpc.worker.threads=64, supervisor.blobstore.download.thread.count=5, task.backpressure.poll.secs=30, drpc.queue.size=128, es.status.recentDate.min.gap=-1, http.agent.version=0.1, urlbuffer.class=com.digitalpebble.stormcrawler.persistence.SimpleURLBuffer, es.status.max.start.offset=500, topology.backpressure.enable=false, supervisor.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, storm.blobstore.inputstream.buffer.size.bytes=65536, topology.shellbolt.max.pending=100, http.agent.description=, drpc.https.keystore.password=*****, nimbus.code.sync.freq.secs=120, logviewer.port=8000, topology.scheduler.strategy=org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy, topology.executor.send.buffer.size=1024, resource.aware.scheduler.priority.strategy=org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy, pacemaker.auth.method=NONE, storm.daemon.metrics.reporter.plugins=["org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter"], es.status.addresses=http://localhost:9200, topology.worker.logwriter.childopts=-Xmx64m, topology.spout.wait.strategy=org.apache.storm.spout.SleepSpoutWaitStrategy, ui.host=0.0.0.0, topology.submitter.principal=, storm.nimbus.retry.interval.millis=2000, nimbus.inbox.jar.expiration.secs=3600, dev.zookeeper.path=/tmp/dev-storm-zookeeper, topology.acker.executors=null, topology.fall.back.on.java.serialization=true, topology.eventlogger.executors=0, http.content.partial.as.trimmed=false, supervisor.localizer.cleanup.interval.ms=600000, storm.zookeeper.servers=["localhost"], nimbus.thrift.threads=64, logviewer.cleanup.age.mins=10080, topology.worker.childopts=-Xmx2g -Djava.net.preferIPv4Stack=true, topology.classpath=null, supervisor.monitor.frequency.secs=3, file.protocol.implementation=com.digitalpebble.stormcrawler.protocol.file.FileProtocol, nimbus.credential.renewers.freq.secs=600, backpressure.znode.timeout.secs=30, http.timeout=10000, topology.skip.missing.kryo.registrations=true, drpc.authorizer.acl.filename=drpc-auth-acl.yaml, es.status.bucket.sort.field=["nextFetchDate" "url"], pacemaker.kerberos.users=[], fetcher.timeout.queue=-1, http.agent.name=Google-Read-Aloud, storm.group.mapping.service.cache.duration.secs=120, blobstore.dir=/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e, topology.testing.always.try.serialize=false, nimbus.monitor.freq.secs=10, storm.health.check.timeout.ms=5000, supervisor.supervisors=[], topology.tasks=null, topology.bolts.outgoing.overflow.buffer.enable=false, storm.messaging.netty.socket.backlog=500, topology.workers=1, pacemaker.base.threads=10, storm.local.dir=/tmp/55b9f681-e0ff-4929-81ff-d3354e5f786e, backpressure.znode.update.freq.secs=15, worker.childopts=-Xmx%HEAP-MEM%m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump, storm.auth.simple-white-list.users=[], topology.disruptor.batch.timeout.millis=1, jsoup.treat.non.html.as.error=true, topology.message.timeout.secs=300, topology.state.synchronization.timeout.secs=60, topology.tuple.serializer=org.apache.storm.serialization.types.ListDelegateSerializer, supervisor.supervisors.commands=[], nimbus.blobstore.expiration.secs=600, logviewer.childopts=-Xmx128m, storm.nimbus.zookeeper.acls.check=true, topology.environment=null, topology.debug=false, sitemap.discovery=false, spout.min.delay.queries=2000, topology.disruptor.batch.size=100, storm.disable.symlinks=false, storm.messaging.netty.max_retries=300, ui.childopts=-Xmx768m, storm.network.topography.plugin=org.apache.storm.networktopography.DefaultRackDNSToSwitchMapping, storm.zookeeper.session.timeout=20000, drpc.childopts=-Xmx768m, drpc.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.connection.timeout=15000, storm.zookeeper.auth.user=null, sitemap.filter.hours.since.modified=-1, textextractor.include.pattern=["DIV[id=\"maincontent\"]" "DIV[itemprop=\"articleBody\"]" "ARTICLE"], storm.meta.serialization.delegate=org.apache.storm.serialization.GzipThriftSerializationDelegate, topology.max.spout.pending=100, storm.codedistributor.class=org.apache.storm.codedistributor.LocalFileSystemCodeDistributor, fetcher.threads.per.queue=1, nimbus.supervisor.timeout.secs=60, protocols=http,https,file, fetcher.server.delay.force=false, nimbus.task.timeout.secs=30, storm.zookeeper.superACL=null, drpc.port=3772, pacemaker.max.threads=50, storm.zookeeper.retry.intervalceiling.millis=30000, es.indexer.settings={"cluster.name" "elasticsearch"}, nimbus.thrift.port=6627, storm.auth.simple-acl.admins=[], topology.component.cpu.pcore.percent=10.0, supervisor.memory.capacity.mb=3072.0, indexer.text.fieldname=content, http.accept.language=en-us,en-gb,en;q=0.7,*;q=0.3, storm.nimbus.retry.times=5, supervisor.worker.start.timeout.secs=120, es.status.sample=false, http.robots.403.allow=true, http.store.headers=false, storm.zookeeper.retry.interval=1000, fetcher.server.delay=1.0, http.agent.url=https://support.google.com/webmasters/answer/1061943, fetcher.metrics.time.bucket.secs=10, es.status.index.name=status, logs.users=null, storm.cluster.metrics.consumer.publish.interval.secs=60, worker.profiler.command=flight.bash, transactional.zookeeper.port=null, drpc.max_buffer_size=1048576, pacemaker.thread.timeout=10, spout.ttl.purgatory=30, task.credentials.poll.secs=30, blobstore.superuser=ubuntu, drpc.https.keystore.type=JKS, topology.worker.receiver.thread.count=1, topology.state.checkpoint.interval.ms=1000, supervisor.slots.ports=[6700 6701 6702 6703], detect.mimetype=true, fetcher.max.crawl.delay.force=false, topology.transfer.buffer.size=1024, sitemap.schedule.delay=-1, storm.health.check.dir=healthchecks, topology.worker.shared.thread.pool.size=4, drpc.authorizer.acl.strict=false, nimbus.file.copy.expiration.secs=600, metadata.track.depth=true, es.status.settings={"cluster.name" "elasticsearch"}, es.status.global.sort.field=nextFetchDate, es.indexer.create=false, worker.profiler.childopts=-XX:+UnlockCommercialFeatures -XX:+FlightRecorder, topology.executor.receive.buffer.size=1024, backpressure.disruptor.low.watermark=0.4, topology.users=[], nimbus.task.launch.secs=120, storm.local.mode.zmq=false, track.anchors=true, storm.messaging.netty.buffer_size=5242880, storm.cluster.state.store=org.apache.storm.cluster_state.zookeeper_state_factory, worker.heartbeat.frequency.secs=1, storm.log4j2.conf.dir=log4j2, ui.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.zookeeper.root=/storm, topology.submitter.user=ubuntu, topology.tick.tuple.freq.secs=null, metadata.track.path=true, drpc.https.port=-1, http.content.limit=65536, storm.workers.artifacts.dir=workers-artifacts, http.accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, supervisor.blobstore.download.max_retries=3, task.refresh.poll.secs=10, topology.metrics.consumer.register=[{"class" "com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer", "parallelism.hint" 1}], storm.exhibitor.port=8080, task.heartbeat.frequency.secs=3, fetcher.max.crawl.delay=30, pacemaker.port=6699, storm.messaging.netty.max_wait_ms=1000, es.status.bulkActions=500, topology.component.resources.offheap.memory.mb=0.0, drpc.http.port=3774, es.indexer.flushInterval=2s, topology.error.throttle.interval.secs=10, storm.messaging.transport=org.apache.storm.messaging.netty.Context, es.status.flushInterval=5s, topology.disable.loadaware.messaging=false, selenium.delegated.protocol=com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol, storm.messaging.netty.authentication=false, topology.component.resources.onheap.memory.mb=128.0, es.indexer.bulkActions=100, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, topology.kryo.register={"com.digitalpebble.stormcrawler.Metadata" nil}, urlfilters.config.file=urlfilters.json, worker.gc.childopts=, nimbus.topology.validator=org.apache.storm.nimbus.DefaultTopologyValidator, es.status.query.timeout=-1, nimbus.seeds=["localhost"], metadata.persist=["_redirTo" "error.cause" "error.source" "isSitemap" "isFeed"], nimbus.queue.size=100000, nimbus.cleanup.inbox.freq.secs=600, storm.blobstore.replication.factor=3, worker.heap.memory.mb=768, logviewer.max.sum.worker.logs.size.mb=4096, pacemaker.childopts=-Xmx1024m, ui.users=null, parser.emitOutlinks=true, transactional.zookeeper.servers=null, supervisor.worker.timeout.secs=30, storm.zookeeper.auth.password=null, textextractor.exclude.tags=["STYLE" "SCRIPT"], storm.blobstore.acl.validation.enabled=false, client.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, indexer.text.maxlength=-1, storm.thrift.socket.timeout.ms=600000, supervisor.childopts=-Xmx256m, fetcher.threads.number=50, topology.worker.max.heap.size.mb=768.0, ui.http.x-frame-options=DENY, backpressure.disruptor.high.watermark=0.9, ui.filter=null, es.metrics.settings={"cluster.name" "elasticsearch"}, selenium.instances.num=1, ui.header.buffer.bytes=4096, topology.min.replication.count=1, selenium.capabilities={"loadImages" false, "takesScreenshot" false, "javascriptEnabled" true}, scheduler.class=com.digitalpebble.stormcrawler.persistence.DefaultScheduler, robots.cache.spec=maximumSize=10000,expireAfterWrite=6h, es.metrics.addresses=http://localhost:9200, topology.disruptor.wait.timeout.millis=1000, es.indexer.concurrentRequests=1, robots.noFollow.strict=true, indexer.md.mapping=["parse.title=title" "parse.keywords=keywords" "parse.description=description" "domain=domain"], storm.nimbus.retry.intervalceiling.millis=60000, topology.trident.batch.emit.interval.millis=50, es.status.recentDate.increase=-1, https.protocol.implementation=com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol, storm.auth.simple-acl.users=[], drpc.invocations.threads=64, es.status.max.urls.per.bucket=2, java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib, es.indexer.index.name=content, ui.port=8080, fetcher.max.queue.size=-1, selenium.setScriptTimeout=0, indexer.canonical.name=canonical, es.status.bucket.field=key, es.status.routing=true, topology.kryo.decorators=[], es.status.max.buckets=50, storm.id=crawler-1-1574269039, topology.name=crawler, fetcher.max.throttle.sleep=-1, storm.exhibitor.poll.uripath=/exhibitor/v1/cluster/list, fetcher.server.min.delay=0.0, storm.messaging.netty.transfer.batch.size=262144, parsefilters.config.file=parsefilters.json, logviewer.appender.name=A1, nimbus.thrift.max_buffer_size=1048576, storm.auth.simple-acl.users.commands=[], http.protocol.implementation=com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol, drpc.request.timeout.secs=600, fetchInterval.error=-1}
6648 [SLOT_1024] INFO o.a.s.d.worker - Worker f5d45862-968e-43f7-97e8-669e3dae7d11 for storm crawler-1-1574269039 on ab823f98-11b3-4021-b8e7-f9da7ebc12f3:1024 has finished loading
6648 [SLOT_1024] INFO o.a.s.d.s.Container - SET worker-user f5d45862-968e-43f7-97e8-669e3dae7d11 ubuntu
6649 [SLOT_1024] INFO o.a.s.d.s.Slot - STATE WAITING_FOR_BLOB_LOCALIZATION msInState: 845 -> WAITING_FOR_WORKER_START msInState: 0 topo:crawler-1-1574269039 worker:f5d45862-968e-43f7-97e8-669e3dae7d11
6649 [SLOT_1024] INFO o.a.s.d.s.Slot - SLOT 1024: Changing current assignment from null to LocalAssignment(topology_id:crawler-1-1574269039, executors:[ExecutorInfo(task_start:8, task_end:8), ExecutorInfo(task_start:12, task_end:12), ExecutorInfo(task_start:2, task_end:2), ExecutorInfo(task_start:18, task_end:18), ExecutorInfo(task_start:6, task_end:6), ExecutorInfo(task_start:16, task_end:16), ExecutorInfo(task_start:10, task_end:10), ExecutorInfo(task_start:14, task_end:14), ExecutorInfo(task_start:4, task_end:4), ExecutorInfo(task_start:9, task_end:9), ExecutorInfo(task_start:7, task_end:7), ExecutorInfo(task_start:3, task_end:3), ExecutorInfo(task_start:1, task_end:1), ExecutorInfo(task_start:11, task_end:11), ExecutorInfo(task_start:13, task_end:13), ExecutorInfo(task_start:19, task_end:19), ExecutorInfo(task_start:5, task_end:5), ExecutorInfo(task_start:15, task_end:15), ExecutorInfo(task_start:17, task_end:17)], resources:WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0), owner:ubuntu)
6653 [SLOT_1024] INFO o.a.s.d.s.Slot - STATE WAITING_FOR_WORKER_START msInState: 4 topo:crawler-1-1574269039 worker:f5d45862-968e-43f7-97e8-669e3dae7d11 -> RUNNING msInState: 0 topo:crawler-1-1574269039 worker:f5d45862-968e-43f7-97e8-669e3dae7d11
7040 [refresh-active-timer] INFO o.a.s.d.worker - All connections are ready for worker ab823f98-11b3-4021-b8e7-f9da7ebc12f3:1024 with id f5d45862-968e-43f7-97e8-669e3dae7d11
7055 [Thread-33-spout-executor[9 9]] INFO o.a.s.d.executor - Opening spout spout:(9)
7056 [Thread-47-spout-executor[10 10]] INFO o.a.s.d.executor - Opening spout spout:(10)
7058 [Thread-19-spout-executor[12 12]] INFO o.a.s.d.executor - Opening spout spout:(12)
7069 [Thread-49-spout-executor[14 14]] INFO o.a.s.d.executor - Opening spout spout:(14)
7071 [Thread-35-spout-executor[11 11]] INFO o.a.s.d.executor - Opening spout spout:(11)
7079 [Thread-21-__metricscom.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[2 2]] INFO o.a.s.d.executor - Preparing bolt __metricscom.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer:(2)
7088 [Thread-23-sitemap-executor[7 7]] INFO o.a.s.d.executor - Preparing bolt sitemap:(7)
7140 [Thread-25-fetcher-executor[3 3]] INFO o.a.s.d.executor - Preparing bolt fetcher:(3)
7141 [Thread-27-__acker-executor[1 1]] INFO o.a.s.d.executor - Preparing bolt __acker:(1)
7143 [Thread-39-spout-executor[16 16]] INFO o.a.s.d.executor - Opening spout spout:(16)
7143 [Thread-31-partitioner-executor[6 6]] INFO o.a.s.d.executor - Preparing bolt partitioner:(6)
7142 [Thread-43-status_metrics-executor[19 19]] INFO o.a.s.d.executor - Preparing bolt status_metrics:(19)
7142 [Thread-55-spout-executor[17 17]] INFO o.a.s.d.executor - Opening spout spout:(17)
7142 [Thread-17-spout-executor[8 8]] INFO o.a.s.d.executor - Opening spout spout:(8)
7142 [Thread-45-parse-executor[5 5]] INFO o.a.s.d.executor - Preparing bolt parse:(5)
7142 [Thread-37-__system-executor[-1 -1]] INFO o.a.s.d.executor - Preparing bolt __system:(-1)
7142 [Thread-53-spout-executor[15 15]] INFO o.a.s.d.executor - Opening spout spout:(15)
7142 [Thread-41-spout-executor[13 13]] INFO o.a.s.d.executor - Opening spout spout:(13)
7142 [Thread-29-status-executor[18 18]] INFO o.a.s.d.executor - Preparing bolt status:(18)
7141 [Thread-51-index-executor[4 4]] INFO o.a.s.d.executor - Preparing bolt index:(4)
7150 [Thread-37-__system-executor[-1 -1]] INFO o.a.s.d.executor - Prepared bolt __system:(-1)
7151 [Thread-27-__acker-executor[1 1]] INFO o.a.s.d.executor - Prepared bolt __acker:(1)
7159 [Thread-31-partitioner-executor[6 6]] INFO c.d.s.b.URLPartitionerBolt - Using partition mode : byHost
7168 [Thread-31-partitioner-executor[6 6]] INFO o.a.s.d.executor - Prepared bolt partitioner:(6)
7258 [Thread-29-status-executor[18 18]] INFO c.d.s.u.URLPartitioner - Using partition mode : byHost
7457 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup BasicURLFilter[com.digitalpebble.stormcrawler.filtering.basic.BasicURLFilter]
7457 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup BasicURLFilter[com.digitalpebble.stormcrawler.filtering.basic.BasicURLFilter]
7458 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup MaxDepthFilter[com.digitalpebble.stormcrawler.filtering.depth.MaxDepthFilter]
7458 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup MaxDepthFilter[com.digitalpebble.stormcrawler.filtering.depth.MaxDepthFilter]
7459 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup BasicURLFilter[com.digitalpebble.stormcrawler.filtering.basic.BasicURLFilter]
7460 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup MaxDepthFilter[com.digitalpebble.stormcrawler.filtering.depth.MaxDepthFilter]
7460 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup BasicURLNormalizer[com.digitalpebble.stormcrawler.filtering.basic.BasicURLNormalizer]
7460 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup BasicURLNormalizer[com.digitalpebble.stormcrawler.filtering.basic.BasicURLNormalizer]
7460 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup HostURLFilter[com.digitalpebble.stormcrawler.filtering.host.HostURLFilter]
7461 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup HostURLFilter[com.digitalpebble.stormcrawler.filtering.host.HostURLFilter]
7462 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup BasicURLNormalizer[com.digitalpebble.stormcrawler.filtering.basic.BasicURLNormalizer]
7462 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup HostURLFilter[com.digitalpebble.stormcrawler.filtering.host.HostURLFilter]
7466 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup RegexURLNormalizer[com.digitalpebble.stormcrawler.filtering.regex.RegexURLNormalizer]
7467 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup RegexURLNormalizer[com.digitalpebble.stormcrawler.filtering.regex.RegexURLNormalizer]
7467 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup RegexURLNormalizer[com.digitalpebble.stormcrawler.filtering.regex.RegexURLNormalizer]
7470 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup RegexURLFilter[com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter]
7471 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup SelfURLFilter[com.digitalpebble.stormcrawler.filtering.basic.SelfURLFilter]
7473 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup RegexURLFilter[com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter]
7474 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup SelfURLFilter[com.digitalpebble.stormcrawler.filtering.basic.SelfURLFilter]
7474 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.ParseFilters - Setup SitemapFilter[com.digitalpebble.stormcrawler.filtering.sitemap.SitemapFilter]
7473 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup RegexURLFilter[com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter]
7475 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup SelfURLFilter[com.digitalpebble.stormcrawler.filtering.basic.SelfURLFilter]
7476 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup SitemapFilter[com.digitalpebble.stormcrawler.filtering.sitemap.SitemapFilter]
7473 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup SitemapFilter[com.digitalpebble.stormcrawler.filtering.sitemap.SitemapFilter]
7482 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.b.FetcherBolt - [Fetcher #-1] : starting at 2019-11-20 16:57:22
7520 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup XPathFilter[com.digitalpebble.stormcrawler.parse.filter.XPathFilter]
7532 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup BasicURLFilter[com.digitalpebble.stormcrawler.filtering.basic.BasicURLFilter]
7532 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup MaxDepthFilter[com.digitalpebble.stormcrawler.filtering.depth.MaxDepthFilter]
7532 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup BasicURLNormalizer[com.digitalpebble.stormcrawler.filtering.basic.BasicURLNormalizer]
7532 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup HostURLFilter[com.digitalpebble.stormcrawler.filtering.host.HostURLFilter]
7539 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup RegexURLNormalizer[com.digitalpebble.stormcrawler.filtering.regex.RegexURLNormalizer]
7539 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup XPathFilter[com.digitalpebble.stormcrawler.parse.filter.XPathFilter]
7540 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup RegexURLFilter[com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter]
7540 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup SelfURLFilter[com.digitalpebble.stormcrawler.filtering.basic.SelfURLFilter]
7540 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup SitemapFilter[com.digitalpebble.stormcrawler.filtering.sitemap.SitemapFilter]
7540 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup LinkParseFilter[com.digitalpebble.stormcrawler.parse.filter.LinkParseFilter]
7540 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup BasicURLFilter[com.digitalpebble.stormcrawler.filtering.basic.BasicURLFilter]
7541 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup MaxDepthFilter[com.digitalpebble.stormcrawler.filtering.depth.MaxDepthFilter]
7541 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup BasicURLNormalizer[com.digitalpebble.stormcrawler.filtering.basic.BasicURLNormalizer]
7541 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup HostURLFilter[com.digitalpebble.stormcrawler.filtering.host.HostURLFilter]
7542 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup RegexURLNormalizer[com.digitalpebble.stormcrawler.filtering.regex.RegexURLNormalizer]
7543 [Thread-45-parse-executor[5 5]] INFO c.d.s.u.URLPartitioner - Using partition mode : byDomain
7543 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup DomainParseFilter[com.digitalpebble.stormcrawler.parse.filter.DomainParseFilter]
7544 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup RegexURLFilter[com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter]
7544 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup SelfURLFilter[com.digitalpebble.stormcrawler.filtering.basic.SelfURLFilter]
7544 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup SitemapFilter[com.digitalpebble.stormcrawler.filtering.sitemap.SitemapFilter]
7544 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup LinkParseFilter[com.digitalpebble.stormcrawler.parse.filter.LinkParseFilter]
7544 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.u.URLPartitioner - Using partition mode : byDomain
7544 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup DomainParseFilter[com.digitalpebble.stormcrawler.parse.filter.DomainParseFilter]
7545 [Thread-23-sitemap-executor[7 7]] INFO c.d.s.p.ParseFilters - Setup CommaSeparatedToMultivaluedMetadata[com.digitalpebble.stormcrawler.parse.filter.CommaSeparatedToMultivaluedMetadata]
7545 [Thread-45-parse-executor[5 5]] INFO c.d.s.p.ParseFilters - Setup CommaSeparatedToMultivaluedMetadata[com.digitalpebble.stormcrawler.parse.filter.CommaSeparatedToMultivaluedMetadata]
7550 [Thread-45-parse-executor[5 5]] INFO o.a.s.d.executor - Prepared bolt parse:(5)
7550 [Thread-23-sitemap-executor[7 7]] INFO o.a.s.d.executor - Prepared bolt sitemap:(7)
7718 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.RobotRulesParser - No agents listed in 'http.robots.agents' property! Using http.agent.name [Google-Read-Aloud]
7722 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.p.RobotRulesParser - No agents listed in 'http.robots.agents' property! Using http.agent.name [Google-Read-Aloud]
7722 [Thread-25-fetcher-executor[3 3]] INFO c.d.s.b.FetcherBolt - Using queue mode : byHost
7792 [Thread-25-fetcher-executor[3 3]] INFO o.a.s.d.executor - Prepared bolt fetcher:(3)
7820 [Thread-39-spout-executor[16 16]] INFO c.d.s.e.p.AbstractSpout - [spout #8] assigned shard ID 8
7823 [Thread-55-spout-executor[17 17]] INFO c.d.s.e.p.AbstractSpout - [spout #9] assigned shard ID 9
7824 [Thread-17-spout-executor[8 8]] INFO c.d.s.e.p.AbstractSpout - [spout #0] assigned shard ID 0
7824 [Thread-41-spout-executor[13 13]] INFO c.d.s.e.p.AbstractSpout - [spout #5] assigned shard ID 5
7825 [Thread-53-spout-executor[15 15]] INFO c.d.s.e.p.AbstractSpout - [spout #7] assigned shard ID 7
7825 [Thread-35-spout-executor[11 11]] INFO c.d.s.e.p.AbstractSpout - [spout #3] assigned shard ID 3
7825 [Thread-35-spout-executor[11 11]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7821 [Thread-39-spout-executor[16 16]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7828 [Thread-35-spout-executor[11 11]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7828 [Thread-39-spout-executor[16 16]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7825 [Thread-47-spout-executor[10 10]] INFO c.d.s.e.p.AbstractSpout - [spout #2] assigned shard ID 2
7829 [Thread-47-spout-executor[10 10]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7829 [Thread-47-spout-executor[10 10]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7825 [Thread-19-spout-executor[12 12]] INFO c.d.s.e.p.AbstractSpout - [spout #4] assigned shard ID 4
7830 [Thread-19-spout-executor[12 12]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7825 [Thread-53-spout-executor[15 15]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7830 [Thread-19-spout-executor[12 12]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7830 [Thread-53-spout-executor[15 15]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7824 [Thread-41-spout-executor[13 13]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7831 [Thread-41-spout-executor[13 13]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7824 [Thread-17-spout-executor[8 8]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7840 [Thread-17-spout-executor[8 8]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7823 [Thread-55-spout-executor[17 17]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7825 [Thread-49-spout-executor[14 14]] INFO c.d.s.e.p.AbstractSpout - [spout #6] assigned shard ID 6
7842 [Thread-49-spout-executor[14 14]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7841 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0xdc zxid:0x3c txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout
7842 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0xdf zxid:0x3d txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout
7842 [Thread-49-spout-executor[14 14]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7843 [Thread-55-spout-executor[17 17]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7845 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0xe6 zxid:0x3e txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout
7839 [Thread-33-spout-executor[9 9]] INFO c.d.s.e.p.AbstractSpout - [spout #1] assigned shard ID 1
7846 [Thread-33-spout-executor[9 9]] ERROR o.a.s.util - Async loop died!
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7846 [Thread-33-spout-executor[9 9]] ERROR o.a.s.d.executor -
java.lang.ClassCastException: class clojure.lang.PersistentVector cannot be cast to class java.lang.String (clojure.lang.PersistentVector is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at com.digitalpebble.stormcrawler.util.ConfUtils.getString(ConfUtils.java:74) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AbstractSpout.open(AbstractSpout.java:188) ~[dev1-0.1.jar:?]
at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.open(AggregationSpout.java:98) ~[dev1-0.1.jar:?]
at org.apache.storm.daemon.executor$fn__10112$fn__10127.invoke(executor.clj:609) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7847 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0xf0 zxid:0x42 txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout
7863 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0x105 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout-last-error Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout-last-error
7863 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0x106 zxid:0x4c txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout-last-error Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout-last-error
7875 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0x111 zxid:0x50 txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout-last-error Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout-last-error
7875 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0x112 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout-last-error Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout-last-error
7875 [ProcessThread(sid:0 cport:2000):] INFO o.a.s.s.o.a.z.s.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x100040091f2000f type:create cxid:0x114 zxid:0x53 txntype:-1 reqpath:n/a Error Path:/storm/errors/crawler-1-1574269039/spout-last-error Error:KeeperErrorCode = NodeExists for /storm/errors/crawler-1-1574269039/spout-last-error
7886 [Thread-19-spout-executor[12 12]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7886 [Thread-17-spout-executor[8 8]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7886 [Thread-49-spout-executor[14 14]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7892 [Thread-39-spout-executor[16 16]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7892 [Thread-35-spout-executor[11 11]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7893 [Thread-47-spout-executor[10 10]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7895 [Thread-55-spout-executor[17 17]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7895 [Thread-41-spout-executor[13 13]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
7895 [Thread-53-spout-executor[15 15]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:788) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.daemon.executor$mk_executor_data$fn__9997$fn__9998.invoke(executor.clj:281) [storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
DigitalPebble
--
You received this message because you are subscribed to the Google Groups "DigitalPebble" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalpebbl...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/digitalpebble/5e39215b-c3b5-4d96-90b0-a6c6391ab0fa%40googlegroups.com.
