Mesh Issues?

[Albert Law]

Hi,

Any ideas on what the following stdout and stderr mean?

WARN [streamSourceWorker-1] 2014-11-14 16:31:31,601
AbstractStreamFileDataSource.java (line 810)
org.jboss.netty.channel.ChannelException: no matching mesh peers
at com.addthis.meshy.Meshy.createSession(Meshy.java:252)
at com.addthis.meshy.SourceHandler.(SourceHandler.java:99)
at com.addthis.meshy.service.stream.StreamSource.(StreamSource.java:62)
at com.addthis.meshy.service.stream.StreamSource.(StreamSource.java:52)
at com.addthis.hydra.task.stream.StreamSourceMeshy$MeshyStreamFile.getInputStream(StreamSourceMeshy.java:561)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource$Wrap.(AbstractStreamFileDataSource.java:492)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource.nextWrappedSource(AbstractStreamFileDataSource.java:614)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource.access$2200(AbstractStreamFileDataSource.java:101)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource$SourceWorker.fill(AbstractStreamFileDataSource.java:800)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource$SourceWorker.run(AbstractStreamFileDataSource.java:842)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

Nov 14, 2014 4:31:31 PM com.google.common.cache.LocalCache$Segment$1 run
WARNING: Exception thrown during refresh
java.util.concurrent.ExecutionException:
org.jboss.netty.channel.ChannelException: no matching mesh peers
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2344)
at com.google.common.cache.LocalCache$Segment$1.run(LocalCache.java:2327)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2322)
at com.google.common.cache.LocalCache$Segment.refresh(LocalCache.java:2385)
at com.google.common.cache.LocalCache$Segment.scheduleRefresh(LocalCache.java:2363)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2183)
at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
at com.addthis.hydra.task.stream.MeshHostScoreCache.get(MeshHostScoreCache.java:92)
at com.addthis.hydra.task.stream.StreamSourceMeshy.selectMeshHost(StreamSourceMeshy.java:408)
at com.addthis.hydra.task.stream.StreamSourceMeshy.nextSource(StreamSourceMeshy.java:355)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource.nextWrappedSource(AbstractStreamFileDataSource.java:610)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource.access$2200(AbstractStreamFileDataSource.java:101)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource$SourceWorker.fill(AbstractStreamFileDataSource.java:800)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource$SourceWorker.run(AbstractStreamFileDataSource.java:842)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.jboss.netty.channel.ChannelException: no matching mesh peers
at com.addthis.meshy.Meshy.createSession(Meshy.java:252)
at com.addthis.meshy.SourceHandler.(SourceHandler.java:99)
at com.addthis.meshy.service.stream.StreamSource.(StreamSource.java:62)
at com.addthis.meshy.service.stream.StreamSource.(StreamSource.java:52)
at com.addthis.hydra.task.stream.MeshHostScoreCache$1.load(MeshHostScoreCache.java:68)
at com.addthis.hydra.task.stream.MeshHostScoreCache$1.load(MeshHostScoreCache.java:66)
at com.google.common.cache.CacheLoader.reload(CacheLoader.java:94)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3525)
at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2321)
... 16 more

Nov 14, 2014 4:31:31 PM com.google.common.cache.LocalCache$Segment$1 run
WARNING: Exception thrown during refresh
java.util.concurrent.ExecutionException:
org.jboss.netty.channel.ChannelException: no matching mesh peers
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2344)
at com.google.common.cache.LocalCache$Segment$1.run(LocalCache.java:2327)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2322)
at com.google.common.cache.LocalCache$Segment.refresh(LocalCache.java:2385)
at com.google.common.cache.LocalCache$Segment.scheduleRefresh(LocalCache.java:2363)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2183)
at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
at com.addthis.hydra.task.stream.MeshHostScoreCache.get(MeshHostScoreCache.java:92)
at com.addthis.hydra.task.stream.StreamSourceMeshy.selectMeshHost(StreamSourceMeshy.java:408)
at com.addthis.hydra.task.stream.StreamSourceMeshy.nextSource(StreamSourceMeshy.java:355)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource.nextWrappedSource(AbstractStreamFileDataSource.java:610)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource.access$2200(AbstractStreamFileDataSource.java:101)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource$SourceWorker.fill(AbstractStreamFileDataSource.java:800)
at com.addthis.hydra.task.source.AbstractStreamFileDataSource$SourceWorker.run(AbstractStreamFileDataSource.java:842)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.jboss.netty.channel.ChannelException: no matching mesh peers
at com.addthis.meshy.Meshy.createSession(Meshy.java:252)
at com.addthis.meshy.SourceHandler.(SourceHandler.java:99)
at com.addthis.meshy.service.stream.StreamSource.(StreamSource.java:62)
at com.addthis.meshy.service.stream.StreamSource.(StreamSource.java:52)
at com.addthis.hydra.task.stream.MeshHostScoreCache$1.load(MeshHostScoreCache.java:68)
at com.addthis.hydra.task.stream.MeshHostScoreCache$1.load(MeshHostScoreCache.java:66)
at com.google.common.cache.CacheLoader.reload(CacheLoader.java:94)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3525)
at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2321)
... 16 more


--
Albert Law
Lead Engineer - Data Acquisition
NewBrand
http://www.newbrandanalytics.com/

[Ian Barfield]

Sorry for the delay on some of these responses. I was on vacation.

These errors look like a mesh node went down in the middle of a job. Did the local mss process die around that time?

--
You received this message because you are subscribed to the Google Groups "hydra-oss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hydra-oss+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Albert Law]

Hi Ian,

We run Hydra on Rackspace boxes. Stewart mentioned that is a terrible
idea due to Rackspace interconnect issues. Is there a way I could
increase Meshy's resilience to a spotty network? Perhaps more
retrying or a longer backoff than 5000ms?

[Ian Barfield]

The stack traces look like the job is failing to connect to the "meshHost" which is almost always localhost. Is that not possible with your setup? I would not expect interconnect issues between two processes on the same box, but maybe rackspace just does that.

[Albert Law]

Hi Ian,

Mmm, I wouldn't think that is normal even on a cloud box. I will have
to look into it.