2.4.0 RC4 upgrade issues

[Sean Clark]

I've upgraded from 2.3 to 2.4RC1 -- encountered 2 issues so far:

(1) apparently my 2.3 setup had errors it never warned about? I got thousands of "Cannot use Date Tiered Compaction with AppendPoints. Please turn off either of them" in log -- setting tsd.storage.enable_appends = false made that go away - although now i am thinking i have to turn on compactions?

(2) One i can't solve is my log filled with this:

ERROR [RpcHandler.logError] - [id: 0x5c208f8e, /192.168.0.35:35486 => /192.168.0.35:4242] Unexpected exception caught while serving [null, sys.numa.foreign_allocs, 1499362984, 0, node=0, host=ncw-web-01]

java.lang.RuntimeException: Should never be here

Caused by: java.lang.IllegalStateException: id=[-12, -29, -96] => name=sys.numa.foreign_allocs, already mapped to proc.meminfo.dirty

at net.opentsdb.uid.UniqueId.addNameToCache(UniqueId.java:328) ~[tsdb-2.4.0RC1.jar:]

Is there some other cache/duplicate setting that is now being triggered? looking at my simple mockup graph, the data i care about appears to be 'there' -- but i was not looking at some of the tcollector metrics in granfana

=Sean

[ManOLamancha]

On Thursday, July 6, 2017 at 10:53:06 AM UTC-7, Sean Clark wrote:

I've upgraded from 2.3 to 2.4RC1 -- encountered 2 issues so far:

(1) apparently my 2.3 setup had errors it never warned about? I got thousands of "Cannot use Date Tiered Compaction with AppendPoints. Please turn off either of them" in log -- setting tsd.storage.enable_appends = false made that go away - although now i am thinking i have to turn on compactions?

When you upgraded, did you enable the date tiered flag? It should be disabled by default and if not I'll need to fix it. Otherwise disable the date tiered and enable the appends as you have to use HBase 2.x with date tiered enabled. I don't think it's ready for prod use. 

(2) One i can't solve is my log filled with this:

ERROR [RpcHandler.logError] - [id: 0x5c208f8e, /192.168.0.35:35486 => /192.168.0.35:4242] Unexpected exception caught while serving [null, sys.numa.foreign_allocs, 1499362984, 0, node=0, host=ncw-web-01]

java.lang.RuntimeException: Should never be here

Caused by: java.lang.IllegalStateException: id=[-12, -29, -96] => name=sys.numa.foreign_allocs, already mapped to proc.meminfo.dirty

at net.opentsdb.uid.UniqueId.addNameToCache(UniqueId.java:328) ~[tsdb-2.4.0RC1.jar:]

It sounds like a UID assignment bug. Try running the UID fcsk tool: http://opentsdb.net/docs/build/html/user_guide/cli/uid.html#fsck

 

Is there some other cache/duplicate setting that is now being triggered? looking at my simple mockup graph, the data i care about appears to be 'there' -- but i was not looking at some of the tcollector metrics in granfana

You'd likely be missing the proc.meminfo.dirty metric as goofed the assignment somehow. 

[Jesus Orosco]

Hello,

I also upgraded my 2.3 TSD to 2.4 RC1 and get the same message as stated above.  I attempted to run the fsck uid tool to no avail.

chuy@tsd1:/etc/opentsdb23$ sudo /usr/share/opentsdb/bin/tsdb uid fsck fix

Exception in thread "main" java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;

at org.hbase.async.HBaseClient.flush(HBaseClient.java:811)

at org.hbase.async.HBaseClient.shutdown(HBaseClient.java:1061)

at net.opentsdb.tools.UidManager.main(UidManager.java:132)

chuy

[ManOLamancha]

On Monday, July 24, 2017 at 10:53:33 AM UTC-7, Jesus Orosco wrote:

Hello,

I also upgraded my 2.3 TSD to 2.4 RC1 and get the same message as stated above.  I attempted to run the fsck uid tool to no avail.

chuy@tsd1:/etc/opentsdb23$ sudo /usr/share/opentsdb/bin/tsdb uid fsck fix

Exception in thread "main" java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;

at org.hbase.async.HBaseClient.flush(HBaseClient.java:811)

at org.hbase.async.HBaseClient.shutdown(HBaseClient.java:1061)

at net.opentsdb.tools.UidManager.main(UidManager.java:132)

This turned out to be a bad tweak (on my part) to the 2.4 RC code trying to use the UTF8 encoding for UID's that didn't work out so well. I rolled it back in the "next" branch so try that out and you shouldn't see these issues.

[Mike Chepaykin]

HI. I have the same error after upgrade - "Cannot use Date Tiered Compaction with AppendPoints". Can anyone make it clear, how to disable that "date tiered flag" in OpenTSDB 2.4RC2?

[ManOLamancha]

On Friday, January 19, 2018 at 4:41:21 AM UTC-8, Mike Chepaykin wrote:

HI. I have the same error after upgrade - "Cannot use Date Tiered Compaction with AppendPoints". Can anyone make it clear, how to disable that "date tiered flag" in OpenTSDB 2.4RC2?

Make sure "tsd.storage.use_otsdb_timestamp" is false in your config. It looks to default to true right now in RC2.