OpenTSDB /api/lookup/search not returning expected results

[Jim]

I have collectd submitting performance data to otsdb version 2.3.  The metric data looks like;

metric name = sys.cpu.idle

tag host=<hostname>

tag dc=SOM

etc.  with a bunch more tags, but those are the relevant ones.  I've confirmed that collectd is correctly tagging the metrics using its debug logging.

I then have Grafana plugged into the otsdb, trying to retrieve the values of the host tag for all metric sys.cpu.idle time series.

I've tried configuring Grafana to use the query tag_values(sys.cpu.idle, host, dc=SOM), but this returns no results.

I then looked at the API call that Grafana was doing to try and replicate the results, or identify the cause of the problem

The API query looks like this;

/api/search/lookup?limit=3000&m=sys.cpu.idle{host=*,dc=SOM}

This results in a JSON response of;

{

• type: "LOOKUP",
• metric: "sys.cpu.idle",
• tags: 

  [

  • {

    • key: "host",
    • value: "*"

    },
  • {

    • key: "dc",
    • value: "SOM"

    }

  ],
• limit: 3000,
• time: 2,
• results: [ ],
• startIndex: 0,
• totalResults: 0

}

Containing an empty list of results.  I know that there should be results to this api call.  But I'm not sure why there isn't.

I've also tried using the use_meta=false following a suggestion by someone in the opentsdb IRC channel;

/api/search/lookup?limit=3000&use_meta=false&m=sys.cpu.idle{host=*,dc=SOM}

This gives me the same empty result.

The suggest API seems to be working however.  If I ask for a tag value that starts with s for example,  I get results.  

/api/suggest?max=1000&q=s&type=tagv

Results response of;

["sda","sda1","sda2","shortterm","slab_recl","slab_unrecl","sleeping"]

To me, this suggests that the API is working as expected, and that I should be getting data to my lookup queries. But at this point I'm stumped.

My configuration as reported by /api/config is below

{

• tsd.core.auto_create_metrics: "false",
• tsd.core.auto_create_tagks: "true",
• tsd.core.auto_create_tagvs: "true",
• tsd.core.connections.limit: "0",
• tsd.core.enable_api: "true",
• tsd.core.enable_ui: "true",
• tsd.core.meta.cache.enable: "false",
• tsd.core.meta.enable_realtime_ts: "true",
• tsd.core.meta.enable_realtime_uid: "false",
• tsd.core.meta.enable_tsuid_incrementing: "false",
• tsd.core.meta.enable_tsuid_tracking: "false",
• tsd.core.plugin_path: "",
• tsd.core.preload_uid_cache: "false",
• tsd.core.preload_uid_cache.max_entries: "300000",
• tsd.core.socket.timeout: "0",
• tsd.core.stats_with_port: "false",
• tsd.core.storage_exception_handler.enable: "false",
• tsd.core.tree.enable_processing: "false",
• tsd.core.uid.random_metrics: "false",
• tsd.http.cachedir: "/tmp/opentsdb",
• tsd.http.query.allow_delete: "false",
• tsd.http.request.cors_domains: "",
• tsd.http.request.cors_headers: "Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since",
• tsd.http.request.enable_chunked: "true",
• tsd.http.request.max_chunk: "20480",
• tsd.http.show_stack_trace: "true",
• tsd.http.staticroot: "/opentsdb/build/staticroot",
• tsd.mode: "rw",
• tsd.network.async_io: "true",
• tsd.network.bind: "0.0.0.0",
• tsd.network.keep_alive: "true",
• tsd.network.port: "4242",
• tsd.network.reuse_address: "true",
• tsd.network.tcp_no_delay: "true",
• tsd.network.worker_threads: "",
• tsd.no_diediedie: "false",
• tsd.query.allow_simultaneous_duplicates: "true",
• tsd.query.enable_fuzzy_filter: "true",
• tsd.query.filter.expansion_limit: "4096",
• tsd.query.skip_unresolved_tagvs: "false",
• tsd.query.timeout: "0",
• tsd.rtpublisher.enable: "false",
• tsd.rtpublisher.plugin: "",
• tsd.search.enable: "false",
• tsd.search.plugin: "",
• tsd.startup.enable: "false",
• tsd.startup.plugin: "",
• tsd.stats.canonical: "false",
• tsd.storage.compaction.flush_interval: "10",
• tsd.storage.compaction.flush_speed: "2",
• tsd.storage.compaction.max_concurrent_flushes: "10000",
• tsd.storage.compaction.min_flush_threshold: "100",
• tsd.storage.enable_appends: "false",
• tsd.storage.enable_compaction: "true",
• tsd.storage.fix_duplicates: "false",
• tsd.storage.flush_interval: "1000",
• tsd.storage.hbase.data_table: "tsdb",
• tsd.storage.hbase.meta_table: "tsdb-meta",
• tsd.storage.hbase.prefetch_meta: "false",
• tsd.storage.hbase.scanner.maxNumRows: "128",
• tsd.storage.hbase.tree_table: "tsdb-tree",
• tsd.storage.hbase.uid_table: "tsdb-uid",
• tsd.storage.hbase.zk_basedir: "/hbase",
• tsd.storage.hbase.zk_quorum: "localhost,x.x.x.x,x.x.x.x",
• tsd.storage.repair_appends: "false",
• tsd.timeseriesfilter.enable: "false",
• tsd.uidfilter.enable: "false"

}

At this point I'm quite stumped,  so any help is greatly appreciated.

[Jonathan Creasy]

Typically for the search API you need a search plugin, like the Elasticsearch one. 

"If the plugin is not configured or enabled, endpoints other than /api/search/lookup will return an exception."

The suggest API is a different endpoint all together, it simply uses the list of UIDS from the tsdb-uid table with

 no knowledge of which tag keys, tag values are attached to which metrics. 

Grafana should be using the suggest API. Can you identify the requests from Grafana in your logs to confirm which API is being used by Grafana?

[Jonathan Creasy]

You probably need realtime_uid turned on, although if it were my environment, I'd probably leave it off and periodically run the metasync command.

http://opentsdb.net/docs/build/html/user_guide/cli/uid.html#metasync

• tsd.core.meta.cache.enable: "false",
• tsd.core.meta.enable_realtime_ts: "true",
• tsd.core.meta.enable_realtime_uid: "false",
• tsd.core.meta.enable_tsuid_incrementing: "false",
• tsd.core.meta.enable_tsuid_tracking: "false",

[Jim]

So after looking at Grafana, I confirmed that Grafana, when using the tag_values(...) method is using the /api/search/lookup url

I've also confirmed that I should have host values being returned, as I can manually create a graph using the same query structure and get returned data using the baked in otsdb query builder ui;

So, do I need to enable the metadata functions to get this to work?  Or is there a way that doesn't result in such a performance hit?  

You've mentioned leaving them off, and manually running the metasync command.  Does this get the same result with less of a performance hit?

[Jim]

This was the eventual solution.  Manually running metasync gives me the expected results when using the api/search/lookup entrypoint.  

I'll research the impact that enabling this in the config would have.  But for the time being, a cronjob seems like it would be sufficient.