question about logging each 'put' request

[Huafeng Lyu]

Hi guys,

I'm new to opentsdb, and am here for help.

My agent sends datapoints to opentsdb's telnet port every minute, and I found some datapoints are missing... According to my agent's log, all datapoints were sent, so I want to know if opentsdb received them at the telnet port. If not received, it's probably a network outage; if received, it's probably an opentsdb writing issue.

So is there an approach to let opentsdb log each put request it receives, as well as the request type (telnet, or http api, etc)?

Or, any other better alternatives to know if opentsdb receives these put/write requests?

And advice is highly appreciated.

Thanks.
--huafeng

[Thibault Godouet]

How about using tcpdump / tshark to record the traffic OpenTSDB receives and use this to double check?
You should then be able to search strings in the pcap using the CLI (e.g. tshark with a filter like 'tcp contains' looking for one put that is missing -- or I suspect you should be able to make tshark print the content of the tcp steam as text and then grep it) or wireshark.

[Huafeng Lu]

Thanks Thibault. Finally I use "tcpdump -qns 0 -A -r <pcap_file>" to get the text and then wrote a python script to get the telnet session data (a bunch of 'put' commands) from it.

--huafeng

[Jonathan Creasy]

Another approach might be to write a realtime plugin, it could be used to emit metrics about the metrics being received.