running custom scripts under tcollector

[Mike Dylan]

Hi Group:

I have many different data repositories that I have to pull data and push it to opentsdb. For some cases I have to execute R scripts, some python, some perl, some php etc.

In order to execute these scripts to write to opentsdb, is the following is true?

1. install tcollector.py framework

2. create this directory /usr/local/tcollector/collector/0

3.Create custom scripts and place them under /usr/local/tcollector/collector/0 directory.

Is the script write to stdout and tcollector takes it from there?

Can somebody please shed some insight into this?

Regards,

Mike

[James Barnes]

Hi Mike,

I'm fairly new to OpenTSDB and TCollector as well so somebody else may be able to provide better answers. That said…

1. Yes
2. The collector/0 directory should already exist and there are many scripts in there. So, no need to create it.
3. You can place scripts in collector/0 or make new directories such as collector/300 which will run every 300 seconds. The collector/0 directory runs continuously so the scripts in that directory must be designed to self-throttle. In all other numbered directories, your script just sends to stdout and terminates. This makes it very easy to test — you get to run your script and look at the output. Then, when you're happy with the metrics, you move it into a numbered directory for how often you'd like it to run.
4. You'll need to start tcollector somewhere in here via '/usr/local/bin/tcollector/startstop start'

This should have the documentation, it looks down at the moment:

http://www.opentsdb.net/tcollector.html

Here was my setup on an AWS Ubuntu server:

Install TCollector

• sudo apt-get update
• sudo apt-get install git-core
• sudo apt-get install openjdk-7-jdk
• sudo apt-get install python-boto
• // cd to desired directory
• cd /usr/local/bin
• // clone tcollector
• sudo git clone https://github.com/OpenTSDB/tcollector.git
• // move into tcollector directory
• cd tcollector
• // turn off default collectors by making them non-executable
• sudo chmod 644 collectors/0/*

Configure TCollector

• // set tsdb host
• sudo vi startstop
• replace commented out line 6: TSD_HOST=ip-10-0-145-200.us-west-2.compute.internal
• :wq
• // create any desired collectors
• e.g. sudo mkdir collectors/180
• e.g. sudo cp blah collectors/180

Launch TCollector

• // start tcollector
• sudo ./startstop start --dedup-interval=0 (we want all points — you may not need this)
• // watch the log
• tail -f /var/log/tcollector.log
• // look for valid connection
• 2014-09-02 19:54:40,656 tcollector[17346] INFO: Selected connection: ip-10-0-145-200.us-west-2.compute.internal:4242
• // enable desired collectors
• e.g. sudo chmod 755 collectors/120/great-stats.py
• // look for valid spawning of collector
• e.g. 2014-09-02 19:57:24,522 tcollector[17346] INFO: spawned great-stats.py (pid=17543)

James

[Mike Dylan]

James,

Thank you for your answer. This is great. Only other question I have is that my server SLES (suse). How do I install this - https://github.com/OpenTSDB/tcollector on suse? I just copy this script to my linux(tcollector.py), create the corresponding directoryies manually and start this script in the background - tcollector.py?

[James Barnes]

The TCollector code is set up so that you git clone it.

sudo git clone https://github.com/OpenTSDB/tcollector.git

You need to install git first if you don't have it. Does your linux have yum? If so, 'sudo yum install git' and then the above git clone command.

James

From: Mike Dylan <seanca...@gmail.com>
Date: Wednesday, October 8, 2014 11:36 AM
To: "open...@googlegroups.com" <open...@googlegroups.com>

[Kieren Hynd]

If you wanted to package it up, you should be able to build an .rpm that might work on SLES from the Git clone by running "make rpm" in the rpm/ directory (and similary, "dpkg-buildpackage" from the top level directory usually spits out a working .deb).

-kieren