in_exec problems

[Paula Keezer]

Hello,  I am trying to get a very simple 'hello world' in_exec call working.  I don't seem to be having much luck.  Here is the source configuration I am using:

  <source>

    type exec

    command python /home/paula/v-dev/myvirtenv/sanbbox1/hello.py

    keys k1,k2,k3

    tag_key k1

    time_key k2

    time_format %Y-%m-%d %H:%M:%S

    run_interval 10s

  </source>

My program sends a tsv to stdout each time it is called.  the tsv looks    like:

debug.blah 2013-07-12 17:31:42 hello

where the fields are seperated by tabs (just cant seem them in google groups)

the td_agent starts just fine and the tsd_agent.log show that all the plug ins are installed and running.  But, nothing calls my hello.py program (I have logging wiithin the program) and no IO gets picked up by in_exec.  

It almost seems like there is no python to run in the environment that td_agent is in.  Since there is no error, I cannot debug (unless I begin instrumenting in_exec)

Any ideas?  examples that anyone has successfully done?

Thanks,  Paula Keezer

[kiyoto]

Hi Paula,

That's strange. What user do you run td-agent as? Is python/your hello.py script visible from that user?

Here is an example configuration that worked for me locally with a similar setup: https://github.com/kiyoto/fluentd-examples/tree/master/test_in_exec

A couple of suggestions that may be helpful.

1. Instead of running a Python script in your virtualenv, try running something simpler (e.g. bash script). If something like that runs, then it might have something to do with your python setup.

2. Also, it would be helpful to know the version of your td-agent.

Thanks!

kiyoto

[Paula Keezer]

Hi Kiyoto,  td-agent is run as td-agent.  

The Changelog indicates Release 0.10.35 - 2013/06/12

a simple bash script does not seem to work either.

Is there any way  to get more verbose td-agent logs?  It does not seem to be complaining about my configuration, however, I know (from my own logging code in python) that the python script is not running.

Thank you for your assitance,

Paula

On Friday, July 12, 2013 5:43:39 PM UTC-7, Paula Keezer wrote:

[Paula Keezer]

Now have sh shell working!  Have correct $PATH. I am using k3 to display session info; example

#!/bin/sh

stuff=`echo $PATH`

echo "debug.stuf\t2013-07-15 00:00:00\t$stuff"

will display the $PATH available to the in_exec script.

When I do the following:

#!/bin/sh

stuff=`python -V`

echo "debug.stuf\t2013-07-15 00:00:00\t$stuff"

I get a 'null' result.  

For some reason , python is not running from the in_exec script

On Friday, July 12, 2013 5:43:39 PM UTC-7, Paula Keezer wrote:

[kiyoto]

Hi Paula,

Good to hear that you made some progress.

>Is there any way  to get more verbose td-agent logs?

Yes, although it is a bit clunky =/

Create a file td-agent in the directory /etc/default with one line "DAEMON_ARGS=-vv". You can do this by

$ sudo echo "DAEMON_ARGS=-vv" > /etc/default/td-agent

Then, you can restart your td-agent. You should now have more verbose logging. I hope this helps. We will improve the documentation around this.

>I get a 'null' result.  

>For some reason , python is not running from the in_exec script

Hmm...it might help to test for "which python" in addition to doing "python -V". Is Python installed for all users, including td-agent?

kiyoto

[Paula Keezer]

Thanks for the details on getting more logging!
I've done 'whereis python' and 'which python'  It uses /usr/bin/python and that works fine from the command line.  It is a link to python2.7.  All have permission set to everyone.  I'm sure I'm missing something!

Paula

On Friday, July 12, 2013 5:43:39 PM UTC-7, Paula Keezer wrote:

[Paula Keezer]

Hi Kiyoto,  I tried the DAEMON_ARGS=vv and the td-agent will not start.  Without that file and command the td-agent rstarts and has two processies which I believe is normal.    Still looking for better logging....

Thanks,

Paula


On Friday, July 12, 2013 5:43:39 PM UTC-7, Paula Keezer wrote:

[kiyoto]

Hi Paula,

>I tried the DAEMON_ARGS=vv and the td-agent will not start.

Did you try "vv" or "-vv" (Note the dash preceding the two v's)?

kiyoto

[Paula Keezer]

Hi Kiyoto,  I have verified that the dash is included. 

With the DAEMON_ARGS=-vv   :  the init.d/td-agent start (or restart) does not succeed and does not show any error.  running init.d/td-agent status shows that ruby is not running

Without the DAEMON_ARGS=-vv   :  The init.d/td-agent starts.  Again, does not show any success or error.  running init.d/td-agent status shows that ruby is running. verified that two processes are running.

But I have good news.  I have solved the in_exec problem I was having with my hello.py.  

1) I tried running td-agent from my users with the command:  

$td-agent -c /etc/td-agent/td-agent.conf -vv

This worked!  so I know td-agent, my conf file, verbose debug and my hello.py were working.

2) I tried a simple sh script using init.d/td-agent start where the  script was:

stuff=$(python -V) 

This did not work on my local user or init.d/td-agent start.  REASON:  python -V returns the python version through stderr.  stderr is not captured by in_exec.  not capturing stderr and storing somewhere is of great concern.

I changed this script to 

stuff=$(python -V 2>&1) and the script worked for both my local user and init.d/td-agent start

Here are my concerns:

1) It is very difficult to debug an in_exec script due to the lack of stderr reporting.  

In my case,  my hello.py includes its own logging so I can see what is happening within my script.  This worked fine when running from my local user because my log file had correct permissions.  When I tried to run this from init.d/td-agent which runs in the td-agent environment, that user did not have permissions to write to my log file.  The hello.py program properly sent an error out stderr however, that message was never visible in the td-agent.log or anywhere else.  

Now that I know this I can be more careful.  However,  I feel that anything that comes out of stderr should go somewhere, maybe a td-agent-stderr.log   If I had been able to see this message I would have immediately known the problem and fixed it!

I do not know if the stderr problem is also an issue when developing an in_ or out_ plugin.  As my team engages more with td-agent, we will be very concerned about how to get stderr messages.  I have a work-around with in_exec (run the program from an sh script and re-direct stderr to stdout [2>&1]).

Thank you very much for your patience with me.  Please do consider how to capture all stderr messages and store them in a log from a td-agent running as td-agent and change the init.d/td-agent start so that it returns a success or failure.

Paula

On Friday, July 12, 2013 5:43:39 PM UTC-7, Paula Keezer wrote:

[Masahiro Nakagawa]

> With the DAEMON_ARGS=-vv   :  the init.d/td-agent start (or restart) does not succeed and does not show any error. 

I found this is the bug of /etc/init.d/td-agent script.

The position of reading /etc/default/td-agent file is wrong.

I will fix tomorrow.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[kiyoto]

Hi Paula,

>Thank you very much for your patience with me.  Please do consider how to capture all stderr messages and store them in a log from a td-agent running as td-agent and >change the init.d/td-agent start so that it returns a success or failure.

Thank you very much for uncovering a bug (as it turned out). I will relay your feedback to core maintainers.

Also, depending on your project timeline, you might be interested in the development of the upcoming v11 (NOT released yet, currently slated to be available in early August).

the v11 repo: https://github.com/fluent/fluentd/tree/v11
...and some ascii-art architecture diagrams (sorry they are still inlined in code):

https://github.com/fluent/fluentd/blob/v11/lib/fluentd/root_agent.rb

https://github.com/fluent/fluentd/blob/v11/lib/fluentd/message_source.rb

https://github.com/fluent/fluentd/blob/v11/lib/fluentd/message_router.rb

[Paula Keezer]

Thank you Masahiro and Kiyoto,

I will look at v11.  We are still in the early stages of becoming fluent in fluentD!

I have another question which I will post in a new topic.

Thanks,

Paula

On Friday, July 12, 2013 5:43:39 PM UTC-7, Paula Keezer wrote: