complex embedded ruby code

[Nicolas Pepinster]

Hello,

I would like to define a host to forward fluentd messages based on an environment variable.

I'm trying to use ruby embedded code.

Here is my config (before my test) with static values :

  <server>
    host myhostA.foo
    port 24224
    weight 60
  </server>
  #Secondary server
  <server>
    host myhostB.foo
    port 24224
    weight 0
  </server>

Here is my test with Ruby embedded code :

  <server>
    host "#{case ENV['NAMESPACE'].slice(0..(ENV['NAMESPACE'].index('-'))) when foo then :myhostA.foo when bar then :myhostA.bar end}"
    port 24224
    weight 60
  </server>
  #Secondary server
  <server>
    host "#{case ENV['NAMESPACE'].slice(0..(ENV['NAMESPACE'].index('-'))) when foo then :myhostB.foo when bar then :myhostB.bar end}"
    port 24224
    weight 0
  </server>

my NAMESPACE env var could take both "foo-ns" and "bar-myns" values.

I got this message when I'm starting fluentd :

/usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/literal_parser.rb:165:in `instance_eval': undefined local variable or method `foo' for #<Binding:0x007f546c821550> (NameError)

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/literal_parser.rb:165:in `instance_eval'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/literal_parser.rb:165:in `eval_embedded_code'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/literal_parser.rb:92:in `scan_double_quoted_string'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/literal_parser.rb:71:in `scan_string'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/literal_parser.rb:64:in `parse_literal'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/v1_parser.rb:130:in `parse_element'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/v1_parser.rb:95:in `parse_element'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/v1_parser.rb:167:in `block in eval_include'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/v1_parser.rb:161:in `each'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/v1_parser.rb:161:in `eval_include'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/v1_parser.rb:145:in `parse_include'

        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.27/lib/fluent/config/v1_parser.rb:113:in `parse_element'

maybe a synthax problem ?

Thanks in advance
 

[Mr. Fiber]

Hi,

I don't write a detailed ruby syntax here, but your ruby code is wrong.

"#{case ENV['NAMESPACE'].split('-').first; when 'foo' then 'myhostA.foo'; when 'bar' then 'myhostA.bar' end}"

Here is one working example.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicolas Pepinster]

Thank you, my code was indeed wrong. It works better now.

Do you have a documentation page for the synthax of the ruby embedded code understanding by fluentd ?

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> Do you have a documentation page for the synthax of the ruby embedded code understanding by fluentd ?

No limitation. All valid ruby code should work.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Nicolas Pepinster]

Ok thank you, case closed :-)

[Nicolas Pepinster]

I'm sorry to reopen this thread but I introduced a little variance by reading a file and it doesn't work.

Here is my code :

host "#{tenant=`cat /tmp/tenant`; case tenant; when 'tenant1' then 'machine1'; when 'tenant2' then 'machine2'; when 'tenant3' then 'machine3'; when 'tenant4' then 'machine4' end}"

When I start my fluentd process with this config, the 'host' variable is empty.

I also tried to read the file differentlty :

tenant=%x(cat "/tmp/tenant"); ->host variable is empty

tenant=IO.binread("/tmp/tenant/); -> get a Fluent::ConfigParseError

It's probably a wrong synthax again... do you have an idea ?

I tested my ruby code with ruby -e command in the fluentd docker container but the behavior is not always the same that when the config is reading by fluentd process.

So, subsidiary question : what's the best way to test ruby code before putting it in the fluentd config ?

Thanks

[Mr. Fiber]

> tenant=`cat /tmp/tenant`;

Does this tenant contain `\n`?
I tested following configuation and it worked.

<source>
  @type forward
  foo "#{tenant=`cat ~/tenant`.chomp; case tenant; when 'tenant1' then 'machine1'; when 'tenant2' then 'machine2'; when 'tenant3' then 'machine3'; when 'tenant4' then 'machine4' end}"
</source>

% RUBYLIB=~/dev/fluentd/fluentd/lib fluentd -c embedded.conf
2016-10-13 09:12:40 +0900 [info]: reading config file path="embedded.conf"
2016-10-13 09:12:40 +0900 [info]: starting fluentd-0.14.7
2016-10-13 09:12:40 +0900 [info]: spawn command to main: /Users/repeatedly/.rbenv/versions/2.3.1/bin/ruby -Eascii-8bit:ascii-8bit /Users/repeatedly/.rbenv/versions/2.3.1/bin/fluentd -c embedded.conf --under-supervisor
2016-10-13 09:12:41 +0900 [info]: reading config file path="embedded.conf"
2016-10-13 09:12:41 +0900 [info]: starting fluentd-0.14.7 without supervision
...[snip]...
2016-10-13 09:12:41 +0900 [info]: gem 'fluentd' version '0.14.7'
2016-10-13 09:12:41 +0900 [info]: adding source type="forward"
2016-10-13 09:12:41 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    foo machine1
  </source>
</ROOT>

> So, subsidiary question : what's the best way to test ruby code before putting it in the fluentd config ?

I don't have good idea because we don't assume such complex ruby code.

We add embedded ruby code feature for "#{ENV['FOO_BAR']}" like case.

I think deployment tool is better for above configuration handling.

--

[Nicolas Pepinster]

Thanks, it works.

There was indeed a carriage return at the end of the file. I fixed the output to /tmp/tenant instaed of using chomp.

I understand and agree your point, it would be better to execute these commands at build time in the deployment tool. 

I can' do that because my use case is deploying Fluentd Docker image on Kubernetes and reading file available only when the image is running.

Anyway, thanks again for your help. 

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.