StreamingLogWriterPlugin - open/addEvent/close doesn't seem to get called

[Russ Smith]

Hi folks,,

Firstly, I think it's very, very likely that I'm doing something wrong, so think of this as an opportunity to point and laugh!

We use Rundeck, and we're investigating ways of taking the log output from Rundeck and creating events that we feed into our event stream platform (Apache Kafka). I found the example for Logstash that someone did, which definitely looks comparable - we'd want to do a little more in terms of validating the schema, but the principle is the same.

However, before I even get to that point, I've hit a blocker. I cannot seem to get a much more straightforward groovy plugin to work as a StreamingLogWriterPlugin at all - writing to a local file, let alone sending it to a network based system.

On looking back through examples, I found the "JsonStreamingLogWriterPlugin.groovy" - which from my reading should write the output from any job into individual job-id named files in a chosen config directory. Once I got that working, I would add some groovy code in to submit the events and messages to Kafka.

However, still no success. I know that the JsonStreamingLogWriterPlugin.groovy file is being read in, because if I break the syntax I get a whole host of error messages in the Rundeck logs. I can also get JsonStreamingLogWriterPlugin.groovy to write a file when it's initiated, by adding the following within the rundeckPlugin(StreamingLogWriterPlugin) closure:

    def startup=new File("/tmp/rdlog","startup.out")

    startup<<'Hello\n'

    startup.withWriter{w->

        w<< 'Hello\n'

    }

When I run a job (I've got an otherwise empty job with "echo running" as a local execution command), it creates the /tmp/rdlog/startup.out file and puts "Hello" into it, so I know that Rundeck has picked it up.

However, that's all it does - it never seems to call the open, addEvent or close closures from within the plugin.

Can anyone point me in the right direction? I'm hoping I've missed something outrageously simple somewhere along the line - a config variable that enables it, or something in the job definition, or whatever.

Many thanks in advance,

Russ

[rac...@rundeck.com]

Hi Russ,

I tested this example plugin, just put it on the libext directory and add the following line to your rundeck-config.properties file to enable it:

rundeck.execution.logs.streamingWriterPlugins=JsonStreamingLogWriterPlugin

Some notes:

1) You can change the file path on line 11 (outpurDir).
2) The plugin saves the data on JSON format, it’s just an example, you can modify it to write on text-plain if you like.

It works in my environment (Rundeck 3.3.8), of course, feel free to modify it.

Hope it helps!

​

[Russ Smith]

Hiya,

You know, it's always the little things - thank you so much for your response, it allowed me to spot where I'd been going wrong. A typo - looks like I'd used rundeck.execution.logs.streamingWriterPlugin instead of rundeck.execution.logs.streamingWriterPlugins!

It's now happily chewing up my disk space as it should be - thanks again! Everyone: always check the little things!

Cheers,

Russ

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/50g2WezU7Zw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/af78e986-6084-4095-b480-deb5064b32b0n%40googlegroups.com.

[rac...@rundeck.com]

Happy to help Russ!

Greetings!