logstash-plugin configure index

[iob]

i have set up a logstash-plugin to forward logs to elasticsearch. for now it works and i can see logs on logstash-jenkins index that
has been set in the plugin. What I want though is to sent to logstash-YY.mm.dd indices. i tried to set the "key" to logstash-* but doesnt seem to work. any idea?

[Josh Miller]

You would want to define the index in the output section of logstash.config, i.e.:

output {

    elasticsearch {

        hosts => [“host"]

        index => “logstash-logs-%{+YYYY.MM.dd}"

    }   

}

RE:  https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index

HTH,

Josh Miller
ITSA Consulting, LLC

skype:  itsecureadmin
https://itsecureadmin.com/

On Mar 17, 2017, at 7:24 AM, iob <jbona...@gmail.com> wrote:

i have set up a logstash-plugin to forward logs to elasticsearch. for now it works and i can see logs on logstash-jenkins index that
has been set in the plugin. What I want though is to sent to logstash-YY.mm.dd indices. i tried to set the "key" to logstash-* but doesnt seem to work. any idea?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/354daa13-1875-46e0-8d76-b1dc9b5b0f85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[iob]

yeah but what should I give to the jenkins' logstash-plugin. i have configured as logstash-%{+YYYY.MM.dd}/jenkins now. Although i cant see anything when I search for type:jenkins

[Josh Miller]

When you say you have it configured, do you mean the “key” field?  I have found that you do not need to set that.  To send the output of Jenkins jobs to logstash, only set:

Jenkins -> Manage Jenkins -> Logstash

1. indexer type to ‘syslog’

2. hostname to logstash

3. port to $logstash_port

If you have a lot of output going to logstash, you might have json parse errors which would prevent proper indexing to elasticsearch.  Check the records in elasticsearch around the time that you would expect to see the Jenkins output as it might not be typed properly.  One way to troubleshoot this is to set the max lines from the job going to logstash to something small, like 5, and then check to see if you are sending data properly.  Also, be sure to use the post build action to send output to logstash.

Best,

Josh Miller
ITSA Consulting, LLC

skype:  itsecureadmin
https://itsecureadmin.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c0c08e8f-4a70-4b2b-abe2-50544d033b59%40googlegroups.com.

[iob]

@Josh thanks a lot from your help so far.

I assume that you mean Jenkins -> Manage Jenkins -> Global Tool Configuration -> Logstash Plugin. Which in this case yes i mean the key. Also I noticed that you suggest indexer type to ‘syslog’ but i have to elasticsearch as I want to push directly to elastic. I am going to try what you suggest. But I wonder whether we do need to give the "key"?

[Josh Miller]

Ok, I typically send through logstash, so yes, you would want to set your indexer type to ‘elasticsearch’ and then fill in your host and port appropriately.  Looking at the help under the key, it looks like you want to set the index and type there:

re:  ELASTICSEARCH: The name and type path. Example: "/indexName/type"

It looks like your key definition should be correct here:  "logstash-%{+YYYY.MM.dd}/jenkins"

Best,

Josh Miller
ITSA Consulting, LLC

skype:  itsecureadmin
https://itsecureadmin.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ec8ce47d-6e76-40be-9225-b2b42570276c%40googlegroups.com.