Log storage in S3 and "live" logging

[Xavier Humbert]

Hi,

My users complain that since the executions logs are stored in S3, they
cannot follow the logs in the activity window, but have to wait for the
termination

My config :

In rundeck-config.properties :

## S3 Log Storage
rundeck.execution.logs.fileStoragePlugin = org.rundeck.amazon-s3


In framework.properties

# ----------------------------------------------------------------
# S3 Storage
# ----------------------------------------------------------------

# AWSCredentialsFile can point to a file which contains `accessKey` and
`secretKey`
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSCredentialsFile
= /etc/rundeck/.aws/credentials
# name of the bucket
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket = rundeck
# path to store the logs
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path =
logs/${job.project}/${job.execid}.log
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.host_base =
storagegrid.XXX
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.endpoint =
storagegrid.XXX
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.pathStyle = True

Is this the expected behaviour ?

Thanks,

Xavier

--
Xavier Humbert
CRT Supervision et Exploitation de Niveau 1
Rectorat de Nancy-Metz
03 83 86 27 39

[rac...@rundeck.com]

Hi Xavier,

Tested on 4.9. I can follow the job execution “in real time”.

What Rundeck version are you using? Plugin version? Do you see any clue in the service.log file?

I tested successfully this job definition:

- defaultTab: nodes description: '' executionEnabled: true id: c1da227e-7031-456d-9df1-b927182e6570 loglevel: INFO name: HelloWorld nodeFilterEditable: false plugins: ExecutionLifecycle: null scheduleEnabled: true sequence: commands: - exec: echo "hi" - exec: sleep 15; echo "henlo world" keepgoing: false strategy: node-first uuid: c1da227e-7031-456d-9df1-b927182e6570

rundeck-config.properties:

rundeck.execution.logs.fileStoragePlugin=org.rundeck.amazon-s3

framework.properties:

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSAccessKeyId=xxx framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSSecretKey=xxx framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket=mybucket framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path=logs/${job.project}/${job.execid}.log

Could you test with this config?

Regards!

​

[Xavier Humbert]

Hi Rainer,

This is Rundeck 4.9, with plugin rundeck-s3-log-plugin-1.0.12.jar. Nothing in service log

I got "Waiting for Workflow State and Log Output to become available..." in nodes view and in log output view, just the progress bar...

Regards

Xavier

Le 1/17/23 14:49, rac...@rundeck.com a écrit :

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/d9e18e66-d122-4202-a7df-16082fc774dfn%40googlegroups.com.

[rac...@rundeck.com]

Hi Xavier,

It seems pretty similar to this, do you have a similar "clustered environment"?

Greetings.

[Xavier Humbert]

Hi Rainer

Actually yes

Regards,

Xavier

Le 1/17/23 19:16, rac...@rundeck.com a écrit :

To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/c827c0dc-374c-431a-a321-a5d00e693674n%40googlegroups.com.

[rac...@rundeck.com]

Hi Xavier,

Please add your use case on that thread, but Rundeck OSS isn't designed to work in clustered environments. That's only possible on Process Automation (formerly "Rundeck Enterprise"). Take a look at this.

Regards!