Best Practices for Purging/Archiving Old Logs/Data?

[Nick Tkach]

Are there any details or suggestions for how to safely purge/move old data from Rundeck?  I'm thinking especially of all the log files.  I'm not finding any details on when exactly it would be safe to move/compress/delete them.  Ideally I'd like to just use a cron to do something along the lines of compress ones that are older than say, 3 days, and delete/move ones that are older than say, 7.  Is it just as simple as doing exactly that?  So something along these lines:

0  1  *  *  *  /usr/bin/find /opt/rundeck/var/logs/ -type f -mtime +10 -exec rm {} \; > /dev/null 2>&1
0  1  *  *  *  /usr/bin/find /opt/rundeck/var/logs/ -type f ! -name "*.gz" -mtime +5 -exec gzip -f {} \; > /dev/null 2>&1

Do you have to also do something in the database side if you move logs around like that?

[Alex Honor]

Hi Nick,

For the log files, you could modify the log4j.properties and use a rolling file appender. That might make it easier to remove log data for a given range. The execution output files are a different matter. If you delete them, looking at history will show a warning that it can't find the output. What's needed is an API and GUI action to delete the execution from the database and the associated output log. It's on our roadmap but haven't got an ETA for that feature.

Thanks

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Alex Honor

[SimplifyOps, Inc | a...@simplifyops.com ]

Be sure to comment and vote on Rundeck Feature Development!

[Nick Tkach]

Cool, I'll give that a try.  When you say "execution output files", not quite sure what files you're talking about.  For example, if we have rundeck configured to use /var/logs for logging, do you mean the files that are in /var/logs/rundeck/*, all subdirectories with the project names?  The files down in there?

Alex Honor

[SimplifyOps, Inc | a....@simplifyops.com ]

[Alex Honor]

The execution output logs are the individual files that end in .rdlog. 

Alex Honor

[SimplifyOps, Inc | a...@simplifyops.com ]

[seld...@gmail.com]

I wrote a perl script some months ago for exactly that scenario.

Is based on the assumption that you are using the default h2 db, but should be no problem to modify for any other rdbms.

[Kayvon Raphael]

Came up with a very janky way to purge old executions. Posted on the github issue but putting here as well, as it was getting very inconvenient to wait for UI to delete executions in batches of 1000. works for now

* on the rundeck server

    * navigate to job directory you want to delete for (e.g. /var/lib/rundeck/logs/rundeck/Project1/job/rundeck_job_id/logs)

    * grab the execution IDs

        * ls -ltrh | head -n XXX | awk '{print $9}' | cut -d. -f 1 | uniq > /tmp/delete_ids.txt

        * remember that XXX should be double what you want to delete, as there are 2 files per execution

    * turn those into comma separated 

        * paste -d, -s /tmp/delete_ids.txt > /tmp/delete_ids_comma.txt

    * open that file and delete leading comma and turn into sql query ( DELETE FROM rundeck.execution WHERE id in (...)  )

    * now delete those log files

        * ls -ltrh | head -n XXX | awk '{print $9}' | xargs rm

and then from somewhere

* mysql -h your_mysql_master -u your_user -pXXX rundeck < /tmp/delete_ids_comma.txt

[danifr]

Hi,

regarding this, I wrote a small python script to delete old execution logs.
Basically, it deletes all the executions older than a specified date (3 months in the script) via the Rundeck API.

https://github.com/danifr/danifr-rundeck/blob/master/py_scripts/deleteoldlogs.py

It's not perfect but it works. It could be a good start point for those of you who are looking for this functionality.
I'm planning on improving it a little bit on the coming days.

Cheers,
Dani.

[Santiago López Denazis]

  Hi guys, I also need to delete old executions periodically.

  Dani, the problem with do that using the Rundeck Api, is that is very slow when you have a lot of executions. Because that, I prefer use a script like the Rapahael's script.

  Best,

--

You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Santiago López Denazis
GNU/Linux SysAdmin
slde...@linux.com
Free your source, free your mind.
----
Por favor, NO utilice formatos de archivo propietarios para el intercambio
de documentos, como DOC y XLS, sino HTML, PDF, TXT, CSV o cualquier otro
que no obligue a utilizar un programa de un fabricante concreto. Vea
http://www.gnu.org/philosophy/no-word-attachments.es.html
--

[Luca Busin]

I was having the same issue so I wrote a bash script to delete old logs and records:

https://gist.github.com/lucabusin/cad36a45764f2fc2e2daa81e2db4186d

This script extends the original script https://gist.github.com/unicolet/af648a97163ce6b44645 mainly to take care of workflow records too.

[Santiago López Denazis]

  Hi, this is the script that I wrote long time ago, I think it's still working: https://gist.github.com/sldenazis/8f845eafa3a7326daeeee03f0d69d5fa