Problem running rake node:del for Puppet Dashboard databse hosted on RDS

[Felipe Salum]

Hi guys,

The command below was working when my puppet dashboard mysql database was running in the same machine as the puppet master.

rake RAILS_ENV=production -f /usr/share/puppet-dashboard/Rakefile node:del name=my-app-server

However now I moved the mysql database to RDS and it doesn't work anymore, my database.yml reflects the change and the Puppet Dashboard web interface works fine accessing the RDS database, but the rake task to delete is broken. Interesting is that node:list works and list all my nodes.

production:
  database: dashboard_production
  username: username
  password: password
  host: something.at.rds.amazonaws.com
  encoding: utf8
  adapter: mysql

Did I miss something during the local mysql to RDS migration ?

Thanks,
Felipe

[Felipe Salum]

I had a friend helping me to debug and looks like it is taking forever to delete the entries from resource status table (a lot of entries).

Is that something we can improve ?

[Ramin K]

On 7/24/2013 1:49 PM, Felipe Salum wrote:
> I had a friend helping me to debug and looks like it is taking forever
> to delete the entries from resource status table (a lot of entries).
>
> Is that something we can improve ?

env RAILS_ENV=production rake reports:prune upto=5 unit=day
env RAILS_ENV=production rake reports:prune:orphaned

You can change the RAILS_ENV and number of day, weeks, months, etc to
match your system and its needs.

Sounds like you've never pruned so it's going to be slow going. Try
walking it forward like so.

env RAILS_ENV=production rake reports:prune upto=3 unit=mon
env RAILS_ENV=production rake reports:prune upto=1 unit=mon
env RAILS_ENV=production rake reports:prune upto=2 unit=wk
env RAILS_ENV=production rake reports:prune upto=5 unit=day

Don't forget to add this to your crontab with the retentions settings
that best fit your system.

Ramin

[Felipe Salum]

I actually do for 2 weeks.

rake RAILS_ENV=production reports:prune upto=2 unit=wk

Ramin

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/7-sjp1Ef1p0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Ramin K]

Doesn't matter, it's the size of the data and the amount you're deleting.

Let's say that your reports:prune upto=2 unit=wk is ultimately going to
delete 100k records because you've never pruned the database before.

reports:prune upto=2 unit=wk, 100k, time: 3600s

By breaking it up, it'll run faster because large deletes can cause poor
performance in Mysql.

reports:prune upto=9 unit=wk, deletes: 9k, time: 280s
reports:prune upto=8 unit=wk, deletes: 11k, time: 250s
reports:prune upto=7 unit=wk, deletes: 10k, time: 251s
reports:prune upto=6 unit=wk, deletes: 12k, time: 239s
reports:prune upto=5 unit=wk, deletes: 11k, time: 205s
reports:prune upto=4 unit=wk, deletes: 11k, time: 180s
reports:prune upto=2 unit=wk, deletes: 22k, time: 200s

Running it nightly with a fully prune data set.
reports:prune upto=2 unit=wk, deletes: 1.5k, time: 25s

** I made up all these numbers up, but the performance ratio should be
roughly correct based on past experience. ymmv. **

Also the Rake task will load the id of the records it plans to delete
into memory. This can be quite large and is another reason to break it
into smaller chunks.

Ramin

> https://groups.google.com/d/__topic/puppet-users/7-__sjp1Ef1p0/unsubscribe
> <https://groups.google.com/d/topic/puppet-users/7-sjp1Ef1p0/unsubscribe>.

> To unsubscribe from this group and all its topics, send an email to

> puppet-users+unsubscribe@__googlegroups.com
> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> To post to this group, send email to puppet...@googlegroups.com

> <mailto:puppet...@googlegroups.com>.
> Visit this group at http://groups.google.com/__group/puppet-users
> <http://groups.google.com/group/puppet-users>.
> For more options, visit https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com.

[Felipe Salum]

I understood, what I wanted to say is that I run a cron job every day for the 2 weeks prune.. and I have less than 2 weeks data in the database right now since I moved it recently to a new server.

I'm running a optimize task, the resource_statuses is really big.

    <mailto:puppet-users@googlegroups.com>.

    Visit this group at http://groups.google.com/__group/puppet-users
    <http://groups.google.com/group/puppet-users>.
    For more options, visit https://groups.google.com/__groups/opt_out
    <https://groups.google.com/groups/opt_out>.



--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send

an email to puppet-users+unsubscribe@googlegroups.com.

To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/7-sjp1Ef1p0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.