Puppet dashboard stuck pending jobs

[Jagga Soorma]

Hi Guys,

I am a new puppet user and wanted some type of monitoring for puppet so deployed puppet-dashboard.  It has been working very well for a few days not, but all of a sudden I start getting pending tasks and they never finish even after restarting all processes.  They keep accumulating and never seem to finish even though the clients are running fine.  I have the puppet-dashboard running on a different server than my puppetmaster.  Is there any way to troubleshoot what seems to be stuck and how to I get it going without having to destroy the dashboard mysql db and recreating it again?  That is the only way I have got it working again but that is not the correct way of doing this :)  Any help would be appreciated.

Thanks!

[Jo Rhett]

Make sure that your dashboard workers are running. If you are using CentOS then "service puppet-dashboard-workers status" will tell you.

On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:

I am a new puppet user and wanted some type of monitoring for puppet so deployed puppet-dashboard.  It has been working very well for a few days not, but all of a sudden I start getting pending tasks and they never finish even after restarting all processes.  They keep accumulating and never seem to finish even though the clients are running fine.  I have the puppet-dashboard running on a different server than my puppetmaster.  Is there any way to troubleshoot what seems to be stuck and how to I get it going without having to destroy the dashboard mysql db and recreating it again?  That is the only way I have got it working again but that is not the correct way of doing this :)  Any help would be appreciated.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Jo Rhett

Net Consonance : net philanthropy to improve open source and internet projects.

[Jagga Soorma]

Yes, the workers are definitely running and I have tried restarting puppet-dashboard-workers a few times as well as rebooting the vm completely without any luck.  Something seems to be stuck somewhere but I can't figure out what.  This has happened a few times now and I have just been recreating from scratch to get it functional again.  Are there any logs for the workers that might tell us why these pending jobs are stuck or not finishing?

Thanks.

[Jagga Soorma]

Anyone out there run into this issue before or have any idea about how to troubleshoot such a problem?  Any help would be appreciated.  I can't keep rebuilding puppet dashboard every time this happens.

Thanks

On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote:

Yes, the workers are definitely running and I have tried restarting puppet-dashboard-workers a few times as well as rebooting the vm completely without any luck.  Something seems to be stuck somewhere but I can't figure out what.  This has happened a few times now and I have just been recreating from scratch to get it functional again.  Are there any logs for the workers that might tell us why these pending jobs are stuck or not finishing?

Thanks.

On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett <jrh...@netconsonance.com> wrote:

Make sure that your dashboard workers are running. If you are using CentOS then "service puppet-dashboard-workers status" will tell you.

On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:

I am a new puppet user and wanted some type of monitoring for puppet so deployed puppet-dashboard.  It has been working very well for a few days not, but all of a sudden I start getting pending tasks and they never finish even after restarting all processes.  They keep accumulating and never seem to finish even though the clients are running fine.  I have the puppet-dashboard running on a different server than my puppetmaster.  Is there any way to troubleshoot what seems to be stuck and how to I get it going without having to destroy the dashboard mysql db and recreating it again?  That is the only way I have got it working again but that is not the correct way of doing this :)  Any help would be appreciated.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Jo Rhett

Net Consonance : net philanthropy to improve open source and internet projects.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

[Esther Hanko]

Try looking in /usr/share/puppet-dashboard/log/.
Did you update the dashboard or puppet master recently? I have ran into similar problems after updates, since I run the dashboard on a different port than the external_nodes script uses. The DASHBOARD_URL in /usr/share/puppet-dashboard/bin/external_node got changed back into localhost:3000 and as a result, all my nodes were seen as unresponsive.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[Felipe Salum]

Are you having the problem after running 'puppet agent --test' a few times or it just happens without any manual run ?

I was having the same problem some days ago, in my case because I was running 'puppet agent --test', more specifically the --show_diff that is used in --test was automatically generating a lot of output and causing the puppet dashboard workers to fail and increase the processing queue.

Look at /usr/share/puppet-dashboard/log/delayed_job.log (or whatever your dashboard location is) and check what error it is giving.

You can clean the queue manually without reinstalling everything:

service puppet-dashboard-workers stop
cd /usr/share/puppet-dashboard
rm -f spool/*
rake jobs:clear RAILS_ENV=production
service puppet-dashboard-workers start

Regards,
Felipe

On Saturday, February 9, 2013 7:11:04 PM UTC-8, Jagga Soorma wrote:

Anyone out there run into this issue before or have any idea about how to troubleshoot such a problem?  Any help would be appreciated.  I can't keep rebuilding puppet dashboard every time this happens.

Thanks

On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote:

Yes, the workers are definitely running and I have tried restarting puppet-dashboard-workers a few times as well as rebooting the vm completely without any luck.  Something seems to be stuck somewhere but I can't figure out what.  This has happened a few times now and I have just been recreating from scratch to get it functional again.  Are there any logs for the workers that might tell us why these pending jobs are stuck or not finishing?

Thanks.

On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett <jrh...@netconsonance.com> wrote:

Make sure that your dashboard workers are running. If you are using CentOS then "service puppet-dashboard-workers status" will tell you.

On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:

I am a new puppet user and wanted some type of monitoring for puppet so deployed puppet-dashboard.  It has been working very well for a few days not, but all of a sudden I start getting pending tasks and they never finish even after restarting all processes.  They keep accumulating and never seem to finish even though the clients are running fine.  I have the puppet-dashboard running on a different server than my puppetmaster.  Is there any way to troubleshoot what seems to be stuck and how to I get it going without having to destroy the dashboard mysql db and recreating it again?  That is the only way I have got it working again but that is not the correct way of doing this :)  Any help would be appreciated.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Jo Rhett

Net Consonance : net philanthropy to improve open source and internet projects.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[Jagga Soorma]

Thanks for pointing me to the log location.  Looks like this might be the cause of the problem:

--
..snip
2013-02-10T10:40:35-0800: [Worker(delayed_job.3 host:ssfpcond01 pid:25048)] Report.create_from_yaml_file failed with ActiveRecord::StatementInvalid: Mysql::Error: Data too long for column 'details' at row 1: INSERT INTO `delayed_job_failures` (`summary`, `read`, `updated_at`, `backtrace`, `details`, `created_at`) VALUES('Importing report report-18121-1179.
yaml', 0, '2013-02-10 18:40:35', '--- \n- /usr/share/puppet-dashboard/vendor/rails/activerecord/lib/active_record/connection_adapters/a
..snip..
--

I am managing my passwd & group file via puppet which at times have huge amount of changes especially in the group file.  This could possibly be causing this error.  Is there a way to increase the size of the details column to enable it to accept larger data?

Has anyone seen this issue before?

Thanks again for your help with this.

[Jagga Soorma]

Looks like that was actually the problem.  Searched google and came across the fix.  Had to change the type of the messages column to mediumtext and the pending jobs started finishing.  Did this by:

alter table report_logs change column message message VARCHAR(65536);

Just in case someone runs into this issue.  Thanks.

[Jesse Martinich]

Your 'alter table' fix worked for me too. Thanks!

[Sean Penticoff]

I ran into this as a result of the output of failed deps on a yum
install and had a couple jobs that just wouldn't go away. After fixing
the condition creating the error I used /usr/bin/rake jobs:clear to
clear out the delayed_job queue and everything was back to normal.

[cko]

That fixed the problem for me. Thanks.

[Julien Nephtali]

I found the solution to my problem here:

http://projects.puppetlabs.com/issues/18411

ALTER TABLE delayed_job_failures MODIFY details BLOB;

that resolved the error.