Puppet-Dashboard All Nodes "Unresponsive", Background Tasks

[Ximena Cardinali]

Hello There,

I've been struggling the last days with this issue. The situation is happening since I've upgrade Puppet to 3.6.0.

- Problem:

On Puppet-Dashboard all Hosts are shown as Unresponsive and thousand of tasks are queued as Failed.

- Environment:

OS: Debian Wheezy

Puppet-Dashboard: 1.2.23

Puppet: 3.6.0

Facter: 2.0.1

Hiera: 1.3.2

- Applied Solutions:

* Solution 1:

cd /usr/share/puppet-dashboard/

- Stop dashboard workers

rm -v spool/*

rake jobs:clear RAILS_ENV=production

- Start dashboard workers.

Good solution, but temporary, because the problem after a few days comes back.

* Solution 2: 

I've also did the following update to the DB:

mysql> ALTER TABLE delayed_job_failures MODIFY details BLOB;

Which also did not work.

Does anyone have any idea of what can be happening there?

[Ximena Cardinali]

Hello,

I had to apply also this other Solution:

mysql> describe report_logs;

+-----------+--------------+------+-----+---------+----------------+

| Field     | Type         | Null | Key | Default | Extra          |

+-----------+--------------+------+-----+---------+----------------+

| id        | int(11)      | NO   | PRI | NULL    | auto_increment |

| report_id | int(11)      | NO   | MUL | NULL    |                |

| level     | varchar(255) | YES  |     | NULL    |                |

| message   | blob         | YES  |     | NULL    |                |

| source    | text         | YES  |     | NULL    |                |

| tags      | text         | YES  |     | NULL    |                |

| time      | datetime     | YES  |     | NULL    |                |

| file      | text         | YES  |     | NULL    |                |

| line      | int(11)      | YES  |     | NULL    |                |

+-----------+--------------+------+-----+---------+----------------+

9 rows in set (0.00 sec)

mysql> ALTER TABLE report_logs MODIFY message VARCHAR(65536);

Query OK, 46574 rows affected, 2 warnings (0.97 sec)

Records: 46574  Duplicates: 0  Warnings: 2

mysql> describe report_logs;

+-----------+--------------+------+-----+---------+----------------+

| Field     | Type         | Null | Key | Default | Extra          |

+-----------+--------------+------+-----+---------+----------------+

| id        | int(11)      | NO   | PRI | NULL    | auto_increment |

| report_id | int(11)      | NO   | MUL | NULL    |                |

| level     | varchar(255) | YES  |     | NULL    |                |

| message   | mediumtext   | YES  |     | NULL    |                |

| source    | text         | YES  |     | NULL    |                |

| tags      | text         | YES  |     | NULL    |                |

| time      | datetime     | YES  |     | NULL    |                |

| file      | text         | YES  |     | NULL    |                |

| line      | int(11)      | YES  |     | NULL    |                |

+-----------+--------------+------+-----+---------+----------------+

9 rows in set (0.00 sec)

mysql> 

For now, everything is working as expected. We will see in a few days, or after puppetdb Upgrade.

X.

[Ramin K]

It looks like you did this command 'alter table report_logs column
message mediumtext;' rather than varchar(65536). mediumtext is what it
should be changed to. Both BLOB and TEXT are 64kb while MEDIUMTEXT is 16MB.

Ramin

> _- Problem:_
> On Puppet-Dashboard all Hosts are shown as *Unresponsive* and
> thousand of tasks are queued as *Failed*.
>
> _- Environment:_

> OS: Debian Wheezy
> Puppet-Dashboard: 1.2.23
> Puppet: 3.6.0
> Facter: 2.0.1
> Hiera: 1.3.2
>

> _- Applied Solutions:_

>
> * Solution 1:
> cd /usr/share/puppet-dashboard/
> - Stop dashboard workers
> rm -v spool/*
> rake jobs:clear RAILS_ENV=production
> - Start dashboard workers.
>
> Good solution, but temporary, because the problem after a few days
> comes back.
>
> * Solution 2:
> I've also did the following update to the DB:
> mysql> ALTER TABLE delayed_job_failures MODIFY details BLOB;
> Which also did not work.
>
> Does anyone have any idea of what can be happening there?
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/da0290f9-5120-4040-b1f4-227878317ebc%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/da0290f9-5120-4040-b1f4-227878317ebc%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Ximena Cardinali]

Hello Ramin,

thanks for your answer.

I just wanted to say that varchar(65536) is mediumtext. As you can see in the table descriptions:

mysql> describe report_logs;

+-----------+--------------+------+-----+---------+----------------+

| Field     | Type         | Null | Key | Default | Extra          |

+-----------+--------------+------+-----+---------+----------------+

| id        | int(11)      | NO   | PRI | NULL    | auto_increment |

| report_id | int(11)      | NO   | MUL | NULL    |                |

| level     | varchar(255) | YES  |     | NULL    |                |

| message   | mediumtext   | YES  |     | NULL    |                |

| source    | text         | YES  |     | NULL    |                |

| tags      | text         | YES  |     | NULL    |                |

| time      | datetime     | YES  |     | NULL    |                |

| file      | text         | YES  |     | NULL    |                |

| line      | int(11)      | YES  |     | NULL    |                |

+-----------+--------------+------+-----+---------+----------------+

9 rows in set (0.00 sec)

mysql> 

Also, thanks a lot for letting me know that BLOB and TEXT are 64kb while MEDIUMTEXT is 16MB. I did not have that in my mind.

Cheers,

X.

> <mailto:puppet-users+unsub...@googlegroups.com>.