Change project settings extremely slow

[Alexander Szele]

We have four Rundeck 5.18.0 instances (two devl, two prod) running on Red Hat Enterprise Linux 9 with MySQL backends. One of the prod instances has a large number of Projects (60+).

When we try to change settings on any of those projects, such as setting Disable Executions, the save process is EXTREMELY slow. Enough that users get confused and re-hit the Save button (which causes a token error, but that's what you get for hitting buttons repeatedly...)

Is this a known issue that is intended to be fixed? Is this just how it is now? (It was only reported to us this week, but we don't know what version the issue started with as it's not a commonly used function.)

Notably, these project changes are near instantaneous on the other three instances, so I suspect this is an artifact of us having such a large number of projects on that one host.

Any insight would be appreciated. Thanks!

Alex

[Ogme]

Hello,

IIRC, the projects structure is split from the jobs structure, so there shouldn't be much difference between 2 instances for this.
I even have a particular instance using on purpose the H2 Db with a load of jobs, which never raised any problem about this while being used daily.

Some pointers and suggestions about the instance with a large amount of projects :

• is there any error regarding the connection pool exhausted ?
  the default configuration has a max size of 10 connections, which might not be enough for your case.
  
  
• In the same manner, any error at all ?
  
  
• any monitoring information regarding the servers ? (cpu, memory,  storage, number of users on rundeck, ...)
  Rundeck has some integrated monitoring information which can be activated.
  It is also a java application, so you can use JMX to retrieve information from the instance, with tools like Jolokia for example. I would suggestion looking first at the memory heap usage, and threads pool.
  
  
• does the DB get regular maintenance applied, covering the indexes ?
  If there are a lot of delete + create, this could help a lot.
  Warning : such maintenance can lock the DB for a while the first time depending of the amount of data.
  
  
• compare the following with one of the other instances working correctly : 
• the dataSource.url parameter (rundeck-config.properties)
• the JDBC connector driver (jar) and its version
  
• the indexes on the Rundeck DB tables.
  An old instance could be using an old JDBC driver, or have some indexes still present from a previous, older version
  
  
• There were rare cases where adding manually an extra index on a table could drastically help.
  Activate the slow query log on the DB engine to have some pointers for the tables which could beneficiate from this.
  
  

Regards,

    Ogme

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rundeck-discuss/38bcb153-bb5c-4a1c-bd19-6a2271a1d4d7n%40googlegroups.com.

[Alexander Szele]

Sorry for the late reply. Regarding suggestions above:
- no errors happening in the logs of any significance, regarding the database or otherwise.

- Active monitoring of the servers. No CPU, Load, or Memory spikes at all and none of those are showing any kind of resource pressure. Memory usage is stable around 5G of 8G available. I haven't tried monitoring rundeck's internal processes or Java details during one of these slow response events. I will see if that's manageable in our instance.

- DBs are maintained regularly by our DBA team handling index maintenance, etc.

- Configs are identical. All four servers deployed by ansible with the only differences being the target databases. They're even using the same DB host, a MySQL enterprise cluster. And they were all deployed around the same time and are maintained on the same schedule with monthly updates.

- We'll check the slow query log the next time we get a report of this.

But overall, I'm getting the impression that this is simply an inefficiency in the application or database design where they simply did not anticipate an environment with this large a number of projects?

Thanks!

[Alexander Szele]

Apologies again for the double reply - minor correction. These are NOT using our central MySQL cluster, but each server has a dedicated standalone MySQL server. All of which are still maintained by the DBA team to the same standards. But each rundeck instance has a dedicated db server. Sorry for the confusion on my part.

[Alexander Szele]

OK. We've found the issue I think.

When we enable or disable executions on a project, there's a VERY slow query that runs to pull all the scheduled jobs from the execution table.

The query is essentially "SELECT ...allcolumns... FROM execution WHERE status='scheduled' and project='<PROJECTNAME';

It currently takes over 30 seconds in our environment as it's scanning all 690,000 rows of the execution table.

There's no index on status/project as a pairing, so the select is scanning the entire table and bogging down the change.

I think. It's really hard to tell what exactly is happening here...

Should I put this into git as an issue to address?