RS with more than one million resources

[carlos...@techex.es]

Dear friends.

Im in front of a RS installation for a photographic archive and the number of resources its over 1 millon and growning arround 10K of new pictures every month.

Even when we update the server up to 32GB the ram and configure the MariaDB over CentOs 7 to work with innoDb_buffer_pool-size to use 24 GB as the 80% the response is far from being the desired in terms of times. I know has no sense to do a search of all the resources but when i do i got the next messaje /var/www/html/fototeca/include/search_functions.php line 817: array_pad(): You may only pad up to 1048576 elements at a time so im afraid that the aplication has a implicit limit at the time to show a number of resources in on search results. Im using ResourceSpace version 7.9.8760 over CentOS 7 and MariaDb 5.6.

I tryed diferent searching and got resutts with set of resources arroun 2000 pictures, and when i try to navigate form one to another using the next or previous arrow i got the waiting animation about by 7 seconds in every picture. Im really dont know if this has to be with the database or could be another things involved that i could optimize to provide more speed and productivity.

Would love to knwo if someone can give me any advice or different aproach to the problem.

Regards

[Allison Stec]

array_pad() does have a limit, and that’s something that php imposes. That being said, the number is large enough where returning that many results is less than helpful.

You can adjust the config option $max_results to limit the results set to something under that of the php limit. The default setting is 200k. This will at least prevent your users from a negative experience.

It sounds like you should review some documentation of optimizing mysql (or hire someone to handle it).

Also, you might try updating to version 8 of RS. There has been a fair amount of work done to improve the data structure from 7.9-8.0. I’d be surprised if you didn’t see a noticeable improvement just from making the switch. I would suggest upgrading to at least 8.1 if possible. If you go this route be sure to follow best practices related to upgrading..back everything up! If it were me and I had the resources, I’d actually clone the system and run a test update first, leaving the current system available and untouched.

Allison Stec

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.com
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carlos Alonso]

Thansk so much for the help.

I'm preparing a RS 8.1 in other system with similar configuration 8 cores and 32GB and i will use the same Database to test the performance as you suggested.

In the mean time i was trying to do some test in order to know what is happening on the system.

As i commented before, i have 1.083.000 Resources and it will incremented 10K per month from now. so i try different variable configuration to see the performance difference.

I observed that if i set :

# Experimental performance enhancement - two pass mode for search results.
# The first query returns only the necessary number of results for the current search results display
# The second query is the same but returns only a count of the full result set, which is used to pad the result array to the correct size (so counts display correctly).
# This means that large volumes of resource data are not passed around unnecessarily, which can significantly improve performance on systems with large data sets.
$search_sql_double_pass_mode=false;

RS is heeding the variable $max_results=200000; if i set the $search_sql_double_pass_mode to true; seems to ignore the $max_results variable .

The values im using for the mariaDB are as follow:

key_buffer_size                 = 512M

myisam_sort_buffer_size         = 64M

join_buffer_size                = 2M

read_buffer_size                = 2M

sort_buffer_size                = 2M

read_rnd_buffer_size            = 2M

table_cache                     = 1024

tmp_table_size                  = 92M

max_heap_table_size             = 92M

max_allowed_packet              = 16M

max_connect_errors              = 100

max_connections                 = 250

query_cache_size                = 192M

query_cache_type                = 1

query_cache_limit               = 5M

thread_cache_size               = 4

innodb_file_per_table           = on

innodb_use_sys_malloc           = 0

innodb_buffer_pool_size         = 24G

innodb_flush_method             = O_DIRECT

innodb_buffer_pool_instances    = 5

innodb_log_buffer_size          = 8M

innodb_log_file_size            = 640M

innodb_log_group_home_dir       = /mnt/media/myibdata

My concern its about the system is behaving as it will not using the mariaDB caché. i set the $max_results=1000; to do some testing and do a search based on a keyword that give me the first 1000 results so its limited. when i open the view page of a resource and press in the next arrow to move to the next resource, the time for the query its the same anytime and im talking about 30 seconds for a so simple search. If icopy and paste the same query in the mariaDB console, i got 12 seconds to get response the first time and 0 seconds the second time, so the "next resource" is allways the same query everytime.

Is there any variable or code in the RS 7.9 that use temp tables i see $use_temp_tables in previews config.php files but not fine in the new versions.

Cheers

To unsubscribe from this group and stop receiving emails from it, send an email to resourcespace+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

=========================================================================
ATENCIÓN! CAMBIO DE UBICACIÓN, ROGAMOS ACTUALICEN SUS BASES DE DATOS
ATTENTION! WE HAVE MOVED OUR FACILITIES, PLEASE UPDATE YOUR RECORDS
=========================================================================
Carlos Alonso
Techex
carlos...@techex.es
www.techex.es
T. 918 591 152
M. 699 996 229
Calle de la Granja, 30
28108 Alcobendas (Madrid) - Spain

AVISO LEGAL: Este mensaje contiene información destinada exclusivamente al usuario de destino, pudiendo
contener información confidencial o protegida legalmente. Si, por un error de envío o transmisión, ha recibido
este mensaje y usted no es el destinatario del mismo, por favor, notifique de este hecho al remitente y no use,
informe, distribuya, imprima, copie o difunda este mensaje bajo ningún medio.
Cualquier opinión en él contenida, es exclusiva de su autor y no representa necesariamente la opinión de
Techex Iberica, S.L. o Techex Comunicaciones y Sistemas, S.L.


LEGAL WARNING:This message contains information intended exclusively for the addressee, it might contain
confidential information or legally protected data. If, by delivery or transmission error this message is received
and the adressee not known, please notify the sender and do not, inform, hand out, print, copy, or circulate the
said message under any circumstances.
Any opinion contained therein, belongs exclusively to its author and does not necessarily represent the opinion
of Techex Iberica, S.L. or Techex Comunicaciones y Sistemas, S.L.

[carlos...@techex.es]

Following Allison recommendation i did some more research and guessed that the bottle neck its more about Apache o PHP that MariaDB, when i copy and paste the query Resourcespace doing in the mariaDB console, the response time are so agile and surprise me so much, but if i do this same query's in RS or even in phpMyadmin, the query time its less than one second and the representation of the result rows can take 30 seconds.

Any idea to set the PHP to sue multiple cores or speed up this installation?

thanks