Videos and server_add module

56 views
Skip to first unread message

Darby Vicker

unread,
Feb 23, 2021, 1:45:51 AMFeb 23
to Gallery 3 Users
Hey everyone,

With my video situation in good shape, I've been attempting to add my latest videos.  I've always used the server_add module for both pictures and videos since I tend to do large batches at a time.   Adding items is more more reliable and I didn't have as many timeout issues if I just rsync the files to my server and then use server_add to add them to my albums.  I am having a problem adding videos.  This always worked just fine with flv files, but something is not working correctly with the new mp4 files.  To be fair, I never tried to add an flv file with the new gallery (3.1.3) installation so I don't know if this is specific to mp4 files or if flv files would have behaved the same way.  

If I use server_add and select just a single mp4 file, the status bar seems to stall and never completes.  But if I wait a couple minutes and hit the "close" button, the video is added properly.  

However, if I select a directory containing several mp4 files (7 in this case), the status bar once again stalls and the message never progresses past "Adding photos / albums (1 of 8)".  Even if I wait quite a long time (~10 minutes), nothing happens.  If I press the close button, I'm left with an empty album and nothing added.  The sizes of the mp4 range from 5 MB to 50 MB, so nothing too crazy.  Those file sizes are very comparable to the flv files I've uploaded previously (with a non-revival version of gallery).  In the past, the status bar would progress through the files at the rate of a few seconds per file, maybe just a bit longer for larger videos.  

I was thinking this was specific to videos but, in retrospect, I don't think I have tried adding any new albums with server_add since I did the 3.1.3 upgrade.  So I also just tried adding a new album (directory) with just 13 jpg's (no videos).  The behavior was almost the same - the process stalls and the status bar never gets past the first file.  In this case, one of the pictures was processed and added to the album.  But I waited more than 5 minutes before closing the uploader, which should have been more than enough time for the 13 pictures in this test.  So I think the server_add module is generally broken.  Any help getting this working again would be greatly appreciated.

Output from var/logs/2021-02-23.log.php show below.  Perhaps that is enough but are there other logs I can check? Please let me know what else I can check or provide to debug this.  

Thanks,
Darby


2021-02-23 05:57:16 +00:00 --- error: Database_Exception: #2006: MySQL server has gone away [ SELECT *

FROM `g3_caches`

WHERE `key` IN ('---') ] in /home/thevic11/public_html/gallery3-3.1.3/system/libraries/Database_Mysqli_Result.php:27

Stack trace:

#0 /home/thevic11/public_html/gallery3-3.1.3/system/libraries/Database_Mysqli.php(79): Database_Mysqli_Result_Core->__construct(false, 'SELECT *\nFROM `...', Object(mysqli), true)

#1 /home/thevic11/public_html/gallery3-3.1.3/system/libraries/Database.php(272): Database_Mysqli_Core->query_execute('SELECT *\nFROM `...')

#2 /home/thevic11/public_html/gallery3-3.1.3/modules/gallery/libraries/MY_Database.php(48): Database_Core->query('SELECT *\nFROM `...')

#3 /home/thevic11/public_html/gallery3-3.1.3/system/libraries/Database_Builder.php(973): Database->query('SELECT *\nFROM `...')

#4 /home/thevic11/public_html/gallery3-3.1.3/modules/gallery/libraries/drivers/Cache/Database.php(105): Database_Builder_Core->execute()

#5 /home/thevic11/public_html/gallery3-3.1.3/system/libraries/Cache.php(141): Cache_Database_Driver->get(Array, true)

#6 /home/thevic11/public_html/gallery3-3.1.3/modules/gallery/models/task.php(58): Cache_Core->get(Array)

#7 /home/thevic11/public_html/gallery3-3.1.3/modules/server_add/controllers/server_add.php(288): Task_Model_Core->log('Skipping invali...')

#8 /home/thevic11/public_html/gallery3-3.1.3/modules/gallery/helpers/task.php(90): Server_Add_Controller::add(Object(Task_Model))

#9 /home/thevic11/public_html/gallery3-3.1.3/modules/server_add/controllers/server_add.php(122): task_Core::run('410')

#10 [internal function]: Server_Add_Controller->run('410')

#11 /home/thevic11/public_html/gallery3-3.1.3/system/core/Kohana.php(331): ReflectionMethod->invokeArgs(Object(Server_Add_Controller), Array)

#12 /home/thevic11/public_html/gallery3-3.1.3/system/core/Event.php(208): Kohana_Core::instance(NULL)

#13 /home/thevic11/public_html/gallery3-3.1.3/application/Bootstrap.php(67): Event_Core::run('system.execute')

#14 /home/thevic11/public_html/gallery3-3.1.3/index.php(123): require('/home/thevic11/...')

#15 {main}

Brad Dutton

unread,
Feb 23, 2021, 11:31:22 AMFeb 23
to Gallery 3 Users
Check your mysql/mariadb logs, the db server is crashing or hitting some sort of limit that the connection is going away.

Darby Vicker

unread,
Feb 24, 2021, 12:25:29 AMFeb 24
to Gallery 3 Users
My site is on a hosting provider and I don't have easy access to the sql logs.  I've asked for them.  

Does it make sense that the the server_add module would break with the 3.1.3 upgrade when it was working fine before that?  As part of the process of evaluating other photo gallery software I was forced to upgrade my site from PHP5 to PHP7.  Would that have changed the interaction with the SQL server?  

Brad Dutton

unread,
Feb 24, 2021, 12:49:12 AMFeb 24
to Gallery 3 Users
PHP 7 uses the "mysqli" module instead of "mysql":
They are supposed to be functionally the same. Other than changing the mysql module being used, gallery shouldn't have any changes that intentionally change the behavior.

Darby Vicker

unread,
Feb 24, 2021, 1:06:50 AMFeb 24
to Gallery 3 Users
In researching this error, I've seen suggestions that increasing timeouts can help.  I've tried putting these in php.ini:

[PHP]
mysql.connect_timeout = 300
mysql.interactive_timeout=300

But that doesn't seem to help.  Do you know if that's the right section?  Should it go in the [Session] section of php.ini?  


Brad Dutton

unread,
Feb 24, 2021, 11:49:23 AMFeb 24
to Gallery 3 Users
I think these would be the applicable options:
https://www.php.net/manual/en/mysqli.configuration.php
The only one I could see helping would be:
mysqli.reconnect=1
That wouldn't fix the problem but it may work around it by creating a new connection after the old one is dropped. You can put the option anywhere the section headers (.e.g [PHP]) aren't actually used for anything.

Darby Vicker

unread,
Feb 24, 2021, 12:43:57 PMFeb 24
to Gallery 3 Users

I just tried with these:

mysql.connect_timeout=300
mysql.interactive_timeout=300
mysqli.reconnect=1


No change - still hangs and I still get the same stack trace I showed before in the logs.  

The php.ini file gets loaded every time you do an operation, correct?  Or would I have have to restart something to make those changes take effect?  

Brad Dutton

unread,
Feb 24, 2021, 12:45:39 PMFeb 24
to Gallery 3 Users
Apache has to be restarted after these changes.

Darby Vicker

unread,
Feb 24, 2021, 1:07:09 PMFeb 24
to Gallery 3 Users
Ahh, that's harder - again, will need the hosting provider to to help on that.  

If anyone else is able to try using the server_add module on a version of revival with php 7, that would be great.  It would be nice to know if this is just a problem on my end with the particular setup of my hosting provider or a more general issue.  

JR

unread,
Feb 24, 2021, 4:30:33 PMFeb 24
to gallery...@googlegroups.com

I'm going to throw in some info on the "timeout" issues that may or may not be helpful.

These connect maximum connect time setting are in seconds, so a setting=300 is 5 minutes, which may seem like plenty. But large files take time to process *after* they hit their destination and the processing must go through the old Gallery code -- and be integrated into the Gallery file system and the various database references for it created. I'm thinking Server_add is not going to send the next file until it gets confirmation that the previous file was properly processed.  So a setting of 600 or more (10 minutes) might help. Bear in mind the setting is the *maximum* time allowed for the SQL transaction to *complete* -- so why not go for a higher ceiling? If the transaction completes and "reports back"  faster than that, it's no harm, no foul, and the next file is sent.

Except, there is this to consider:

"mysql.connect_timeout applies only to establishing a connection to mySQL, while default_socket_timeout applies to all subsequent communication after the connection has been established. So, if you already have an established connection, the connect_timeout parameter will be ignored once you send your query. But if a subsequent transaction take more 300 seconds to acknowledge, the socket_timeout parameter -- if it is not set to higher than 300 seconds --will be triggered and kill the transaction."

So, maybe take a look at default_socket_timeout as well? The PHP manual states that the default for it is only 60 (1 minute).

More info on this setting at:

https://www.php.net/manual/en/filesystem.configuration.php

The above is the actual official PHP manual and keyword searching it can turn up all kinds of valuable info (most of which goes way over my head) which is why I'm only pointing out this stuff :)

--J.R.

--
WHEN USING AN EMAIL PROGRAM to reply to this message, click REPLY TO LIST or REPLY TO ALL so your reply goes out to everyone in the group. If you click REPLY or REPLY TO SENDER Google will *only* send your reply to the original author (not recommended).
 
To post a NEW MESSAGE to the group, send an new email to:
gallery...@googlegroups.com
 
To view or sign in to this group on the web, use this URL:
https://groups.google.com/forum/#!forum/gallery-3-users
---
You received this message because you are subscribed to the Google Groups "Gallery 3 Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gallery-3-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gallery-3-users/e0f4fbf8-5de1-4413-b693-f7b0c67a3f23n%40googlegroups.com.

Darby Vicker

unread,
Feb 25, 2021, 1:44:41 AMFeb 25
to Gallery 3 Users
Tech support wasn't super helpful - they wouldn't give me access to the sql logs directly because I'm on a shared server.  They eventually sent me some info:

2021-02-24 9:30:32 140291337217792 [Warning] Aborted connection 9135004 to db: 'thevic11_gallery3' user: 'thevic11_gallery' host: 'localhost' (Got timeout reading communication packets) 
2021-02-24 9:34:56 140282546280192 [Warning] Aborted connection 9139610 to db: 'thevic11_gallery3' user: 'thevic11_gallery' host: 'localhost' (Got timeout reading communication packets) 
2021-02-24 9:36:18 140291484628736 [Warning] Aborted connection 9140945 to db: 'thevic11_gallery3' user: 'thevic11_gallery' host: 'localhost' (Got timeout reading communication packets) 
2021-02-24 21:10:00 140282543576832 [Warning] Aborted connection 9864940 to db: 'thevic11_gallery3' user: 'thevic11_gallery' host: 'localhost' (Got timeout reading communication packets)

I'm not sure how helpful that is. 

However, I'm making some progress on figuring out the problem.  I've been trying to add the same directory of videos this entire time in my tests that are stalling.  I tried adding another set of videos and, low and behold, some of them added properly before the process stalled again.  I eventually figured out this is a file size issue.  Any file over 20MB stalls the process and causes the database errors.  So, I could use some help figure out where the 20 MB file limit is being imposed.  There are probably a few different layers where this limit could be imposed - apache, php, sql, others?  I see in the php.ini at the top level of gallery, I have these: 

upload_max_filesize = 20M
post_max_size = 100M

The same values are imposed in the .htaccess.  I wouldn't think the upload size is being imposed here since the file is already on the server.  But it does correlate with the size limit that looks like is being imposed.  I tried increasing upload max size to 100M in both the php.ini and .htacess and adding a 75 MB video but it still stalled and produced the DB connection loss messages in the logs.  I don't think these are the right values to change anyway because I still have the gallery source from the pre-revival upgrade and those values are also 20M and 100M in both the php.ini and .htaccess.  I just checked and most of my videos from 2019 are over 20MB - some are as large as 300MB.  I can keep poking around in the old source to see if I can find where these limits changed but if anyone can point me in the right direction, that would be great.  

Also, would it make sense to try and trap these kind of errors?  Knowing what the imposed limits are, should the code check the file size and throw a warning if the file size exceeds the limits?  

Brad Dutton

unread,
Feb 25, 2021, 11:26:05 AMFeb 25
to Gallery 3 Users
Using server_add the max post and upload sizes shouldn't matter as the files are already on the server. These would only be used if you use the normal upload process. In which case these 3 settings matter:
  php_value upload_max_filesize      80M
  php_value post_max_size            100M
  php_value memory_limit             200M

If you can view any other apache or php logs check there to see if anything is happening outside of the gallery. Some of these limits wouldn't be visible to the application itself as the server would stop execution before gallery could report anything.

Darby Vicker

unread,
Mar 4, 2021, 1:28:47 AMMar 4
to Gallery 3 Users
I was able to get php logs turned on so that's good but the aren't showing any additional errors when the server_add module loses its connection.  

Using the code in the accepted answer here:  

https://stackoverflow.com/questions/33250453/how-to-solve-general-error-2006-mysql-server-has-gone-away

I was able to verify that my timeouts are pretty small.  That code also shows the values before and after and confirms that I can change the values.  Here is the same confirmation of small timeout values from mysql directly:

MariaDB [(none)]> SHOW VARIABLES LIKE  '%timeout%' ;
+-----------------------------+----------+
| Variable_name               | Value    |
+-----------------------------+----------+
| connect_timeout             | 10       |
| deadlock_timeout_long       | 50000000 |
| deadlock_timeout_short      | 10000    |
| delayed_insert_timeout      | 300      |
| innodb_flush_log_at_timeout | 1        |
| innodb_lock_wait_timeout    | 50       |
| innodb_rollback_on_timeout  | OFF      |
| interactive_timeout         | 30       |
| lock_wait_timeout           | 86400    |
| net_read_timeout            | 30       |
| net_write_timeout           | 60       |
| slave_net_timeout           | 60       |
| thread_pool_idle_timeout    | 60       |
| wait_timeout                | 30       |
+-----------------------------+----------+

So if I wanted to try and make these kinds of calls within the gallery code to increase the timeouts:

$results = $db->query("SET session wait_timeout=28800", FALSE);
$results = $db->query("SET session interactive_timeout=28800", FALSE);

Where is the best place to do that?  Or should I be setting these values somewhere else?  
Reply all
Reply to author
Forward
0 new messages