Memory issues (v2.53)

[brzic...@gmail.com]

Hi folks,

trying to upload a 10 GB file using the command line :

sudo php symfony digitalobject:load --index /home/xxx/CSV.csv

I ran into this issue :

mmap() failed: [12] Cannot allocate memory

PHP Fatal error:  Out of memory (allocated 12582912) (tried to allocate 9588163448 bytes) in /usr/share/nginx/atom/lib/task/digitalobject/digitalObjectLoadTask.class.php on line 269

 

Fatal error: Out of memory (allocated 12582912) (tried to allocate 9588163448 bytes) in /usr/share/nginx/atom/lib/task/digitalobject/digitalObjectLoadTask.class.php on line 269

Is this a hardware limit or a php limit ? What can I do here ?

The file is too big for uploading via the AtoM GUI.

Thanks

[Karl Goetz]

On Tue, 26 Nov 2019 11:01:20 -0800 (PST)
brzic...@gmail.com wrote:

> Hi folks,
>
> trying to upload a 10 GB file using the command line :
>
> sudo php symfony digitalobject:load --index /home/xxx/CSV.csv

Hi,
Could I confirm confirm you are trying to load a 10 gigabyte CSV file?

You should split that up, no matter what PHP's memory limit...

> I ran into this issue :
>

> * mmap() failed: [12] Cannot allocate memory*
>
> *PHP Fatal error: Out of memory (allocated 12582912) (tried to allocate
> 9588163448 bytes) in
> /usr/share/nginx/atom/lib/task/digitalobject/digitalObjectLoadTask.class.php
> on line 269*
>
> * Fatal error: Out of memory (allocated 12582912) (tried to allocate
> 9588163448 bytes) in
> /usr/share/nginx/atom/lib/task/digitalobject/digitalObjectLoadTask.class.php
> on line 269*

>
>
> Is this a hardware limit or a php limit ? What can I do here ?

Its a php limit; you'll need to set it in different locations depending on how you want to use it, the two likely options are below.

/etc/php/7.0/cli/php.ini /etc/php/7.0/fpm/php.ini

thanks,

--
Karl Goetz
Technical Services Officer - eResearch, Information Technology Services
University of Tasmania & Tasmanian Partnership for Advanced Computing

Mail: University of Tasmania, Private Bag 69, Hobart, Tasmania 7001
Delivery: TT Flynn Street, Sandy Bay, Tasmania 7005



University of Tasmania Electronic Communications Policy (December, 2014).
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

[Dan Gillean]

Hi there, 

I agree with Karl - considering that the digital object load task expects a 2-column CSV, a 10GB CSV file is HUGE, and you should definitely break that up into smaller imports! And yes, the digital object load task is not currently supported via the user interface anyway, so 

It is likely a PHP configuration issue, so you could look at adjusting some of the execution limit values in your php.ini file, as Karl suggests. See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/installation/execution-limits/
  

That said, it could also be that you've exhaused all available memory on the server... how much memory does this server have allocated?

Let us know how it goes! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/20191127093852.1d1a042d%40minister.

[brzic...@gmail.com]

Sorry folks, I was not clear.

The CSV file is just a single line pointing to a single 10GB .mp4 file

[brzic...@gmail.com]

The server has 12 GM of memory allocated...

 

[brzic...@gmail.com]

OK. Made some changes to the /etc/php/7.2/cli/php.ini file.

Getting a new error :

Checksum values did not validate: /usr/share/nginx/atom/uploads/r/null/b/8/6/

That seems like a corrupt file. What would be a good way to deal with this ?

-recreate file (time intensive)

-recreate checksum (what tool would you folks suggest?)

Thanks

 

[cindy mclellan]

I'm getting the same error. Any mp4 files that I try to upload over 4 GB:

Checksum values did not validate: /usr/share/nginx/atom/uploads/r/null/b/8/6/

[Dan Gillean]

Hi there both,

I don't think I've seen this particular error before with the load task.  

Are you trying to replace an existing digital object, or upload a new one to a description that hasn't previously had one attached?

Are you using the description CSV, or the 2-column CSV we recommend creating for the command-line digital object load task?

Have you tried looking in that directory (/usr/share/nginx/atom/uploads/r/null/b/8/6/) to see if something is there?

Is the target description linked to a repository record? (I ask because it looks like it is trying to upload to uploads/r/null, which suggests no related repository. It's possible that if you link a repository record, the next upload will try in a different place and may avoid this strange conflict). 

Any further info you can provide on your exact steps will be helpful - thanks! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4edcfd9c-7c1b-4529-9cb3-bb37f572a530%40googlegroups.com.

[brzic...@gmail.com]

Are you trying to replace an existing digital object, or upload a new one to a description that hasn't previously had one attached?

Add a new one to a description that has not previously had one attached - but I had the php memory error on the first attempt.

 

Are you using the description CSV, or the 2-column CSV we recommend creating for the command-line digital object load task?

Using 2-column CSV 

Have you tried looking in that directory (/usr/share/nginx/atom/uploads/r/null/b/8/6/) to see if something is there?

Yes, the directory exists but it is empty .

Is the target description linked to a repository record?

Yes, the object ID was obtained by using an SQL query

[brzic...@gmail.com]

I ask because it looks like it is trying to upload to uploads/r/null, which suggests no related repository. It's possible that if you link a repository record, the next upload will try in a different place and may avoid this strange conflict-

All my uploads are in ..uploads/r/null/…

my folder structure is:

..uploads/r

..uploads/tmp

..uploads/r/null

..uploads/r/null/0

to

..uploads/r/null/f

[Dan Gillean]

Hi again, 

I will ask our devs for further thoughts, but based on some of your other posts, I'm wondering if there is a language issue at play here.

What is the default installation culture of your site? You can check this in apps/qubit/config/settings.yml

• https://www.accesstomemory.org/docs/latest/admin-manual/customization/config-files/#apps-qubit-config-settings-yml  
  

What language are you using in the interface, and in your imports? Is it the same as the default culture, or different? If different, have you added that language to Admin > Settings > i18n languages and reindexed?

Does your repository record have a title added in all the cultures you are using in your site? For example, if you are using both English and Spanish for example, can you flip the interface to each culture and enter edit mode, and see a title in your repository record? Or is it just using culture fallback and displaying the source title for some?

My suspicion is that your installation culture might be set to en, but you are using AtoM with a different culture, or vice versa, and this is putting everything in a null folder because in the installation culture, your repository is untitled. I haven't yet tested to confirm this, but worth checking. 

Anyway, I will check to see if our team have any further suggestions. 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/26ee4988-31f5-4cdd-9fea-f67d307f166b%40googlegroups.com.

[brzic...@gmail.com]

Hi Dan,

I deleted the empty folder at /usr/share/nginx/atom/uploads/r/null/b/8/6/

and ran the symphony command again.

Now I am back to the original memory error.

mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated 12582912) (tried to allocate 9588163448 bytes) in /usr/share/nginx/atom/lib/task/digitalobject/digitalObjectLoadTask.class.php on line 269

Fatal error: Out of memory (allocated 12582912) (tried to allocate 9588163448 bytes) in /usr/share/nginx/atom/lib/task/digitalobject/digitalObjectLoadTask.class.php on line 269

I had previously made these changes to the /etc/php/7.2/cli/php.ini file

memory_limit = -1 (this was the default and no changes were made)

post_max_size = changed from 8M to 72M

upload_max_filesize = changed from 2M to 64 M

As I am not using the browser I left /etc/php/7.2/fpm/php.ini as is..

Please note that running symphony with smaller video sizes works fine...

[brzic...@gmail.com]

Hi Dan,

just to answer your questions: 

What is the default installation culture of your site? You can check this in apps/qubit/config/settings.yml

• default_Culture: en (verified)
  

What language are you using in the interface, and in your imports? Is it the same as the default culture, or different? If different, have you added that language to Admin > Settings > i18n languages and reindexed?

 

Interface uses English. Have French as option but I don't think it was ever used by my archivist. No other languages listed.

Please note that over the years I have gone through three upgrades/migrations - with relevant SQL restores. I remember a long time ago we had more than 2 languages. Maybe that the null directory issue stems from way back - not that it ever created any problems.

Still think this is primarily a php memory issue (see my previous post) …

Thanks for following up.

 

[Dan Gillean]

Hi there, 

Ok, good to know there are not ongoing language issues at the moment at least. Without knowing the full history of your server and AtoM's use, it's hard to say what's happened here - but let's return to the original issue. 

I will ask our team for further suggestions, but I think you may need to make additional adjustments to the php.ini file if the mp4 file is 10GB in size. See for example the following two variables from our documentation: 

post_max_size:
Sets max size of post data allowed. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize. Generally speaking, memory_limit should be larger than post_max_size. When an integer is used without specifying the unit, the value is measured in bytes - shorthand (in bytes [B], kilobytes [K], megabytes [M] or gigabytes [G]) may also be used to specify bigger values.

More info: https://secure.php.net/manual/ini.core.php#ini.post-max-size

upload_max_filesize:
The maximum size of an uploaded file. When an integer is used without specifying the unit, the value is measured in bytes - shorthand (in bytes [B], kilobytes [K], megabytes [M] or gigabytes [G]) may also be used to specify bigger values.

More info: https://secure.php.net/manual/ini.core.php#ini.upload-max-filesize

If your PHP file is setting the upload size limit to 64MB, but you are trying to upload a 10GB file, that may be part of the issue at play here! Note as well what it says under the post_max_size - "To upload large files, this value must be larger than upload_max_filesize."

You will probably also need to adjust the max_execution_limit value as well - a 10GB file is not going to finish uploading in 30 seconds, which is the default. 

Finally, if possible, you might consider locally creating a lower-resolution version, to make upload a bit easier. Another possibility would be splitting the video up into parts, and uploading these parts to different lower-level sibling descriptions, as was done in our demo data here: 

• https://demo.accesstomemory.org/mining-and-smelting-of-canadian-nickel  
  

Anyway, start by trying to make more adjustments in the php.ini file, restarting services and see if that helps. I will see if I can gather any other suggestions. 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/9a5f61ce-b809-4bc9-a179-9b68ebc7d4f5%40googlegroups.com.

[brzic...@gmail.com]

Thanks for that Dan,

just to confirm that I am editing the correct php.ini file: /etc/php/7.2/cli/php.ini

There are additional php.ini files in /etc/php/7.2/fpm but those do not relate to the command-line process right ?

I also presume that I do not need to change anything in /etc/php/7.2/fpm/pool.d/atom.conf ?

Thanks

.

[Dan Gillean]

Hi again, 

That's a good point! I checked with a developer who told me the following: 

"If he followed our install instructions then nginx will use the atom.conf settings for running PHP scripts when fielding web requests. My experience is that the settings in atom.conf will override the others for web requests to AtoM's web controllers."

So from the sounds of it, you will definitely need to modify the atom.conf PHP pool. I think the php.ini file afftects things globally, so if you are running any other PHP applications on the same server, changing things there could also affect the execution limits in other applications. You may need to make the changes in both places! Try it out, and let us know how it goes. 

Cheers,  

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/29e2d35d-8ca8-4a0e-b426-f47a9c19fad3%40googlegroups.com.

[brzic...@gmail.com]

Hi Dan and Happy New Year,

I changed values in the following files:

/etc/php/7.2/cli/php.ini

/etc/php/7.2/fpm/php.ini

/etc/php/7.2/fpm/pool.d/atom.conf

These are the new values I set:

post_max_size=11G

upload_max_filesize=10G

max_execution_limit=300

With a 9.6 GB file still got the memory allocation error. Can you think of another location for php configuration and any other settings ?

Would be curious to find out what is the maximum file size that folks have been able to upload (either via command line or GUI). Anybody ?

Thanks

[Karl Goetz]

On Mon, 6 Jan 2020 13:11:34 -0800 (PST)
brzic...@gmail.com wrote:

> Hi Dan and Happy New Year,
>
> I changed values in the following files:
> /etc/php/7.2/cli/php.ini
> /etc/php/7.2/fpm/php.ini
> /etc/php/7.2/fpm/pool.d/atom.conf
>
> These are the new values I set:
> post_max_size=11G
> upload_max_filesize=10G
> max_execution_limit=300

I had a quick look at digitalObjectLoadTask.class.php and it appears to use https://www.php.net/manual/en/function.file-get-contents.php
for reading the file contents. No maximum filesize is provided but from the manual there isn't a default maximum.

You mentioned the system as 12gb of ram - does that include swap? If 12 is the total (including swap) and you want to upload a 9-10gb file
you could easily be running out of memory.

> With a 9.6 GB file still got the memory allocation error. Can you think of
> another location for php configuration and any other settings ?
>
> Would be curious to find out what is the maximum file size that folks have
> been able to upload (either via command line or GUI). Anybody ?
>
> Thanks
>

For clarity, is it exactly the same error as in your original post?

> * mmap() failed: [12] Cannot allocate memory*
>
>
> *PHP Fatal error: Out of memory (allocated 12582912) (tried to allocate
> 9588163448 bytes) in
> /usr/share/nginx/atom/lib/task/digitalobject/digitalObjectLoadTask.class.php
> on line 269*

JFTR, this error says your file is 8.9gb is size, not the 9.6 you mentioned previously.

:~$ python
Python 2.7.16 (default, Apr 6 2019, 01:42:57)
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 9588163448.0 / 1024 / 1024 / 1024
8.929673068225384
>>> (9588163448.0 + 12582912) / 1024 / 1024 / 1024
8.941391818225384
>>>

thanks,
kk

[brzic...@gmail.com]

OK. Did another test. This time I was monitoring memory usage while attempting to load...

System on idle has 4 GB of memory used (out of 12GB plus 1 GB swap)

When I try to upload a 5 GB file memory jumps up to 9 GB (out of 12GB plus 1 GB swap). No memory allocation problems shown.

From this it is clear that if I try to upload a 10 GB file the system will look for 14GB  (out of 12GB plus 1 GB swap). This therefore must create the memory error. The only fix for this is to increase my system memory.

But the real issue is apparently this:

Checksum values did not validate: /usr/share/nginx/atom/uploads/r/null/b/8/6/…..

Symfony creates a new destination folder but does not put anything in it....

Checksum compares the input file size and the size of the file in the destination folder. Since they are different checksum fails.

So what could be preventing the file copy ?

 

[Karl Goetz]

On Tue, 7 Jan 2020 13:41:58 -0800 (PST)
brzic...@gmail.com wrote:

> OK. Did another test. This time I was monitoring memory usage while
> attempting to load...
>
> System on idle has 4 GB of memory used (out of 12GB plus 1 GB swap)
>
> When I try to upload a 5 GB file memory jumps up to 9 GB (out of 12GB plus
> 1 GB swap). No memory allocation problems shown.
>
> From this it is clear that if I try to upload a 10 GB file the system will
> look for 14GB (out of 12GB plus 1 GB swap). This therefore must create the
> memory error. The only fix for this is to increase my system memory.
>
> But the real issue is apparently this:

> *Checksum values did not validate:
> /usr/share/nginx/atom/uploads/r/null/b/8/6/…..*

> Symfony creates a new destination folder but does not put anything in it....
> Checksum compares the input file size and the size of the file in the
> destination folder. Since they are different checksum fails.
>
> So what could be preventing the file copy ?
>
>
>

If you're interested in reading the code, the relevant code appears to be in lib/model/QubitDigitalObject.php , but I'm not able to say
with any real confidence why you are having this problem - your best bet is to wait for someone from Artefactual to chip in.

thanks,

[José Raddaoui]

Hi there,

Did you experience the checksum validation error with the 5 GB or with the 10 GB file upload? I'd expect that kind of error if you had the memory issue. We've created this ticket to investigate the memory usage on large file uploads.

Best regards.

[brzic...@gmail.com]

With the 10GB file I get the memory allocation problem (as I should) - system needs 14 GB and only has 13 GB available

With the 5GB (where there is sufficient memory allocated) I get the checksum error.

But no checksum errors with smaller files...

[José Raddaoui]

Hi again,

I have been able to reproduce the checksum validation problem and reported another issue. Both issues are closely related as the checksum error comes from loading the entire file on memory. If you have a access to the AtoM code and want to give a try to a work in progress fix you could use this change. It should address the two issues but only on the digital object load task. Please notice that the process still generates a reference video and a thumbnail image from the master video after it's copied, so it will still be a demanding process.

Best regards.