Unable to create SHM segment, mmap failed with errno=13

[Gaurav Behl]

Hi All,

I'm facing this error in one of my env. while other env. works w/o any issue. Hence, I'm leaning towards the way env. configuration is done rather than ModPagespeed configuration. The server is RHEL 7.7 with apache 2.4.x

Could anyone provide insights on when/why this would happen?

Any kind of pointers/help is appreciated.

Log snippet::

[Mon Nov 02 20:26:49.433118 2020] [pagespeed:error] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] Unable to create SHM segment [1]globalstatistics, mmap failed with errno=13.

[Mon Nov 02 20:26:49.433140 2020] [pagespeed:warn] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] Problem during shared memory setup; statistics functionality unavailable.

[Mon Nov 02 20:26:49.433288 2020] [pagespeed:error] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] Unable to create SHM segment [1]SharedCircularBuffer.:0, mmap failed with errno=13.

[Mon Nov 02 20:26:49.433488 2020] [pagespeed:error] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] Unable to create SHM segment [1]/var/cache/mod_pagespeed/metadata_cache, mmap failed with errno=13.

[Mon Nov 02 20:26:49.433502 2020] [pagespeed:error] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] SharedMemCache: can't create segment /var/cache/mod_pagespeed/metadata_cache of size 10764288

[Mon Nov 02 20:26:49.433527 2020] [pagespeed:warn] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] Unable to initialize shared memory cache: /var/cache/mod_pagespeed.

[Mon Nov 02 20:26:49.433580 2020] [pagespeed:error] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] Unable to create SHM segment [1]/var/cache/mod_pagespeed flush /named_locks, mmap failed with errno=13.

[Mon Nov 02 20:26:49.433599 2020] [pagespeed:error] [pid 15679:tid 140552950315136] [mod_pagespeed 1.13.35.2-0 @15679] Unable to create memory segment for locks.

Thanks

[Longinos]

Hi

I made a Google search with "mmap failed with errno 13" and all results are related to file permission....

[Gaurav Behl]

Thanks Longinos, I've confirmed that 

.so file, log folder and cache folder have appropriate RWX permissions & ownership is same user through which httpd is running

Still no luck..

[Longinos]

In the code, the message " nable to create SHM segment %s, mmap failed with errno=%d. " comes from a test to map files to memory.

https://github.com/apache/incubator-pagespeed-mod/blob/d8d13e85f8356e361e9faa4091f305202415f0f8/pagespeed/kernel/thread/pthread_shared_mem.cc#L170

So the issues, in some way, is file related.

[Longinos]

Hi

Have you solved the issue?

Reading the code, seems that the test is done opening /dev/zero as stated here https://github.com/apache/incubator-pagespeed-mod/blob/d8d13e85f8356e361e9faa4091f305202415f0f8/pagespeed/kernel/thread/pthread_shared_mem.cc#L149

Have your system this device?

[Gaurav Behl]

Thanks Longinos, I've asked my infra team partners to validate this one as I don't have access to VM. 
Still working on it, would update the findings.

[Gaurav Behl]

As mentioned on web in couple of places, the issue gets fixed after disabling SELinux - I can confirm this after testing with my managed/hosting services provider. But this is not a solution I'm looking for. However, it confirms that write access to "some file/folder" is the issue as mentioned by Longinos and shared here https://stackoverflow.com/questions/17202741/why-does-mmap-fail-with-permission-denied-for-the-destination-file-of-a-file-c 

"Or MAP_SHARED was requested and PROT_WRITE is set, but fd is not open in read/write (O_RDWR) mode. Or PROT_WRITE is set, but the file is append-only."

which resonates with line #164 of pthread_shared_mem.cc  char* base = reinterpret_cast<char*>(mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0));

A similar issue is still open here - https://github.com/apache/incubator-pagespeed-mod/issues/1936

There is no /dev/zero mount in my setup but yes there is another mount under '/mnt'  on which apache is installed and I'm assuming that's where the issue is.

The other env. where it works fine doesn't have any additional/external mounts and hence no issue in read/write access and pagespeed works perfectly fine.

Since, shared memory is not available, I tried to disable ModPagespeedStatistics & set ModPagespeedDefaultSharedMemoryCacheKB 0 which got rid of couple of errors but I'm not sure if this is a good approach. Would this impact any filters or image processing as I don't see my image filters being applied with this config.

Is there any clean solution to proceed forward in current setup? Is it even worth to set up a PROD solution by disabling shared memory somehow via configs or otherwise is it better to discard this module and look for other technologies? I'm trying to understand the value-add of half baked features due to underlying constraints. Can someone share insights pls.?

Thanks

[Longinos]

Hi

As far as I see in the code /dev/zero is nedeed. Even in the line #164 from pthread_shared_mem.cc there is a fd, if you look at line #153 there is int fd = open("/dev/zero", O_RDWR);

The /dev/zero is a special device that fill a file whit null char and is used to create the shm. Here is a way to recreate the file.

Maybe you can recreate the device and try if it works.

I use in my own config DefaultSharedMemoryCacheKB= 0 and no CreateSharedMemoryMetadataCache config at all , and have no errors, but then you need a external cache, like redis or memcached. Whit this config you can share all the caches between various servers, but introduces some plus RTT and some  latency times.

This makes that all pagespeed caches (metadata cache and resources cache) are stored in the external cache, with this config the file cache still must be set in place because pagespeed refuses to start if not, it need to store allmost 1 file ( !clean!time!) in it.

For Centos exists a rpm wiht selinux config for nginx pagespeed, maybe you take a look at their files and serve as guide to config the selinux for apache.

Here you can read about selinux and nginx, pagespeed. I have not found any similar post for apache, but may help.