More WooCommerce fun with Aoache 2.4 (this time with PHP 7)

[Lewis G Rosenthal]

Hey, Paul...

Although I haven't confirmed the same error under PHP 8.1, I suspect the
issue is the same.

WooCommerce 4.5.5 (last released 4.5 build) does not produce the error,
however, upgrading to 4.6.0 produces an immediate failure, which is logged:

Fatal error: Uncaught IntlException: Collator::__construct():
collator_create: unable to open ICU collator in <blah>

I'm not sure what's missing, though various web sources point to having to
reinstall the icu4c package (I assume that icu and libicu which we have are
icu4c, as we do not have specific icu4c packages available). I am loading
the intl.dll PHP module, as well.

The failing code seems to stem from includes/class-wc-core-functions.php
near line 1782, specifically:

/**
* Sort array according to current locale rules and maintaining index
association.
* By default tries to use Collator from PHP Internationalization Functions
if available.
* If PHP Collator class doesn't exists it fallback to removing accepts
from a array
* and by sorting with `uasort( $data, 'strcmp' )` giving support for ASCII
values.
*
* @since 4.6.0
* @param array $data List of values to sort.
* @param string $locale Locale.
* @return array
*/
function wc_asort_by_locale( &$data, $locale = '' ) {
// Use Collator if PHP Internationalization Functions (php-intl) is
available.
if ( class_exists( 'Collator' ) ) {
$locale = $locale ? $locale : get_locale();
$collator = new Collator( $locale );
$collator->asort( $data, Collator::SORT_STRING );
return $data;
}

$raw_data = $data;

array_walk(
$data,
function ( &$value ) {
$value = remove_accents( html_entity_decode( $value
) );
}
);

uasort( $data, 'strcmp' );

foreach ( $data as $key => $val ) {
$data[ $key ] = $raw_data[ $key ];
}

return $data;
}

I can confirm that *not* loading the intl.dll module allows the code to fall
back and render pages properly. I'm just not sure if I can safely disable
intl functions without breaking something else.

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS, EA
Rosenthal & Rosenthal, LLC www.2rosenthals.com
visit my IT blog www.2rosenthals.net/wordpress
-------------------------------------------------------------

[Paul Smedley]

Hey Lewis!

Please try https://smedley.id.au/tmp/php-7.4.32-os2-intl-debug-20221210.zip

This uses the bww versions of icu*.dll rather than mu own self-compiled
static libs. It also bumps icu to a newer version.

Not saying it will help, but it might.

A nice little distraction before a beach walk on a 35C day (before
dropping back to 23C tomorrow).

This afternoon will likely be spent under the airconditioning, manually
applying misapplied patches to qt6-webengine :)

Cheers,

Paul

[Lewis G Rosenthal]

Thanks, Paul!

This does fix the issue. Currently, I'm up to WooCommerce 5.7.0 on WordPress
5.5. I can't get to 5.8.0, however, as apparently, something for the rest
API in WooCommerce is needed from a newer WordPress, and I have to sort out
why the newer WordPress didn't want to run (perhaps an older plugin, though
I have been careful to upgrade with all plugins disabled).

When time allows, if you can build the intl module for 8.1, I'll give that a
test, as well.

Glad you're feeling more yourself. :-) Enjoy the good weather while it lasts!

Cheers

[Paul Smedley]

https://smedley.id.au/tmp/php-8.1.12-os2-intl-debug-20221210.zip

[Lewis G Rosenthal]

Thanks. This one fixes the intl issue for PHP 8. Of course, there are other
plugin compatibility issues with PHP 8, but these may not be insurmountable.
We'll see. ;-)

[Lewis G Rosenthal]

So far, I am up to WordPress 6.1.1 (latest release version as of today) +
WooCommerce 6.4.0. Applying 6.5.0 and requesting any page (admin or front
end) gets me a crash (SYS3171) in PHP 7.4.32's opcache.dll, which takes
Apache down (I have not tested this particular issue under PHP 8.1.12, yet).
Here is the data from POPUPLOG.OS2 (no .TRP files anywhere to be seen):

12-11-2022 12:19:11 SYS3171 PID 0349 TID 0003 Slot 00c6
J:\APPS\APACHE24\BIN\HTTPD.EXE
c0000005
13f78436
P1=00000002 P2=029cfff4 P3=XXXXXXXX P4=XXXXXXXX
EAX=2167c044 EBX=21dcb018 ECX=00000d04 EDX=21dcb018
ESI=00000d04 EDI=21c7649c
DS=0053 DSACC=d0f3 DSLIM=bfffffff
ES=0053 ESACC=d0f3 ESLIM=bfffffff
FS=150b FSACC=00f3 FSLIM=00000030
GS=0000 GSACC=**** GSLIM=********
CS:EIP=005b:13f78436 CSACC=d0df CSLIM=bfffffff
SS:ESP=0053:029cffcc SSACC=d0f3 SSLIM=bfffffff
EBP=029d0024 FLG=00010216

OPCACHE.DLL 0001:00038436

Possibly related:

https://developer.woocommerce.com/2022/04/04/breaking-change-notice-woocommerce_emogrifier-hook/

I'm not getting anything useful in the PHP error log or in the Apache logs,
though I am still seeing that SYS0003 dbd:error upon Apache startup:

[Sun Dec 11 12:19:14.506000 2022] [mpm_mpmt_os2:notice] [pid 845:tid 1]
AH00206: Apache/2.4.54 (OS/2) OpenSSL/1.1.1q PHP/7.4.32 configured --
resuming normal operations
[Sun Dec 11 12:19:14.815000 2022] [dbd:error] [pid 846:tid 1] (OS 3)SYS0003:
The system cannot find the path specified. : AH00633: failed to initialise
[Sun Dec 11 12:19:14.815000 2022] [dbd:crit] [pid 846:tid 1] (OS 3)SYS0003:
The system cannot find the path specified. : AH00636: child init failed!
[Sun Dec 11 12:19:14.825000 2022] [dbd:error] [pid 847:tid 1] (OS 3)SYS0003:
The system cannot find the path specified. : AH00633: failed to initialise
[Sun Dec 11 12:19:14.825000 2022] [dbd:crit] [pid 847:tid 1] (OS 3)SYS0003:
The system cannot find the path specified. : AH00636: child init failed!

This seems unrelated, however, as I get this all the time, and the database
is (obviously) accessible.

More on what's triggering the opcache.dll crash when I have more time to
test (traveling today), though I have to suspect the replacement of the
emogrifier class, and the new cssinliner class in WooCommerce 6.5.0 (per the
linked notice).

Cheers

[Paul Smedley]

Hey mate,

Are you able to test with 8.1 and specifically the fix from https://mantis.smedley.id.au/view.php?id=773

Cheers,

Paul

---

From: Lewis G Rosenthal <lgros...@2rosenthals.com>
Sent: Monday, 12 December 2022 04:46
To: Apache2 Mailing List
Subject: Re: [Apache/2] More WooCommerce fun with Aoache 2.4 (this time with PHP 7)

--
You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apache2+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/apache2/63961E89.3080100%402rosenthals.com.

[Lewis Rosenthal]

I'll give that a try before I start picking apart this new class.

Thanks for the pointer, Paul.

>>You received this message because you are subscribed to the Google Groups &quot;Apache for OS/2&quot; group.

[Steven Levine]

In <63961E89...@2rosenthals.com>, on 12/11/22
at 01:16 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi guys,

>8.1.12, yet). Here is the data from POPUPLOG.OS2 (no .TRP files anywhere
>to be seen):

All you get with stack overflows is a popuplog. Exceptq reports require a
usable stack.

>12-11-2022 12:19:11 SYS3171 PID 0349 TID 0003 Slot 00c6
>J:\APPS\APACHE24\BIN\HTTPD.EXE
>c0000005
>13f78436
>P1=00000002 P2=029cfff4 P3=XXXXXXXX P4=XXXXXXXX
>EAX=2167c044 EBX=21dcb018 ECX=00000d04 EDX=21dcb018
>ESI=00000d04 EDI=21c7649c
>DS=0053 DSACC=d0f3 DSLIM=bfffffff
>ES=0053 ESACC=d0f3 ESLIM=bfffffff
>FS=150b FSACC=00f3 FSLIM=00000030
>GS=0000 GSACC=**** GSLIM=********
>CS:EIP=005b:13f78436 CSACC=d0df CSLIM=bfffffff
>SS:ESP=0053:029cffcc SSACC=d0f3 SSLIM=bfffffff
>EBP=029d0024 FLG=00010216

>Possibly related:
>https://developer.woocommerce.com/2022/04/04/breaking-change-notice-woocommerce_emogrifier-hook/

Unlikely. This breaking change will get you a method not found failure.

From an lxlite -c: exemap

Stack obj:ESP: 3:007D0000

## Base Size
R W E Res Dis Shr Pre Inv Swp Rsd Loc A16 32B Cnf IOP
3 00030000 007D0000 ¹ ¹ ¹


From the popuplog

P1=00000002 P2=029cfff4 P3=XXXXXXXX P4=XXXXXXXX

and

SS:ESP=0053:029cffcc SSACC=d0f3 SSLIM=bfffffff

The stack size in decimal is

[d:\tmp]hexmath -t 7D0000 = 8192000

The probably could be runaway recursion in opcache or just insufficient
stack for the design.

You can try bumping the stack to 12MB with

exehdr /STACKOS2:0xc00000 httpd.exe

If the trap persists, there is a recursion problem.

Steven

--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------

[Steven Levine]

In <63961E89...@2rosenthals.com>, on 12/11/22
at 01:16 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi,

>I'm not getting anything useful in the PHP error log or in the Apache
>logs, though I am still seeing that SYS0003 dbd:error upon Apache
>startup:

This is an odd one.

>[Sun Dec 11 12:19:14.506000 2022] [mpm_mpmt_os2:notice] [pid 845:tid 1]
>AH00206: Apache/2.4.54 (OS/2) OpenSSL/1.1.1q PHP/7.4.32 configured --
>resuming normal operations
>[Sun Dec 11 12:19:14.815000 2022] [dbd:error] [pid 846:tid 1] (OS 3)SYS0003:
>The system cannot find the path specified. : AH00633: failed to initialise
>[Sun Dec 11 12:19:14.815000 2022] [dbd:crit] [pid 846:tid 1] (OS 3)SYS0003:
>The system cannot find the path specified. : AH00636: child init failed!
>[Sun Dec 11 12:19:14.825000 2022] [dbd:error] [pid 847:tid 1] (OS 3)SYS0003:
>The system cannot find the path specified. : AH00633: failed to initialise
>[Sun Dec 11 12:19:14.825000 2022] [dbd:crit] [pid 847:tid 1] (OS 3)SYS0003:
>The system cannot find the path specified. : AH00636: child init failed!

This is from:

modules\database\mod_dbd.c:646
rv = apr_reslist_create(&group->reslist,
cfg->nmin, cfg->nkeep, cfg->nmax,
apr_time_from_sec(cfg->exptime),
dbd_construct, dbd_destruct, group,
group->pool);
if (rv != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, APLOGNO(00633)
"failed to initialise");
return rv;
}

What is odd is that sys0003 is directory not found and I don't yet see where in the code path this could occur.

[Lewis G Rosenthal]

[It would have been nice had I proofread the subject before starting this
thread. Oh, well.]

On 12/11/22 08:58 pm, Steven Levine wrote:
> In <63961E89...@2rosenthals.com>, on 12/11/22
> at 01:16 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:
>
> Hi guys,
>
>> 8.1.12, yet). Here is the data from POPUPLOG.OS2 (no .TRP files anywhere
>> to be seen):
> All you get with stack overflows is a popuplog. Exceptq reports require a
> usable stack.

Of course. I used to know that. That explains a lot.

>> 12-11-2022 12:19:11 SYS3171 PID 0349 TID 0003 Slot 00c6
>> J:\APPS\APACHE24\BIN\HTTPD.EXE
>> c0000005
>> 13f78436
>> P1=00000002 P2=029cfff4 P3=XXXXXXXX P4=XXXXXXXX
>> EAX=2167c044 EBX=21dcb018 ECX=00000d04 EDX=21dcb018
>> ESI=00000d04 EDI=21c7649c
>> DS=0053 DSACC=d0f3 DSLIM=bfffffff
>> ES=0053 ESACC=d0f3 ESLIM=bfffffff
>> FS=150b FSACC=00f3 FSLIM=00000030
>> GS=0000 GSACC=**** GSLIM=********
>> CS:EIP=005b:13f78436 CSACC=d0df CSLIM=bfffffff
>> SS:ESP=0053:029cffcc SSACC=d0f3 SSLIM=bfffffff
>> EBP=029d0024 FLG=00010216
>> Possibly related:
>> https://developer.woocommerce.com/2022/04/04/breaking-change-notice-woocommerce_emogrifier-hook/
> Unlikely. This breaking change will get you a method not found failure.
>

Better that than taking down the server, obviously. My point was that as
that was entirely new code, I wondered whether it was requesting something
unexpected. Perhaps not, then.

> From an lxlite -c: exemap
>
> Stack obj:ESP: 3:007D0000
>
> ## Base Size
> R W E Res Dis Shr Pre Inv Swp Rsd Loc A16 32B Cnf IOP
> 3 00030000 007D0000 ¹ ¹ ¹
>
>
> From the popuplog
>
> P1=00000002 P2=029cfff4 P3=XXXXXXXX P4=XXXXXXXX
>
> and
>
> SS:ESP=0053:029cffcc SSACC=d0f3 SSLIM=bfffffff
>
> The stack size in decimal is
>
> [d:\tmp]hexmath -t 7D0000 = 8192000
>
> The probably could be runaway recursion in opcache or just insufficient
> stack for the design.
>
> You can try bumping the stack to 12MB with
>
> exehdr /STACKOS2:0xc00000 httpd.exe
>
> If the trap persists, there is a recursion problem.
>

Okay. I'll go back down to the garage and grab that machine from the trunk
(this hotel is HUGE...five city blocks). I'll make the change and will
follow up.

Thanks, Steve!

[Steven Levine]

In <63969A55...@2rosenthals.com>, on 12/11/22
at 10:04 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi,

>[It would have been nice had I proofread the subject before starting this
> thread. Oh, well.]

And here I thought that Aoache stood for Apache with Opcache. :-)

[Lewis G Rosenthal]

Hi...

On 12/11/22 10:16 pm, Steven Levine wrote:
> In <63969A55...@2rosenthals.com>, on 12/11/22
> at 10:04 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:
>
> Hi,
>
>
>> [It would have been nice had I proofread the subject before starting this
>> thread. Oh, well.]
> And here I thought that Aoache stood for Apache with Opcache. :-)
>

;-)

You're too kind, my friend.

[Lewis G Rosenthal]

Hi, again...

After a successful change in stack size, starting httpd gets me a hard hang,
followed by a WPS exit and a blinking cursor. I needed to power cycle to
restart. I couldn't even Ctrl-Alt-Num-Num.

Attempting to set the stack to 10MB gets me:

SYS0189: The operating system cannot run J:\APPS\APACHE24\BIN\HTTPD.EXE.
httpd stopped at 11 Dec 2022 22:47:38
31 +++ Call Syssleep 2;
REX0004: Error 4 running J:\apps\apache24\startup.cmd, line 31: Program
interrupted

I get the same error if I try to bump the stack to anything much above 8MB.
Interestingly, at 8196, I get a hard lockup.

Note that in none of these cases have I opened the browser; this is just
starting the daemon.

PS - I used your patched exehdr.exe which doesn't complain about not being
able to find the message file. :-)

[Paul Smedley]

I can look to compile an executable with a larger stack when I get home in a couple of hours...

Cheers,

Paul

---

From: Lewis G Rosenthal <lgros...@2rosenthals.com>

Sent: Monday, 12 December 2022 14:44
To: apa...@googlegroups.com

Subject: Re: [Apache/2] More WooCommerce fun with Aoache 2.4 (this time with PHP 7)

--
You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to apache2+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/apache2/6396AAA9.6070109%402rosenthals.com.

[Steven Levine]

In <6396AAA9...@2rosenthals.com>, on 12/11/22
at 11:14 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi there,

>>> You can try bumping the stack to 12MB with
>>> exehdr /STACKOS2:0xc00000 httpd.exe
>>> If the trap persists, there is a recursion problem.

>After a successful change in stack size, starting httpd gets me a hard
>hang, followed by a WPS exit and a blinking cursor. I needed to power
>cycle to restart. I couldn't even Ctrl-Alt-Num-Num.

It only looked successful. We will not be suggesting this method of
adjusting the stack size again. Turns out this switch option may never
have worked. Once you know it doesn't work, if you look carefully at the
exehdr output, it's obvious why:

Module: httpx

Module type: Program
Number of memory pages: 00000002 (2)
Initial CS:EIP: object 1 offset 00000000
Initial SS:ESP: object 3 offset 000c0000
Automatic data object: 2
Stack allocation: 000c0000 (786432) bytes

no. virtual virtual map map flags
address size index size
0001 00010000 00000040 00000001 00000001 EXECUTABLE, READABLE, 32-bit 0002
00020000 000000a0 00000002 00000001 READABLE, WRITEABLE, 32-bit 0003
00030000 007d0000 00000003 00000000 READABLE, WRITEABLE, 32-bit

The stack allocation got changed, but the size of object 3, which is where
the stack lives did not. As you found this does not lead to a good
outcome, although the hard hang is a tad unexpected.

Lxlite -c:exemap detects the problem and fails complaining that the
executable format is invalid.

I can probably patch the executable to get the object size in sync with
the stack size, but it's probably better to wait for Paul build a new
httpd.exe.

>PS - I used your patched exehdr.exe which doesn't complain about not
>being able to find the message file. :-)

Are you sure about this? I have no record of ever patching exehdr.exe.
Of course, it's always possible one of the cats ate my notes.

[Paul Smedley]

https://smedley.id.au/tmp/httpd-exe-20221212.zip

[Steven Levine]

In <4066de00-28f6-d8f3...@smedley.id.au>, on 12/12/22
at 05:30 PM, Paul Smedley <pa...@smedley.id.au> said:

Hi,

>https://smedley.id.au/tmp/httpd-exe-20221212.zip

Thanks.

BTW, I figured out why I could not patch the broken 12MB version that
exehdr produced. It was the usual I'm blind. Somehow I patched a byte
that mattered, which of course, should have been left unsullied.

[Lewis G Rosenthal]

Hi...

On 12/12/22 11:53 am, Steven Levine wrote:
> In <4066de00-28f6-d8f3...@smedley.id.au>, on 12/12/22
> at 05:30 PM, Paul Smedley <pa...@smedley.id.au> said:
>
> Hi,
>
>> https://smedley.id.au/tmp/httpd-exe-20221212.zip
> Thanks.
>
> BTW, I figured out why I could not patch the broken 12MB version that
> exehdr produced. It was the usual I'm blind. Somehow I patched a byte
> that mattered, which of course, should have been left unsullied.
>

With the latest stack size change to Apache, I now get:

PHP 7:

12-14-2022 17:53:42 SYS3171 PID 008a TID 0003 Slot 00c2

J:\APPS\APACHE24\BIN\HTTPD.EXE
c0000005

0c898436
P1=00000002 P2=02dffff4 P3=XXXXXXXX P4=XXXXXXXX
EAX=2167a044 EBX=21c2e018 ECX=00000d04 EDX=21c2e018
ESI=00000d04 EDI=21dda49c

DS=0053 DSACC=d0f3 DSLIM=bfffffff
ES=0053 ESACC=d0f3 ESLIM=bfffffff
FS=150b FSACC=00f3 FSLIM=00000030
GS=0000 GSACC=**** GSLIM=********

CS:EIP=005b:0c898436 CSACC=d0df CSLIM=bfffffff
SS:ESP=0053:02dfffcc SSACC=d0f3 SSLIM=bfffffff
EBP=02e00024 FLG=00010216

OPCACHE.DLL 0001:00038436

------------------------------------------------------------

12-14-2022 17:53:46 SYS3171 PID 008b TID 0003 Slot 00c5

J:\APPS\APACHE24\BIN\HTTPD.EXE
c0000005

0c898436
P1=00000002 P2=02dffff4 P3=XXXXXXXX P4=XXXXXXXX
EAX=2167a044 EBX=21c2e018 ECX=00000d04 EDX=21c2e018
ESI=00000d04 EDI=21dda49c

DS=0053 DSACC=d0f3 DSLIM=bfffffff
ES=0053 ESACC=d0f3 ESLIM=bfffffff
FS=150b FSACC=00f3 FSLIM=00000030
GS=0000 GSACC=**** GSLIM=********

CS:EIP=005b:0c898436 CSACC=d0df CSLIM=bfffffff
SS:ESP=0053:02dfffcc SSACC=d0f3 SSLIM=bfffffff
EBP=02e00024 FLG=00010216

OPCACHE.DLL 0001:00038436

And with PHP 8:

12-14-2022 16:19:48 SYS3171 PID 007e TID 0003 Slot 00a2

J:\APPS\APACHE24\BIN\HTTPD.EXE
c0000005

0d4868cd
P1=00000002 P2=02dffff8 P3=XXXXXXXX P4=XXXXXXXX
EAX=21c6808c EBX=21928018 ECX=21c4e00c EDX=00000344
ESI=00000d04 EDI=02e1e060

DS=0053 DSACC=d0f3 DSLIM=bfffffff
ES=0053 ESACC=d0f3 ESLIM=bfffffff
FS=150b FSACC=00f3 FSLIM=00000030
GS=0000 GSACC=**** GSLIM=********

CS:EIP=005b:0d4868cd CSACC=d0df CSLIM=bfffffff
SS:ESP=0053:02dfffdc SSACC=d0f3 SSLIM=bfffffff
EBP=02e00034 FLG=00010202

PHP.DLL 0001:004b68cd

[Paul Smedley]

Hey Lewis,

Interesting..... I wonder how hard it would be for me to reproduce this locally?

Cheers,

Paul

---

From: Lewis G Rosenthal <lgros...@2rosenthals.com>

Sent: Thursday, 15 December 2022 09:27

To: apa...@googlegroups.com
Subject: Re: [Apache/2] More WooCommerce fun with Aoache 2.4 (this time with PHP 7)

Hi...

On 12/12/22 11:53 am, Steven Levine wrote:

> In <4066de00-28f6-d8f3-d99...@smedley.id.au>, on 12/12/22

--
You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apache2+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/apache2/639A54CF.8080601%402rosenthals.com.

[Lewis Rosenthal]

Well, I can zip up the web content easily enough, but the db will have to be empty. Let me think on that one. I should be able to dump the schema for you to reload. I don't know how much you'll have to have in it to be able to launch the admin pages.

On December 14, 2022 6:16:03 PM EST, Paul Smedley <pa...@smedley.id.au> wrote:
>Hey Lewis,
>
>Interesting..... I wonder how hard it would be for me to reproduce this locally?
>
>Cheers,
>
>Paul
>________________________________
>From: Lewis G Rosenthal <lgros...@2rosenthals.com>
>Sent: Thursday, 15 December 2022 09:27
>To: apa...@googlegroups.com
>Subject: Re: [Apache/2] More WooCommerce fun with Aoache 2.4 (this time with PHP 7)
>
>>Hi...
>>
>>On 12/12/22 11:53 am, Steven Levine wrote:

>>> In <4066de00-28f6-d8f3...@smedley.id.au>, on 12/12/22

>>You received this message because you are subscribed to the Google Groups &quot;Apache for OS/2&quot; group.

[Steven Levine]

In <639A54CF...@2rosenthals.com>, on 12/14/22
at 05:57 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi guys,

>PHP 7:

>12-14-2022 17:53:42 SYS3171 PID 008a TID 0003 Slot 00c2
>J:\APPS\APACHE24\BIN\HTTPD.EXE
>c0000005
>0c898436
>P1=00000002 P2=02dffff4 P3=XXXXXXXX P4=XXXXXXXX
>EAX=2167a044 EBX=21c2e018 ECX=00000d04 EDX=21c2e018
>ESI=00000d04 EDI=21dda49c
>DS=0053 DSACC=d0f3 DSLIM=bfffffff
>ES=0053 ESACC=d0f3 ESLIM=bfffffff
>FS=150b FSACC=00f3 FSLIM=00000030
>GS=0000 GSACC=**** GSLIM=********
>CS:EIP=005b:0c898436 CSACC=d0df CSLIM=bfffffff
>SS:ESP=0053:02dfffcc SSACC=d0f3 SSLIM=bfffffff
>EBP=02e00024 FLG=00010216

>OPCACHE.DLL 0001:00038436

Still stack overflow. Based on the numbers, I am assuming the low end of
the thread stack is 02e00000. When the function was entered ESP was
02e00024 because ESP gets copied to EBP. When the exception occured
ESP=02dfffcc. The difference is the size of the local variables.

I'm sure a process dump would show a very deep call stack.

My math says the trap is at:

ext\opcache\Optimizer\zend_inference.c:77

To verify this, crank up php under the the debugger and run until
opcache.dll is loaded. Then use Theseus to figure where opcache.dll
object 1 is loading in memory. This may or may not be the same address as
when run via modphp. Add 38436 to this address and you will get the trap
address. Set a breakpoint at the trap address and then view the code at
the breakpoint.

This view code at breakpoint method is a workaround for the fact that
idebug will not let you direcly view code at an arbitrary address.

[Steven Levine]

In <639A54CF...@2rosenthals.com>, on 12/14/22
at 05:57 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi guys,

A small update...

>PHP 7:

>12-14-2022 17:53:42 SYS3171 PID 008a TID 0003 Slot 00c2
>J:\APPS\APACHE24\BIN\HTTPD.EXE
>c0000005
>0c898436
>P1=00000002 P2=02dffff4 P3=XXXXXXXX P4=XXXXXXXX
>EAX=2167a044 EBX=21c2e018 ECX=00000d04 EDX=21c2e018
>ESI=00000d04 EDI=21dda49c
>DS=0053 DSACC=d0f3 DSLIM=bfffffff
>ES=0053 ESACC=d0f3 ESLIM=bfffffff
>FS=150b FSACC=00f3 FSLIM=00000030
>GS=0000 GSACC=**** GSLIM=********
>CS:EIP=005b:0c898436 CSACC=d0df CSLIM=bfffffff
>SS:ESP=0053:02dfffcc SSACC=d0f3 SSLIM=bfffffff
>EBP=02e00024 FLG=00010216

>OPCACHE.DLL 0001:00038436

My math says the trap is at:

ext\opcache\Optimizer\zend_inference.c:77

The debugger modifies this slightly to

ext\opcache\Optimizer\zend_inference.c:173

This is just the optimizer confusing the debugger a bit because

ext\opcache\Optimizer\zend_inference.c:77

defines the macro referenced by the code.

I'd recommend a process dump as the next step. Since the trap is in a
DLL, we need a Extended (X) style dump. It will be helpful if Paul
provides an opcache.map. I have tools that will use a .map file when the
number of symbols exceeds what a .sym file is capable of.

It might be worth some effort to see if you get the php cli to trap the
same way, although this might not be so easy. This would make debugging
simplier.

It's possible to debug this with idebug, but setup will a bit complicated.

Let's see what a process dump reveals.

[Paul Smedley]

Hey guys,

https://smedley.id.au/tmp/opcache.map

Cheers,

Paul

[Lewis G Rosenthal]

Hi...

Thanks for this.

Okay, with this in place, and rigged for an eXtended dump style, I managed
to get what appears to be a valid dump of the opcache.dll crash.

Try:

ftp://ftp.2rosenthals.com/pub/Dump/opcache-stack-overflow-php7-4-32-2022-12-15.7z

which is:

32478414 Dec 15 16:04 opcache-stack-overflow-php7-4-32-2022-12-15.7z

This was triggered by reloading the WP admin page. Not knowing what might be
exposed in that file, I won't leave it on the server too long, but you
should both be able to reach it without me doing anything fancy.

Cheers

[Steven Levine]

In <639B8E0E...@2rosenthals.com>, on 12/15/22
at 04:13 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi all,

>> https://smedley.id.au/tmp/opcache.map

Just to verify. This if for 7.4.32?

>Okay, with this in place, and rigged for an eXtended dump style,

You don't need to .map file in place because it has not effect on
capturing the dump. You need it where pmdf can file it.

>I
>managed to get what appears to be a valid dump of the opcache.dll crash.

>ftp://ftp.2rosenthals.com/pub/Dump/opcache-stack-overflow-php7-4-32-2022-12-15.7z
>which is:
>32478414 Dec 15 16:04 opcache-stack-overflow-php7-4-32-2022-12-15.7z

It's fine. It says we are looping at:

02e00024: Frame 02e00084 Ret zend_ssa_remove_block + b3a

More later after dinner. We need to convert this to a source code
location.

02e00024: Frame 02e00084 Ret zend_ssa_remove_block + b3a

[Paul Smedley]

Hey Steven,

On 16/12/22 12:30, Steven Levine wrote:
> In <639B8E0E...@2rosenthals.com>, on 12/15/22
> at 04:13 PM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:
>
> Hi all,
>
>>> https://smedley.id.au/tmp/opcache.map
>
> Just to verify. This if for 7.4.32?

Correct - that seemed to be the version Lewis was working on, and was
the version of the sigsegv that you snipped :)

Cheers,

Paul

[Steven Levine]

In <d008f282-070f-943d...@smedley.id.au>, on 12/16/22

at 02:13 PM, Paul Smedley <pa...@smedley.id.au> said:

Hi,

>>>> https://smedley.id.au/tmp/opcache.map
>> Just to verify. This if for 7.4.32?
>Correct - that seemed to be the version Lewis was working on, and was
>the version of the sigsegv that you snipped :)

I was pretty sure it was a match. The timestamp of the map was a couple
of days older than the binaries, so I figured I should check. It seems to
match up OK in pmdf.

The trap is in the optimizer

zend_optimize_script ->
zend_dfa_analyze_op_array ->
zend_ssa_find_sccs ->
zend_ssa_check_scc_var ->
trap

The optimizer part of opcache is compiling and optimizing the php code to
bytecode before stashing the bytecode in the cache. The code reminds me
of how much graph theory I no longer know. :-)

What the function labels say is the optimizer is scanning the parse tree
(DFA) to find the the strongly connected components (SCCs).In this case
the components are variables defined in the php code. The results of this
scan will allow variables to be dropped and/or merged where possible.
zend_ssa_check_scc_var is recursive and in this case it recurses until the
stack overflows. It is too soon to know if there is a problem in the
parse tree or the code that is scanning the parse tree.

Lewis, I recommend you experiment with opcache settings and see if we can
find one the prevents the recursion. See

https://www.php.net/manual/en/opcache.configuration.php

At this point, I am going to assume we have a complexity problem of some
sort and that opcache is working for less complex pages. Is this a
reasonable assumption.

You might try blacklisting the failing script and see what others fail.

[Lewis G Rosenthal]

Hi...

Okay, so easy things to test:

opcache.enable = 0

I can confirm that I am able to load the admin page (albeit slowly) with the
opcache disabled (no Apache crashes).

Enabling opcache logging, and increasing verbosity to 4 (and re-enabling
opcache, obviously), doesn't seem to get me anything specific up to the
point of the crash.

Creating a blacklist, and blacklisting plugins/woocommerce/ (all files under
the woocommerce plugin tree) works, and avoids the crash, but hasn't yet
narrowed the search. So... I'm working on narrowing things down.

More as the operation progresses.

[Lewis G Rosenthal]

Okay, so the two failing scripts are:

woocommerce/vendor/composer/jetpack_autoload_classmap.php
woocommerce/vendor/composer/jetpack_autoload_filemap.php

With these two blacklisted, the crash does not occur (they both need to be
blacklisted, however, e.g.: [...]/vendor/composer/j).

Now to determine what changed between 6.4.0 and 6.5.0...

The answer to that is "a lot." The files are radically different between the
two versions. Both files are automatically generated by
automattic/jetpack-autoloader (this is also under the vendor subdirectory).

Picking these files apart to figure out what is now killing the opcache is a
tedious proposition. If it's just a matter of blacklisting these two
scripts, then no big deal, but I think we should try to determine why these
appear to cause us difficulty on OS/2 as opposed to other platforms.

[Steven Levine]

In <639C9C2F...@2rosenthals.com>, on 12/16/22
at 11:26 AM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:

Hi,

>Okay, so the two failing scripts are:

>woocommerce/vendor/composer/jetpack_autoload_classmap.php
>woocommerce/vendor/composer/jetpack_autoload_filemap.php

Good progress.

>With these two blacklisted, the crash does not occur (they both need to
>be blacklisted, however, e.g.: [...]/vendor/composer/j).

>Picking these files apart to figure out what is now killing the opcache
>is a tedious proposition.

I would start by creating a testing.ini and enabling opcache for th php
cli and trying

php -c testing.ini -f jetpack_autoload_classmap.php

Configure testing.ini as needed and create a testing.php wrapper, if
jetpack_autoload_classmap.php will not load on its own.

If we can get this setup to trap the same way, running under the debugger
will be much easier. I already have a starter set of breakpoints.

>but I think we should try to determine
>why these appear to cause us difficulty on OS/2 as opposed to other
>platforms.

Definitely. Also, it appears that the optimizer code moved into the core
for 8.x. Since the stack overflow persists, it would appear that part of
the code that is affecting us has not changed. The one benefit of 8.x is
that opcache is more conifigurable.

[Lewis G Rosenthal]

Hi...

On 12/16/22 12:25 pm, Steven Levine wrote:
> In <639C9C2F...@2rosenthals.com>, on 12/16/22
> at 11:26 AM, Lewis G Rosenthal <lgros...@2rosenthals.com> said:
>
> Hi,
>
>> Okay, so the two failing scripts are:
>> woocommerce/vendor/composer/jetpack_autoload_classmap.php
>> woocommerce/vendor/composer/jetpack_autoload_filemap.php
> Good progress.

Up to that point, yes. :-)

>> With these two blacklisted, the crash does not occur (they both need to
>> be blacklisted, however, e.g.: [...]/vendor/composer/j).
>> Picking these files apart to figure out what is now killing the opcache
>> is a tedious proposition.
> I would start by creating a testing.ini and enabling opcache for th php
> cli and trying
>
> php -c testing.ini -f jetpack_autoload_classmap.php
>
> Configure testing.ini as needed and create a testing.php wrapper, if
> jetpack_autoload_classmap.php will not load on its own.
>

The script is a long "return array()" so without the caller, it doesn't tell
us much. IOW, I can run it, but as there's no caller, the return is ignored;
the script simply exists upon encountering the return statement. This is
normal and expected.

I'll have to trace it back.

> If we can get this setup to trap the same way, running under the debugger
> will be much easier. I already have a starter set of breakpoints.

Harder in practice (getting it to trap) than it may appear.

>> but I think we should try to determine
>> why these appear to cause us difficulty on OS/2 as opposed to other
>> platforms.
> Definitely. Also, it appears that the optimizer code moved into the core
> for 8.x. Since the stack overflow persists, it would appear that part of
> the code that is affecting us has not changed. The one benefit of 8.x is
> that opcache is more conifigurable.
>

True.

[Massimo S.]

Il 09/12/2022 22:03, Paul Smedley ha scritto:
> Hey Lewis!
>
> Please try https://smedley.id.au/tmp/php-7.4.32-os2-intl-debug-20221210.zip

Hi Paul,

for now i'm going to stay with 7.4.32 do you suggest me to uprade to this build of INTL dll?

(i run 4WP websites)

thanks

massimo