PHP updates

[Paul Smedley]

Hi All,

Thanks to Steven and dmik, I now have a testfix to libcx, and newer PHP
versions seem to be working OK with apache 2.4.x

Note: I have no plans to make builds of these available for apache 2.2.x
- apache 2.2 was made end of life some years ago, so updates make no sense.

The recommended apache2 2.4.x build at this time is:
https://smedley.id.au/tmp/httpd-2.4.48-os2-20210619-debug.zip

The available test versions of PHP are:
https://smedley.id.au/tmp/php-7.3.25-os2-20210727.zip
https://smedley.id.au/tmp/php-7.4.20-os2-20210807.zip
https://smedley.id.au/tmp/php-8.0.7-os2-20210808.zip

Note: the libcx test fix from
https://github.com/bitwiseworks/libcx/issues/95#issuecomment-892614609
is required for these builds.

I've (minimally) tested wordpress 5.8 and mantis with these builds.

phpmyadmin seems to have some issues, I can login, but there are some
errors I need to work through - to be honest - I'm not sure if these are
configuration related, bugs with the PHP port, or both.

Any and all feedback appreciated.

Cheers,

Paul.

[Paul Smedley]

PHP 8.0.9:
https://smedley.id.au/tmp/php-8.0.9-os2-20210808.zip

[David McKenna]

 Thanks Paul! Version 7.4.20 won't start unless I remark out the Curl extension. POPUPLOG says:

08-08-2021  12:54:50  SYS2070  PID 0081  TID 0001  Slot 00d6
C:\PROGRAMS\APACHE24\BIN\HTTPD.EXE
CURL->CURL._curl_version_info
127


 But it seems to work about as well as 7.3.25. phpMyAdmin seems to work well, but Wordpress has numerous problems - have to reload pages constantly, some don't render at all, can't install plugins, and there seems to be some kind of memory leak going on - eventually my server runs out of memory.

 Version 8.0.9 loads fine, but phpMyAdmin gives all kinds of errors complaining about 'iconv'. Wordpress runs about the same as the php7 versions...

Regards,

[David McKenna]

 Pretty sure the rendering issues with Wordpress are due to our outdated browsers. Went to a windows machine using the latest Chrome and everything seems to render well there. Still can't install plugins or update, and it sucks up a lot of memory on the server...

[Paul Smedley]

Hi David,

That might make sense, I've been doing my testing with Chrome under Linux. I'll try and play around this week with updates/plugins.

Cheers,

Paul

--
You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apache2+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/apache2/00251474-0f2c-40f6-88a2-bcb2e557ff1an%40googlegroups.com.

[Steven Levine]

In <45d33aab-0fd6-6b57...@smedley.id.au>, on 08/08/21
at 09:55 AM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

After a switching to a proper mantis distribution and tweaking php.ini to
better match php 7.x

php -c . -f i:\sla_dev2_browse\mantisbt-2.25.2.git\index.php

gets much further with 7.3.25 and 7.4.20 and would probably run as
expected with a proper mysql setup.

I knew that mysql was deprecated, but did not know that it and mcrypt were
fully obsolete in 7.x.

It will be interesting to see if the libcx defect was causing the wget
timeouts.

Steven

--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------

[Steven Levine]

In <45d33aab-0fd6-6b57...@smedley.id.au>, on 08/08/21
at 09:55 AM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

It appears that the libcx defect may have also been the cause of most, but
not all, of the wget retries and truncated GET requests.

Testing locally

wget http://localhost/big-test-file

where big-test-file is about 30MB usually runs without any retries and
when there are retries I've not yet seen more than one retry per wget
request.

Looking at the iptrace output, it looks like the same early eof as before,
but not as often.

[Paul Smedley]

Hi Steven,

That's great news! 

Cheers,

Paul

--
You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apache2+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/apache2/6111d797.1c69fb81.ce549.a58aSMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[David McKenna]

Hi Paul,

  Just a little update. By making sure ALL DLL's are loaded high in both Apache and PHP, I have been able to run without crashing the server due to running out of memory. Still can't use php8 with phpMyAdmin though - I get dozens of this error when I try:

iconv(): Wrong encoding, conversion from UTF-8 to UTF-16BE is not allowed.

 No trouble there with Wordpress or php7. When I google that error, everything that comes up says this is an indication the php iconv module is not enabled, but it is definitely enabled, because if I do disable it, phpMyAdmin will not work at all and an error message says to enable iconv. So something not right with the iconv module in php8?

 Another thing I notice - when starting Apache, the error log always has this error at the beginning:

[Sat Aug 14 09:57:23.548000 2021] [mpm_mpmt_os2:error] [pid 1392:tid 1] AH00200: DosGetNamedSharedMem returned 0

 Is that anything to be concerned about?

Regards,

[David McKenna]

 Paul,

  Just noticed that the file iconv.dll in php8 is significantly smaller (67,240) than the one in php7 (1,348,917) - maybe a clue?

Regards,

[Paul Smedley]

Hi Dave,

On 15/8/21 1:27 am, David McKenna wrote:
>   Just noticed that the file iconv.dll in php8 is significantly smaller
> (67,240) than the one in php7 (1,348,917) - maybe a clue?

Maybe... but it doesn't explain why phpMyAdmin is working for me :/ I'm
guessing that it's decided to link against the libc iconv rather than
GNU iconv. I'll try and take a look at that later on today.

> On Saturday, August 14, 2021 at 10:54:42 AM UTC-4 David McKenna wrote:
>
> Hi Paul,
>
>   Just a little update. By making sure ALL DLL's are loaded high in
> both Apache and PHP, I have been able to run without crashing the
> server due to running out of memory. Still can't use php8 with
> phpMyAdmin though - I get dozens of this error when I try:
>
> iconv(): Wrong encoding, conversion from UTF-8 to UTF-16BE is not
> allowed.
>
>  No trouble there with Wordpress or php7. When I google that error,
> everything that comes up says this is an indication the php iconv
> module is not enabled, but it is definitely enabled, because if I do
> disable it, phpMyAdmin will not work at all and an error message
> says to enable iconv. So something not right with the iconv module
> in php8?
>
>  Another thing I notice - when starting Apache, the error log
> always has this error at the beginning:
>
> [Sat Aug 14 09:57:23.548000 2021] [mpm_mpmt_os2:error] [pid 1392:tid
> 1] AH00200: DosGetNamedSharedMem returned 0
>
>  Is that anything to be concerned about?

I don't think so - I'll need to check the code.

Cheers,

Paul

[Paul Smedley]

Hi All,

On 15/8/21 8:43 am, Paul Smedley wrote:
>
> On 15/8/21 1:27 am, David McKenna wrote:
>>    Just noticed that the file iconv.dll in php8 is significantly
>> smaller (67,240) than the one in php7 (1,348,917) - maybe a clue?
>
> Maybe... but it doesn't explain why phpMyAdmin is working for me :/ I'm
> guessing that it's decided to link against the libc iconv rather than
> GNU iconv. I'll try and take a look at that later on today.

Confirmed.... I'll investigate why this afternoon/evening.

Cheers,

Paul

[Paul Smedley]

Hi Guys,

I think that https://smedley.id.au/tmp/php-8.0.9-os2-20210815.zip should
fix this.

Cheers,

Paul

[David McKenna]

 Paul,

 Gave this new one a try and it is different, but now I get a whole bunch of new errors in phpMyAdmin (using php8). They are always the same - I'll attach an example...

[Paul Smedley]

Hi David,

Interesting. I did switch to Komh's newer build of libiconv from Hobbes, which allowed php 8's configure to find libiconv. I'll take another look this afternoon when I get home. Out for Sunday lunch right now :)

Cheers,

Paul

--

You received this message because you are subscribed to the Google Groups "Apache for OS/2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apache2+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/apache2/8c92daa2-5d10-4e3d-bd11-2398abd95d86n%40googlegroups.com.

[Steven Levine]

In <8c92daa2-5d10-4e3d...@googlegroups.com>, on 08/14/21
at 06:34 PM, David McKenna <davidmc...@gmail.com> said:

Hi all,

The error report:

>Notice in .\vendor\twig\twig\src\Extension\CoreExtension.php#1004
>iconv(): Unknown error (9)
>Backtrace
>CoreExtension.php#1004: iconv(
>string 'UTF-8',
>string 'UTF-16BE',
>string '?',
>)

Is correct, but misleading. If I guessed right the failure is the return
statement in:

1036 function twig_convert_encoding($string, $to, $from)
1037 {
1038 if (!\function_exists('iconv')) {
1039 throw new RuntimeError('Unable to convert encoding: required
function iconv() does not exist. You should install ext-iconv or
symfony/polyfill-iconv.');
1040 }
1041
1042 return iconv($from, $to, $string);
1043 }

To test iconv outside of the app, you can use:

php-cli -r"echo(iconv('UTF-8', 'UTF-16BE', '?'));"

and

php-cli -r"echo(bin2hex(iconv('UTF-8', 'UTF-16BE', '?')));"

The former outputs

" ?"

without the quotes. The later outputs

003f

which is correct for UFT16.

This works as expected for php 7.4.20 and 5.6.40. However for php 8.0.9,
we get

>php-cli -r"echo(bin2hex(iconv('UTF-8', 'UTF-16BE', '?')));"

Warning: iconv(): Wrong encoding, conversion from "UTF-8" to "UTF-16BE" is
not allowed in Command line code on line 1

The small testcase:

php-cli -r"iconv('UTF-8', 'UTF-16BE', '?');"

should make debugging this one relatively easy.

[Steven Levine]

In <CAEtV2=BN=StRNaVTYqFqZUQjf=4thosHvgHMvo...@mail.gmail.com>,
on 08/15/21
at 12:19 PM, Paul Smedley <pa...@smedley.id.au> said:

Hi,

>which allowed php 8's configure to find libiconv. I'll take another look
>this afternoon when I get home. Out for Sunday lunch right now :)

Enjoy, I am digesting our dinner of sauted shrimp and asparagus.

For some reason:

ext\iconv\iconv.c:442
cd = iconv_open(out_charset, in_charset);

is failing and the PHP_ICONV_ERR_WRONG_CHARSET return is generating the
error message. The line number may be off in the above.

The error is odd to be sure. The 7.4.20 and 8.0.9 code are essentially
the same for the iconv_open call.

[Steven Levine]

In <c086ad78-1638-4ff0...@googlegroups.com>, on 08/14/21

at 07:54 AM, David McKenna <davidmc...@gmail.com> said:

Hi all,

> Another thing I notice - when starting Apache, the error log always has
>this error at the beginning:

>[Sat Aug 14 09:57:23.548000 2021] [mpm_mpmt_os2:error] [pid 1392:tid 1]
>AH00200: DosGetNamedSharedMem returned 0

> Is that anything to be concerned about?

Good catch. It's a porting error. We have

mpm\mpmt_os2\mpmt_os2.c:188
else {
/* Parent process or DosGetNamedSharedMem problem */
int rc;
is_parent_process = TRUE;
// 2013-03-24 SHL
if (rc != ERROR_FILE_NOT_FOUND)
ap_log_error(APLOG_MARK, APLOG_ERR, APR_FROM_OS_ERROR(rc), s,
APLOGNO(00200)

The spurious:

int rc;

is hiding the actual error code which will be ERROR_FILE_NOT_FOUND unless
something bad happened. The int rc was not present in 2.2, but snuck in
sometime while Paul was porting 2.4.

The code reports the unexpected rc value and continues on, so there's no
effect on server operation.

Paul, please make it go away. :-)

[Steven Levine]

In <2cd0b0fb-fbd6-4052...@googlegroups.com>, on 08/14/21

at 08:57 AM, David McKenna <davidmc...@gmail.com> said:

Hi all,

> Just noticed that the file iconv.dll in php8 is significantly smaller
>(67,240) than the one in php7 (1,348,917) - maybe a clue?

It is. The big iconv.dll embeds all the coversion tables. The small one
probably expects to find them in external files or just got built wrong.
The big iconv.dll lists the missing UTF-16BE and many more.

00024AD4
6011987·MACCROATIAN·BIG-FIVE·WINDOWS-936·UTF-16BE·KOI8-U·KOI8-RU·WINDOWS-1250·MS-GREEK·WINDOWS-1253·MACGREEK
00024B40
·CSPC850MULTILINGUAL·ISO_8859-7:2003·JIS0208·IBM367·ISO-2022-JP-1·ISO-2022-JP-MS·JIS_X0212·CSEUCTW·CSSHIFTJI

[Paul Smedley]

Hi Guys,

On 15/8/21 12:13 pm, Steven Levine wrote:
> To test iconv outside of the app, you can use:
>
> php-cli -r"echo(iconv('UTF-8', 'UTF-16BE', '?'));"
>
> and
>
> php-cli -r"echo(bin2hex(iconv('UTF-8', 'UTF-16BE', '?')));"
>
> The former outputs
>
> " ?"
>
> without the quotes. The later outputs
>
> 003f
>
> which is correct for UFT16.
>
> This works as expected for php 7.4.20 and 5.6.40. However for php 8.0.9,
> we get
>
>> php-cli -r"echo(bin2hex(iconv('UTF-8', 'UTF-16BE', '?')));"
>
> Warning: iconv(): Wrong encoding, conversion from "UTF-8" to "UTF-16BE" is
> not allowed in Command line code on line 1
>
> The small testcase:
>
> php-cli -r"iconv('UTF-8', 'UTF-16BE', '?');"
>
> should make debugging this one relatively easy.

Unfortunately I'm struggling to reproduce this:

[U:\DEV\php-8.0.9\php8]php-cli -r"iconv('UTF-8', 'UTF-16BE', '?');"

Give no output at all - no warning either.

Any ideas?

Cheers,

Paul

[Paul Smedley]

Hi Again,

ok:
<?php
error_reporting(E_ALL);
ini_set('display_errors', '1');
echo iconv("ISO-8859-1", "UTF-8", "This is a test.");
?>

Gives me:
[U:\DEV\php-8.0.9\php8]php-cli e:\web\iconv.php

Notice: iconv(): Unknown error (9) in E:\web\iconv.php on line 4

Same test works on linux.

More to come when I get time.

Cheers,

Paul

[David McKenna]

 Hi Steven,

  Thanks for the great explanations - have been reading up on using php from the command line so your example was a great help.

 In regards to the Apache mpmt_os2 error - if I understand correctly, that error should be 'File Not Found'? If so, I bet it is the pid file, because it is never there when apache starts... so really just a warning.

Regards,

[Steven Levine]

In <6f392a76-a0a8-d974...@smedley.id.au>, on 08/15/21
at 05:37 PM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

>[U:\DEV\php-8.0.9\php8]php-cli -r"iconv('UTF-8', 'UTF-16BE', '?');"
>Give no output at all - no warning either.
>Any ideas?

This one is not supposed to give output. It's for use under the debugger.
The returned value goes into the bit bucket. The plan was to set a
breakpoint at iconv_open and see why it failed.

Your command line is fine too, It just does more than needed when running
under the debugger.

However, this wall suggested as before I understood that iconv.dll was
missing the conversion tables, so there's no need for the debugger yet.

[Steven Levine]

In <9d1d4b11-e68a-42bf...@googlegroups.com>, on 08/15/21
at 06:33 AM, David McKenna <davidmc...@gmail.com> said:

Hi David,

> Thanks for the great explanations - have been reading up on using php
>from the command line so your example was a great help.

For complex (i.e. multi-line testcases), I typically just write a snippet
of php and run it with -f. -r is handy for one liners. Be careful with
quotes percent and other characters special to the shell. The OS/2 shells
are not as functional as the Linux shells and different quoting is
required. For example:

php-cli -c . -r"printf('%s', 'hello');"

will fail because the %s gets lost. I run 4OS2, so my workaround is

php-cli -c . `-r"printf('%s', 'hello');"`

which protects the special characters from the shell. For cmd.exe you can
double the %.

The reason for the -c . is so that php will find a localized php.ini in
the per version test directories. If find this makes testing easier.

> In regards to the Apache mpmt_os2 error - if I understand correctly,
>that error should be 'File Not Found'?

Yes, but not for the reason you guessed.

server\mpm\mpmt_os2\mpmt_os2.c:142
rc = DosGetNamedSharedMem((PPVOID)&parent_info, listener_shm_name,
PAG_READ);

is checking if the httpd instance is the parent or a child. The shared
memory object will not exist at this time for the parent. The check is
for ERROR_FILE_NOT_FOUND because we discovered that this is what the API
returns when we added the check back in 2013. My memory is not good
enough to recall exactly why we added the check.

Regarding the return value ERROR_FILE_NOT_FOUND, I often have to remind
folks that the vast majority of the OS/2 docs are over 25 years old and
can and do include errors and omissions.

[Paul Smedley]

Hi All,

On 15/8/21 1:34 pm, Steven Levine wrote:
> In <c086ad78-1638-4ff0...@googlegroups.com>, on 08/14/21
> at 07:54 AM, David McKenna <davidmc...@gmail.com> said:
> The spurious:
>
> int rc;
>
> is hiding the actual error code which will be ERROR_FILE_NOT_FOUND unless
> something bad happened. The int rc was not present in 2.2, but snuck in
> sometime while Paul was porting 2.4.
>
> The code reports the unexpected rc value and continues on, so there's no
> effect on server operation.
>
> Paul, please make it go away. :-)

Fixed locally, and rebuilding now..

Cheers,

Paul

[Paul Smedley]

Hi Guys,

On 16/8/21 12:09 am, Steven Levine wrote:
> In <6f392a76-a0a8-d974...@smedley.id.au>, on 08/15/21
> at 05:37 PM, Paul Smedley <pa...@smedley.id.au> said:
>
> Hi Paul,
>
>> [U:\DEV\php-8.0.9\php8]php-cli -r"iconv('UTF-8', 'UTF-16BE', '?');"
>> Give no output at all - no warning either.
>> Any ideas?
>
> This one is not supposed to give output. It's for use under the debugger.
> The returned value goes into the bit bucket. The plan was to set a
> breakpoint at iconv_open and see why it failed.
>
> Your command line is fine too, It just does more than needed when running
> under the debugger.
>
> However, this wall suggested as before I understood that iconv.dll was
> missing the conversion tables, so there's no need for the debugger yet.

Please try https://smedley.id.au/tmp/php-8.0.9-os2-iconv-20210817.zip

I noticed that iconv.a has libiconv and libiconv_open

PHP uses iconv and iconv_open

I'm not seeing how/where iconv is mapped to libiconv and iconv_open is
mapped to libiconv_open

So... php was configured to use libiconv, but the calls were going to
the libiconv.

hacked around currently with:
#ifdef __OS2__
#define iconv_open libiconv_open
#define iconv libiconv
#endif

and my simple testcase works.

Cheers,

Paul

[Steven Levine]

In <3b3ff9f7-b461-437c...@smedley.id.au>, on 08/17/21

at 07:38 PM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

>Please try https://smedley.id.au/tmp/php-8.0.9-os2-iconv-20210817.zip

This one passes my simplistic testcases. In addition, a diff of the
contents imply it knows how to convert all the charsets the 7.4.20
iconv.dll knows about.

>I noticed that iconv.a has libiconv and libiconv_open
>PHP uses iconv and iconv_open

>I'm not seeing how/where iconv is mapped to libiconv and iconv_open is
>mapped to libiconv_open

>So... php was configured to use libiconv, but the calls were going to
>the libiconv.

>hacked around currently with:
>#ifdef __OS2__
>#define iconv_open libiconv_open
>#define iconv libiconv
>#endif

It appears that KO's iconv.h is supposed to handle this:

iconv.h:69
#ifndef LIBICONV_PLUG
#define iconv_open libiconv_open
#endif

iconv.h:79
#ifndef LIBICONV_PLUG
#define iconv libiconv
#endif

Is it possible you are not picking up KO's iconv.h?

I don't see where LIBICONV_PLUG might be defined.

Sometimes I wonder if Martin's Hobbes reorg is helping all that much.
They messed up the timestamp in

Libiconv 1.16 (2006-12-26)

so it took me minute or two to figure out that the timestamp was borked
and it really was newer than the netlabs-rel build.

[David McKenna]

 Paul,

  Your fixed iconv.dll works like a charm... phpMyAdmin no longer complains. Might be my imagination, but it seems like php8 is snappier than previous versions. Rendering pages in Wordpress and phpMyAdmin seems very fast now...

Regards,

[Paul Smedley]

Hi Dave,

Thanks for confirming. I'll try fix iconv a bit more elegantly today,
then I'll upload a new package including the fix.

I also got cmake configure completed with the mysql 5.7.x source - I
just need to finish applying all the diffs that wouldn't apply cleanly.

It would be nice to get us onto current Apache, MySQL and PHP :)

Cheers,

Paul

> www.scoug.com <http://www.scoug.com> www.arcanoae.com
> <http://www.arcanoae.com> www.warpcave.com <http://www.warpcave.com>
> ----------------------------------------------------------------------

>
> --
> You received this message because you are subscribed to the Google
> Groups "Apache for OS/2" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to apache2+u...@googlegroups.com

> <mailto:apache2+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/apache2/1eea101e-8f67-401a-a35f-a4e6dd62ccd0n%40googlegroups.com
> <https://groups.google.com/d/msgid/apache2/1eea101e-8f67-401a-a35f-a4e6dd62ccd0n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Steven Levine]

In <1eea101e-8f67-401a...@googlegroups.com>, on 08/17/21
at 01:53 PM, David McKenna <davidmc...@gmail.com> said:

Hi,

> Your fixed iconv.dll works like a charm... phpMyAdmin no longer
>complains. Might be my imagination, but it seems like php8 is snappier
>than previous versions. Rendering pages in Wordpress and phpMyAdmin
>seems very fast now...

This might be related to ongoing heap optimization. The php7 heap is
almost a total rewrite of what we had for php5. I suspect php8 got more
of the same.

[Steven Levine]

In <92537076-9177-b147...@smedley.id.au>, on 08/18/21

at 06:55 AM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

>It would be nice to get us onto current Apache, MySQL and PHP :)

We are definitely getting closer. :-)

Steven

--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> Warp/DIY/BlueLion etc.

www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------

[Paul Smedley]

Hi Steven,

Possible - but pre-processor output (https://smedley.id.au/tmp/iconv.pp)
shows I'm using the right one, as does inserting a strategic #error xx

> I don't see where LIBICONV_PLUG might be defined.

Me neither :(

> Sometimes I wonder if Martin's Hobbes reorg is helping all that much.
> They messed up the timestamp in
>
> Libiconv 1.16 (2006-12-26)
>
> so it took me minute or two to figure out that the timestamp was borked
> and it really was newer than the netlabs-rel build.

I was confused too :(

Cheers,

Paul

[Steven Levine]

In <7f557c91-f42a-b005...@smedley.id.au>, on 08/18/21

at 11:56 AM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

>Possible - but pre-processor output (https://smedley.id.au/tmp/iconv.pp)
>shows I'm using the right one, as does inserting a strategic #error xx

Is iconv.pp with or without your hack? I see:

iconv.pp:14183
cd = libiconv_open(out_charset, in_charset);

which looks like what's needed.

>> I don't see where LIBICONV_PLUG might be defined.
>Me neither :(

Unless LIBICONV_PLUG is defined, I would expect KO's iconv.h to do the
right thing.

You can always check what's really defined with

iconv.h:69
#ifdef LIBICONV_PLUG
#error fooey1
#endif
#ifndef LIBICONV_PLUG
# error fooey2
#define iconv_open libiconv_open
#endif

Of course, this is best done with your hack disabled.

>I was confused too :(

I feel better now. :-)

[Paul Smedley]

Hey guys,

On 18/8/21 3:57 pm, Steven Levine wrote:
> In <7f557c91-f42a-b005...@smedley.id.au>, on 08/18/21
> at 11:56 AM, Paul Smedley <pa...@smedley.id.au> said:
>
> Hi Paul,
>
>> Possible - but pre-processor output (https://smedley.id.au/tmp/iconv.pp)
>> shows I'm using the right one, as does inserting a strategic #error xx
>
> Is iconv.pp with or without your hack? I see:
>
> iconv.pp:14183
> cd = libiconv_open(out_charset, in_charset);
>
> which looks like what's needed.
>
>>> I don't see where LIBICONV_PLUG might be defined.
>> Me neither :(
>
> Unless LIBICONV_PLUG is defined, I would expect KO's iconv.h to do the
> right thing.
>
> You can always check what's really defined with
>
> iconv.h:69
> #ifdef LIBICONV_PLUG
> #error fooey1
> #endif
> #ifndef LIBICONV_PLUG
> # error fooey2
> #define iconv_open libiconv_open
> #endif
>
> Of course, this is best done with your hack disabled.

This shows LIBICONV_PLUG isn't defined, but I found that php_config.h
has a #define for ICONV_ALIASED_LIBICONV - this isn't currently being
set to 1; forcing it to 1 fixes things. Off to see what's happening with
configure....

Cheers,

Paul

[Steven Levine]

In <7f557c91-f42a-b005...@smedley.id.au>, on 08/18/21

at 11:56 AM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

You probably already know this, but when hunting for #defines I often find

gcc -E -dM

helpful.

[Paul Smedley]

Full rebuild... https://smedley.id.au/tmp/php-8.0.9-os2-20210819.zip

On 18/8/21 6:23 am, David McKenna wrote:

[David McKenna]

 Thanks Paul!... so far, so good with this new build. One thing I noticed yesterday (not with this build) is after using Wordpress for a long while, shared high memory (according to mem /v) was at only 92M (on a 4GB machine). Not clear to me what is using up memory. If I stop apache all memory is returned, so a restart clears it. Maybe  a settings issue - need to investigate more...

[Steven Levine]

In <28eb8c24-ce2f-48f3...@googlegroups.com>, on 08/19/21
at 03:25 AM, David McKenna <davidmc...@gmail.com> said:

Hi David,

>while, shared high memory (according to mem /v) was at only 92M (on a
>4GB machine). Not clear to me what is using up memory. If I stop apache
>all memory is returned, so a restart clears it. Maybe a settings issue
>- need to investigate more...

This is a known issue and it's not limited to the OS/2 platform. All the
mpm modues support configuration variables to deal with this. For our mpm
module, the usual fix is to restart the worker processes after some number
of requests.

The main source of memory usage is memory fragmentation. The apache heap
is based on pools which miminizes true leaks. Pools help avoid
fragmentation, but do not eliminate it.

That said, if a transaction manages to terminate without releasing its
pool, there will be leaks. If you can identify specfic instances where
this occurs, tickets would be good.

Here's an annotated httpd-mpm.conf from one of my test setups.

extra\httpd-mpm.conf:92
# OS/2 MPM
# StartServers: Number of server processes to maintain
# MinSpareThreads: Minimum number of idle threads per process, #
to handle request spikes
# MaxSpareThreads: Maximum number of idle threads per process #
MaxRequestsPerChild: Maximum number of connections per server process #
MaxConnectionsPerChild: Maximum number of connections per server process
<IfModule mpm_mpmt_os2_module>
# default ap_thread_stack_size is 128 * 1024 (128K) asof 2013-03-09
ThreadStackSize 131024
# DEFAULT_START_DAEMON is 2 as of 2014-12-22 SHL
# 2020-02-12 SHL FIXME debug
# StartServers 2
StartServers 1
# DEFAULT_MIN_SPARE_THREAD is 5 as of 2014-12-22 SHL
# MinSpareThreads 5
MinSpareThreads 1
# DEFAULT_MAX_SPARE_THREAD is 10 as of 2014-12-22 SHL
# MaxSpareThreads 10
MaxSpareThreads 1
# ap_max_requests_per_child initialized to 0
# 2015-01-05 SHL Updated
# MaxRequestsPerChild 1000
# 2019-08-18 SHL Sync with 2.24 renames
# MaxConnectionsPerChild 1000
# 2019-11-06 SHL FIXME to be gone - testing
MaxConnectionsPerChild 5
</IfModule>

This setup is used for debugging, so threads and servers are purposely
limited so that I can debug the worker process with idebug.

A more typical production setup is:

extra\httpd-mpm.conf:100
<IfModule mpm_mpmt_os2_module>
ThreadStackSize 65536
StartServers 2
MinSpareThreads 5
MaxSpareThreads 10
MaxConnectionsPerChild 1000
</IfModule>

To tune these values for your mix, you can use apache's ab.exe. There's
also the siege tool. We used this extensively to stabilize 2.2. Paul's
build is at

http://smedley.id.au/tmp/siege-3.0.9-os2-20141231.zip

The current release is 3.14. The changelog does not show any updates that
we really need.

Steven

--
----------------------------------------------------------------------
"Steven Levine" <ste...@earthlink.net> Warp/DIY/BlueLion etc.

www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------

[David McKenna]

 Hi Steven,

  Thanks for the settings tips and insights into how apache uses memory - good info!

"That said, if a transaction manages to terminate without releasing its
pool, there will be leaks. If you can identify specfic instances where
this occurs, tickets would be good."

 Would I expect a message in the apache error log if this happens? Or do I need to use theseus to catch the leak...

 I'll look at ab and siege over the weekend. Also looking forward to a new build of MySQL :-)

[Steven Levine]

In <7ba781f6-9227-4704...@googlegroups.com>, on 08/19/21

at 01:18 PM, David McKenna <davidmc...@gmail.com> said:

Hi David,

> Would I expect a message in the apache error log if this happens? Or do
>I need to use theseus to catch the leak...

If the leak is bad enough to cause httpd/php to fail, it should get logged
or reported with an exceptq report.

Theseus is useful to watch for trends, but it is unlikely tell you the
source of specific leaks, although there have been exceptions to this
generality. Theseus can sometimes help you know when the leaks occur so
you know where to look in the logs.

My guess is that when we are ready to apply focus to leak redemediation,
we will need to use some of the pool debugging tools that the apache
runtime provides.

>Also looking forward to a
>new build of MySQL :-)

Definitely. One more fun thing to occupy my time. :-)

[Paul Smedley]

Hey Guys,

On 20/8/21 7:03 am, Steven Levine wrote:
> In <7ba781f6-9227-4704...@googlegroups.com>, on 08/19/21
> at 01:18 PM, David McKenna <davidmc...@gmail.com> said:
>> Also looking forward to a
>> new build of MySQL :-)
>
> Definitely. One more fun thing to occupy my time. :-)

I'm hoping to get a build of 5.7.35 soon. I have today off (gotta love a
9 day fortnight at work) so fingers crossed I'll finish patching it and
get a build complete today. Currently getting to 12% built even without
applying all the failed patches :P

Cheers,

Paul

[Paul Smedley]

Hi All,

OK, I'm nearly built.... currently I've stubbed around our lack of the
support for the mysql timer api's:
ie timer

The posix implementation of the mysql timer api is:
https://github.com/mysql/mysql-server/blob/mysql-5.7.35/mysys/posix_timers.c
or for Windows
https://github.com/mysql/mysql-server/blob/mysql-5.7.35/mysys/win_timers.c

posix uses:
- timer_create
- timer_settime
- timer_delete

I don't believe anyone has ported these yet for OS/2.

Cheers,

Paul

[Paul Smedley]

Hi Again,

Not withstanding the above....

A set of diffs are at https://smedley.id.au/tmp/mysql-5.7.35.diff

A zip of binaries is at
https://smedley.id.au/tmp/mysql-5.7.35-os2-20210820.zip

NB: the binaries are completely untested with the exception of mysql.exe

The extent of testing with mysql.exe is to attempt a connection to a
mysql server. There are issues with keyboard handling of passwords -
specify the password on the command line for now. Once connected - a
minimal set of commands seemed to work ok.

Any feedback appreciated - I'll try do some proper testing (including
mysqld) here tomorrow or Sunday. Given the aforementioned issues with
the timer api - issues are expected :)

Cheers,

Paul

[David McKenna]

Hi Paul,

  Thanks for the MySQL build... can't get it to start though, crashes with a trap. I'll attach...

[Paul Smedley]

Hi Dave,

Thanks for the TRP - no surprise this is connected to my 'stubbed' timer
implementation - last two objects in the Call Stack are:

EBP Address Module Obj:Offset Nearest Public Symbol
-------- --------- -------- ------------- -----------------------
Trap -> 00E0B217 MYSQLD 0001:00DFB217 my_os2thread.c#292
_pthread_join + 17 0001:00DFB200 (my_os2thread.obj)

Offset Name Type Hex Value
ÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ ÄÄÄÄÄÄÄÄÄ
8 thread pointer to type 0x25A 0
12 status pointer to void 0

019AF9F8 00E4CCFA MYSQLD 0001:00E3CCFA _my_timer_deinitialize
+ 1A 0001:00E3CCE0 (os2_timers.c.obj)

I'll try and look at this later today. We're off to our holiday house
this morning to do some gardening before guests arrive at 2pm.

Cheers,

Paul

> --
> You received this message because you are subscribed to the Google
> Groups "Apache for OS/2" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to apache2+u...@googlegroups.com
> <mailto:apache2+u...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/apache2/ca77ce2d-0c6e-4580-8e23-d4c16762b4e1n%40googlegroups.com
> <https://groups.google.com/d/msgid/apache2/ca77ce2d-0c6e-4580-8e23-d4c16762b4e1n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[David McKenna]

 Hey Paul,

  Enjoy the garden and guests! Maybe we should start a new thread for this - no longer about PHP updates...

Regards,

[Paul Smedley]

Hi Dave,

Note I did tweak the subject :) We got the gardening done and got home
before the guests arrived :)

I'll try investigate the timer API's tomorrow.

Cheers,

Paul

> <https://groups.google.com/d/msgid/apache2/ca77ce2d-0c6e-4580-8e23-d4c16762b4e1n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/apache2/ca77ce2d-0c6e-4580-8e23-d4c16762b4e1n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Apache for OS/2" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to apache2+u...@googlegroups.com
> <mailto:apache2+u...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/apache2/046cb48c-db51-483d-8cfd-260520a60851n%40googlegroups.com
> <https://groups.google.com/d/msgid/apache2/046cb48c-db51-483d-8cfd-260520a60851n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Paul Smedley]

Hi all,

OK I tweaked my stubs for the timer API's, now I get further...

mysqld.exe: Can't create/write to file 'C\ibiRLugv' (Errcode: 2 - No
such file or directory)
2021-08-21T08:41:25.045000Z 0 [ERROR] InnoDB: Unable to create temporary
file; errno: 2

Pretty obvious that it's related to drive letters and cleaning up paths
- just need to work out how to patch it now.

Cheers,

Paul

[David McKenna]

Hi All,

  It's happened 3 times now while messing with Wordpress from a Windows machine - the apache server computer crashes in AFINETK. The system is currently not set up to do system dumps - is there any utility in trying to set up and capture one for this?

[Steven Levine]

In <70107550-7348-4287...@googlegroups.com>, on 08/21/21
at 12:17 PM, David McKenna <davidmc...@gmail.com> said:

Hi David,

Assuming you are running ArcaOS, see:

https://www.arcanoae.com/wiki/how-to-get-a-trap-dump/

[David McKenna]

Thanks Steven, I'll get it set up, try to capture a dump, then post a ticket at Arcanoae's bug tracker...

Regards,

[Steven Levine]

In <0a0bda1f-2c37-46eb...@googlegroups.com>, on 08/22/21

at 05:36 AM, David McKenna <davidmc...@gmail.com> said:

Hi David,

>Thanks Steven, I'll get it set up, try to capture a dump, then post a
>ticket at Arcanoae's bug tracker...

Mysql is not supported by ArcaNoae. It is supported by Paul, with a bit
of help from tne rest of us. Please post the ticket to Paul's mantis.

https://mantis.smedley.id.au

[David McKenna]

OK... will do, thanks.

[David McKenna]

 Well, I memlimit-ed the server to 1GB and have been running Apache 2.4, php 8.09, and MySQL 5.6 for about a week now, but can't get it to trap in AFINETK again yet - will keep trying. But it is worth noting that since using Steven's suggested httpd-mpm.conf settings, even when memlimited, the server does not exhaust the available memory...

[Steven Levine]

In <bc977238-7b6a-4c29...@googlegroups.com>, on 08/28/21

at 10:06 AM, David McKenna <davidmc...@gmail.com> said:

Hi David,

> Well, I memlimit-ed the server to 1GB and have been running Apache 2.4,
>php 8.09, and MySQL 5.6 for about a week now, but can't get it to trap in
> AFINETK again yet

Have you tried siege yet? It's the best stress test we have once ou have
it set up. If you are going to use it, perhaps Paul can be talked in to
building v3.1.4.

The AFINETK trap is not specifically related to apache. It is most likely
a subtle serialization error in the driver that triggers when the driver
is stressed. If it was not subtle, we would probably already have
isolated it and patched it. :-)

>But it is worth noting that since
>using Steven's suggested httpd-mpm.conf settings, even when memlimited,
>the server does not exhaust the available memory...

The limit does a good job of working around the fragmentation, at very
little cost. It took a bit of work to get it working right. We also had
some issues with graceful shutdown, but I think these have all been
resolved.

BTW, I have not had a chance to get back to working on wget timeout issue.
Can you make this happen? If I wget a sufficiently large file, wget is
going to restart intermittently. What's happening is the httpd thinks
it's done sending the file and it's not. The most recent libcx fixes a
defect that could trigger this behavior. However, I've not verified that
this was the only possible trigger.

Thanks,

[David McKenna]

Hi Steven,

  Thanks for the nudge to try siege, I had been side-tracked trying to get the server to trap in AFINETK, so haven't been testing other things. I finally figured siege out, ran it from my desktop, and in about 30 seconds the server trapped :-) So I have a nice big (7zipped) dump file. I can log a ticket on Paul's site, but not sure how to upload. Advice appreciated...

  I use Peter Moylan's  ftp server for file transfers, so never even thought about using Apache for file transfers. But I'll look into setting it up for such and give wget a shot... what size have you seen failures at?

Regards,

[Paul Smedley]

Hi David,

On 29/8/21 8:03 am, David McKenna wrote:
> Hi Steven,
>
>   Thanks for the nudge to try siege, I had been side-tracked trying to
> get the server to trap in AFINETK, so haven't been testing other things.
> I finally figured siege out, ran it from my desktop, and in about 30
> seconds the server trapped :-) So I have a nice big (7zipped) dump file.
> I can log a ticket on Paul's site, but not sure how to upload. Advice
> appreciated...

I sent you a Dropbox file request link - once uploaded - I can make it
available to Steven if necessary.

Cheers,

Paul

[David McKenna]

Paul,

 Dump file uploaded...

Regards,

[Steven Levine]

In <967457db-92f3-4aa0...@googlegroups.com>, on 08/28/21
at 03:33 PM, David McKenna <davidmc...@gmail.com> said:

Hi guys,

>I finally figured siege out, ran it from my desktop, and in about 30
>seconds the server trapped :-)

In afinetk or somewhere else? :-) If the trap is in afinetk, this should
get a AcraNoae mantis ticket. The

https://www.arcanoae.com/wiki/how-to-upload-a-dump-file/

link in

https://www.arcanoae.com/wiki/how-to-get-a-trap-dump/

explains how to upload the dump file to Arca Noae's FTP site.

>thought about using Apache for file transfers. But I'll look into setting
> it up for such and give wget a shot... what size have you seen failures
>at?

My last testcase was about 30MB. There's nothing to set up. Wget is
going to do your standard http GET to fetch the file.

My test driver is the trivail:

rexxtry `do 50 ; 'wget http://localhost/big-test-file' ; end`

The relevant ticket on Paul's mantis is 738.

[David McKenna]

 Hey Steven,

  How can I tell from the dump what trapped? The server made the dump file, then rebooted. I did not witness the trap on the server, since I was at my desktop at the time. Looked at the dump file, but it is not clear to me what happened. Have to admit that dump files are totally new to me...

  In your 'wget http://localhost/big-test-file' example, where on the server would I put the 'big-test-file' to be wget-ted?

Regards,

[Steven Levine]

In <5eae4233-b6fb-451b...@googlegroups.com>, on 08/28/21
at 07:26 PM, David McKenna <davidmc...@gmail.com> said:

Hi David,

> How can I tell from the dump what trapped?

You need to up pmdf to analyze OS/2 dump files. See section 5 of my
venerable

http://www.warpcave.com/os2diags/ProcDumpRef.txt

>The server made the dump
>file, then rebooted. I did not witness the trap on the server, since I
>was at my desktop at the time.

This is why the dump facility works the way it does. :-)

desktop at the time. Looked at the dump file, but it is not clear to me
what happened. Have to admit that dump files are totally new to me...

>Have to admit that dump files are totally new to me...

Don't worry about that. There's a steep learning curve. I intend to take
a look at the dump file, once it is uploaded to the Arca Noae FTP. Of
course, if you have questions, just ask.

> In your 'wget http://localhost/big-test-file' example, where on the
>server would I put the 'big-test-file' to be wget-ted?

The above says the file is in the document root. It's really no different
than index.html or any other file. I guess I should ask how much do you
know about the Apache server configuration? I don't want to be telling
you stuff you already know.

[wkit...@windstream.net]

On 8/28/21 10:26 PM, David McKenna wrote:
>   In your 'wget http://localhost/big-test-file' example, where on the server
> would I put the 'big-test-file' to be wget-ted?

that's the server's root directory so you'd put the file in its DocumentRoot...

remember, the URL format is protocol://host/directory/tree/path... so if the URL
is http://localhost/foo/bar/some.file and DocumentRoot is defined in the conf
file as

DocumentRoot X:/myOS2Apache/htdocs

then you would physically place some.file in directory
X:/myOS2Apache/htdocs/foo/bar...

--
NOTE: No off-list assistance is given without prior approval.
*Please keep mailing list traffic on the list where it belongs!*

[David McKenna]

 Thanks for the explanation guys... I was thinking that an upload directory would have to be set up in a conf file or something, but apparently not.

[David McKenna]

 Hi Steven,

  Just created a ticket (3177) at arcanoae.com for the AFINETK trap, and uploaded the 7zipped dumpfile..

[Steven Levine]

In <598c97bf-bb67-4ac7...@googlegroups.com>, on 08/29/21

at 06:14 AM, David McKenna <davidmc...@gmail.com> said:

Hi David,

> Thanks for the explanation guys... I was thinking that an upload
>directory would have to be set up in a conf file or something, but
>apparently not.

You could do that, but there's no need in this case.

[wkit...@windstream.net]

On 8/29/21 9:14 AM, David McKenna wrote:
>  Thanks for the explanation guys...

you're welcome...

> I was thinking that an upload directory
> would have to be set up in a conf file or something, but apparently not.

that depends on what you are calling an "upload directory" and it depends on
your usage...

for example, if you are running some perl/php/python based application that lets
people/bots upload files, then it may very well have one or more upload
directories defined in a config file or database... how the upload is
accomplished may be as simple as a http "put"... hopefully that upload directory
is protected and not visible to others or if it is visible, files in it cannot
be downloaded from there by the average random on the 'net...

if you are really old-school and (still) run a text terminal oriented BBS from
the dial-up days before the internet was a thing, then the BBS configuration
will have one or more upload directories... if the BBS is more modern and has a
web front end, it may also allow uploads into defined upload directories... i
still run one and we handle uploads inside telnet/ssh/rlogin pipes as well as
via web and ftp...

speaking of FTP <wink> <wink>

let's also not forget that to some people, copying a file on their local hard
drive from one directory to another is "uploading" to them... i've heard many,
over the years, say they were "uploading a file from their floppy to their HD"
when they meant they were copying it... i gave up trying to correct and educate
them a few decades ago ;)

[Steven Levine]

In <3868a533-7f6f-4ced...@googlegroups.com>, on 08/29/21

at 07:40 AM, David McKenna <davidmc...@gmail.com> said:

Hi David,

> Just created a ticket (3177) at arcanoae.com for the AFINETK trap, and
>uploaded the 7zipped dumpfile..

The dump file loads fine. David Azarewicz is looking at it. Your trap is
not in the same location as the afinetk in ticket 2736, but it might be
occurring for a similar reason.

We shall see.

[David McKenna]

 Hi wkitty42,

  Yeah, upload/download are often confusing - I probably should have said a 'transfer directory', that is: a folder specified for transferring files whether in or out.

Regards,

[David McKenna]

Hi Steven,

 I've tried your wget test script, but get this:

[C:\]rexxtry `do 50 ; 'wget http://localhost/big-test-file' ; end`
  Oooops ! ... try again.     Unmatched "/*" or quote
  rc = 6 ......................................... REXXTRY.CMD on OS/2

No matter, I ran wget http://192.168.21.2/bind.zip (a 50M file) manually about 25 times now, and have not had a failure yet. Not sure how often you were seeing it fail, but it seems good to me. I am using the latest libc and libcx files from the netlabs-exp repository on both machines.

Regards,

[Steven Levine]

In <6b8bc98a-32e9-45c4...@googlegroups.com>, on 08/29/21
at 05:45 PM, David McKenna <davidmc...@gmail.com> said:

Hi,

>[C:\]rexxtry `do 50 ; 'wget http://localhost/big-test-file' ; end`
> Oooops ! ... try again. Unmatched "/*" or quote
> rc = 6 ......................................... REXXTRY.CMD on OS/2

I suspect I neglected to mention the above will only run under 4OS2. In
4OS2, the backquotes act like single quotes in *ix. Turns out in this
case, they were not needed. The following:

rexxtry do 50 ; 'wget http://localhost/big-test-file' ; end

will work for both 4os2 and cmd.exe.

[David McKenna]

Hi Steven,

  Still no luck with this:

[C:\]rexxtry do 50; 'wget http://192.168.21.2/bind.zip'; end

  Oooops ! ... try again.     Unmatched "/*" or quote
  rc = 6 ......................................... REXXTRY.CMD on OS/2

 I tried that and the original version in 4OS/2 as well and get the same error.

[David McKenna]

 Hi Steven,

  Guess I spoke too soon - I did have one attempt at 'wget http://192.168.21.2/bind.zip' fail (twice) during download, but each time wget recovered and managed to continue to the end. I checked the downloaded file and it is good. Anything I can do to help pinpoint the problem?

Regards,

[Steven Levine]

In <b8c9e396-d8a7-4120...@googlegroups.com>, on 08/30/21

at 04:17 AM, David McKenna <davidmc...@gmail.com> said:

Hi David,

> Still no luck with this:

>[C:\]rexxtry do 50; 'wget http://192.168.21.2/bind.zip'; end
> Oooops ! ... try again. Unmatched "/*" or quote
> rc = 6 ......................................... REXXTRY.CMD on OS/2

You are making me think too much. You are running Classic REXX, so this
needs to be:

rexxtry `do 50; 'wget http:/'||'/192.168.21.2/bind.zip'; end`

and you need to run it under 4OS2. Classic REXX does not allow //
anywhere in a command line argument and figuring out how to escape the ||
in CMD.EXE is beyond my current patience level. :-)

[Steven Levine]

In <25e0e132-5ff9-43bd...@googlegroups.com>, on 08/30/21

at 11:37 AM, David McKenna <davidmc...@gmail.com> said:

Hi,

> Guess I spoke too soon - I did have one attempt at 'wget
>http://192.168.21.2/bind.zip' fail (twice) during download, but each time
> wget recovered and managed to continue to the end.

This matches what I see here. wget will recover eventually because the
default retry count is unlimited. If the same error occurs using the
browser, you will end up with a silently truncated file.

>Anything I can do to help pinpoint the
>problem?

You could settle in with the debugger and figure out where httpd or libcx
is going astray. :-)

The most likey place to start is:

server\core.c:4911
if ((status = apr_file_open(&fd, r->filename, APR_READ |
APR_BINARY

where the file gets opened for read. This file handle is inserted in the
brigade at:

server\core.c:4938
e = apr_brigade_insert_file(bb, fd, 0, r->finfo.size,
r->pool);

The file is send down the line for processing at

server\core.c:4955
status = ap_pass_brigade(r->output_filters, bb);

where in the fullness of time the file get read and sent on its way to the
requester.

I'm glad we had this conversion. I notice that that we have:

server\core.c:4940
#if APR_HAS_MMAP
if (d->enable_mmap == ENABLE_MMAP_OFF) {
(void)apr_bucket_file_enable_mmap(e, 0);
}
#endif

You httpd.conf should have

#EnableMMAP off

Please change this to

EnableMMAP=off

and see if this is sufficient to resolve the wget failures.

If you get tired of watching the screen to spot errors, I'm sure I can
come up with some REXX that will pause on error.

Thanks,

[Paul Smedley]

Hi Guys,

On 31/8/21 5:25 am, Steven Levine wrote:
> I'm glad we had this conversion. I notice that that we have:
>
> server\core.c:4940
> #if APR_HAS_MMAP
> if (d->enable_mmap == ENABLE_MMAP_OFF) {
> (void)apr_bucket_file_enable_mmap(e, 0);
> }
> #endif
>
> You httpd.conf should have
>
> #EnableMMAP off
>
> Please change this to
>
> EnableMMAP=off
>
> and see if this is sufficient to resolve the wget failures.

I doubt this is the problem - apr.h has:
#define APR_HAS_MMAP 0

Cheers,

Paul

[David McKenna]

Hi Steven,

 For now, I'll leave the debugger to your expertise :-) Tried your latest script and that worked well, thanks. Using that seems to show failures more often than I saw when running wget manually - about 1 failure in 3 attempts or so, where manual was 1 in about 30.

 Tried adding 'EnableMMAP=off' to httpd.conf, but Apache would complain about a 'syntax error' on that when starting. Changed to 'EnableMMAP off' which worked. Running your script with that enabled, wget would fail just about every time.

 What might be good is if you could change the script to count the number of either failed or succeeded attempts to wget...

Regards,

[Steven Levine]

In <37ffa2e5-f552-3302...@smedley.id.au>, on 08/31/21
at 06:57 PM, Paul Smedley <pa...@smedley.id.au> said:

Hi Paul,

>I doubt this is the problem - apr.h has:
>#define APR_HAS_MMAP 0

Interesting. httpd lied to me. I wonder if this is worth an apache
bugzilla:

httpd -t -L -d.. |& less

tells me:

EnableMMAP (core.c)
Controls whether memory-mapping may be used to read files
Allowed in *.conf anywhere and in .htaccess
when AllowOverride includes FileInfo

Also

>httpd -t -d..
Syntax OK

and

>httpd -d..
Apache/2.4.48 (OS/2) PHP/7.4.20

This gives all the evidence of indicating the command is valid, but

server\core.c:4940
#if APR_HAS_MMAP
if (d->enable_mmap == ENABLE_MMAP_OFF) {
(void)apr_bucket_file_enable_mmap(e, 0);
}
#endif

which is certainly going to make the EnableMMAP setting a nop.

Anyway, this means the short reads are not a mmap issue. It's most like
an internal httpd issue.

Would you consider a patch to wrap

server\core.c:4489
AP_INIT_TAKE1("EnableMMAP", set_enable_mmap, NULL, OR_FILEINFO,
"Controls whether memory-mapping may be used to read files"),

in an #ifdef to avoid this confusion in the future?

The default initial value should be fine:

server\core.c:198
conf->enable_mmap = ENABLE_MMAP_UNSET;

[Lewis G Rosenthal]

Hi...

On 08/31/21 12:42 pm, Steven Levine wrote:
> In <37ffa2e5-f552-3302...@smedley.id.au>, on 08/31/21
> at 06:57 PM, Paul Smedley <pa...@smedley.id.au> said:
>
> Hi Paul,
>
>> I doubt this is the problem - apr.h has:
>> #define APR_HAS_MMAP 0
> Interesting. httpd lied to me. I wonder if this is worth an apache
> bugzilla:
>
> httpd -t -L -d.. |& less
>
> tells me:
>
> EnableMMAP (core.c)
> Controls whether memory-mapping may be used to read files
> Allowed in *.conf anywhere and in .htaccess
> when AllowOverride includes FileInfo
>

And this is directive is documented here:

https://httpd.apache.org/docs/2.4/mod/core.html#enablemmap

This has been around for some time (2.2, at least), though I've never had
occasion to play with it. That said, the description does state:

By default, when the handling of a request requires access to the data
within a file -- for example, when delivering a server-parsed file using
mod_include -- Apache memory-maps the file if the OS supports it.


which led me to believe that we likely ignored it, so as I say, I never
really considered it.

> Also
>
>> httpd -t -d..
> Syntax OK
>
> and
>
>> httpd -d..
> Apache/2.4.48 (OS/2) PHP/7.4.20
>
> This gives all the evidence of indicating the command is valid, but
>
> server\core.c:4940
> #if APR_HAS_MMAP
> if (d->enable_mmap == ENABLE_MMAP_OFF) {
> (void)apr_bucket_file_enable_mmap(e, 0);
> }
> #endif
>
> which is certainly going to make the EnableMMAP setting a nop.
>

That's what I thought.

> Anyway, this means the short reads are not a mmap issue. It's most like
> an internal httpd issue.
>
> Would you consider a patch to wrap
>
> server\core.c:4489
> AP_INIT_TAKE1("EnableMMAP", set_enable_mmap, NULL, OR_FILEINFO,
> "Controls whether memory-mapping may be used to read files"),
>
> in an #ifdef to avoid this confusion in the future?
>
> The default initial value should be fine:
>
> server\core.c:198
> conf->enable_mmap = ENABLE_MMAP_UNSET;
>

--
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS, EA
Rosenthal & Rosenthal, LLC www.2rosenthals.com
visit my IT blog www.2rosenthals.net/wordpress
-------------------------------------------------------------

[Steven Levine]

In <CAEtV2=AO0SNB9a+HwrUCjuMYobbA...@mail.gmail.com>,
on 08/10/21
at 07:56 PM, Paul Smedley <pa...@smedley.id.au> said:

Hi all,

I finally carved out some quality time with debugger to understand what is
causing the wget retries and the cutoff GET responses.

What causes this that the select in apr_socket_sendv at:

network_io\os2\sendrecv.c:138
fds = sock->socketdes;
rv = select(&fds, 0, 1, 0, sock->timeout >= 0 ?
sock->timeout/1000 : -1);

sometimes returns 0 because the write socket is not quite ready to write
even though the prior write returned to the caller. This is somewhat
expected because the socket is running non-blocking and the socket timeout
was set to 0 by ap_core_output_filter at:

server\core_filters.c:452
apr_status_t ap_core_output_filter(ap_filter_t *f, apr_bucket_brigade
*new_bb) server\core_filters.c:501
apr_socket_timeout_get(sock, &sock_timeout);
apr_socket_timeout_set(sock, 0);

This makes sense for the unix semantics of aprsocket_sendv where the stack
implements sock_sendv natively. If the sock_sendv cannot send the entire
vector, it rewrites the vector so that apr_sock_sendv can retry.

Our implementation of apr_sock_sendv differs quite a bit from the unix
implemenatation so the possibility to the select error does not exist.

It appears I can mostly avoid the error by patching the timeout a larger
value, but I'm pretty sure this is the right fix because it still see some
retries. These show up in the logs as byte filter reports when wget
requests the missing data.

What's really needed is retry loop of some sort and to fail the GET
request when the retries fail.

Currently when he select fails, the client thinks that the GET request
completed without error which is not good.

I'm thinking about how best to patch this.