IBM HTTP Server error log messages

1,140 views

Skip to first unread message

Neo Chen

unread,

Apr 2, 2007, 5:28:21 AM4/2/07

to netk...@googlegroups.com

IBM HTTP Server error log messages

Messages and resolutions

Cross-platform messages

Message: [emerg] (9)Bad file number: fcntl: F_SETLKW: Error getting accept lock, exiting! Perhaps you need to use the LockFile directive to place your lock file on a local disk!
The "Bad file number" message indicates that the IBM HTTP Server file mutex descriptor has been closed incorrectly. This has been encountered on Solaris with IBM HTTP Server 1.3 and a third-party module.

Make a change to httpd.conf to work around this problem; add AcceptMutex sysvsem to the bottom of httpd.conf. Also, there are likely changes to the Solaris kernel semaphore tuning. Refer to these notes.
Message: [warn] (22)Invalid argument: setsockopt: (TCP_NODELAY)
[error] (22)Invalid argument: getsockname
[warn] (22)Invalid argument: apr_socket_opt_set: (TCP_NODELAY)
[info] (22)Invalid argument: apr_socket_addr_get(APR_LOCAL)
This message is logged when the client drops the connection right after IBM HTTP Server has started processing it but before IBM HTTP Server has started reading the request. It is to be expected that some subset of clients will drop the connection at unexpected times (this could be a real user at a web browsers pressing the stop button), and this is one of the types of messages that can be logged because of that.

Unfortunately, the error returned from the kernel (EINVAL) has a text description that makes the failure look like a programming error, but that is how these calls are rejected when the TCP connection has already been dropped by the client.

Some load balancers test availability of the server on a regular basis in such a way that IBM HTTP Server often encounters a failure on one of these calls. Current maintenance levels of IBM HTTP Server 1.3 have lowered the severity of these messages and included the client IP address in the log message.
(info) (9) bad file number: core_output_filter: writing data to the network. The significance with this message is the text "bad file number." If it says something like "Broken pipe" or "Connection reset", it is a different issue.
To correct some problems which lead to the "bad file number" message, the customer needs PQ85834 or later for 2.0.42.2, or 2.0.47.
[info] [client 1.2.3.4] (32)Broken pipe: client stopped connection before rflush completed
[info] (32)Broken pipe: core_output_filter: writing data to the network
This message is logged when the client drops the connection while IBM HTTP Server is writing the response. It is to be expected that some subset of clients will drop the connection while IBM HTTP Server is writing to it. The user could have pressed the stop button or closed the browser or there could have been a temporary network problem.

Another normal condition where this may occur is when a .pdf file is requested from a browser with the Adobe plug-in installed. The browser first requests the entire file, then when it sees the type of response the plug-in takes over, cancels the initial request (though IBM HTTP Server may not have written the entire response yet), then re-issues the request for the .pdf file in the form of a range request.

A rare condition where these messages may occur is if there is some application layer issue (e.g., browser incompatibility with application response) which leads to the browser abruptly dropping the connection to IBM HTTP Server when it receives certain response data. In order to diagnose the problem, network traces or mod_net_trace are normally needed so that IBM HTTP Server support can see what was sent to the browser. For SSL connections, mod_net_trace is preferred since it traces the data in unencrypted form.
[debug] D:\Build\WWWIHS1328\apache\ibm\modules\afpa\mod_ibm_afpa.c(595): (2)No such file or directory: afpa_init: hafpa = 0, cfg->init_afpa = 1, cfg->bBaseServerConfig = 1, cfg->afpaEnabled = 0
This is a debug message and is not formatted for customer consumption. The key issue here is the value printed for "cfg->afpaEnabled." If the value is 0, then AFPA is disabled. Otherwise, AFPA is enabled.
/opt/IBMHttpServer/bin/httpd: relocation error: /opt/IBMHttpServer/bin/httpd: undefined symbol: apr_table_compress
This is a symptom of IBM HTTP Server finding the wrong version of the libapr-0.so at runtime. libapr-0.so is provided with the IBM HTTP Server installation in the lib/ subdirectory, and is found by a platform-specific environment variable in bin/envvars.
- Verify that the library path set in bin/envvars includes the libs/ subdirectory if you have altered your installation.
- If you've modified the apachectl script, or are using some other custom method to start IHS, make sure it's sourcing the proper bin/envvars file
/opt/IBMHttpServer/bin/httpd: relocation error:
/opt/IBMHttpServer/bin/httpd: symbol __pthread_atfork, version GLIBC_2.0 not defined in file libpthread.so.0 with link time reference
This is an issue seen with IBM HTTP Server 1.3.28 on RedHat 3 Update 4/x86 and SLES 9/x86 caused by the vendor dropping some programming interfaces which were available in prior releases. IBM HTTP Server will continually try to fork new child processes but each fork will fail with these messages. A local circumvention is to avoid the use of LDAP in /etc/nsswitch.conf. Cumulative e-fix PK16139 and later have changed IBM HTTP Server to avoid the problem.
[notice] caught SIGTERM, shutting down
This message is a normal shutdown message for IBM HTTP Server. If a user or automated task (e.g., cron job) does "apachectl stop" or sends SIGTERM to the IBM HTTP Server parent process, this message will be written.

Note: The SIGTERM "signal" shouldn't be confused with other signals that indicate problems, such as SIGSEGV, SIGBUS, SIGABRT, and SIGILL. SIGTERM is the normal mechanism to tell a program to terminate on Unix, so apachectl sends SIGTERM to the IBM HTTP Server parent process to tell it to go away (and to clean up other IBM HTTP Server processes).
[warn] child process 26354 still did not exit, sending a SIGTERM and/or [error] child process 26354 still did not exit, sending a SIGKILL
A child process is not exiting as quickly as expected during shutdown. During shutdown the parent will tell the child processes to exit. If after a short amount of time a child has still not exited, the parent will send the SIGTERM signal to the child process again and write the message to the log. If after another interval the child process still does not exit, then the parent process forcefully brings down the child by sending the SIGKILL signal to the child process.
[error] could not make child process 11540 exit, attempting to continue anyway
A child process hasn't exited even after sending the kill signal to the process. Web server termination has to continue even though the process hasn't exited, because there is no further action that can be taken automatically.

If the process exits within a few more seconds, the expected cause of the message is high system load at the time of termination, and there is no operational problem.

If the process does not exit on its own within one minute, the expected cause is that a thread in that process is blocked in the operating system kernel. Operating system support may need to investigate if this is a recurring problem. IBM HTTP Server support can investigate if hang documentation is provided. When running the hang collector tool, specify auto for the parent process id and - for the non-SSL port.
[alert] No active workers found... Apache is exiting!
Check for error messages earlier in the error log. One or more of those error messages should explain the problem.
[Wed Sep 10 15:20:03 2003] [crit] the listener thread didn't exit
This is sometimes written when a child process exits. It means that one of the threads in a child process didn't exit in the cleanest possible manner, but it doesn't indicate any operational problem. This is a debug message in current levels of IBM HTTP Server.
[Tue Sep 21 23:55:02 2004] [emerg] (22)Invalid argument: apr_proc_mutex_unlock failed. Attempting to shutdown process gracefully.
This message can be ignored if it occurs at the time of a web server restart (apachectl graceful or apachectl restart). It commonly occurs when the accept mutex type is SysV sem and there are multiple listening sockets.

The log level of this message has been changed to debug if it appears during restart with the following maintenance levels:

Release Fix level

2.0.42.2 PK07831 or later

2.0.47.1 PK07831 or later

6.0 6.0.2.1 or later
[warn] new file descriptor 1025 is too large; you probably need to rebuild Apache with a larger FD_SETSIZE (currently 1024)
To resolve the problem:
- V2.0.42 - upgrade to PQ85834 or later
- V2.0.47 - upgrade to 2.0.47.1 or later
(24)Too many open files: file permissions deny server access: /opt/IBMHttpServer/htdocs/en_US/index.html
The ulimits for number of file descriptors per process could be too low, or there could be a file descriptor leak. Stop IBM HTTP Server, increase the hard and soft limits for number of file descriptors, start IBM HTTP Server again, and see if the problem reappears.
[error] [client 1.2.3.4] (11)Resource temporarily unavailable: couldn't spawn child process: /path/to/cgi
This message occurs when the operation system cannot spawn (fork()) another child process for the web server for a CGI request. The request will fail with a 500 error. The specific cause is usually one of the following:
- limits on processes for the user which started the web server may be exceeded
- system-wide limits on processes may be exceeded
- there may be a system resource shortage
More information may be available in the operating system documentation for the fork() subroutine.

AIX

The AIX documentation for fork() is http://publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/libs/basetrf1/fork.htm. The applicable limit is maxuproc. The following commands show how to display the current value and set it to a new value:
```
# lsattr -El sys0 -a maxuproc
maxuproc 2048 Maximum number of PROCESSES allowed per user True
# chdev -l sys0 -a maxuproc=4096
sys0 changed
```
[Sat Dec 13 11:52:48 2003] [warn] long lost child came home! (pid 11380)
This can occur with piped loggers (e.g., rotatelogs) during a graceful restart. One of these messages may appear for every piped logger.

With levels of IBM HTTP Server 2.0 prior to interim fix PK01070, it can also occur during steady state for httpd processes which begin exiting due to MaxSpareThreads or MaxRequestsPerChild processing, but take a long time to exit.
[alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
This is a tuning issue. The operating system failed a request by the web server to create another thread. The operating system can be tuned to allow more threads, or the web server can be tuned to create fewer threads per process so that it doesn't encounter this issue.

Configration to check:
1. value of ThreadsPerChild in the worker MPM configuration in serverroot/conf/httpd.conf
  It may be too high for current operating system limits on memory or processes or threads. If the operating system limits cannot be changed, reduce the value of ThreadsPerChild and adjust other MPM directives accordingly.
2. AIX: value of LDR_CNTRL variable in serverroot/bin/envvars file
  This controls how much of the address space can be used for heap storage, and heap storage is the main requirement on AIX for being able to create a new thread.
  
  The MAXDATA setting within LDR_CNTRL should be at least 0x60000000 in order to allow a large number of threads. Example:
```
LDR_CNTRL="MAXDATA=0x80000000" ; export LDR_CNTRL ...
```
3. Linux: Adjust stack size via ulimit -s or /etc/security/limits.conf
  For every thread created, Linux allocates the stack size as displayed by ulimit -s in the processes address space. If ThreadsPerChild * (ulimit -s) is greater than 2GB the address space will be exhausted (OS defaults may be as high as 10MB).
  
  If a very large number of threads are required, reduce the stack size for the IBM HTTP Server environment. A stack size of 128K should be sufficient for IBM HTTP Server and the WebSphere plug-in. If third-party modules are used in the configuration, a larger stack may be required.
4. HP-UX: Ensure that the max_thread_proc kernel setting is at least five more than the setting of ThreadsPerChild. max_thread_proc is the maximum number of threads in a process. A related kernel setting is nkthread, the maximum number of threads in the system.
  max_thread_proc and nkthread can be displayed or modified by the sam program. With some levels of HP-UX, changing the value requires a kernel rebuild and a reboot of the system.
  Changing the setting to more than the absolute minimum will allow future web server configuration changes without having to modify this kernel setting. Other applications may benefit as well.
[crit] (17)File exists: unable to create scoreboard "/opt/IBMIHS/logs/xxxx" (name-based shared memory failure)
The xxxxx portion of the message varies based on the value of the ScoreBoardFile directive.

Either there are multiple web server instances configured with the same ScoreBoardFile directive or there is a single instance configured with ScoreBoardFile and it did not shutdown cleanly.

The recommended solution in either case is to remove the ScoreBoardFile directive and restart.

The only situation where ScoreBoardFile is needed is if a third-party application which reads the web server shared memory is in use and that application specifies that the ScoreBoardFile directive is required. That is very rare.
[error] server reached MaxClients setting, consider raising the MaxClients setting
At the time the message was written, there were not any worker threads, or child processes in V1.3, available to process additional requests. I.e., all child processes/threads are already in use handling existing requests. The message is only written once during the life of the server, so the peak condition could have happened more than once.

This is an important message to watch out for when there is a symptom such as a hung server. In some types of hang conditions (e.g., all IBM HTTP Server child process/threads tied up waiting for app server to respond, and app server isn't responding because the servlet processing the request is hung), this "MaxClients" message will be written to the error log.

If there is no other symptom, the customer may wish to increase MaxClients to increase the peak capacity of IBM HTTP Server, but unless any users are having problems with long response times then perhaps nothing can be done. The customer can get a good idea of how many of the IBM HTTP Server child processes/threads are in use at a given time by activating mod_status and checking the server-status page.

mod_mpmstats for IBM HTTP Server V2.0 and above provides better reporting of this condition.
[info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 4 idle, and 24 total children
or
[info] server seems busy, (you may need to increase StartServers, ThreadsPerChild or Min/MaxSpareThreads), spawning 8 children, there are around 20 idle threads, and 5 total children
This message appears when a lot of child processes have to be created at about the same time to handle increased load. If it occurs very rarely, it may be triggered by a an infrequent but drastic increase in load. If it occurs relatively frequently, the process management tuning needs to be changed, as follows:

IBM HTTP Server 1.3

Increase MaxSpareServers to be a larger percentage of MaxClients.

IBM HTTP Server 2.0 and above

Increase MaxSpareThreads to be a larger percentage of MaxClients.
[crit] (67)Address already in use: make_sock: could not bind to address nnn.nnn.nnn.nnn port 80
This can occur at startup or restart if some other process is listening to the specified port and optional address.
[emerg] (69)Network is down: apr_accept: giving up.
This can be reported on some platforms if the network layer or a specific interface is no longer active. IBM HTTP Server will exit if all child processes receive this same error.

This problem has been encountered on AIX when performing a load balancer takeover. An AIX APAR fix is available to disable reporting the action to applications via the ENETDOWN error:
- AIX 5.2: IY85386
- AIX 5.3: IY85206
Refer to the text of the appropriate AIX APAR for a temporary circumvention.
Ouch! malloc failed in malloc_block()
This occurs if a storage allocation fails. The child process then exits.

The storage allocation can fail due to reaching an OS limit for memory, or because a plug-in module is asking for an inappropriate amount of storage.

In later IBM HTTP Server 1.3 (PQ87084 for 1.3.19.6 or 1.3.26.2, or IBM HTTP Server 1.3.28), the number of bytes for the failing allocation is traced to make it easier to decide the condition.
[error] [client 127.0.0.1] request failed: error reading the headers
This message can be triggered by one of the following protocol errors encountered while reading a request from the client:
- the number of request header fields exceeds the server's limit (400 will be logged in access log)
- the length of a request header field exceeds the server's limit (400 will be logged in access log)
- a request header field is missing a colon separator (400 will be logged in access log)
The exact cause is written in the error message sent to the client, but it is not logged anywhere unless the "error-notes" note for the request is logged via mod_log_config. (This would be rather voluminous and is not recommended except for brief testing periods.) The note can be logged by adding %{error-notes}n to your log format string.

Even without logging the error-notes note, you can consult the access log to see the URL of the request which failed and see what error message was returned to the browser. Look for requests which failed with 400 with the same timestamp as the messages in the error log.
[error] SSL0108E: Initialization error, GSK internal error.
If using 2.0.42 on Linux/PPC: This can be caused by GSkit getting installed without the OpenSSL functions. The root cause shows up in an strace of server startup like this:
```
open("/usr/local/ibm/gsk6//icc/icclib/libicclib.so", O_RDONLY) = -1 ENOENT (No such file or directory)
```
To resolve: In the directory where IBM HTTP Server was unpacked there should be files iccrte.tar and osslrte.tar, as well as gskit.sh and the GSkit RPM. Run this to re-install GSkit with the OpenSSL function:
```
# rpm --erase gsk6bas
# export ICC_PKG_DIR=&ltany_non_empty_string>
# ./gskit.sh 1
```
This bypasses a license dialog. If you install IBM HTTP Server normally via the GUI, you'll have a chance to see the license and accept it.

The message can appear on other platforms when there is a problem with the GSKit installation or there is a system library compatibility issue.
[error] SSL0404E: I/O failed, insufficient storage.
IHS 2.0.47.1 requires cumulative fix PK07831 or later to be compatible with GSKit versions 7.0.3.12 and higher.
piped log program '(null)' failed unexpectedly
This message can appear with IBM HTTP Server 2.0 and above at shutdown or restart time when piped log programs such as rotatelogs are configured. It does not result in an operational problem.
[error] [client nnn.nnn.nnn.nnn] (13)Permission denied: access to some url failed because search permissions are missing on a component of the path
To find the exact permission problem, first determine which file should be served for the specified url. As an example, if the url is /, this would usually result in serving file index.html in the IBM HTTP Server DocumentRoot directory. With the following configuration in httpd.conf
```
DocumentRoot /opt/local/HTTPServer/htdocs/en_US
...
User nobody
Group nobody
```
the actual file to be served would be /opt/local/HTTPServer/htdocs/en_US/index.html, and the user/group nobody/nobody must be able to read and search each component of the path up through index.html. So run these commands and verify from the permissions of each component of the path that user/group nobody can read and search that component.
```
ls -ld /
ls -ld /opt
ls -ld /opt/local
ls -ld /opt/local/HTTPServer
ls -ld /opt/local/HTTPServer/htdocs
ls -ld /opt/local/HTTPServer/htdocs/en_US
ls -l /opt/local/HTTPServer/htdocs/en_US/index.html
```
If the directory where you choose to install IBM HTTP Server is not within a directory created by the system, you may need to add read and search permissions to that directory so that the web server user/group can access the files.
[crit] SSL0600S: Unable to connect to session ID cache
or
[crit] (146)Connection refused: SSL0600S: Unable to connect to session ID cache
or
[crit] (13)Permission denied: SSL0600S: Unable to connect to session ID cache
See sidd connect failures.
[warn] SSL0265W: Client did not supply a certificate.
This message is received when SSLClientAuth required is specified, but IBM HTTP Server did not receive a client certificate during the SSL handshake. Additionally a 403 Forbidden error will appear at the browser. If the browser has a certificate installed, verify that the certificate authority (CA) which created the client certificate has a signer certificate installed in IBM HTTP Server's key database (.kdb) file. If not, use iKeyman to import the signer certificate information from the client certificate.
[error] [client nnn.nnn.nnn.nnn] (13)Permission denied: unable to connect to cgi daemon after multiple tries: /path/to/cgi
This message can be received for CGI requests with IBM HTTP Server 2.0 or above on Unix and Linux systems. mod_cgid implements the CGI feature in these levels of IBM HTTP Server. At IBM HTTP Server initialization, mod_cgid creates a Unix socket in the filesystem which is used when executing CGI scripts. The default name for this Unix socket is logs/cgisock within the ServerRoot directory. (Example name: /usr/IBMIHS/logs/cgisock) The name of this Unix socket in the filesystem can be changed with the ScriptSock directive. Example:
```
ScriptSock /tmp/scriptsock
```
There are two requirements for the filesystem path of this Unix socket:
1. If there are multiple instances of IBM HTTP Server running on this machine, the different instances must be configured to use different Unix sockets. When the different instances have the same value for the ServerRoot directive, the ScriptSock directive must be used in each configuration file to specify a unique name. Otherwise, security exposures can occur if different web servers have different credentials, since CGI requests may end up being executed by the wrong IBM HTTP Server instance. With cumulative e-fix PK01070 or later for IBM HTTP Server 2.0.x, mod_cgid will refuse to execute a CGI request if this configuration error exists, since unexpected privileges could be used otherwise.
2. The entire path to the socket must be accessible by the web server user id (e.g., nobody, or whatever is specified by the User and Group directives).
  Assume the following configuration:
```
User nobody
Group nobody
ScriptSock /usr/IBMIHS/logs/scriptsock
```
  If the permission denied error is written to the error log when a CGI request is made, the expected cause is that there is a filesystem permissions issue which prevents the web server user id from accessing the Unix socket. Perform the following checks:
  1. Run "ls -ld /usr" and verify that user/group nobody has permissions to read (r) and search (x) the directory.
  2. Run "ls -ld /usr/IBMIHS" and verify that user/group nobody has permissions to read (r) and search (x) the directory.
  3. Run "ls -ld /usr/IBMIHS/logs" and verify that user/group nobody has permissions to read (r) and search (x) the directory.
  4. Run "ls -l /usr/IBMIHS/logs/scriptsock" and verify that user/group nobody has permissions to read (r) and write (w) the Unix socket file.
  Here is an example where the logs directory has been made readable only by root. In this case, the permission denied error would occur when trying to process a CGI request. The fix would be to use the ScriptSock directive to specify a Unix socket file that the web server user id could access, or to make the logs directory readable by the web server user id.
```
[trawick@b80-2 /]$ ls -ld /usr
drwxr-xr-x  63 bin      bin            1536 Mar  7 16:15 /usr
[trawick@b80-2 /]$ ls -ld /usr/IBMIHS
drwxr-xr-x  22 root     system          512 May 19 2004  /usr/IBMIHS
[trawick@b80-2 /]$ ls -ld /usr/IBMIHS/logs
drwx------   2 root     system          512 May 19 2004  /usr/IBMIHS/logs
```
If the problem persists, send in the following information to IBM support:
1. IBM HTTP Server conf file
2. IBM HTTP Server error log covering a time when CGI requests failed
3. output of ls -ld on every directory component of the path to the Unix socket
4. output of ls -l on the Unix socket itself
[error] [client 127.0.0.1] Invalid method in request unprintable-chars
This problem is almost always caused by an SSL request being handled by a virtual host which is not SSL-enabled. It will be accompanied by a message similar to the following in the access log:
```
127.0.0.1 - - [15/Apr/2005:11:42:51 -0400] "\x80\x8c\x01\x03\x01" 501 310 - -
```
Here are some common points of confusion which can lead to this issue:
1. The client may not have specified the expected server IP address. It may have reached the host using an IP address which is not covered by any of the <VirtualHost> directives.
2. If the <VirtualHost> directive used in httpd.conf specifies a host name instead of an IP address, it may not match all IP addresses for that host name (apply PK01070 or later for IBM HTTP Server 2.0.x to fix that) and/or the web server may get different IP addresses from the DNS for that host name than clients would see.
  One cause for this is a defect in IBM HTTP Server 2.0.x, resolved by PK01070 or later, which respected only the first returned IP address for the host name.
  
  Another cause for this is when the DNS used by IBM HTTP Server returns different information than the DNS used by the client.
To diagnose the configuration problem, make these configuration changes:
1. Specify only IP addresses in the <VirtualHost > directives for SSL-enabled virtual hosts. Do not use host names.
2. Add the target IP address (%A) to the access log format so that the IP address used by the client on the failing request can be seen.
3. Log which vhost was selected.
  1. Add SetEnv vhostname MAIN to the main scope of httpd.conf.
  2. Add SetEnv vhostname UNIQUE-NAME to each VirtualHost container. Make sure UNIQUE-NAME is unique for each virtual host.
  3. Add the vhostname (%{vhostname}e) to the access log format.
Example log format with both of these changes made:
```
LogFormat "%h %l %u %t \"%r\" %>s %b %A %{vhostname}e" common
```
Example setting of vhostname:
```
...
SetEnv vhostname MAIN
...
<VirtualHost 192.168.1.15:8443>
...
SetEnv vhostname vhost8443
</VirtualHost>

<VirtualHost 192.168.1.1:443>
...
SetEnv vhostname vhost443
</VirtualHost>
```
Now restart IBM HTTP Server and try the request again. Check the access log for the destination IP address and the vhost name:
```
127.0.0.1 - - [15/Apr/2005:11:58:24 -0400] "\x80\x8c\x01\x03\x01" 501 310 127.0.0.1 MAIN
```
In this example, the IP address in next to last column is the IP address used by the client. So you have to check if there is an SSL-enabled virtual host which specifies that IP address (or *) on the <VirtualHost > directive.

The last column is the virtual host which was actually chosen. In this example, it is the main scope of httpd.conf, because the IP address used by the client did not match the IP address on any <VirtualHost > directives. Whichever virtual host was chosen, verify that there is no SSLEnable directive for it, and decide if the client sent the request to wrong vhost (due to bad link?) or if the SSLEnable directive is supposed to be specified or if another VirtualHost container is appropriate.
[error] [client nnn.nnn.nnn.nnn] Directory index forbidden by rule: /usr/IBMIHS/docroot/path/
A request has been issued from the browser, and that request maps to an actual directory on the web server machine under DocumentRoot, and there is no active DirectoryIndex directive/document to return a document such as index.html to the browser, so IBM HTTP Server tries to send a directory listing back to the client. However, the Indexes option is not active for the directory so an error is returned.

To resolve the problem, you need to determine what response was supposed to be returned on this request.

If the request is not expected, check for links on your site which specify that URL.

If a directory listing was supposed to be returned, enable the Indexes option for that directory. Example:
```
<Directory /usr/IBMIHS/docroot/path/>
Options +Indexes
</Directory>
```
If a static file such as index.html was supposed to be returned, enable that as the DirectoryIndex for that directory. Example:
```
<Directory /usr/IBMIHS/docroot/path/>
DirectoryIndex index.html
</Directory>
```
If the request is supposed to map to a WebSphere request, check the WebSphere plug-in configuration file and make sure there is a rule to send requests for that URL to WebSphere.
[warn] [client nnn.nnn.nnn.nnn] mod_was_ap20: /some/URL not found
This message is issued by the WebSphere plug-in, and it indicates a plug-in processing error, due either to the plug-in implementation or to the plug-in configuration. A common cause of this error is fixed by APAR PK01215 (WebSphere plug-in). It can affect ErrorDocument processing as well as some other types of IBM HTTP Server processing. A WebSphere plug-in update including that fix needs to be applied before the error can be investigated further.
[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
[emerg] (28)No space left on device: Couldn't create accept lock
Check your system sysv semaphore usage and limits
- Linux: Compare ipcs -l with ipcs -u,
- Solaris: Compare sysdef | grep -i shm with ipcs -s
- HPUX: Compare grep -i SHM /etc/conf/master.d/sysv-sem with ipcs -s
- AIX: (Dynamically adjusted; error should not occur on that platform)
[error] [client aaa.bbb.ccc.ddd] internal error: bad expires code: (null), referer: http://www.example.com/welcome.html
This problem with IBM HTTP Server 2.0.42.x/2.0.47.x is limited to extra error message logging. It is resolved by any maintenance after PQ85834. The customer should apply the current recommended update.
[alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2513
setuid() can return error code 11 when the NPROC rlimit for the user would be exceeded. Edit /etc/security/limits.conf or use ulimit to increase limit on number of processes.
[error] mod_ibm_ldap: unable to authenticate the web server for realm 'Example LDAP Realm': Encoding error.
If this is Solaris, verify that the SUNWuiu8 package is installed:
```
# pkginfo SUNWuiu8
system      SUNWuiu8       Iconv modules for UTF-8 Locale
```
[crit] Unable to set default LDAPCodePages directory relative to ServerRoot.

Release	Fix level
2.0.42.2	PK07831 or later
2.0.47.1	PK07831 or later
6.0	6.0.2.1 or later

If the ServerRoot directive has been changed from the default value, copy (or symlink) the codepages directory from the installation path into the new ServerRoot

[error] SSL0209E: SSL Handshake Failed, ERROR processing cryptography.

If this message appears shortly after an IBM HTTP Server shutdown, the message can be ignored (SSL request using crypto hardware, blocked during IBM HTTP Server shutdown)
This message can also occur when IHS has just created a new child process, under the following circumstances
1. IHS is configured to use a cryptographic accelerator
2. GSkit level is prior to 7.0.3.27
3. A second SSL handshake is started immediately after the start of the first SSL handshake of an IHS child process
If this message appears continuously during normal SSL request processing, verify crypto accelerator configuration is correct. If using pkcs11, one possibility is that the IHS user needs to be added to the pkcs11 group.
Otherwise, contact your cryptographic accelerator vendor to gather a trace of PKCS11 activity to determine the error returned by the accelerator.
- User ihsadmin
- Group pkcs11

[warn] proxy: No protocol handler was valid for the URL /example/url.js. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
The usual cause of this is the lack of a LoadModule directive to load mod_proxy_http.so when supporting HTTP proxy. Uncomment, or add, the following directive:
```
LoadModule proxy_http_module modules/mod_proxy_http.so
```
If the specified URL is handled via FTP, activate mod_proxy_ftp.so instead.

This message can also be seen with some levels of IBM HTTP Server when SSL is used for the proxy connection, and an error occurs setting up the connection. If the necessary proxy modules are already loaded and the proxy connections are https, apply one of the following fixes:
- IBM HTTP Server 2.0.47.1: PK07831 or later
- IBM HTTP Server 6.0: 6.0.2.1 or later
[notice] suEXEC mechanism enabled (wrapper: /usr/IBMIHS/bin/suexec)
This message indicates that suEXEC could be used if configured in httpd.conf. In order to configure it, the SuexecUserGroup directive is coded. See this suexec how-to for more information.

Windows-specific messages

Cannot load IHS_PATH/afpaplugin.dll into server: The specified module could not be found.
This error message may occur when the customer is trying to use IBM HTTP Server with the Websphere Application Server high speed external caching feature and has configured the wrong version of afpaplugin.

The afpaplugin.dll file, which is a component of Webshpere Application Server, is only for use with IBM HTTP Server 1.3.x. The afpaplugin_20.dll file, which is installed via the Websphere 6.x Plugins installation, is required when using IBM HTTP Server 2.0.x and above.

When using IBM HTTP Server 2.0.42 or 2.0.47, cumulative e-fix PQ94086 or later must be applied. With IBM HTTP Server 6.x no additional fix pack is required.
The following error messages indicate that some other vendor's software may be installed which does not correctly implement AcceptEx or other Winsock functions:
- (OS 6)The handle is invalid. : winnt_accept: AcceptEx failed.
- (OS 64)The specified network name is no longer available.: winnt_accept: Asynchronous AcceptEx failed
- (OS 121)The semaphore timeout period has expired. : winnt_accept: Asynchronous AcceptEx failed.
- (OS 10106)The requested service provider could not be loaded or initialized.
- (OS 10038)An operation was attempted on something that is not a socket.
Other symptoms that might be experienced are slow IHS response time or the appearance that IHS has hung.

Try adding Win32DisableAcceptEx to the IHS configuration file and restart IHS.

Note: AFPA must be disabled before adding the Win32DisableAcceptEx directive. See technote Disabling AFPA (fast cache accelerator) in IBM HTTP Server for details of how to disable AFPA.

z/OS-specific messages

mod_auth_saf

[error] [client ip-address] 
  (errno)error-string (errno2=errno2): SAF user 
  userid: authentication failure for "request-uri": Password 
  Mismatch

The server logs this message when an invalid password is specified. Common error conditions are:

errno value	errno2 value	meaning
111	0xnnnn0000	The password is of a valid length but is not valid.
121	0xnnnn02A7 (JRPasswordLenError)	SAF reports that the password has an invalid length. (mod_auth_saf did not make this determination.)

[error] [client ip-address] 
  (errno)error-string (errno2=errno2): SAF user 
  userid not found: request-uri

The server logs this message when an invalid user id is specified. Common error conditions are:

errno value	errno2 value	meaning
143	0xnnnn05DD	The user id is of a valid length but is not defined to SAF.
121	0xnnnn02A6 (JRUserNameLenError)	SAF reports that the user id has an invalid length. (mod_auth_saf did not make this determination.)

[debug] (168)user [username]: SAF authentication failure 401-168 for [request-uri]: password has expired

The server logs this message when a user has entered a correct SAF password, but the password has expired. A call to the function __passwd() returned EMVSEXPIRE. Access is denied.

To enable updating of expired SAF passwords using the server, code the directive "AuthSAFExpiration". For example:
```
    AuthSAFExpiration on
```
You may also tailor the message to be presented at the client by coding some brief text instead of "on". For example:
```
    AuthSAFExpiration "EXPIRED PW: oldpw/newpw/newpw"
```
You can also use the optional AuthSAFReEnter directive to further tailor the next password message issued at the client. For example:
```
    AuthSAFReEnter "New PW again:"
```
[crit] [client ip-address] (157)EDC5157I An internal error has occurred. (errno2=0x090C02AF): user userid: SAF authentication failure 500 for request-uri: function is not supported

The server logs this message when a call to the function __passwd() returns EMVSERR.

Check the errnoJr value for the exact cause. The most common cause, as seen in the example above where errno2 is 02AF, is that you have not marked for program control one or more modules that the server loads. You can use the command "extattr +ap [modulename]" to mark a module program controlled.
[info] [client ip-address] (errno-value)errno-description (errno2=errno2-value) user userid: saf_check_pw=401 for request-uri: Other SAF failure; errno2=yyyyzzzz

The server logs this message when a call to the function __passwd() returns an error which the server does handles with specific messages. The combination of errno and errno2 explain the problem.

UNIX System Services Messages and Codes describes all errno and errno2 values. C/C++ Run-Time Library Reference contains documentation for the __passwd() function.
- z/OS V1R6.0 UNIX System Services Messages and Codes
- z/OS V1R6.0 C/C++ Run-Time Library Reference
[error] [client ip-address] access to [request-uri] failed: [requirement]
whatever
The server logs this message when it denies access to a resource based on the options in a "Require" directive in httpd.conf.

If access should have been granted, review the options in the "Require" directive.
[info] [client ip-address] access to [request-uri] failed; reason: user [username] is not in the "Require" list

The server logs this message when it denies access to a resource because a "Require" directive in httpd.conf does not specifically include the username or a group containing the username.

If access should have been granted, review the options in the "Require" directive.
[info] (xxx)rc=[return code] attempting to reset UID

The server logs this message when it has run a request under SAFRunAs control and is attempting to delete the SAF security environment and return to the server's default user ID (UID).

See the documentation for errno xxx and the pthread_security_np() function in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.

mod_ldap and mod_auth_ldap

[warn] [client 9.37.242.143] [16777781] auth_ldap authenticate: user [xxxx] authentication failed; URI /ldapdir/index.html [User not found][No such object]
The server logs this message if auth_ldap cannot find the requested dn or user, but mod_ldap is not authoritative.

The following messages will also be logged if no module is authoritative and the user is not found by any authorization module:
```
[crit] [client 9.37.242.143] check_user_id hook called
[crit] [client 9.37.242.143] configuration error: couldn't check user. No user file?: /ldapdir/index.html
```
See http://publib.boulder.ibm.com/httpserv/manual60/mod/mod_auth_ldap.html#authldapauthoritative. Each authorization module only authenticates based on its own knowledge. If directive AuthLDAPAuthoritative off is specified, mod_auth_ldap will not reject the request, but will allow it to be passed to other possible authorization modules. Normally, mod_auth_ldap is the only authorization module configured for a request and AuthLDAPAuthoritative is set to on.
[warn] [client 9.76.147.159] [43] auth_ldap authenticate: user [username] authentication failed; URI [path/to/ldap/directory] [LDAP: ssl connections not supported][N/A]
The server logs this message if an LDAP session was requested but failed.

If an SSL connection to the LDAP server is required, the url specified in AuthLDAPURL must begin with ldaps://, non-SSL must begin with ldap://. Additional messages which describe the problem may have appeared earlier in the error log.
[error] (121)EDC5121I Invalid argument. (errno2=0x07200316): LDAP cache: error while creating a shared memory segment: EDC5121I Invalid argument. (errno2=0x07200316)
[error] (153)EDC5153I Catalog obtain error. (errno2=0xC5C7082A): LDAP cache: error while creating a shared memory segment: EDC5153I Catalog obtain error. (errno2=0xC5C7082A)
Ensure that the size specified by LDAPSharedCacheSize is valid.

Ensure that the file specified in LDAPSharedCacheFile is valid.

Delete the file specified in LDAPSharedCacheFile and try starting IBM HTTP Server again.

If a LDAPSharedCacheFile directive is specified remove it; if it is not specified then add it. If the directive is specified, named shared memory will be used; otherwise, anonymous shared memory will be used.
[crit] [return code] LDAP cache: failed to set mutex permissions
The server may log this message if the User or Group directive in httpd.conf does not have appropriate permissions.

See the documentation for semctl() IPC_SET operation in the z/OS C/C++ Run-Time Library Reference, document number SA22-7821.
[crit] LDAP: Invalid LDAPTrustedCAType directive - KDB_FILE or SAF_KEYRING type required or mismatched
The server logs this message if the KDB file, SAF keyring, or password file are invalid.
- KDB file requirements: The file extension must be .kdb and the associated associated password file must be have the same name as the kdb file, but with a file extension of .sth.
- SAF keyring requirements: The name must be a maximum of 8 characters and meet all RACF naming conventions. There is no password file.
[warn] LDAP: SSL initialization failed
[notice] LDAP: SSL support unavailable
The server logs one or both of these messages if SSL was not requested, or fails. If SSL support is required, check for additional messages in the log and directives such as LDAPTrustedCA and LDAPTrustedCAType.

Messages from third-party modules

Contact the supplier of any third-party modules in the configuration in order to address these messages.

SemCounter ERROR: Cannot remove semaphore: 655433
[notice] msgsnd() FAILED can't send, ipc msg queue is full (EAGAIN=Resource temporarily unavailable)
[error] [client 1.2.3.4] (13)Permission denied: Could not open file /apps/opt/IBMHTTPD-SSO/logs/filter.52954
This message is known to originate from SiteMinder.

Reply all

Reply to author

Forward

0 new messages