db2val and FCM

[frit...@gmail.com]

I have a 10.1 Infosphere warehouse setup and I'm having a rather peculiar issue now and again when I reboot my nodes (I have 2).

After rebooting both nodes and making sure they come up cleanly, I try to do db2start. DB2 comes up on the 6 partitions on node 1 but won't come up on the 6 partitions on node 2.

In the DB2 diag on node 2, I repeatedly see this error:

Warning: waiting for the FCM shared memory set to be created seems to be taking longer than expected. Will continue to wait for another minute. Check to ensure that your configuration will in fact create an FCM shared memory segment.

Finally the db2start times out, unable to start any of the partitions on node 2.

Thinking maybe I have some kind of config issue, I run db2val and get this:

DBI1336E The partitioned database environment validation for the
instance db2inst1 failed. Reason code = 3.

Reason code 3 says that the start and end ports for FCM are not reserved properly. So, I look at the service file on both nodes and they're the same:

DB2_inst1 60008/tcp
DB2_inst1_END 60056/tcp

I'm not sure if the db2val thing is a red-herring or what because the whole startup issue with FCM only happens sometimes and usually another reboot or 2 clears it up, but it's kind of maddening that I have to just keep rebooting to get this to work. Once I do get db2 up and running everything seems to be OK, I just don't understand why the startup is messed up.

Any advice?

Thanks,
Greig Wise

[frit...@gmail.com]

Maybe more context would be helpful. A more complete error message from db2diag.log:

2013-04-16-16.14.29.301623-240 E3391788E602 LEVEL: Severe
PID : 7251 TID : 139989669111552 PROC : db2ckpwd 7
INSTANCE: inst1 NODE : 007
HOSTNAME: host.domain.com
EDUID : 1 EDUNAME: db2sysc 7
FUNCTION: DB2 UDB, oper system services, sqlockpw_daemon_main, probe:2
DATA #1 : <preformatted>

[Ian]

This is really a question you should be asking IBM with a PMR.

However: Is the instance called "db2inst1" or "inst1" ? The DBI1336E
message refers to it as "db2inst1", but this db2diag.log entry shows
"inst1".

The entries in /etc/services need to match the actual instance name.
If your instance is named "db2inst1"

DB2_db2inst1 60008/tcp
DB2_db2inst1_END 60056/tcp

But if it's called "inst1", then the ports can be named like you show.
i.e.,

DB2_inst1 60008/tcp
DB2_inst1_END 60056/tcp


I'm not sure if this has anything to do with the errors you're
getting, but it pops out as a possibility.

I'm not quite sure why you are reserving 49 ports for the FCM - you
mention that you have 2 servers with 6 logical partitions each.
Hopefully you aren't ever running with 49 logical database partitions
on a single host!

Unrelated to your specific problem, but while I'm talking about
/etc/services entries: it's generally considered best practice to
explicitly reserve every FCM port in /etc/services, like:

DB2_db2inst1 60008/tcp
DB2_db2inst1_1 60009/tcp
DB2_db2inst1_2 60010/tcp
DB2_db2inst1_3 60011/tcp
DB2_db2inst1_4 60012/tcp
...
DB2_db2inst1_47 60055/tcp
DB2_db2inst1_END 60056/tcp

This is not required, but it helps system administrators know that
every port between the "DB2_db2inst1" and the "DB2_db2inst1_END" port
may be used by DB2.

[frit...@gmail.com]

The actual instance name is different than what I posted, I just changed it as to not reveal anything "proprietary". But the actual instance name matches what is in the services file so that is not the issue. Also, I have actually tried enumerating all the services as you suggest, but that doesn't seem to help either.

I think that the db2val thing turns out to be a red herring because I have subsequently run it as root rather than the instance owner and in this case it does not complain about FCM.

So the question remains why I get the error about waiting for FCM shared memory creation at db2start time. I actually do have a PMR with IBM on this one, but I don't yet have resolution. I was trying to see if maybe someone here had a similar experience and could share some suggestions while I'm waiting on IBM.

Thanks,
Greig Wise