incremental propagation gets stuck with UPDATE_FULL_RESYNC_NEEDED

[Dave Steiner]

I've turned on incremental propagation for my two test Kerberos machines but
continually tries to do a full sync but doesn't.

Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_get_updates_1,
UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success,
client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX
Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_full_resync_1,
spawned resync process 1227, client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX
Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_get_updates_1,
UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success,
client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX
Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_full_resync_1,
spawned resync process 1230, client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX
Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_get_updates_1,
UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success,
client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX
Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_full_resync_1,
spawned resync process 1233, client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX
Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_get_updates_1,
UPDATE_FULL_RESYNC_NEEDED; Incoming SerialNo=0; Outgoing SerialNo=N/A, success,
client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX
Apr 29 16:50:40 host1 kadmind[19112](Notice): Request: iprop_full_resync_1,
spawned resync process 1236, client=kiprop/host2.rutgers.edu@DEFAULT_REALM,
service=kiprop/master1.rutgers.edu@DEFAULT_REALM, addr=XXX.XX.XXX.XX

Sometimes I get the spawned resync and sometimes I don't. But it never seems to
do anything. I still have the same entries when I do kproplog on the master and
the slave. There is no dumped file that I can find.

Before starting this (as I had worked with iprop a few months back) did a full
kprop and deleted the principal.ulog files to start fresh.

One odd thing about our setup is we have multiple realms. As far as I can tell
from previously playing with iprop is that it doesn't work on multiple realms.
But at this time, I just want to iprop my default realm.

The kdc.conf looks like:

[realms]
DEFAULT_REALM = {
.....
kadmind_port = 760
kdc_ports = 88
......
iprop_enable = true
iprop_port = 860
}

On the slave server, the kprop is started as

kpropd -r DEFAULT_REALM -f /path/data/DEFAULT_REALM/from_master -F
/path/data/DEFAULT_REALM/principal -P 860 -S

Any ideas why (1) it thinks it needs to do a full resync (kproplog shows one new
update on the master), and (2) why it's not doing the full resync? What can I
check to see why it's not working.

-ds

[Nico Williams]

On Mon, Apr 29, 2013 at 4:09 PM, Dave Steiner <ste...@oit.rutgers.edu> wrote:
> I've turned on incremental propagation for my two test Kerberos machines but
> continually tries to do a full sync but doesn't.

What version of MIT krb5 are you using?

> Before starting this (as I had worked with iprop a few months back) did a full
> kprop and deleted the principal.ulog files to start fresh.

BTW, there's a kproplog -R option to reset the ulog now. You should use that.

> One odd thing about our setup is we have multiple realms. As far as I can tell
> from previously playing with iprop is that it doesn't work on multiple realms.
> But at this time, I just want to iprop my default realm.

Multiple realms in one KDB principal file? Or just multiple realms on a host?

IIUC krb5kdc supports multiple realms in a single KDB just fine, but
kadmind doesn't, and kadmind plays a big role in iprop.

> Any ideas why (1) it thinks it needs to do a full resync (kproplog shows one new
> update on the master), and (2) why it's not doing the full resync? What can I
> check to see why it's not working.

Can you truss/strace the kadmind (and follow fork and exec) and see
what's happening? It's probably a misconfiguration that will be come
evident as soon as you see open(2) return some ENOENT in the
truss/strace output.

Nico
--

[Dave Steiner]

On 4/29/2013 7:34 PM, Nico Williams wrote:
> On Mon, Apr 29, 2013 at 4:09 PM, Dave Steiner <ste...@oit.rutgers.edu> wrote:
>> I've turned on incremental propagation for my two test Kerberos machines but
>> continually tries to do a full sync but doesn't.
> What version of MIT krb5 are you using?

Just upgraded to 1.11.2 (from 1.9.2)

>
>> Before starting this (as I had worked with iprop a few months back) did a full
>> kprop and deleted the principal.ulog files to start fresh.
> BTW, there's a kproplog -R option to reset the ulog now. You should use that.

Thanks!

>
>> One odd thing about our setup is we have multiple realms. As far as I can tell
>> from previously playing with iprop is that it doesn't work on multiple realms.
>> But at this time, I just want to iprop my default realm.
> Multiple realms in one KDB principal file? Or just multiple realms on a host?
>
> IIUC krb5kdc supports multiple realms in a single KDB just fine, but
> kadmind doesn't, and kadmind plays a big role in iprop.

Multiple realms in a single kdc.conf. Running multiple kadmind's on
different ports. That has worked fine for normal propagation.

>
>> Any ideas why (1) it thinks it needs to do a full resync (kproplog shows one new
>> update on the master), and (2) why it's not doing the full resync? What can I
>> check to see why it's not working.
> Can you truss/strace the kadmind (and follow fork and exec) and see
> what's happening? It's probably a misconfiguration that will be come
> evident as soon as you see open(2) return some ENOENT in the
> truss/strace output.

I will try this tomorrow and let you know. Thanks!

-ds

>
> Nico
> --

[Dave Steiner]

> Nico
> --

Nothing from the truss stood out as a problem although I did get the following
(probably from the kprop):

15522: lstat64("/tmp/kproptktCIaquE", 0xFFBFF770) Err#2 ENOENT
15522: unlink("/tmp/kproptktCIaquE") Err#2 ENOENT

which seems kind of odd.

I'm wondering about my ports. I start the kpropd on the slave with '-P 860' and
have the 'iprop_port =860' in the kdc.conf files on both master and slave. Is
that correct?

-ds

[Dave Steiner]

I think I've found the problem after looking at the truss output further.

> 3129: connect(8, 0x00026A60, 16, SOV_DEFAULT) Err#146
> ECONNREFUSED
> 3129: close(8) = 0
> 3129: fstat64(2, 0xFFBFF718) = 0
> 3129: write(2, " k p r o p", 5) = 5
> 3129: write(2, " : ", 2) = 2
> 3129: write(2, 0x0002A420, 18) = 18
> 3129: C o n n e c t i o n r e f u s e d
> 3129: write(2, " ", 1) = 1
> 3129: write(2, 0x00014B6C, 26) = 26
> 3129: w h i l e c o n n e c t i n g t o s e r v e r
> 3129: write(2, "\r", 1) = 1
> 3129: write(2, "\n", 1) = 1
> 3129: _exit(1)

Now I just need to find out why the connection is refused. I was able
to run the normal kprop command before I started iprop.

-ds

[Nico Williams]

On Wed, May 1, 2013 at 11:20 PM, Dave Steiner <ste...@oit.rutgers.edu> wrote:
> Because we have multiple realms, we run several kpropd's with the -P. When
> I run kprop I give the port. But when kprop is run from kadmin for
> incremental propagation, where is it going to get the correct port number
> from?

In src/kadmin/server/ipropd_svc.c I see:

/* XXX Yuck! */
if (getenv("KPROP_PORT")) {
pret = execl(kprop, "kprop", "-f", dump_file, "-P",
getenv("KPROP_PORT"), clhost, NULL);
} else {
pret = execl(kprop, "kprop", "-f", dump_file, clhost, NULL);
}

There's your answer. Either from KPROP_PORT in the environment, or by
having a per-kadmind instance krb5.conf and KRB5_CONFIG in the
environment.

Ideally all of the KDC-side daemons/tools would support multi-realm
operation, but kadmind doesn't quite at this time.

Nico
--

[Dave Steiner]

Because we have multiple realms, we run several kpropd's with the -P.
When I run kprop I give the port. But when kprop is run from kadmin for
incremental propagation, where is it going to get the correct port
number from?

-ds

[Nico Williams]

iprop and kprop use different ports.

Nico
--

[Dave Steiner]

On 5/2/2013 12:25 AM, Nico Williams wrote:
> On Wed, May 1, 2013 at 11:20 PM, Dave Steiner <ste...@oit.rutgers.edu> wrote:

>> Because we have multiple realms, we run several kpropd's with the -P. When
>> I run kprop I give the port. But when kprop is run from kadmin for
>> incremental propagation, where is it going to get the correct port number
>> from?

> In src/kadmin/server/ipropd_svc.c I see:
>
> /* XXX Yuck! */
> if (getenv("KPROP_PORT")) {
> pret = execl(kprop, "kprop", "-f", dump_file, "-P",
> getenv("KPROP_PORT"), clhost, NULL);
> } else {
> pret = execl(kprop, "kprop", "-f", dump_file, clhost, NULL);
> }
>
> There's your answer. Either from KPROP_PORT in the environment, or by
> having a per-kadmind instance krb5.conf and KRB5_CONFIG in the
> environment.
>
> Ideally all of the KDC-side daemons/tools would support multi-realm
> operation, but kadmind doesn't quite at this time.
>
> Nico
> --

Would the following work?

Since I only care about turning on iprop for one realm, I can just have
that kpropd use the default 754 port instead of 860 like I do now.
Would I then have to change iprop_port for that realm to 754 in the
kdc.conf also? Or does iprop happen over a different port from kprop?

-ds

[Dave Steiner]

Since I start multiple kpropd's (for our multiple realms) in the startup
script and give them different ports, the easiest fix was to just give
the one realm I want to iprop the port 754. That worked just fine.
Don't ask me why we didn't use 754 originally ... I inherited the setup. ;-)

Now I need to turn back on normal propagation for the other realms but
that hopefully won't be a problem.

Nico, thanks for all your help!

-ds