IDTOKENS and defrag daemon

[Carsten Aulbert]

Hi all,

with the recent switch to v25.0, we finally moved away from a pool
password to idtokens which seemed to have worked fine for standard
operations - maybe by accident ;-)

When starting playing around with the defrag daemon, I started to see
these errors on our central manager where I enabled defrag:

07/21/26 12:03:48 ERROR: AUTHENTICATE:1003:Failed to authenticate with
any method|AUTHENTICATE:1004:Failed to authenticate using
SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using
IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS

(sorry for the noise, I should remove SCITOKENS and FS)

I was able to fix this by adding /etc/condor/password.d/POOL to the
**EP** triggered by this debug log from DefragLog:

07/21/26 12:32:57 HANDSHAKE: in handshake(my_methods = 'TOKEN,SCITOKENS')
07/21/26 12:32:57 HANDSHAKE: handshake() - i am the client
07/21/26 12:32:57 HANDSHAKE: sending (methods == 6144) to server
07/21/26 12:32:57 HANDSHAKE: server replied (method = 2048)
07/21/26 12:32:57 IDTOKENS: Examining /etc/condor/tokens.d/condor for
valid tokens from issuer BLA.
07/21/26 12:32:57 Server sent status indicating not OK.
07/21/26 12:32:57 PW: Client received ERROR from server, propagating
07/21/26 12:32:57 Can't send null for random string.
07/21/26 12:32:57 AUTHENTICATE: method 2048 (IDTOKENS) failed.

If I understand this correctly, the central manager's defrag is acting
as the "client" and thus I need to configure a (signing) password on the
EPs I wish to drain? Or am I thoroughly confused and overlooking
something as this is the opposite of my mental model.

Cheers

Carsten

--
Dr. Carsten Aulbert, Max Planck Institute for Gravitational Physics,
Callinstraße 38, 30167 Hannover, Germany, Phone +49 511 762 17185

[Thomas Hartmann]

Hi Carsten,

can you check what capabilities/scope the idtoken has for the
machine/daemon where the defrag is running? I would guess that it should
have at least READ and ADMIN

Cheers,
Thomas

[Carsten Aulbert]

Hi Thomas,

On 7/21/26 15:37, Thomas Hartmann wrote:
> can you check what capabilities/scope the idtoken has for the machine/
> daemon where the defrag is running? I would guess that it should have at
> least READ and ADMIN

No scope at all when listed via condor_token_list, which hopefully means
all capabilities? It's an ancient token, I don't remember how I created
it back at @1694080679 (and never looked into it as it just worked for
regular operations).

[John M Knoeller]

No scope in the token means no narrowing.

Whether it has READ and/or ADMIN will depend on whether the identity of the token is in the ALLOW_READ and/or ALLOW_ADMIN list of the Startd config.

-tj

________________________________________
From: 'Carsten Aulbert' via HTCondor Users
Sent: Tuesday, July 21, 2026 12:07 PM
To: Thomas Hartmann
Cc: htcondo...@g-groups.wisc.edu
Subject: Re: [HTCondor-users] IDTOKENS and defrag daemon

Hi Thomas,

Cheers

Carsten

--
Archives for older messages are found at https://www-auth.cs.wisc.edu/lists/htcondor-users/
---
You received this message because you are subscribed to the Google Groups "HTCondor Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to htcondor-user...@g-groups.wisc.edu.
To view this discussion visit https://groups.google.com/a/g-groups.wisc.edu/d/msgid/htcondor-users/0aeaa8f0-88b2-4be6-9d65-ce1755a4cb05%40aei.mpg.de.

[Carsten Aulbert]

Hi tj,

Finally time to look into this again.

On 7/21/26 20:38, John M Knoeller wrote:
> No scope in the token means no narrowing.

ok

> Whether it has READ and/or ADMIN will depend on whether the identity of the token is in the ALLOW_READ and/or ALLOW_ADMIN list of the Startd config.

very basic and not that secure:

target node has
ALLOW_ADMINISTRATOR = condor@* condor@password root@*
ALLOW_DAEMON = condor@* condor@password
ALLOW_NEGOTIATOR = condor@* condor@password
ALLOW_READ = *
ALLOW_WRITE = *

(taken from condor_config_val -sum)

The token on the machine running defrag daemon has

"sub":"con...@atlas.local"

thus, I would expect this to work but in DefragLog I get

07/27/26 07:56:55 SECMAN: required authentication with startd
sl...@a8201.atlas.local failed, so aborting command DRAIN_JOBS.
07/27/26 07:56:55 ERROR: AUTHENTICATE:1003:Failed to authenticate with

any method|AUTHENTICATE:1004:Failed to authenticate using
SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using
IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS

07/27/26 07:56:55 Failed to send request to drain
sl...@a8201.atlas.local: Failed to start DRAIN_JOBS command to
sl...@a8201.atlas.local

Thanks for any hints/insight :)

[John M Knoeller]

Best guess would be that the STARTD is unable to validate the token because it does not have a signing key that matches the one that was used the issue the token.

Try temporarily enabling this config variable on the startd. (don't forget to reconfig the startd)

STARTD_DEBUG = $(STARTD_DEBUG) D_SECURITY:2

This generates a lot of output, so you don't want to leave this debug level turned on long term, but it should be able to give some insight into why the token cannot be used to authenticate the drain command.

-tj

________________________________________
From: 'Carsten Aulbert' via HTCondor Users

Sent: Monday, July 27, 2026 3:45 AM
To: John M Knoeller

Cc: htcondo...@g-groups.wisc.edu
Subject: Re: [HTCondor-users] IDTOKENS and defrag daemon

Hi tj,

(taken from condor_config_val -sum)

"sub":"con...@atlas.local"

--

Archives for older messages are found at https://www-auth.cs.wisc.edu/lists/htcondor-users/
---
You received this message because you are subscribed to the Google Groups "HTCondor Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to htcondor-user...@g-groups.wisc.edu.

To view this discussion visit https://groups.google.com/a/g-groups.wisc.edu/d/msgid/htcondor-users/2b4683a9-0311-46f9-9ee3-6602714d7c72%40aei.mpg.de.