RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
APEL Martin
Hi Cole,
No problem regarding the delay. I’m happy that you try your best to help me 😊
You are right that I was stepping through the code of condor_store_cred, not through the schedd, so this was a futile attempt.
Unfortunately setting UID_DOMAIN didn’t help. I was under the assumption, that this setting has no effect on Windows. In any case here are my complete settings on Windows:
Condor_config:
use SECURITY : recommended(SYSTEM, Administrator@*, MAL1@*)
##--------------------------------------------------------------------
## Settings from the the installer questions
##--------------------------------------------------------------------
INSTALL_USER = MAL1
CONDOR_HOST = master.dsone.3ds.com
UID_DOMAIN = 3ds.com
CONDOR_ADMIN = marti...@3ds.com
SMTP_SERVER = mail.3ds.com
DAEMON_LIST = MASTER SCHEDD
Condor_config.local:
FILESYSTEM_DOMAIN = Simpack
UID_DOMAIN = 3ds.com
SEC_DEFAULT_AUTHENTICATION = REQUIRED
SEC_READ_AUTHENTICATION = REQUIRED
SEC_WRITE_AUTHENTICATION = REQUIRED
SEC_DEFAULT_AUTHENTICATION_METHODS = IDTOKENS, NTSSPI
SEC_CLIENT_AUTHENTICATION_METHODS = IDTOKENS, NTSSPI
SEC_READ_AUTHENTICATION_METHODS = IDTOKENS, NTSSPI
SEC_WRITE_AUTHENTICATION_METHODS = IDTOKENS, NTSSPI
ALLOW_READ = *@3ds.com/*.3ds.com,*@DSONE/*.3ds.com
ALLOW_WRITE = *@3ds.com/*.3ds.com,*@DSONE/*.3ds.com
ALLOW_DAEMON = ro...@3ds.com/*.3ds.com,con...@3ds.com/*.3ds.com
One additional oddity I noticed: Most of the time I get ‘Operation failed because it is not allowed’, but every now and then the operation takes longer and the error changes to ‘Operation failed. Make sure your ALLOW_WRITE setting includes this host.’. I don’t know if this helps in any way.
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Thursday, June 25, 2026 5:17 PM
To: APEL Martin <Marti...@3ds.com>; htcondo...@cs.wisc.edu
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, Sorry for the delayed response . I was out of the office for the past two days. The server (i. e. the Schedd) should be the one that maps the incoming user to a canonical user[@ domain] during authentication. I assume you were stepping
Hi Martin,
Sorry for the delayed response . I was out of the office for the past two days. The server (i.e. the Schedd) should be the one that maps the incoming user to a canonical user[@domain] during authentication. I assume you were stepping through the tool code and not the Schedd (I would not recommend). Looking back on your last tool debug output, it appears the final HTCondor identity that failed was MAL1.3ds.com@none. I would assume the intent would be for the final identity to be MA...@3ds.com. If this is the case, you should be able to achieve this without a map file by setting UID_DOMAIN = 3ds.com. This will get the bare username for all users via NTSSPI and set the domain name to 3ds.com thus resulting in the final identity of MA...@3ds.com.
-Cole Bollig
From: APEL Martin <Marti...@3ds.com>
Sent: Tuesday, June 23, 2026 7:23 AM
To: Cole Bollig <cabo...@wisc.edu>;
htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Cole,
The cluster master is running on Linux, as are all executor nodes. Submission can occur from Linux and Windows, on Windows jobs run under the user nobody. Therefore there’s no credd running.
I continued to investigate a bit and took a look at the source code of the authentication. There’s one strange thing I don’t understand: It should be possible to map usernames via the CERTIFICATE_MAPFILE config variable, which I have tried for testing. However from the debug output it seems as if this map file is never read, which I verified using ProcessMonitor. I built condor on Windows myself (by the way: specifying CMake build type DEBUG still seems to generate a RelWithDebInfo build, at least the output directory had this name) and stepped through the code with a debugger and authenticator_ is always NULL in line 491 of authenticate.cpp, so the map file is never consulted.
Thanks
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Monday, June 22, 2026 5:52 PM
To: APEL Martin <Marti...@3ds.com>;
htcondo...@cs.wisc.edu
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, There should be default mappings to allow NTSSPI map users correctly, so you shouldn't need a map file (there may need to be some other bits). But things are interesting due to the Windows AP into a linux pool. Are the user jobs supposed
Hi Martin,
There should be default mappings to allow NTSSPI map users correctly, so you shouldn't need a map file (there may need to be some other bits). But things are interesting due to the Windows AP into a linux pool. Are the user jobs supposed to run as a particular user on the EP side or can they run as nobody? Just trying to get a big picture idea about your pool setup.
-Cole
From: APEL Martin <Marti...@3ds.com>
Sent: Monday, June 22, 2026 9:32 AM
To: Cole Bollig <cabo...@wisc.edu>;
htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Cole,
Unfortunately this doesn’t work either. This is the log with a slightly different error message now:
E:\users\apel>condor_store_cred -d:D_SECURITY add
Account: MAL1@DSONE
CredType: password
Enter password:
06/22/26 16:29:18 STORE_CRED: In mode 100 'add', user is "MAL1@DSONE"
06/22/26 16:29:18 SECMAN: command 479 STORE_CRED to local schedd from TCP port 57767 (blocking).
06/22/26 16:29:18 SECMAN: new session, doing initial authentication.
06/22/26 16:29:18 SECMAN: Auth methods: NTSSPI,TOKEN,PASSWORD
06/22/26 16:29:18 AUTHENTICATE: setting timeout for <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=LP15-MAL1-CEM.dsone.3ds.com&noUDP&sock=schedd_15596_7b7c> to 20.
06/22/26 16:29:18 HANDSHAKE: in handshake(my_methods = 'NTSSPI,TOKEN,PASSWORD')
06/22/26 16:29:18 HANDSHAKE: handshake() - i am the client
06/22/26 16:29:18 HANDSHAKE: sending (methods == 2576) to server
06/22/26 16:29:18 HANDSHAKE: server replied (method = 16)
06/22/26 16:29:18 Authentication was a Success.
06/22/26 16:29:18 AUTHENTICATION: setting default map to (null)
06/22/26 16:29:18 AUTHENTICATION: post-map: current FQU is '(null)'
06/22/26 16:29:18 AUTHENTICATE: Exchanging keys with remote side.
06/22/26 16:29:18 AUTHENTICATE: Result of end of authenticate is 1.
06/22/26 16:29:18 SECMAN: generating AES key for session with local schedd...
06/22/26 16:29:18 SECMAN: successfully enabled encryption!
06/22/26 16:29:18 SECMAN: successfully enabled message authenticator!
06/22/26 16:29:18 SECMAN: FAILED: Received "DENIED" from server for user MAL1.3ds.com@none using method NTSSPI.
06/22/26 16:29:18 ERROR: SECMAN:2010:Received "DENIED" from server for user MAL1.3ds.com@none using method NTSSPI.
06/22/26 16:29:18 STORE_CRED: Failed to start STORE_CRED command. Unable to contact local schedd
Operation failed.
Make sure your ALLOW_WRITE setting includes this host.
I started playing around with the authentication map file, which I previously did not use at all, as my primary auth method is IDTOKEN, which doesn’t use this file. However I could not get it to map my Windows username to the one under Linux yet (according to the log output).
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Monday, June 22, 2026 4:25 PM
To: APEL Martin <Marti...@3ds.com>;
htcondo...@cs.wisc.edu
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, Oops, I did mean D_SECURITY. It looks like the client and server (since they share the same configuration) are using TOKEN authentication resulting in the condor user since that is the first method in the list. Try adding NTSSPI to
Hi Martin,
Oops, I did mean D_SECURITY. It looks like the client and server (since they share the same configuration) are using TOKEN authentication resulting in the condor user since that is the first method in the list. Try adding NTSSPI to the start of the list.
-Cole Bollig
From: APEL Martin <Marti...@3ds.com>
Sent: Monday, June 22, 2026 9:02 AM
To: Cole Bollig <cabo...@wisc.edu>;
htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Cole,
You probably meant -debug:D_SECURITY instead of S_SECURITY 😉
This is the generated output:
E:\users\apel>condor_store_cred -debug:D_SECURITY add
Account: MAL1@DSONE
CredType: password
Enter password:
06/22/26 15:59:35 STORE_CRED: In mode 100 'add', user is "MAL1@DSONE"
06/22/26 15:59:35 SECMAN: command 479 STORE_CRED to local schedd from TCP port 61582 (blocking).
06/22/26 15:59:35 SECMAN: new session, doing initial authentication.
06/22/26 15:59:35 SECMAN: Auth methods: TOKEN,NTSSPI,PASSWORD
06/22/26 15:59:35 AUTHENTICATE: setting timeout for <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=LP15-MAL1-CEM.dsone.3ds.com&noUDP&sock=schedd_24784_2504> to 20.
06/22/26 15:59:35 HANDSHAKE: in handshake(my_methods = 'TOKEN,NTSSPI,PASSWORD')
06/22/26 15:59:35 HANDSHAKE: handshake() - i am the client
06/22/26 15:59:35 HANDSHAKE: sending (methods == 2576) to server
06/22/26 15:59:35 HANDSHAKE: server replied (method = 2048)
06/22/26 15:59:35 IDTOKENS: Examining C:\Users\mal1\.condor\tokens.d\mal1.token for valid tokens from issuer none.
06/22/26 15:59:35 Ignoring token as it was signed with key POOL (not known to the server).
06/22/26 15:59:35 getTokenSigningKey(): for id=LOCAL, pool=0 v84mode=0 reading C:\Condor\tokens.sk\LOCAL
06/22/26 15:59:35 Authentication was a Success.
06/22/26 15:59:35 AUTHENTICATION: setting default map to condor@password
06/22/26 15:59:35 AUTHENTICATION: post-map: current FQU is 'condor@password'
06/22/26 15:59:35 AUTHENTICATE: Exchanging keys with remote side.
06/22/26 15:59:35 AUTHENTICATE: Result of end of authenticate is 1.
06/22/26 15:59:35 SECMAN: generating AES key for session with local schedd...
06/22/26 15:59:35 SECMAN: successfully enabled encryption!
06/22/26 15:59:35 SECMAN: successfully enabled message authenticator!
06/22/26 15:59:35 SESSION: client duplicated AES to BLOWFISH key for UDP.
06/22/26 15:59:35 SECMAN: added session LP15-MAL1-CEM:20884:1782136775:4 to cache for 60 seconds (3600s lease).
06/22/26 15:59:35 SECMAN: startCommand succeeded.
Operation failed because it is not allowed
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Monday, June 22, 2026 3:48 PM
To: APEL Martin <Marti...@3ds.com>;
htcondo...@cs.wisc.edu
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, What does the command with -debug: S_SECURITY say? -Cole From: APEL Martin <Martin. APEL@ 3ds. com> Sent: Monday, June 22, 2026 8: 34 AM To: Cole Bollig <cabollig@ wisc. edu>; htcondor-users@ cs. wisc. edu <htcondor-users@ cs. wisc. edu>
Hi Martin,
What does the command with -debug:S_SECURITY say?
-Cole
From: APEL Martin <Marti...@3ds.com>
Sent: Monday, June 22, 2026 8:34 AM
To: Cole Bollig <cabo...@wisc.edu>;
htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Cole,
I have restarted all services on the Windows machine and checked with condor_config_val, that NTSSPI was there.
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Monday, June 22, 2026 3:25 PM
To: APEL Martin <Marti...@3ds.com>;
htcondo...@cs.wisc.edu
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, I believe the authentication is choosing PASSWORD which is POOL password and thus trying as user condor (I do want to admit that the auth code is not my forte). When you added NTSSPI did you do a condor_reconfig? Without a reconfiguration
Hi Martin,
I believe the authentication is choosing PASSWORD which is POOL password and thus trying as user condor (I do want to admit that the auth code is not my forte). When you added NTSSPI did you do a condor_reconfig? Without a reconfiguration the Schedd will not have NTSSPI in its authentication list resulting in the client tool to still use PASSWORD authentication since the server side, i.e. the schedd, will pick the first matching option they both support.
-Cole Bollig
From: APEL Martin <Marti...@3ds.com>
Sent: Monday, June 22, 2026 1:54 AM
To: Cole Bollig <cabo...@wisc.edu>;
htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Cole,
Thanks for the hint, unfortunately it didn’t help. What confuses me are these messages in SchedLog:
06/22/26 08:43:56 (pid:13476) WARNING: store_cred() for user username@WinDomainName attempted by user condor, rejecting
It seems as if the interactive invocation of condor_store_cred is carried out as user ‘condor’, not me.
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Friday, June 19, 2026 4:42 PM
To: APEL Martin <Marti...@3ds.com>;
htcondo...@cs.wisc.edu
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, For the windows hosts, try adding NTSSPI to the list of authentication methods (before password). -Cole Bollig From: APEL Martin <Martin. APEL@ 3ds. com> Sent: Friday, June 19, 2026 2: 32 AM To: Cole Bollig <cabollig@ wisc. edu>;
Hi Martin,
For the windows hosts, try adding NTSSPI to the list of authentication methods (before password).
-Cole Bollig
From: APEL Martin <Marti...@3ds.com>
Sent: Friday, June 19, 2026 2:32 AM
To: Cole Bollig <cabo...@wisc.edu>;
htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Cole,
condor_config_val -dump authentication_methods returns
# Configuration from machine: LP15-MAL1-CEM.dsone.3ds.com
# Parameters with names that match authentication_methods:
SEC_CLIENT_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD
SEC_DEFAULT_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD
SEC_READ_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD
SEC_WRITE_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD
# Contributing configuration file(s):
# C:\Condor\condor_config
# C:\Condor\condor_config.local
I had added the PASSWORDS methods only for Windows, but it seems it doesn’t help in any way.
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Thursday, June 18, 2026 10:10 PM
To: APEL Martin <Marti...@3ds.com>;
htcondo...@cs.wisc.edu
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, For the windows submit host what does condor_config_val -dump authentication_methods return? -Cole From: APEL Martin <Martin. APEL@ 3ds. com> Sent: Thursday, June 18, 2026 7: 57 AM To: Cole Bollig <cabollig@ wisc. edu>; htcondor-users@ cs. wisc. edu
Hi Martin,
For the windows submit host what does condor_config_val -dump authentication_methods return?
-Cole
From: APEL Martin <Marti...@3ds.com>
Sent: Thursday, June 18, 2026 7:57 AM
To: Cole Bollig <cabo...@wisc.edu>;
htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Cole,
Thank you for your quick response. Regarding your questions:
1. I run condor_store_cred add without any additional parameters
2. Adding the -debug:D:HOST_NAME generates the following output (I have replaced domain and usernames):
condor_store_cred -debug:D_HOSTNAME add
06/18/26 08:29:16 NETWORK_INTERFACE=* matches Ethernet fe80::d7db:b5ca:797a:9984, Ethernet 169.254.55.6, Ethernet 4 fe80::f680:c67b:aff5:9b, Ethernet 4 169.254.71.157, Ethernet 2 fe80::2328:e55a:26cb:dadd, Ethernet 2 192.168.208.34, Wi-Fi fe80::f224:7aa9:83bb:1dc2, Wi-Fi 169.254.98.211, Local Area Connection* 1 fe80::894c:4326:1ed2:e343, Local Area Connection* 1 169.254.118.174, Local Area Connection* 12 fe80::b59:9f8e:7b22:bbff, Local Area Connection* 12 169.254.172.248, Bluetooth Network Connection fe80::d6c5:e04c:b18:7898, Bluetooth Network Connection 169.254.89.57, Loopback Pseudo-Interface 1 ::1, Loopback Pseudo-Interface 1 127.0.0.1, vEthernet (nat) fe80::660:43c2:243:4ea, vEthernet (nat) 192.168.224.1, choosing IP 192.168.208.34
06/18/26 08:29:16 hostname: hostname.dnsdomainname
06/18/26 08:29:16 I am: hostname: hostname, fully qualified doman name: hostname.dnsdomainname, IP: 192.168.208.34, IPv4: 192.168.208.34, IPv6:
06/18/26 08:29:16 Trying to getting network interface information after reading config
06/18/26 08:29:16 NETWORK_INTERFACE=* matches Ethernet fe80::d7db:b5ca:797a:9984, Ethernet 169.254.55.6, Ethernet 4 fe80::f680:c67b:aff5:9b, Ethernet 4 169.254.71.157, Ethernet 2 fe80::2328:e55a:26cb:dadd, Ethernet 2 192.168.208.34, Wi-Fi fe80::f224:7aa9:83bb:1dc2, Wi-Fi 169.254.98.211, Local Area Connection* 1 fe80::894c:4326:1ed2:e343, Local Area Connection* 1 169.254.118.174, Local Area Connection* 12 fe80::b59:9f8e:7b22:bbff, Local Area Connection* 12 169.254.172.248, Bluetooth Network Connection fe80::d6c5:e04c:b18:7898, Bluetooth Network Connection 169.254.89.57, Loopback Pseudo-Interface 1 ::1, Loopback Pseudo-Interface 1 127.0.0.1, vEthernet (nat) fe80::660:43c2:243:4ea, vEthernet (nat) 192.168.224.1, choosing IP 192.168.208.34
06/18/26 08:29:16 NETWORK_INTERFACE=* matches Ethernet fe80::d7db:b5ca:797a:9984, Ethernet 169.254.55.6, Ethernet 4 fe80::f680:c67b:aff5:9b, Ethernet 4 169.254.71.157, Ethernet 2 fe80::2328:e55a:26cb:dadd, Ethernet 2 192.168.208.34, Wi-Fi fe80::f224:7aa9:83bb:1dc2, Wi-Fi 169.254.98.211, Local Area Connection* 1 fe80::894c:4326:1ed2:e343, Local Area Connection* 1 169.254.118.174, Local Area Connection* 12 fe80::b59:9f8e:7b22:bbff, Local Area Connection* 12 169.254.172.248, Bluetooth Network Connection fe80::d6c5:e04c:b18:7898, Bluetooth Network Connection 169.254.89.57, Loopback Pseudo-Interface 1 ::1, Loopback Pseudo-Interface 1 127.0.0.1, vEthernet (nat) fe80::660:43c2:243:4ea, vEthernet (nat) 192.168.224.1, choosing IP 192.168.208.34
06/18/26 08:29:16 hostname: hostname.dnsdomainname
06/18/26 08:29:16 I am: hostname: hostname, fully qualified doman name: hostname.dnsdomainname, IP: 192.168.208.34, IPv4: 192.168.208.34, IPv6:
Account: username@WinDomainName
CredType: password
Enter password:
06/18/26 08:29:22 STORE_CRED: In mode 100 'add', user is "username@WinDomainName"
06/18/26 08:29:22 New Daemon obj (schedd) name: "", pool: "", addr: ""
06/18/26 08:29:22 Neither name nor addr specified, using local values - name: "hostname.dnsdomainname", full host: "hostname.dnsdomainname"
06/18/26 08:29:22 Finding classad for local daemon, SCHEDD_DAEMON_AD_FILE is "C:\Condor\spool/.schedd_classad"
06/18/26 08:29:22 Found Name in ClassAd, using "hostname.dnsdomainname"
06/18/26 08:29:22 Daemon client (schedd) address determined: name: "hostname.dnsdomainname", pool: "", alias: "hostname.dnsdomainname", addr: "<192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>"
06/18/26 08:29:22 Found SCHEDDIpAddr in ClassAd, using "<192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>"
06/18/26 08:29:22 Found CondorVersion in ClassAd, using "$CondorVersion: 25.11.0 2026-06-10 BuildID: 920473 GitSHA: 7f5259d9 $"
06/18/26 08:29:22 Found CondorPlatform in ClassAd, using "$CondorPlatform: x86_64_Windows10 $"
06/18/26 08:29:22 Found Machine in ClassAd, using "hostname.dnsdomainname"
06/18/26 08:29:22 Checking if <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4> is a sinful address
06/18/26 08:29:22 <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4> is a sinful address!
06/18/26 08:29:22 Using port 9618 based on address "<192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>"
06/18/26 08:29:22 Found address 1 candidates:
06/18/26 08:29:22 -410 192.168.208.34:9618
06/18/26 08:29:22 Considering address candidate 192.168.208.34:9618.
06/18/26 08:29:22 Found compatible candidate 192.168.208.34:9618.
06/18/26 08:29:22 Destroying Daemon object:
06/18/26 08:29:22 Type: 3 (schedd), Name: hostname.dnsdomainname, Addr: <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>
06/18/26 08:29:22 FullHost: hostname.dnsdomainname, Host: hostname, Pool: , Port: 9618
06/18/26 08:29:22 IsLocal: Y, IdStr: local schedd, Error:
06/18/26 08:29:22 --- End of Daemon object info ---
Operation failed because it is not allowed
3. The master node as well as all execution nodes of the cluster run on Linux. Submit hosts are both Linux and Windows. On Windows I do not use the ‘run_as_owner’ feature.
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Wednesday, June 17, 2026 5:26 PM
To: htcondo...@cs.wisc.edu
Cc: APEL Martin <Marti...@3ds.com>
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, Some questions to help look into this: What specifically are you running on the command line to store credentials? Can you run the same command with the -debug: D_HOSTNAME option and share the resulting output (feel free to cleanse
Hi Martin,
Some questions to help look into this:
1. What specifically are you running on the command line to store credentials?
2. Can you run the same command with the -debug:D_HOSTNAME option and share the resulting output (feel free to cleanse and/or send directly)?
3. You mentioned having a mixed pool. What OS are the Aps (submit hosts)?
Cheers,
Cole Bollig
From: HTCondor-users <htcondor-us...@cs.wisc.edu>
on behalf of APEL Martin via HTCondor-users <htcondo...@cs.wisc.edu>
Sent: Wednesday, June 17, 2026 5:17 AM
To: htcondo...@cs.wisc.edu <htcondo...@cs.wisc.edu>
Cc: APEL Martin <Marti...@3ds.com>
Subject: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
ZjQcmQRYFpfptBannerEnd
We have recently upgraded our HTCondor cluster from 8.9 to 25.11. The cluster contains Linux as well as Windows machines. All authentication and authorization is configured to use IDTOKENS, which works fine under Linux. However when using the same approach on Windows any submissions fail and tell me, that I need to use condor_store_cred.
When I run condor_store_cred add I get an error ‘Operation failed because it is not allowed’ after entering the password.
SchedLog contains entries such as
06/17/26 11:58:38 (pid:25592) WARNING: store_cred() for user user@domain attempted by user condor, rejecting
I have to add that the DNS domain is not identical to the Windows domain name. I have tried adding both domains to the ‘ALLOW_*’ configurations, I have tried enabling PASSWORD authentication, but nothing seems to help. I have a token for user@winDomain in my tokens.d directory, which allows me to run e.g. condor_status. I tried the same with a token for user@dnsdomain, condor_status works here as well.
But condor_store_cred continues to fail in all these cases.
Any help would be very much appreciated.
Martin
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
Cole Bollig
Sent: Friday, June 26, 2026 1:31 AM
To: Cole Bollig <cabo...@wisc.edu>; HTCondor Users Mailing LIst Forward <htcondo...@cs.wisc.edu>
Szabolcs Horvátth
SEC_CLIENT_AUTHENTICATION_METHODS = $(SEC_DEFAULT_AUTHENTICATION_METHODS) CLAIMTOBE
ZjQcmQRYFpfptBannerEnd
We have recently upgraded our HTCondor cluster from 8.9 to 25.11. The cluster contains Linux as well as Windows machines. All authentication and authorization is configured to use IDTOKENS, which works fine under Linux. However when using the same approach on Windows any submissions fail and tell me, that I need to use condor_store_cred.
When I run condor_store_cred add I get an error ‘Operation failed because it is not allowed’ after entering the password.
SchedLog contains entries such as
06/17/26 11:58:38 (pid:25592) WARNING: store_cred() for user user@domain attempted by user condor, rejecting
I have to add that the DNS domain is not identical to the Windows domain name. I have tried adding both domains to the ‘ALLOW_*’ configurations, I have tried enabling PASSWORD authentication, but nothing seems to help. I have a token for user@winDomain in my tokens.d directory, which allows me to run e.g. condor_status. I tried the same with a token for user@dnsdomain, condor_status works here as well.
But condor_store_cred continues to fail in all these cases.
Any help would be very much appreciated.
Martin
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.
Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer https://www.3ds.com/privacy-policy/contact/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-us...@cs.wisc.edu with a
subject: Unsubscribe
The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/
APEL Martin
Hi Cole,
I had tried putting NTSSPI in the front in various combinations with other changes, but it never worked. However I just tried again and this time it seems to work. I suspect that only in conjunction with setting the UID domain or maybe by writing the DSONE in upper-case in the allow statements this works.
But it’s not over yet: I can successfully call ‘condor_store_cred add’ now, but when I try to run condor_submit, it still doesn’t work. When I run it for the first time I only get
ERROR: Failed to create proc,
running it a second time changes the error message to
ERROR: No credential stored for MAL1@DSONE
Correct this by running:
condor_store_cred add
In the schedlog I have found the following assertion failure
06/30/26 08:45:21 (pid:20564) ERROR "Assertion ERROR on (ownerInfo != nullptr)" at line 3879 in file C:\condor\execute\dir_8312\sources\src\condor_schedd.V6\qmgmt.cpp
So this looks like a real bug, not just a configuration issue.
Thanks,
Martin
From: Cole Bollig <cabo...@wisc.edu>
Sent: Monday, June 29, 2026 4:42 PM
To: APEL Martin <Marti...@3ds.com>; HTCondor Users Mailing LIst Forward <htcondo...@cs.wisc.edu>
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, Things might be getting tripped up with the authentication method ordering. Try NTSSPI,IDTOKENS. Also, we may be able to figure out more by looking at the server side of the conversion i. e. the Schedd log. Requires SCHEDD_DEBUG =
APEL Martin
Hi Szabolcs,
Thanks for the hint. I saw this ‘CLAIMTOBE’ in the documentation and plan to use it as a last resort, if everything else fails. But as long as Cole is willing to help us figure out the reason I will continue to try.
Thanks,
Martin
From: Szabolcs Horvátth <szabolcs...@gmail.com>
Sent: Monday, June 29, 2026 5:49 PM
To: HTCondor-Users Mail List <htcondo...@cs.wisc.edu>
Cc: APEL Martin <Marti...@3ds.com>
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Hi Martin, We had a very similar upgrade path from 8. 4 to 25. 2, using linux servers, startds, but some windows submitters. In our case we had to add these to the windows condor local config files to make it work, no other authentication was
John M Knoeller
Sent: Tuesday, June 30, 2026 1:52 AM
Archives for older messages are found at https://www-auth.cs.wisc.edu/lists/htcondor-users/
---
You received this message because you are subscribed to the Google Groups "HTCondor Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to htcondor-user...@g-groups.wisc.edu.
To view this discussion visit https://groups.google.com/a/g-groups.wisc.edu/d/msgid/htcondor-users/ecc73ba6f8544734b1c3571da0df2fcf%403ds.com.
APEL Martin
Hi John, hi Cole,
What about mixed pools: Should NTSSPI be listed on Linux machines as first auth method as well (if this works at all)?
condor_store_cred query tells me, that a valid credential is stored.
The assert unfortunately has no accompanying stack trace, so I cannot provide that.
When I ran condor_qusers for the first time, the displayed list was empty. I have then added my user explicitly via ‘condor_qusers -add’, afterwards condor_qusers listed myself like this:
USER OS_USER ENABLED MAX_RUN JOBS:Idle Running Held Removed Completed
MA...@3ds.com MAL1@DSONE yes default 0 0 0 0 0
Which seems correct to me.
However condor_submit still generates the same error message and assertion failure.
Hope this helps,
Martin
From: John M Knoeller <joh...@cs.wisc.edu>
Sent: Tuesday, June 30, 2026 7:32 PM
To: Cole Bollig <cabo...@wisc.edu>; HTCondor Users Mailing LIst Forward <htcondo...@cs.wisc.edu>; APEL Martin <Marti...@3ds.com>
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
In a windows pool, NTSSPI should always be the first authentication method in the list for all daemons. It is simply not possible to use IDTOKEN for all authentication on Windows, because there are cases like submitting jobs where a valid Windows
John M Knoeller
User = "MA...@3ds.com"
Owner = "MAL1"
OSUser = "MAL1@DSONE"
Sent: Wednesday, July 1, 2026 1:10 AM
To: John M Knoeller <joh...@cs.wisc.edu>; Cole Bollig <cabo...@wisc.edu>; HTCondor Users Mailing LIst Forward <htcondo...@cs.wisc.edu>
APEL Martin
Hi John,
These are the relevant lines from condor_qusers -long:
OsUser = "MAL1@DSONE"
Owner = "MAL1"
User = MA...@3ds.com
NTDomain = "DSONE"
So exactly what you expected.
This is the log up until the failed assertion:
07/01/26 16:41:59 (pid:12404) ******************************************************
07/01/26 16:41:59 (pid:12404) ** condor_schedd.exe (CONDOR_SCHEDD) STARTING UP
07/01/26 16:41:59 (pid:12404) ** C:\Condor\bin\condor_schedd.exe
07/01/26 16:41:59 (pid:12404) ** SubsystemInfo: name=SCHEDD type=SCHEDD(4) class=DAEMON(1)
07/01/26 16:41:59 (pid:12404) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON
07/01/26 16:41:59 (pid:12404) ** $CondorVersion: 25.11.0 2026-06-10 BuildID: 920473 GitSHA: 7f5259d9 $
07/01/26 16:41:59 (pid:12404) ** $CondorPlatform: x86_64_Windows10 $
07/01/26 16:41:59 (pid:12404) ** PID = 12404
07/01/26 16:41:59 (pid:12404) ** Log last touched 7/1 16:41:55
07/01/26 16:41:59 (pid:12404) ******************************************************
07/01/26 16:41:59 (pid:12404) Using config source: C:\Condor\condor_config
07/01/26 16:41:59 (pid:12404) Using local config sources:
07/01/26 16:41:59 (pid:12404) C:\Condor\condor_config.local
07/01/26 16:41:59 (pid:12404) config Macros = 66, Sorted = 66, StringBytes = 2546, TablesBytes = 2432
07/01/26 16:41:59 (pid:12404) CLASSAD_CACHING is ENABLED
07/01/26 16:41:59 (pid:12404) Daemon Log is logging: D_ALWAYS D_ERROR D_STATUS
07/01/26 16:41:59 (pid:12404) SharedPortEndpoint: listener already created.
07/01/26 16:41:59 (pid:12404) DaemonCore: command socket at <192.168.1.23:9618?addrs=192.168.1.23-9618+[2003-fe-470b-3f00-8873-4ffb-989f-67d6]-9618&alias=LP15-MAL1-CEM.dsone.3ds.com&noUDP&sock=schedd_20104_be64>
07/01/26 16:41:59 (pid:12404) DaemonCore: private command socket at <192.168.1.23:9618?addrs=192.168.1.23-9618+[2003-fe-470b-3f00-8873-4ffb-989f-67d6]-9618&alias=LP15-MAL1-CEM.dsone.3ds.com&noUDP&sock=schedd_20104_be64>
07/01/26 16:41:59 (pid:12404) Daemon history file: C:\Condor\spool/schedd_daemon_history
07/01/26 16:41:59 (pid:12404) History file rotation is enabled.
07/01/26 16:41:59 (pid:12404) Maximum history file size is: 20971520 bytes
07/01/26 16:41:59 (pid:12404) Number of rotated history files is: 2
07/01/26 16:41:59 (pid:12404) config super users : condor, SYSTEM
07/01/26 16:41:59 (pid:12404) JOB_TRANSFORM_PelicanRetry setup as transform rule #1 :
NAME PelicanRetry
REQUIREMENTS (stringListIMember("pelican",WantTransferPluginMethods) || stringListIMember("osdf",WantTransferPluginMethods) || stringListIMember("stash",WantTransferPluginMethods)) && isUndefined(PelicanRetryEnabled)
EVALMACRO retry_delay = $(PelicanRetry_MinimumDelay) + random($(PelicanRetry_RandomDelay))
SET PelicanRetryEnabled true
SET PelicanRetryDelay $(retry_delay)
07/01/26 16:41:59 (pid:12404) Reloading job factories
07/01/26 16:41:59 (pid:12404) Loaded 0 job factories, 0 were paused, 0 failed to load
07/01/26 16:41:59 (pid:12404) TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s
07/01/26 16:41:59 (pid:12404) TransferQueueManager upload 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load
07/01/26 16:41:59 (pid:12404) TransferQueueManager download 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load
07/01/26 16:42:16 (pid:12404) Owner MAL1@dsone has no JobQueueUserRec
07/01/26 16:42:16 (pid:12404) Creating pending JobQueueUserRec for owner MAL1@dsone
07/01/26 16:42:16 (pid:12404) ERROR "Assertion ERROR on (ownerInfo != nullptr)" at line 3879 in file C:\condor\execute\dir_8312\sources\src\condor_schedd.V6\qmgmt.cpp
I had restarted all condor services immediately before the attempt, so there should not be anything cached from previous attempts.
Thanks,
John M Knoeller
07/01/26 16:42:16 (pid:12404) Owner MAL1@dsone has no JobQueueUserRec
07/01/26 16:42:16 (pid:12404) Creating pending JobQueueUserRec for owner MAL1@dsone
07/01/26 16:42:16 (pid:12404) ERROR "Assertion ERROR on (ownerInfo != nullptr)" at line 3879 in file
the socket authenticates as "MAL1@dsone" it does not find the User record whose name is "MA...@3ds.com", so it tries to create a new user record. Then when creates the user record it splits the authenticated identity into MAL1 and DSONE, and appends UID_DOMAIN to MAL1 to get "MA...@3ds.com" which becomes official name of the User record.
Sent: Wednesday, July 1, 2026 9:44 AM
APEL Martin
Hi John,
This did the trick! I had to regenerate a few tokens on the Linux side, because they store the domain name as well, but after I could successfully run jobs from Linux and Windows again.
Thanks very much for your help, also to you, Cole!
Martin
From: John M Knoeller <joh...@cs.wisc.edu>
Sent: Wednesday, July 1, 2026 5:21 PM
To: APEL Martin <Marti...@3ds.com>; Cole Bollig <cabo...@wisc.edu>; HTCondor Users Mailing LIst Forward <htcondo...@cs.wisc.edu>
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11
Ok that is it. These 3 lines tell the tale. 07/01/26 16: 42: 16 (pid: 12404) Owner MAL1@ dsone has no JobQueueUserRec 07/01/26 16: 42: 16 (pid: 12404) Creating pending JobQueueUserRec for owner MAL1@ dsone 07/01/26 16: 42: 16 (pid: 12404) ERROR "Assertion