Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Weird error with ADFS integration

199 views
Skip to first unread message

Anders Jelnes

unread,
Aug 22, 2024, 4:37:58 AM8/22/24
to RCDevs Security
Hi,
A customer with licenses is experiencing a weird error.

Setup is:
WAP on server 2019 fully patched
ADFS on server 2019 fully patched
Latest ADFS plugin from RCDevs on ADFS server (1.0.14)
Latest Webadm Enterprise edition (2.3.20)

Error description:
ADFS will stop communicating with OTP server giving the error:
"The OpenOTP server could not be contacted. Check your network connection."

If I restart the ADFS service on the ADFS server communication will be reestablished for a while, sometimes for days sometimes hours and sometimes only minutes.

No obvious log entries on ADFS server or OTP server.

I'm at a loss over if this is a bug or misconfiguration (on my behalf) in the ADFS service or if it's the RCDevs ADFS plugin

Spyridon Gouliarmis (RCDevs)

unread,
Aug 23, 2024, 8:28:31 AM8/23/24
to RCDevs Security
Hello Anders,

can you show us the debug log from the plugin? You can start producing it in c:\RCDEVS-LOGS\ by setting debug_mode to 4 in HKEY_LOCAL_MACHINE\SOFTWARE\RCDevs\OpenOTP-AP , and waiting for the next crash to happen.

I investigated a similar issue at a customer's and the only interesting thing I could find in the Event Log was that the ADFS server lost network connectivity regularly (at least with its DCs). Maybe you'll be more lucky than I was with the Event Log.

Also, isn't the ADFS service restarted twice automatically? What happens then?

Spyridon Gouliarmis (RCDevs)

unread,
Aug 23, 2024, 8:30:07 AM8/23/24
to RCDevs Security
Restart the ADFS service after changing debug_log, BTW.

Anders Jelnes

unread,
Aug 23, 2024, 9:49:52 AM8/23/24
to RCDevs Security
Hi Spyridon,

Service recovery is set to restart twice, but I'm not sure it's actually restarted, as the service is actually running when the error appears. When the error appears at the ADFS web interface I do a service restart and comm's are up again... for a period.

Also ADFS service in itself seems to work as the customer has OTP excemptions when logging in from internal networks (per ADFS authentication policies), and there are no problems or errors when on internal networks.

I've followed your instructions and enabled debug mode + restarted ADFS service..... so now it's a waiting game ;-)

I'll return with data when I see the error again.

Best regards 
Anders 

fredag den 23. august 2024 kl. 14.30.07 UTC+2 skrev Spyridon Gouliarmis (RCDevs):
Restart the ADFS service after changing debug_log, BTW.

On Friday, August 23, 2024 at 2:28:31 PM UTC+2 Spyridon Gouliarmis (RCDevs) wrote:
Hello Anders,

can you show us the debug log from the plugin? You can start producing it in c:\RCDEVS-LOGS\ by setting debug_mode to 4 in HKEY_LOCAL_MACHINE\SOFTWARE\RCDevs\OpenOTP-AP , and waiting for the next crash to happen.

I investigated a similar issue at a customer's and the only interesting thing I could find in the Event Log was that the ADFS server lost network connectivity regularly (at least with its DCs). Maybe you'll be more lucky than I was with the Event Log.

Also, isn't the ADFS service restarted twice automatically? What happens then?

Spyridon Gouliarmis (RCDevs)

unread,
Aug 23, 2024, 10:46:48 AM8/23/24
to RCDevs Security
Sorry, for our ADFS AP, the right value is debug_mode 1. 4 is for the credential provider and authentication package.

Anders Jelnes

unread,
Aug 23, 2024, 11:24:24 AM8/23/24
to RCDevs Security
Debug mode set to 1 and ADFS service restarted, I see the RCDevsLogs folder now.

Will return when error and log data is available.

Anders Jelnes

unread,
Aug 26, 2024, 4:10:12 AM8/26/24
to RCDevs Security
Hi Spyridon,

So this morning it happened again.

I did the following:
1. tried a login using the ADFS service (https://FQDN/ECP)
2. same error
3. restart ADFS service 
4. login again same login using ADFS service (https://FQDN/ECP
5. login succeeded with TOTP
6. tried a different login using ADFS service (https://outlook.office.com) (debug logfile open)
7. ADFS service crashed
8. started ADFS service and tried same login again  (https://outlook.office.com) (debug file closed, just in case the ADFS service was bitching at an open logfile.. who knows)
9. login succeeded

I'm not sure at what point the error started occurring prior to this morning.

All the above steps performed this morning, logfile attached with anonymized email/UN.

BR
Anders

ADFS-Log - Copy.txt

Spyridon Gouliarmis (RCDevs)

unread,
Aug 26, 2024, 10:39:18 AM8/26/24
to RCDevs Security
Can you try again with the release candidate you can find here: https://ext.rcdevs.com/share/s/QENXXfXWLHs9MER

It should produce more debug logs between "Service provider ID : urn:federation:MicrosoftOnline" and the moment of the crash.

It also has a fix for a crash in the internal library responsible for contacting OpenOTP, so with a bit of luck you won't see any more crashes in the first place.
 provider ID : urn:federation:MicrosoftOnline

Anders Jelnes

unread,
Aug 27, 2024, 5:26:05 AM8/27/24
to RCDevs Security
Hi Spyridon,
I'll give it a go one of the coming days.... need to schedule a service window with the customer first. I'll return with data when done.

.... thanks for the help so far :-)

Anders Jelnes

unread,
Aug 28, 2024, 3:44:01 AM8/28/24
to RCDevs Security
Hi Spyridon,

Just to let you know... I just implemented the Release Candidate, did some rudimentary testing, all seems ok and I can se an extra logfile nanohttp.txt besides the previous ADFS-Log.txt.

I'll keep an eye on the environment and let you know the results, if no errors show in the next 14 days I'll give you a heads up that all is OK otherwise I'll write as soon as I se an error.

Thanks again :-) 

Anders Jelnes

unread,
Aug 28, 2024, 3:44:15 AM8/28/24
to RCDevs Security
Hi Spyridon,

Had an error this morning when testing, not sure though if it's related to the current issue as the error was something along the lines of ADFS server not able to serve request at the moment.

The error went away without restarting the service, I just closed the browser window and tried again, all went well on second attempt.

I've attached log files 

tirsdag den 27. august 2024 kl. 11.26.05 UTC+2 skrev Anders Jelnes:
ADFS-Log - Copy.txt
nanohttp - Copy.txt

Anders Jelnes

unread,
Aug 28, 2024, 3:44:21 AM8/28/24
to RCDevs Security
Event log showed this error:
Faulting application name: Microsoft.IdentityServer.ServiceHost.exe, version: 10.0.17763.4644, time stamp: 0x93c9137a
Faulting module name: ucrtbase.dll, version: 10.0.17763.6189, time stamp: 0xbc3e3f37
Exception code: 0xc0000005
Fault offset: 0x0000000000045ff7
Faulting process id: 0x17cc
Faulting application start time: 0x01daf899daa5cab6
Faulting application path: C:\Windows\ADFS\Microsoft.IdentityServer.ServiceHost.exe
Faulting module path: C:\Windows\System32\ucrtbase.dll
Report Id: 7b6d275b-a3a3-40e1-b110-a91a2e77d1ef
Faulting package full name:
Faulting package-relative application ID: 

tirsdag den 27. august 2024 kl. 11.26.05 UTC+2 skrev Anders Jelnes:

Anders Jelnes

unread,
Aug 28, 2024, 7:58:20 AM8/28/24
to RCDevs Security
Had the error again at 13:39 Europe/Copenhagen (time of test).

Attached the logfiles 

ADFS-Log - Copy.txt
nanohttp - Copy.txt

Spyridon Gouliarmis (RCDevs)

unread,
Aug 28, 2024, 10:24:20 AM8/28/24
to RCDevs Security
Alright, can you try the same nextcloud link as before? It should have a new .msi, with the same name.

No fixes this time, just more logs where the previous ones stop.

Anders Jelnes

unread,
Aug 29, 2024, 8:36:49 AM8/29/24
to RCDevs Security
done... installed log enhanced version, restarted entire server, checked that ADFS and MFA comms are up and working, all ok.... waiting for incident and will return with data.

Anders Jelnes

unread,
Sep 2, 2024, 3:41:15 AM9/2/24
to RCDevs Security
Good morning,

Error this morning, restarted ADFS service, tried again - succes

Attached logfiles

ADFS-Log - Copy.txt
nanohttp - Copy.txt

Spyridon Gouliarmis (RCDevs)

unread,
Sep 4, 2024, 8:45:03 AM9/4/24
to RCDevs Security
I apologize, the previous version did not output anything in the nanohttp file (as can be seen from the dates inside).

The version currently behind the nextcloud link should now output its logs properly.

Anders Jelnes

unread,
Sep 9, 2024, 9:45:48 AM9/9/24
to RCDevs Security
Hi Spyridon,

I'll install the version available at the NC link today and return with log data as soon as the error manifests itself again.

Sorry I've been away for some time.

Anders Jelnes

unread,
Sep 9, 2024, 9:54:30 AM9/9/24
to RCDevs Security
It's installed now, did some test logins failed two times each time I did a restart of the ADFS service, third time it showed the token field.

Just to be sure I'll let it run until next it fails and then upload the data.

Anders Jelnes

unread,
Sep 16, 2024, 5:27:46 AM9/16/24
to RCDevs Security
Hi Spyridon,

Attached logfiles... there should be several errors followed by service restarts (ADFS service) followed by succesful attempts.

Let me know if you need any further info from me :-)
nanohttp - Copy.txt
ADFS-Log - Copy.txt

Spyridon Gouliarmis (RCDevs)

unread,
Sep 18, 2024, 9:51:04 AM9/18/24
to RCDevs Security
FYI, we've managed to replicate the issue and are figuring out a fix.

Anders Jelnes

unread,
Sep 18, 2024, 11:17:56 AM9/18/24
to rcdevs-t...@googlegroups.com

sounds fantastic


--
You received this message because you are subscribed to the Google Groups "RCDevs Security" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rcdevs-technic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rcdevs-technical/4d1f5f0e-81b4-4b57-8e3b-0d9e95a13e93n%40googlegroups.com.

Spyridon Gouliarmis (RCDevs)

unread,
Sep 20, 2024, 3:16:26 AM9/20/24
to RCDevs Security
The usual link (https://ext.rcdevs.com/share/s/QENXXfXWLHs9MER) now contains a fixed installer that should not crash anymore. Can you try it?

Anders Jelnes

unread,
Sep 20, 2024, 8:29:16 AM9/20/24
to RCDevs Security
just installed it.... will monitor and test and return with results

Anders Jelnes

unread,
Sep 23, 2024, 7:57:02 AM9/23/24
to RCDevs Security
Error still persists.
Tried this morning, ADFS service had crashed and was not running. Started service and tried login... succes ..... waited a while and tried login again same error no communication with mfa server.

Logs attached

nanohttp.txt
ADFS-Log.txt

Anders Jelnes

unread,
Sep 23, 2024, 7:57:02 AM9/23/24
to RCDevs Security
Don't now if it's any help, but the following error appeared in Windows Application Event  log just about the time when i tried a login this morning:
Log Name:      Application
Source:        Application Error
Date:          23-09-2024 10:29:40
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      *****-ADFS01.****.local
Description:

Faulting application name: Microsoft.IdentityServer.ServiceHost.exe, version: 10.0.17763.4644, time stamp: 0x93c9137a
Faulting module name: ucrtbase.dll, version: 10.0.17763.6189, time stamp: 0xbc3e3f37
Exception code: 0xc0000005
Fault offset: 0x0000000000045ff7
Faulting process id: 0x1728
Faulting application start time: 0x01db0b5ba50ad167

Faulting application path: C:\Windows\ADFS\Microsoft.IdentityServer.ServiceHost.exe
Faulting module path: C:\Windows\System32\ucrtbase.dll
Report Id: b51aa5a5-d62c-47cc-81b3-2ad57b8ccbf2

Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2024-09-23T08:29:40.144759900Z" />
    <EventRecordID>8519845</EventRecordID>
    <Channel>Application</Channel>
    <Computer>****-ADFS01.****.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Microsoft.IdentityServer.ServiceHost.exe</Data>
    <Data>10.0.17763.4644</Data>
    <Data>93c9137a</Data>
    <Data>ucrtbase.dll</Data>
    <Data>10.0.17763.6189</Data>
    <Data>bc3e3f37</Data>
    <Data>c0000005</Data>
    <Data>0000000000045ff7</Data>
    <Data>1728</Data>
    <Data>01db0b5ba50ad167</Data>
    <Data>C:\Windows\ADFS\Microsoft.IdentityServer.ServiceHost.exe</Data>
    <Data>C:\Windows\System32\ucrtbase.dll</Data>
    <Data>b51aa5a5-d62c-47cc-81b3-2ad57b8ccbf2</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

fredag den 20. september 2024 kl. 14.29.16 UTC+2 skrev Anders Jelnes:

Yoann Traut (RCDevs)

unread,
Sep 23, 2024, 11:53:28 AM9/23/24
to RCDevs Security
Hello,

The logs you provided indicate that the plugin upgrade did not work as expected, and the test version we provided is not being used. Could you please remove the ADFS plugin, unregister it, and reinstall it using the version we provided?

Regards

Anders Jelnes

unread,
Sep 24, 2024, 10:10:59 AM9/24/24
to RCDevs Security
Sure, but could you provide instructions on how to "unregister" it, do you mean uninstall it, reboot (just to be sure), and then install again?

It makes sense that the plugin did not take effect... see attached screenshot
2024-09-23 18_13_03.png

Anders Jelnes

unread,
Sep 24, 2024, 10:11:02 AM9/24/24
to RCDevs Security
I uninstalled all entries from program manager, rebooted, reinstalled and rebooted again.

Checked if working after last reboot, succes.

.... and now also remembered to set debug mode 1 in reg ;-)

Will provide data and result within a week.



mandag den 23. september 2024 kl. 17.53.28 UTC+2 skrev Yoann Traut (RCDevs):

Anders Jelnes

unread,
Oct 4, 2024, 4:08:57 AM10/4/24
to RCDevs Security
Hi,
Just to let you know.... No errors since last post.... seems you nailed it ;-)

Let me know if you need any log files to confirm 

Spyridon Gouliarmis (RCDevs)

unread,
Oct 10, 2024, 9:03:19 AM10/10/24
to RCDevs Security
FYI, the new release at https://www.rcdevs.com/downloads/download/?file=Plugins%2FOpenOTP_ADFS-1.0.15.0-x64.zip includes the fix for your issue.

Don't forget to disable the logs.
Reply all
Reply to author
Forward
0 new messages