Preauthentication issues using NTLM

[iva...@gmail.com]

Hi.

We are currently working on making a RavenDB based application and we are having trouble with making bulk inserts work properly.

We're currently using RavenDB server and client version 3.5.5-35223, windows authentication enabled and it's running as a windows service.

 

We have a testapp that does a bunch of bulk inserts to simulate a real world scenario. It's set up this way:

 

            DocumentStore.HttpMessageHandlerFactory = () => new WebRequestHandler

            {
                UnsafeAuthenticatedConnectionSharing = true,
                PreAuthenticate = true,
                AllowAutoRedirect = false,
                Credentials = CredentialCache.DefaultNetworkCredentials
            };

 

 

There are two issues at hand:

 

1) If we enable preauthenticate and authentication uses NTLM instead of Kerberos (for instance when the server and client is on the same machine, or the client credentials used is on a different domain) bulk insert doesn't work at all. No errors are thrown, but a fiddler trace shows that 0 documents are bulk inserted, even though the POST request get a 202 ACCEPTED response. I can also see that Preauthenticate isn't working because of the excessive amounts of 401s we are getting. Normal inserts / requests are working fine with NTLM, but we're still getting a lot of 401s. Running single or multithreaded makes no difference.

 

 

{"Documents":0,"IsSerializationError":false,"Completed":true,"Faulted":false,"Canceled":false,"State":{"Progress":"Imported 0 documents"},"Exception":null}

When kerberos auth is used (client is on a different machine), everything is working perfectly. The session is set up, and no unnecessary authentications happen. In our scenario running ravenDB on a different machine isn't really viable.

 

2) When preauthenticate is disabled we see weird things. As expected, get lots of 401s for every bulk insert (and other requests as well), which in turn impacts performance and makes our AD / Splunk guys angry.

In high traffic situations we have seen that the client gets confused and does weird things. What we're currently observing is errors like "Could not get token for bulk insert" on some of the bulk inserts, bad request etc.(although it happens pretty seldom). In a different application that uses RavenDB 3.0 with the same configuration I've also seen other weird things like SOAP data in request headers, request payloads with incomplete data (cut off) and requests meant for the ravendb server going to other webservices. This behaviour is observed both in dev, test and prod (everything on running on different hardware), and is fixed by enabling preauthentication. NTLM / Kerberos makes no difference in this scenario.

 

If we disable UnsafeAuthenticatedConnectionSharing this doesn't happen, but we get port exhaustion problems.

Any help regarding this would be greatly appreciated.

[Oren Eini (Ayende Rahien)]

1) What are the _exact_ requests and response you are seeing (the request url and the status code).

2)  Bulk Insert isn't actually using windows auth. It uses a windows auth request to generate a single use token and then uses that.

That is likely the reason for the bad token you are seeing. 

Is it possible that someone else is using the same connections as RavenDB in your case, you mentioned SOAP calls, do you know who put that data in?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[iva...@gmail.com]

I can upload fiddler traces if you have some place a bit more private i can send it to.
The traffic is piped through fiddler with system.net defaultproxy in our test application. The issues are replicated both with and without fiddler. Stream and decode is enabled, otherwise fiddler is running in it's default state.

1)

Bulk insert to local machine (NTLM) with preauthenticate enabled:
1

The only thing that's inserted into the database is a -1, which is not expected.



Bulk insert to local machine (NTLM) without preauthenticate:


And the database now has the expected data:



Bulk insert to remote machine (Kerberos), without preauthenticate:


It works like expected, but with lots of 401s. The Authorization token confirms that kerberos is now used.

Bulk insert remote machine (Kerberos) with preauthenticate:


Kerberos is used, preauthentication is working and the db has the expected content.

[iva...@gmail.com]

As for issue #2

 

On the other application it was solved by enabling preauthenticate. It has the RavenDB instance on a remote server, so kerberos + preauthentication and everything is fine. If we disable preauthentication strangeness starts ensuing when load gets high. That application mostly use regular inserts to ravendb which is probably why we see different behaviours in high traffic situations. The soap data was meant for a WCF webservice the application uses. I agree that it seems like some kind of connection pool or memory starts getting reused somehow, but the developers were adament on that the application isn't the issue.

 

Found a screenshot (FQDNs stripped) from when i was troubleshooting the other application, where it tried to do a https connect to a partial ravendb request url. It's a bit of WTF.

 

What i'd like to do is sort out all our issues preauthentication since everything seems to boil down to that. If preauthentication is working, everything is good. So the next question is probably - why doesn't NTLM preauthentication work, or is there a way to force kerberos when the client and server is on the same machine.

[Oren Eini (Ayende Rahien)]

That might be related to system configuration and likely need some consult from your system.

With regards to Bulk Insert, IIRC, they are using separate connection per bulk insert, and NTLM pre-auth relies on the connection level.

Kerberos does things differently, so that is likely why it works.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[iva...@gmail.com]

If bulk insert uses separate connections NTLM will be futile, yes... But we're seeing the same behaviour on regular gets / inserts as well. Replicating the auth setup with my own IIS + curl and it works fine (only 1 401). If i point the curl to the ravendb binding it doesn't work. Might seem like RavenDB doesn't send the headers needed for the NTLM auth to persist.

Anyway, instead of relying on NTLM since it's a no go for bulk inserts we will generate an API key and use it for those localhost requests.

[iva...@gmail.com]

With binding i mean the ravendb service. RavenDB isn't running on IIS in this scenario.