Cannot bulk insert any more

[Daniel González García]

Hi,

When I was using previous version (.2230) of the client and server, I was able to perform bulk inserts without any problem.

When I changed to the newest version (.2260) I am not able to do it anymore in some environments running exactly the same code in all of them.

When I capture the offeding traffic I get:

{
  "Url": "/bulkInsert?",
  "Error": "System.ObjectDisposedException: Safe handle has been closed\r\n   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)\r\n   at Microsoft.Win32.Win32Native.GetTokenInformation(SafeTokenHandle TokenHandle, UInt32 TokenInformationClass, SafeLocalAllocHandle TokenInformation, UInt32 TokenInformationLength, UInt32& ReturnLength)\r\n   at System.Security.Principal.WindowsIdentity.GetTokenInformation(SafeTokenHandle tokenHandle, TokenInformationClass tokenInformationClass)\r\n   at System.Security.Principal.WindowsIdentity.get_User()\r\n   at System.Security.Principal.WindowsIdentity.GetName()\r\n   at System.Security.Principal.WindowsIdentity.get_Name()\r\n   at Raven.Database.Server.Security.MixedModeRequestAuthorizer.AuthorizeOSingleUseAuthToken(IHttpContext context, String token) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Server\\Security\\MixedModeRequestAuthorizer.cs:line 69\r\n   at Raven.Database.Server.HttpServer.SetThreadLocalState(IHttpContext ctx) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Server\\HttpServer.cs:line 900\r\n   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Server\\HttpServer.cs:line 837\r\n   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\\Builds\\RavenDB-Stable\\Raven.Database\\Server\\HttpServer.cs:line 609"
}

I have restarted the raven db service without any luck.

I would appreciate any inputs, because this feature is key to our solution.

Thanks in advance

[Oren Eini (Ayende Rahien)]

This has been fixed already.
Are you doing that in 2360?

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Daniel González García]

Both client and server are running .2360.
I tripled checked:

• Raven.Server.exe: file version 2.0.3.2360, product version 2.0.3 / 4ab4776
• Raven.Client.Lightweight.dll: file version 2.0.3.2360, product version 2.0.3 / 4ab4776

As it had to do with some authentication, I have recreated both the system database and "the" database from scratch, re-added the domain account that needs access to the system database and granted access to "the" database. Restarted the service with the same error.

[Oren Eini (Ayende Rahien)]

Hm,
This was fixed here:
https://github.com/ayende/ravendb/commit/27706fa87b7772fab538c4b77f40648f248532c1

I'll issue another unstable for 2.0 for the merge for this fix

[Daniel González García]

Does that mean that we will have to roll an unstable version of the client?
Where could I download that version?

The sad thing is that behavior is only observed in our development environment. Our staging and pre-production work fine with the same codebas and dependencies.

[Oren Eini (Ayende Rahien)]

It will be there in a few hours: http://hibernatingrhinos.com/builds/ravendb-unstable-v2.0/

This is a server side change.

The problem relates to your exact production environment, if you are on a domain / not, etc.

[Daniel González García]

Thanks for providing a fix so promptly. Looking forward to downloading it.

Strange thing is that the environment has not changed in months.
Db server still on a domain, db still running with the same credentials, clients still in the domain and running the same credentials.
But in our other (also static in terms of changes) environments we have not seen the problem (yet).
And I hope we never see the problem, because deploying an unstable database server sends chills down my spine. And stakeholders do not know that yet....

[Oren Eini (Ayende Rahien)]

Please the build here: http://hibernatingrhinos.com/builds/ravendb-unstable-v2.0/2363-unstable
2363

[Daniel González García]

Unfortunately, I am still getting the same error.

Based on the github link you provided, I checked (dissasembled) the Raven.Database.dll code and the stable version (.2360) does not contain the piece of code present in the commit.
The unstable version from the build you provided (.2363) does. But still, does not solve the problem.

When I upgrade the server, I only stop the service and replace all the binaries in the folder that contains the server. I do not have to reinstall the service, do I?

[Daniel González García]

A bit more information: the database server is running as a service under the "Local System" credentials

[Oren Eini (Ayende Rahien)]

No, just upgrading the binaries should suffice.

What do you mean, 2363 doesn't solve the problem?
What is the error you get?

[Daniel González García]

Exactly, .2363 does not solve my problem.

I get exactly the same error as I was getting with .2360:

{
  "Url": "/bulkInsert?",
  "Error": "System.ObjectDisposedException: Safe handle has been closed\r\n   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)\r\n   at Microsoft.Win32.Win32Native.GetTokenInformation(SafeTokenHandle TokenHandle, UInt32 TokenInformationClass, SafeLocalAllocHandle TokenInformation, UInt32 TokenInformationLength, UInt32& ReturnLength)\r\n   at System.Security.Principal.WindowsIdentity.GetTokenInformation(SafeTokenHandle tokenHandle, TokenInformationClass tokenInformationClass)\r\n   at System.Security.Principal.WindowsIdentity.get_User()\r\n   at System.Security.Principal.WindowsIdentity.GetName()\r\n   at System.Security.Principal.WindowsIdentity.get_Name()\r\n   at Raven.Database.Server.Security.MixedModeRequestAuthorizer.AuthorizeOSingleUseAuthToken(IHttpContext context, String token) in c:\\Builds\\RavenDB-Unstable-v2.0-s\\Raven.Database\\Server\\Security\\MixedModeRequestAuthorizer.cs:line 116\r\n   at Raven.Database.Server.HttpServer.SetThreadLocalState(IHttpContext ctx) in c:\\Builds\\RavenDB-Unstable-v2.0-s\\Raven.Database\\Server\\HttpServer.cs:line 900\r\n   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\\Builds\\RavenDB-Unstable-v2.0-s\\Raven.Database\\Server\\HttpServer.cs:line 837\r\n   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\\Builds\\RavenDB-Unstable-v2.0-s\\Raven.Database\\Server\\HttpServer.cs:line 609"

[Oren Eini (Ayende Rahien)]

Can you check that OnetimeToken.User has return new WindowsPrincipal(new WindowsIdentity(windowsUserToken.Value));

[Oren Eini (Ayende Rahien)]

As well as what is the value of windowsUserToken ?

[Daniel González García]

Yes, the property return a new instance of WindowsPrincipalbsed on the WindowsIDentity with the value of the token (if the token has a value).
Otherwise returns the "cached" IPrincipal.

This is the piece of disassembled code:

public IPrincipal User { get { if (this.windowsUserToken.HasValue) { return new WindowsPrincipal(new WindowsIdentity(this.windowsUserToken.Value)); } return this.user; }

[Daniel González García]

I am not sure what you mean.

What is the value of the variable at runtime?
How could I get that? Attaching a debugger to the ravendb process?

[Daniel González García]

Oh, if you mean, the setter code, is (again, decompiled):

WindowsPrincipal windowsPrincipal = value as WindowsPrincipal;
if (windowsPrincipal != null) {
this.user = null;
this.windowsUserToken = new IntPtr?(((WindowsIdentity) windowsPrincipal.Identity).Token);
...

[Oren Eini (Ayende Rahien)]

Yes, attached a debugger.

[Daniel González García]

I will be doing that today as soon as possible. Have other "fires" burning atm...

[Daniel González García]

Ok,

I attached the VS debugger to the running service. I loaded the symbols from the pdbs contained in the unstable build.

Unfortunately, the exception did not pop up and I do not know how to set a breakpoint in the method that is causing trouble.

Any hints?

[Oren Eini (Ayende Rahien)]

We think that this happens because under some condition, an aysnc task is running under the same thread as the main one.

We are investigating a fix right now

[georgiosd]

Just to verify that 2363 doesn't work here either

[Daniel González García]

Ok, thanks for your time. Looking forward to hearing about that fix.

[georgiosd]

Any progress on this?

[Kijana Woodard]

I _think_ this thread relates as it's a similar behavior:

https://groups.google.com/forum/#!searchin/ravendb/async$20block/ravendb/Vi2iC6xkxcI/tbHEV_b2fGcJ

[Georgios Diamantopoulos]

Looks like it's fixed in the next build, thanks - for the record, I wasn't using an async bulk insert.

---

Date: Fri, 24 May 2013 11:47:46 -0500
Subject: Re: [RavenDB] Cannot bulk insert any more
From: kijana....@gmail.com
To: rav...@googlegroups.com

You received this message because you are subscribed to a topic in the Google Groups "ravendb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/2DA_DQyL99w/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

That doesn't matter, internally bulk insert is always async.

[Oren Eini (Ayende Rahien)]

Okay, we need to setup some time on Skype to reproduce and analyze this.

When can you do that?

On Mon, May 27, 2013 at 12:13 PM, Martin Kristensen <martin...@gmail.com> wrote:

Hi,

And thank you for helping. I'm a colleague to Daniel and I have just tried to update raven for latest version (2370), both client and server. Though I still get exact same error as with the previous version of RavenDb.

This is the exception I receive: 

"InternalServerError Internal Server Error\r\n{\r\n  \"Url\": \"/bulkInsert?\",\r\n  \"Error\": \"System.ObjectDisposedException: Safe handle has been closed\\r\\n   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)\\r\\n   at Microsoft.Win32.Win32Native.GetTokenInformation(SafeTokenHandle TokenHandle, UInt32 TokenInformationClass, SafeLocalAllocHandle TokenInformation, UInt32 TokenInformationLength, UInt32& ReturnLength)\\r\\n   at System.Security.Principal.WindowsIdentity.GetTokenInformation(SafeTokenHandle tokenHandle, TokenInformationClass tokenInformationClass)\\r\\n   at System.Security.Principal.WindowsIdentity.get_User()\\r\\n   at System.Security.Principal.WindowsIdentity.GetName()\\r\\n   at System.Security.Principal.WindowsIdentity.get_Name()\\r\\n   at Raven.Database.Server.Security.MixedModeRequestAuthorizer.AuthorizeOSingleUseAuthToken(IHttpContext context, String token) in c:\\\\Builds\\\\RavenDB-Stable\\\\Raven.Database\\\\Server\\\\Security\\\\MixedModeRequestAuthorizer.cs:line 69\\r\\n   at Raven.Database.Server.HttpServer.SetThreadLocalState(IHttpContext ctx) in c:\\\\Builds\\\\RavenDB-Stable\\\\Raven.Database\\\\Server\\\\HttpServer.cs:line 900\\r\\n   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx) in c:\\\\Builds\\\\RavenDB-Stable\\\\Raven.Database\\\\Server\\\\HttpServer.cs:line 837\\r\\n   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx) in c:\\\\Builds\\\\RavenDB-Stable\\\\Raven.Database\\\\Server\\\\HttpServer.cs:line 609\"\r\n}\r\n"}

and inner exception: The remote server returned an error: (500) Internal Server Error.

[Martin Kristensen]

Right now we have a focus on correcting minor fixes for the customer so this issue has been down priotized but I can take time and look into it tomorrow if it fits your schedule?

[Oren Eini (Ayende Rahien)]

I am going on vacation, so that wouldn't work.

How about Monday with Fitzchak?

[Kijana Woodard]

"I am going on vacation"

Well deserved. Enjoy!

[Martin Kristensen]

I think Monday will do fine. I'm sitting in +1 Timezone (in case that affects available time). 

You can catch me on mak at vertica dot dk

So I will send my skype alias that way or you can send yours.

Thank you for helping.