RavenDB: Smuggler: Import Error -> Exception : System.Net.Http.HttpRequestException: An error occurred while sending the request.

[Ueli Sauter]

Hi all

We have exported our Database called "DynamiteLAB" using Smuggler
- F:\RavenDB\Tools\Raven.Smuggler out http://localhost:8080 "F:\BackupPath\current.dump" --database=DynamiteLAB

After that we got this error while improting the dump on another server:
- F:\RavenDB\Tools\Raven.Smuggler in http://localhost:8080 current.dump --database=DynamiteLAB

Starting to import file: current.dump
Begin reading indexes
Done with reading indexes, total: 12
Begin reading documents
Done with reading documents, total: 14966
Begin reading attachments
Importing attachment foo.jpg
~~~~~~~ TRUNCATED ~~~~~~~
Importing attachment bar.png
Done with reading attachments, total: 6126
Begin reading transformers
Done with reading transformers, total: 0
Begin reading identities
Done with reading identities, total: 86
Imported 14'868 documents and 6'126 attachments, deleted 0 documents and 0 attachments in 460.836 s
Failed to write all results to a server, probably something happened to the server. Exception : System.Net.Http.HttpRequestException: An error
occurred while sending the request. ---> System.Net.WebException: Die Anfrage wurde abgebrochen:
Die Anfrage wurde abgebrochen..
   bei System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
   bei System.Net.Http.HttpClientHandler.StartGettingResponse(RequestState state)
   bei System.Net.Http.HttpClientHandler.<>c__DisplayClass8.<GetRequestStreamCallback>b__6(Task task)
   --- Ende der internen Ausnahmestapelüberwachung ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Raven.Client.Connection.Implementation.HttpJsonRequest.<>c__DisplayClass86_0.<<ExecuteRawRequestAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   bei Raven.Client.Connection.Implementation.HttpJsonRequest.<RunWithAuthRetry>d__38`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Raven.Client.Connection.Implementation.HttpJsonRequest.<ExecuteRawRequestAsync>d__86.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   bei Raven.Client.Document.RemoteBulkInsertOperation.<StartBulkInsertAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Raven.Client.Document.RemoteBulkInsertOperation.<DisposeAsync>d__44.MoveNext()
An error occurred while sending the request.

Note: Sorry, parts of the exeption are in German.

The dump is about 1.3 GB.
The export and the import take about 10 minutes each.
We are using the newest stable "3.5.3-patch-35187 - 2017/01/23" of both the server and the tools (smuggler)
Deployes as Windows Service.
Running on Windows Server 2012 R2
Storage Engine is "Default" (Esent or Voron)

Actually, we could not find any problems after the import. The number of objects is correct. So it seems that the export/import-procedure was working perfectly fine. But why is there an error? Is this a Bug?

Please tell if you need more information.

Thank you very much for your time.

Best Regards

Ueli

[Oren Eini (Ayende Rahien)]

It looks like a timeout after the request is over. Likely it waited on the server to respond but it didn't for some reason.

If the data is there (and the info says yes), you can safely ignore it

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ueli Sauter]

Thanks for your estimation.

Regarding the import/export-scenario this sounds very good and definetly supports our expectation of a successfull procedure.
Regarding the product "RavenDB" itself this is still not really a satisfying solution... Why is there an error when it can be ignored safely?

Is this a Bug?
Can we provide you any additional information?
Should we open up a bug-report within the issue-tracker?

Thanks & best Regards

André

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

If you can import this with Fiddler avaialble, that would give a lot more information, yes.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ueli Sauter]

Well... Today I tried to make the RavenDB.Smuggler.exe to forward traffic over fiddler but failed. Sorry. There is no "--proxy"-option and it was also not working when setting systemwide-proxy-settings in Internet Explorer.

In case you would want me to create a fiddler-file for that, it would be great to tell me how to do it. I mean its easy for regular browser traffic but not for the traffic that gets made by Smuggler.exe.

Besides, the database is about 1.3 GB. The fiddler-file would even be a bit bigger so I guess the solution would be that easily to find. And besides, it would also not be possible to just send you the complete file because it contains all our data objects. Therefor you shlould also please specify in what exactly would be interested in inside the fiddler-file.

Thanks & best regards

[Oren Eini (Ayende Rahien)]

You shouldn't have to do anything if Fiddler is open, it will automatically use it.

If you are running on local host, use localhost.fiddler instead.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ueli Sauter]

Hi Oren

When giving Smuggler the URL http://localhost.fiddler:8080/... it was actually working properly to capture all the traffic in fiddler. Thanks for the help.

But when running it all thrue fiddler, it took easely twice as long and showed another exception at the end.

~~~~TRUNCATED ~~~~
Importing attachment media/furttalmesse/innen_aussen.pdf
Done with reading attachments, total: 6254

Begin reading indexes
Done with reading indexes, total: 12

Begin reading transformers
Done with reading transformers, total: 0
Begin reading identities

Done with reading identities, total: 88
Imported 15'151 documents and 6'254 attachments, deleted 0 documents and 0 attachments in 1'097.376 s


Failed to write all results to a server, probably something happened to the server. Exception : Raven.Abstractions.Connection.ErrorResponseException: The client failed to send a complete request on this NEW connection before the timeout period elapsed; 1076240 bytes were read from client.

---> System.InvalidOperationException: Could not parse json:
The client failed to send a complete request on this NEW connection before the timeout period elapsed; 1076240 bytes were read from client.
---> Raven.Imports.Newtonsoft.Json.JsonReaderException: Unexpected character encountere
d while parsing value: T. Path '', line 0, position 0.
   bei Raven.Imports.Newtonsoft.Json.JsonTextReader.ParseValue() in C:\Builds\RavenDB-3.5-Patch\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:Zeile 935.
   bei Raven.Imports.Newtonsoft.Json.JsonTextReader.ReadInternal() in C:\Builds\RavenDB-3.5-Patch\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:Zeile 393.
   bei Raven.Imports.Newtonsoft.Json.JsonTextReader.Read() in C:\Builds\RavenDB-3.5-Patch\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:Zeile 314.
   bei Raven.Json.Linq.RavenJObject.Load(JsonReader reader) in C:\Builds\RavenDB-3.5-Patch\Raven.Abstractions\Json\Linq\RavenJObject.cs:Zeile 191.
   bei Raven.Json.Linq.RavenJObject.Parse(String json) in C:\Builds\RavenDB-3.5-Patch\Raven.Abstractions\Json\Linq\RavenJObject.cs:Zeile 276.

   --- Ende der internen Ausnahmestapelüberwachung ---

   bei Raven.Json.Linq.RavenJObject.Parse(String json) in C:\Builds\RavenDB-3.5-Patch\Raven.Abstractions\Json\Linq\Raven
JObject.cs:Zeile 282.
   bei Raven.Client.Extensions.HttpJsonRequestExtensions.<AssertNotFailingResponse>d__0.MoveNext()

   --- Ende der internen Ausnahmestapelüberwachung ---

   bei Raven.Client.Extensions.HttpJsonRequestExtensions.<AssertNotFailingResponse>d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Raven.Client.Document.RemoteBulkInsertOperation.<StartBulkInsertAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Raven.Client.Document.RemoteBulkInsertOperation.<DisposeAsync>d__44.MoveNext()

The client failed to send a complete request on this NEW connection before the timeout period elapsed; 1076240 bytes were read from client.

 StatusCode: 408, ReasonPhrase: 'Request Timed Out', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Connection: close
  Timestamp: 12:56:58.637
  Cache-Control: must-revalidate, no-cache
  Date: Tue, 21 Feb 2017 11:56:58 GMT
  Content-Type: text/html; charset=UTF-8
}

F:\Export,Import>pause

As an attachment, you can find the fiddler-file in form of a password-protected zip-file.

Please notice: I have deleted almost every attachment, replaced the body from within the "bulkInsert" (documents) and have also deleted some of the indexes that have been created from within the capture. Therefor, the fiddler-file should only include content that you really need to troubleshoot the issue.

I hope this helps.

Thanks and best Regards

André

[Ueli Sauter]

BTW:
I could not find an email-address of you, therefor, I sent you the password via the contact-form over here: https://ravendb.net/contact/contact

Thanks

[Maxim Buryak]

Hi,

I have a suspicion about the reson of the error, can you try performing the import without the attachments?

You can do that by adding  "--operate-on-types=Indexes,Documents" to the params.

For more details, look here: 

https://ravendb.net/docs/article-page/3.5/csharp/server/administration/exporting-and-importing-data

Best Regards,

Hibernating Rhinos Ltd  

Maxim Buryak l Core Team Developer l Mobile:+972-54-217-7751

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/  

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ueli Sauter]

Hi Oren, Hi Maxim

Thanks for your answer. Sorry for ansering so late but we are currently only working on weekends on this project....

Here is the console-output:

F:\Export,Import>F:\RavenDB\Tools\Raven.Smuggler in http://localhost.fiddler:8080 current.dump --database=DynamiteLAB --operate-on-types=Indexes,Documents

Starting to import file: current.dump

Begin reading documents
Done with reading documents, total: 15151
Begin reading attachments
Done with reading attachments, total: 0

Begin reading indexes
Done with reading indexes, total: 12
Begin reading transformers
Done with reading transformers, total: 0
Begin reading identities
Done with reading identities, total: 88

Imported 15'151 documents and 0 attachments, deleted 0 documents and 0 attachments in 103.466 s

Failed to write all results to a server, probably something happened to the server. Exception : Raven.Abstractions.Connection.ErrorResponseException:
The client failed to send a complete request on this NEW connection before the timeout period elapsed; 1076240 bytes were read from client.
---> System.InvalidOperationException: Could not parse json:
The client failed to send a complete request on this NEW connection before the timeout period elapsed; 1076240 bytes were read from client.

---> Raven.Imports.Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: T. Path '', line 0, position 0.

   bei Raven.Imports.Newtonsoft.Json.JsonTextReader.ParseValue() in C:\Builds\RavenDB-3.5-Patch\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:Zeile 935.
   bei Raven.Imports.Newtonsoft.Json.JsonTextReader.ReadInternal() in C:\Builds\RavenDB-3.5-Patch\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:Zeile 393.
   bei Raven.Imports.Newtonsoft.Json.JsonTextReader.Read() in C:\Builds\RavenDB-3.5-Patch\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:Zeile 314.
   bei Raven.Json.Linq.RavenJObject.Load(JsonReader reader) in C:\Builds\RavenDB-3.5-Patch\Raven.Abstractions\Json\Linq\RavenJObject.cs:Zeile 191.
   bei Raven.Json.Linq.RavenJObject.Parse(String json) in C:\Builds\RavenDB-3.5-Patch\Raven.Abstractions\Json\Linq\RavenJObject.cs:Zeile 276.
   --- Ende der internen Ausnahmestapelüberwachung ---

   bei Raven.Json.Linq.RavenJObject.Parse(String json) in C:\Builds\RavenDB-3.5-Patch\Raven.Abstractions\Json\Linq\RavenJObject.cs:Zeile 282.

   bei Raven.Client.Extensions.HttpJsonRequestExtensions.<AssertNotFailingResponse>d__0.MoveNext()
   --- Ende der internen Ausnahmestapelüberwachung ---
   bei Raven.Client.Extensions.HttpJsonRequestExtensions.<AssertNotFailingResponse>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Raven.Client.Document.RemoteBulkInsertOperation.<StartBulkInsertAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   bei System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Raven.Client.Document.RemoteBulkInsertOperation.<DisposeAsync>d__44.MoveNext()
The client failed to send a complete request on this NEW connection before the timeout period elapsed; 1076240 bytes were read from client.

 StatusCode: 408, ReasonPhrase: '

Request Timed Out', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Connection: close
  Timestamp: 20:06:22.876
  Cache-Control: must-revalidate, no-cache
  Date: Sun, 05 Mar 2017 19:06:22 GMT

  Content-Type: text/html; charset=UTF-8
}

F:\Export,Import>pause

Drücken Sie eine beliebige Taste . . .

So it once again reached a time-limit, it seems...

Please notice that the console-line "Begin reading attachments" was the one that took so long. So I think that the switch "--operate-on-types=Indexes,Documents" is realy only not importing the attachments but still reading all the attachments. This also takes some time. Therefor this switch seems not to help a lot.

The corresponding fiddler-file can be found attached to this post. The password has once again been sent via the contact-form, because I did not write down the password from last time. -.- This time I wrote it down for eventual further file-transfer.

Best regards

André

[Ueli Sauter]

Hi Oren, Hi Maxim

An additional small update. I just tried to delete all the attachments but one. Then exported it all and re-imported it.

This is the successfull console-output:

F:\Export,Import>F:\RavenDB\Tools\Raven.Smuggler in http://localhost.fiddler:8080 current.dump --database=DynamiteLAB

Starting to import file: current.dump
Begin reading documents

Done with reading documents, total: 8898
Begin reading attachments
Importing attachment TRUNCATED.png
Done with reading attachments, total: 1

Begin reading indexes
Done with reading indexes, total: 12
Begin reading transformers
Done with reading transformers, total: 0
Begin reading identities
Done with reading identities, total: 88

Imported 8'898 documents and 1 attachments, deleted 0 documents and 0 attachments in 3.576 s
Wrote 1'008 [2'496 kb] (total 8'898) documents to server gzipped to 431 kb in 6.4 sec.

F:\Export,Import>pause
Drücken Sie eine beliebige Taste . . .

So it really seems to have to do with the amout of attachments. Dont know if it is the size or the time it uses to import, but since it is working perfectly fine after deleting all attachments but one, it can only be the amount or time, I guess...

You can also find the corresponding fiddler-file attached to this post (same password).

Please tell he how I can futher support you.

Thanks & best regards

André

[Oren Eini (Ayende Rahien)]

Yes, it looks like we aren't heartbeating the connection while this is going on, and if reading the attachments takes long enough, the request will die.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Oren Eini (Ayende Rahien)]

http://issues.hibernatingrhinos.com/issue/RavenDB-6487

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Ueli Sauter]

Hi Oren, Hi Maxim

Thanks for your investigation.

I am wondering, if this also solves the initial request/error. We will see... Looking forward to play arround with the new Smuggler.exe as soon as the fix is available.

Thanks in advance & best regards

André

http://issues.hibernatingrhinos.com/issue/RavenDB-6487

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Ueli Sauter]

Hi all

The bug seems to be fixed, right?
- https://github.com/ravendb/ravendb/pull/1822

When can we expect a new Smuggler-version?
Can I maybe support you by testing it somehow?

[Oren Eini (Ayende Rahien)]

This has been fixed and is part of the latest build

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.