Raven.Server.exe crashed during backup
136 views
Skip to first unread message
Paul Hinett
Jul 31, 2012, 7:00:32 PM7/31/12
to rav...@googlegroups.com
I was running a script against my ravendb server, cleaning up some
unique constraint docs that have got messed up...nothing major, then my
db server crashed. Raven.Backup.exe was also 50% through a scheduled
full backup of the database too.
I had plenty of memory available on the server too.
Here is the event log:
Application: Raven.Server.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the
.NET Runtime at IP 000007FEF67E26FE (000007FEF67E0000) with exit code
80131506.
--
Faulting application name: Raven.Server.exe, version: 1.0.960.0, time
stamp: 0x4fca12e8
Faulting module name: clr.dll, version: 4.0.30319.269, time stamp:
0x4ee9d6e1
Exception code: 0xc0000005
Fault offset: 0x00000000000026fe
Faulting process id: 0x414
Faulting application start time: 0x01cd6e974052c2ee
Faulting application path: C:\RavenDb\Server\Raven.Server.exe
Faulting module path:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: e239f20c-db60-11e1-8a49-002590773ddf
---
The above probably doesn't give much away, but it did just restart the
service unexpectedly. I had some other event log entries saying the log
files had been replayed successfully or something along them lines too.
My indexes and data was fine when it did come back online.
Not sure if this is related or not, but i have a scheduled task setup to
backup my tenant database 3 times per day to date/timed folders, each
day probably 1 or 2 times out of 3 the backup never seems to complete
and stops half way though from looking at the backup folders (there is
only a Indexs + IndexDefinitions folders, no new folder)...i'm just
wondering weather it's failing due to the server restarting unexpectedly.
Using stable 960, running as a windows service.
Paul
unique constraint docs that have got messed up...nothing major, then my
db server crashed. Raven.Backup.exe was also 50% through a scheduled
full backup of the database too.
I had plenty of memory available on the server too.
Here is the event log:
Application: Raven.Server.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the
.NET Runtime at IP 000007FEF67E26FE (000007FEF67E0000) with exit code
80131506.
--
Faulting application name: Raven.Server.exe, version: 1.0.960.0, time
stamp: 0x4fca12e8
Faulting module name: clr.dll, version: 4.0.30319.269, time stamp:
0x4ee9d6e1
Exception code: 0xc0000005
Fault offset: 0x00000000000026fe
Faulting process id: 0x414
Faulting application start time: 0x01cd6e974052c2ee
Faulting application path: C:\RavenDb\Server\Raven.Server.exe
Faulting module path:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: e239f20c-db60-11e1-8a49-002590773ddf
---
The above probably doesn't give much away, but it did just restart the
service unexpectedly. I had some other event log entries saying the log
files had been replayed successfully or something along them lines too.
My indexes and data was fine when it did come back online.
Not sure if this is related or not, but i have a scheduled task setup to
backup my tenant database 3 times per day to date/timed folders, each
day probably 1 or 2 times out of 3 the backup never seems to complete
and stops half way though from looking at the backup folders (there is
only a Indexs + IndexDefinitions folders, no new folder)...i'm just
wondering weather it's failing due to the server restarting unexpectedly.
Using stable 960, running as a windows service.
Paul
Oren Eini (Ayende Rahien)
Aug 1, 2012, 7:49:55 AM8/1/12
to rav...@googlegroups.com
When you say, "db server crashed" , do you mean the service? The actual service?
I am not sure what we can do about: "
an internal error in the .NET Runtime ", at least not unless we have a repro.
About the backups, are you running incremental or full? What do you mean, doesn't stop?
Paul Hinett
Aug 1, 2012, 8:24:03 AM8/1/12
to rav...@googlegroups.com
It looks as if the actual raven.server.exe crashed, i'm not sure how it
restarted it self though unless thats by design...maybe this was a
one-off, i'll keep an eye over it.
Backups are full backups, i just run a test backup from the command
prompt and got the following message which is why it's failing. Not
sure why it works sometimes though and not others...what can i do to
fix this?
Failed to complete backup because: Could not find file
'C:\RavenDb\Server\Tenats\house-mixes\backupTempDirectories\711f...\_v_mun.del
Paul
> Faulting application path: C:\RavenDb\Server\Raven.__Server.exe
> Faulting module path:
> C:\Windows\Microsoft.NET\__Framework64\v4.0.30319\clr.dll
> Report Id: e239f20c-db60-11e1-8a49-__002590773ddf
restarted it self though unless thats by design...maybe this was a
one-off, i'll keep an eye over it.
Backups are full backups, i just run a test backup from the command
prompt and got the following message which is why it's failing. Not
sure why it works sometimes though and not others...what can i do to
fix this?
Failed to complete backup because: Could not find file
'C:\RavenDb\Server\Tenats\house-mixes\backupTempDirectories\711f...\_v_mun.del
Paul
> Faulting module path:
> C:\Windows\Microsoft.NET\__Framework64\v4.0.30319\clr.dll
> Report Id: e239f20c-db60-11e1-8a49-__002590773ddf
Oren Eini (Ayende Rahien)
Aug 1, 2012, 9:55:35 AM8/1/12
to rav...@googlegroups.com
This is using 960, right?
Paul Hinett
Aug 1, 2012, 9:57:09 AM8/1/12
to rav...@googlegroups.com
Yes.
On 01 August 2012 14:55:35, Oren Eini (Ayende Rahien) wrote:
> This is using 960, right?
>
> On Wed, Aug 1, 2012 at 3:24 PM, Paul Hinett
> <pa...@ukcreativedesigns.com <mailto:pa...@ukcreativedesigns.com>> wrote:
>
> It looks as if the actual raven.server.exe crashed, i'm not sure
> how it restarted it self though unless thats by design...maybe
> this was a one-off, i'll keep an eye over it.
>
> Backups are full backups, i just run a test backup from the
> command prompt and got the following message which is why it's
> failing. Not sure why it works sometimes though and not
> others...what can i do to fix this?
>
> Failed to complete backup because: Could not find file
> 'C:\RavenDb\Server\Tenats\__house-mixes\__backupTempDirectories\711f...\___v_mun.del
>
> Paul
>
>
>
> On 01 August 2012 <tel:2012> 12:49:55, Oren Eini (Ayende Rahien)
> <pa...@ukcreativedesigns.com
> <mailto:pa...@ukcreativedesigns.com>
> <mailto:paul@__ukcreativedesigns.com
> C:\RavenDb\Server\Raven.____Server.exe
> Faulting module path:
> C:\Windows\Microsoft.NET\____Framework64\v4.0.30319\clr.dll
> Report Id: e239f20c-db60-11e1-8a49-____002590773ddf
On 01 August 2012 14:55:35, Oren Eini (Ayende Rahien) wrote:
> This is using 960, right?
>
> On Wed, Aug 1, 2012 at 3:24 PM, Paul Hinett
> <pa...@ukcreativedesigns.com <mailto:pa...@ukcreativedesigns.com>> wrote:
>
> It looks as if the actual raven.server.exe crashed, i'm not sure
> how it restarted it self though unless thats by design...maybe
> this was a one-off, i'll keep an eye over it.
>
> Backups are full backups, i just run a test backup from the
> command prompt and got the following message which is why it's
> failing. Not sure why it works sometimes though and not
> others...what can i do to fix this?
>
> Failed to complete backup because: Could not find file
>
> Paul
>
>
>
> On 01 August 2012 <tel:2012> 12:49:55, Oren Eini (Ayende Rahien)
> wrote:
>
> When you say, "db server crashed" , do you mean the service? The
> actual service?
>
> I am not sure what we can do about: " an internal error in the
> .NET
> Runtime ", at least not unless we have a repro.
>
> About the backups, are you running incremental or full? What
> do you
> mean, doesn't stop?
>
> On Wed, Aug 1, 2012 <tel:2012> at 2:00 AM, Paul Hinett
>
> When you say, "db server crashed" , do you mean the service? The
> actual service?
>
> I am not sure what we can do about: " an internal error in the
> .NET
> Runtime ", at least not unless we have a repro.
>
> About the backups, are you running incremental or full? What
> do you
> mean, doesn't stop?
>
> <pa...@ukcreativedesigns.com
> <mailto:pa...@ukcreativedesigns.com>
> <mailto:paul@__ukcreativedesigns.com
> Faulting module path:
> C:\Windows\Microsoft.NET\____Framework64\v4.0.30319\clr.dll
> Report Id: e239f20c-db60-11e1-8a49-____002590773ddf
Oren Eini (Ayende Rahien)
Aug 2, 2012, 4:29:42 AM8/2/12
to rav...@googlegroups.com
Can you check the log, it will have the full message there.
<mailto:paul@ukcreativedesigns.com>
<mailto:paul@__ukcreativedesigns.com
Oren Eini (Ayende Rahien)
Aug 2, 2012, 4:30:15 AM8/2/12
to rav...@googlegroups.com
Next build will also have better diagnostics for a possible failure mode.
Paul Hinett
Aug 13, 2012, 12:11:40 PM8/13/12
to rav...@googlegroups.com
Just coming back to this, i still am having the issue of some backups
not starting.
Here is a the exception from the log file:
System.IO.FileNotFoundException: Could not find file
'C:\RavenDb\Server\Tenants\house-mixes\BackupTempDirectories\c3dc919746cb4d168c3c2ef4e2ff86fd\_in31_d9.del'.
File name:
'C:\RavenDb\Server\Tenants\house-mixes\BackupTempDirectories\c3dc919746cb4d168c3c2ef4e2ff86fd\_in31_d9.del'
at System.IO.__Error.WinIOError(Int32 errorCode, String
maybeFullPath)
at System.IO.FileInfo.get_Length()
at Raven.Database.Backup.DirectoryBackup.Prepare() in
c:\Builds\RavenDB-Stable\Raven.Database\Backup\DirectoryBackup.cs:line
134
at Raven.Storage.Esent.Backup.BackupOperation.Execute(Object
ignored) in
c:\Builds\RavenDB-Stable\Raven.Storage.Esent\Backup\BackupOperation.cs:line
75
As i said, it works 2 out of 3 times, with no consistency of when it
fails (so it seems).
On 02 August 2012 09:30:15, Oren Eini (Ayende Rahien) wrote:
> Next build will also have better diagnostics for a possible failure mode.
>
> On Thu, Aug 2, 2012 at 11:29 AM, Oren Eini (Ayende Rahien)
> <aye...@ayende.com <mailto:aye...@ayende.com>> wrote:
>
> Can you check the log, it will have the full message there.
>
>
> On Wed, Aug 1, 2012 <tel:2012> at 4:57 PM, Paul Hinett
> <pa...@ukcreativedesigns.com <mailto:pa...@ukcreativedesigns.com>>
> wrote:
>
> Yes.
>
>
> On 01 August 2012 <tel:2012> 14:55:35, Oren Eini (Ayende
> <pa...@ukcreativedesigns.com
> <mailto:pa...@ukcreativedesigns.com>
> <mailto:paul@__ukcreativedesigns.com
>
> Paul
>
>
>
> On 01 August 2012 <tel:2012> <tel:2012 <tel:2012>>
> <mailto:paul@>__ukcreativedesig__ns.com
> <http://ukcreativedesigns.com>
>
> <mailto:paul@__ukcreativedesigns.com
> C:\RavenDb\Server\Raven.______Server.exe
> Faulting module path:
>
> C:\Windows\Microsoft.NET\______Framework64\v4.0.30319\clr.dll
> Report Id:
> e239f20c-db60-11e1-8a49-______002590773ddf
not starting.
Here is a the exception from the log file:
System.IO.FileNotFoundException: Could not find file
'C:\RavenDb\Server\Tenants\house-mixes\BackupTempDirectories\c3dc919746cb4d168c3c2ef4e2ff86fd\_in31_d9.del'.
File name:
'C:\RavenDb\Server\Tenants\house-mixes\BackupTempDirectories\c3dc919746cb4d168c3c2ef4e2ff86fd\_in31_d9.del'
at System.IO.__Error.WinIOError(Int32 errorCode, String
maybeFullPath)
at System.IO.FileInfo.get_Length()
at Raven.Database.Backup.DirectoryBackup.Prepare() in
c:\Builds\RavenDB-Stable\Raven.Database\Backup\DirectoryBackup.cs:line
134
at Raven.Storage.Esent.Backup.BackupOperation.Execute(Object
ignored) in
c:\Builds\RavenDB-Stable\Raven.Storage.Esent\Backup\BackupOperation.cs:line
75
As i said, it works 2 out of 3 times, with no consistency of when it
fails (so it seems).
On 02 August 2012 09:30:15, Oren Eini (Ayende Rahien) wrote:
> Next build will also have better diagnostics for a possible failure mode.
>
> On Thu, Aug 2, 2012 at 11:29 AM, Oren Eini (Ayende Rahien)
> <aye...@ayende.com <mailto:aye...@ayende.com>> wrote:
>
> Can you check the log, it will have the full message there.
>
>
> <pa...@ukcreativedesigns.com <mailto:pa...@ukcreativedesigns.com>>
> wrote:
>
> Yes.
>
>
> On 01 August 2012 <tel:2012> 14:55:35, Oren Eini (Ayende
> Rahien) wrote:
>
> This is using 960, right?
>
> On Wed, Aug 1, 2012 <tel:2012> at 3:24 PM, Paul Hinett
>
> This is using 960, right?
>
> <pa...@ukcreativedesigns.com
> <mailto:pa...@ukcreativedesigns.com>
> <mailto:paul@__ukcreativedesigns.com
> <mailto:pa...@ukcreativedesigns.com>>> wrote:
>
> It looks as if the actual raven.server.exe crashed,
> i'm not sure
> how it restarted it self though unless thats by
> design...maybe
> this was a one-off, i'll keep an eye over it.
>
> Backups are full backups, i just run a test backup
> from the
> command prompt and got the following message which is
> why it's
> failing. Not sure why it works sometimes though and not
> others...what can i do to fix this?
>
> Failed to complete backup because: Could not find file
>
> 'C:\RavenDb\Server\Tenats\____house-mixes\____backupTempDirectories\711f...\_____v_mun.del
>
> It looks as if the actual raven.server.exe crashed,
> i'm not sure
> how it restarted it self though unless thats by
> design...maybe
> this was a one-off, i'll keep an eye over it.
>
> Backups are full backups, i just run a test backup
> from the
> command prompt and got the following message which is
> why it's
> failing. Not sure why it works sometimes though and not
> others...what can i do to fix this?
>
> Failed to complete backup because: Could not find file
>
>
> Paul
>
>
>
> On 01 August 2012 <tel:2012> <tel:2012 <tel:2012>>
> 12:49:55, Oren Eini (Ayende Rahien)
>
> wrote:
>
> When you say, "db server crashed" , do you mean
> the service? The
> actual service?
>
> I am not sure what we can do about: " an internal
> error in the
> .NET
> Runtime ", at least not unless we have a repro.
>
> About the backups, are you running incremental or
> full? What
> do you
> mean, doesn't stop?
>
> On Wed, Aug 1, 2012 <tel:2012> <tel:2012
>
> wrote:
>
> When you say, "db server crashed" , do you mean
> the service? The
> actual service?
>
> I am not sure what we can do about: " an internal
> error in the
> .NET
> Runtime ", at least not unless we have a repro.
>
> About the backups, are you running incremental or
> full? What
> do you
> mean, doesn't stop?
>
> <mailto:pa...@ukcreativedesigns.com>
> <mailto:paul@__ukcreativedesigns.com
> <mailto:pa...@ukcreativedesigns.com>>
> <mailto:paul@
> <mailto:paul@__ukcreativedesigns.com
> <mailto:pa...@ukcreativedesigns.com>>
> <mailto:paul@>__ukcreativedesig__ns.com
> <http://ukcreativedesigns.com>
>
> <mailto:paul@__ukcreativedesigns.com
> Faulting module path:
>
> C:\Windows\Microsoft.NET\______Framework64\v4.0.30319\clr.dll
> Report Id:
> e239f20c-db60-11e1-8a49-______002590773ddf
Oren Eini (Ayende Rahien)
Aug 16, 2012, 12:06:36 PM8/16/12
to rav...@googlegroups.com
Thanks, I changed the backup process to be more robust.
On Mon, Aug 13, 2012 at 7:11 PM, Paul Hinett <pa...@ukcreativedesigns.com> wrote:
Just coming back to this, i still am having the issue of some backups not starting.
Here is a the exception from the log file:
System.IO.FileNotFoundException: Could not find file 'C:\RavenDb\Server\Tenants\house-mixes\BackupTempDirectories\c3dc919746cb4d168c3c2ef4e2ff86fd\_in31_d9.del'.
File name: 'C:\RavenDb\Server\Tenants\house-mixes\BackupTempDirectories\c3dc919746cb4d168c3c2ef4e2ff86fd\_in31_d9.del'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileInfo.get_Length()
at Raven.Database.Backup.DirectoryBackup.Prepare() in c:\Builds\RavenDB-Stable\Raven.Database\Backup\DirectoryBackup.cs:line 134
at Raven.Storage.Esent.Backup.BackupOperation.Execute(Object ignored) in c:\Builds\RavenDB-Stable\Raven.Storage.Esent\Backup\BackupOperation.cs:line 75
As i said, it works 2 out of 3 times, with no consistency of when it fails (so it seems).
Next build will also have better diagnostics for a possible failure mode.
On Thu, Aug 2, 2012 at 11:29 AM, Oren Eini (Ayende Rahien)<aye...@ayende.com <mailto:aye...@ayende.com>> wrote:On Wed, Aug 1, 2012 <tel:2012> at 4:57 PM, Paul Hinett
Can you check the log, it will have the full message there.
wrote:
Yes.
On 01 August 2012 <tel:2012> 14:55:35, Oren Eini (AyendeOn Wed, Aug 1, 2012 <tel:2012> at 3:24 PM, Paul Hinett
Rahien) wrote:
This is using 960, right?
<pa...@ukcreativedesigns.com
<mailto:paul@ukcreativedesigns.com>
<mailto:paul@ukcreativedesigns.com>
<mailto:paul@__ukcreativedesigns.com
<mailto:paul@ukcreativedesigns.com>>
<mailto:paul@
<mailto:paul@>__ukcreativedesig__ns.com
<http://ukcreativedesigns.com>
Reply all
Reply to author
Forward
0 new messages