Windows x64 bareos-fd is crashing during backup
Vasya Pupkin
I have localized Windows version. That`s why I`m sending xml windows event viewer representation.
Please tell me if C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_bareos-fd.exe_2d6044da24cdba66748b38e452eacc144abe4356_0b502d10 is of any use.
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-01-17T15:46:25.000000000Z" />
<EventRecordID>87606</EventRecordID>
<Channel>Application</Channel>
<Computer>XXX-BOX.lan</Computer>
<Security />
</System>
- <EventData>
<Data>bareos-fd.exe</Data>
<Data>1.0.0.0</Data>
<Data>5c1b5fed</Data>
<Data>bareos-fd.exe</Data>
<Data>1.0.0.0</Data>
<Data>5c1b5fed</Data>
<Data>c0000005</Data>
<Data>000000000000ee04</Data>
<Data>1a68</Data>
<Data>01d4ae7656bd96be</Data>
<Data>C:\Program Files\Bareos\bareos-fd.exe</Data>
<Data>C:\Program Files\Bareos\bareos-fd.exe</Data>
<Data>0b3877fa-1a6f-11e9-8100-448a5b8407b9</Data>
</EventData>
</Event>
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Windows Error Reporting" />
<EventID Qualifiers="0">1001</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-01-17T15:46:25.000000000Z" />
<EventRecordID>87607</EventRecordID>
<Channel>Application</Channel>
<Computer>XXX-BOX.lan</Computer>
<Security />
</System>
- <EventData>
<Data />
<Data>0</Data>
<Data>APPCRASH</Data>
<Data>No Data</Data>
<Data>0</Data>
<Data>bareos-fd.exe</Data>
<Data>1.0.0.0</Data>
<Data>5c1b5fed</Data>
<Data>bareos-fd.exe</Data>
<Data>1.0.0.0</Data>
<Data>5c1b5fed</Data>
<Data>c0000005</Data>
<Data>000000000000ee04</Data>
<Data />
<Data />
<Data />
<Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_bareos-fd.exe_2d6044da24cdba66748b38e452eacc144abe4356_0b502d10</Data>
<Data />
<Data>0</Data>
<Data>0b3877fa-1a6f-11e9-8100-448a5b8407b9</Data>
<Data>4</Data>
</EventData>
</Event>
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Bareos" />
<EventID Qualifiers="0">0</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-01-17T15:46:52.000000000Z" />
<EventRecordID>87609</EventRecordID>
<Channel>Application</Channel>
<Computer>XXX-BOX.lan</Computer>
<Security />
</System>
- <EventData>
<Data>Bareos ERROR:</Data>
<Data>lib/bsock_tcp.cc:417 Wrote 6 bytes to File daemon:localhost:9102, but only 0 accepted.</Data>
</EventData>
</Event>
Vasya Pupkin
2019-01-17 18:41:00 bareos-dir JobId 1702: Start Backup JobId 1702, Job=xxx-box.2019-01-17_18.41.00_05
2019-01-17 18:41:00 bareos-dir JobId 1702: Connected Storage daemon at fs32.lan:9103, encryption: ECDHE-PSK-CHACHA20-POLY1305
2019-01-17 18:41:01 bareos-dir JobId 1702: Created new Volume "xxx-box.2019-01-17-18-41.aii_1702" in catalog.
2019-01-17 18:41:01 bareos-dir JobId 1702: Using Device "disk-fs32-r6s1" to write.
2019-01-17 18:41:01 bareos-dir JobId 1702: Probing... (result will be saved until config reload)
2019-01-17 18:41:01 bareos-dir JobId 1702: Connected Client: xxx-box at xxx-box.lan:9102, encryption: PSK-AES256-CBC-SHA
2019-01-17 18:41:01 bareos-dir JobId 1702: Handshake: Immediate TLS 2019-01-17 18:41:01 bareos-dir JobId 1702: Encryption: PSK-AES256-CBC-SHA
2019-01-17 18:41:06 xxx-box JobId 1702: DIR and FD clocks differ by 5 seconds, FD automatically compensating.
2019-01-17 18:41:02 bareos-dir JobId 1702: Sending Accurate information.
2019-01-17 18:41:07 fs32-sd JobId 1702: Labeled new Volume "xxx-box.2019-01-17-18-41.aii_1702" on device "disk-fs32-r6s1" (/_bareos).
2019-01-17 18:41:07 fs32-sd JobId 1702: Wrote label to prelabeled Volume "xxx-box.2019-01-17-18-41.aii_1702" on device "disk-fs32-r6s1" (/_bareos)
2019-01-17 18:41:07 bareos-dir JobId 1702: Max Volume jobs=1 exceeded. Marking Volume "xxx-box.2019-01-17-18-41.aii_1702" as Used.
2019-01-17 18:41:06 xxx-box JobId 1702: Created 28 wildcard excludes from FilesNotToBackup Registry key
2019-01-17 18:41:07 xxx-box JobId 1702: Connected Storage daemon at fs32.lan:9103, encryption: PSK-AES256-CBC-SHA
2019-01-17 18:41:13 xxx-box JobId 1702: Generate VSS snapshots. Driver="Win64 VSS", Drive(s)="C"
2019-01-17 18:41:13 xxx-box JobId 1702: VolumeMountpoints are not processed as onefs = yes.
2019-01-17 18:46:20 bareos-dir JobId 1702: Fatal error: Network error with FD during Backup: ERR=Connection reset by peer
2019-01-17 18:46:20 fs32-sd JobId 1702: Fatal error: stored/append.cc:173 Error reading data header from FD. ERR=Connection reset by peer
2019-01-17 18:46:20 fs32-sd JobId 1702: Releasing device "disk-fs32-r6s1" (/_bareos).
2019-01-17 18:46:20 bareos-dir JobId 1702: Fatal error: No Job status returned from FD.
2019-01-17 18:46:20 bareos-dir JobId 1702: Error: Bareos bareos-dir 18.2.4rc2 (18Dec18):
Build OS: Linux-4.4.92-6.18-default debian Debian GNU/Linux 9.6 (stretch)
JobId: 1702
Job: xxx-box.2019-01-17_18.41.00_05
Backup Level: Incremental, since=2019-01-16 13:00:06
Client: "xxx-box" 18.2.4rc2 (18Dec18) Microsoft Windows 7 Professional Service Pack 1 (build 7601), 64-bit,Cross-compile,Win64
FileSet: "win-all" 2018-07-27 16:59:34
Pool: "aii" (From Job IncPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "disk-fs32-r6" (From Job resource)
Scheduled time: 17-Jan-2019 18:41:00
Start time: 17-Jan-2019 18:41:01
End time: 17-Jan-2019 18:46:20
Elapsed time: 5 mins 19 secs
Priority: 12
FD Files Written: 0
SD Files Written: 0
FD Bytes Written: 0 (0 B)
SD Bytes Written: 0 (0 B)
Rate: 0.0 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: yes
Volume name(s): xxx-box.2019-01-17-18-41.aii_1702
Volume Session Id: 1
Volume Session Time: 1547739520
Last Volume Bytes: 221 (221 B)
Non-fatal FD errors: 1
SD Errors: 0
FD termination status: Error
SD termination status: Running
Termination: *** Backup Error ***
Vasya Pupkin
2019-01-17 21:07:56 bareos-dir JobId 1709: Connected Storage daemon at fs32.lan:9103, encryption: ECDHE-PSK-CHACHA20-POLY1305
2019-01-17 21:07:56 bareos-dir JobId 1709: Created new Volume "xxx-box.2019-01-17-21-07.aii_1709" in catalog.
2019-01-17 21:07:56 bareos-dir JobId 1709: Using Device "disk-fs32-r6s1" to write.
2019-01-17 21:07:56 bareos-dir JobId 1709: Probing... (result will be saved until config reload)
2019-01-17 21:07:56 bareos-dir JobId 1709: Connected Client: xxx-box at xxx-box.lan:9102, encryption: PSK-AES256-CBC-SHA
2019-01-17 21:07:56 bareos-dir JobId 1709: Handshake: Immediate TLS 2019-01-17 21:07:56 bareos-dir JobId 1709: Encryption: PSK-AES256-CBC-SHA
2019-01-17 21:08:02 xxx-box JobId 1709: DIR and FD clocks differ by 5 seconds, FD automatically compensating.
2019-01-17 21:07:57 bareos-dir JobId 1709: Sending Accurate information.
2019-01-17 21:08:02 fs32-sd JobId 1709: Labeled new Volume "xxx-box.2019-01-17-21-07.aii_1709" on device "disk-fs32-r6s1" (/_bareos).
2019-01-17 21:08:02 fs32-sd JobId 1709: Wrote label to prelabeled Volume "xxx-box.2019-01-17-21-07.aii_1709" on device "disk-fs32-r6s1" (/_bareos)
2019-01-17 21:08:02 bareos-dir JobId 1709: Max Volume jobs=1 exceeded. Marking Volume "xxx-box.2019-01-17-21-07.aii_1709" as Used.
2019-01-17 21:08:02 xxx-box JobId 1709: Created 28 wildcard excludes from FilesNotToBackup Registry key
2019-01-17 21:08:02 xxx-box JobId 1709: Connected Storage daemon at fs32.lan:9103, encryption: PSK-AES256-CBC-SHA
2019-01-17 21:08:09 xxx-box JobId 1709: Generate VSS snapshots. Driver="Win64 VSS", Drive(s)="C"
2019-01-17 21:08:09 xxx-box JobId 1709: VolumeMountpoints are not processed as onefs = yes.
2019-01-17 21:14:24 bareos-dir JobId 1709: Fatal error: Network error with FD during Backup: ERR=Connection reset by peer
2019-01-17 21:14:24 fs32-sd JobId 1709: Fatal error: stored/append.cc:173 Error reading data header from FD. ERR=Connection reset by peer
2019-01-17 21:14:24 fs32-sd JobId 1709: Releasing device "disk-fs32-r6s1" (/_bareos).
2019-01-17 21:14:24 bareos-dir JobId 1709: Fatal error: No Job status returned from FD.
2019-01-17 21:14:24 bareos-dir JobId 1709: Error: Bareos bareos-dir 18.2.4rc2 (18Dec18):
Build OS: Linux-4.4.92-6.18-default debian Debian GNU/Linux 9.6 (stretch)
Job: xxx-box.2019-01-17_21.07.54_04
Backup Level: Incremental, since=2019-01-16 13:00:06
Client: "xxx-box" 18.2.4rc2 (18Dec18) Microsoft Windows 7 Professional Service Pack 1 (build 7601), 64-bit,Cross-compile,Win64
FileSet: "win-all" 2018-07-27 16:59:34
Pool: "aii" (From Job IncPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "disk-fs32-r6" (From Job resource)
Start time: 17-Jan-2019 21:07:57
End time: 17-Jan-2019 21:14:24
Elapsed time: 6 mins 27 secs
Priority: 12
FD Files Written: 0
SD Files Written: 0
FD Bytes Written: 0 (0 B)
SD Bytes Written: 0 (0 B)
Rate: 0.0 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: yes
Volume Session Id: 1
Volume Session Time: 1547748438
Last Volume Bytes: 999,936,057 (999.9 MB)
Vasya Pupkin
*setdebug client=xxx-box level=200 trace=1
Connecting to Client xxx-box at xxx-box.lan:9102
2000 OK setdebug=200 trace=1 hangup=0 timestamp=0 tracefile=C:\Windows\TEMP/xxx-box.trace
*run yes job=xxx-box
Job queued. JobId=1713
Last string in trace before crash:
xxx-box (150): filed/fd_plugins.cc:339-1713 No bplugin_list: GeneratePluginEvent ignored.
Andreas Rogge
>
> *setdebug client=xxx-box level=200 trace=1
> Connecting to Client xxx-box at xxx-box.lan:9102
> 2000 OK setdebug=200 trace=1 hangup=0 timestamp=0 tracefile=C:\Windows\TEMP/xxx-box.trace
> *run yes job=xxx-box
Best Regards,
Andreas
--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-0
http://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz
Vasya Pupkin
This host stops crashing. Nothing have changed. I was sleeping. Time passed by. 12 hours. Odd.
My bad. I`ve deleted crash tracefile. But I have lots bareos-fd* dirs in C:\ProgramData\Microsoft\Windows\WER\ReportQueue\. I have lots of them from different hosts. Some of them with memory dumps. Please look at the pictures attached. Are these dumps of any help?
"Luckily" I have other hosts crashing.
Thank you for your help!
Vasya Pupkin
I`ve sent you archive with traces directly to your email. Archive is too large(9 Mbytes) to post it at group.
Thanks!
richman...@gmail.com
this is not bareos crash. This is windows behavior for "not answering
applications"
disable "Windows Error Reporting Service " and/or increase
HungAppTimeout in system registry
Vasya Pupkin
Please look at https://stackoverflow.com/questions/17168982/exception-error-c0000005-in-vc
Exception code c0000005 is the code for an access violation. That means that your program is accessing (either reading or writing) a memory address to which it does not have rights. Most commonly this is caused by:
Accessing a stale pointer. That is accessing memory that has already been deallocated. Note that such stale pointer accesses do not always result in access violations. Only if the memory manager has returned the memory to the system do you get an access violation.
Reading off the end of an array. This is when you have an array of length N and you access elements with index >=N.
To solve the problem you'll need to do some debugging. If you are not in a position to get the fault to occur under your debugger on your development machine you should get a crash dump file and load it into your debugger. This will allow you to see where in the code the problem occurred and hopefully lead you to the solution. You'll need to have the debugging symbols associated with the executable in order to see meaningful stack traces.