Export function; request for statistical data

[Peter Lowham]

i, along with others are looking at issues with the 'Export' function.  I'm learning more about it each time you guys send me feedback.  To that end, if some of you who are using 'Export' could volunteer some data for me to process, I would be most obliged.  I'm trying to understand whether the conditions apply to all B2 models or particular ones; for example, my calculations are base on an early model (2016) B2.  As I don't have access to the newer models I cannot tell if these figures are valid for later models.

Data Requested

Data                                                   Example (my B2)

B2 Model                                                  B2 1TB

Model Year                                                 2016

Backup Device                                        1 GB HDD                (could be HDD/ USB Memory Stick/ SSD etc)

No. of Artists                                                 611

No. of Albums                                             1455

No. of .wav files                                                0

No. of .flac files                                           5664

No. of .mp3 files                                        14131

No. of .aac files                                                 0

Disk space used (GB)                                   213

Export Backup Time taken (Hrs)                  8.75

Require multiple Export runs to complete? (how many runs) 3

Thank you all in advance,

Peter.

[Stephen Winding]

Peter,

Here are screen shots of my backup. I found the day on all folders of Dec.31,1979 very strange. Mac? I don't think so since it is just reading the drive. Music is 161.45GB seems right. Playlist 3kb. I thought the Export didn't  backup Playlist. Funny only one name comes up Christmas. I have other like Classic, Soul, Folk etc. All Music seems to be here. It also plays on my iMac. Anything else you need. I guess I can call this a successful backup. But I'm not willing to try a restore yet if I don't need it. Usually I use backups when transferring  data to a new machine. Don't fix it if any broken so to speak.

Regards

Steve

[David Wheeler]

Hi Steve - me too with this odd date!

Using a Mac to display contents of a Samsung T5 500Gb Solid State Disc used to store b2bexport from USB C on a recent Brennan B2.  T5 was formatted from B2 using USB C, then exported 7831 tracks with artwork etc.  Here's the info summary:

Kind:        Volume 

Created:   31 December 1979 at 23:00

Modified:   31 December 1979 at 23:00

Format:      MS-DOS (FAT32)

Capacity:   499.98 GB

Available:   293.71 GB (11.7 MB purgeable)

Used:         206,286,356,480 bytes (206.29 GB on disk)

odd indeed

David Wheeler. (please ignore earlier premature posting)

[Davywhizz]

Hi Peter

You and I have had a couple of emails about this already but I'll add my latest data to the pot. Incidentally, like Stephen, I've also noticed the date of  December 31 1979 when looking at the properties of my backup drives (on a Windows 10 laptop). Coincidentally, though irrelevantly, it's a memorable date for me: my band had a New Year's gig at a military base and we were held at gunpoint afterwards for trying to help a drunken Corporal go AWOL. Apparently he did it all the time, so the guards were waiting for him at the gate. How we all laughed.

Anyway, I've recently done two clean B2 backups (ie wiping the drives first) to different drives on consecutive days and got exactly the same results:

B2 Model - 480G

Year  - 2020

Backup - 1TB (two of them, both bought from the Brennan shop)

Artists - 373

Albums - 634

All tracks are FLAC

Disc space used - 219GB

Time  -  from memory, around six hours each time

Runs - only one each time

The B2 was, as above, showing 1007 folders (artists plus albums) and the backup drives both showed 1009. I took this to be correct, as it  surely includes the music and playlist folders within the export? 

I've only had my B2 since mid-March and only ever used the basic Brennan export instructions. I had one problem, with the very first backup (at only about 100 CDs): the backup stalled on 528 tracks. At the time I put it down to a very brief power cut we sometimes get here in the sticks, often just a few seconds. After a bit of panic over a frozen B2,  I rebooted and the backup started again at the point it left off, finishing without any trouble. There was nothing wrong with track 528. 

I've obviously been lucky so far, especially as I'm now into much bigger backups. I don't think I do anything out of the ordinary, other than I've always used USB A for backup rather than C. 

My CDs have been ripped in various ways: internal B2 drive, external drive into USB C and, increasingly, using dBpoweramp on my laptop and transferred mostly via USB stick, once by NAS. I've edited the album art, titles etc on the UI and also on the laptop before transfer. I try to remember to do a full, clean backup if I've done any new edits.

Thanks for persevering with this Peter. I'm sure it will prove to be really helpful to us all.

[Peter Lowham]

Hi Davywhizz,

Thank you very much for your data set; this is really useful.

First though, that's a great story!  I'm thinking we should set up a 'Funny Stories' thread on this forum.  It would be a real morale booster especially during these times of isolation!

Like you, I ended up at gunpoint in Bratislava airport in January 1987, while going on a skiing trip to Bulgaria.  It was a Czech soldier on the other end of the Kalasnikov, but that's another story and I lived to tell the tale!  My friends who were with me on that trip still give me a hard time about that incident now, and that was over 30 years ago!

Anyway, back to your stats.

First off can I ask you how many music tracks you have?  Also, do you have some, or a lot of, classical music in your collection? I.m tuning my spreadsheet as I receive more information, and these are relevant factors.

I have keyed your data set into my 'Brennan Export Calculator' spreadsheet.  For your data, this is giving me a predicted 'Export' time of of 6.4 hours which is close to your estimate of 6 hours so that is good.

Your calculation of the number of folders is also good, with one small correction to your assumption.  The difference of 2 folders is down to the content of the \b2Export\music' directory.  The additional two items in this directory are the '.' and '..' entries which are created and used by the B2 (Linux) and the PC and the Mac.  These two entries are always in any directory and are used by the Linux/PC/Mac for interal and external reference purposes.

For example in Linux, at command line level, you sometimes need to to go up to the parent directory and you would use the command 'cd ..' to change into the parent directory. 

I reckon that your 'Export' is in the danger zone and is close to the crash point, so I would advise you to begin using the 4 hour segment time sometime soon, especially if you are adding any more music to your collection.  Your later model B2 will run for longer before crashing, but others who have a collection close to yours in terms of size have already hit the crash point.  Your 480G model is the first that I have stats for so that might be a new factor.

The other thing that I have just found is that it is highly advisable to reboot your B2 once the 'Export' is completed.  I have noticed that the B2 does not appear to be releasing the memory buffers after the 'Export' is complete.  This might not be noticable to the user because the lighter duty processes like playing music do not need that much memory but if you were to start ripping soon after the 'Export' has completed you might find that the B2 locks up.

Anyway, thanks again for the story and the data.

Regards,

Peter.

[Peter Lowham]

Hi Stephen,

At first sight, your Export figures seem to be reasonable.  However, if you have the time, if you could supply the data as shown below, I would be able to give you better feedback.  Just replaced my data below with your equivalent and I'll get back to you fairly quickly.

                             Data Requested

Data                                                   Example (my B2)

B2 Model                                                  B2 1TB

Model Year                                                 2016

Backup Device                                        1 GB HDD                (could be HDD/ USB Memory Stick/ SSD etc)

No. of Artists                                                 611

No. of Albums                                             1455

No. of .wav files                                                0

No. of .flac files                                           5664

No. of .mp3 files                                        14131

No. of .aac files                                                 0

Disk space used on B2 (GB)                        213

Export Backup Time taken (Hrs)                  8.75

Require multiple Export runs to complete? (how many runs) 3

Disk space used on USB backup device (GB)

Regards,

Peter.

[Davywhizz]

Hi Peter

I had 9358 tracks at that stage. I hadn't tackled the classical albums by then, but have since ripped and loaded them. We don't have a lot on CD so that will be it for the B2. I'm ripping all compilations/classical on my laptop with dBpoweramp as I find it easier to edit the listings. 

I've added maybe another 50 CDs in total since those two full backups and did incremental backups both times, which look OK. I've probably got about another 100 CDs to do, so  the B2 will have close to 800 in total. This is why I bought the 480G model as I knew it would be more than enough. We probably won't buy many CDs once I'm done unless I can't find something on Spotify.

I always save and switch off the B2 after backing up, but only because I need to remove the external drive (which I use in USB A) to replace the wifi dongle in order to use the UI. Thus I always get an accidental reboot as you suggest. 

I've got your detailed backup instructions printed off and will give them a go next time I do a full backup. Thanks again Peter.

[Peter Lowham]

Hi All Mac users,

On looking at the 'Export' folder creation date of 31 December 1979, I'm wondering if this date is the Mac O/S 'Day 0'.  All computer systems have a 'Day 0' set (but each one uses a different date for 'Day '0') and then used a date counter field from 'Day 0'  for date calculations, counting both forwards and backwards from that date.  It is stored as an integer.

It might be that the Mac cannot read this date counter field correctly because the folders were created under Linux (the B2's operating system) and therefore uses the value '0' as a default.  This could give you the date that we are seeing.

Perhaps some Mac O/S techie could let us know?

For example, my favourite O/S is called 'Pick' and its 'Day 0' is 31st December 1967. The date counter was a 4 digit field (up to 9999) and we realised, fortunately in good time, that our customers' systems would show 17th May 1995 (Day 9999) on that day but would show 31st December 1967 on the next day, because the date counter would roll over to 10000 but with no capability to store the 5th digit.

We issued a software patch to all customers which was to make the counter field 5 digits, and all was well.

Regards,

Peter.

[Stephen Winding]

Hi Peter,

Sorry here's Data, at least I'm not crazy about Dec 31 1979.

Model    B2 2T

Model Year Feb. 26 2020

Backup 2T Toshiba Canvio

# of Artist 210

# of Albums 542

# of Flac files 7410

# of wav files 0

# of mp3 files 0

# of acc files 0

Disk Space on B2 used 28.01GB Disk space used on Backup 161.45GB

Export a little less then 5hrs ( 1 backup only )

Regards

Steve

[Peter Lowham]

Hi Stephen,

Thanks for the data and here are my results.

The spreadsheet predicts a volume size of 155 GB so your actual file size of 161.45 GB looks good.

The spreadsheet predicts an 'Export' run time of 4.75 hours so that also looks good.

It also predicts the number of folders in the '\b2Export\music' folder should be 752 or greater. 

The one figure that I would look into is B2 'Disk Space used' which is showing as 28,01 GB.  I think that this is being mis-reported by the B2.

I would also say that your 'Export' time is heading into the danger zone, and that if you are adding more music to the B2, then I would look at doing the 'Export' in 2 cycles.

Regards,

Peter.

[Peter Lowham]

Hi Davywhizz,

Thanks for your update.  Let me know how you get on.

Regards,

Peter.

[Stephen Winding]

Peter,

I noticed that Davywhizz is getting the December 31 1979 on a Windows 10 laptop. Just a little for food for thought.

Regards

Steve

[Daniel Taylor]

On Monday, June 8, 2020 at 3:03:19 PM UTC-4, Stephen Winding wrote:

December 31 1979 on a Windows 10 laptop

I wonder if that indicates that the date is not set right on the B2.

[Stephen Winding]

Hi Peter,

Knowing I am In the danger zone. How does this affect incremental backup. Or should we be doing a fresh backup instead. 

Regards

Steve

[Peter Lowham]

Hi Stephen,

Did your previous 'Export'(s) complete without the B2 crashing?  If so, then your existing '\b2\Export\' should be good and you can continue to use it as for 'incremental' backups.

However, one very important point to note is that if you have added any new music from an 'external' source (such as NAS) and you have run 'Scan Disk' since the last 'Export' then you MUST start a new fresh  'Export'. This is because the B2's 'b2db' file is inextricably linked to its 'Export' file so any external music file additions will cause a change to 'b2db', thus renderibg the 'Export' unsuitable for 'incremental' use.

Regards,

Peter.

[Mark Fishman]

I usually use the bulk upload function on the web UI to add music to my B2, and that properly appends to b2db.

BUT:
I also use Andy C's b2db sorting program to alphabetize my b2db file periodically. That will force me to do a full Export instead of an incremental, according to your analysis.

Robocopy is looking better and better.

[Peter Lowham]

Hi Mark,

Yes, you are correct, in my view.  As you state, if you use the 'bulk upload function' this will APPEND to the 'b2db' file which the 'Export' function will deal with perfectly correctly, and the 'incremental' nature of the 'Export' will be preserved.

However, if the ORDER of the existing 'b2db' file is changed in any way, and you try to use an existing  'Export' in 'incremental' mode, the 'Export' process will become totally confused and lost, leading to corrupted 'Export' files.

Robocopy is a really good application because it mirrors the source unconditionally.  The only drawback is that it is a lot slower (for a wired Cat5 B2 connection about 5 x slower than the Export).  Now that I have found a reliable way to use Export, I will use it to create the occasional 'base' builds and then use Robocopy for the day-to-day incremental backups.

Regards,

Peter.

[Margaret Hall]

This morning completed a backup & here's the result:

B2 Model                                                  B2 2TB

Model Year                                               Acquired new from Brennan Feb.2018

Backup Device                                        1 TB Freecom HDD (bought earlier this year I think)              

No. of Artists                                                 720

No. of Albums                                             1637

No. of .wav files                                                0

No. of .flac files                                           19243

No. of .mp3 files                                                0

No. of .aac files                                                 0

Disk space used (GB)                                   489.71

Export Backup Time taken (Hrs)                    12 hrs.& c.40 mins. (+/- 10 mins)

Require multiple Export runs to complete? (how many runs)  No - ran straight through

All seems to be there - random tracks play on my iMAC (which also states 'created 31 Dec.1979'); info. as follows:

Format: MS_Dos (FAT 32)

Capacity: 999.96 GB

Available: 498.04 GB

Used: 501.92 GB.

Note: The Freecom has always loaded without interruption, tho' it usually takes more than 1 effort to get it formatted on the B2 (worked this time at 3rd attempt, after I'd switched the B2 on & off at the wall). Have also used an old Buffalo (4-5 yrs. old) as a 2nd backup, & this stops a few times & has to be re-started, but is only 500GB capacity, so I may not try to use it again.

[Peter Lowham]

Hi Margaret,

Thank you for the stats, these are a great help to us.

From my spreadsheet estimator I have a couple of observations to make.

First, your '\b2Export\music\' folder should have a minimum of 2359 folders, there could be more, but this figure is the absolute minimum number.

Next, your 'Export' backup time is less than I expected, I estimated 14.5 hours.  Given that your Export ran in one run, could you do some checking on your Export file to make sure that the music tracks content is really all there?  The Export file folder 'Properties' should give a 'Disk Size' figure which should be close to the B2's figure of 490 GB that you have provided.

Another query, if you have the time.  Does your collection contain some or a lot of classical music?  My reason for asking is my 'Disk space used' calculator works well for some of the data sets, but is less accurate for others, and I'm wondering if classical music tracks are another factor.

Finally, your comments on the Format function are very interesting.  I too have come across a problem where one of my three external disks just will not format on the B2.  I've put this on my list to be looked at after the Export problem is fully resolved.

Regards,

Peter.

[Davywhizz]

Morning Peter

I've just completed my first full B2 export using your new instructions and here are the numbers:

2 Model                                                  B2 480G

Model Year                                                 2020

Backup Device                                        1 TB HDD                

No. of Artists                                                 385

No. of Albums                                               691

No. of .wav files                                                0

No. of .flac files                                           10370

No. of .mp3 files                                                0

No. of .aac files                                                 0

Disk space used (GB)                                   244

Export Backup Time taken (Hrs)                   After four hours, 6400 tracks, so 1600 per hour. Sorry, I didn't stay up for the second run

Require multiple Export runs to complete?     2 (by choice)

The folders on the B2 (artists plus albums) add up to 1076 and the HDD properties show 1078, which indicates the export is complete. There are 12061 files in the export including the album art etc.

I followed your instructions pretty much exactly, but wasn't sure if I still needed to format the drive as well as cleaning it on my laptop. Your instructions read to me as either/or. I did both, assuming it wouldn't hurt.

Last week I did two full backups to different HDDs using the original Brennan instructions and both seemed to work fine on a slightly smaller volume of data. It wasn't a lot of extra effort to do it with the revised method so I'll use it from now on as the volumes go up.

[Peter Lowham]

Hi Davywhizz,

Thank you for these data; the picture of Export backups is getting better each time.

My spreadsheet estimations give:

Expected time to run Export: 7 hours 10 minutes.

Number of Folders in Export: 1078  (ideal figure - can be greater, if there are 'junk files' in the collection).

When your collection gets to 12000 tracks you will cross the 8 hour Export time.

You don't need to format the drive before doing a new, clean Export.  Just deleting the 'b2Export' directory is perfectly fine. Also, if by any chance there is a directory named 'hardfi', delete that as well, before starting a new Export.

Another additional piece of information is that if you are renaming Artists and/or Albums, and you are using 'Incremental' Export, then you will have a duplicate copy of each renamed folder in the Export.  This is just the way Export works.  So if you are renaming folders, you should take a new clean full Export in order to dispose of the duplicate folders.

I recommended the use of 4 hours maximum per Export run based on my testing which has been on the older B2 (2016 model).  My B2 would run an Export for 5 - 6 hours before crashing (every time) so I settled on 4 hours to be on the (very) safe side.  I ran a test Export yesterday, NOT using the 4 hour rule and, sure enough, the Export crashed after 5 hours and 50 minutes.

I think that the newer models, such as yours will run Export for a longer period, probably up to 6 hours without crashing., However, I am loath to make the recommendation unless I can test it, and I don't have the later model to do the tests.

The difference (dependent on various factors) between the earlier and later models is approximately:

Export backup rate on early models  24 GB per hour

Export backup rate on later models  34 GB per hour

Regards and thanks again,

Peter.

[Daniel Taylor]

I have completed an Export to another hard disk that I was able to get the B2 to format.

My B2 is a newer edition.  I bought it in late 2018 - Nov or Dec, I think.

Software Version Oct 4 2019 10:29:32
B2 Model:                       B2 500 GB HDD, replaced by 2 TB HDD
Backup Device:
Artists:                         364
Albums:                        536
WAV files:                        0
FLAC files:                  7282
MP3 files:                     134
AAC files:                        0
Disc space used:             199.83 GB
No. of runs to complete:          2
Export Backup Time:              ~6 hours (rough estimate)
    1st run:  4:55      was nearly done when I pressed STOP
    2nd run: ~1:00    0.5 < t < 1.4 

I'm glad I did the Export.  When compared to my other two manuallly backed up disks, I found some disparities.  Also, a few songs that I thought I'd imported onto the B2 were missing from the albums that were Imported.  That's a head-scratcher.

[Peter Lowham]

Hi Daniel,

Thank you very much for these data and I have processed the set as follows.

My estimations for your Export backup are:

Estimated Export run time : 5.85 hours

Minimum number of folders : 902 ( could be greater than this depending on the .'cleanliness' of the music collection).

Based on the figures that you have supplied,  your Export looks good.

Regards,

Peter.

[Mike W]

Hi Peter, just a quick question, are you gathering the data with an end objective or just giving individual feedback for each responder?

It would make sense for the B2 to do a final data comparison check having completed the export and report on any errors.

Will the export function be renamed "backup" at any point? I wonder how many users of backup there are who need backwards compatibility with the Jb7, might be worth an ask, this would enable B2 users to use the much more obvious backup function.

Good work ny the way

Regards

Mike

[Daniel Taylor]

One thing I forgot to mention.  The date stamp on each of my backup files is 1/1/1980.  (And the clock is set correctly.)  Curious that it's one day more recent than the 12/31/1979 that was found on another backup.

[Peter Lowham]

Hi Mike,

My data collection came into being some time after I had read a number of posts in the forum where it was apparent that some users were having problems wirth Export, in particular where the Export did not complete, or it did appear to complete but was found to have music files missing.  That was in late 2019, but initially I did not get involved as I was not using Export then (I use NAS).  When I saw thse posts it discouraged me from using the function.

However, I began to look at Export again after I purchased my BB1, because NAS is a lot slower than Export, and I wanted to find a more efficient process to do full music rebuilds.  That was when I found that my B2 could not complete an Export; the B2 always crashed at between 5 and 6 hours running (my Export is a 9 hour run).

The end objective is to assist the Brenan team to idendify and fix the root problem, so the Export functions fully and correctly.

The main resons for the data gathering were/ are:

1. To establish the success or failure of a given Export, especially in regard to larger collections.

2. To try to identify any pattern or reason for a failure or for missing files.

3. To provide a comparison of Export run times (Expected Time v Actual Time) in order to establish the completeness of the run.

4. To advise on how many 'Export Runs' should a given Export be broken into.

With reference to your queries, those are in the hands of the Brennan team.  In the meantime, we have established a work-around (Using the rule that 4 hours should be the maximum Export run time -  then reboot and restart Export - repeat - until Export is complete ) and that is showing to be effective at this time.

Regards,

Peter.

[Peter Lowham]

Hi Daniel,

I think that I have seen the 1/1/1980 date in another post recently, but I cannot recall where.  I'll have a look around and will let you know if I find it. 

Regards,

Peter.

[Peter Lowham]

Hi Folks,

Just to keep everyone updated as to what is happening with the 'Export/ Import' issues, the problem has been identified by the Brennan team.

A fix for the problem is being tested right now and if all is well, it will be released shortly as a 'Web Upgrade'.  So hang in there!

Regards,

Peter.

[Brennan Support]

Hello Peter

Thanks for the feedback about import - I have now applied the fix to import and restore - I have not had time to test with a big data set and the stats I could see seemed a bit mixed so there may be more to do.

The new import fix is available as a web upgrade.

Martin

[Peter Lowham]

Hi Martin, (releted and reposted due to typos and grammatical errors!)

I've just upgraded to 'B2B Jun 29 2020', and have kicked off an Import.  I'll see how it goes through today; it should complete in about 3 hours time if all is well.  I'll send you an update as soon as I have the result.

I've had a look and test of the new '.flac + .mp3' option.  This is really good!  I've been maintaining a separate '.mp3' collection because I have a couple of devices that won't play '.flac'.  This is a really neat and tidy solution; thank you for this!

One small point with this option; when the process completes, after the conversion of the last track in the 'mirror' folder, it exits without deleting the last '.wav' track.

Regards,

Peter.

[Keith]

Good afternoon Peter.

It is good to see your research reflected in the recent update.

I had already supplied data on one backup.  I include it again with two more recent backups, the latest being after the software upgrade.

5 June

162 artists, 449 albums, 0 wav, 7399 flac, 96 mp3, 0 aac, size 156Gb

11 June back up 27 mins

184 artists, 497 albums, 0 wav, 8056 flac, 96 mp3, 0 aac, size 168Gb

30 June back up 1h14 mins

226 artists, 654 albums, 0 wav, 9246 flac, 96 mp3, 0 aac, size 202.6Gb

27 July back up 3h36mins

242 artists, 784 albums, 0 wav, 9941 flac, 96 mp3, 0 aac, size 228.8Gb

The B2 is a Feb 2020 model 480Gb SSD.

The backups are to the previous backup location so the B2 is probably applying an incremental backup each time.

I now have now completed capturing all my CDs, cassettes and LPs but in future I expect the odd one to surface from time to time hiding on a bookshelf.

My next task is to try and understand the Sonos logic for artwork and to overcome its tendency to name artists and albums differently to that shown on the B2.  Probably something to do with tags.

regards

Keith

[Daniel Taylor]

On Tuesday, July 28, 2020 at 8:58:42 AM UTC-4, Keith wrote:

My next task is to try and understand the Sonos logic for artwork and to overcome its tendency to name artists and albums differently to that shown on the B2.  Probably something to do with tags.

Bingo!  

[Peter Lowham]

Hi Keith,

Thanks very much for your data; I'm still collecting and analysing these data for various reasons and this is very useful.

Can I ask you what your software version is; 'B2 xxxxx' or 'B2B xxx' as there is a noticeable 'Export' and 'Import' data transfer rate depending on which of these you are running on? 

Are you using 'Export' or 'Backup' to back up your files?  If you don't need JB7 compatiblity, 'Export' is the recommended back up method.

You are correct is assuming that the backups are incremental, if you are using the same backup devive and location each time. 

Regarding the tags, the Brennan does not use any tag data at all; it uses the artist name/album name and track name data only.  Sonos, on the other hand, does use tag data, and that is why there are differences between the two devices.

Regards,

Peter.

[Keith]

Yes, I use Export to C.

The B2 version is B2B 21 July 2020.  It was updated from version 11 May 2020 v 10.10.33

Keith

[Peter Lowham]

Hi Keith,

That's great, thanks again.

Regards,

Peter.

[Graham Keith]

Can someone please direct me to Andy C's Excel file to sort my albums.  I've got them as a Text Doc via a memory stick, copied on to a spreadsheet, but they're showing in some random number order.  To use the Excel Alpha sort, I'll have to go through each one and delete the number reference before I can do an alpha sort.  Is there a smarter way?

Graham 

[Peter Lowham]

Hi Graham,

Is this the link that you are looking for?

https://groups.google.com/d/msg/brennanb2/H8z874Z1jvg/eIVAPR6DAQAJ

Regards,

Peter.