Sorting your music library & analysing the accuracy of your Backup

1,087 views
Skip to first unread message

AndyC

unread,
Feb 21, 2021, 11:50:38 AM2/21/21
to Brennan Forum

Hi All,

I’m pleased to release the latest update of my Music Sorting & Backup Analysis app – Version 8.1.

https://www.dropbox.com/s/g6piekjcfpbuo82/b2%2BBackUpSort8.1.zip?dl=0

To existing users, this release adds a few bug fixes and some extra features such as the ability to group Artists and Albums together under Themes, such as <Party> or [Holidays] etc.

To any who are new to this, it’s a freeware app compiled using QB64.org that enables you to sort the order in which your music library plays and hence the order in which it is displayed by the GUI. It’s much more comprehensive than the b2’s built-in A-Z Sorting command. You use it by downloading the b2db file from your b2, reordering it in your Windows PC using this app and then reloading b2db & rebooting. There are full instructions in the Read Me file. It works for all versions of Windows from XP onwards.

It arranges your artists and albums (within a given artist) to display and thus play alphabetically by default. You have the option to override this so it plays in a different chosen order, e.g. chronologically or thematically or in any other order you choose. On average, the app takes less than 30 seconds to analyse & reorder your b2db file ready for it to be reloaded into your b2.

------------------------------------

Additionally & very usefully, it can also analyse a recent b2 Backup on an external HDD or SSD by comparing it with the b2’s current library to see how ‘fit for purpose’ it is. The problem is that b2’s Export function is incremental and so any renaming of your library causes additional copies of that music to be backed up under their new names, but the originals are not deleted. Therefore you effectively have 2 copies of the same music in the Backup but you likely have forgotten all your edits so don’t know what to delete. This app solves that by also exporting a text file telling you where the unwanted duplicates are in the Backup.

This means you no longer have to completely delete an older backup and start again, you can instead analyse it and then manually edit it on your PC using the data my app provides. This saves having to commit to a time-consuming new full Backup and also saves on rewrite cycles if using an SSD (as they have a limited number of these by design).

One other thing it does whilst analysing your Backup is to record the duration of each album and track for your information.

--------------------------------------

For those who generally play their libraries in a random order or use lots of playlists, the Sorting part of this app is probably not of much use. But, for others like me who don’t do these things, it will hopefully improve yet further your enjoyment of your b2.

And the Backup verification feature is useful to every user who likes to know they have a Plan B in place should their b2’s HDD go AWOL.

PMB

unread,
Feb 22, 2021, 5:21:48 AM2/22/21
to Brennan Forum
Hi Andy C,

Thank you for posting this. Will check it out.

Paul
Brennan Support.

James Bottomley

unread,
Feb 27, 2021, 1:41:48 PM2/27/21
to Brennan Forum
Hi Andy, this sounds great and I'm going to try it.

Will it allow me to sort by date order added from the most recent downwards?

thanks
James

AndyC

unread,
Feb 28, 2021, 11:57:44 AM2/28/21
to Brennan Forum
Hi James,
 
As the Read Me file explains, the program’s default is a full alphabetic sort but a numerical prefix over-rides this. It looks for a leading number, possibly followed by a single letter suffix and then followed by a space. After that, it’s alphabetic sorting.
 
Earlier versions of the program only used partial alphabetic sorting so the letter suffix was designed to cope with double Cds and box sets - e.g
 
2A This Album Cd1
2B This Album Cd2
 
Now, even if you omitted the letter suffix, this example would still sort correctly using the current full alphabetical sorting. Then the numerical prefix would simply position these 2 albums correctly in the full album list.)
--------------------------------------------------------------------------------------------
 
So, to sort by date, you can do what I do – it’s nice & simple as no sorting is done after the initial number n-
 
1 Tubular Bells (1973)
2 Hergest Ridge (1974)
3 Ommadawn (8-10-75)
4 Stuttgart – Live 2.3.76
etc.
 
I originally wrote the program with the prefix representing the number of albums any artist could have and I set the limit at 999. If therefore you use the date as the prefix, it has to be less than this.
 
So this won’t work:
1973 Tubular Bells
1974 Hergest Ridge
1975 Ommadawn
 
But this will:
73 Tubular Bells
74 Hergest Ridge
75 Ommadawn
 
If there is more than one album from a given year, you could use the letter suffix -
 
73 Tubular Bells
74 Hergest Didge
75A (Oct) Ommadawn
75B (Dec) Live at BBC Cd1
75C (Dec) Live at BBC Cd2
78 Incantations
 
or you could number the albums within a year
 
73 Tubular Bells
74 Hergest Didge
75 (1 - Oct) Ommadawn
75 (2 - Dec) Live at BBC Cd1
75 (3 - Dec) Live at BBC Cd2
78 Incantations
 
Within ‘75’ the alphabetic sorting will sort using the numbers 1,2 & 3 after the leading bracket. (Unfortunately the months of the year aren’t spelt in alphabetical order, otherwise it would be a lot simpler!)
 
If the album date is 2000 or beyond, it would have to be represented by 100, 101 etc.
 
I can do an update to allow a full year date or some other dating system to be used if you need it. Please let me know what you have in mind and I’ll see how best to accommodate it.
 
All the best, Andy

Mark Fishman

unread,
Feb 28, 2021, 6:59:36 PM2/28/21
to Brennan Forum
Andy, what happens if the 4-digit year is included in () at the start?
e.g., 
(2000) Album 1
(2003) Album 2
...etc.


riley1...@gmail.com

unread,
Mar 1, 2021, 12:12:30 PM3/1/21
to Brennan Forum
AndyC this looks really useful but unfortunatley for me i have already put my albums in chronological order by artist with the full year, ie 1982 so i guess I can not currently use it. Hopefully one day you will increase the year field to 4 digits.
Where i have a number of albums from the same year I use spaces to get them into the correct order, so for the example you use:
75A (Oct) Ommadawn
75B (Dec) Live at BBC Cd1
75C (Dec) Live at BBC Cd2
i would have
75   Ommadawn   (3 spaces)
75  Live at BBC Cd1   (2 spaces)
75 Live at BBC Cd2    (1 space)

That may not suit everyone but i am happy that i know they are in the correct order, not sure if you "app" works that way or not but it works on the B2.
Regards Paul

AndyC

unread,
Mar 1, 2021, 4:11:43 PM3/1/21
to Brennan Forum
Thanks Mark & Paul, this is very useful in trying to cater for all tastes.

1. In the case of Paul's example of 75^^^Ommadawn etc., ( ^ represents a space), this works fine. My app sorts exactly as Windows Explorer does, by comparing the 2 strings character by character and a space goes ahead of any letter. Just to be 100% sure, I've just tried it on my library with correct results - I never thought of using that method of sorting before!

2. Mark's example of (2000) Album 1, (2001) Album 2 is a bit more complicated. Another user asked me to accommodate 'categories' of music together e.g. Holiday music or Party music. So I incorporated a feature where a leading bracket causes the Artist or Album to go at the end of the library, but to be sorted alphabetically thereafter e'g' (Party) Album 1, (Party) Album 2. I allowed any type of bracket, such as { or ( or [ or < to enclose the key word - <Party>, {Party} etc. This is explained in the Read Me file. So putting the year in brackets will trigger this feature.

If instead the year is just listed e.g. 2000 Album 1, 2001 Album 2, then, as I explained to James, there's a current limit of 999 on the leading number as I never envisaged it to be a date, rather a count of an artist's albums and I thought getting to even 100 would be exceptional.

I'm just writing a new version to remove this limit and allow any leading number up to 2 billion. Thereafter each entry is sorted alphabetically as per usual. This means that an entry could be month specific eg 2000-03 Album 1, 2000-10 Album 2 etc. The leading number stops at the dash or space or whatever (non-numeric) character and it's alphabetical thereafter, hence the need for the first 9 months to have a leading zero.

Whilst I'm doing the update, does anyone else have any numbering or dating schemes they'd like included if possible? Andy

riley1...@gmail.com

unread,
Mar 1, 2021, 5:10:39 PM3/1/21
to Brennan Forum
Andy, how about a check to see if there are any "albums" with "coverart" - unless there is already a way of checking for this, thanks Paul

AndyC

unread,
Mar 7, 2021, 3:25:12 PM3/7/21
to Brennan Forum
Hi All,

I've updated my app to Version 8.2 so it will now reads leading numbers on Artist & Album names up to 9,999. Therefore leading years will always be sorted correctly, even when mixed with other leading number. Consider the following album list of a certain Artist -

2 Early Album
4 Bit Later Album               The first 2 album are given number to promote them to the top of the list - they are sorted numerically
1975 Album A
2004  Album B                    A double space after the date inconspicuously moves this ahead of the next entry.
2004 Album C                     The bulk of the Albums are date stamped.
Fantastic Album                 Next come the normal album names
Next Album                         Alphabetical sorting - "N" comes after "F"
[1986] Smooth Album       A leading bracket - [ { ( < - forces the album to the end of the list, sorted alphabetically thereafter.
[Party] Dancing Album      This sorts after the earlier entry because "1" is ahead of "P" in the ASCII code - numbers 1st, then capital letters then lower case.
 
The default sorting is alphabetical just like Windows Explorer, with a leading number of up to 4 digits being sorted beforehand. 

I've also included a feature that is an improvement on Explorer - the ability to suppress any leading word with a full stop. In my app, "Neil Young" sorts under "N" whereas "Neil. Young" sorts under "Y" without having to write it as "Young, Neil" as per Explorer & iTunes Sorting. 

Also a leading "A" or "The" is automatically suppressed in my app - "The Beatles" is sorted under "B" by default (- unless a leading number is added of course). In this case, an added full stop turns the feature off - "The. Beatles" sorts under "T". This is useful when the leading "The" is actually part of the band or album name and you expect to find it under "T" e.g. "The. Four Tops". 

All this is explained in the Read Me file that accompanies the 64 Bit & 32 Bit executable apps. The DropBox link to the ZIP file is -

Paul's suggestion to check all albums for the existence of Cover Art is interesting. Of course, the cover art .jpg isn't included in b2db, it's only on the HDD and therefore also on any BackUp. Looking for these files whilst analysing a Back up should add a very small increase to the overall run time. The list of albums missing cover art would be output as a text file. I'll produce an update to do this in a week or 2 - am a bit busy at present.

All the best, Andy

AndyC

unread,
Mar 9, 2021, 5:25:06 PM3/9/21
to Brennan Forum
Warren recently sent me this via my private email -
 Andy, I was with you until the... "If you want to know the time duration of each album & track as well, plug in a recent Backup made using 'USB Functions > Export to C'. Then the app add this info to the listings in both the main text files, as well as analysing the Backup for accuracy and completeness."
You're going to need to elaborate a bit or explain better for me to follow exactly what you meant--or exactly "the what & the how" of what you were instructing in that couple sentences...
Communication being both a pitch and a catch... I didn't successfully catch!
I haven't had time to give your software contribution a tryout so perhaps if I had, it would be brilliant communication. Regards, Warren.
-------------------------------------------------------------------
Others might need a bit more clarification too, so here goes. The original purpose of my app was to reorder the music library in my b2 so that in played and displayed in the UI in a more logical (to my eyes) order. It does this by reordering the 'master database' text file - b2db - & Martin kindly added commands to allow b2db to be exported and imported. This was about 5 years ago and I've been using & sharing successive upgrades of the App with any who also want to do this. This is still its main function. The Read Me file included in the ZIP download explains this fully and is worth a read before using the App.

Yes, I know that turning on A-Z sorting achieves more or less the same thing, and for most people this is enough. But my App does allow a more thorough job to be done and has more customising options, as the Read Me file explains. Once you read this, it should be quite easy to apply it to your needs.

During the lock down early last year, I had time on my hands and used it to tackle the problem of how to verify the accuracy of a Back Up. Theoretically a recent Back Up of b2's music library on an external HDD should be an exact copy of what's on the b2's internal HDD or SSD. However, there had been some discussion in the Forum at the time of b2's Back Up process terminating prematurely due to a software bug. This bug was fixed last summer but it left me wondering how I could be 100% sure that my Back Up is 'fit for purpose' if disaster should strike my b2's internal HDD. I then realised that the solution was to use the App to compare the music in the Back Up with the music as listed in b2db. So my App has an option to plug a recent backup into your Windows PC and then my App compares it with b2db whilst it is reordering the latter. The App's home screen asks just one question - "Do you want to analyse your Back Up?" If you enter "y" for yes, it auto-detects the external HDD and gets on with the job. (The default is "No" so just hitting Enter makes it run without doing the Back up analysis.)

This analysis does increase the App run time up from 1 minute (or less) for a straight b2db reorder to maybe 15 minutes when analysing the backup as well (1,850 Albums). This 15 minutes sounds slow but the App runs automatically so go and make a cup of coffee whilst it's running! Also it speeds up noticeably if a fast SSD is being analysed and/or if a 2nd run is being done and Windows has already cached the contents of the external HDD or SSD. (Obviously a fast PC also helps.) When this analysis is done, the App outputs another text file called Back Up Problems.txt. This lists tracks missing for the Back Up and extra unwanted tracks in the Back Up that aren't in the b2. This latter is useful because b2 Back Ups using 'Export to C' are incremental and, if you edit any track or album names, the next backup adds these new names (& their music) but doesn't delete the old ones. Also, if you delete music, it doesn't get deleted from the Back Up.  So your Back Up swells with duplicates & unwanted music. 

Brennan recommend deleting a Back Up after a while and starting again. But this is time consuming and also SSDs don't like being erased and overwritten too often. So, instead, armed with this text file listing the problems in the Back Up, you can easily delete the duplicates using your PC and Windows Explorer. When you've finished, run the App again and it will congratulate you on a job well done if you've got the editing of the Back Up all correct!

Whilst logging all the tracks in the backup ready to compare them, with the tracks in b2db, I also realised that I could also record the duration of each track so this could be added to the album & track text file lists, just like you get on the back of most CDs. The duration of Non-compressed WAV tracks is proportional to the file size but compressed tracks like FLAC, MP3 & AAC needed a bit more detective work. The duration information is actually encoded in the file header and just needed reading & decoding. So, now when you analyse your Back Up using my App, it automatically adds track duration information to the track list text file. And because the maths is easy, it also tells you how long each album is, and just for fun, how long your entire music collection is. (My collection would play for just over 74 days continuously before repeating any music.) All this is also explained in the Read Me file.
-------------------------------------------------------------------------------------
One word of caution - special characters in artist, album and track names are encoded differently by Windows and can cause false errors. For example, the humble apostrophe ' (as in "Don't Cry" for example) is encoded in at least 3 different ways by keyboards around the world and often MusicBrainz serves up album information that includes characters that are different from the ones that your keyboard generates. The consequence is that my App reports this track as missing from the Back Up and also reports it as an unwanted extra track in the Back Up. The solution is either to pair up such reports and ignore them as 'false positives' or edit your library to remove such characters and rerun everything - you only have to do this once. It's irritating but a consequence of the 'Global Village' we all live in. I understand that a future b2 software update will improve the b2's handling of such characters so maybe this situation will improve as a consequence.

Sorry for the long reply but i hope this helps a bit. Happy Brennaning! Andy

AndyC

unread,
Apr 17, 2021, 3:46:45 PM4/17/21
to Brennan Forum
COVER ART detection now included in latest update.

I've now added Cover Art detection to the latest update to my A-Z Sorting & Backup checking program. 

Earlier versions generated a more detailed version of the b2's Album list (Settings > Maintenance > List Albums). Now, with the latest update, it also tells you if there is any cover art associated with a given Album and what size it is - e'g 250x254. 

Additionally it generates a new text file that simply lists all the Cover Art in your b2 in the same A-Z order as the music. Hopefully this will assist you in adding Cover Art to other Albums in your music library.

The program detects the Cover Art files in your Back Up rather than from your b2's HDD using NAS. I feel this is safer than letting an automated program rummage around inside the b2's HDD. The latter has a 'Here Be Dragons' feel to it that might just provoke instability and the need for more Scan Disc-ing. Provided your Back Up is up to date, the information therein should be exactly the same as in your b2.
-----------------------------------------------------------------------------
Interestingly, I've discovered that if you update a piece of Cover Art because you've found a better image, b2 doesn't delete the unused one(s) so you can have multiple Cover Art files associated with a particular album. I assume b2 has an internal database that tells it which file to actually display. I haven't tried using NAS to delete the unwanted files so don't know what would happen if you deleted the file that b2 actually wants to display. ("If in doubt, run Scan Disc" would apply here too, I guess!) In truth, the files usually take up very little disc space, so it's not really a big deal to leave the duplicates in place.

I understand that Cover Art files can be .jpg, .jpeg, & .png. My program detects these and also .bmp files as well, although I don't think b2 currently will accept .bmp files. If anyone knows of any other file types that can be used as Cover Art, please let me know and I'll add detection of them to the program.

Detecting the size of each piece of Cover Art from its file header didn't turn out as simple as I'd hoped. (What a surprise!) I've found 3 different types of .jpg header so far, including one that is simply an HTML address pointing to the actual file's address on the internet. Similarly I have so far found that .bmp files can have 2 different types of headers. 

If you use the program and have a Cover Art file that it can't work out the size of (it will either report it as 'unreadable' or display the wrong size), please email me the file and I'll look into it. My personal email is on the Read Me instructions included in the download.

The latest version of my program is 8.6. The following link will take you to the free download -


Just to recap, as usual 32 & 64 bit versions are included and it will work on any version of Windows from XP onwards.

All the best, Andy

PMB

unread,
Apr 19, 2021, 4:07:44 AM4/19/21
to Brennan Forum
Hi Andy,

Thanks for update.


Anyone using Andy's program and have you any feedback to share?

Paul
Brennan Support.

Mark Fishman

unread,
Apr 19, 2021, 6:12:07 AM4/19/21
to Brennan Forum
I have been using Andy's program for a few months, although I haven't tried the latest version yet. The sorting of the b2db file is a great feature, since as a classical music lsitener I don't use the random functiuon and don't want stuff playing in an unpredictable sequemce.

The very best feature (IMHO) is the comparison of the backup files with the current b2db file! It catches any differences, including naming inconsistencies that are due to how Linux and Windows differ in their handling of non-ASCII characters, and flags long filenames that might cause problems.

Thanks, Andy! -- m.

Peter Lowham

unread,
Apr 19, 2021, 8:31:44 AM4/19/21
to Brennan Forum
Hi All,

I'm just about to start using Andy's program, so I'll be getting to grips with it very shortly.

Thank you Andy for all the effort that you are putting into this!

Regards,
Peter.

Warren Knauer

unread,
Apr 19, 2021, 9:23:12 PM4/19/21
to Brennan Forum
@AndyC,

Would you please consider writing a program that would read the B2 database and output a tab-delimited file (preferred, or a CSV format) that would import correctly into Microsoft Excel to columns for sorting—or pivot table summary.  I’m certainly hopeful you could re-use some of your existing coding.  The minimum data to have included in each text record output into the tab-delimited file:

SONG TITLE <tab> TIME DURATION <tab> ARTIST <tab> ALBUM TITLE

(The order of these fields left-to-right would not matter because of Excel provisions for manipulating the data.  I’d prefer tab-delimited over CSV but either would be fantastic.  Possibly adding a field containing True|False or Yes|No for album COVER ART would be great too.)

I propose this because it would help B2 users in multiple ways:  you could see how many versions of particular “songs” you have, many different Excel-enabled filters, sorts & searches, and possibly do some other useful pivot-table summaries or tabulations. 

Frankly, the most compelling reason for such an output file may be:  I want to build playlists with durations that can fit a pretty narrow window or length of time.  Another benefit would be merging track-level info from the file your coding would produce with tag info from elsewhere.  Then use pivot tables to make other Playlists that take duration, genre, recording date, etc (from whatever tag source) into consideration.    

I would be willing to try donating an XLSX file that uses the data, copes with the Time Duration maths—for at least some of the stated purposes.  I once made a file something like this that used data from iTunes--before pivot tables came along...

Many thanks for your consideration!

Warren


Warren Knauer

unread,
Apr 19, 2021, 11:12:59 PM4/19/21
to Brennan Forum
@AndyC, By the way--were you to be interested in my preceding post, I should probably add Excel requires time entry in h:mm:ss format without exception, so far as I know.  After entry, the times can be displayed in mm:ss or other formats but that does not relieve the required entry format.  Traditional Excel pain in the arse there...  Same is true when seeing times summed in a pivot table--setting a display format of [h:]mm:ss would at least take care of changing 77:03 to 1:17:03 for you... or there are plenty other display options...

Mark Fishman

unread,
Apr 20, 2021, 6:53:52 AM4/20/21
to Brennan Forum
The b2db file does not contain all the information that you ask for. For example, neither track duration nor the presence/absence of cover art is in that file. So you'd still need a drive with a backup of your B2's music, and not only would every directory have to be listed (to look for art), but every track has to be opened and examined to find the duration information.

Warren Knauer

unread,
Apr 20, 2021, 4:32:55 PM4/20/21
to Brennan Forum
I was really hoping track duration was there (cover art yes|no was just an impromptu "would be helpful")

Mark Fishman

unread,
Apr 20, 2021, 5:17:53 PM4/20/21
to Brennan Forum
Andy's program does (already) look at track durations, but it does so by examining a backup drive that you attach to your PC with a backup of your B2 music. So he can get that info. It's just not as simple as parsing the b2db file.

AndyC

unread,
Apr 20, 2021, 6:09:59 PM4/20/21
to Brennan Forum
Hi Warren (and Mark),

Interesting! I was thinking of doing something with Playlists next so this may well be it.

I think the simplest way be meet this request would be to use all the data already in my program to output an extra file with a .csv suffix whilst it is running and generating b2db-new & all the other text files. CSV files have a very simple structure that Excel will know what to do with. (More importantly, Excel shouldn't be tempted to throw up any error messages concerning corrupted file data, which would always be a concern if instead I tried to create a complicated .xls or .xlsx file.)

I need to know exactly what fields each line of entry will contain, and the best order. Are they:- 1. Track name, 2. Album, 3. Artist, 4. Duration, 5. Cover Art (Y/N).

The first 3 are normal text fields without any backslash ('\') delimiters I assume.

Track name: In b2db, this included a leading number e.g. 03 Comfortably Numb. Are the leading numbers to be included or do they need trimming off?

Duration: Inside my program, the time duration for each track is read from the file header in the Back Up & stored in secs. My program then outputs it as Hr:min:sec to enhance readability. Would you need 3 fields - hrs, mins, secs? Or one colon-delimited field - hr:min:sec? (Having briefly played with a test .csv, formatting a cell as Time results in Excel storing data as H:MM:SS, and the converse is true - when Excel finds such a data structure whilst opening a .csv file, it formats that cell as Time)

Cover Art: Is Y/N sufficient or do you need to know the file's display size as well - ( 450 x 500 )

PlayLists: The b2 command - Playlist Menu > Save Playlists outputs a folder containing one text file for each Playlist, listing the Albums (& tracks if an incomplete album is included) that are in it. Do we need to try and include this information also, so that there's a Playlist field which indicates the name of the playlist (if any) that the track is already included in. For this to work, users would have to carry out the Save Playlists procedure beforehand and copy the resultant folder into my program's working folder so it can be read. This might be a complication too far for some users and so I would only be willing to include this field if it served some good or even vital purpose. Does it?

Sorting all the tracks into A-Z order isn't a problem but will require the Artist & Album fields to be included in the sorting, just as Excel can be asked to do. This will add a time overhead to the run time - is it necessary or would you actually let Excel do the sorting based on a particular column of data according to any particular requirement you had at the time?

Any input from anyone else is also welcome so.

When I understand what this .csv file should contain, I can get to work. It'll take a few weeks of course as this is a hobby & I'm quite busy otherwise.

All the best, AndyC

Warren Knauer

unread,
Apr 20, 2021, 9:51:58 PM4/20/21
to Brennan Forum

My recommendation or preference for a .CSV extract is a first step, with (best?) consistency with B2 implementation, 

  1. Artist (included/repeated in each Track record)
  2. Album (included/repeated in each Track record)
  3. Track name (YES, include numeric prefixing “track number” as that can easily be removed in Excel IF desired)
  4. Duration (single field using the required time input text format for Excel to identify as a time “hh:mm:ss” where the “hh” will usually be 0)
  5. Cover art (“FALSE” when none is found or “450 x 500”—e.g.  I prefer just having a single column but, again, an opinion)

To expedite matters, defaulting all sorting to Excel seems vastly superior (or other software B2 users might read the .CSV file with).  For a second later step, doing something more with Playlists might be a benefit?  I would rather not see the first (.CSV extract) delayed for this bit.  I do show (below) how I want to work up Draft playlists within Excel.

I'm attempting to attach several screenshots from Excel. "INPUT" tab shows data imported in and the Time formatted.  "data" tab is an expansion on INPUT with additional data added--Plus I have presumed an assumption to strip away ".flac" from track names but to populate that into a separate column (only because some B2 users might like that?--I personally have no opinion on that).  Then a couple more columns to support building a new Playlist.  The other two shots show a couple Pivot Tables, one that summarizes which have missing cover art plus one where a Playlist has been attempted to fit a one-hour target.

I'm hoping I've not confused...?  (the attachments may not be in the proper order)

Cheers,

Warren

Screen Shot 2021-04-20 at 9.27.12 PM.png
Screen Shot 2021-04-20 at 9.28.16 PM.png
Screen Shot 2021-04-20 at 9.27.41 PM.png
Screen Shot 2021-04-20 at 9.28.03 PM.png

Warren Knauer

unread,
Apr 20, 2021, 9:53:07 PM4/20/21
to Brennan Forum
I would have uploaded the XLSX but haven't researched rules on that...

AndyC

unread,
Apr 22, 2021, 4:06:49 AM4/22/21
to Brennan Forum
Hi Warren,

Thanks for this - the 4 screen shots arrived OK.

Leave it with me and I'll have a look at what can be done. It'll take a couple of weeks probably so please be patient.

All the best, AndyC

Warren Knauer

unread,
Apr 22, 2021, 11:04:02 AM4/22/21
to Brennan Forum
@AndyC--Great!  Certainly there is no time pressure from me.
I am going to risk the wrath of Google Groups and/or the Administrators of this forum & upload my "mock-up" XLSX and an accompanying .CSV mockup file--in hopes you can refer to & benefit from...  There are no Macros or any other nonsense in the XLSX file.  It was created using Excel 360 or whatever they're calling it now.  It has an INPUT sheet that presumes to be the result of importing the CSV file--then a Data sheet where that imported data has been supplemented--then a couple example pivot tables (one to help highlight missing album art & one to show an approach to evaluate files to go into a draft playlist).  (The .CSV file was actually created from the INPUT tab contents)
The data is in tribute to the soon-to-be released 50th anniversary Deja vu set...  :-)
Best regards,
Warren

extract_spec.csv
forAndy.xlsx

arw...@gmail.com

unread,
Apr 26, 2021, 10:13:55 AM4/26/21
to Brennan Forum
I've been trying this excellent wee programme over the weekend, and it's great at sorting, thank you, AndyC! I have found that although there are dire warnings about it not backing up files with diacritical marks or other non-standard characters, and that it reports both that files like that are missing and that they are unwanted in the export, they are, in fact, included in the export and not duplicated. I have attached a screenshot of the report from your programme, an extract of Backup problems.txt and of a listing of the export on my USB backup drive. I can confirm that, for instance, 'Der Jäger' in Die schöne müllerin plays from the backup file properly, although it should fail on at least three grounds! Clearly, I haven't checked all 1042 reported files, but none that I have checked is missing from the backup. For the sake of completeness, the stats shown on my B2 front panel match those in the main panel of your programme. Incidentally, I have been exporting for backup since I got my B2 about 5 years ago, and I have never noticed missing tracks or albums.

It seems, then, that Export works better than intended, at least for me.

Andrew
Screenshot (6).png

AndyC

unread,
Apr 26, 2021, 3:32:37 PM4/26/21
to Brennan Forum
Hi Andrew,

Thanks for your feedback. Yes, this problem that you are having is the common one of Encoders & encoding.

Your b2 backs up all your music to your external HDD or USB. Then, when my program comes to compare what's in the back up with what's in the b2's internal HDD, it has to do this via Windows. And this is where the problem comes in.

When Windows encounters unusual characters in any text, it uses Encoders to interpret them. Try this - open a non-.doc or .docx file in Word (by changing the file extension to .doc if necessary - ignore the warning that Nanny Windows pops up with) and you'll likely get a dialog box asking you to choose which encoder you want to use. And there are over 100 of them, covering most of the various languages on this little planet of our. My program compares the character string of a given Artist\Album\Track as found in your b2db internal database file with the equivalent in the Back Up read using Windows Command's DIR command. If the 2 strings aren't EXACTLY equal, it will of course report a discrepancy and this may well be a 'false negative' due to 'foreign' characters such as you are experiencing.

This problem is discussed in the Read Me file that accompanies my program. I've found by experience that even the humble apostrophe (as in "that's") can have at least 3 different encodings depending on the country in which the computer that was used to enter data into the music look up database is located.

Your b2 also isn't too happy with these characters. You'll notice that when a track name containing a 'foreign' character scrolls across the screen, the display includes one or 2 extra spaces, making it look a bit funny. This is because these character are encoded using 2 or 3 bytes per character, rather than the more usual one byte per character. For example, the Crosby, Stills, Nash & Young album "Deja Vu", if written correctly with the French vowel characters, becomes 12 bytes long instead of the original 7. If you display it in Notepad, you can barely recognise it!

I believe that Martin Brennan is working on encoding issues with the b2 for a future software update. In the meantime, unless someone can suggest a different solution to this problem, you have 2 choices -
1. Manually check the list of 'missing' music from the back up and, when satisfied that they are all 'false negatives', ignore them.
2. Delete each 'extra' artist, album or track from your back up, manually edit each 'problem' artist album & track name in your b2 using the GUI to replace the 'foreign' characters with normal ASCII characters, and then do another back up. Do another Scan Disc, rerun my program and it should report far fewer problems.

I fully realise that neither solution is ideal. I note from your screen shot that you have a fair amount of German & French music and this often includes many accented letters. There's a lot of editing there if you use Option 2 (and some people want their music names displayed as the composer intended so don't want to do this). However I've gone down the Option 2 route and so my comparison works well. Also the scrolling display on my b2 looks a lot better too.

I'm not 100% happy with my program's current solution to this problem but can't think of a better solution right now. Maybe Martin's future updates will improve matters.

I'm aware too that this problem with European languages is probably a 'walk in the park' compared to the equivalent issues with Far Eastern keyboards!! LOL

All the best, Andy.

arw...@gmail.com

unread,
Apr 27, 2021, 4:21:08 AM4/27/21
to Brennan Forum
Thanks for the full explanation, Andy. Most of the problem characters are as a result of my editing names to include ones which the B2 will display, i.e., to replace the double spaces. I use a website called Typeit to find them and copy/paste, something I mentioned on here years ago. It works, and its only drawback is that it trips up your programme +/- Windows. Given there aren’t so far any missing files in my backup, I’m happy. It seems to me you may have a job on to find a workaround, though - good luck!

AndyC

unread,
May 16, 2021, 1:18:55 PM5/16/21
to Brennan Forum
NEW UPDATE RELEASED

I'm pleased to release Version 8.7 of my b2 A-Z sorting and Back UP analysis program for free download. It works on Windows XP onwards and 32 & 64 Bit versions are included in the ZIP file, plus a Read Me file. The link is -

The main improvement is that it generates an extra new file - Music Library.csv. This contains every track in your b2's music library in CSV format that will load easily into Excel, Numbers and other spreadsheets. The number of rows in the spreadsheet equals the track total in your library (Press INFO on the remote to find this) - plus one, the column heading titles. 

Depending on whether you have your BackUp plugged in at the time of running the program, you will get either the basic or enhanced version of this .csv file -
1. Basic - This lists - Artist, Album, No Of Tracks, Track (including the track number prefix if present)
2. Enhanced - This lists Artist, Album, No Of Tracks, Album Duration, Track, Track Duration, Album Artwork.

The 2 Duration fields display the time in h:mm:ss format. I tried omitting the h: bit for track duration to simplify the display but Excel gave inconsistent results as it tried to guess what the type of data was. Adding the h: bit in front seems to have cured this to a great extent - you may still have to play with the formatting of these cells columns depending on your needs. (Incidentally I do have one track over an hour long anyway - Mike Oldfield's Amarok album on CD is just one track of 1:00:02 duration!)

The Artwork field is the last column as it is possible to have more than one artwork file for any given album. In these cases, extra columns contain the extra artwork. The cell entries are either the artwork size - e.g. '450x500' - or 'FALSE' if artwork can't be found. In the event that a time duration can't be found, the cell entry is also 'FALSE'. Hopefully this choice of 'nil entry' word will aid advanced sorting of data using the spreadsheet's functions & formulae.

As ever, I've tested Version 8.7 extensively on my library but you never know, so please advise me of any errors, glitches or requests for enhancement. Thanks.

I note that I released Version 1.0 of my little program exactly 5 years ago in May 2016 and all it did back then was sort your music library into A-Z order using the b2db file. It still does that of course, but now there's a whole lot more besides - 7 other text files of information and the new .csv file to help you curate your music library. The Read Me file that's included in the release gives full instructions and info, especially for newbies. One of the text files is a simple reference file that tells you what's in the other files, as most contain multiple categories of related information.

All the best, AndyC

PS The next release hopefully will give the option of using NAS to access track duration & artwork data instead of using a recent Back Up. This is probably a good couple of months away though...

Warren Knauer

unread,
May 18, 2021, 5:19:12 PM5/18/21
to Brennan Forum
Andy and all--
I have to say I am so excited to try new changes, this weekend??--sadly I have been derailed by USB thumb drive issue.  Getting a new one...

Thanks Andy!!

Warren Knauer

unread,
May 20, 2021, 6:02:47 PM5/20/21
to Brennan Forum
AndyC and all:
V8.7 looks to be great work, congratulations--more to the point, Thank You!  A few comments and questions follow...

I did not have focus on or strong interest in the "sort" elements of your prior work but the duration info has been of strong interest to me, if you recall, for making Playlists that are good groupings BUT also fit into a variety of predetermined overall durations.  Surprises from my first use of V8.7 included the amount of issues resulting from tracks or albums with "peculiar" embedded characters.  Honestly, that was something I had not yet been concerned over as I'm much too new to the B2 specifically and mass ripping in general.  I'd ripped selected CDs in the past, but had never had the ambition of ripping the entirety till buying the B2.

I am going to need to understand what's driving missing durations.  Well over a thousand tracks affected for me...  
Do compact discs have track duration data stored on them?--within each track or in some overall header-level?  
or do devices determine/calculate that--surely not?  
And what for files stored as mp3?  m4a?  wav?
And as I am using pivot tables in Excel, I think the album durations will not matter to me as the tracks will be summed for me.

If eliminating the "problem" characters will eliminate loss of track duration data, then I'll be OK.  If there are more issues, I need to understand.
Thanks AndyC!
Best regards to all the other addicted music lovers,
Warren



Mark Fishman

unread,
May 20, 2021, 6:26:13 PM5/20/21
to Brennan Forum
Audio Compact Discs do not contain files or information apart from the audio, which is in a "stream" format. There is a Table of Contents, separate from the audio data, that tells the CD player where in the stream to locate the start of each portion of the audio that we call a "track". This location is a frame boundary (the audio stream consists of frames of a fixed size that contain audio samples plus some error-detection bits). The CD player calculates the duration of each "track" by subtracting the start of one track from the start of the one following it, and multiplying by the duration of a frame.

Ripped audio is stored in files. Since the audio information on a CD, and thus in the ripped file, is stored as PCM values at a fixed sample rate, the length in bytes of an uncompressed file converts directly to a duration in minutes and seconds. This duration can also be stored in the header or other tags of file formats that support such things, e.g., FLAC, mp3, m4a, ogg, and so on.

Andy's program opens the files and tries to find where in the header for each type might be the duration information. Apparently it isn't always in exactly the same position relative to the start of the file, so it can be tricky to find.

I don't think this process should be affected by filenames that have non-ASCII characters in them, but you could test that by renaming some of the "problem" files to see if another pass of the program picks up any different information.

-- m.

On Thursday, May 20, 2021 at 6:02:47 PM UTC-4 knauer...@gmail.com wrote:
AndyC and all:
V8.7 looks to be great work, congratulations--more to the point, Thank You!  A few comments and questions follow...

I did not have focus on or strong interest in the "sort" elements of your prior work but the duration info has been of strong interest to me, if you recall, for making Playlists that are good groupings BUT also fit into a variety of predetermined overall durations.  Surprises from my first use of V8.7 included the amount of issues resulting from tracks or albums with "peculiar" embedded characters.  Honestly, that was something I had not yet been concerned over as I'm much too new to the B2 specifically and mass ripping in general.  I'd ripped selected CDs in the past, but had never had the ambition of ripping the entirety till buying the B2.

I am going to need to understand what's driving missing durations.  Well over a thousand tracks affected for me...  
Do compact discs have track duration data stored on them?--within each track or in some overall header-level?  
or do devices determine/calculate that--surely not?  
And what for files stored as mp3?  m4a?  wav?
And as I am using pivot tables in Excel, I think the album durations will not matter to me as the tracks will be summed for me.

If eliminating the "problem" characters will eliminate loss of track duration data, then I'll be OK.  If there are more issues, I need to understand.
Thanks AndyC!
Best regards to all the other addicted music lovers,
Warren



Warren Knauer

unread,
May 21, 2021, 11:40:23 AM5/21/21
to Brennan Forum
Thank you Mark--very slightly curious why the process of getting identification & info from Musicbrainz (or wherever) does not include grabbing "published" track durations.  That's the point in the overall Brennan sequence that available album art is grabbed, right? (rhetorical)  Nonessential so that is probably the answer to my curiousity!

Mark Fishman

unread,
May 21, 2021, 11:56:43 AM5/21/21
to Brennan Forum
The online databases generally do not contain track durations directly. Here's an example from gnudb (which "adopted" freedb when the original freedb "went away"):
You can see it contains the offsets to the track frames, but not actual minutes:seconds sort of thing. MusicBrainz works similarly, although it uses a different DiscID calculation. Since the timings are not actually "there", they can't be "grabbed" -- they would have to be calculated.

If you have NAS enabled on your B2, you will see that the folders for tracks actually ripped ON the B2 also contain a TOC file -- that has the same track offsets as the CD or the database. It's used to computer the DiscID so the database lookup can be done.

One could use the TOC, no matter where it comes from, to compute track durations -- but it wasn't part of the B2 software design.

AndyC

unread,
May 22, 2021, 12:49:26 PM5/22/21
to Brennan Forum
Hi Warren & Mark,

Just to add a little to your thoughts, my program does indeed get its time information from the header in each track in the Back Up. I believe it is very robust in finding this information once it has identified the file type and, in my testing, the only issues I've ever found concern these confounded 'unusual' characters, about which I've already written quite a bit. (If anyone has a track whose time can't be read yet has no 'unusual' characters in its name (or the Artist or album), please email me details and a copy of the tracks and I'll look into it. Thanks)

Warren, if you look at your 1,000 or so 'misses' where the time info isn't present, I'm guessing that the Artist, Album or track name more often than not has a simple apostrophe in it - as in "Don't Go". This little character has at least 3 different ASCII codings, depending on the home country of the person who entered the info into MusicBrainz. Your Windows PC sees a different ASCII code in the name in your b2db file from the same name in your Back Up. Hence my program can't match the 2 character strings and reports a 'miss'. It's intensely irritating.

Solutions:
1. Go through your entire music library and edit each apostrophe in the GUI by deleting it and reentering it from your keyboard. (Often you can see the shape of the pixels making up the apostrophe change slightly when you hit 'Enter' to make the edit.) Then do the usual - Scan Disk and Export to C. You should then find that most of the time information can now be found. However the BackUp Problems.txt file will report lots of duplicates in your Back Up as your b2's incremental "Export to C"  has just added the 'new' tracks alongside the old ones. You have to manually delete the old ones - or format the Back Up disk and start the Export process again.
2. I'm currently working on a version of my program that gets its time info from the b2's internal HDD via NAS. Hopefully this will minimise the unusual characters problem. In fact a Beta is running on my PC now - and I've hit the first little snag - NAS is very slow! Getting the time from 2,000 albums via Back UP generally took about 15 - 20 minutes. With NAS, just I've worked out that the same process will take about 4 hours!
3. So, maybe I'll forget NAS (or leave it as an option for hard souls with time on their hands) and instead try a 'nearly is good enough' approach to the comparison process between names in b2db and in the Back Up. This means that, if every character in both name strings matches EXCEPT ONE, the program will then accept it as a match and get the time info. This could thrown up other problems with different tracks with similar names but I think it could work.

If only the b2 only used the standard ASCII character set and ignored the extended characters, these problems would be minimal. However there'd then be quite a few owners with French, German, Gaelic etc. music who wouldn't be happy that the track names now aren't right. I believe Martin is working on this right now but it's going to be difficult to please everybody, I fear.

All the best, Andy

PS NAS has just taken an hour to read the time info from 465 Albums. However, my b2 is about 5 years old so maybe newer b2s with faster Raspberry Pis and SSDs will go a lot quicker. Here hoping!

Peter Lowham

unread,
May 22, 2021, 1:36:26 PM5/22/21
to Brennan Forum
Hi Andy,

First of all thank you very much for your efforts in developing this software package; I'm now using it and finding that it is extremely useful for the 'data cleansing' of my collection!

I have about 250 (in a collection of 20,500 tracks) 'character problems' to resolve, but most of those are, as you say, the 'apostrophe' issue.  I'm just looking at using 'Notepad++' to do a bulk 'find and replace' which Notepad++ should do with relatively little pain.  So I'm good with that!

Regarding the statement "If only the b2 only used the standard ASCII character set" is it not the case that it is the 'MusicBrainz' site (or previously FreeDb) that is the source of these characters?  I don't know precisely how the Brennan album and track naming works, so perhaps someone out there can enlighten me!

On the 'slow NAS' topic, the early and late B2 models have the same performance characteristics, same processor, same memory capacity, etc. so the later model will show no difference in performance.  I believe that the reason for the switch over was purely that the Ras-Pi 1 Model B (early B2) went out of production in 2016, and the Ras-Pi 1 Model B+ (late B2) was a direct replacement and won't be be phased out until 2026.

Regards,
Peter.

Warren Knauer

unread,
May 22, 2021, 7:54:15 PM5/22/21
to Brennan Forum
AndyC, Peter--thanks for adding more insights.  Peter, I use Notepad++ but didn't realize it could be used to edit filenames on the B2?  Is it possible one of us has misunderstood the solution approach Andy explained?

Andy, you referred to home country setting and based on my IT background (retired now), you are no doubt correct.  Also, you mention apostrophe as a big contributor, and I expect that is also true.  Be aware though, it will not be the only bugaboo as some might say.  So I am skeptical that the scheme of one-character difference may catch many but it will certainly not resolve the problem fully.

You may recall such things as em dash and en dash (not sure if those are Mac-specific terms, I think not).  Also, there are multiple kinds of smart quotes (the curly buggers, not the rigid upright ones).  There are also multiple kinds of space characters.  And of course the truckloads of things such as accent characters, graves, dual characters (I forget the true name), and many more.

My perspective on this may not be helpful but I'm trying.  It certainly would be best for this to be as robust as possible with machine execution time being a lower priority since an overnight session is always a reasonable option. Otherwise, there's temptation to sit there and watch the damned thing churn!

I have changed entries that failed to collect a time for entries of mine down to Buffalo Springfield.  I noticed that in some cases, only the album title was problematic.  Not sure that your placing the album total duration in the output is a factor there.  I would be fine if that were dropped since Excel can do the math for us.  In some cases, an album title had some songs with smart quote apostrophes (or other problems) BUT also has tracks with apostrophes which were (apparently) not the curly fellows because those tracks had times.  Of course, the best example of an album-level issue had to be the Billy Joel album whose title was in Cyrillic characters!

Re-running the Export now to see if the ones I edited on the B2 and then removed from the backup are now in the backup with the names as amended and no longer among the no-duration list when the analysis is completed.  I know this export re-run will finish soon but I may not get another update chance here till later Sunday (I'm in USA Eastern time).

Andy--this is good stuff you're doing. I will put in a word with the Nobel committee!
Warren

AndyC

unread,
May 23, 2021, 5:27:10 PM5/23/21
to Brennan Forum
Hi guys,

Thanks for the comments. I thought that getting time data from NAS rather than from a Back up would improve the 'apostrophe' situation but it didn't! Amazingly, comparing b2db with the internal HDD from which it was obtained still throws up discrepancies with unusual characters simply because the 2 are compared via Windows and its built in converters. Also NAS works about 10x slower than a Back Up on an external drive - I'm not joking, it took 4 solid hours to get data from 2,000 odd albums, instead of less than 15 minutes.

So I've abandoned the NAS approach, for these reasons and also because probing the b2's guts whilst it's doing other things has an uncomfortable "Here be dragons" feel to it!

Anyway, I've had another idea that I think should be productive. So don't go deleting those rogue apostrophes just yet, give me a few days to see what I can come up with please.

All the best, AndyC

Warren Knauer

unread,
May 24, 2021, 11:15:06 AM5/24/21
to Brennan Forum
Ouch--I already sunk the time into doing the clean-up.  I found all but one entry had one or more characters in the track (or album) title.  The only one that did not was a track from the John Mayer CD "Room for Squares" (I don't care about this one)--interestingly, the track was labeled 00 untitled.flac  (I don't recall the track number)
So I edited it and retyped the entirety of 00 untitled to be certain of content.  That did not cause the duration to be captured--in fact, no change at all.
Since then, I added maybe 25 new tracks but did not go through the process again yet.
Regards,
Warren 

Warren Knauer

unread,
May 24, 2021, 12:46:51 PM5/24/21
to Brennan Forum
Checked on this John Mayer thing and it is an oddity.  Apparently I have a re-release that added a track ("3x5") bringing to 13 BUT inexplicably there are 14 tracks.  From Wikipedia... "On the Columbia re-release, "St. Patrick's Day" is consistently listed as track 14; track 13 is an unlisted song with a duration of 0:04, and sometimes even 0:00."  
Presuming I will get around to just deleting it as there seems to be no content.

AndyC

unread,
Jun 28, 2021, 5:22:21 PM6/28/21
to Brennan Forum

Hi All,

I’m pleased to release the latest version 9.1 of my music sorting and backup checking program:-  https://www.dropbox.com/s/bi19ca0i62nbz8s/b2%2BBackUpSort9.1.zip?dl=0

For anybody who doesn’t know what this is, it’s a free Windows program that adds a few ‘bells & whistles’ to managing your music library in your b2 & increasing the pleasure of b2 ownership. It’s been in existence for about 5 years now, works with Windows XP, 7, 8 & 10 and I’m adding more features with each release. It mainly does the following:

1. It sorts your whole library (as it appears in the central column of the UI) into A-Z order + a few tweaks beyond that to let you really personalise the order. (This of course is the b2’s ‘master’ playlist.)

2. It generates a Clean Up.txt file that identifies any unknown Artists, Albums and Tracks to aid editing (for CDs that aren't in the MusicBrainz database) & tells you where any non-FLAC tracks are.

3. It provides comprehensive text files of all your Albums and Tracks & other library statistics.

4. It provides a spreadsheet-friendly CSV file of your whole music library’s contents.

5. If you plug a recent external HDD Back Up into your PC, it will also tell you of any problems with the completeness of the Back Up - either missing tracks or excess tracks left over from previous editing sessions. (In the event of having to a Restore from the Back Up, this is pretty important.)

6. It records the track & album times (such as you often get on the back of CD cases). This is especially useful if planning playlists etc. with specific total playing times in mind. These times also appear in the CSV file. (This time data isn't 100% complete due to problems trying to read artist, album & track names that contain 'unrecognised' characters - generally from foreign languages. For more info, see below!)

7. The ZIP file includes 32 & 64 Bit versions of the program, plus a comprehensive Read Me file explaining it all. (RTFM!)

NB: It’s not to everyone’s taste and your b2 works fine without it. However I like it and am pleased to freely share it with anyone else who is like minded. If you like it too and want other features adding, please email me and I’ll see what I can do.

-----------------------------------------------------------------------------

Since the previous release (Version 8.7), I’ve been working on the problem of the difficult-to-recognise characters in the names of Artists, Albums and Tracks. These are typically French, German or Gaelic letters with accents on them – such as “Déjà vu” in French. These letters are rendered differently as they pass through operating systems (via encoders) and thus the character string describing the Artist\Album\Track of a piece of music in a Back Up is different from that in the b2’s database. Therefore, an attempt to verify that a track is successfully Backed Up by comparing the 2 character strings always returns a negative, even when the track IS actually successfully in the Back Up.

(Incidentally, I was amused recently to see that this problem is commonplace. I paired my iPhone with the infotainment system of a brand new Volvo XC60 and the Volvo OS screen said “Andy’s iPhone paired successfully”. The couple of sub-menus later, the next screen was where my phone was actually chosen to be accessed: It asked “Select Andy*^#s iPhone?”, where 3 weird symbols replaced the simple apostrophe!)

Firstly, I solved the problem of Back Up verification via character string comparison by appreciating that any Artist\Album\Track string has a lot of redundant information in it. Therefore, if it is compressed by deleting all potentially unrecognised characters, there’s still more than enough information left to make a unique positive verification that a Back Up track exists. This also neatly gets round the other problem that Volvo's OS illustrated – that a single byte character such as an apostrophe is often rendered as a new character of 2 or 3 bytes (in extended ASCII), thereby altering the character string length. Comparing unequal length strings will always return a negative; however, by deleting all the ‘dodgy’ bytes, the strings end up equal in length again & this problem is got round very simply. 

-------------------------------------------

A completely separate but associated problem concerns reading the time duration of each track. This is done by accessing information in that track’s header in the Back Up. In previous versions of the program, the Artist\Album\Track string came from b2’s internal database, the b2db file & numerous tracks with unrecognised characters in their names couldn't be accessed. In an early attempt to fix this in a recent unreleased beta of my program, I tried reading the time data from the b2’s internal HDD using NAS. However the results weren’t much better (most unread tracks remained unread) and, worse still, the process was about 20x slower than using an external HDD – it actually took 4 hours to read about 20,000 tracks instead of the usual 10 - 15 minutes. So I abandoned the NAS route – I have to say I wasn't at all unhappy about this turn of events as I wasn’t totally comfortable accessing data from within the b2 whilst it was busy doing other things & might have got upset by the intrusion! As another Forum member might say – “Here Be Dragons!”

So I went back to using the Back Up. In my next attempt to fix this, I assumed that if I read this Artist\Album\Track string from the FAT of the external Back Up HDD instead of b2db, the unrecognised characters might just be rendered the same and so each previously unreadable track in the Back Up could now be read successfully. To do this, I had to make a major rewrite of the core of the program. As programmers know, such a core rewrite invariably affects other processes being run within the program and this also has to be debugged. So it took a bit longer than expected – as usual!

Eventually, I got the new core working OK and launched the program expecting to see most of the missing time data filled in on the Album & Track lists. No such luck! The problem remained exactly the same – reading the FAT of the HDD via Windows and then using the exact same information to open the track in it failed 100% because at some point an encoder or 2 in my PC still changed the unrecognised characters and shuffled things up again. This really was an ‘Aaaaarghh’ moment when I thought the whole crazy encoder system had beaten me.

Another explored avenue was to create a look up table of all the encoders, so that unrecognised characters can be converted back again – a sort of reverse-encoder process. The problem is there aren’t just one or two encoders, there are dozens if not hundreds and more are being added so it just gets too unwieldy. Another failure!

 I suppose all this was to be expected as there’s a truism that goes – “Any problem worthy of attack, shows its worth by fighting back!” So I thought about this problem some more and have now come up with a fix: It’s not a 100% fix but it has remedied the vast majority of the problem & enables the times of most of these tracks to be read now using Version 9.1.

 The QB64 language in which my program is written allows access to the DOS shell (i.e. the old DOS Command.exe command line program which is still there, even in Win10). The DOS command COPY allows wildcards (* & ?) in file (i.e. track) names. So, to read a previously unreadable track, I simply had to copy it using the wildcards to obscure the unrecognised characters and name the copy file as something readable. Then this copy file could be read.

Therefore “COPY /Y  Artist\Album\01*.flac  g:\tempfile.flac” gives a readable new file located in the root directory of the Back Up HDD well away from the actual music library. It is deleted straight after reading to tidy things up and leave the HDD as it was found. (The program of course firstly checks that the Back Up HDD isn’t write-protected before proceeding.)

Most of the unrecognised character problems are likely to be found in the Track names, so hopefully this solves at least 80% of the issues. The program already outputs a Clean Up text file as described in Bullet Point 2 above: It now also lists the locations of tracks whose times were read successfully but which have unrecognised characters in the track names only. This aids any user who wants to edit them via the UI to make a Restore of the library using the Back Up a more robust process.

Sadly, this method isn’t a complete solution as COPY won’t allow wildcards in directory names, so this technique can’t be used for unrecognised characters in Artist & Album names. Therefore the time data of albums such as CSNY’s album ‘Déjà vu’ currently remains unreadable unless the owner replaces the 1st 2 vowels with their English equivalents.

I’m exploring a few other ideas to tackle this issue and hopefully they’ll appear in Version 9.2 fairly soon. Meanwhile, I hope you enjoy using the program. (Incidentally, whilst I was rewriting the core, I also improved the on-screen layout to improve the presentation of information too. There's lots to watch whilst it's running!). 

As ever, please let me know of any issues that crop up during use of this program. (My email address is in the Read Me file)

All the best, AndyC

Reply all
Reply to author
Forward
0 new messages