Problem with DropSync on Sonoma

[Mark Roseman]

Sync is not retaining the date and time of original files but updating the file dates to the the date of the sync. 

This was a bit of a disaster as I had not realized what was happening and when I synced back lost all the original file date info. Luckily I had a recent Time Machine backup and could restore. How do I get the copied files to retain the timestamp of the original.

thanks

Mark

[Roger Emnett]

Has this problem been resolved??

Roger

[ira cooke]

Mark and I are investigating this.  I have tested on Sonoma with no problems so it may be something specific to his system.

[ira cooke]

Or possibly a bug related to FAT32 formatted drives.  I will report back here after more testing. 

[ira cooke]

I can confirm that there is a problem with syncing on Sonoma.

The problem is much deeper than DropSync.  It affects the basic unix copy (cp) command as well as rsync (the engine that DropSync uses). 

The specific details of the issue are that files will not retain their modification times when syncing to drives formatted as FAT32 or exFAT.  The issue is specific to Sonoma and even on Sonoma mac natively formatted (HFS or APFS) drives are unaffected.  

I will need to file a bug report with apple and hope that this is fixed in a future Sonoma update.  I am hopeful that they will fix it as this is a fairly serious bug with implications well beyond DropSync.

For now my advice is either to either avoid updating to Sonoma or to avoid using FAT32 or exFAT drives if you are using Sonoma

[Lyle M. (XizorBlackSun)]

I'm having an issue on Apple Silicon/Sonoma as well. I have "only scan changed directories" checked under General in the Configure window, but whenever there is a change to ANY file in that directory, it starts a complete scan that takes hours and grinds away on my external hard drive. (External drive is a 10TB G-Drive formatted as Mac OS Extended (Journaled)). This worked fine on my Intel-based iMac on Ventura; I would make a change and it would pick up the change and copy just that change. On my Mac Studio w/ Sonoma, I've set up my Configure window *exactly* like I had it on my Intel iMac, but on my M2 Max Mac Studio, it just grinds away and scans for hours. Any help would be greatly appreciated.

[ira cooke]

Hi Lyle,

This sounds like quite a different issue. 

On a new computer drop sync will need to perform a full sync the first time through.  Importantly, that first sync needs to be successful or it will flag that a full re-scan is needed. 

Perhaps try testing with a smaller subfolder first to see if you can get it working there before progressing to your 10Tb.  

Best wishes

Ira

[Lyle M. (XizorBlackSun)]

Hi Ira,

I have performed full scans several times over the course of the past week. The configuration scans a folder on my external HDD and backs the files up to my NAS.

The first few scans looked fine, one of which took an entire day to scan (about 1TB worth of files). Now, it seems like every time I make any kind of change, it spends hours re-scanning the entire folder instead of just scanning/updating the changed files.

Before, on my previous iMac (2017 Intel iMac running Ventura), I had this running perfectly. I would make a change, it would run for a few seconds, back up the relevant files, and then stop. Same External HDD, same NAS. It's just a different computer running the scan.

(See screenshot below)

Any thoughts on why this might be happening on MacOS Sonoma (M2 Max silicon) vs performing properly on MacOS Ventura (Intel)?

-Lyle

[Roger Emnett]

Ira and Lyle

Don't know if this relevant but for what it is worth I recently went from a Macbook Pro 2017 Intel running Ventura to a Macbook Air M1 and I have went through all the OS updates from Ventura to Sonoma 14.1 with the same 2 TB SSD Mac OS Extended (Journaled) with over 1 TB of music files and so far have not had any problems,

Roger  

--
You received this message because you are subscribed to a topic in the Google Groups "DropSync" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dropsync/Ll4qK_QN-4Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dropsync+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dropsync/38daef9f-04e5-4842-abec-f26e0cb741c5n%40googlegroups.com.

--

Roger Emnett

419-424-3271

[Lyle M. (XizorBlackSun)]

It's a very strange issue. I looked at the details of the most recent backup run, and it seems to be updating the Permissions and Extended Attributes for each and every folder in the configured backup folder. And it happens every time I make a change to *anything* in the folder.

(This is just a small truncated list):

[Lyle M. (XizorBlackSun)]

Could it be that the files that I am backing up originated on a different Mac, and the current Mac has a (slightly) different user name?

I mean... I guess the solution is staring me in the face here. I could just remove the --xattr and --perms from the Advanced window. I care about modification dates, but not user information, since this is mainly for archival/crash restoration purposes.

-Lyle

[Ira Cooke]

Hi Lyle,

If this is happening over and over again it means that the update is not successfully modifying the extended attributes and permissions at the destination.

Your idea about the different user name is certainly a possibility. When you actually run a sync do you see a bunch of errors in the log. You will have a hierarchy of files/folders that is mostly owned by your previous username so when you (new user name) tries to perform an update it will probably fail. Unfortunately I suspect you will see quite a few problems along these lines due to the username change. If you really want to get things working smoothly again you might need to perform a fresh copy. Assuming the destination is just a backup you could presumably delete those files and then perform a fresh copy. Of course, this comes with the usual caveat to take great care .. ie make a backup of the backup first.

Best wishes
Ira

> On 9 Nov 2023, at 4:35 am, Lyle M. (XizorBlackSun) <lyl...@gmail.com> wrote:
>
> Could it be that the files that I am backing up originated on a different Mac, and the current Mac has a (slightly) different user name?
>
> I mean... I guess the solution is staring me in the face here. I could just remove the --xattr and --perms from the Advanced window. I care about modification dates, but not user information, since this is mainly for archival/crash restoration purposes.
>
> -Lyle
>
> On Tuesday, November 7, 2023 at 12:23:52 PM UTC-8 Lyle M. wrote:
> It's a very strange issue. I looked at the details of the most recent backup run, and it seems to be updating the Permissions and Extended Attributes for each and every folder in the configured backup folder. And it happens every time I make a change to *anything* in the folder.
>
> (This is just a small truncated list):
>
>
>

> On Monday, November 6, 2023 at 6:07:40 PM UTC-8 raem...@gmail.com wrote:
> Ira and Lyle
>
> Don't know if this relevant but for what it is worth I recently went from a Macbook Pro 2017 Intel running Ventura to a Macbook Air M1 and I have went through all the OS updates from Ventura to Sonoma 14.1 with the same 2 TB SSD Mac OS Extended (Journaled) with over 1 TB of music files and so far have not had any problems,
>
> Roger
>
> On Mon, Nov 6, 2023 at 2:42 PM Lyle M. (XizorBlackSun) <lyl...@gmail.com> wrote:
> Hi Ira,
>
> I have performed full scans several times over the course of the past week. The configuration scans a folder on my external HDD and backs the files up to my NAS.
>
> The first few scans looked fine, one of which took an entire day to scan (about 1TB worth of files). Now, it seems like every time I make any kind of change, it spends hours re-scanning the entire folder instead of just scanning/updating the changed files.
>
> Before, on my previous iMac (2017 Intel iMac running Ventura), I had this running perfectly. I would make a change, it would run for a few seconds, back up the relevant files, and then stop. Same External HDD, same NAS. It's just a different computer running the scan.
>
> (See screenshot below)
>
>
>

> You received this message because you are subscribed to the Google Groups "DropSync" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dropsync+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dropsync/61bd723e-48c2-4b43-8ca4-d00a681a29f3n%40googlegroups.com.

[Lyle McNaughton]

Hi Ira,

Thanks for your reply. This backup *was* a fresh copy that I just made in a new folder on the NAS. The external HDD is the same as it was; I unplugged it from my 2017 iMac, and plugged it into my 2023 M2 Max Mac Studio.

Even removing the --xattr and --perms from the Advanced tab hasn't helped; it still ruins a full scan every time. The "Extended Attributes" portion of the log is gone now, but the "Permissions" is still being updated every time for every folder.

Here's my "Advanced" configuration (only copying from Left --> Right):

Are any of these safe to remove if I'm just interested in having a backup of the files & modification dates? (For the record, this is the same configuration I had on my Intel iMac, except on that one I still had --xattr and --perms set for L-->R). I'm guessing I could just keep "Update", "Modification Times", and "Delete After".

Here's a copy of the same log excerpt from just a few minutes ago:

Thanks for all of your help in this.

-Lyle

To view this discussion on the web visit https://groups.google.com/d/msgid/dropsync/448FD7EF-7F4F-43D5-AB7B-7B76F164B1F9%40gmail.com.

[Ira Cooke]

Hi Lyle,

Thinking about this a bit more I’ve realised that you will need to fix the username on the source as well. All those files will also be owned by your previous user. When a new copy is created on the destination it will be the new user resulting in a mismatch.

In order for the auto-sync to work the syncs must be successful, but I am guessing that you still have files owned by two different users in the mix and this is leading to many errors. Can you click “Save Report” on one of your completed syncs (not a preview) and send it to me privately. I can take a look.

Unfortunately you may need to find a way to update the user on your 10G source drive. There are command-line ways to do this but perhaps the simplest would be if you have another drive to use temporarily. If you can copy to that, and then copy back it may fix the issue.

With so much data this will be a big operation. Sorry. I do sympathise. Changing username seems like a small thing but it can cause real headaches when managing files across multiple drives / devices.


Best wishes
Ira

> On 9 Nov 2023, at 7:59 am, Lyle McNaughton <lyl...@gmail.com> wrote:
>
> Hi Ira,
>
> Thanks for your reply. This backup *was* a fresh copy that I just made in a new folder on the NAS. The external HDD is the same as it was; I unplugged it from my 2017 iMac, and plugged it into my 2023 M2 Max Mac Studio.
>
> Even removing the --xattr and --perms from the Advanced tab hasn't helped; it still ruins a full scan every time. The "Extended Attributes" portion of the log is gone now, but the "Permissions" is still being updated every time for every folder.
>
> Here's my "Advanced" configuration (only copying from Left --> Right):
>

> <Screenshot 2023-11-08 at 1.54.27 PM.png>
>
> Are any of these safe to remove if I'm just interested in having a backup of the files & modification dates? (For the record, this is the same configuration I had on my Intel iMac, except on that one I still had --xattr and --perms set for L-->R). I'm guessing I could just keep "Update", "Modification Times", and "Delete After".
>
> Here's a copy of the same log excerpt from just a few minutes ago:
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/dropsync/CALgK3PW1oH1M3rB7_DbV2Hfi0Eh1Z5pgVXsLzMf3hq1FZTV%3D9g%40mail.gmail.com.

[Lyle M. (XizorBlackSun)]

Hi Ira,

After going back and looking at my previous machine, I realized that the usernames are identical between machines, same iCloud account, etc. However, I didn't copy/transfer the account to the new machine; I simply set it up as a new user with the same name. So I suppose that would technically be a different user.

If I understand what you're saying correctly, the solution would be to attach a newly-formatted external drive (formatted as Mac OS Extended (Journaled), not exFAT or Fat32), and then copy the files I want to back up to my NAS from the old external drive to the newly-formatted external drive under the new user account? 

Thanks again for all of your help and advice on this. If I'd known the permissions thing would be such a pickle, I would have done it that way in the first place!

-Lyle

[Ira Cooke]

Hi Lyle,

Actually if the usernames are identical then I’m less certain that this is your issue.

When you perform a full sync do you see errors in the log? If you can send me the log with those errors that would be great.

Best wishes
Ira

> To view this discussion on the web visit https://groups.google.com/d/msgid/dropsync/e600ad83-1c5e-44f7-9e46-9890ceb208b0n%40googlegroups.com.

[Lyle M. (XizorBlackSun)]

Hi Ira,

I do not see any errors in the copy log. The sync is clean each time, even if it is doing WAY more than it needs to. So at the very least, my data is backed up.

So after our last exchange, I wanted to make sure that my Mac's username owned the files in question. I cleared my external SSD drive, reformatted it as APFS, copied the folder to be backed up onto the newly formatted APFS drive, cleared the backup on the NAS, and re-performed a fresh DropSync 3 copy from the SSD to the NAS. The copy took a while (~2 days over USB-C 3.0 to NAS via WiFi), but it finished successfully without errors.

HOWEVER. I am still faced with the same exact issue as before. Making one change to a file in iTunes (whose media files live on the external SSD) starts a scan that updates the ENTIRE copy on the NAS, both folders AND files. It seems to modify Permissions, Modification Times, and Extended Attributes randomly on both files and folders, with no rhyme or reason as to why. No folders or files have been updated at the source, but the scan seems to think everything has changed. Also, even when I don't make any changes to the files in the "watched" folder, the backup scan will still run every couple of hours, even though it is set to "only scan changed directories". Each sync preview takes between 10-30 minutes, and each sync update takes between 30-50 minutes, and it updates just about every file and folder.

I am attaching screenshots of my configure window setup. Perhaps I'm missing something in the config that's causing this to run unnecessarily?

If you see anything in my configuration that might be an issue, I would appreciate any advice you have.

I have really loved the way this program worked for me in the past, but I'm wondering if it's just not going to work for me in Sonoma anymore.

Thanks again for all of your help,

-Lyle

[Lyle M. (XizorBlackSun)]

Also, I've been reading about the issues with rsync and the case sensitivity of different file systems/platforms, but my APFS-formatted SSD is case-insensitive, as is my Synology NAS. So I don't think that's the issue. My previous installation was using a Mac OS Extended (Journaled) format, which was also case-insensitive.

Is there a way to update the version of rsync that DropSync 3 uses? Wondering if that might fix the issue.

-Lyle

[Ira Cooke]

Hi Lyle,

It’s not possible to update the rsync that DropSync is using. To be honest I don’t think there is anything wrong with it here so I doubt an update will change things.

Thinking a bit more about your comments. I think an issue is the fact that you are including your iTunes directory in the sync. iTunes and some other system-controlled directories have lots of files in them that are subject to control of the OS. Importantly the files can be modified by background processes (ie without you seeing them .. ie without rhyme or reason as you say). This fits the pattern you are seeing and it might also be something that has changed with sonoma.

Unfortunately there isn’t a good solution. My recommendation is probably to use a separate folder-pair for your documents (things you control) and do not incldue photos, iTunes etc in your folder pair. If you still want to backup iTunes or Photos etc you might want to do that on a schedule rather than using the auto-backup option.

Best wishes
Ira

> On 16 Nov 2023, at 7:36 am, Lyle M. (XizorBlackSun) <lyl...@gmail.com> wrote:
>
> Also, I've been reading about the issues with rsync and the case sensitivity of different file systems/platforms, but my APFS-formatted SSD is case-insensitive, as is my Synology NAS. So I don't think that's the issue. My previous installation was using a Mac OS Extended (Journaled) format, which was also case-insensitive.
>
> Is there a way to update the version of rsync that DropSync 3 uses? Wondering if that might fix the issue.
>
> -Lyle
>
>
> On Wednesday, November 15, 2023 at 1:19:18 PM UTC-8 Lyle M. wrote:
> Hi Ira,
>
> I do not see any errors in the copy log. The sync is clean each time, even if it is doing WAY more than it needs to. So at the very least, my data is backed up.
>
> So after our last exchange, I wanted to make sure that my Mac's username owned the files in question. I cleared my external SSD drive, reformatted it as APFS, copied the folder to be backed up onto the newly formatted APFS drive, cleared the backup on the NAS, and re-performed a fresh DropSync 3 copy from the SSD to the NAS. The copy took a while (~2 days over USB-C 3.0 to NAS via WiFi), but it finished successfully without errors.
>
> HOWEVER. I am still faced with the same exact issue as before. Making one change to a file in iTunes (whose media files live on the external SSD) starts a scan that updates the ENTIRE copy on the NAS, both folders AND files. It seems to modify Permissions, Modification Times, and Extended Attributes randomly on both files and folders, with no rhyme or reason as to why. No folders or files have been updated at the source, but the scan seems to think everything has changed. Also, even when I don't make any changes to the files in the "watched" folder, the backup scan will still run every couple of hours, even though it is set to "only scan changed directories". Each sync preview takes between 10-30 minutes, and each sync update takes between 30-50 minutes, and it updates just about every file and folder.
>
> I am attaching screenshots of my configure window setup. Perhaps I'm missing something in the config that's causing this to run unnecessarily?
>
>
>
>
>
>
>
>
>
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/dropsync/a8e9afa1-ea7c-490f-847a-acdc0d3187dfn%40googlegroups.com.

[Lyle M. (XizorBlackSun)]

Hi Ira,

It definitely seems like a problem with either how Sonoma works, or how the M2 silicon architecture works with Sonoma.

Funny thing, I decided I would just set the backup tasks to run every 24 hours instead of whenever a change was made. What I discovered, though, was that when it runs on that 24hr schedule, ONLY the changed files get scanned/transferred, instead of the entire directory (which is what was happening when I selected "whenever a change occurs in the source folder").

I have six processes that run, to back up the most important things to me (music and photos) -- all of my other documents are backed up via Time Machine. Two processes to back up the Music Library files to my NAS and a separate external drive; two processes to back up the Media folder to NAS and external drive, and two processes to back up the Photos LIbrary to NAS and external drive. All are working well on the 24hr schedule, which is fine for me, since I don't make that many changes from day to day. It has just been an adjustment since it worked so well on Intel/Ventura.

Thanks again for all of your help and thought that you have put into this issue.

One question -- is there a way to schedule the time for the 24 hr backups to run? At the moment, they all start around the same time, which I'm assuming is because that's around the time of day I checked all of the boxes on the "Sync every _____ hours/minutes" box. I'd prefer to run them over the course of the day, just to prevent wear & tear on the drives. If not, no big deal, but I figured I'd ask.

Again, thanks for all your help on this.

-Lyle

[Ira Cooke]

Dear Lyle,

Yes there is a way to schedule timed syncs with much more flexibility. This isn’t actually built into the app but takes advantage of its scriptability.

You can read more about how to set this up here

https://www.mudflatsoftware.com/dropsync3/docs/scripting.html

Best wishes
Ira

> To view this discussion on the web visit https://groups.google.com/d/msgid/dropsync/4d4561cd-49ed-4a3f-9085-b4ac971e1783n%40googlegroups.com.