Lightroom version of duplicates

[James Gray]

I ran into an issue with Lightroom that makes no sense to me.  Maybe somebody knows a reason for this.  Or maybe it is a bug.

I discovered by accident tonight that I had some missing raw files in my main Photo folder.  I am sure it was due to human error.  I found the missing files in my backup folder that LR does not know about.  I copied them to my main Photo folder.  I found that LR did not have those raw files imported/cataloged.  So, I tried to synchronize the individual folders.  It did not work.  LR told me the number of missing files that matched my manual counts.  However, LR would not import them!  I did some Google searches and found a workaround.  To me, it is a kludge.  The kludge is to allow LR to import suspected duplicates.  To me, that makes no sense as those raw files were not duplicates of any images in my catalog.  Why does LR work this way?  I have told LR to treat jpegs and corresponding raw files as separate files.

Jim Gray

[Keith Bauer]

Jim:

Based on the info you provided this is likely the issue.

The files that you think are missing aren't really missing.  They are just in a different physical folder than the one you are trying to synchronize.  That would explain why when you try to sync, you are told the files are already in your catalog.

To locate them:

Set you search context to All Photographs (just click on All Photographs under the Catalog section on the left panel of LR)

Click on Text in the Filter Bar

Put in the file name of one of the files that you think is missing and do the text based search.

The results will show the file (or possibly files) if there are more than one that match the search criteria.

Right Click on the file from the search results and choose  Go to Folder in Library.

That will take you to the folder in your library structure on the left panel where the file physically resides.

My guess is that folder will not be the folder where you think the files should reside.  

--
Remember: Please sign your posts with your real name.
---
You received this message because you are subscribed to the Google Groups "elcc-chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elcc-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAGLETg7X-Ti5QStOH4Gux8K9XqnMHthLmbL9xqEA-NNGqOW8wg%40mail.gmail.com.

[James Gray]

Hi Keith,

Thanks for taking the time to respond.  Let me first say that I do have instances like you describe.  Like many of us, my initial "organization" of my digital images left a lot to be desired.  In the last few years, I have better organized my (and my wife's) images.  I have one large external drive that contains my "Main Photo" folder where all new digital images are loaded since about 2015.  I have not attempted to thoroughly clean up the old organization.  Instead, I am counting on Lightroom to create a logical organization when there is some chaos in the physical location of some files.  I have 5 drives with images on them.  Besides "Main Photo" there is "Backup of Main Photo", "Originals raw-jpeg", latest jpegs-plus, J-Photo (which contains old images that are cataloged.  Almost all of the images in the LR catalog are either in Main Photo or J-Photo.  A lot of the images on J-Photo are duplicated in Main Photo.  In each of those cases, LR has cataloged one of them.  In any case, I am quite aware of the fact that when I am in one folder in LR that LR will say there are missing images, but LR won't import them because the same image is already in the LR catalog.

That is not what is going on in the situation I ran into yesterday.  I used an app called "Find Any File" to find the missing files.  They were in the "Backup of Main Photo".  I have already used Bridge to copy each of the missing raw files into the appropriate folder in "Main Photo".  Then I attempted to use LR to Synchronize the folder.  LR reported the number of missing files that matched my manual count.  It would not import the missing raw files until I unchecked the "Don't import Suspected Duplicates".  Everything looked OK.  At your suggestion, I set the search context to all photographs.  I  did a spot check of one of the files that were missing.  The result from the text-based search is that it found the image I imported yesterday that could be a duplicate.  However, it also found a catalog entry of the same filename that it says is missing.  This is the msg from LR:

"_DSC0925.ARW" could not be used because the original file could not be found.  Would you like to locate it?

Then it shows the path to what it says is the old location.  I don't understand how it came up with the old location.  I suppose the senior moment that resulted in the accidental deletion of the original raw file was longer than I realized.  Now I wonder what I should do now and what I should have done.  If I had not copied the files back into the folder they should have been in, I would not have wanted LR to find the file in the backup folder.  I do not know if clicking Cancel will delete the "virtual duplicate".  Since there were about 50 missing raw files I thought I had taken care of.  I wonder if I have about 50 of these "virtual duplicates" I need to clean up. if so, I need to figure out how.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAPY2%3Dz-8EQL-hxup4sY%2B_EE_9ZQSSLNjDdwm8Wr9pn8YSfL1HA%40mail.gmail.com.

[Keith Bauer]

_DSC0925.ARW" could not be used because the original file could not be found.  Would you like to locate it?

That message means that you deleted or moved the original file outside of LightRoom, i.e. using Windows File Explorer, or Mac Finder.  ALL file management should be done using Lightroom to avoid this exact problem.

When I teach LR, this is the analogy that I use.

The LR catalog is exactly like the card catalog in a physical Library.  Whether that card catalog is electronic or physical like it was in the old days, the entries in the catalog simply reference the real books on the shelf.  If you go to the shelf, and move or remove a book, the catalog will no longer be accurate.  In LightRoom that means using Windows File Explorer or Mac Finder to move, rename or delete images will leave the LR catalog in a state with Missing Images.  Solution.... Never do that.

Your question about deleting the "virtual duplicate".  Be careful with the word Virtual as you can have "Virtual Copies" created inside of lightroom.  I think you are just referencing images that are missing because the catalog and the physical files are not in sync.  You can delete the missing files the same way as deleting any image in LR.  You'll just get a message that says the original file can't be found and the catalog entry will be removed.

If you want to find all missing files in your entire catalog, go to the Library Menu option at the top of the screen and choose "Find All Missing Photos".  That will create a collection which has all catalog entries that are like the one you referenced above.  You can then decide if you just want to delete those catalog entries or go figure out where they are and re-sync them.

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAGLETg5NC5Lg3sF9jzZiGZNFAzwt%2BgVGqOi4VnDay1WCKzHQsg%40mail.gmail.com.

[James Gray]

Hi Keith,

Thank you again for your reply.

I put the words "virtual duplicate" in quotes knowing that Virtual Copy is a concept in Lightroom and I hope I did not confuse anyone.

I worked as a database software engineer from 1984 to 2019 and understand what the catalog in LR is.  As I see it, LR is an outstanding database system.

This has nothing to do with the original issue of this thread.  Last week I had a hard disk act flakey.  I could not access anything on the drive using Bridge, Photoshop, and some other software.  I tried the disk utility first aid on the drive.  The utility said it could not do first aid and I should try to copy the data.  After unplugging the drive for over 5 minutes I found that Finder would work.  I will confess that I had forgotten that LR pointed to some of the images on this drive.  I had a spare larger drive.  So I used Finder to copy the folders to the larger drive.  It seemed to work fine, so I disconnected the flakey drive and attached the larger drive with the files.  I confess I do not know how I could have used LR to avoid the missing files.  I will point out that the vast majority of the images on the larger drive are not imported into LR as they are duplicates of images on Main Photo.

I will add that up to this point Library/Synchronize folder has worked perfectly for me when I have added or deleted images.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAPY2%3Dz9EN4u%3DOedOnyZZTck6UKrQj%2B7Cso7ZNgxyzmjaqwJdrw%40mail.gmail.com.

[James Gray]

I have tracked down all the supposedly missing files that Lightroom said existed.  The cleanup was a lot easier than I thought it would be.  The problem I have with trying to use LR for ALL file management is that LR only knows about one copy of each image file.  I have multiple copies of each image I care about.  I need to use Bridge or Finder to manage those files that are not in the LR catalog.  I still think each photographer should have at least 3 copies of each image they consider important.  It is too bad there is not an easy-to-spot field in the metadata that tells if an image is cataloged in LR.

Jim

[Keith Bauer]

Jim:

When I said use LR for ALL data management, that obviously means for all images that are managed in the LR catalog.  Clearly it doesn't mean use LR to manage backups.  

Agree that it is important to have multiple backups.  I maintain 5 copies of all images and I NEVER touch Finder or Bridge for data management on any backup drive.

If you don't have, or know about Carbon Copy Cloner, you need to learn about it and use it.  It is a backup tool for the Mac (only Mac) that creates perfect clones of any "Master" image drive to any number of backup drives.

Any new images imported to LR, any changed images in LR, any renamed images will be cloned to the backup drive(s) when you run the backup software.   Any deleted images in LR will be deleted from the backup drive(s) when you run the backup software.   You never need to deal with manual intervention on backup drives to make sure that any and all changes you made to your master images on master drive(s) are cloned to the backup drive(s).  

Carbon Copy Cloner can be found at https://bombich.com

They offer a free 30 day trial and when you choose to buy it, it will set you back a whopping $39.99

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAGLETg51o-zVG3yOMngNXP5pOBk98kB4FUig%3DnjKTbcpLix_CA%40mail.gmail.com.

[James Gray]

Hi Keith,

Thank you for responding again.  I have thought about this and was planning to ask about this.  You have answered most of my questions.  I am hoping some other individuals will describe their way of managing backups and other intentional redundancy.  I would not like the feature of Carbon Copy Cloner that you say will delete from the backup drives any image that is deleted from the Master folder (the one I call Main Photo).  To me, the main reason for backups is to protect against hardware failure and loss, but also to protect you from human error.   I cannot count the number of times I have had to go back to a backup to retrieve a file that was accidentally deleted or messed up beyond recognition.  Some of these were not my errors but those of my wives and sons.  I will take a look at Carbon Copy Cloner.  I assume there are options for configuring it differently than the way you have it configured.  You apparently are far less error-prone than me or the people I have lived with.

One reason I am still learning to use Lightroom is I only implemented it in 2020.  I originally loaded it in 2011 and when I started to try to use it, I got messages that I was sure were going to cause me to run out of disk space.  At the time I was unable to find out what Lightroom was actually doing in "importing" images.  I really dislike the term "importing" as a former database programmer and database manager.  What Lightroom is really doing is cataloging the images.  Clearly, Lightroom can simultaneously import and catalog new images.  I will also say that I heard too many horror stories from people using Lightroom that intimidated me.  Just to be clear, I am very glad I implemented Lightroom.  I think it is great.

In your very first reply on this thread, you suggested that a reason why LR would not import some images in a folder was because the images were cataloged from a different folder.  I do have a lot of that.  In other words, I have a lot of folders where some images in the folder are cataloged in LR and other images in those folders are not in the LR catalog.  Those image files cannot be imported because the exact same image file is already in the LR catalog from a different folder.  The only way to manage all of the images in one of those folders is by using Bridge or Finder.  The only way to attempt to clean up such issues depends on using Bridge or Finder.  Doing the cleanup is very time-consuming and error-prone.  I can see no reason to attempt a full cleanup when Lightroom can provide a logical structure to all of my images without having to have a clean logical structure to all of my physical image files.

I will add another thing.  I have found that Lightroom is much more flexible and forgiving than I was given the impression between 2011 and 2020.  I have a hybrid system of using Bridge and Lightroom and it works well for me.  It is very easy to import image files that I have saved from Adobe Camera Raw and Photoshop.  It is also easy for LR to find deleted files and delete them from the catalog.  Yes, there are a few hiccups.  I am still learning.  I would say for current Bridge users that there can be advantages to a hybrid system.  You may find it works best for you and most other users to never touch image files with Bridge, Finder, or Explorer.  It works for me to continue using Bridge and it could work for some others who do not want to do a lock stock and barrel switch from Bridge to Lightroom.

I hope this helps some others who read this.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAPY2%3Dz_BPM_yks06KKgh3XNu2d_regqEsuO76XpjYtACV%2B7XsA%40mail.gmail.com.

[Keith Bauer]

" I would not like the feature of Carbon Copy Cloner that you say will delete from the backup drives any image that is deleted from the Master folder (the one I call Main Photo). "

Jim:  As I noted in my e-mail about Carbon Copy Cloner, I suggest learning about and seeing what it can and can't do.  There is a feature that is called the Safety Net which addresses this exact concern.

All I can tell you is I've implemented Carbon Copy Cloner on dozens and dozens of client machines (Macs).  It is hands down the best backup solution for Macs.  Of course it requires understanding what it does and how to make it the correct custom solution for any image organizational method.  

Good luck with your organization of your images.  

[Colin Barnett]

I've enjoyed this detailed discussion of the inner workings of LR. I want to mention the two-part strategy I use for backup. I use BackBlaze to back up my and my wife's computer. It does it automatically as you save files. There are several features I like about it. If you need a file, you can restore it easily. If the amount of files you need is large, they will mail you a thumb drive or a hard drive. When you copy the files, you can send the hardware back or keep it if you want to buy it. In addition, you can activate an option where you have access to your files back in time, for up to a year. So if you are, for example, working on a database and you want to have a copy of the database from yesterday or last month, you can do that with BackBlaze. The entire process is seamless; I forget that I am even backing up my files.

The second part of my backup is to use Goodsync to backup my and my wife's files to an external drive. Goodsync runs locally on my computer and backs up any computer attached to my home network. I run Goodsync manually most nights; it is a one-click process. You can also schedule automatic backups. Goodsync copies your new files and deletes any files that you have deleted on your computer.

With BackBlaze and Goodsync I am well protected against hard drive failures. And in case of a fire, I only need to grab my external drive and I have all my files. If my house is destroyed when I am not home, I need to buy a new computer and restore all my files from the hard drive that BackBlaze will send me.

With Lightroom, I make backups during import, but I don't really regard these as actual backups of my images because they are just the RAW or JPG files with no post-processing. Moreover, I delete a lot of imported photos after I review them. This means that probably 70% of my LR backups are of files that I have already decided I don't want. But I don't go into those backed up files to delete them.

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAGLETg5GvtPihWBagda5Frkpfv9Hq3%3DjaBKOJuHP7yLJrcPf%3Dg%40mail.gmail.com.

--
______________________________________
Colin Barnett
5715 Lost Dutchman NE
Albuquerque NM 87111
443-996-3501
colinba...@gmail.com
http://www.flickr.com/photos/cbarnett88/

[James Gray]

Hi Colin,

Thanks for responding.  At this point, I am more inclined toward Carbon Copy Cloner.  I also save my raw and initial jpegs to a separate drive.  Any type of file redundancy and duplication can function as a backup.

I think I have a clue why I have quite a few cataloged images distributed in different folders is related to how I imported all of my files in 2020.  At that time I had more than 100,000 image files distributed across 5 hard drives.  I should have been more patient in importing the different drives.  I think some cataloging was going on in the background that was not complete when I tried to import images from a 2nd, 3rd, etc drive.  I hope that makes sense.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/af77ff00-ec7e-10d9-8d82-bb9c9da4a578%40gmail.com.

[James Gray]

Thanks for wishing me luck.  The process is slow mostly because I would rather post-process images than organize them.

I did look at the Carbon Copy Cloner website and it does appear to have the flexibility I need.  I can also see I need to do more organizing before implementing a new backup method.

I appreciate your feedback as it helped me a lot.

Jim

--

Remember: Please sign your posts with your real name.
---
You received this message because you are subscribed to the Google Groups "elcc-chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elcc-chat+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elcc-chat/CAPY2%3Dz_Scjp95OcbU8Gwf4wO6ZmctbaZ_2Y%3DPEfoUOGi5hPauw%40mail.gmail.com.