Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Comparing differing directory trees
14 views
Skip to first unread message
Dave Higton
Apr 28, 2013, 12:01:37 PM4/28/13
to
I have roughly 1600 photos from various cameras, stored on the
Iyonix's hard drive and again on two NAS drives. Although the
two NAS copies should be identical, there are differences in
directory structure between Iyonix and NAS. Also, when the NAS
drives were used on the NSLU2 (before I changed to a Synology
USBStation 2), the file date/time stamps don't always match.
The things I can rely on are the file name and exact file size.
Any suggestions as to what RISC OS apps might help me compare
the contents?
I'd like to remove them from the Iyonix and keep them only on
NAS, but be sure everything has been copied over.
Dave
Iyonix's hard drive and again on two NAS drives. Although the
two NAS copies should be identical, there are differences in
directory structure between Iyonix and NAS. Also, when the NAS
drives were used on the NSLU2 (before I changed to a Synology
USBStation 2), the file date/time stamps don't always match.
The things I can rely on are the file name and exact file size.
Any suggestions as to what RISC OS apps might help me compare
the contents?
I'd like to remove them from the Iyonix and keep them only on
NAS, but be sure everything has been copied over.
Dave
Steve Fryatt
Apr 28, 2013, 1:57:44 PM4/28/13
to
On 28 Apr, Dave Higton wrote in message
<e77d89435...@my.inbox.com>:
> I have roughly 1600 photos from various cameras, stored on the Iyonix's
> hard drive and again on two NAS drives. Although the two NAS copies
> should be identical, there are differences in directory structure between
> Iyonix and NAS. Also, when the NAS drives were used on the NSLU2 (before
> I changed to a Synology USBStation 2), the file date/time stamps don't
> always match. The things I can rely on are the file name and exact file
> size.
>
> Any suggestions as to what RISC OS apps might help me compare the
> contents?
DirSync, which can be found via PackMan. It will compare two directories
(and their children) using a range of options, and you don't have to
synchronise (any or all of) them at the end. I can't see a "don't compare
datestamps" option, but you can be flexible to handle issues with NASs.
--
Steve Fryatt - Leeds, England
http://www.stevefryatt.org.uk/
<e77d89435...@my.inbox.com>:
> I have roughly 1600 photos from various cameras, stored on the Iyonix's
> hard drive and again on two NAS drives. Although the two NAS copies
> should be identical, there are differences in directory structure between
> Iyonix and NAS. Also, when the NAS drives were used on the NSLU2 (before
> I changed to a Synology USBStation 2), the file date/time stamps don't
> always match. The things I can rely on are the file name and exact file
> size.
>
> Any suggestions as to what RISC OS apps might help me compare the
> contents?
(and their children) using a range of options, and you don't have to
synchronise (any or all of) them at the end. I can't see a "don't compare
datestamps" option, but you can be flexible to handle issues with NASs.
--
Steve Fryatt - Leeds, England
http://www.stevefryatt.org.uk/
Dave Symes
Apr 28, 2013, 3:14:15 PM4/28/13
to
In article <mpro.mlz8js09...@stevefryatt.org.uk>,
Indeedy, I use DirSync between this SARPC and my NAS (Synology DS110j) and
very good it is.
I use it to check/compare on the fly, but I have also created Hotlist I
use for compare/backups between SARPC and the NAS.
(After created, accessed easily with the Adjust button over DirSync
Iconbar icon.)
Dave
FWIW: I also use it from a VRPC install on which I have a mirror of this
ancient SARPC.
D.
--
Dave Triffid
very good it is.
I use it to check/compare on the fly, but I have also created Hotlist I
use for compare/backups between SARPC and the NAS.
(After created, accessed easily with the Adjust button over DirSync
Iconbar icon.)
Dave
FWIW: I also use it from a VRPC install on which I have a mirror of this
ancient SARPC.
D.
--
Dave Triffid
Tony Moore
Apr 28, 2013, 3:48:51 PM4/28/13
to
On 28 Apr 2013, Steve Fryatt <ne...@stevefryatt.org.uk> wrote:
> On 28 Apr, Dave Higton wrote in message
> <e77d89435...@my.inbox.com>:
>
> > I have roughly 1600 photos from various cameras, stored on the
> > Iyonix's hard drive and again on two NAS drives. Although the two
> > NAS copies should be identical, there are differences in directory
> > structure between Iyonix and NAS. Also, when the NAS drives were
> > used on the NSLU2 (before I changed to a Synology USBStation 2), the
> > file date/time stamps don't always match. The things I can rely on
> > are the file name and exact file size.
> >
> > Any suggestions as to what RISC OS apps might help me compare the
> > contents?
>
> DirSync, which can be found via PackMan. It will compare two
> directories (and their children)
True, but if there are 'differences in the directory structure between
> On 28 Apr, Dave Higton wrote in message
> <e77d89435...@my.inbox.com>:
>
> > I have roughly 1600 photos from various cameras, stored on the
> > Iyonix's hard drive and again on two NAS drives. Although the two
> > NAS copies should be identical, there are differences in directory
> > structure between Iyonix and NAS. Also, when the NAS drives were
> > used on the NSLU2 (before I changed to a Synology USBStation 2), the
> > file date/time stamps don't always match. The things I can rely on
> > are the file name and exact file size.
> >
> > Any suggestions as to what RISC OS apps might help me compare the
> > contents?
>
> DirSync, which can be found via PackMan. It will compare two
> directories (and their children)
Iyonix and NAS', the directories would have to be compared one by one.
> using a range of options, and you don't have to synchronise (any or
> all of) them at the end. I can't see a "don't compare datestamps"
> option
different date/timestamp
Tony
Evan Clark
Apr 28, 2013, 7:35:16 PM4/28/13
to
In article <e77d89435...@my.inbox.com>,
depending on how your photos are organised and what other files may
be in the same directories:
Using Locate, search for all objects excluding Applications and
Directories and including the filetype(s) of your photos.
From the results window, save the path names and use this as the
basis to create an obey file. Using your favourite text editor, add
'info ' to the start of each line and at the end of each line add
redirection to send the output to another file, e.g. { >>
<path>.<filename> } (except just a single '>' on the first line).
Run the obey file and sort the lines in the resulting output file
(you may want to edit out the permissions, filetype, time and date
from each line before sorting).
Repeat this for each device and compare the results with SideDiff.
(Thanks to Steve Fryatt and Harriet Bazley for their invaluable
applications Locate and SideDiff.)
Hope this is comprehendible.
Evan.
Dave Higton <da...@davehigton.me.uk> wrote:
> I have roughly 1600 photos from various cameras, stored on the
> Iyonix's hard drive and again on two NAS drives. Although the
> two NAS copies should be identical, there are differences in
> directory structure between Iyonix and NAS. Also, when the NAS
> drives were used on the NSLU2 (before I changed to a Synology
> USBStation 2), the file date/time stamps don't always match.
> The things I can rely on are the file name and exact file size.
> Any suggestions as to what RISC OS apps might help me compare
> the contents?
I'd try something like this - you may have to make adaptations
> I have roughly 1600 photos from various cameras, stored on the
> Iyonix's hard drive and again on two NAS drives. Although the
> two NAS copies should be identical, there are differences in
> directory structure between Iyonix and NAS. Also, when the NAS
> drives were used on the NSLU2 (before I changed to a Synology
> USBStation 2), the file date/time stamps don't always match.
> The things I can rely on are the file name and exact file size.
> Any suggestions as to what RISC OS apps might help me compare
> the contents?
depending on how your photos are organised and what other files may
be in the same directories:
Using Locate, search for all objects excluding Applications and
Directories and including the filetype(s) of your photos.
From the results window, save the path names and use this as the
basis to create an obey file. Using your favourite text editor, add
'info ' to the start of each line and at the end of each line add
redirection to send the output to another file, e.g. { >>
<path>.<filename> } (except just a single '>' on the first line).
Run the obey file and sort the lines in the resulting output file
(you may want to edit out the permissions, filetype, time and date
from each line before sorting).
Repeat this for each device and compare the results with SideDiff.
(Thanks to Steve Fryatt and Harriet Bazley for their invaluable
applications Locate and SideDiff.)
Hope this is comprehendible.
Evan.
Jeremy Nicoll - news posts
Apr 29, 2013, 3:20:27 PM4/29/13
to
Dave Higton <da...@davehigton.me.uk> wrote:
>The things I can rely on are the file name and exact file size.
And hopefully content!
>The things I can rely on are the file name and exact file size.
I'm out of touch with RO apps these days but I'm fairly sure there's a CLI
tool that generates a file hash/checksum of a file; I'd be inclined to use
*repeat (or similar) to iterate that down through the file trees concerned.
I'd think I'd try to create a listing with <checksum> <filepath> on each
line, and sort the whole thing by the checksum value. Then merge the three
listings and then remove each item that had the same checksum value listed
three times.
--
Jeremy C B Nicoll - my opinions are my own.
Email sent to my from-address will be deleted. Instead, please reply
to newsre...@wingsandbeaks.org.uk replacing "aaa" by "284".
Nat Queen
Apr 29, 2013, 8:04:23 PM4/29/13
to
In message <mpro.mm1723...@wingsandbeaks.org.uk.invalid>
Jeremy Nicoll - news posts
my website (http://www.queen.clara.net/pgp/acorn.html). The command
"md5 <file>" produces the MD5 hash of a file, plus the filename.
> I'd think I'd try to create a listing with <checksum> <filepath> on each
> line, and sort the whole thing by the checksum value. Then merge the three
> listings and then remove each item that had the same checksum value listed
> three times.
That should be possible, using the 'cipher' module.
A possible alternative, perhaps more useful for single files, is Paul
Vigay's application !CRC (http://www.vigay.com/software/index.html),
which calculates a 32-bit checksum.
Nat
--
Dr. N.M. Queen | Phone: +44 121 414 6595 Fax: +44 121 414 3389
School of Mathematics | PGP-encrypted e-mail preferred. Public key at
Univ. of Birmingham | http://www.queen.clara.net and on keyservers.
Birmingham B15 2TT, UK | Info: http://www.queen.clara.net/pgp/pgp.html
Jeremy Nicoll - news posts
<jn.nntp....@wingsandbeaks.org.uk> wrote:
> Dave Higton <da...@davehigton.me.uk> wrote:
>>The things I can rely on are the file name and exact file size.
> And hopefully content!
> I'm out of touch with RO apps these days but I'm fairly sure there's a CLI
> tool that generates a file hash/checksum of a file; I'd be inclined to use
> *repeat (or similar) to iterate that down through the file trees concerned.
Several such utilities exist. Try the 'cipher' module, available from
> Dave Higton <da...@davehigton.me.uk> wrote:
>>The things I can rely on are the file name and exact file size.
> And hopefully content!
> I'm out of touch with RO apps these days but I'm fairly sure there's a CLI
> tool that generates a file hash/checksum of a file; I'd be inclined to use
> *repeat (or similar) to iterate that down through the file trees concerned.
my website (http://www.queen.clara.net/pgp/acorn.html). The command
"md5 <file>" produces the MD5 hash of a file, plus the filename.
> I'd think I'd try to create a listing with <checksum> <filepath> on each
> line, and sort the whole thing by the checksum value. Then merge the three
> listings and then remove each item that had the same checksum value listed
> three times.
A possible alternative, perhaps more useful for single files, is Paul
Vigay's application !CRC (http://www.vigay.com/software/index.html),
which calculates a 32-bit checksum.
Nat
--
Dr. N.M. Queen | Phone: +44 121 414 6595 Fax: +44 121 414 3389
School of Mathematics | PGP-encrypted e-mail preferred. Public key at
Univ. of Birmingham | http://www.queen.clara.net and on keyservers.
Birmingham B15 2TT, UK | Info: http://www.queen.clara.net/pgp/pgp.html
davehi...@gmail.com
May 1, 2013, 8:26:18 AM5/1/13
to
In the end, there didn't seem to be any set of apps that would
do what I want without considerable difficulty. I really do
mean that the directory structures are different. Really
different. I opted to write something in BASIC, which seems
to be doing a good job.
I realised that something like 300 photos were hidden in plain
sight - from my first digital camera, and I had never copied
them to NAS!
I'm sceptical as to whether it is necessary to compare file
checksums or hashes. The file names are still those assigned
by the cameras and are unique. If the leafname and size are
identical, I take it that the contents are identical too. If
we can't copy a file and get a perfect copy every time, there
is no value in storage.
Dave
Jeremy Nicoll - news posts
May 2, 2013, 2:11:58 AM5/2/13
to
davehi...@gmail.com wrote:
> I'm sceptical as to whether it is necessary to compare file checksums or
> hashes. The file names are still those assigned by the cameras and are
> unique. If the leafname and size are identical, I take it that the
> contents are identical too.
Perhaps; checking CRCs/hashes is a quick way to find out though.
> I'm sceptical as to whether it is necessary to compare file checksums or
> hashes. The file names are still those assigned by the cameras and are
> unique. If the leafname and size are identical, I take it that the
> contents are identical too.
However I didn't realise from your original description that (despite
directory structures differing) the files' leafnames were the same in each
set of files.
If for example you have (as I do for reasons I needn't go into) sets of
files with for example one set just having mainly names like
mypage
and another has mainly
mypage.htm
and another has mainly
mypage.html,faf
etc, then it's nothing like as simple to work out where the duplicates are.
I'm working through a set of backups at the moment trying to weed out
duplicates; in some cases I just use CRC/hashes, while in others (eg for
JPEGs which can easily be identified by their first few bytes) I'm bulk
renaming them to strip whatever leafname suffixes they have and set a
standard form. Then normal compares work ok.
Dave Higton
May 2, 2013, 3:18:25 PM5/2/13
to
In message <mpro.mm5qjy...@wingsandbeaks.org.uk.invalid>
if present, I truncated it to before the slash. As it happens, the
"extensions" seemed to be the same on both drives, so it made no
difference.
Dave
> davehi...@gmail.com wrote:
>
> > I'm sceptical as to whether it is necessary to compare file checksums or
> > hashes. The file names are still those assigned by the cameras and are
> > unique. If the leafname and size are identical, I take it that the
> > contents are identical too.
>
> Perhaps; checking CRCs/hashes is a quick way to find out though.
>
> However I didn't realise from your original description that (despite
> directory structures differing) the files' leafnames were the same in each
> set of files.
>
> If for example you have (as I do for reasons I needn't go into) sets of
> files with for example one set just having mainly names like
>
> mypage
>
> and another has mainly
>
> mypage.htm
>
> and another has mainly
>
> mypage.html,faf
>
> etc, then it's nothing like as simple to work out where the duplicates are.
> I'm working through a set of backups at the moment trying to weed out
> duplicates; in some cases I just use CRC/hashes, while in others (eg for
> JPEGs which can easily be identified by their first few bytes) I'm bulk
> renaming them to strip whatever leafname suffixes they have and set a
> standard form. Then normal compares work ok.
Oh yes, I foresaw that problem. I looked for a slash in the name;
>
> > I'm sceptical as to whether it is necessary to compare file checksums or
> > hashes. The file names are still those assigned by the cameras and are
> > unique. If the leafname and size are identical, I take it that the
> > contents are identical too.
>
> Perhaps; checking CRCs/hashes is a quick way to find out though.
>
> However I didn't realise from your original description that (despite
> directory structures differing) the files' leafnames were the same in each
> set of files.
>
> If for example you have (as I do for reasons I needn't go into) sets of
> files with for example one set just having mainly names like
>
> mypage
>
> and another has mainly
>
> mypage.htm
>
> and another has mainly
>
> mypage.html,faf
>
> etc, then it's nothing like as simple to work out where the duplicates are.
> I'm working through a set of backups at the moment trying to weed out
> duplicates; in some cases I just use CRC/hashes, while in others (eg for
> JPEGs which can easily be identified by their first few bytes) I'm bulk
> renaming them to strip whatever leafname suffixes they have and set a
> standard form. Then normal compares work ok.
if present, I truncated it to before the slash. As it happens, the
"extensions" seemed to be the same on both drives, so it made no
difference.
Dave
0 new messages