How do you back up your work?

[Dallas]

I clone my whole drive and keep the drive off-site. Trouble is... at 5
megs a shot I'm filling up my drive pretty fast and I don't feel like
going to the trouble of upgrading to a new disk and moving everything
onto it, blah, blah, blah.

It strikes me that backing up photos is different than backing up a
drive with an operating system, all you need to do is back up the
actual files.

I have a few thoughts, a USB disk drive, a USB solid state drive,
several large flash drives... even burning DVDs (after all those are
non-volatile and can be read on any machine.)

I'm liking an idea of putting all my photos on a portable USB drive and
using it as the working photo drive... just put all photos on it, then
perhaps using one of the Internet based "cloud" backup services to
backup everything on that drive.

That would cover virus destruction, computer theft, total loss in a
fire and drive failure.


--
Dallas

[Alan Browne]

On 2012-03-07 17:14 , Dallas wrote:
>
> I clone my whole drive and keep the drive off-site. Trouble is... at 5
> megs a shot I'm filling up my drive pretty fast and I don't feel like
> going to the trouble of upgrading to a new disk and moving everything
> onto it, blah, blah, blah.
>
> It strikes me that backing up photos is different than backing up a
> drive with an operating system, all you need to do is back up the
> actual files.
>
> I have a few thoughts, a USB disk drive, a USB solid state drive,
> several large flash drives... even burning DVDs (after all those are
> non-volatile and can be read on any machine.)

DVD's, even if kept in benign conditions (dry, cool, dark) have a life
expectancy of about 5 - 10 years. Plan on re-burning or migrating the
data in 5 years.

There are "Gold" standard DVD's that should keep for 100+ years (cool,
dry, dark) but be sure they are manufactured in Taiwan. The Indian ones
have proven faulty to date.

New storage technology will be coming out in the next few years that
should exceed 1000 years.

> I'm liking an idea of putting all my photos on a portable USB drive and
> using it as the working photo drive... just put all photos on it, then
> perhaps using one of the Internet based "cloud" backup services to
> backup everything on that drive.

For the volume of photos I have, scans, raw, finished in various sizes,
crops, printer preps, that would be a horrendous amount of cloud. It
would precipitate and people would drown in pixels.

> That would cover virus destruction, computer theft, total loss in a
> fire and drive failure.

[1] First off I have automatic backups 2x daily onto a "on beyond RAID"
drive (Drobo). This machine has 4 drives and I can hot swap at any time
and with any sized disk (equal or larger size). So if a drive fails,
the light turns red - I can still work - no losses, and then pull the
"red" drive and put in a new drive. Restoration is automatic and
transparent. And I can keep working through it all.

Further, I can upgrade any of the 4 drives at any time without shutting
down the unit. Just pull any drive out and slap in a new one. The
drive then rebuilds the redundant data. When the 4 lights are green
again, I can then add another drive... and so on.

[2] Secondly, all my photo work (projects) are on individual DVD sets.
(1 or more disks).

[3] Thirdly, all my photo work is also stored on 2 other HD's with those
volumes being updates on a project by project basis.

The sole weakness in my system at present is I have no offsite storage.
I should really keep some drives over at my buddy's house or across
the street at my neighbour's house. And rotate these every couple months.

--
"I was gratified to be able to answer promptly, and I did.
I said I didn't know."
-Samuel Clemens.

[(PeteCresswell)]

Per Dallas:

>I clone my whole drive and keep the drive off-site. Trouble is... at 5
>megs a shot I'm filling up my drive pretty fast and I don't feel like
>going to the trouble of upgrading to a new disk and moving everything
>onto it, blah, blah, blah.

I use a half-dozen naked one-terabyte drives.

Instead of being in USB wrappers, I pop them in to a sled in the
PC that is made for that purpose.

The application I use is called Second Copy. It backs up
everything I tell it about to whatever drive is in the sled and I
tell it to keep 10 old copies of anything that changes.

I probably ought to shuffle the drives more often, but I do it
when I think of it.

The drive being used goes into my car. The drive in my car goes
in to my garden shed, another drives lives at one remote
location.

The other drives sit on top of my desk.

Right after I pop a new drive in to the sled and just before I
remove a drive from the sled I run a ChkDsk on the drive. They
are usually OK, but not always.

If that seems like overkill, consider:

- You can fry 2-3 drives just from a flaky card and
not realizing that the card is bad until several drives
have been ruined. Been there done that.... and that's
part of the reason for the remote drive - to save me from
my own stupidity. The rule there is that if I ever get
down to the remote drive, I back it up on another PC before
I bring it home.

- Drives are cheap. 1-TB drives are less than a hundred bucks
and smaller drives are even cheaper.


Also, something else that I'm prone to rant about: keep your
data and your system on separate drives - or at least in separate
partitions. That reduces your exposure to malware and makes
restoring your sys from a known good image without losing data
much easier.

There's more.... but that's the essence of it for data.

System is another rant.

-
--
Pete Cresswell

[mike]

I'm not a pro, I would only loose memories if all my digital pics
disappeared so I look for a quick, cheap and simple solution.

I keep all my pics in folders by year on my PCs HD which is
automatically backed up weekly, cover for HD failure.

I backup the pics onto DVD at the end of the year and keep the DVDs at
my mums house in case of fire/theft of the PC.

I back up recent (i.e. since the last DVD) pics & important files onto a
portable HD every week or so or so and keep that in a fire safe along
with insurance policies etc. This is in case of the PC being stolen.

Worst case scenario and the house burns down and it's hot enough to
destroy the contents of the safe, I will loose a years pics. If that
happens I wont be worrying about pictures ....

I switched to DVD media from CDR a five years or so ago so I'll have to
consider re-burning them soon.

Mike

[Dallas]

Alan Browne wrote:

> Further, I can upgrade any of the 4 drives at any time without
> shutting down the unit. Just pull any drive out and slap in a new
> one. The drive then rebuilds the redundant data. When the 4 lights
> are green again, I can then add another drive... and so on.

Salivating... that's a backup system.

But yeah, you'd be wiped out in a fire. I do keep a third clone of the
HD at the bank in a safe deposit box. In theory, I should swap it once
a month, but it winds up being more like 2-3 months. Still, much
better than losing it all.

--
Dallas

[Dallas]

(PeteCresswell) wrote:

> The rule there is that if I ever get
> down to the remote drive, I back it up on another PC before
> I bring it home.

That is wise... if you get infected with sophisticated malware you
can't let any device connect to the infected machine until it's
certified clean.

I had hardware going out that acted just like a virus on the hard
drive. For all practical purposes, it had to be treated like a virus.
I wish I could have that 50 or so hours back spent on that endeavor.

--
Dallas

[Robert Coe]

On Thu, 08 Mar 2012 17:23:10 -0600, "Dallas" <Cybnorm@spam_me_not.Hotmail.Com>
wrote:

You get 95% of the benefit by just throwing your next-to-last backups, on
whatever media you have available, into the trunk of your car. I've never
understood why so many people find that riduculously simple procedure to be
beyond their reach. It's far from the most sophisticated backup method, but it
allows you to sleep nights if you have nothing better.

Bob

[tcroyer]

I have a Carbonite account to cover most of my system. But, since Carbonite
doesn't back up external drives, once a week, I manually copy that (with
most of my new photo work) to a removable, portable 1T drive.

If you have all your work on an internal drive, Carbonite makes the back up
date essentially invisible. Cost is OK, too: $59 per year.

--
Tom Royer
If you're not free to fail, you're not free. -- Gene Burns
"Dallas" <Cybnorm@spam_me_not.Hotmail.Com> wrote in message
news:evydncAXfeBCQMrS...@earthlink.com...

[Pete A]

On 2012-03-09 14:37:48 +0000, Joel said:

> I just burn mine to DVDs. Stories I read from some experts sharing what
> they do
>
> 1. They backup to a hard drive
>
> 2. Then they worry that the drive may go bad so they backup to other drive
>
> 3. With 2 hard drives that feel little safer so they find some other thing
> to worry by worrying that the house may get fired. So they backup to
> another hard drive to keep at the spouse's parent house (his/her parent may
> be too far or some other reason), then they worry that they may get divorced
> and will never see the backup again
>
> 4. And they backup to the 3rd or 4th hard drives.

5. Then they worry that the hard drive interface (SCSI, IDE etc.) will
become obsolete.

[Alan Browne]

On 2012-03-08 18:23 , Dallas wrote:
> Alan Browne wrote:
>
>> Further, I can upgrade any of the 4 drives at any time without
>> shutting down the unit. Just pull any drive out and slap in a new
>> one. The drive then rebuilds the redundant data. When the 4 lights
>> are green again, I can then add another drive... and so on.
>
> Salivating... that's a backup system.
>
> But yeah, you'd be wiped out in a fire.

Something I do plan to rectify shortly. It's the "do" part I have to
get round to... (Just need to buy a few 2 TB disks to setup a rotation.
I have a "toaster" style drive (EZ Dock-2) that's appropriate for the
scheme.

> I do keep a third clone of the
> HD at the bank in a safe deposit box. In theory, I should swap it once
> a month, but it winds up being more like 2-3 months. Still, much
> better than losing it all.

I also have a fireproof safe in the basement with room for a few hard
drives. I could do that as well. It's down on the concrete in an area
that would be last to get "hot" in a fire. So that plus 30 minutes
could be enough...

[Alan Browne]

I'd be concerned with the continuous vibration applied to the hard drives.

For CD/DVD's the humidity would be a concern, and perhaps heat as well
depending on local conditions.

[Alan Browne]

Hard disks are not a long term strategy so obsolescence is not a
particular issue.

Further, while obsolescence was an issue for past formats, it is less
and less an issue going forward if for sheer volume of machines out
there alone.

[tony cooper]

A lady from this area is in critical condition in the hospital. Her
house was on fire, she ran out safely, but ran back in to rescue a cat
and was overcome by smoke and passed out.

Completely understandable if she had run back in to rescue her
photograph archives on disks, but a cat? You can always get another
cat.


--
Tony Cooper - Orlando, Florida

[Robert Coe]

On Fri, 09 Mar 2012 19:43:42 -0500, Alan Browne
<alan....@FreelunchVideotron.ca> wrote:

: On 2012-03-08 21:53 , Robert Coe wrote:
: > On Thu, 08 Mar 2012 17:23:10 -0600, "Dallas"<Cybnorm@spam_me_not.Hotmail.Com>
: > wrote:
: > : Alan Browne wrote:
: > :
: > :> Further, I can upgrade any of the 4 drives at any time without
: > :> shutting down the unit. Just pull any drive out and slap in a new
: > :> one. The drive then rebuilds the redundant data. When the 4 lights
: > :> are green again, I can then add another drive... and so on.
: > :
: > : Salivating... that's a backup system.
: > :
: > : But yeah, you'd be wiped out in a fire. I do keep a third clone of the
: > : HD at the bank in a safe deposit box. In theory, I should swap it once
: > : a month, but it winds up being more like 2-3 months. Still, much
: > : better than losing it all.
: >
: > You get 95% of the benefit by just throwing your next-to-last backups, on
: > whatever media you have available, into the trunk of your car. I've never
: > understood why so many people find that riduculously simple procedure to be
: > beyond their reach. It's far from the most sophisticated backup method, but it
: > allows you to sleep nights if you have nothing better.
:
: I'd be concerned with the continuous vibration applied to the hard drives.

Don't the heads on a hard drive lock when you turn it off? In some early PCs
you were supposed to run a command to lock the heads if you were going to move
the computer. Then they came out with drives that locked the heads
automatically; it seemed a great step forward.

: For CD/DVD's the humidity would be a concern, and perhaps heat as well
: depending on local conditions.

You think? I'd be surprised if got so hot in Montréal that heat would be a
problem in the trunk. In the main part of the car, maybe.

In any case, there are better ways to handle backups, but any protection is
better than nothing. Yesterday a complex of four $1,000,000+ houses in a
Boston suburb burned to the ground in minutes. The wind was so high that there
was nothing the firefighters could do.

Bob

[Savageduck]

The lesson here is, always make sure you have a backup cat in a
fireproof safe, or off site.


--
Regards,

Savageduck

[Alan Browne]

Or keep the cat in a fireproof safe.

My SO would go for her cat, not a doubt.

[Alan Browne]

Certainly. But vibration has other effects on electromechanical devices
as well that could damage the unit over time. Most HD's are extremely
robust - but vibration and time...

>
> : For CD/DVD's the humidity would be a concern, and perhaps heat as well
> : depending on local conditions.
>
> You think? I'd be surprised if got so hot in Montréal that heat would be a
> problem in the trunk. In the main part of the car, maybe.

I wasn't being specific to anywhere.

There's humidity to consider as well.

> In any case, there are better ways to handle backups, but any protection is
> better than nothing. Yesterday a complex of four $1,000,000+ houses in a
> Boston suburb burned to the ground in minutes. The wind was so high that there
> was nothing the firefighters could do.

I'm not saying the car is a bad idea, it isn't. One should just
consider all factors.

A similar fire here (more like $25M+ total) happened a few years ago
when a roof fire started by the roofers spread very fast across several
buildings. Winds spread it. All but one of the buildings were still
under construction.

[Pete A]

Geographically diverse redundancy using three locations is, I believe,
considered to be a highly fault-tolerant system. Such triple-redundant
systems have an extremely low probability of failure provided that
there is no common point of failure which has been overlooked.

If one stores a HDD in a car then attempts to use it before giving it
several hours to stabilize to the new environment, it will be operating
well outside of its design limits (for goodness sake, it was probably
initially shipped in a cargo hold that went well below its minimum
storage specifications). HDDs are designed to be most reliable when
continuously powered in an environment of moderate (and slowly
changing) temperature and humidity.

The traditional CRC-16 sector checksum used on most mass storage
devices is a completely useless guarantee of data integrity. If it was
useful, CRC-16 would be used for integrity instead of MD5 (poor), SHA-1
(moderate), AES-256 (quite good). Even Zip files use the CRC-32 plus a
recording of the file size to detect data corruption, which is actually
very robust for reasonably low error-rate systems; unfortunately it has
no redundancy to recover a corrupted file, but that was not its
intended purpose - the recipient simply asks the provider to send the
file again.

CD and DVD media were designed to provide reasonably accurate data
retrieval to inherently fault-tolerant systems, not to provide
archive-quality mission-critical data integrity. Digital audio and
video are transmitted via media with an alarmingly high error rate: as
the error rate increases, the decoding algorithms try to provide
graceful degradation, which is tailored to human perception rather than
data integrity.

I would suggest to anyone using CDs, DVDs, HDDs, or other mass storage
devices for backup that they seriously consider using RAR (or an
equivalent) because it adds redundancy that seems to cope quite well
with recovery from errors on both mass storage devices and the Internet.

Also make sure you that have a computer with ECC memory and an OS that
_correctly_ responds to it. Without that combination, you may be
backing-up corrupted data. This is one of the common points of failure
that I referred to in my opening paragraph. The occasional hardware
glitch is not as uncommon as we would like to believe. Some calibrate
their monitors quite frequently to ensure colour fidelity, but do these
same people frequently run exhaustive computer hardware test software?
If the reader has a tendency to paranoia, I strongly recommend that you
never run these very long test programs.

[Dallas]

Savageduck wrote:

> The lesson here is, always make sure you have a backup cat in a
> fireproof safe, or off site.

Na.. no need.. cats automatically back themselves up. To retrieve the
backup, just put a bowl of food at the back door and you'll have a new
cat in no time.

--
Dallas

[Dallas]

Robert Coe wrote:

> You get 95% of the benefit by just throwing your next-to-last
> backups, on whatever media you have available, into the trunk of your
> car.

Until someone steals your car. :- )

But, seriously, I do have data on my drive that I wouldn't want to be
in a criminal's hands.

--
Dallas

[Savageduck]

Strange, I have had criminals in hand who I wouldn't want on my drive.

--
Regards,

Savageduck

[PeterN]

And I've met criminals who;
wouldn't want to be on your drive; and/or
wouldn't want you to see the data on their drive.

--
Peter

[(PeteCresswell)]

Per Dallas:

This thread is getting better and better... -)
--
Pete Cresswell

[Burt Johnson]

I wrote a blog entry on tha a while back for our photography club. The
answer is fairly long and involved, so here is a link to it, rather than
just repeating everything:


http://mindstormphoto.squarespace.com/blog/2011/12/6/computer-how-and-wh
y-to-backup-your-computers.html

hmmm.. that is pretty long. Here is a 'tinyUrl' version of the same
link:

http://tinyurl.com/6uaw8df

--
- Burt Johnson - MindStorm, Inc.
Gallery: http://www.mindstormphoto.com/
Blog: http://mindstormphoto.squarespace.com/
500px: http://500px.com/mindstormphoto

[(PeteCresswell)]

Per Burt Johnson:

>
>http://mindstormphoto.squarespace.com/blog/2011/12/6/computer-how-and-wh
>y-to-backup-your-computers.html
>
>hmmm.. that is pretty long. Here is a 'tinyUrl' version of the same
>link:
>
> http://tinyurl.com/6uaw8df

That made a lot of sense - and now I'm thinking maybe a second
NAS box in a neighbor's garage or closet... or even at one of the
sites I serve - where I have a PC anyhow...

But I can't figure out how CrashPlan differs from using a file
copy utility like SecondCopy that can copy to an FTP destination.

Can anybody elucidate?
--
Pete Cresswell

[Alan Browne]

You should write a book.

[Charles E. Hardwidge]

"Pete A" <pete3....@nospam.ntlworld.com> wrote in message
news:2012031018292941470-pete3attkins@nospamntlworldcom...

> Geographically diverse redundancy using three locations is, I believe,

> considered to be a highly fault-tolerant system. [...]

> [...] HDDs are designed to be most reliable when continuously powered in

> an environment of moderate (and slowly changing) temperature and humidity.
>

> I would suggest to anyone using CDs, DVDs, HDDs, or other mass storage
> devices for backup that they seriously consider using RAR (or an
> equivalent) because it adds redundancy that seems to cope quite well with
> recovery from errors on both mass storage devices and the Internet.

Really surprised this kind of thing isn't a FAQ as it's all a solved problem
but there you go. Might be worth tacking on VPN for site to site back ups
and full disc encryption.

I've found disc drive temperature tends to be around 10 degrees Celsius
above ambient temperature. Around 25C is good with reliability dropping
slowly off as you approach 40C. Anything approaching 50C is trouble.

An alternative to RAR archives is PAR recovery files developed for
transferring files over usenet. The advantage is PAR can work with original
files & directory structures.

--
Charles E. Hardwidge

[Charles E. Hardwidge]

"tony cooper" <tony.co...@gmail.com> wrote in message
news:8tell71tt5jpba3sm...@4ax.com...

> A lady from this area is in critical condition in the hospital. Her
> house was on fire, she ran out safely, but ran back in to rescue a cat
> and was overcome by smoke and passed out.
>
> Completely understandable if she had run back in to rescue her
> photograph archives on disks, but a cat? You can always get another
> cat.

Your narrative cropping is a little tight. Did she rescue the cat?

--
Charles E. Hardwidge

[Charles E. Hardwidge]

"(PeteCresswell)" <x...@y.Invalid> wrote in message
news:d2bpl7tjpmnf952tm...@4ax.com...

> That made a lot of sense - and now I'm thinking maybe a second
> NAS box in a neighbor's garage or closet... or even at one of the
> sites I serve - where I have a PC anyhow...

Definitely think this is the way to go for a lot of people and even
business.

--
Charles E. Hardwidge

[Burt Johnson]

First, CrashPlan does differential, automated backups. That is, you
back up 2TB of data (as I do) and then change 50GB of it (which often
happens on a daily basis for me). CP will upload that 50GB and I then
have a full backup again.

Second, CP is smart about disk copies. Do you have the same file in
more than one location? CP figures that out and only uploads one copy,
and a pointer to where it should go in those other places.

Perhaps most important, CrashPlan acts like a remote TimeMachine, giving
you historic copies as well as current ones. Need to recover a file as
it stood last December? No FTP copy will handle that, but CP will.

Of course, I have a TimeMachine local copy, and when I need to go back
in time, that is my first line of defense. However, if the TM disk does
not have what I need, or the house burned down / was burglarized, I fo
to the CrashPlan backup.

[Burt Johnson]

Joel <Jo...@NoSpam.com> wrote:

> A stolen hard drive? You can't seem to enjoy a normal life, can you?

If someone breaks into your house and steals your compter, you can
assume they will take every high tech device in the same room --
including your backup disk.

#1 - If you don't have an offsite backup, that means you are hosed and
have just lost everything.

#2 - The criminal can potentially get your bank info. Of course, if
they have the computer already, the extra disk copy is probably not the
weakest link...

#3 - But if they steal the disk (only) from a car, and the disk is not
strongly password protected, you probably have bank info, etc there.
Not sure I would want a criminal to have that info...

Personally I use CrashPlan, which prevents lots of these problems. See
my response to the orig post for more info there.

[Doug McDonald]

I back up new files such as photos or MP3s I get
immediately on two different brands of DVD-ROMs,
one Taiyo Yuden. I do the same to the DNA files I process
for people.

I back up (clone) entire hard disks twice a year and keep three
back ones, one in my office, one at home, and one in
a storage room at work. For this I use Ghost 11 on
a DVD-ROM; this makes bootable disks. I normally
store the old disk and put the clone in my computer ...
the Ghost process partially defragments the disk.

I have actually had one instance where the main disk died
and the first backup was partially unreadable so I
had to boot the second backup and restore files from
the CD-ROM backups (this happened before DVD-ROMS).

I'm paranoid about this, and for good reason ... I've seen
people who had major disasters.

Doug McDonald

[Wolfgang Weisselberg]

Pete A <pete3....@nospam.ntlworld.com> wrote:
> On 2012-03-09 14:37:48 +0000, Joel said:
>> "Dallas" <Cybnorm@spam_me_not.Hotmail.Com> wrote:

>>> I clone my whole drive and keep the drive off-site. Trouble is... at 5

[...]

> 5. Then they worry that the hard drive interface (SCSI, IDE etc.) will
> become obsolete.

You're mixing up archival and backup again.

Archival: yes, you might worry about that.
Backup: you really think the HDD-interface of your backups (which
should by definition be *recent*) will become obsolete within
days or *shudder* months?

-Wolfgang

[Wolfgang Weisselberg]

tcroyer <t...@solidus-ts.com> wrote:
> I have a Carbonite account to cover most of my system. But, since Carbonite
> doesn't back up external drives, once a week, I manually copy that (with
> most of my new photo work) to a removable, portable 1T drive.

Does Carbonite encrypt data on your system, or can Carbonite (and
everyone breaking in there, every court's subpoena, etc.) read
your data?

Additionally, Carbonite doesn't work with my OS.

-Wolfgang

[Wolfgang Weisselberg]

Pete A <pete3....@nospam.ntlworld.com> wrote:

> If one stores a HDD in a car then attempts to use it before giving it
> several hours to stabilize to the new environment, it will be operating
> well outside of its design limits (for goodness sake, it was probably
> initially shipped in a cargo hold that went well below its minimum
> storage specifications).

Something like outside -40 to +70°C?

> HDDs are designed to be most reliable when
> continuously powered in an environment of moderate (and slowly
> changing) temperature and humidity.

HDDs don't like shock when operating, condensing and other wetness
and tons of spinup-spindown actions, especially when they have
to use the rotational energy as power source to finish writing
a block and store the heads in a power-out emergency.

> The traditional CRC-16 sector checksum used on most mass storage
> devices is

used nowhere. HDDs of this millennium have several levels of ECC
data, otherwise they wouldn't be able to use such dense magnetic
patterns with any sort of reliability. Same with CDs, even more
so with CDs in data format, same with DVDs, same with BlueRay,
same with Flash memory of any kind, same with magnetic tape.
Even caches in CPUs have ECC. Only main RAM doesn't have ECC
usually (because that would increase the cost by 1/8th). In the
good olden times RAM each 8 byte hat a parity bit (which is more
than enough to make ECC out of them in larger blocks) ...

> a completely useless guarantee of data integrity.

1 out of 2^16 (65,536) is actually pretty good in detecting
data errors if done well.

> If it was
> useful, CRC-16 would be used for integrity instead of MD5 (poor), SHA-1
> (moderate), AES-256 (quite good).

These are used to detect malicious changes by intelligent, whily
opponents, not data corruption (which is random and doesn't try
to hide). Completely different story. Even going in that
direction shows you're merely dropping buzzwords without
understanding what you are saying.

> Even Zip files use the CRC-32 plus a
> recording of the file size to detect data corruption, which is actually
> very robust for reasonably low error-rate systems; unfortunately it has
> no redundancy to recover a corrupted file, but that was not its
> intended purpose - the recipient simply asks the provider to send the
> file again.

Actually, ZIP is a compression format.

All data channels you'd use for transmission of data are very low
error systems (by design, by checksums and resends, by enough
ECC), same as the storage systems you'd use. In practical use
only broken systems would be any other very low error rate systems.

Again you prove you don't see the woods, barking up the
(wrong) tree.

> CD and DVD media were designed to provide reasonably accurate data
> retrieval to inherently fault-tolerant systems, not to provide
> archive-quality mission-critical data integrity. Digital audio and
> video are transmitted via media with an alarmingly high error rate: as
> the error rate increases, the decoding algorithms try to provide
> graceful degradation, which is tailored to human perception rather than
> data integrity.

Again, CD and DVD were designed to be pressed and counter typical
usage damage (scratches etc.), for mass media use. The error
rate isn't alarmingly high: The output is completely correct
for almost everything. You should measure how often your HDD
does error correction, especially when it's a partial response
maximum likelyhood system!

Anyway (you got that partially right) a corrupted sector of 1/75th
second can be faded out without being noticeable in audio CDs.
With video data there's immediately visible artifacts.

> I would suggest to anyone using CDs, DVDs, HDDs, or other mass storage
> devices for backup that they seriously consider using RAR (or an
> equivalent) because it adds redundancy that seems to cope quite well
> with recovery from errors on both mass storage devices and the Internet.

Again a case of too little information and too much buzzwords.
For CDs and DVDs and BlueRays, the correct solution is dvdisaster,
which works more intelligent than PAR, since these media can
fail partially. (2 partially failed media kill your PAR, but
can usually be repaired by dvdisaster.)

For HDDs, partial failure is rare, so PARs saved to the same
HDD are useless. You either live with HDDs being failable and
the chance that both your computer and the backup HDDs failing
simultaneously being a really low chance, or you use RAID (or
similar data spreading) so you can stand losing n (n usually
being 1 or 2) drives out of every set of m backup disks.

> Also make sure you that have a computer with ECC memory and an OS that
> _correctly_ responds to it.

You need a BIOS that does scrubbing. The only OS part that's
"needed" if at all is alerting you to the fact if a certain
memory region begins failing. Actually memtest86+ does a
good job detecting that.

> Without that combination, you may be
> backing-up corrupted data.

It's much more likely that you fail because you never checked your
backup worked. It's much more likely you backed up corrupted data
because your hard drive/controller/cable/... corrupted the data.

> This is one of the common points of failure
> that I referred to in my opening paragraph.

'common' as in 'happens to 100 people in the world per
year'. Many more get murdered. Many more back up corrupted
data due to other reasons than RAM being bad.

99.99% of corrupted data being backed up is because the data
is *already* corrupted on your computer.

> The occasional hardware
> glitch is not as uncommon as we would like to believe.

I've seen it happen. UPS[1], several hard drives[2],
RAM[3][4], several graphics cards[5], ethernet port on
mainboard[6], probably soundcard on mainboard[7],
USB-miniUSB-cable[8], Ethernet cable[9] ...

> Some calibrate
> their monitors quite frequently to ensure colour fidelity, but do these
> same people frequently run exhaustive computer hardware test software?

Why should they? Most such errors are clearly visible:
- a corrupted JPEG caused by some glitch or persistent error is
clearly visibly broken
- a corrupted program crashes
- malfunctioning RAM causes clearly visible reliability problems
(like corrupted JPEG displays and program crashes)
- broken ethernet ports or cables cause no data transferred
- no sound is clearly audible
- freezing HDs or graphics cards etc. cause computer lock ups
and once you get such problems it's time to investigate.

Monitors are *known* to drift away from calibration and near
impossible to catch with the blank eye (even when using comparison
prints), *but* the drift causes, ah, imperfect results which can
be clearly noticable when it's too late and lots of work needs
to be redone.

Clearly, monitors need to be hardware checked. Just like
HDDs are (S.M.A.R.T.).

> If the reader has a tendency to paranoia, I strongly recommend that you
> never run these very long test programs.

If you suffer from paranoia, get treatment.


-Wolfgang

[1] testing it's worn-out batteries automatically and not being
able to switch back to main power in time to prevent a
computer crash.
[2] no data loss, though, though funny things like the firmware
crashing and freezing the computer (as it waited infinitely
for necessary pieces of data or code) after several weeks,
later days and finally refusing to start up every time. Yes,
I usually run my computers 24/7/365.25
[3] that was a known production error in a PDA (pre-flash
memory times): the refresh cycles were not frequent enough
or something. I got it repaired out of warranty for free.
[4] a pair of RAM sticks wouldn't run reliably at full bus speed
when both were inserted. Since that was a new computer anyway,
memtest86 (no '+' back then) caught the error reliably.
Replaced them a few times (for free, of course), until the
dealer gave up and I got 2 from the same production run.
Which worked fine for the lifetime of the computer.
[5] For unknown reasons started to overheat, more prominently
when asked to do 3D graphics. Directing additional air
at them helped, replacing them cured the computer lock ups
(partial, i.e. graphic subsystem only and full lockups, btw.)
The last one was this year.
[6] just went and died. Windows machine, i.e. some games only
and rarely running, usually without network. No idea when
it died. I had a USB-Ethernet converter for my XOs which for
now lives on that machine. Running Linux I could download
the Windows driver for that on said machine. (Linux had
everything on board.)
[7] Untested, but sound sometime back failed. Didn't add a
soundcard board to test the theory yet. Since it's for work
and photoediting sound's a low priority ...
[8] the ssh-via-USB connection to my ebook reader (openinkpot)
failed after several seconds, every time. Switching the USB
cable solved the problem. Would never have thought it could
be the USB cable, so it needed a hint to change it ...
[9] Pings were going out, but not coming in. Cable used to work
previously, it was probably experiencing shear forces at some
point which partially broke it.

[Wolfgang Weisselberg]

tony cooper <tony.co...@gmail.com> wrote:
> On Fri, 09 Mar 2012 19:41:29 -0500, Alan Browne

>>I also have a fireproof safe in the basement with room for a few hard
>>drives. I could do that as well. It's down on the concrete in an area
>>that would be last to get "hot" in a fire. So that plus 30 minutes
>>could be enough...

> A lady from this area is in critical condition in the hospital. Her
> house was on fire, she ran out safely, but ran back in to rescue a cat
> and was overcome by smoke and passed out.

> Completely understandable if she had run back in to rescue her
> photograph archives on disks, but a cat? You can always get another
> cat.

You can always get another photograph archive, just download
random pix via Google and Wikipedia. But each cat is different
from any other cat and cannot be replaced like any random photograp
archive can.

-Wolfgang

[tony cooper]

Follow-up: The lady died yesterday.

[David Dyer-Bennet]

Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

> Pete A <pete3....@nospam.ntlworld.com> wrote:

>> The traditional CRC-16 sector checksum used on most mass storage
>> devices is
>
> used nowhere. HDDs of this millennium have several levels of ECC
> data, otherwise they wouldn't be able to use such dense magnetic
> patterns with any sort of reliability. Same with CDs, even more
> so with CDs in data format, same with DVDs, same with BlueRay,
> same with Flash memory of any kind, same with magnetic tape.
> Even caches in CPUs have ECC. Only main RAM doesn't have ECC
> usually (because that would increase the cost by 1/8th). In the
> good olden times RAM each 8 byte hat a parity bit (which is more
> than enough to make ECC out of them in larger blocks) ...

I did invest in ECC memory for my file server. I figure I look at
things after editing them and writing them to disk, so ECC errors on my
editing system will get caught immediately, but I want higher
reliability out of my file server.

It's not just 1/8 more for extra bits, it also adds some time to the
basic memory cycle.

>> a completely useless guarantee of data integrity.
>
> 1 out of 2^16 (65,536) is actually pretty good in detecting
> data errors if done well.

Ctein once tracked down an intermittent data problem with one of his
disks to a cable problem. Going back and recomputing, basically the
bad cable was raising the undetected error rate from 1 in 10^14 all the
way to 1 in 10^13. And he noticed this, and was able to track it down.

This leaves me feeling 1 in 65536 is not very good, just off the top of
my head .
--
David Dyer-Bennet, dd...@dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

[Pete A]

You know me well enough by now, Wolfgang :-)

With that in mind, how about the recent reliability of cloud storage.
Its name conjures up the phrase "pie in the sky". I can pay to backup
my data somewhere in the ether and a "ruling", or even just a company
whim, can make my data inaccessible - iDisk being a soon to be
fulfilled example.

And what about the plethora of hardware that lies in its response to
fsync() or equivalent API call: the list of IDE and USB drives that ACK
a low-level flush command after they've written the data to their cache
instead of after it is successfully committed to permanent storage is
despicable.

And what about the fact that certain mass storage devices have been
shown to be not fully compatible with motherboard chipsets. I know this
one very well: it took both me and the computer supplier months of hard
work to figure out the source of data errors. Hey, MD5 may not be very
strong evidence, but it's strong enough to wallop hardware vendors into
admitting the truth when used enough times.

Of course I'm confused over the terms "archival" and "backup" - the
computer itself hasn't the faintest idea of what the software is
requesting it to do and the computer certainly doesn't give a rat's
arse if it's doing what the unsuspecting user is expecting it to do.

The humble fly is far more intelligence than your computer. So, next
time a fly lands on your face after it has be eating something gross,
use it as a lesson to learn how much intelligent creatures in the
hardware and software industry really care about your health,
well-being, data integrity, archival, and backup storage.

[Savageduck]

Sorry to hear that.

How is the cat doing?

--
Regards,

Savageduck

[Pete A]

Having partially composed myself from stitches of laughter: get some
software and a suitable drive to show you the massive number of C1 and
C2 errors from pressed CDs and laser recorded CDs. I have several
pressed music CDs where the C1 errors have caused so many C2 errors
that good CD players eventually revert to "STOP" mode and most players
either get stuck ("broken record") or keep skipping across the disc.
These CDs do what CDs always do - get worse as they get older.

Next test: record CDs at different recording speeds and notice how the
C1 errors massively increase towards the rated recording speed. You
will also notice that some CDs increase the C1 errors when the
recording speed is reduced below their optimal recording speed.

The cross-interleaved Reed-Solomon code (CIRC) encoding and decoding is
a masterpiece of engineering design. One of the Philips test CDs has a
sequence of "drilled holes" of various sizes across the disc. Sadly,
very few players pass the full set of tests. I say "sadly" because for
a reason unknown to me, it is legal to sell CD players and recorders
that fall a long way short of the original standards/specifications
even though the devices are stated by their manufacturer to conform to
a <Colour> Book Standard.

The CIRC is to combat surface and substrate regular pattern defects
(usually visibly identifiable) such as scratches, fingerprints, and
hairs. These cause bursts of errors. CIRC increases the "Hamming
distance" between adjacent errors caused by typical media defects
thereby reducing the amount of forward error correction required to
provide reasonable data recovery. CIRC does nothing to help correct
underlying problems with the substrate, which degrades with time.

It is well-known that Linux has various tools to read CDs and DVDs with
the lowest possible error-rate, but these algorithms are not used by
regular CD/DVD applications. Why? Because it can take many hours (even
days) to recover reasonably accurate data from CDs and DVDs. Yes, I
know data CDs and DVDs have a so-called data protection layer on top of
the raw disc data just as TCP has a so-called guarantee of accurate
data transmission. We all know that TCP delivers corrupted files far
more often than it should, which according to Wolfgang this should be
almost impossible.

The Shannon-Hartley theorem of data transmission cannot be overcome by
using any form of clever digital technology. The probability of having
unrecoverable errors is always greater than zero and always increases
towards maximum media read speed. If you think that by ticking a check
box in a typically application's options you are guaranteed accurate
data recording and retrieval then you do not understand the
implications of having inaccurate data on your computer system. Either
that or you employ enough lawyers to work around data corruptions as
they occur.

I shall not bother to address each of your points, Wolfgang, because
you have completely failed to understand why the existing CRC
mechanisms used for data integrity have proved themselves to archaic to
the point of being next to useless for any other purposes than cost
reduction and marketing hype.

[tony cooper]

The newspapers haven't covered that. No pet obit listings.

[Burt Johnson]

And I've seen LOTS of cases where the disk died "the day before I was
going to fully back it up!"...

That is why an automated, daily incremental backup is so important. I
have a local one made hourly (TimeMachine) and a remote one done daily
(CrashPlan).

No muss, no fuss, no forgetting, no extra work. Once it is set up, it
can be forgotten until needed. It is then also easy to recover just a
few files that were deleted 3 months ago when needed (happened to my
wife recently).

[tcroyer]

--
Tom Royer
If you're not free to fail, you're not free. -- Gene Burns
"Wolfgang Weisselberg" <ozcv...@sneakemail.com> wrote in message
news:9ah639-...@ID-52418.user.berlin.de...

> tcroyer <t...@solidus-ts.com> wrote:
>> I have a Carbonite account to cover most of my system. But, since
>> Carbonite
>> doesn't back up external drives, once a week, I manually copy that (with
>> most of my new photo work) to a removable, portable 1T drive.
>
> Does Carbonite encrypt data on your system, or can Carbonite (and
> everyone breaking in there, every court's subpoena, etc.) read
> your data?

From the Carbonite web site:

Bank-level privacy protection
To prevent unauthorized access to your files, we encrypt them with 128-bit
Blowfish encryption before they even leave your computer. Then they're
transmitted to our data centers using Secure Socket Layer (SSL) technology,
the same security technology used in online banking and e-commerce
transactions. Your files remain encrypted on our servers. We even pay a
professional security firm to test our intrusion defenses to make sure no
one else can access your files.

[Alan Browne]

On 2012-03-09 10:38 , tcroyer wrote:
<snip>

Don't top post.

[Wolfgang Weisselberg]

Pete A <pete3....@nospam.ntlworld.com> wrote:
> On 2012-03-14 12:50:55 +0000, Wolfgang Weisselberg said:
>> Pete A <pete3....@nospam.ntlworld.com> wrote:

[major snip, since Pete A does not answer lots of things
where he's clearly wrong]

>> Again, CD and DVD were designed to be pressed and counter typical
>> usage damage (scratches etc.), for mass media use. The error
>> rate isn't alarmingly high:

> Having partially composed myself from stitches of laughter: get some
> software and a suitable drive to show you the massive number of C1 and
> C2 errors from pressed CDs and laser recorded CDs.

"massive number"? Compared to what, the number of errors
that can be corrected?

You might also note that if the CD plays, the output of the
CD data is beween zero and not noticeable.

> I have several
> pressed music CDs where the C1 errors have caused so many C2 errors
> that good CD players eventually revert to "STOP" mode and most players
> either get stuck ("broken record") or keep skipping across the disc.
> These CDs do what CDs always do - get worse as they get older.

Old CD, right?
Yep, the early CDs literally rust internally and thus have a
rather short lifetime.

> Next test: record CDs at different recording speeds and notice how the
> C1 errors massively increase towards the rated recording speed. You
> will also notice that some CDs increase the C1 errors when the
> recording speed is reduced below their optimal recording speed.

How good a given recordable CD and a burner work together (measured
by C1 and C2 error rates) varies *a lot* from burner to burner
and from lot to lot.

> The cross-interleaved Reed-Solomon code (CIRC) encoding and decoding is
> a masterpiece of engineering design. One of the Philips test CDs has a
> sequence of "drilled holes" of various sizes across the disc. Sadly,
> very few players pass the full set of tests. I say "sadly" because for
> a reason unknown to me, it is legal to sell CD players and recorders
> that fall a long way short of the original standards/specifications
> even though the devices are stated by their manufacturer to conform to
> a <Colour> Book Standard.

Actually, it seems legal to sell music CDs that
- very creatively don't follow the Red Book Standard (and
thus play on any given player just by pure chance)
- install rootkit software (looking at you, Sony!)

> The CIRC is to combat surface and substrate regular pattern defects
> (usually visibly identifiable) such as scratches, fingerprints, and
> hairs. These cause bursts of errors. CIRC increases the "Hamming
> distance" between adjacent errors caused by typical media defects
> thereby reducing the amount of forward error correction required to
> provide reasonable data recovery. CIRC does nothing to help correct
> underlying problems with the substrate, which degrades with time.

Just as, say, heart medicine products are to combat various heart
illnesses and conditions .... but do nothing to help correct
the underlying problems with the tissue, which degrades with
time and will stop functioning within usually 100 years from
production.

Error correction helps to work with the inevitably existing
error rates, even when they increase over time. Proof: The
CD wouldn't even be playable without error correction.

> It is well-known that Linux has various tools to read CDs and DVDs with
> the lowest possible error-rate, but these algorithms are not used by
> regular CD/DVD applications. Why? Because it can take many hours (even
> days) to recover reasonably accurate data from CDs and DVDs. Yes, I
> know data CDs and DVDs have a so-called data protection layer on top of
> the raw disc data just as TCP has a so-called guarantee of accurate
> data transmission.

The various tools you mention often work around the limits of,
say the Red Book Standard (no proper sector adressing possible,
one long spiral (think 33,3 rpm record of olden days) instead of
tracks and sectors (think floppy disk, hard disk)). Worse, it's
not needed for most cases, it's only needed for damaged/'special'
disks, where the reader's ECC doesn't cancel the errors good
enough for the purpose used.

And TCP doesn't guarantee 100% error free transmission. Never was
designed for it. It only used 16 bit checksums (one for the TCP
header and data, one for the IP header). It's a virtual
circuit, not a virtual error free connection.

> We all know that TCP delivers corrupted files far
> more often than it should, which according to Wolfgang this should be
> almost impossible.

So do you have any statistics how often a TCP transmission
delivers wrong data, and have you compared that to the to be
expected error rates? No?

BTW, I haven't noticed "TCP" delivering corrupted files.
That may, of course, be because applications layering on TCP
may use their own error checking data.

> The Shannon-Hartley theorem of data transmission cannot be overcome by
> using any form of clever digital technology.

Actually, sending any kind of data below the channel capacity
can be done at arbitrary low error rates. You want an error
rate of 1 error in the lifetime of the universe? Sure. Can
be done.

> The probability of having
> unrecoverable errors is always greater than zero and always increases
> towards maximum media read speed.

I think you are wrong here. >0, sure, but arbitrary low (if
the engineers make that effort) and I don't think it's read
speed related. Or would you say the probability of unrecoverable
errors per bit read is larger on the outer rim of a HDD than on
the inner rim?

Additionally a well designed medium can have a maximum read
speed so that the signals at that speed are read best and are
worse at any slower speeds. (magnet patterns passing under a
reading head elict a stronger signal when they pass faster.
In fact, at speed 0 they don't elict a signal. So increasing
the speed *improves* the signal.)

> If you think that by ticking a check
> box in a typically application's options you are guaranteed accurate
> data recording and retrieval then you do not understand the
> implications of having inaccurate data on your computer system.

If you think that I'm a GUI constrained checkbox clicker you
are sadly mistaken.

Do you have looked at the undetectable error rate of your hard
drives (according to the manufacturer) recently? Of course
you have or will have inaccurate data on your computer
system. What will you do about it?

> Either
> that or you employ enough lawyers to work around data corruptions as
> they occur.

Lawyers restore undetectabble errors? That's a new one.

> I shall not bother to address each of your points, Wolfgang, because
> you have completely failed to understand why the existing CRC
> mechanisms used for data integrity have proved themselves to archaic to
> the point of being next to useless for any other purposes than cost
> reduction and marketing hype.

You are the one to use *communication channel bandwidth* theorems
on *storage media* where the channel isn't the bottleneck of
reliability or speed.

OK, what's your payable, scaleable suggestion for the
"existing CRC mechanisms" in, say, hard disks (where they
don't even exist in the form you claim)?

>> The output is completely correct
>> for almost everything. You should measure how often your HDD
>> does error correction, especially when it's a partial response
>> maximum likelyhood system!

[major snip, since Pete A does not answer lots of things
where he's clearly wrong]

-Wolfgang

[Dallas]

Burt Johnson wrote:

> However, if the TM disk does not have what I need, or the house

burned down / was burglarized, I go to the CrashPlan backup.

But does the CrashPlan make a bootable clone of the hard disk?

If the answer is 'no' then recovering from a burned down house would
look like this:

1) Buy new computer with OS installed.
2) Purchase all my software again because install CDs were lost in the
fire.. i.e. MS Office, Photoshop, Outlook, etc.
3) Install all the Applications, connect to your ISP ect.
4) Recover your data from CrashPlan and restore it to your new computer
in correct locations.

If you have an off site clone of a hard drive, you'd just install it in
a new box. Now, I have no illusions about trying to boot that hard
drive in a new computer... in theory you'd boot into safe mode and
begin the process of installing new drivers. But, completing that,
you'd have all your programs and data ready to go.

--
Dallas

[Burt Johnson]

And you would lose all the data from the last time you made the clone.
You certainly won't be making a clone daily, or taking your backup
offsite daily.

Go ahead and make your clone if you really want, but make sure you also
have an automated daily offsite backup, so the work you are doing today
is safe, not just what you did 6 months ago.

The 'burned down house' scenario is the extreme end, and has a low
probability of being needed. Not zero, and we have had major regional
fires burn within a few doors of our houses on two occasions (two
different houses for that matter).

if that really does happen, rebuilding the computer would be one of the
minor issue to address, and I would rather know that I lost NO data at
all, rather than maybe saving a few hours of recovery, but not having
anything I worked on for the past xx months.

[Mark F]

On Tue, 20 Mar 2012 19:43:49 -0500, "Dallas"
<Cybnorm@spam_me_not.Hotmail.Com> wrote:

> Burt Johnson wrote:
>
> > However, if the TM disk does not have what I need, or the house
> burned down / was burglarized, I go to the CrashPlan backup.
>
> But does the CrashPlan make a bootable clone of the hard disk?
>
> If the answer is 'no' then recovering from a burned down house would
> look like this:
>

Beforehand:
1. Use your favorite stand-alone utility to make a single file image
of your system disk to a file on a drive.
2. Use your favorite archiving program (WINRAR?) to split the
single file into multiple files and put them on a new recovery
drive. (Backup and restore of a 40GB file over the web with
throttled backups isn't fun.)
3. Repeat 1 and 2 so you wind up with a single disk can be used
to make clones for all of your system disks.
4. stick ISO's of all of your software on the recovery drive as
you get the software. (You won't use these for disaster recovery
but it will help when you move to a new operating system
or get a new computer.)
5. Connect the drive with the split images to one system for backup
to the cloud. (Clone this drive or the one with the unsplit
image backups and keep copies at your friends or in your safe
deposit box.)
NOTE: Encrypt or use self encrypting disks for the backups as you
desire.

> 1) Buy new computer with OS installed.

Only need one for all of your computers, just to un WINRAR the
split-image files.

> 2) Purchase all my software again because install CDs were lost in the
> fire.. i.e. MS Office, Photoshop, Outlook, etc.

All ISO's of all of these were kept in my step 4, but you don't
use them for recovery.

> 3) Install all the Applications, connect to your ISP ect.
> 4) Recover your data from CrashPlan and restore it to your new computer
> in correct locations.
>
> If you have an off site clone of a hard drive, you'd just install it in
> a new box.

Correct: my additional steps show a practical way of getting
files of a practical size to backup to the cloud so you can recover
from your bank also being under water.

> Now, I have no illusions about trying to boot that hard
> drive in a new computer... in theory you'd boot into safe mode and
> begin the process of installing new drivers. But, completing that,
> you'd have all your programs and data ready to go.

In addition, everyone probably has many programs that were only
available for installation from the web with no way to save the
installation files. Many of these are no longer available, have to
be installed in the same order to get things to work the same way,
etc. Basically, no hope of installing any system from scratch and
having it work the same way. (Some people install from scratch
regularly and learn to deal with the most common things that mess up
or can't be installed on an operating system that can be installed
on currently available computers.)

[David Dyer-Bennet]

"Dallas" <Cybnorm@spam_me_not.Hotmail.Com> writes:

> Burt Johnson wrote:
>
>> However, if the TM disk does not have what I need, or the house
> burned down / was burglarized, I go to the CrashPlan backup.
>
> But does the CrashPlan make a bootable clone of the hard disk?
>
> If the answer is 'no' then recovering from a burned down house would
> look like this:
>
> 1) Buy new computer with OS installed.
> 2) Purchase all my software again because install CDs were lost in the
> fire.. i.e. MS Office, Photoshop, Outlook, etc.
> 3) Install all the Applications, connect to your ISP ect.
> 4) Recover your data from CrashPlan and restore it to your new computer
> in correct locations.

Buy just the OS; you'll install the rest of the applications from your
archived installation files after you've restored your data. (Most of
the applications I got on CD have upgraded over the net a few times by
now, so I've got files in store. And I create files documenting serial
numbers that live in the same directory.)

> If you have an off site clone of a hard drive, you'd just install it in
> a new box. Now, I have no illusions about trying to boot that hard
> drive in a new computer... in theory you'd boot into safe mode and
> begin the process of installing new drivers. But, completing that,
> you'd have all your programs and data ready to go.

All my data files live on my fileserver. The backups of that are
directly readable drives (for any Solaris server; they're ZFS
filesystems). I can get to them without building a new server, simply
by booting any x86 computer from a Solaris LiveCD.

[Wolfgang Weisselberg]

Pete A <pete3....@nospam.ntlworld.com> wrote:
> On 2012-03-14 12:55:11 +0000, Wolfgang Weisselberg said:

>> Pete A <pete3....@nospam.ntlworld.com> wrote:
>>> On 2012-03-09 14:37:48 +0000, Joel said:
>>>> "Dallas" <Cybnorm@spam_me_not.Hotmail.Com> wrote:

>>>>> I clone my whole drive and keep the drive off-site. Trouble is... at 5
>> [...]

>>> 5. Then they worry that the hard drive interface (SCSI, IDE etc.) will
>>> become obsolete.

>> You're mixing up archival and backup again.

>> Archival: yes, you might worry about that.
>> Backup: you really think the HDD-interface of your backups (which
>> should by definition be *recent*) will become obsolete within
>> days or *shudder* months?

> You know me well enough by now, Wolfgang :-)

> With that in mind, how about the recent reliability of cloud storage.
> Its name conjures up the phrase "pie in the sky". I can pay to backup
> my data somewhere in the ether and a "ruling", or even just a company
> whim, can make my data inaccessible - iDisk being a soon to be
> fulfilled example.

Deoending on the company that you trust your data with, the
major problem will not be that data is lost, but that the
company or product folds. But any other backup system has the
same problem, it's not 100%. HDDs die, tapes break, CDs, DVDs,
BlueRays deteriorate, break, become unreadable, ...

So don't use one cloud based system as the sole storage for
important data.

> And what about the plethora of hardware that lies in its response to
> fsync() or equivalent API call: the list of IDE and USB drives that ACK
> a low-level flush command after they've written the data to their cache
> instead of after it is successfully committed to permanent storage is
> despicable.

True. Try not to buy them.

> And what about the fact that certain mass storage devices have been
> shown to be not fully compatible with motherboard chipsets. I know this
> one very well: it took both me and the computer supplier months of hard
> work to figure out the source of data errors. Hey, MD5 may not be very
> strong evidence, but it's strong enough to wallop hardware vendors into
> admitting the truth when used enough times.

Well, one of the two isn't compatible enough to the standards.
And MD5 is very strong evidence: a different MD5 does mean
different input data.

Weaker it may be in cryptographic terms. But that means stuff like
replacing the original with a fake that has the identical MD5 ...

> Of course I'm confused over the terms "archival" and "backup"

Why?

> - the
> computer itself hasn't the faintest idea of what the software is
> requesting it to do and the computer certainly doesn't give a rat's
> arse if it's doing what the unsuspecting user is expecting it to do.

Your body also has not the faintest idea what your brain is
trying to tell it, and doesn't give a rat's ass either.

> The humble fly is far more intelligence than your computer.

That completely depends on what software you're using.
There's a lot of rather intelligent programs that do tons of
things a fly couldn't even comprehend. There are learning
programs (e.g. learning to classify between spam and
non-spam), and programs that learn what you like to see on
TV and act upon it ...

> So, next
> time a fly lands on your face after it has be eating something gross,
> use it as a lesson to learn how much intelligent creatures in the
> hardware and software industry really care about your health,
> well-being, data integrity, archival, and backup storage.

Non-sequitur: Completely unconnected to each other.

-Wolfgang

[Wolfgang Weisselberg]

Dallas <Cybnorm@spam_me_not.Hotmail.Com> wrote:
> Burt Johnson wrote:

>> However, if the TM disk does not have what I need, or the house
>> burned down / was burglarized, I go to the CrashPlan backup.

> But does the CrashPlan make a bootable clone of the hard disk?

Not necessary.

> If the answer is 'no' then recovering from a burned down house would
> look like this:

> 1) Buy new computer with OS installed.

You just lost *all* your hardware, including the bootable clone
copy (if you have one). Without a computer with OS you won't be
able to access computer data. After all, you need to access your
CrashPlan account in some way.

> 2) Purchase all my software again because install CDs were lost in the
> fire.. i.e. MS Office, Photoshop, Outlook, etc.

Aren't they bound to your hardware? In which case they won't
work with your new computer anyway ... whether your house burned
down or you just upgraded your computer.

Other software, you just copy the files and settings to the right
place (and that's where the Windows registry sucks 10s of miles
sized asteroids through thin straws).

> 3) Install all the Applications, connect to your ISP ect.

Connecting to your ISP is step 1.3, downloading your data is
1.6 and then you do have all your applications.

> 4) Recover your data from CrashPlan and restore it to your new computer
> in correct locations.

> If you have an off site clone of a hard drive,

you have lost your access to it. The key to the bank vault
is lost.

> you'd just install it in
> a new box.

Wrong.
1) Buy a new computer without an OS and without a harddrive.
(have fun finding one)
2) fetch the clone drive.
3) Install the clone drive into the computer. Replace the
mainboard, since the hard drive doesn't connect to the mainboard
any more.
4) Try to boot the computer.
5) Hope you have all the drivers for everything with your computer.
Install them.
6) Connect to the internet. No problem, but ... your whole system
is months or even years out of date! Including the AV software.
It takes longer just to update a new Windows than the usual
timespan between attacks on the holes it patches on a given IP,
so ...
7) ... try to update all your software faster than an attack
gets you.
8) Try to make sure your computer has not been attacked yet.
9) buy all the stuff you bought again since you made the last
clone and install it.
10) Cry for all the rest you didn't backup since then ...

> Now, I have no illusions about trying to boot that hard
> drive in a new computer... in theory you'd boot into safe mode and
> begin the process of installing new drivers. But, completing that,
> you'd have all your programs and data ready to go.

The good thing about theory and praxis is that in theory they
both behave the same. In praxis ... well, that's different.

-Wolfgang

[David Dyer-Bennet]

Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

> Dallas <Cybnorm@spam_me_not.Hotmail.Com> wrote:

>> 2) Purchase all my software again because install CDs were lost in the
>> fire.. i.e. MS Office, Photoshop, Outlook, etc.
>
> Aren't they bound to your hardware? In which case they won't
> work with your new computer anyway ... whether your house burned
> down or you just upgraded your computer.

No, with the possible exception of Windows itself (depends on what kind
of license you have; the default that comes with big-name hardware is
rather limited, but I've rarely had that license). But everything else
you can move to the new computer (I've moved Photoshop, Bibble Pro,
Photo Mechanic, Thumbs Plus, various plugins, Picture Window Pro, and
oodles of other stuff across a number of computers).

> Other software, you just copy the files and settings to the right
> place (and that's where the Windows registry sucks 10s of miles
> sized asteroids through thin straws).

Funny, because they invented the registry to make problems from people
moving stuff around smaller.

>> 3) Install all the Applications, connect to your ISP ect.
>
> Connecting to your ISP is step 1.3, downloading your data is
> 1.6 and then you do have all your applications.
>
>> 4) Recover your data from CrashPlan and restore it to your new computer
>> in correct locations.
>
>> If you have an off site clone of a hard drive,
>
> you have lost your access to it. The key to the bank vault
> is lost.

You've probably got your ID (it lives in my wallet in my pocket, so if I
survive my ID probably does too), you can walk into the bank and get
them to being a locksmith to let you into your vault.

I think you get two copies of the key, anyway. One can live at home,
one in your car, or something. Or in a drawer at work.

>> you'd just install it in
>> a new box.
>
> Wrong.
> 1) Buy a new computer without an OS and without a harddrive.
> (have fun finding one)

Trivially easy, I've bought that way frequently.

> 2) fetch the clone drive.
> 3) Install the clone drive into the computer. Replace the
> mainboard, since the hard drive doesn't connect to the mainboard
> any more.

No, install the proper drivers for the new mainboard. It may well do
this itself (Windows generally does).

> 4) Try to boot the computer.
> 5) Hope you have all the drivers for everything with your computer.
> Install them.
> 6) Connect to the internet. No problem, but ... your whole system
> is months or even years out of date! Including the AV software.
> It takes longer just to update a new Windows than the usual
> timespan between attacks on the holes it patches on a given IP,
> so ...
> 7) ... try to update all your software faster than an attack
> gets you.

I've never had problems with this when configuring new computers or
doing motherboard replacements. The key is, your computer shouldn't
ever be *directly exposed* on the internet, where other things can
connect to it (unless it's a hardened server system). But mostly at
home you're behind NAT, in which case outside systems can't reach yours.

So then, just avoid browsing anywhere except Windows Update until you're
updated.

Or if you're badly paranoid, download them on another computer onto a
DVD.

> 8) Try to make sure your computer has not been attacked yet.
> 9) buy all the stuff you bought again since you made the last
> clone and install it.



> 10) Cry for all the rest you didn't backup since then ...

Which is why automatic backups are good, yes. And why automatic
off-site is especially good.

Still, I brought 16GB of data back from a photo trip to the zoo
yesterday, and that takes a while to backup via my outbound Internet
bandwidth.

>> Now, I have no illusions about trying to boot that hard
>> drive in a new computer... in theory you'd boot into safe mode and
>> begin the process of installing new drivers. But, completing that,
>> you'd have all your programs and data ready to go.
>
> The good thing about theory and praxis is that in theory they
> both behave the same. In praxis ... well, that's different.

I've had consistently good results with moterhboard upgrades, though,
which is what this is (the hard part).

[David Dyer-Bennet]

Never have *just one* backup copy, is the rule. This rule is
medium-agnostic.

And remember that if you lose your backup -- you haven't lost your data
yet.

>> And what about the plethora of hardware that lies in its response to
>> fsync() or equivalent API call: the list of IDE and USB drives that ACK
>> a low-level flush command after they've written the data to their cache
>> instead of after it is successfully committed to permanent storage is
>> despicable.
>
> True. Try not to buy them.

Can be a nasty problem, especially because you may not notice until much
later. I dunno, can fraud suits be brought against them? It seems to
me they're not actually implementing the standards they claim to be
implementing.

[Pete A]

Fraud? This is my interpretation of EULAs: I cannot install the
OS/hardwar driver/software/service unless I actively agree to what in
essence says "You agree that you are installing a bug-ridden consumer
toy that will occasionally fail and destroy your data. This product is
for entertainment/amusement purposes only therefore the vendor cannot
be held responsible for any data loss and/or leakage of personal
information to third parties." Have I missed something?

[David Dyer-Bennet]

You've missed that such contract of adhesion are specifically forbidden
in many states.

[Pete A]

Obviously not in the states where it is of utmost importance:

<http://www.bbc.co.uk/news/technology-17486847>

I don't see any mention of suing the software/harware vendors.

[Mark F]

On Mon, 26 Mar 2012 21:38:50 +0100, Pete A

This product can be completely removed at our option without prior
notice. We can remove your ability to access your data from the
cloud, which was the only reason that you installed the product in the
first place, but we retain the right to keep access to the data for
ourselves.

[Pete A]

On 2012-03-18 13:58:16 +0000, Wolfgang Weisselberg said:

> Pete A <pete3....@nospam.ntlworld.com> wrote:
>> On 2012-03-14 12:50:55 +0000, Wolfgang Weisselberg said:
>>> Pete A <pete3....@nospam.ntlworld.com> wrote:
>
> [major snip, since Pete A does not answer lots of things
> where he's clearly wrong]
>
>>> Again, CD and DVD were designed to be pressed and counter typical
>>> usage damage (scratches etc.), for mass media use. The error
>>> rate isn't alarmingly high:
>
>> Having partially composed myself from stitches of laughter: get some
>> software and a suitable drive to show you the massive number of C1 and
>> C2 errors from pressed CDs and laser recorded CDs.

Sorry for the delay in replying.

> "massive number"? Compared to what, the number of errors
> that can be corrected?
>
> You might also note that if the CD plays, the output of the
> CD data is beween zero and not noticeable.

<http://www.cdmasteringservices.com/digitalerrors.htm>

I was talking about the many pressed and user-recorded CDs that
massively exceed 220 C1 errors/s (by massively, I mean many thousands
of C1 errors/s) and bursts of C2 errors that are uncorrectable leading
to clearly audible distortion.

>> I have several
>> pressed music CDs where the C1 errors have caused so many C2 errors
>> that good CD players eventually revert to "STOP" mode and most players
>> either get stuck ("broken record") or keep skipping across the disc.
>> These CDs do what CDs always do - get worse as they get older.
>
> Old CD, right?
> Yep, the early CDs literally rust internally and thus have a
> rather short lifetime.

Those pressed CDs caused the problem from the first day I bought them,
which was within a month of each one being released. All I was aware of
at the time was considerable audio distortion. I initially assumed it
was caused by terrible recording quality. One was so bad I took it back
to the shop: none of the players in the shop could play it at all.

The audio quality of the CDs have indeed got worse over time even
though the quality of the players has improved so I guess the discs
have deteriorated with time.

>> Next test: record CDs at different recording speeds and notice how the
>> C1 errors massively increase towards the rated recording speed. You
>> will also notice that some CDs increase the C1 errors when the
>> recording speed is reduced below their optimal recording speed.
>
> How good a given recordable CD and a burner work together (measured
> by C1 and C2 error rates) varies *a lot* from burner to burner
> and from lot to lot.

Very true.

>> The cross-interleaved Reed-Solomon code (CIRC) encoding and decoding is
>> a masterpiece of engineering design. One of the Philips test CDs has a
>> sequence of "drilled holes" of various sizes across the disc. Sadly,
>> very few players pass the full set of tests. I say "sadly" because for
>> a reason unknown to me, it is legal to sell CD players and recorders
>> that fall a long way short of the original standards/specifications
>> even though the devices are stated by their manufacturer to conform to
>> a <Colour> Book Standard.
>
> Actually, it seems legal to sell music CDs that
> - very creatively don't follow the Red Book Standard (and
> thus play on any given player just by pure chance)

Yep (I like your wording of that).

> - install rootkit software (looking at you, Sony!)

Luckily, I avoided installing a rootkit.

>> The CIRC is to combat surface and substrate regular pattern defects
>> (usually visibly identifiable) such as scratches, fingerprints, and
>> hairs. These cause bursts of errors. CIRC increases the "Hamming
>> distance" between adjacent errors caused by typical media defects
>> thereby reducing the amount of forward error correction required to
>> provide reasonable data recovery. CIRC does nothing to help correct
>> underlying problems with the substrate, which degrades with time.
>
> Just as, say, heart medicine products are to combat various heart
> illnesses and conditions .... but do nothing to help correct
> the underlying problems with the tissue, which degrades with
> time and will stop functioning within usually 100 years from
> production.
>
> Error correction helps to work with the inevitably existing
> error rates, even when they increase over time. Proof: The
> CD wouldn't even be playable without error correction.

It would be much less useful than a length of string and two tin cans.

>> It is well-known that Linux has various tools to read CDs and DVDs with
>> the lowest possible error-rate, but these algorithms are not used by
>> regular CD/DVD applications. Why? Because it can take many hours (even
>> days) to recover reasonably accurate data from CDs and DVDs. Yes, I
>> know data CDs and DVDs have a so-called data protection layer on top of
>> the raw disc data just as TCP has a so-called guarantee of accurate
>> data transmission.
>
> The various tools you mention often work around the limits of,
> say the Red Book Standard (no proper sector adressing possible,
> one long spiral (think 33,3 rpm record of olden days) instead of
> tracks and sectors (think floppy disk, hard disk)). Worse, it's
> not needed for most cases, it's only needed for damaged/'special'
> disks, where the reader's ECC doesn't cancel the errors good
> enough for the purpose used.

True.

> And TCP doesn't guarantee 100% error free transmission. Never was
> designed for it. It only used 16 bit checksums (one for the TCP
> header and data, one for the IP header). It's a virtual
> circuit, not a virtual error free connection.

That's an interesting point. My ISP offers free storage space if I
install the software, which just seems to be a fancy wrapper to an FTP
client. AFAIK it gives no more reliable data transmission than TCP
itself.

>> We all know that TCP delivers corrupted files far
>> more often than it should, which according to Wolfgang this should be
>> almost impossible.
>
> So do you have any statistics how often a TCP transmission
> delivers wrong data, and have you compared that to the to be
> expected error rates? No?

Guilty as charged! Most applications don't/cannot report TCP errors so
I have no data. Am I alone in finding that an application such as RAR
needs to use its redundancy files far more frequently than one would be
comfortable with?

> BTW, I haven't noticed "TCP" delivering corrupted files.
> That may, of course, be because applications layering on TCP
> may use their own error checking data.

As above, over the years I've experienced far too many data errors over FTP.

>> The Shannon-Hartley theorem of data transmission cannot be overcome by
>> using any form of clever digital technology.
>
> Actually, sending any kind of data below the channel capacity
> can be done at arbitrary low error rates. You want an error
> rate of 1 error in the lifetime of the universe? Sure. Can
> be done.

Yep.

>> The probability of having
>> unrecoverable errors is always greater than zero and always increases
>> towards maximum media read speed.
>
> I think you are wrong here. >0, sure, but arbitrary low (if
> the engineers make that effort) and I don't think it's read
> speed related. Or would you say the probability of unrecoverable
> errors per bit read is larger on the outer rim of a HDD than on
> the inner rim?

I used to know a fair bit about this, but I've forgotten most of it. My
brain error rate is rapidly increasing at it gets towards the end :-(

CD/DVD has the luxury of changing RPM to account for this problem. HDD
manufacturers have gone a long way to overcome it, but read/write speed
still deteriorates from beginning to end of the disc.

> Additionally a well designed medium can have a maximum read
> speed so that the signals at that speed are read best and are
> worse at any slower speeds. (magnet patterns passing under a
> reading head elict a stronger signal when they pass faster.
> In fact, at speed 0 they don't elict a signal. So increasing
> the speed *improves* the signal.)

Yep, but there will always be an optimal speed range for minimum error rate.

>> If you think that by ticking a check
>> box in a typically application's options you are guaranteed accurate
>> data recording and retrieval then you do not understand the
>> implications of having inaccurate data on your computer system.
>
> If you think that I'm a GUI constrained checkbox clicker you
> are sadly mistaken.

I don't think that for one moment. I should've said "people" instead of "you".

> Do you have looked at the undetectable error rate of your hard
> drives (according to the manufacturer) recently? Of course
> you have or will have inaccurate data on your computer
> system. What will you do about it?

I've chosen to not let it bother me any more. If I personally lose
"important" data I'll get over it and generate some more after the
event.

>> Either
>> that or you employ enough lawyers to work around data corruptions as
>> they occur.
>
> Lawyers restore undetectabble errors? That's a new one.

My mind often wanderers off into hyperspace instead of staying focused.

>> I shall not bother to address each of your points, Wolfgang, because
>> you have completely failed to understand why the existing CRC
>> mechanisms used for data integrity have proved themselves to archaic to
>> the point of being next to useless for any other purposes than cost
>> reduction and marketing hype.
>
> You are the one to use *communication channel bandwidth* theorems
> on *storage media* where the channel isn't the bottleneck of
> reliability or speed.
>
> OK, what's your payable, scaleable suggestion for the
> "existing CRC mechanisms" in, say, hard disks (where they
> don't even exist in the form you claim)?

As above.

>>> The output is completely correct
>>> for almost everything. You should measure how often your HDD
>>> does error correction, especially when it's a partial response
>>> maximum likelyhood system!
>
> [major snip, since Pete A does not answer lots of things
> where he's clearly wrong]

Thank you for your detailed reply, Wolfgang.

I learn from being wrong and absolutely nothing from being right. I
apologize for directing my animosity towards you. Rather than
attempting to explain why I do this sometimes, I feel it would be far
better for me to address my problem rather than to make excuses for it.

Pete.

[Wolfgang Weisselberg]

David Dyer-Bennet <dd...@dd-b.net> wrote:
> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:
>> Dallas <Cybnorm@spam_me_not.Hotmail.Com> wrote:

>>> 2) Purchase all my software again because install CDs were lost in the
>>> fire.. i.e. MS Office, Photoshop, Outlook, etc.

>> Aren't they bound to your hardware? In which case they won't
>> work with your new computer anyway ... whether your house burned
>> down or you just upgraded your computer.

> No, with the possible exception of Windows itself (depends on what kind
> of license you have; the default that comes with big-name hardware is
> rather limited, but I've rarely had that license).

OK, then it's more usual with games than with productive
software.

> But everything else
> you can move to the new computer (I've moved Photoshop, Bibble Pro,
> Photo Mechanic, Thumbs Plus, various plugins, Picture Window Pro, and
> oodles of other stuff across a number of computers).

So you don't need a disk image or install DVD for them?

>> Other software, you just copy the files and settings to the right
>> place (and that's where the Windows registry sucks 10s of miles
>> sized asteroids through thin straws).

> Funny, because they invented the registry to make problems from people
> moving stuff around smaller.

There are better systems, for example "keep everything in the
program's directory" system. Or the "/etc/$PROGRAM" model,
which with "$PROGRAM.d" files allows even secondary programs to
add configuration to the primary program, without having to be
clever with the original configuration.[1]

Additionally, both models are trivially seen and changed with
your usually file system tools. With clear text config, it's
easy to see what they're doing and there's ample space for long
comments on the options and commented out options, too, so the
file is self-documenting and easy to change with any text editor.


Contrast and compare the registry, where you need special tools,
litte if any contained documentation and where real life has
proven cleaning up the registry is anything but trivial.

>>> 3) Install all the Applications, connect to your ISP ect.

>> Connecting to your ISP is step 1.3, downloading your data is
>> 1.6 and then you do have all your applications.

>>> 4) Recover your data from CrashPlan and restore it to your new computer
>>> in correct locations.

>>> If you have an off site clone of a hard drive,

>> you have lost your access to it. The key to the bank vault
>> is lost.

> You've probably got your ID (it lives in my wallet in my pocket, so if I
> survive my ID probably does too), you can walk into the bank and get
> them to being a locksmith to let you into your vault.

> I think you get two copies of the key, anyway. One can live at home,
> one in your car, or something. Or in a drawer at work.

Large fire (due to earthquake). The local bank's vault
burned/was flooded/crushed by rubble. :-)

>>> you'd just install it in
>>> a new box.

>> Wrong.
>> 1) Buy a new computer without an OS and without a harddrive.
>> (have fun finding one)

> Trivially easy, I've bought that way frequently.

Gateway? Dell?

No, you have to shop for that, talk to the local small
computer store or know your sources.

>> 2) fetch the clone drive.
>> 3) Install the clone drive into the computer. Replace the
>> mainboard, since the hard drive doesn't connect to the mainboard
>> any more.

> No, install the proper drivers for the new mainboard. It may well do
> this itself (Windows generally does).

I doubt a proper driver can install a physical PATA connector.

>> 4) Try to boot the computer.
>> 5) Hope you have all the drivers for everything with your computer.
>> Install them.
>> 6) Connect to the internet. No problem, but ... your whole system
>> is months or even years out of date! Including the AV software.
>> It takes longer just to update a new Windows than the usual
>> timespan between attacks on the holes it patches on a given IP,
>> so ...
>> 7) ... try to update all your software faster than an attack
>> gets you.

> I've never had problems with this when configuring new computers or
> doing motherboard replacements. The key is, your computer shouldn't
> ever be *directly exposed* on the internet, where other things can
> connect to it (unless it's a hardened server system). But mostly at
> home you're behind NAT, in which case outside systems can't reach yours.

Well, yes, Windows should never be exposed to the open
internet. That was one of the points.

> So then, just avoid browsing anywhere except Windows Update until you're
> updated.

Is there a digital signature, and is it checked automatically?

> Or if you're badly paranoid, download them on another computer onto a
> DVD.

If I'm paranoid, I won't use Windows. :-)

>> 8) Try to make sure your computer has not been attacked yet.
>> 9) buy all the stuff you bought again since you made the last
>> clone and install it.

>> 10) Cry for all the rest you didn't backup since then ...

> Which is why automatic backups are good, yes. And why automatic
> off-site is especially good.

Yep. Very, very true.

> Still, I brought 16GB of data back from a photo trip to the zoo
> yesterday, and that takes a while to backup via my outbound Internet
> bandwidth.

And that's why local backups are also valuable.

>>> Now, I have no illusions about trying to boot that hard
>>> drive in a new computer... in theory you'd boot into safe mode and
>>> begin the process of installing new drivers. But, completing that,
>>> you'd have all your programs and data ready to go.

>> The good thing about theory and praxis is that in theory they
>> both behave the same. In praxis ... well, that's different.

> I've had consistently good results with moterhboard upgrades, though,
> which is what this is (the hard part).

So have I, but then I don't use Windows outside the games box,
preferring a real OS.

-Wolfgang

[1] A program might add itself to the task scheduler, add
configuration to the event monitor checker and so on.

[Wolfgang Weisselberg]

Pete A <pete3....@nospam.ntlworld.com> wrote:
> On 2012-03-18 13:58:16 +0000, Wolfgang Weisselberg said:
>> Pete A <pete3....@nospam.ntlworld.com> wrote:
>>> On 2012-03-14 12:50:55 +0000, Wolfgang Weisselberg said:
>>>> Pete A <pete3....@nospam.ntlworld.com> wrote:

>> [major snip, since Pete A does not answer lots of things
>> where he's clearly wrong]

>>>> Again, CD and DVD were designed to be pressed and counter typical
>>>> usage damage (scratches etc.), for mass media use. The error
>>>> rate isn't alarmingly high:

>>> Having partially composed myself from stitches of laughter: get some
>>> software and a suitable drive to show you the massive number of C1 and
>>> C2 errors from pressed CDs and laser recorded CDs.

> Sorry for the delay in replying.

>> "massive number"? Compared to what, the number of errors
>> that can be corrected?

>> You might also note that if the CD plays, the output of the
>> CD data is beween zero and not noticeable.

> <http://www.cdmasteringservices.com/digitalerrors.htm>

> I was talking about the many pressed and user-recorded CDs that
> massively exceed 220 C1 errors/s (by massively, I mean many thousands
> of C1 errors/s) and bursts of C2 errors that are uncorrectable leading
> to clearly audible distortion.

"many"? As in several hundred or thousand all in all?
OK, I have not yet encountered such a problem with pressed disks
(though I've quite often seen problems with burned CDs and DVDs),
that may skew my oppinion.

>>> I have several
>>> pressed music CDs where the C1 errors have caused so many C2 errors
>>> that good CD players eventually revert to "STOP" mode and most players
>>> either get stuck ("broken record") or keep skipping across the disc.
>>> These CDs do what CDs always do - get worse as they get older.

>> Old CD, right?
>> Yep, the early CDs literally rust internally and thus have a
>> rather short lifetime.

> Those pressed CDs caused the problem from the first day I bought them,
> which was within a month of each one being released. All I was aware of
> at the time was considerable audio distortion. I initially assumed it
> was caused by terrible recording quality. One was so bad I took it back
> to the shop: none of the players in the shop could play it at all.

OK, you got a few bad results from the pressing.
I don't think that's much more than the usual problem with
mass productions: some lemons are always found.

> The audio quality of the CDs have indeed got worse over time even
> though the quality of the players has improved so I guess the discs
> have deteriorated with time.

All disks deteriorate over time.

>> Actually, it seems legal to sell music CDs that
>> - very creatively don't follow the Red Book Standard (and
>> thus play on any given player just by pure chance)

> Yep (I like your wording of that).

Thanks :-)

>> - install rootkit software (looking at you, Sony!)

> Luckily, I avoided installing a rootkit.

Luckily, the only use for Windows is playing games.
(OK, that's true for me, not for everyone.)
And then my music taste is rather far from mainstream.

>> Error correction helps to work with the inevitably existing
>> error rates, even when they increase over time. Proof: The
>> CD wouldn't even be playable without error correction.

> It would be much less useful than a length of string and two tin cans.

Well, you _could_ use it as a mirror for signalling over much
longer distances --- if the sun shines.

>> And TCP doesn't guarantee 100% error free transmission. Never was
>> designed for it. It only used 16 bit checksums (one for the TCP
>> header and data, one for the IP header). It's a virtual
>> circuit, not a virtual error free connection.

> That's an interesting point. My ISP offers free storage space if I
> install the software, which just seems to be a fancy wrapper to an FTP
> client. AFAIK it gives no more reliable data transmission than TCP
> itself.

But for IP connections a very low error rate is necessary.
One would expect an error rate of 2^25-2^30 or better. With 16
bit checksums, the undetected error rate would be 2^41 at worst.

Additionally a bad connection would immediately show up, as
practically *all* packets that are corrupted will show up as
broken. (There's no intelligence to corrupt the packets in a way
that they seem OK.) This high error rate will ring alarm bells.

>>> We all know that TCP delivers corrupted files far
>>> more often than it should, which according to Wolfgang this should be
>>> almost impossible.

>> So do you have any statistics how often a TCP transmission
>> delivers wrong data, and have you compared that to the to be
>> expected error rates? No?

> Guilty as charged! Most applications don't/cannot report TCP errors so
> I have no data. Am I alone in finding that an application such as RAR
> needs to use its redundancy files far more frequently than one would be
> comfortable with?

I'm not using RAR. But I know something about burned CDs and
DVDs. That's why I talk about dvdisaster
http://dvdisaster.net/en/
whenever the talk comes to these disks as storage.

>> BTW, I haven't noticed "TCP" delivering corrupted files.
>> That may, of course, be because applications layering on TCP
>> may use their own error checking data.

> As above, over the years I've experienced far too many data errors over FTP.

Hmmm. In that case I'd look into your home network and into your
connection to the provider. Maybe even the mainboard-HD-cable.

>>> The probability of having
>>> unrecoverable errors is always greater than zero and always increases
>>> towards maximum media read speed.

>> I think you are wrong here. >0, sure, but arbitrary low (if
>> the engineers make that effort) and I don't think it's read
>> speed related. Or would you say the probability of unrecoverable
>> errors per bit read is larger on the outer rim of a HDD than on
>> the inner rim?

> I used to know a fair bit about this, but I've forgotten most of it. My
> brain error rate is rapidly increasing at it gets towards the end :-(

Upgrade to MS Brain 1 SP 3.

Or switch to GNU Encephalon: more features, less bugs, free ---
and your thoughts are actually yours again.

> CD/DVD has the luxury of changing RPM to account for this problem. HDD
> manufacturers have gone a long way to overcome it, but read/write speed
> still deteriorates from beginning to end of the disc.

But the beginning is at the outer rim ... and thus the
sectors pass fastest under the heads.

>> Additionally a well designed medium can have a maximum read
>> speed so that the signals at that speed are read best and are
>> worse at any slower speeds. (magnet patterns passing under a
>> reading head elict a stronger signal when they pass faster.
>> In fact, at speed 0 they don't elict a signal. So increasing
>> the speed *improves* the signal.)

> Yep, but there will always be an optimal speed range for minimum error rate.

True. So "increasing the speed increases the error rate"
isn't the whole story, at least.

>> Do you have looked at the undetectable error rate of your hard
>> drives (according to the manufacturer) recently? Of course
>> you have or will have inaccurate data on your computer
>> system. What will you do about it?

> I've chosen to not let it bother me any more. If I personally lose
> "important" data I'll get over it and generate some more after the
> event.

I see.

>>> Either
>>> that or you employ enough lawyers to work around data corruptions as
>>> they occur.

>> Lawyers restore undetectabble errors? That's a new one.

> My mind often wanderers off into hyperspace instead of staying focused.

Don't you need your heart replaced to do that and survive?[1]

>>> I shall not bother to address each of your points, Wolfgang, because
>>> you have completely failed to understand why the existing CRC
>>> mechanisms used for data integrity have proved themselves to archaic to
>>> the point of being next to useless for any other purposes than cost
>>> reduction and marketing hype.

>> You are the one to use *communication channel bandwidth* theorems
>> on *storage media* where the channel isn't the bottleneck of
>> reliability or speed.

>> OK, what's your payable, scaleable suggestion for the
>> "existing CRC mechanisms" in, say, hard disks (where they
>> don't even exist in the form you claim)?

> As above.

Ignore the problem?
OK, that is one solution, if it's not really a problem.

You could use a RAID (allowing for one or even 2 failed disks in
the stack) with constant checking of the data (e.g. via more
ECC data embedded in the data stream by the RAID driver) and
reconstruct and repair if an error is detected.

That's an external solution.

Alternatively you could say "HDD manufacturers should
further decrease the undetectable error rate".

> Thank you for your detailed reply, Wolfgang.

> I learn from being wrong and absolutely nothing from being right. I
> apologize for directing my animosity towards you. Rather than
> attempting to explain why I do this sometimes, I feel it would be far
> better for me to address my problem rather than to make excuses for it.

I am humbled. Of course I accept your apology, please do accept
mine --- I tend to formulate too harsh and tend to be much closer
to personal attacks than I should.


-Wolfgang

[1] Superluminal, Vonda N. McIntyre

[Andrew Reilly]

On Thu, 29 Mar 2012 23:30:06 +0200, Wolfgang Weisselberg wrote:

>> CD/DVD has the luxury of changing RPM to account for this problem. HDD
>> manufacturers have gone a long way to overcome it, but read/write speed
>> still deteriorates from beginning to end of the disc.
>
> But the beginning is at the outer rim ... and thus the sectors pass
> fastest under the heads.

I'm not familiar with DVD (have a suspicion that it does start at the
outside) but CD definitley starts at the inside. That's why small form-
factor CDs (CD singles) work.

Cheers,

--
Andrew

[David Dyer-Bennet]

Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

> David Dyer-Bennet <dd...@dd-b.net> wrote:
>> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:
>>> Dallas <Cybnorm@spam_me_not.Hotmail.Com> wrote:
>
>>>> 2) Purchase all my software again because install CDs were lost in the
>>>> fire.. i.e. MS Office, Photoshop, Outlook, etc.
>
>>> Aren't they bound to your hardware? In which case they won't
>>> work with your new computer anyway ... whether your house burned
>>> down or you just upgraded your computer.
>
>> No, with the possible exception of Windows itself (depends on what kind
>> of license you have; the default that comes with big-name hardware is
>> rather limited, but I've rarely had that license).
>
> OK, then it's more usual with games than with productive
> software.

Probably; I have little experience with games, especially expensive
games.

>> But everything else
>> you can move to the new computer (I've moved Photoshop, Bibble Pro,
>> Photo Mechanic, Thumbs Plus, various plugins, Picture Window Pro, and
>> oodles of other stuff across a number of computers).
>
> So you don't need a disk image or install DVD for them?

No, just the install files and product keys.

>>> Other software, you just copy the files and settings to the right
>>> place (and that's where the Windows registry sucks 10s of miles
>>> sized asteroids through thin straws).
>
>> Funny, because they invented the registry to make problems from people
>> moving stuff around smaller.
>
> There are better systems, for example "keep everything in the
> program's directory" system. Or the "/etc/$PROGRAM" model,
> which with "$PROGRAM.d" files allows even secondary programs to
> add configuration to the primary program, without having to be
> clever with the original configuration.[1]

That's where Windows started out; the moved away from that because of
massive problems.

Or at least so they said; I tend to agree with you that it's a better
system.

However, the problems involved shared DLLs, which is somewhat a hard
issue.

> Additionally, both models are trivially seen and changed with
> your usually file system tools. With clear text config, it's
> easy to see what they're doing and there's ample space for long
> comments on the options and commented out options, too, so the
> file is self-documenting and easy to change with any text editor.

I tend to agree, but of course most people don't *have* a text editor
they know how to use.

> Contrast and compare the registry, where you need special tools,
> litte if any contained documentation and where real life has
> proven cleaning up the registry is anything but trivial.

But you can easily find everything referencing a particular DLL.

>>>> 3) Install all the Applications, connect to your ISP ect.
>
>>> Connecting to your ISP is step 1.3, downloading your data is
>>> 1.6 and then you do have all your applications.
>
>>>> 4) Recover your data from CrashPlan and restore it to your new computer
>>>> in correct locations.
>
>>>> If you have an off site clone of a hard drive,
>
>>> you have lost your access to it. The key to the bank vault
>>> is lost.
>
>> You've probably got your ID (it lives in my wallet in my pocket, so if I
>> survive my ID probably does too), you can walk into the bank and get
>> them to being a locksmith to let you into your vault.
>
>> I think you get two copies of the key, anyway. One can live at home,
>> one in your car, or something. Or in a drawer at work.
>
> Large fire (due to earthquake). The local bank's vault
> burned/was flooded/crushed by rubble. :-)

Yes. My "favorite" example (I'm actually very sad it happened, but
it's a great example) was Jacques Loewe's photographs from the Kennedy
White House. He had very sensibly stored them in a bank safe-deposit
vault, since they were clearly valuable. Unfortunately, that bank vault
was under the World Trade Center.

You certainly can lose access to things.

My encrypted password database lives in my Dropbox; and the way dropbox
works, there's a very recent copy of it on at least 4 computers in two
cities, plus my phone. Plus, outside of Dropbox, my backup disks. Plus
the thumb drive I carry in my pocket (less recent; but it contains the
most important passwords at least in their current forms).

I'd like to have a copy of my data backed up in cloud storage as well,
but it's currently too expensive and too slow (I've got an ADSL line, 7
megabits in but slightly less than 1 out). Friends living within a mile
of me got the option of up to 100 megabit fiber internet to their
houses, but that's not available on my street (it's a special case, a
company doing wireless throughout the city was installing a new fiber
trunk).

>>>> you'd just install it in
>>>> a new box.
>
>>> Wrong.
>>> 1) Buy a new computer without an OS and without a harddrive.
>>> (have fun finding one)
>
>> Trivially easy, I've bought that way frequently.
>
> Gateway? Dell?
>
> No, you have to shop for that, talk to the local small
> computer store or know your sources.

I've never bought a computer from Dell or Gateway, they have all come
from General Nanosystems (or, formerly, Tran Micro) in the last couple
of decades. But if for some reason I couldn't avoid Dell, it's
trivially easy to overwrite the existing installation, throw out an
included hard drive, or whatever. Those things are not problems,
they're merely expenses.

>>> 2) fetch the clone drive.
>>> 3) Install the clone drive into the computer. Replace the
>>> mainboard, since the hard drive doesn't connect to the mainboard
>>> any more.
>
>> No, install the proper drivers for the new mainboard. It may well do
>> this itself (Windows generally does).
>
> I doubt a proper driver can install a physical PATA connector.

I'm not running anything that antiquated; and if I were, I'd take the
opportunity to correct it if I was doing a complete replacement.

>>> 4) Try to boot the computer.
>>> 5) Hope you have all the drivers for everything with your computer.
>>> Install them.
>>> 6) Connect to the internet. No problem, but ... your whole system
>>> is months or even years out of date! Including the AV software.
>>> It takes longer just to update a new Windows than the usual
>>> timespan between attacks on the holes it patches on a given IP,
>>> so ...
>>> 7) ... try to update all your software faster than an attack
>>> gets you.
>
>> I've never had problems with this when configuring new computers or
>> doing motherboard replacements. The key is, your computer shouldn't
>> ever be *directly exposed* on the internet, where other things can
>> connect to it (unless it's a hardened server system). But mostly at
>> home you're behind NAT, in which case outside systems can't reach yours.
>
> Well, yes, Windows should never be exposed to the open
> internet. That was one of the points.
>
>> So then, just avoid browsing anywhere except Windows Update until you're
>> updated.
>
> Is there a digital signature, and is it checked automatically?

No idea. I've never had a problem, and there isn't any alternative.

>> Or if you're badly paranoid, download them on another computer onto a
>> DVD.
>
> If I'm paranoid, I won't use Windows. :-)

Most security professionals I know use Windows.

>>> 8) Try to make sure your computer has not been attacked yet.
>>> 9) buy all the stuff you bought again since you made the last
>>> clone and install it.
>
>>> 10) Cry for all the rest you didn't backup since then ...
>
>> Which is why automatic backups are good, yes. And why automatic
>> off-site is especially good.
>
> Yep. Very, very true.
>
>> Still, I brought 16GB of data back from a photo trip to the zoo
>> yesterday, and that takes a while to backup via my outbound Internet
>> bandwidth.
>
> And that's why local backups are also valuable.
>
>>>> Now, I have no illusions about trying to boot that hard
>>>> drive in a new computer... in theory you'd boot into safe mode and
>>>> begin the process of installing new drivers. But, completing that,
>>>> you'd have all your programs and data ready to go.
>
>>> The good thing about theory and praxis is that in theory they
>>> both behave the same. In praxis ... well, that's different.
>
>> I've had consistently good results with moterhboard upgrades, though,
>> which is what this is (the hard part).
>
> So have I, but then I don't use Windows outside the games box,
> preferring a real OS.
>
> -Wolfgang
>
> [1] A program might add itself to the task scheduler, add
> configuration to the event monitor checker and so on.

We really need a better way to let programs do occasional checks for
updates and such, without adding infinite background tasks.

[David Dyer-Bennet]

Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

> Pete A <pete3....@nospam.ntlworld.com> wrote:

>> CD/DVD has the luxury of changing RPM to account for this problem. HDD
>> manufacturers have gone a long way to overcome it, but read/write speed
>> still deteriorates from beginning to end of the disc.
>
> But the beginning is at the outer rim ... and thus the
> sectors pass fastest under the heads.

HDDs no longer have the same number of sectors per track; the linear
bit-density is now constant. So the inner tracks bring the same sector
around again faster, but the outer tracks make more sectors accessible
without moving the access arm.

>>> Do you have looked at the undetectable error rate of your hard
>>> drives (according to the manufacturer) recently? Of course
>>> you have or will have inaccurate data on your computer
>>> system. What will you do about it?
>
>> I've chosen to not let it bother me any more. If I personally lose
>> "important" data I'll get over it and generate some more after the
>> event.
>
> I see.

In my case -- ZFS keeps its own, large, checksums on every filesystem
block (data and metadata). I check each week to see if there are
errors. If there are, it can recover the data from the other drive
(2-way mirror). If it can't, it can notify me and I can try to recover
it from an old snapshot, or from one of the backups (which also include
old snapshots).

Errors at the level that doesn't catch, I just lose the data. I hate
that, in theory (never happened). Well, for some photos I could try to
recover from CDS and DVDs I burned, and I should burn some for more
recent work.

[Alan Browne]

DVD first layer starts at the inner rim and works outward - the disk
spins quickly.

As the head moves outward, the disk speed reduces. Data density on the
disk is constant and speed change keeps data rate constant.

The 2nd layer begins at the outer edge and moves inward - disk speed
increases again. You usually see a brief ( < 1 s) pause during the
layer switch. Good mastering will hide that by putting the switch at a
scene change, but even then it's often perceptible.

I've often thought that when the "feature" is first run the reader
should go read the first few seconds of the 2nd layer start and store it
in RAM such that the layer switch event can be made imperceptible while
the optics re-focus.

[landon...@gmail.com]

I do all my editing on files stored on my actual workstation's HDD. Since I use a Mac all I need to do is backup my home directory and I'm sure all my stuff is being backed up. No more rogue settings or files somewhere else like with Windows. Worst case I reinstall the operating system, install my software again and restore my home directory. Backing up the entire operating system would be useful but I'm not that concerned about it. I know I can get the OS back if I need to.

Now how do I back it up? I have an apple script that runs every Sunday night (at like 2AM) that remounts my filer if it has gone idle and has disconnected due to power saving then rsync's my home directory to my NAS filer. The NAS filer is a D-Link DNS-321 that's a few years old. It's setup with two 1TB disks in RAID 1. If one dies I just pop the disk out and put in a new one. The RAID rebuilds. RAID 1 is mirroring (a copy of every bit is on both disks).

I haven't gone as far as making a copy and storing it offsite yet but I've ben thinking about it. All I'd need to do is copy the contents of this 1TB RAID 1 array to a 1TB disk and send it somewhere safe.

Pretty simple setup that I think is effective.

[Wolfgang Weisselberg]

Ah, brain fart on my part. I was talking about HDDs, not
CD/DVD/BlueRay. You're certainly right for CDs, and I think
it's the same for DVDs (at least that's what the reflection
change in partially written DVDs shows).

OTOH, if there's corrosion going on in burned/burnable DVDs, that
should start from the outside (where the carrier goes straight
to the end), not from the inside (where there's a lot of plastic
towards the hole.

-Wolfgang

[Pete A]

On 2012-03-29 22:30:06 +0100, Wolfgang Weisselberg said:

> [Sorry for the snip - I've saved your thoughtful post for reading later.]

>
>> Thank you for your detailed reply, Wolfgang.
>
>> I learn from being wrong and absolutely nothing from being right. I
>> apologize for directing my animosity towards you. Rather than
>> attempting to explain why I do this sometimes, I feel it would be far
>> better for me to address my problem rather than to make excuses for it.
>
> I am humbled. Of course I accept your apology, please do accept
> mine --- I tend to formulate too harsh and tend to be much closer
> to personal attacks than I should.

Thank you very much for saying that, Wolfgang.

I've decided to stop interacting with alt.photography and the rec.photo
newsgroups because I need to put my very limited energy into some new
and exciting mini projects/opportunities that have been presented to me
(most of them are not related to photography).

I do not wish to lose contact with you because you have helped me
enormously to become a better person and to gain a better quality of
life. If you ever feel like sending me an email, just remove the
"nospam" and the following dot from my email address.

Best regards,

Pete

[Wolfgang Weisselberg]

David Dyer-Bennet <dd...@dd-b.net> wrote:
> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:
>> David Dyer-Bennet <dd...@dd-b.net> wrote:
>>> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:


>>>> Other software, you just copy the files and settings to the right
>>>> place (and that's where the Windows registry sucks 10s of miles
>>>> sized asteroids through thin straws).

>>> Funny, because they invented the registry to make problems from people
>>> moving stuff around smaller.

>> There are better systems, for example "keep everything in the
>> program's directory" system. Or the "/etc/$PROGRAM" model,
>> which with "$PROGRAM.d" files allows even secondary programs to
>> add configuration to the primary program, without having to be
>> clever with the original configuration.[1]

> That's where Windows started out; the moved away from that because of
> massive problems.

Maybe they didn't know how to do it right. Wouldn't be a first
for MS.

> Or at least so they said; I tend to agree with you that it's a better
> system.

Oh, well, they also said (and error messaged) that only
MS-DOS would work with it's early Windows 3.x. That wasn't
the truth, of course.

> However, the problems involved shared DLLs, which is somewhat a hard
> issue.

Shared libraries are nothing new, and available in other systems
as well. Of course, there one can upgrade a library that's in use
(if in doubt, the filespace is not released even if the name is
gone, as long as there are filehandles on the old library) and most
upgrades are downward compatible (unless a major change happens).
(Note: a reboot is not needed! At most a restart of the daemon
is needed, if the daemon would otherwise continue to access an
old library with a security hole.)

MS is supposed to be a company with tons of intelligent people,
they should be able to lick the DLL problem all by themselves.

>> Additionally, both models are trivially seen and changed with
>> your usually file system tools. With clear text config, it's
>> easy to see what they're doing and there's ample space for long
>> comments on the options and commented out options, too, so the
>> file is self-documenting and easy to change with any text editor.

> I tend to agree, but of course most people don't *have* a text editor
> they know how to use.

Notepad. Terrible, but it's there.

>> Contrast and compare the registry, where you need special tools,
>> litte if any contained documentation and where real life has
>> proven cleaning up the registry is anything but trivial.

> But you can easily find everything referencing a particular DLL.

And that helps in which way? Is there an automatic deinstaller
for unneeded DLLs?

Other systems know what packages are installed on the systems,
their requirements, etc. and can handle all that stuff and all
upgrades (of many thousands of packages) automatically.

> My encrypted password database lives in my Dropbox; and the way dropbox
> works, there's a very recent copy of it on at least 4 computers in two
> cities, plus my phone.

So all it needs is a delete order, by mistake or code error,
and your password database on Dropbox is gone.

> I'd like to have a copy of my data backed up in cloud storage as well,
> but it's currently too expensive and too slow (I've got an ADSL line, 7
> megabits in but slightly less than 1 out). Friends living within a mile
> of me got the option of up to 100 megabit fiber internet to their
> houses, but that's not available on my street (it's a special case, a
> company doing wireless throughout the city was installing a new fiber
> trunk).

Yep, they're trying to get 4800 fibre trunk subscribers here ---
they're not going to get more than 2000 (closing in a couple days).

They're not even getting close to where I live. I might have
subscribed, even though we have over a year minimum runtime on the
DSL contract. And yes, I want more upload, download's good enough.

>>>>> you'd just install it in
>>>>> a new box.

>>>> Wrong.
>>>> 1) Buy a new computer without an OS and without a harddrive.
>>>> (have fun finding one)

>>> Trivially easy, I've bought that way frequently.

>> Gateway? Dell?

>> No, you have to shop for that, talk to the local small
>> computer store or know your sources.

> I've never bought a computer from Dell or Gateway, they have all come
> from General Nanosystems (or, formerly, Tran Micro) in the last couple
> of decades. But if for some reason I couldn't avoid Dell, it's
> trivially easy to overwrite the existing installation, throw out an
> included hard drive, or whatever. Those things are not problems,
> they're merely expenses.

Well, if it's merely expenses, I could buy Microsoft (or
Google) and take a workstation or two and throw the rest of
the company away. ;->

>>>> 2) fetch the clone drive.
>>>> 3) Install the clone drive into the computer. Replace the
>>>> mainboard, since the hard drive doesn't connect to the mainboard
>>>> any more.

>>> No, install the proper drivers for the new mainboard. It may well do
>>> this itself (Windows generally does).

>> I doubt a proper driver can install a physical PATA connector.

> I'm not running anything that antiquated; and if I were, I'd take the
> opportunity to correct it if I was doing a complete replacement.

A friend of mine just (2 days ago) upgraded his computer.
The floppy drive and the DVD burner and DVD reader no longer fit
to the motherboard.

A clone drive might well be a PATA drive. Or a SATA drive when
SATA's gonna be phased out for whatever comes next.

>>>> 7) ... try to update all your software faster than an attack
>>>> gets you.

>>> I've never had problems with this when configuring new computers or
>>> doing motherboard replacements. The key is, your computer shouldn't
>>> ever be *directly exposed* on the internet, where other things can
>>> connect to it (unless it's a hardened server system). But mostly at
>>> home you're behind NAT, in which case outside systems can't reach yours.

>> Well, yes, Windows should never be exposed to the open
>> internet. That was one of the points.

>>> So then, just avoid browsing anywhere except Windows Update until you're
>>> updated.

>> Is there a digital signature, and is it checked automatically?

> No idea. I've never had a problem, and there isn't any alternative.

Well, the thing is that if a blackhat poisons a DNS cache and
gives it wrong information re: Microsofts update domain names (so
the IP points to a computer of the blackhats choice and delivers
... bad updates, that would cause a lot of computers to try to
get the bad updates.

If they are digitally signed and checked for that, the bad update
doesn't get applied ...


But of course there *is* an alternative; it's called a Red Hat
install CD[1]. Or an Ubuntu live CD. Or even a Mac + OS X.

>>> Or if you're badly paranoid, download them on another computer onto a
>>> DVD.

>> If I'm paranoid, I won't use Windows. :-)

> Most security professionals I know use Windows.

Most photographers use point&shoot cameras, too.

> We really need a better way to let programs do occasional checks for
> updates and such, without adding infinite background tasks.

Look at the system Debian (and Ubuntu, and a couple others) use:
one system to resolve dependencies, clean out what no longer is
needed, check for updates and apply it, extensible (just add the
info where the repository is), several interfaces, including GUI,
fully commandline capable, fully scriptable, triggered (if wanted)
by cron (usually a very flexible "task scheduler"), ..., backed
by known critical bug warning, etc.

One single process that's occasionally run (no new background
task), and does as much or as little as you want.


All it needs is MS adapting such a thing and telling the software
makers "this sort of URL with that sort of directories and
summary files".


-Wolfgang

[1] The 4 'R's of Windows:
Retry
Restart
Reboot
Reinstall
Red Hat

[David Dyer-Bennet]

Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

> David Dyer-Bennet <dd...@dd-b.net> wrote:

>> My encrypted password database lives in my Dropbox; and the way dropbox
>> works, there's a very recent copy of it on at least 4 computers in two
>> cities, plus my phone.
>
> So all it needs is a delete order, by mistake or code error,
> and your password database on Dropbox is gone.

Not exactly. If it's an error by me, I can recover it (Dropbox keeps
old versions). If it's a code error, I may or may not be able to
recover it from them depending on the details of the code error.

And my laptop is mostly powered down, so I can probbly recover the copy
there.

Otherwise, I have to fall back on the backup copies (I bring a copy to
my fileserver regularly).

>> I'd like to have a copy of my data backed up in cloud storage as well,
>> but it's currently too expensive and too slow (I've got an ADSL line, 7
>> megabits in but slightly less than 1 out). Friends living within a mile
>> of me got the option of up to 100 megabit fiber internet to their
>> houses, but that's not available on my street (it's a special case, a
>> company doing wireless throughout the city was installing a new fiber
>> trunk).
>
> Yep, they're trying to get 4800 fibre trunk subscribers here ---
> they're not going to get more than 2000 (closing in a couple days).
>
> They're not even getting close to where I live. I might have
> subscribed, even though we have over a year minimum runtime on the
> DSL contract. And yes, I want more upload, download's good enough.

Maybe I'll have to move out to one of the rural towns that's getting
massinve bradband upgrades!

>>>>>> you'd just install it in
>>>>>> a new box.
>
>>>>> Wrong.
>>>>> 1) Buy a new computer without an OS and without a harddrive.
>>>>> (have fun finding one)
>
>>>> Trivially easy, I've bought that way frequently.
>
>>> Gateway? Dell?
>
>>> No, you have to shop for that, talk to the local small
>>> computer store or know your sources.
>
>> I've never bought a computer from Dell or Gateway, they have all come
>> from General Nanosystems (or, formerly, Tran Micro) in the last couple
>> of decades. But if for some reason I couldn't avoid Dell, it's
>> trivially easy to overwrite the existing installation, throw out an
>> included hard drive, or whatever. Those things are not problems,
>> they're merely expenses.
>
> Well, if it's merely expenses, I could buy Microsoft (or
> Google) and take a workstation or two and throw the rest of
> the company away. ;->

Yes, you could.

Different order of magnitude problem, though.

The OS only adds $50-100 to the workstation price, last I checked
(buying the OS separately is more than that). So that's the size of the
maximum expense. And as I say, I've found it easy to buy systems
without OS when I wanted them.

>>>>> 2) fetch the clone drive.
>>>>> 3) Install the clone drive into the computer. Replace the
>>>>> mainboard, since the hard drive doesn't connect to the mainboard
>>>>> any more.
>
>>>> No, install the proper drivers for the new mainboard. It may well do
>>>> this itself (Windows generally does).
>
>>> I doubt a proper driver can install a physical PATA connector.
>
>> I'm not running anything that antiquated; and if I were, I'd take the
>> opportunity to correct it if I was doing a complete replacement.
>
> A friend of mine just (2 days ago) upgraded his computer.
> The floppy drive and the DVD burner and DVD reader no longer fit
> to the motherboard.
>
> A clone drive might well be a PATA drive. Or a SATA drive when
> SATA's gonna be phased out for whatever comes next.

Could be -- but for this very reason, you should choose your clone drive
to be easy to use in the event it's needed.

>>>>> 7) ... try to update all your software faster than an attack
>>>>> gets you.
>
>>>> I've never had problems with this when configuring new computers or
>>>> doing motherboard replacements. The key is, your computer shouldn't
>>>> ever be *directly exposed* on the internet, where other things can
>>>> connect to it (unless it's a hardened server system). But mostly at
>>>> home you're behind NAT, in which case outside systems can't reach yours.
>
>>> Well, yes, Windows should never be exposed to the open
>>> internet. That was one of the points.
>
>>>> So then, just avoid browsing anywhere except Windows Update until you're
>>>> updated.
>
>>> Is there a digital signature, and is it checked automatically?
>
>> No idea. I've never had a problem, and there isn't any alternative.
>
> Well, the thing is that if a blackhat poisons a DNS cache and
> gives it wrong information re: Microsofts update domain names (so
> the IP points to a computer of the blackhats choice and delivers
> ... bad updates, that would cause a lot of computers to try to
> get the bad updates.
>
> If they are digitally signed and checked for that, the bad update
> doesn't get applied ...
>

Sure, that's better. It just hasn't happened anywhere I've heard of yet,
so it's not a high priority to me to protect against it.

Note that I'm not at all sure Microsoft *doesn't* use such digital
signatures; I just don't know offhand that they do.

> But of course there *is* an alternative; it's called a Red Hat
> install CD[1]. Or an Ubuntu live CD. Or even a Mac + OS X.

"The package said 'requires windows 95 or better'. So I installed
Linux."

>>>> Or if you're badly paranoid, download them on another computer onto a
>>>> DVD.
>
>>> If I'm paranoid, I won't use Windows. :-)
>
>> Most security professionals I know use Windows.
>
> Most photographers use point&shoot cameras, too.

But most professional photographers do not use them for their main
work.

(And I did mean *computer* security professionals, not the broader
class.)

>> We really need a better way to let programs do occasional checks for
>> updates and such, without adding infinite background tasks.
>
> Look at the system Debian (and Ubuntu, and a couple others) use:
> one system to resolve dependencies, clean out what no longer is
> needed, check for updates and apply it, extensible (just add the
> info where the repository is), several interfaces, including GUI,
> fully commandline capable, fully scriptable, triggered (if wanted)
> by cron (usually a very flexible "task scheduler"), ..., backed
> by known critical bug warning, etc.
>
> One single process that's occasionally run (no new background
> task), and does as much or as little as you want.

Sure, I've used yum and apt-get, and pkg-add over on Solaris. Oh, and
CPAN.

> All it needs is MS adapting such a thing and telling the software
> makers "this sort of URL with that sort of directories and
> summary files".

Yep.

[Wolfgang Weisselberg]

David Dyer-Bennet <dd...@dd-b.net> wrote:
> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

>> David Dyer-Bennet <dd...@dd-b.net> wrote:

>>> My encrypted password database lives in my Dropbox; and the way dropbox
>>> works, there's a very recent copy of it on at least 4 computers in two
>>> cities, plus my phone.

>> So all it needs is a delete order, by mistake or code error,
>> and your password database on Dropbox is gone.

> Not exactly. If it's an error by me, I can recover it (Dropbox keeps
> old versions).

Unless you also purge the old versions.

> If it's a code error, I may or may not be able to
> recover it from them depending on the details of the code error.

Yep.

> And my laptop is mostly powered down, so I can probbly recover the copy
> there.

Oops, it just fell down and is broken.

> Otherwise, I have to fall back on the backup copies (I bring a copy to
> my fileserver regularly).

Well ... :-)

>>> I'd like to have a copy of my data backed up in cloud storage as well,
>>> but it's currently too expensive and too slow (I've got an ADSL line, 7
>>> megabits in but slightly less than 1 out). Friends living within a mile
>>> of me got the option of up to 100 megabit fiber internet to their
>>> houses, but that's not available on my street (it's a special case, a
>>> company doing wireless throughout the city was installing a new fiber
>>> trunk).

>> Yep, they're trying to get 4800 fibre trunk subscribers here ---
>> they're not going to get more than 2000 (closing in a couple days).

^^^^
correction, that should read 2800
^

>> They're not even getting close to where I live. I might have
>> subscribed, even though we have over a year minimum runtime on the
>> DSL contract. And yes, I want more upload, download's good enough.

> Maybe I'll have to move out to one of the rural towns that's getting
> massinve bradband upgrades!

Hmmm, we recently got new cabling in the house, all paid for by
an phone/TV/internet provider. Includes analog and basic (ad paid
and tax paid --- same as analog) TV. (The hook? The house has to
stay with them for 7 or so years.) Since the cable connection's
there, 150 MBit down and 5 MBit up (and no cap) for 35 EUR/month
(which is less than what I used to pay for 2 MBit). However,
as we don't have 1000Mbit-Ethernet, 100MBit/s is fine and will
save 10 EUR/month.

Still, the fibre trunk would have been able for 200MBit down
(not that I need that) and a whopping 100 MBit up --- which would
really play nice with Online Backups.

>>>>>>> you'd just install it in
>>>>>>> a new box.

>>>>>> Wrong.
>>>>>> 1) Buy a new computer without an OS and without a harddrive.
>>>>>> (have fun finding one)

>>>>> Trivially easy, I've bought that way frequently.

>>>> Gateway? Dell?

>>>> No, you have to shop for that, talk to the local small
>>>> computer store or know your sources.

>>> I've never bought a computer from Dell or Gateway, they have all come
>>> from General Nanosystems (or, formerly, Tran Micro) in the last couple
>>> of decades. But if for some reason I couldn't avoid Dell, it's
>>> trivially easy to overwrite the existing installation, throw out an
>>> included hard drive, or whatever. Those things are not problems,
>>> they're merely expenses.

>> Well, if it's merely expenses, I could buy Microsoft (or
>> Google) and take a workstation or two and throw the rest of
>> the company away. ;->

> Yes, you could.

> Different order of magnitude problem, though.

Sure.

> The OS only adds $50-100 to the workstation price, last I checked
> (buying the OS separately is more than that). So that's the size of the
> maximum expense. And as I say, I've found it easy to buy systems
> without OS when I wanted them.

I don't want to pay the Microsoft tax, when they're working
against me and my wishes. Should the US have paid appeasement
money in the billions to the USSR during the cold war, instead
of letting them know that the US might not win --- but the USSR
surely wouldn't either, if they pulled a fast one?

>>>>>> 2) fetch the clone drive.
>>>>>> 3) Install the clone drive into the computer. Replace the
>>>>>> mainboard, since the hard drive doesn't connect to the mainboard
>>>>>> any more.

>>>>> No, install the proper drivers for the new mainboard. It may well do
>>>>> this itself (Windows generally does).

>>>> I doubt a proper driver can install a physical PATA connector.

>>> I'm not running anything that antiquated; and if I were, I'd take the
>>> opportunity to correct it if I was doing a complete replacement.

>> A friend of mine just (2 days ago) upgraded his computer.
>> The floppy drive and the DVD burner and DVD reader no longer fit
>> to the motherboard.

>> A clone drive might well be a PATA drive. Or a SATA drive when
>> SATA's gonna be phased out for whatever comes next.

> Could be -- but for this very reason, you should choose your clone drive
> to be easy to use in the event it's needed.

I choose to be able to download the OS and almost all applications
from the Debian mirrors. That's a very distributed and resilient
backup.

>>>>> So then, just avoid browsing anywhere except Windows Update until you're
>>>>> updated.

>>>> Is there a digital signature, and is it checked automatically?

>>> No idea. I've never had a problem, and there isn't any alternative.

>> Well, the thing is that if a blackhat poisons a DNS cache and
>> gives it wrong information re: Microsofts update domain names (so
>> the IP points to a computer of the blackhats choice and delivers
>> ... bad updates, that would cause a lot of computers to try to
>> get the bad updates.

>> If they are digitally signed and checked for that, the bad update
>> doesn't get applied ...

> Sure, that's better. It just hasn't happened anywhere I've heard of yet,
> so it's not a high priority to me to protect against it.

> Note that I'm not at all sure Microsoft *doesn't* use such digital
> signatures; I just don't know offhand that they do.

They might --- to fight "piracy". Just in case the Somalians
capture a freighter full of MS DVDs.

>> But of course there *is* an alternative; it's called a Red Hat
>> install CD[1]. Or an Ubuntu live CD. Or even a Mac + OS X.

> "The package said 'requires windows 95 or better'. So I installed
> Linux."

Indeed. I switched to Linux before 95 was born.

>>>>> Or if you're badly paranoid, download them on another computer onto a
>>>>> DVD.

>>>> If I'm paranoid, I won't use Windows. :-)

>>> Most security professionals I know use Windows.

>> Most photographers use point&shoot cameras, too.

> But most professional photographers do not use them for their main
> work.

So shold security professionals.
Of course, if "use" implies hacking into, breaking the system's
security or finding the latest holes, that's another matter.
And if "use" implies trying to harden the darn thing ("It's
possible to make Windows secure, in much the same way as it's
possible to make a bullet-proof vest out of cheese--you just need
an awful lot of cheese and the end result doesn't smell good.
--Jim"), well, they're prostituting themselves, so they
better get well paid ... but they have my sympathies.

It's a very restful thing that firewalls, packet inspectors,
security loggers, intrusion detection systems and their ilk
are usually not run on Windows.

> (And I did mean *computer* security professionals, not the broader
> class.)

That was what I assumed.

>>> We really need a better way to let programs do occasional checks for
>>> updates and such, without adding infinite background tasks.

>> Look at the system Debian (and Ubuntu, and a couple others) use:
>> one system to resolve dependencies, clean out what no longer is
>> needed, check for updates and apply it, extensible (just add the
>> info where the repository is), several interfaces, including GUI,
>> fully commandline capable, fully scriptable, triggered (if wanted)
>> by cron (usually a very flexible "task scheduler"), ..., backed
>> by known critical bug warning, etc.

>> One single process that's occasionally run (no new background
>> task), and does as much or as little as you want.

> Sure, I've used yum and apt-get, and pkg-add over on Solaris.

And MS doesn't manage.

> Oh, and CPAN.

Yep, CPAN is fun --- if the author knew about "use strict" and
"use warnings". Some don't. Then sometimes you have to reinvent
the wheel.

>> All it needs is MS adapting such a thing and telling the software
>> makers "this sort of URL with that sort of directories and
>> summary files".

> Yep.

So tell me again why they're rich and have the resources?

-Wolfgang

--
"That's our advantage at Microsoft; we set the standards and we can
change them." -- Karen Hargrove, Microsoft (quoted in the Feb 1993
Unix Review editorial)

[David Dyer-Bennet]

Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

> David Dyer-Bennet <dd...@dd-b.net> wrote:
>> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:
>
>>> David Dyer-Bennet <dd...@dd-b.net> wrote:
>
>>>> My encrypted password database lives in my Dropbox; and the way dropbox
>>>> works, there's a very recent copy of it on at least 4 computers in two
>>>> cities, plus my phone.
>
>>> So all it needs is a delete order, by mistake or code error,
>>> and your password database on Dropbox is gone.
>
>> Not exactly. If it's an error by me, I can recover it (Dropbox keeps
>> old versions).
>
> Unless you also purge the old versions.
>
>> If it's a code error, I may or may not be able to
>> recover it from them depending on the details of the code error.
>
> Yep.
>
>> And my laptop is mostly powered down, so I can probbly recover the copy
>> there.
>
> Oops, it just fell down and is broken.

I can remove the SSD drive and access it on some random other computer
using a USB dongle and booting from a Linux Live CD. SSD drives have
pretty high G-force tolerance!

This is kind of a fun game, actually.

>> Otherwise, I have to fall back on the backup copies (I bring a copy to
>> my fileserver regularly).
>
> Well ... :-)

I should automate that process, though.

(Just brought a new backup to work today to swap for the old off-site copy.)

I didn't wire for gigabit copper ethernet in 1995 when we bought this
house :-). But I do have a gigabit switch on the shelf with my
fileserver and desktop machine, so those two communicate very fast, and
my laptop does when I plug it in down there.

> Still, the fibre trunk would have been able for 200MBit down
> (not that I need that) and a whopping 100 MBit up --- which would
> really play nice with Online Backups.

Yes, it certainly would. Well, I've known my country is a bit backwards
on broadband and especially fiber. And poorer neighborhoods in a city
are especially bad.

I don't either, and in practice I've only paid for Microsoft OSs that
I've actually used (and I don't feel like I should complain about
that). I'm just pointing out that even if, because of locality or
urgency or changing policies or something, you can't avoid the Microsoft
OS, that's a philosophical issue and an expense rather than a real
show-stopping problem.

>>>>>>> 2) fetch the clone drive.
>>>>>>> 3) Install the clone drive into the computer. Replace the
>>>>>>> mainboard, since the hard drive doesn't connect to the mainboard
>>>>>>> any more.
>
>>>>>> No, install the proper drivers for the new mainboard. It may well do
>>>>>> this itself (Windows generally does).
>
>>>>> I doubt a proper driver can install a physical PATA connector.
>
>>>> I'm not running anything that antiquated; and if I were, I'd take the
>>>> opportunity to correct it if I was doing a complete replacement.
>
>>> A friend of mine just (2 days ago) upgraded his computer.
>>> The floppy drive and the DVD burner and DVD reader no longer fit
>>> to the motherboard.
>
>>> A clone drive might well be a PATA drive. Or a SATA drive when
>>> SATA's gonna be phased out for whatever comes next.
>
>> Could be -- but for this very reason, you should choose your clone drive
>> to be easy to use in the event it's needed.
>
> I choose to be able to download the OS and almost all applications
> from the Debian mirrors. That's a very distributed and resilient
> backup.

Yes, that's a pretty decently reliable source on the scale of these
things.

>>>>>> So then, just avoid browsing anywhere except Windows Update until you're
>>>>>> updated.
>
>>>>> Is there a digital signature, and is it checked automatically?
>
>>>> No idea. I've never had a problem, and there isn't any alternative.
>
>>> Well, the thing is that if a blackhat poisons a DNS cache and
>>> gives it wrong information re: Microsofts update domain names (so
>>> the IP points to a computer of the blackhats choice and delivers
>>> ... bad updates, that would cause a lot of computers to try to
>>> get the bad updates.
>
>>> If they are digitally signed and checked for that, the bad update
>>> doesn't get applied ...
>
>> Sure, that's better. It just hasn't happened anywhere I've heard of yet,
>> so it's not a high priority to me to protect against it.
>
>> Note that I'm not at all sure Microsoft *doesn't* use such digital
>> signatures; I just don't know offhand that they do.
>
> They might --- to fight "piracy". Just in case the Somalians
> capture a freighter full of MS DVDs.

Their security work has a large reactive component, but when they were
setting up the auto-update scheme, it must have been blazingly obvious
that having it compromised would be the worst PR disaster imaginable, so
I kind of hope they took good precautions.

>>> But of course there *is* an alternative; it's called a Red Hat
>>> install CD[1]. Or an Ubuntu live CD. Or even a Mac + OS X.
>
>> "The package said 'requires windows 95 or better'. So I installed
>> Linux."
>
> Indeed. I switched to Linux before 95 was born.

I had my first Linux install whenever kernel 0.99pl13 was current, as I
remember it. But it's never been able to replace my desktop system --
while the Gimp is capable, it doesn't support what I need nearly as well
as Photoshop (for restoration and fine printing). I *can* run Bibble
Pro on Linux, and in fact have when my (previous work) work laptop was
running linux (saved me carrying two laptops on trips).

>>>>>> Or if you're badly paranoid, download them on another computer onto a
>>>>>> DVD.
>
>>>>> If I'm paranoid, I won't use Windows. :-)
>
>>>> Most security professionals I know use Windows.
>
>>> Most photographers use point&shoot cameras, too.
>
>> But most professional photographers do not use them for their main
>> work.
>
> So shold security professionals.
> Of course, if "use" implies hacking into, breaking the system's
> security or finding the latest holes, that's another matter.
> And if "use" implies trying to harden the darn thing ("It's
> possible to make Windows secure, in much the same way as it's
> possible to make a bullet-proof vest out of cheese--you just need
> an awful lot of cheese and the end result doesn't smell good.
> --Jim"), well, they're prostituting themselves, so they
> better get well paid ... but they have my sympathies.

If you're writing a book on the topic, your publisher is going to want a
DOC file, and may well send it back annotated, too; for example. It's
just where the world is.

> It's a very restful thing that firewalls, packet inspectors,
> security loggers, intrusion detection systems and their ilk
> are usually not run on Windows.

Yes.

>>>> We really need a better way to let programs do occasional checks for
>>>> updates and such, without adding infinite background tasks.
>
>>> Look at the system Debian (and Ubuntu, and a couple others) use:
>>> one system to resolve dependencies, clean out what no longer is
>>> needed, check for updates and apply it, extensible (just add the
>>> info where the repository is), several interfaces, including GUI,
>>> fully commandline capable, fully scriptable, triggered (if wanted)
>>> by cron (usually a very flexible "task scheduler"), ..., backed
>>> by known critical bug warning, etc.
>
>>> One single process that's occasionally run (no new background
>>> task), and does as much or as little as you want.
>
>> Sure, I've used yum and apt-get, and pkg-add over on Solaris.
>
> And MS doesn't manage.

Partly because they aren't actually preparing a full distribution in the
Linux sense; they only provide the OS, and a few application packages.

>> Oh, and CPAN.
>
> Yep, CPAN is fun --- if the author knew about "use strict" and
> "use warnings". Some don't. Then sometimes you have to reinvent
> the wheel.

Sure, there's essentially no curation of the collection; I was thinking
of the dependency handling and automatic installation, as being related
to the binary systems you had brought up. (Lots of useful modules there,
though.)

>>> All it needs is MS adapting such a thing and telling the software
>>> makers "this sort of URL with that sort of directories and
>>> summary files".
>
>> Yep.
>
> So tell me again why they're rich and have the resources?

Relentless focus on the bottom line?

[Wolfgang Weisselberg]

David Dyer-Bennet <dd...@dd-b.net> wrote:
> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:
>> David Dyer-Bennet <dd...@dd-b.net> wrote:
>>> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:

>>>> David Dyer-Bennet <dd...@dd-b.net> wrote:

>>>>> My encrypted password database lives in my Dropbox; and the way dropbox
>>>>> works, there's a very recent copy of it on at least 4 computers in two
>>>>> cities, plus my phone.

>>>> So all it needs is a delete order, by mistake or code error,
>>>> and your password database on Dropbox is gone.

>>> Not exactly. If it's an error by me, I can recover it (Dropbox keeps
>>> old versions).

>> Unless you also purge the old versions.

>>> If it's a code error, I may or may not be able to
>>> recover it from them depending on the details of the code error.

>> Yep.

>>> And my laptop is mostly powered down, so I can probbly recover the copy
>>> there.

>> Oops, it just fell down and is broken.

> I can remove the SSD drive and access it on some random other computer
> using a USB dongle and booting from a Linux Live CD. SSD drives have
> pretty high G-force tolerance!

Ok, it didn't fall down, the computer was switched on (and the
SSD build in) while lightning struck the power cable somewhere
too close.

> This is kind of a fun game, actually.

It is.

>>> Otherwise, I have to fall back on the backup copies (I bring a copy to
>>> my fileserver regularly).

>> Well ... :-)

> I should automate that process, though.

Yep.

Ah, yes, in a couple of years we'll all need gigabit ethernet
for our internet connection ... but we'll already be all
wireless then. :-)

>> Still, the fibre trunk would have been able for 200MBit down
>> (not that I need that) and a whopping 100 MBit up --- which would
>> really play nice with Online Backups.

> Yes, it certainly would. Well, I've known my country is a bit backwards
> on broadband and especially fiber. And poorer neighborhoods in a city
> are especially bad.

And I used to have a good static IP at home. Then I moved.
Time for IPV6. Time for fibre to the home, or whatever
better comes along next.

>>> The OS only adds $50-100 to the workstation price, last I checked
>>> (buying the OS separately is more than that). So that's the size of the
>>> maximum expense. And as I say, I've found it easy to buy systems
>>> without OS when I wanted them.

>> I don't want to pay the Microsoft tax, when they're working
>> against me and my wishes. Should the US have paid appeasement
>> money in the billions to the USSR during the cold war, instead
>> of letting them know that the US might not win --- but the USSR
>> surely wouldn't either, if they pulled a fast one?

> I don't either, and in practice I've only paid for Microsoft OSs that
> I've actually used (and I don't feel like I should complain about
> that). I'm just pointing out that even if, because of locality or
> urgency or changing policies or something, you can't avoid the Microsoft
> OS, that's a philosophical issue and an expense rather than a real
> show-stopping problem.

Well, appeasement doesn't work well, as history shows. Paying
Danegeld only compounds the problem: you'll never get rid of
the Dane.

>>>>>>> So then, just avoid browsing anywhere except Windows Update until you're
>>>>>>> updated.

>>>>>> Is there a digital signature, and is it checked automatically?

>>>>> No idea. I've never had a problem, and there isn't any alternative.

>>>> Well, the thing is that if a blackhat poisons a DNS cache and
>>>> gives it wrong information re: Microsofts update domain names (so
>>>> the IP points to a computer of the blackhats choice and delivers
>>>> ... bad updates, that would cause a lot of computers to try to
>>>> get the bad updates.

>>>> If they are digitally signed and checked for that, the bad update
>>>> doesn't get applied ...

>>> Sure, that's better. It just hasn't happened anywhere I've heard of yet,
>>> so it's not a high priority to me to protect against it.

>>> Note that I'm not at all sure Microsoft *doesn't* use such digital
>>> signatures; I just don't know offhand that they do.

>> They might --- to fight "piracy". Just in case the Somalians
>> capture a freighter full of MS DVDs.

> Their security work has a large reactive component, but when they were
> setting up the auto-update scheme, it must have been blazingly obvious
> that having it compromised would be the worst PR disaster imaginable, so
> I kind of hope they took good precautions.

Microsoft has a good spindoctoring team:
| "No one's jumped off the top of the [Windows] building here, so I guess
| that's a pretty good indicator that it can't be all that bad," said Rob
| Bennett, Microsoft group product manager.
http://tinyurl.com/2qpyzj

>>>> But of course there *is* an alternative; it's called a Red Hat
>>>> install CD[1]. Or an Ubuntu live CD. Or even a Mac + OS X.

>>> "The package said 'requires windows 95 or better'. So I installed
>>> Linux."

>> Indeed. I switched to Linux before 95 was born.

> I had my first Linux install whenever kernel 0.99pl13 was current, as I
> remember it. But it's never been able to replace my desktop system --
> while the Gimp is capable, it doesn't support what I need nearly as well
> as Photoshop (for restoration and fine printing). I *can* run Bibble
> Pro on Linux, and in fact have when my (previous work) work laptop was
> running linux (saved me carrying two laptops on trips).

Maybe I am easy to please. Linux or Unix with GNU-tools pleases
me, agrees with me and I feel at home there.
Windows doesn't please me for some reason.

>>>>>>> Or if you're badly paranoid, download them on another computer onto a
>>>>>>> DVD.

>>>>>> If I'm paranoid, I won't use Windows. :-)

>>>>> Most security professionals I know use Windows.

>>>> Most photographers use point&shoot cameras, too.

>>> But most professional photographers do not use them for their main
>>> work.

>> So shold security professionals.
>> Of course, if "use" implies hacking into, breaking the system's
>> security or finding the latest holes, that's another matter.
>> And if "use" implies trying to harden the darn thing ("It's
>> possible to make Windows secure, in much the same way as it's
>> possible to make a bullet-proof vest out of cheese--you just need
>> an awful lot of cheese and the end result doesn't smell good.
>> --Jim"), well, they're prostituting themselves, so they
>> better get well paid ... but they have my sympathies.

> If you're writing a book on the topic, your publisher is going to want a
> DOC file, and may well send it back annotated, too; for example. It's
> just where the world is.

I've seen enough "this book was written ..." and "set with ..."
to assume that at least some publishers are saner than that.

And if that fails, there are some export tools to read DOC,
even if one doesn't want LibreOffice.

>> It's a very restful thing that firewalls, packet inspectors,
>> security loggers, intrusion detection systems and their ilk
>> are usually not run on Windows.

> Yes.

Imagine your life support popping up a window ...

>>>>> We really need a better way to let programs do occasional checks for
>>>>> updates and such, without adding infinite background tasks.

>>>> Look at the system Debian (and Ubuntu, and a couple others) use:
>>>> one system to resolve dependencies, clean out what no longer is
>>>> needed, check for updates and apply it, extensible (just add the
>>>> info where the repository is), several interfaces, including GUI,
>>>> fully commandline capable, fully scriptable, triggered (if wanted)
>>>> by cron (usually a very flexible "task scheduler"), ..., backed
>>>> by known critical bug warning, etc.

>>>> One single process that's occasionally run (no new background
>>>> task), and does as much or as little as you want.

>>> Sure, I've used yum and apt-get, and pkg-add over on Solaris.

>> And MS doesn't manage.

> Partly because they aren't actually preparing a full distribution in the
> Linux sense; they only provide the OS, and a few application packages.

That's not a viable excuse for such a big company.

>>> Oh, and CPAN.

>> Yep, CPAN is fun --- if the author knew about "use strict" and
>> "use warnings". Some don't. Then sometimes you have to reinvent
>> the wheel.

> Sure, there's essentially no curation of the collection; I was thinking
> of the dependency handling and automatic installation, as being related
> to the binary systems you had brought up. (Lots of useful modules there,
> though.)

Yep, that too.

>>>> All it needs is MS adapting such a thing and telling the software
>>>> makers "this sort of URL with that sort of directories and
>>>> summary files".

>>> Yep.

>> So tell me again why they're rich and have the resources?

> Relentless focus on the bottom line?

Ferengi-programming and Borg-marketing?


-Wolfgang

[David Dyer-Bennet]

And, when the asteroid hits Minneapolis, it'll probably get my backup at
work, my backups in the fire safe at home, and my primary disks at home
(both halves of the mirror pair). And me.

My old optical disks (not fully current) down in Northfield might
survive, depending a lot on the asteroid in question :-). But nobody
will much care if I and all my friends in Minneapolis are gone.

Certainly having things online and powered on puts them at risk, which
is why I use USB external drives and take them offline when I'm not
backup up to them.

I could have gotten a fire safe with USB pass-through, so I could back
up to a drive in the safe -- but only a USB-powered drive, and those
aren't big enough for a single-drive backup. If I put my own power
pass-through in I would void the warranty, and quite likely actually
impair the fire (and water) protection seriously; and wouldn't know for
sure since I'm not in a position to do suitable tests. So I didn't.

>>>> Otherwise, I have to fall back on the backup copies (I bring a copy to
>>>> my fileserver regularly).
>
>>> Well ... :-)
>
>> I should automate that process, though.
>
> Yep.

And that kind of thing is one of the clear values of conversations like
this, highlighting areas where meaningful improvements can be made at
affordable prices (merely a small amount of work, in that case).

I need to replace the lost thumb drive; I used to have a recent copy in
my pocket all the time.

I still do (and DNS pointing at it); only the one on the router, but I
can map ports through as needed.

I used to have a block of 8 (meaning 6 usable) at home, and used them.

>>>> The OS only adds $50-100 to the workstation price, last I checked
>>>> (buying the OS separately is more than that). So that's the size of the
>>>> maximum expense. And as I say, I've found it easy to buy systems
>>>> without OS when I wanted them.
>
>>> I don't want to pay the Microsoft tax, when they're working
>>> against me and my wishes. Should the US have paid appeasement
>>> money in the billions to the USSR during the cold war, instead
>>> of letting them know that the US might not win --- but the USSR
>>> surely wouldn't either, if they pulled a fast one?
>
>> I don't either, and in practice I've only paid for Microsoft OSs that
>> I've actually used (and I don't feel like I should complain about
>> that). I'm just pointing out that even if, because of locality or
>> urgency or changing policies or something, you can't avoid the Microsoft
>> OS, that's a philosophical issue and an expense rather than a real
>> show-stopping problem.
>
> Well, appeasement doesn't work well, as history shows. Paying
> Danegeld only compounds the problem: you'll never get rid of
> the Dane.

It's a complex benefit analysys; short-term vs. long-term, just me
vs. everybody. Can get to a "belling the cat" scenario pretty easily.

I put Cygwin on my windows boxes, and Emacs, and Perl, and then it's
okay.

I use Photoshop adjustment layers a lot, with layer masks. Most
commonly curves adjustment layers, though not exclusively. It lets me
make major or minor modifications to many aspects of my treatment of a
photo without going back to the beginning and starting over (and without
re-working the bits). I just can't work that way anywhere else. (This
is only for restoration or "fine" printing; for event photos for the web
or whatever Bibble pro does fine, I don't put that level of work into
each and every snapshot).

>>>>>>>> Or if you're badly paranoid, download them on another computer onto a
>>>>>>>> DVD.
>
>>>>>>> If I'm paranoid, I won't use Windows. :-)
>
>>>>>> Most security professionals I know use Windows.
>
>>>>> Most photographers use point&shoot cameras, too.
>
>>>> But most professional photographers do not use them for their main
>>>> work.
>
>>> So shold security professionals.
>>> Of course, if "use" implies hacking into, breaking the system's
>>> security or finding the latest holes, that's another matter.
>>> And if "use" implies trying to harden the darn thing ("It's
>>> possible to make Windows secure, in much the same way as it's
>>> possible to make a bullet-proof vest out of cheese--you just need
>>> an awful lot of cheese and the end result doesn't smell good.
>>> --Jim"), well, they're prostituting themselves, so they
>>> better get well paid ... but they have my sympathies.
>
>> If you're writing a book on the topic, your publisher is going to want a
>> DOC file, and may well send it back annotated, too; for example. It's
>> just where the world is.
>
> I've seen enough "this book was written ..." and "set with ..."
> to assume that at least some publishers are saner than that.
>
> And if that fails, there are some export tools to read DOC,
> even if one doesn't want LibreOffice.

Does LibreOffice handle the annotation protocol? It can certainly
export a perfectly good DOC file, but when it comes back marked up, how
does that work?

>>> It's a very restful thing that firewalls, packet inspectors,
>>> security loggers, intrusion detection systems and their ilk
>>> are usually not run on Windows.
>
>> Yes.
>
> Imagine your life support popping up a window ...

Not really THAT much worse than logging an IO error to the system log,
though.

>>>>>> We really need a better way to let programs do occasional checks for
>>>>>> updates and such, without adding infinite background tasks.
>
>>>>> Look at the system Debian (and Ubuntu, and a couple others) use:
>>>>> one system to resolve dependencies, clean out what no longer is
>>>>> needed, check for updates and apply it, extensible (just add the
>>>>> info where the repository is), several interfaces, including GUI,
>>>>> fully commandline capable, fully scriptable, triggered (if wanted)
>>>>> by cron (usually a very flexible "task scheduler"), ..., backed
>>>>> by known critical bug warning, etc.
>
>>>>> One single process that's occasionally run (no new background
>>>>> task), and does as much or as little as you want.
>
>>>> Sure, I've used yum and apt-get, and pkg-add over on Solaris.
>
>>> And MS doesn't manage.
>
>> Partly because they aren't actually preparing a full distribution in the
>> Linux sense; they only provide the OS, and a few application packages.
>
> That's not a viable excuse for such a big company.

I suspect that they'd be barred on anti-trust grounds from even trying
to do it, precisely because they *are* such a big company.

>>>> Oh, and CPAN.
>
>>> Yep, CPAN is fun --- if the author knew about "use strict" and
>>> "use warnings". Some don't. Then sometimes you have to reinvent
>>> the wheel.
>
>> Sure, there's essentially no curation of the collection; I was thinking
>> of the dependency handling and automatic installation, as being related
>> to the binary systems you had brought up. (Lots of useful modules there,
>> though.)
>
> Yep, that too.
>
>
>>>>> All it needs is MS adapting such a thing and telling the software
>>>>> makers "this sort of URL with that sort of directories and
>>>>> summary files".
>
>>>> Yep.
>
>>> So tell me again why they're rich and have the resources?
>
>> Relentless focus on the bottom line?
>
> Ferengi-programming and Borg-marketing?

yes!

[Wolfgang Weisselberg]

David Dyer-Bennet <dd...@dd-b.net> wrote:
> Wolfgang Weisselberg <ozcv...@sneakemail.com> writes:
>> David Dyer-Bennet <dd...@dd-b.net> wrote:

>>> This is kind of a fun game, actually.

>> It is.

> And, when the asteroid hits Minneapolis, it'll probably get my backup at
> work, my backups in the fire safe at home, and my primary disks at home
> (both halves of the mirror pair). And me.

It'll be a major flood or an earthquake and gaslines rupturing
or a hurricane. You'll all be evacuated, but you won't be able
to grab your backups. Mostly because you're out of town and they
won't let you in.

It could also be a zombie outbreak. You better drop
everything and run!

> My old optical disks (not fully current) down in Northfield might
> survive, depending a lot on the asteroid in question :-). But nobody
> will much care if I and all my friends in Minneapolis are gone.

The dust will render them --- and the drives to read them ---
useless.

> Certainly having things online and powered on puts them at risk, which
> is why I use USB external drives and take them offline when I'm not
> backup up to them.

The powerful self-aware computer virus will sleep in your
computer until you re-attach the drives ...

> I could have gotten a fire safe with USB pass-through, so I could back
> up to a drive in the safe -- but only a USB-powered drive, and those
> aren't big enough for a single-drive backup.

And they're liable to get too many volts over the cable.

>>>>> Otherwise, I have to fall back on the backup copies (I bring a copy to
>>>>> my fileserver regularly).

>>>> Well ... :-)

>>> I should automate that process, though.

>> Yep.

> And that kind of thing is one of the clear values of conversations like
> this, highlighting areas where meaningful improvements can be made at
> affordable prices (merely a small amount of work, in that case).

> I need to replace the lost thumb drive; I used to have a recent copy in
> my pocket all the time.

Easy won, easy lost.

>>>> Still, the fibre trunk would have been able for 200MBit down
>>>> (not that I need that) and a whopping 100 MBit up --- which would
>>>> really play nice with Online Backups.

>>> Yes, it certainly would. Well, I've known my country is a bit backwards
>>> on broadband and especially fiber. And poorer neighborhoods in a city
>>> are especially bad.

>> And I used to have a good static IP at home. Then I moved.
>> Time for IPV6. Time for fibre to the home, or whatever
>> better comes along next.

> I still do (and DNS pointing at it); only the one on the router, but I
> can map ports through as needed.

> I used to have a block of 8 (meaning 6 usable) at home, and used them.

Where is IP V6 when you need it?

>>>>> The OS only adds $50-100 to the workstation price, last I checked
>>>>> (buying the OS separately is more than that). So that's the size of the
>>>>> maximum expense. And as I say, I've found it easy to buy systems
>>>>> without OS when I wanted them.

>>>> I don't want to pay the Microsoft tax, when they're working
>>>> against me and my wishes. Should the US have paid appeasement
>>>> money in the billions to the USSR during the cold war, instead
>>>> of letting them know that the US might not win --- but the USSR
>>>> surely wouldn't either, if they pulled a fast one?

>>> I don't either, and in practice I've only paid for Microsoft OSs that
>>> I've actually used (and I don't feel like I should complain about
>>> that). I'm just pointing out that even if, because of locality or
>>> urgency or changing policies or something, you can't avoid the Microsoft
>>> OS, that's a philosophical issue and an expense rather than a real
>>> show-stopping problem.

>> Well, appeasement doesn't work well, as history shows. Paying
>> Danegeld only compounds the problem: you'll never get rid of
>> the Dane.

> It's a complex benefit analysys; short-term vs. long-term, just me
> vs. everybody.

Yep, a politician (or sect leader or marketing specialist) can
even make mass suicide sound rational and the very best of ideas.

Microsoft has lots of all of the above.

> Can get to a "belling the cat" scenario pretty easily.

The cat learns how to move without the bell making any sound.
Nothing won.

>>> I had my first Linux install whenever kernel 0.99pl13 was current, as I
>>> remember it. But it's never been able to replace my desktop system --
>>> while the Gimp is capable, it doesn't support what I need nearly as well
>>> as Photoshop (for restoration and fine printing). I *can* run Bibble
>>> Pro on Linux, and in fact have when my (previous work) work laptop was
>>> running linux (saved me carrying two laptops on trips).

>> Maybe I am easy to please. Linux or Unix with GNU-tools pleases
>> me, agrees with me and I feel at home there.
>> Windows doesn't please me for some reason.

> I put Cygwin on my windows boxes, and Emacs, and Perl, and then it's
> okay.

Okay like a root canal without pain management, rather than
life-long torture, yes.

> I use Photoshop adjustment layers a lot, with layer masks. Most
> commonly curves adjustment layers, though not exclusively. It lets me
> make major or minor modifications to many aspects of my treatment of a
> photo without going back to the beginning and starting over (and without
> re-working the bits). I just can't work that way anywhere else. (This
> is only for restoration or "fine" printing; for event photos for the web
> or whatever Bibble pro does fine, I don't put that level of work into
> each and every snapshot).

Photoshop doesn't necessitate Windows.

I have no idea. A quick google turns up
http://docs.libreoffice.org/sw/html/classsw_1_1annotation_1_1AnnotationMenuButton.html
which looks like it might.

>>>> It's a very restful thing that firewalls, packet inspectors,
>>>> security loggers, intrusion detection systems and their ilk
>>>> are usually not run on Windows.

>>> Yes.

>> Imagine your life support popping up a window ...

> Not really THAT much worse than logging an IO error to the system log,
> though.

You've just changed your breath rate.
[Accept] [Decline] [Exit]

And of course everything stops working till you click the
right button. Can you say "General Failure" without air?

I'd rather have an I/O error to the mouse or display logged in
my breathing machine ...

>>>>>>> We really need a better way to let programs do occasional checks for
>>>>>>> updates and such, without adding infinite background tasks.

>>>>>> Look at the system Debian (and Ubuntu, and a couple others) use:
>>>>>> one system to resolve dependencies, clean out what no longer is
>>>>>> needed, check for updates and apply it, extensible (just add the
>>>>>> info where the repository is), several interfaces, including GUI,
>>>>>> fully commandline capable, fully scriptable, triggered (if wanted)
>>>>>> by cron (usually a very flexible "task scheduler"), ..., backed
>>>>>> by known critical bug warning, etc.

>>>>>> One single process that's occasionally run (no new background
>>>>>> task), and does as much or as little as you want.

>>>>> Sure, I've used yum and apt-get, and pkg-add over on Solaris.

>>>> And MS doesn't manage.

>>> Partly because they aren't actually preparing a full distribution in the
>>> Linux sense; they only provide the OS, and a few application packages.

>> That's not a viable excuse for such a big company.

> I suspect that they'd be barred on anti-trust grounds from even trying
> to do it, precisely because they *are* such a big company.

Sheesh, they could just let every programmer add their own URLs
at their own servers. No anti-trust, no only way, just one
convenient way to do it.

>>>>>> All it needs is MS adapting such a thing and telling the software
>>>>>> makers "this sort of URL with that sort of directories and
>>>>>> summary files".

>>>>> Yep.

>>>> So tell me again why they're rich and have the resources?

>>> Relentless focus on the bottom line?

>> Ferengi-programming and Borg-marketing?

> yes!

Not a single foot, not a single inch to the Borg!

-Wolfgang