[9fans] mass storage jukebox

[Andrew]

So being a hardware freak... Ive managed to salvage a pair of optical disk
jukeboxes. One of them is a Sony CXD8515Q and uses apparently run-of-the-
mill "rewritable optical disks". The other jukebox happens to be of HP
descent, which i havent opened yet, and Im wondering if there is support
for either of these things under plan 9? I certainly wouldnt mind having
an actual jukebox for my backups. Of course next question in line is,
where can i get more of these expensive opti-disks? The cheapest ive
found per storage unit is $10 per gigabyte, and that is with 9gb disks,
the lower end of the spectrum has 1.2 gb disks for $25-35, and I of
course dont have money for 10+ optical disks to fill the jukebox up.

Thanks,
Andrew

[Russ Cox]

Just buy a big disk and run a pseudo-worm.
Optical media just doesn't make sense anymore.

Russ

[Geoff Collyer]

I think optical media still make sense, especially at ebay prices. I
like having automatic backup that isn't subject to head crashes. I
recently bought an HP 80EX there for $500 and am about to switch my
main file system to it, probably tonight. I'll also be adding a pair
of 80GB IDE disks to act as a mirrored cached, since I only have one
optical drive and don't want to thrash the jukebox to death. More
recently I bought an HP 40FX for $150 on ebay, but it hasn't arrived
yet (I was hoping to get an HP 1200EX with 10 drives, comparable to
emelie, but bigger, but the bidding got out of hand).

The main requirement is that you have exactly the model of crummy,
ancient, lousy, oxide-flaking, 30-kilobit-per-second Sony jukebox that
1127 has or had for bootes and fornax (the one driven by
/sys/src/fs/dev/sony.c) or that you have pretty much any old SCSI-2
MMC jukebox, such as the HPs. I just plugged in the 80EX, configured
it and it worked (I did enable conf.dumpreread out of paranoia, but so
far it's reported no errors):

filsys main c[w2w3]j(w4w5)(l<0-15>l<16-31>)
filsys dump o
filsys other w0

: cpu; lc /n/fsb
Directories:
29000 386 68000 68020 960
acme adm alpha arm cron
dist i lib mail mips
mnt n net.gig power rc
rls sparc sys tmp usr

Files:
LICENSE NOTICE
: cpu; lc /n/fsbdump/2002
Directories:
0917 0918 0919 0920 0921
0925 0926 0927 0928 0929
0930 1001 1002 10021 1003
10031 1004 10041 10042 10043
1005 10051 1006 1007 10071
1008 10081 1009 1010 1011
1012 1013 1014 1015 10151
1016
: cpu;

[Andrew]

thats not the point, i just want to make use of the jukebox for the fun
of it. A hobby if you will. Sure, I _could_ go and just buy a 160gb ide
drive and be done with this, but thats no fun, making use of older
equipment, in bizarre sadistic sort of way is fun for me. obviously
on a full scale modern system id want modern hardware, but my plan 9
implementation is anything but modern by today's standards, which is
fine for me and the other interested parties because this is just for
enthusiasts sake.

[Lyndon Nerenberg]

>I think optical media still make sense, especially at ebay prices.

So did I, until I priced out new gear. For $2000 (Canadian) I was able
to build a 480 TB RAID fileserver. Legacy hardware just isn't worth it.

Now if Mark would just write the glue to make the 2400A talk to
the fileserver kernel ... ;-)

--lyndon

[Andrew]

well the HP jukebox happens to be a 20XT, upon a closer inspection has
a total of 16 slots inside for disks. The other sony jukebox has 20,
id prefer the sony of course, do you have any pointers on what to look
for purchasing disks, or on how to determine if either of these are
compatible?

thanks.

[Lyndon Nerenberg]

>So did I, until I priced out new gear. For $2000 (Canadian) I was able
>to build a 480 TB RAID fileserver. Legacy hardware just isn't worth it.

I was smoking WHAT? Make that a 480 *G*B RAID.

It's still cheap compared to the cost of building goo around
an optical array.

--lyndon

[William Josephson]

On Wed, Oct 16, 2002 at 11:15:00PM -0400, Russ Cox wrote:

> Just buy a big disk and run a pseudo-worm.
> Optical media just doesn't make sense anymore.

Except for off-site backups, I'd second russ's
recommendations. The best argument I've heard
for the worm and the dump was an economic one:
by the time you've used all the available
storage, a newer, bigger one is on the market.
I think the same argument now applies to disks,
possibly with the mirror patch to the fs kernel
or using hardware raid. For off-site backups,
I've been using 700MB CDRs with dump9660. If
it weren't for Window's bone-headed determination
not to read ISO 9660 filesystems on DVD-R's I'd
have already replaced the CDs with DVDs (I keep
running away screaming from the UDF standard).
Once the 40-50GB DVD "standard" becomes a reality,
I think DVD+Venti may provide a reasonable
alternative to copying my data over the network
to several fileservers along the east coast.

[Geoff Collyer]

RAID sounds like an appealling alternative to magneto-optical at
first. But if you bought all the disks in your RAID array at the same
time, they're all going to fail at about the same time, for
sufficiently fuzzy values of ``about''. If you keep one hot spare, as
seems to be typical, when one disk dies, you need to replace it before
a second one dies. There's also the possibility of multiple
concurrent failures to due earthquakes, something being dropped on the
disk array, nearby lightning strike, kittens pulling the power cord
out of the wall with their teeth, etc. But if you're on two weeks'
vacation (or four weeks' if you work somewhere other than the US), you
may not get back home in time to replace the dead disk before a second
one dies. You might, but I'm paranoid and I've been collecting files
for 30 years, and I'd rather not lose them. Also during those 30
years, I've seen and heard an awful lot of magnetic disk head crashes.
I really, really like the lack of optical disk head crashes.

I also like automatic backups that don't involve me doing anything.
Once DVD-RAM jukeboxes exist and drop way, way down in price, they'd
be a reasonable alternative to M-O disks, though DVD-RAM disks still
aren't write-once. I don't think any of the current contenders for
DVD standard are write-once and yet allow incremental writing without
major penalty (megabytes thrown away); true? The DVD standards
mud-wrestling / squabbles / wars don't help, but everybody wants a
royalty. Philips apparently made a lot of money off the Compact
Cassette and the CD.

As for jukebox compatibility, if you have manuals, see if they mention
SCSI-2 MMC (multi-media command set) support. There's an MMC subset
for jukeboxes. If you can't tell, plug the jukebox into a file server
and see if it's recognised when you boot the file server and configure
it to expect the jukebox. The first device in the config string after
"j" is the robot arm and the rest are M-O drives inside the jukebox.
I think failure to recognise the jukebox will produce a flurry of SCSI
error messages.

For optical disks, you want magneto-optical disks that match the
drives in your jukebox. HP drives can write one generation back and
read two generations back. The usual sizes are 1.3, 2.6, 5.2 and I
think 9.4 GB (I don't have any of the 9.x GB drives). There may have
been a 650MB size in antiquity. The disks are standardised by ISO and
the standards can be found on www.iso.ch, though you probably have to
pay to read them. At least you should be able to get the numbers of
the relevant standards from the ISO web site. Note too that sector
size is a function of total size. For 2.6GB disks, sectors are 1024
bytes; for 5.2GB disks, sectors are 2048 bytes. I suspect that 9.xGB
disks use 4096-byte sectors. Make sure that your file server block
size (RBUFSIZE) is at least as big as your M-O disk sector size. This
probably won't be an issue for most people until at least the next
generation of M-O disks, assuming that they use 8192-byte sectors.

[Lyndon Nerenberg]

I've seen (and heard) disks explode, too. Being in the machine room when
it happens is fun. But that only happens once or twice in a lifetime for
most of us. I've become paranoid about exploding disks only lately, now
that I'm doing most of my work from home. Tape backups are much more of
an effort than throwing money at hardware. I will trust the drives to
keep running (in a RAID) more than I will trust a tape drive, or
anything else that requires operator intervention. Especially when I'm
on vacation. MO requires a monkey to do the backup, as does tape. RAID
doesn't. I'll take my chances.

--lyndon.

[Andrew]

not that MO is better then a RAID, or vice versa, but isnt the point of
a jukebox to take the place of said backup-gnome for an extended amount
of time?

[Fco.J.Ballesteros]

A (cheap) mirrowed fake worm (as said before in the thread)
is quite nice for backups. However, for peace of mind,
I just keep an up-to-date copy of everything in my laptop, instead
of burning CDs; all in all, I want to run Plan 9 standanlone in the laptop
when disconnected.

Of course, that doesn't save the dumps, but it's just
in the case of a disaster that breaks both disks in the worms.

BTW, I'm very concerned about moving the data in the worms to
venti as soon as it becomes our std file server. (I wouldn't like
to keep the current fs just to service the dumps in that case).
Is out there any plan to easy the transition and automate the conversion
of files in the dump to venti? If not, I'd like to help there.

[Geoff Collyer]

Exactly; an M-O jukebox doesn't require any manual intervention to
take backups. The Plan 9 file server dumps to M-O jukeboxes
automatically and unattended. I'll take an M-O jukebox over RAID
*any* day.

And the problem isn't so much exploding disks as magnet disk head
crashes: the disk head, flying at, what, a few micrometers?, above the
disk platter and 3,600 - 10,000 RPM (60 - 167 revolutions per second)
typically, and often continuously for years, drops and slaps the oxide
on the surface of the platter, stripping or at least damaging the
oxide from multiple tracks. (There's a great poster that used to be
in machine rooms showing a single smoke particle dwarfing the
head-to-surface gap.) If you're really unlucky, the head doesn't
bounce back, but adheres to the surface, taking out the entirety of
the tracks spanned by the head. If you're really, really unlucky, the
head drops during a seek or the head can still move across the disk
surface after adhering somewhat (or coming detached from the disk arm
as a result of the crash), so a few good seeks can take out many or
all of the tracks on a platter. A good bump to the drive used to be
able to crash heads. Admittedly, modern sealed (non-removable) disks
seem to be more resilient than older ones, but, for example, I had 4
or 5 Seagate SCSI disks die due to head crashes shortly after moving
to California (and this with a moving company that claimed to know how
to move computers), so head crashes are still not once-in-a-lifetime
events, especially with IDE disks, which typically have only 3-year
warranties.

[Boyd Roberts]

Geoff Collyer wrote:

>I've seen and heard an awful lot of magnetic disk head crashes.
>

I've seen remarkably few magnetic disk head crashes. Although, I was
seeing about one a month as soon as a Storageworks RAID array was
installed. Good thing those horrible RZ-4? were RAID 5. Bad thing
was that they were so unreliable that that you needed to RAID them.

Maybe it was all a convenient marketing ploy ...

[r...@vitanuova.com]

i've thought for a while that it would be nice to have the optical
jukebox, but mirror it with a hard disk, so you'd get the speed of
access of the hard disk, along with the crash resistance of the
optical disk.

when we moved to the jukebox, it was great in theory (and great fun to
hear the jukebox chatter as one did "yesterday -d") but in practise
certain access patterns (e.g. searching through the history) became
prohibitively expensive.

i thought that a big cache would solve this, but actually blocks get
modified in place, so the current cache rarely holds yesterday's
blocks.

would the mirroring code in the fileserver be able to deal with such a
disparity in access times?

[Geoff Collyer]

I was afraid that a huge cache might not be enough. Mirroring should
do the job, if you use a big mag. disk as your master and mirror it
onto the jukebox, like this:

filsys main ch1{h3j(w4w5)(l<0-31>)}
filsys dump o
filsys other {h0h2}

This also mirrors the "other" device, from h0 to h2. h1 can be a
fairly small cache. The mirror device ({}) writes the rightmost
sub-device first and proceeds leftward, but reads the leftmost
sub-device first and proceeds rightward if necessary. Thus in the
above configuration, writes to the worm device would go to the
jukebox, j(w4w5)(l<0-31>), first and then be mirrored to h3 (4th IDE
disk). However, h3 would be read first, and only if a read error were
detected would the jukebox be read. So yes, the mirror device is
probably the right solution.

Now I just need a quick way (faster than fs-block-at-a-time) to copy
my already-populated first side (l0) to h3 before I start the
{h3j(w4w5)} mirror. I wonder if scuzz is up to the job?