Build your own storage - blog from BackBlaze
Jayarama Shenoy
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/#more-150
A question to those who'd think of doing something similar themselves - at what price point would there be an opportunity for a vendor to be a value add?
Backbalze's pod is not too different than Sun's Thumper (aka X4550). Some plumbing details are different, CPU & memory configs are different and so is Linux/JFS vs Solaris/ZFS. You can argue that in each case, Sun's options are superior, but that and support/warranty were not enough to offset the $8K vs $50K.
So what price point would've made sense? Any one from Backblaze on this list?
With Windows Live, you can organize, edit, and share your photos. Click here.
Peglar, Robert
Petabytes, sure. Petabytes of integral data, nope. No DIF == no integrity.
From the article: “…When you strip away the marketing terms and fancy logos from any storage solution, data ends up on a hard drive.”
Can’t argue with that. But what data? The data that you think you wrote, at the address you think you wrote it to, at the LUN or ID you think you sent it to? How do you know?
But let’s look a little deeper here. They use two power supplies, 760W rated each. It looks like the unit must have both supplies up to function, as they have subdivided duties. Lose one supply, half your drives disappear, and if you lose half the drives in the scheme they use (see below) your box is dead. Not very good if you have a halfway decent SLA. But perhaps they are assuming cloud vendors will not have SLAs with teeth – which is certainly the case today.
Then there are disk rebuilds and sparing. No hot spares configured, or at least described. Looks like they do three sets of double parity 13+2. So all drives are holding necessary information. No mention of how one spares out a drive, so you have to assume entire unit downtime to replace a drives and allow a rebuild to begin. Speaking of which, how long does that take? Rebuilding a 1.5TB drive – and the SiL chipset they use cannot recognize white space, so they have to rebuild the entire drive – takes tens of hours, easily. Have you ever rebuilt a drive from a 13+2 set? It’s ugly. I have to read 19.5 TB of data to rebuild a single 1.5TB drive. How long does it take to read 19.5 TB of data? Even if the box could write data at max speed to the spare, which is 120MB/sec or 432 GB/hour, it would take 4 hours. But that doesn’t count the read time – and it takes 45 hours to read 19.5 TB of data @ 432 GB/hr – never mind the compute time needed to recalculate double parity on that entire set. That’s the problem.
Then there is the mechanicals. Quite literally, they use rubber bands to ‘dampen vibration’. Now, that’s certainly novel, but I wouldn’t trust my data to that. What’s next, duct tape? I bet this box goes 30 rads easily on the vibration test. They also mount all the drives in the same orientation, a sure-fire way to induce torque and stress. I would also not want to be a drive in the rearmost group of 15, given their depiction of fans – and they use fans, not blowers.
On average, they will lose 2 drives every year (4 per 100 per annum) and by year 5, it will probably rise to 6-8 per year, given their use of consumer-level drives. So, over the course of 5 years, you may have to replace roughly half the drive population. Did they mention that in their cost analysis?
Seagate themselves says this drive is not best adapted for array-based storage. Look at their website for this drive and see their recommended applications – PCs, desktop (not array) RAID and personal external storage. This is because of the duty cycle that these drives are built to. Is this array powered on and active all the time? Verbatim from the drive manual:
Normal I/O duty cycle for desktop personal computers. Operation at excessive I/O duty cycle may degrade product reliability.
The paper mentions expense. Sure, I am worried about expense too. I am worried about the biggest expense of all – humans. I need a server admin to manage and babysit this machine. How many of these can one admin manage? The answer is, much less than I would expect of a cloud storage array. Management is completely by hand. Compare to storage arrays which are intelligent and need no management beyond initial provisioning.
Just replicate data to protect it, right? To what degree of replication? Is 2X enough? I know several cloud providers that replicate 3X using arrays that are far more reliable than this, not made of nylon shims and rubber bands. So, now that petabyte becomes $351,000, and I also need triple the humans to manage triple the units. Plus, that is not counting any drive replacement cost over time, and for 3 units (135 drives) I will probably have to replace 60-70 drives over 5 years for another 9 grand or so. This also assumes there is no cost for a human to replace a drive, an ‘extraordinary event’.
But let’s be very generous and say they can build and operate/maintain this box for 5 years for $500 grand. That’s certainly cheaper than commercial arrays. But again, for me, the biggest issue is integrity. At petabyte scale, it’s a huge problem for any ATA technology. Just ask the guys at CERN, even with FC drives that didn’t do DIF it’s a huge problem.
Just the UCR errors alone at petabyte scale with this box are scary. This drive is only 1 in 10^14. 1*10^14 bits is 1.25*10^13 bytes, or 125 TB. This means if you spin and read a petabyte of this stuff, you get 8 errors which are uncorrectable. Murphy says those 8 errors are in the data you really needed.
Good news – cheap storage. Bad news – cheap storage. This thing is a nice home-built whitebox. It would do well to stay there. You wouldn’t run a serious cloud compute infrastructure on dirt-cheap whitebox servers; likewise, you shouldn’t run a serious cloud storage infrastructure on dirt-cheap whitebox storage.
Rob
No virus
found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.74/2339 - Release Date: 09/03/09
05:50:00
Jayarama Shenoy
Subject: [ Cloud Computing ] Re: Build your own storage - blog from BackBlaze
Date: Thu, 3 Sep 2009 13:17:27 -0500
From: Robert...@xiotech.com
To: cloud-c...@googlegroups.com
Windows Live: Make it easier for your friends to see what you’re up to on Facebook. Find out more.
Peglar, Robert
Figures below actually should be: 1.25 * 10^13 bytes is 12.5 TB, not 125 TB. This means there are 80 unrecoverable errors per petabyte, not 8. Sorry for any confusion. It actually makes the problem worse…
Rob
Peglar, Robert
It is absolutely true. ZFS can’t and doesn’t know about addressing and LUN/ID errors at the disk level. It can check the integrity of the data (payload) as do some other filesystems, but that’s only 1/3rd of the problem. Now, if ZFS appended the T10 DIF field to each 512 at the initiator, your answer would be correct, but ZFS does no such thing. It does scrubbing and “resilvering”, but that is not DIF. The former merely checks to see if blocks _can_ be read and the latter makes an educated guess on a block mismatch which one is right and attempts to rewrite it.
But again, that is just the data, not the addressing. DIF prevents such errors.
Jayarama Shenoy
So while a file system that also checked block level integrity on top of it's own checksum would be better (more usually is), ZFS in this application would be good enough.
DIF itself might not be fail proof either (phantom writes, but you can read the ZFS blog as well for other examples) and so on and so forth. My point is that a lack of T10 DIF in itself does not damn BackBlaze's solution for data integrity (And, in theory, ZFS could've cost them just as much as whatever they chose).
My original question - since it is now 3 emails deep - is: what price will this market bear to improve on the home brewed stuff. One company voted that 8X was not the answer, still curious to know what they think it might be. (I guess there are no Backblaze people on here, sigh).
Subject: [ Cloud Computing ] Re: Build your own storage - blog from BackBlaze
Peglar, Robert
>>> If you know for certain your data is corrupted
And therein lies the rub.
Lack of DIF at petabyte-scale does indeed damn it for integrity. You are correct, there are even more checks beyond T10 DIF that could be done, but in terms of the actual SCSI CDB, DIF covers the 3 inputs that the initiator supplies; payload, address/length (and its subsequent V->P translation), and LUN.
As for market pricing, what value do you place on your data? It’s really an exercise in risk management. Look at it this way; how much would it cost you to recreate/restore that petabyte? What is the time value of not having access to that petabyte? If a petabyte of data is only worth $117,000 to you, that’s one thing. For most companies, though, their petabyte is worth far, far more than that.
Then, there is the issue of storage performance – which is an entirely different (but related) topic. Cost of space is one thing, cost of time is very much another. There is absolutely no mention of performance in the post.
If they really wanted massive petabytes on the cheap, without regard to latency or performance, they should use the technology which draws no power when not doing I/O and can be purchased for under $30,000/raw PB as opposed to the $80,000/raw PB they spend now. The power savings alone would be significant.
Version: 8.5.409 / Virus Database: 270.13.76/2344 - Release Date: 09/03/09
18:05:00
Jacob Farmer
My question -- Does anyone out there have a reference design (or better yet a product) that can achieve real economies of scale and tell a good story for hosting Petabytes of general purpose data? Even better, does anyone have a good story for backing up such a beast?
I've seen some interesitng "object" storage models that manage large collections of files, but does anyone have a general purpose file system that can scale to the moon and still be affordable and resilient? If not, does anyone want to speculate as to what such a file system might look like?
-> Jacob
________________________________
From: cloud-c...@googlegroups.com on behalf of Peglar, Robert
Sent: Fri 9/4/2009 6:42 AM
To: cloud-c...@googlegroups.com
Subject: [ Cloud Computing ] Re: Build your own storage - blog from BackBlaze
And therein lies the rub.
Lack of DIF at petabyte-scale does indeed damn it for integrity. You are correct, there are even more checks beyond T10 DIF that could be done, but in terms of the actual SCSI CDB, DIF covers the 3 inputs that the initiator supplies; payload, address/length (and its subsequent V->P translation), and LUN.
As for market pricing, what value do you place on your data? It's really an exercise in risk management. Look at it this way; how much would it cost you to recreate/restore that petabyte? What is the time value of not having access to that petabyte? If a petabyte of data is only worth $117,000 to you, that's one thing. For most companies, though, their petabyte is worth far, far more than that.
Can't argue with that. But what data? The data that you think you wrote, at the address you think you wrote it to, at the LUN or ID you think you sent it to? How do you know?
Then there are disk rebuilds and sparing. No hot spares configured, or at least described. Looks like they do three sets of double parity 13+2. So all drives are holding necessary information. No mention of how one spares out a drive, so you have to assume entire unit downtime to replace a drives and allow a rebuild to begin. Speaking of which, how long does that take? Rebuilding a 1.5TB drive - and the SiL chipset they use cannot recognize white space, so they have to rebuild the entire drive - takes tens of hours, easily. Have you ever rebuilt a drive from a 13+2 set? It's ugly. I have to read 19.5 TB of data to rebuild a single 1.5TB drive. How long does it take to read 19.5 TB of data? Even if the box could write data at max speed to the spare, which is 120MB/sec or 432 GB/hour, it would take 4 hours. But that doesn't count the read time - and it takes 45 hours to read 19.5 TB of data @ 432 GB/hr - never mind the compute time needed to recalculate double parity on that entire set. That's the problem.
Then there is the mechanicals. Quite literally, they use rubber bands to 'dampen vibration'. Now, that's certainly novel, but I wouldn't trust my data to that. What's next, duct tape? I bet this box goes 30 rads easily on the vibration test. They also mount all the drives in the same orientation, a sure-fire way to induce torque and stress. I would also not want to be a drive in the rearmost group of 15, given their depiction of fans - and they use fans, not blowers.
On average, they will lose 2 drives every year (4 per 100 per annum) and by year 5, it will probably rise to 6-8 per year, given their use of consumer-level drives. So, over the course of 5 years, you may have to replace roughly half the drive population. Did they mention that in their cost analysis?
Normal I/O duty cycle for desktop personal computers. Operation at excessive I/O duty cycle may degrade product reliability.
Just replicate data to protect it, right? To what degree of replication? Is 2X enough? I know several cloud providers that replicate 3X using arrays that are far more reliable than this, not made of nylon shims and rubber bands. So, now that petabyte becomes $351,000, and I also need triple the humans to manage triple the units. Plus, that is not counting any drive replacement cost over time, and for 3 units (135 drives) I will probably have to replace 60-70 drives over 5 years for another 9 grand or so. This also assumes there is no cost for a human to replace a drive, an 'extraordinary event'.
But let's be very generous and say they can build and operate/maintain this box for 5 years for $500 grand. That's certainly cheaper than commercial arrays. But again, for me, the biggest issue is integrity. At petabyte scale, it's a huge problem for any ATA technology. Just ask the guys at CERN, even with FC drives that didn't do DIF it's a huge problem.
Just the UCR errors alone at petabyte scale with this box are scary. This drive is only 1 in 10^14. 1*10^14 bits is 1.25*10^13 bytes, or 125 TB. This means if you spin and read a petabyte of this stuff, you get 8 errors which are uncorrectable. Murphy says those 8 errors are in the data you really needed.
Rob
From: cloud-c...@googlegroups.com [mailto:cloud-c...@googlegroups.com] On Behalf Of Jayarama Shenoy
Sent: Thursday, September 03, 2009 10:36 AM
To: cloud-c...@googlegroups.com
Subject: [ Cloud Computing ] Build your own storage - blog from BackBlaze
Very interesting post on making Petabytes on a budget for cloud storage.
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/#more-150
A question to those who'd think of doing something similar themselves - at what price point would there be an opportunity for a vendor to be a value add?
Backbalze's pod is not too different than Sun's Thumper (aka X4550). Some plumbing details are different, CPU & memory configs are different and so is Linux/JFS vs Solaris/ZFS. You can argue that in each case, Sun's options are superior, but that and support/warranty were not enough to offset the $8K vs $50K.
So what price point would've made sense? Any one from Backblaze on this list?
________________________________
No virus found in this incoming message.
________________________________
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.74/2339 - Release Date: 09/03/09 05:50:00
________________________________
Jeff Darcy
> I've seen some interesitng "object" storage models that manage large collections of files, but does anyone have a general purpose file system that can scale to the moon and still be affordable and resilient? If not, does anyone want to speculate as to what such a file system might look like?
couple of dozen parallel filesystems - e.g. Lustre, PVFS2, GlusterFS,
Ceph - that can scale very high in terms of capacity and performance.
There are fewer that can scale across distance, primarily due to the
difficulty of maintaining consistency across high-latency
low-reliability links. That brings up the question of whether most
people really need a true "general purpose" filesystem - which is
actually rather "special purpose" in terms of the very detailed
semantics that such a thing must preserve. Many people don't need
strong or fine-grained consistency. Many people don't need a
hierarchical namespace with full support for hard and soft links, atomic
rename, read/write to removed files, etc. Many people don't need full
VFS-layer integration, though it's easily had via mechanisms like FUSE
and allows applications to run without using special libraries so long
as they remain within the "semantic envelope" defined by whatever
non-filesystem thing is at the other end (e.g. Amazon S3 for s3fs). The
more of these features one is willing to give up, the more options one
gains wrt distributing stuff on a global scale. You can start with a
traditional filesystem and water down its semantics to make it perform
reasonably at global scale, or you can start with a distributed
key/value store and enrich its semantics until it's "enough like" a
traditional filesystem for your own purposes. It's all a matter of what
tradeoffs make sense for a particular application or user.
Jayarama Shenoy
Thanks for re-setting. From the Tier1 vendors, I thought Sun's Thumper was the closest box product (and it seemed to have had it's share of teething troubles initially - around the mechanical design aspects of it).
For the file system, wouldn't Hadoop/GFS be the thing you described. (Also need to mention MogileFS and it's derivatives). These are not strictly Posix compliant file systems, but they appear to get the job done (and interfacing to them in your applications does not appear to be that hard).
I suspect (from Backblaze's description) that they may be using a roll-their-own 'file system' and would use MogileFS as a guess to what it looks like.
Jay
ps I also make my living in the 'enterprise storage world', but am inclined to believe that the Backblaze box will hold together far better than people give it credit for. And the robustness that the box itself may lack might be made up above this layer.
I would've done this box just a bit differently (and correct some of the potential shortcomings without the expenditure of too much money). My original question was to determine if there's enough money in it to make it worth my time to think more about this problem.
Subject: [ Cloud Computing ] If not Backblaze, than what? --- changing the discussion a little
Date: Fri, 4 Sep 2009 10:38:53 -0400
From: jfa...@cambridgecomputer.com
To: cloud-c...@googlegroups.com
Sent: Fri 9/4/2009 6:42 AM
To: cloud-c...@googlegroups.com
Subject: [ Cloud Computing ] Re: Build your own storage - blog from BackBlaze
>>> If you know for certain your data is corrupted
And therein lies the rub.
Lack of DIF at petabyte-scale does indeed damn it for integrity. You are correct, there are even more checks beyond T10 DIF that could be done, but in terms of the actual SCSI CDB, DIF covers the 3 inputs that the initiator supplies; payload, address/length (and its subsequent V->P translation), and LUN.
As for market pricing, what value do you place on your data? It’s really an exercise in risk management. Look at it this way; how much would it cost you to recreate/restore that petabyte? What is the time value of not having access to that petabyte? If a petabyte of data is only worth $117,000 to you, that’s one thing. For most companies, though, their petabyte is worth far, far more than that.
Then, there is the issue of storage performance – which is an entirely different (but related) topic. Cost of space is one thing, cost of time is very much another. There is absolutely no mention of performance in the post.
From the article: “…When you strip away the marketing terms and fancy logos from any storage solution, data ends up on a hard drive.”
Can’t argue with that. But what data? The data that you think you wrote, at the address you think you wrote it to, at the LUN or ID you think you sent it to? How do you know?
But let’s look a little deeper here. They use two power supplies, 760W rated each. It looks like the unit must have both supplies up to function, as they have subdivided duties. Lose one supply, half your drives disappear, and if you lose half the drives in the scheme they use (see below) your box is dead. Not very good if you have a halfway decent SLA. But perhaps they are assuming cloud vendors will not have SLAs with teeth – which is certainly the case today.
Then there are disk rebuilds and sparing. No hot spares configured, or at least described. Looks like they do three sets of double parity 13+2. So all drives are holding necessary information. No mention of how one spares out a drive, so you have to assume entire unit downtime to replace a drives and allow a rebuild to begin. Speaking of which, how long does that take? Rebuilding a 1.5TB drive – and the SiL chipset they use cannot recognize white space, so they have to rebuild the entire drive – takes tens of hours, easily. Have you ever rebuilt a drive from a 13+2 set? It’s ugly. I have to read 19.5 TB of data to rebuild a single 1.5TB drive. How long does it take to read 19.5 TB of data? Even if the box could write data at max speed to the spare, which is 120MB/sec or 432 GB/hour, it would take 4 hours. But that doesn’t count the read time – and it takes 45 hours to read 19.5 TB of data @ 432 GB/hr – never mind the compute time needed to recalculate double parity on that entire set. That’s the problem.
Then there is the mechanicals. Quite literally, they use rubber bands to ‘dampen vibration’. Now, that’s certainly novel, but I wouldn’t trust my data to that. What’s next, duct tape? I bet this box goes 30 rads easily on the vibration test. They also mount all the drives in the same orientation, a sure-fire way to induce torque and stress. I would also not want to be a drive in the rearmost group of 15, given their depiction of fans – and they use fans, not blowers.
On average, they will lose 2 drives every year (4 per 100 per annum) and by year 5, it will probably rise to 6-8 per year, given their use of consumer-level drives. So, over the course of 5 years, you may have to replace roughly half the drive population. Did they mention that in their cost analysis?
Seagate themselves says this drive is not best adapted for array-based storage. Look at their website for this drive and see their recommended applications – PCs, desktop (not array) RAID and personal external storage. This is because of the duty cycle that these drives are built to. Is this array powered on and active all the time? Verbatim from the drive manual:
Normal I/O duty cycle for desktop personal computers. Operation at excessive I/O duty cycle may degrade product reliability.
The paper mentions expense. Sure, I am worried about expense too. I am worried about the biggest expense of all – humans. I need a server admin to manage and babysit this machine. How many of these can one admin manage? The answer is, much less than I would expect of a cloud storage array. Management is completely by hand. Compare to storage arrays which are intelligent and need no management beyond initial provisioning.
Just replicate data to protect it, right? To what degree of replication? Is 2X enough? I know several cloud providers that replicate 3X using arrays that are far more reliable than this, not made of nylon shims and rubber bands. So, now that petabyte becomes $351,000, and I also need triple the humans to manage triple the units. Plus, that is not counting any drive replacement cost over time, and for 3 units (135 drives) I will probably have to replace 60-70 drives over 5 years for another 9 grand or so. This also assumes there is no cost for a human to replace a drive, an ‘extraordinary event’.
But let’s be very generous and say they can build and operate/maintain this box for 5 years for $500 grand. That’s certainly cheaper than commercial arrays. But again, for me, the biggest issue is integrity. At petabyte scale, it’s a huge problem for any ATA technology. Just ask the guys at CERN, even with FC drives that didn’t do DIF it’s a huge problem.
Just the UCR errors alone at petabyte scale with this box are scary. This drive is only 1 in 10^14. 1*10^14 bits is 1.25*10^13 bytes, or 125 TB. This means if you spin and read a petabyte of this stuff, you get 8 errors which are uncorrectable. Murphy says those 8 errors are in the data you really needed.
Good news – cheap storage. Bad news – cheap storage. This thing is a nice home-built whitebox. It would do well to stay there. You wouldn’t run a serious cloud compute infrastructure on dirt-cheap whitebox servers; likewise, you shouldn’t run a serious cloud storage infrastructure on dirt-cheap whitebox storage.
Rob
From: cloud-c...@googlegroups.com [mailto:cloud-c...@googlegroups.com] On Behalf Of Jayarama Shenoy
Sent: Thursday, September 03, 2009 10:36 AM
To: cloud-c...@googlegroups.com
Subject: [ Cloud Computing ] Build your own storage - blog from BackBlaze
Very interesting post on making Petabytes on a budget for cloud storage.
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/#more-150
A question to those who'd think of doing something similar themselves - at what price point would there be an opportunity for a vendor to be a value add?
Backbalze's pod is not too different than Sun's Thumper (aka X4550). Some plumbing details are different, CPU & memory configs are different and so is Linux/JFS vs Solaris/ZFS. You can argue that in each case, Sun's options are superior, but that and support/warranty were not enough to offset the $8K vs $50K.
So what price point would've made sense? Any one from Backblaze on this list?
With Windows Live, you can organize, edit, and share your photos. Click here.
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.74/2339 - Release Date: 09/03/09 05:50:00
Windows Live: Make it easier for your friends to see what you’re up to on Facebook. Find out more.
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.74/2339 - Release Date: 09/03/09 05:50:00
Windows Live: Make it easier for your friends to see what you’re up to on Facebook. Find out more.
No virus found in this incoming message.
Warren Davidson
http://www.objectivity.com/cloud-computing/default.asp
Warren
From: Jacob Farmer <jfa...@cambridgecomputer.com>
To: cloud-c...@googlegroups.com
Sent: Friday, September 4, 2009 7:38:53 AM
Subject: [ Cloud Computing ] If not Backblaze, than what? --- changing the discussion a little
Jayarama Shenoy
To know *where* your data went bad (and perhaps a bit clue on why), things like DIF are needed. If you are in a structured data environment, there isn't a file system and DIF (or equivalent) is a must for integrity. Not the case in this instance.
Regards
Jay
p.s. We don't know Backblaze overall architecture, but a fair bet that they've baked in service reliability over this quasi-experimental box. Think carefully about their application. This box, while not a perfect storage system by a long shot, has limitations that are not terribly exposed by it. Last post, as I said.
Subject: [ Cloud Computing ] Re: Build your own storage - blog from BackBlaze
Windows Live: Keep your friends up to date with what you do online. Find out more.
Peglar, Robert
One last time as well. DIF is still essential regardless if the filesystem itself does payload checking. The problem is not that the given filesystem will have bad data integrity of its own(ed) data, but can munge _someone else’s_ data.
If the FS, say, orders a write to LUN 5 but it ends up on LUN 6, without DIF that write potentially destroyed someone else’s data, and your filesystem has no idea what just happened or who it happened to. Same thing can happen if the error is addressing (V->P) instead of LUN. Silent corruption of other people’s data.
The poor soul who comes along later expecting integral data on LUN 6 is in for, as they say, a world of hurt. Sure, your filesystem can detect and fix its own, but how about the other guy?
Thanks and this is the last post on this. Onward.
Version: 8.5.409 / Virus Database: 270.13.83/2352 - Release Date: 09/07/09
18:03:00
Jayarama Shenoy
Technically you could serve unstructured data over a file system (and many unified systems do, including from respected enterprise vendors) and that simplifies snapshots etc - but many people (incl me) are not a particular fans of SAN data delivered over a file server (think latency).
So it would be commonly possible to have situations where structured and unstructured data co-exist, and this failure mode is important.
Jay
Subject: [ Cloud Computing ] Re: Build your own storage - blog from BackBlaze
Zero0n3
or storage systems.
With that said, what if their app verifies the data during any
operation?
So poor soul B who's data just got written over without any ack. of it
happening now wants to grab his stored data.
Application requests Data for poorSoul B. application grabs it from
Node 1 and node 3. checks to see if it is the same... it now isn't as
node 3 was the one that was silently written over.
So now it needs to decide _what_ to do. it now goes out to node 8 and
checks to see if node 1 or node 3 matches. (lets assume node 1 and 3
were initially chosen because of latency)
so it quickly checks node 8 == 1 ? and 8 == 3 ? it sees that node 8
and 1 are a match, so it uses node 1 to retrieve said data.
the application also now knows that something is wrong with node 3.
and logs any and all data it finds out about it.
What I am simply saying is that without seeing ANY of their code,
there is no way we can tell how secure the data is, how much error
checking they have, what type of hardware failure issues they have,
etc.
Hell, the ONLY thing they really told us was how to build the storage
node. They didn't mention if they had compute nodes, or checksum
nodes, etc etc.
you are making assumptions based on 1 piece of a complicated puzzle.
Google did the same thing with GoogleFS how many years ago? they
designed a entire system on the premise that they were working with
commodity hardware. This company is no different, just doing it with
a home brew, commodity built, storage node.
I don't mean to be rude, but looking at their team ( https://www.backblaze.com/team.html
) I have a feeling they have thought of and came up with solutions to
everything that has been discussed in this thread.
>
> >>> If you know for certain your data is corrupted
>
> And therein lies the rub.
>
> Lack of DIF at petabyte-scale does indeed damn it for integrity. You
> are correct, there are even more checks beyond T10 DIF that could be
> done, but in terms of the actual SCSI CDB, DIF covers the 3 inputs that
> the initiator supplies; payload, address/length (and its subsequent V->P
> translation), and LUN.
>
> As for market pricing, what value do you place on your data? It's
> really an exercise in risk management. Look at it this way; how much
> would it cost you to recreate/restore that petabyte? What is the time
> value of not having access to that petabyte? If a petabyte of data is
> only worth $117,000 to you, that's one thing. For most companies,
> though, their petabyte is worth far, far more than that.
>
>
> Petabytes, sure. Petabytes of integral data, nope. No DIF == no
> integrity.
>
>
> Can't argue with that. But what data? The data that you think you
> wrote, at the address you think you wrote it to, at the LUN or ID you
> think you sent it to? How do you know?
>
> But let's look a little deeper here. They use two power supplies, 760W
> rated each. It looks like the unit must have both supplies up to
> function, as they have subdivided duties. Lose one supply, half your
> drives disappear, and if you lose half the drives in the scheme they use
> (see below) your box is dead. Not very good if you have a halfway
> decent SLA. But perhaps they are assuming cloud vendors will not have
> Then there are disk rebuilds and sparing. No hot spares configured, or
> at least described. Looks like they do three sets of double parity
> 13+2. So all drives are holding necessary information. No mention of
> how one spares out a drive, so you have to assume entire unit downtime
> to replace a drives and allow a rebuild to begin. Speaking of which,
> rebuild a single 1.5TB drive. How long does it take to read 19.5 TB of
> data? Even if the box could write data at max speed to the spare, which
> is 120MB/sec or 432 GB/hour, it would take 4 hours. But that doesn't
> 432 GB/hr - never mind the compute time needed to recalculate double
>
> Then there is the mechanicals. Quite literally, they use rubber bands
> to 'dampen vibration'. Now, that's certainly novel, but I wouldn't
> trust my data to that. What's next, duct tape? I bet this box goes 30
> rads easily on the vibration test. They also mount all the drives in
> the same orientation, a sure-fire way to induce torque and stress. I
> would also not want to be a drive in the rearmost group of 15, given
> On average, they will lose 2 drives every year (4 per 100 per annum) and
> by year 5, it will probably rise to 6-8 per year, given their use of
> consumer-level drives. So, over the course of 5 years, you may have to
> replace roughly half the drive population. Did they mention that in
> their cost analysis?
>
> Seagate themselves says this drive is not best adapted for array-based
> storage. Look at their website for this drive and see their recommended
> to. Is this array powered on and active all the time? Verbatim from
> the drive manual:
>
> Normal I/O duty cycle for desktop personal computers. Operation at
> excessive I/O duty cycle may degrade product reliability.
>
> The paper mentions expense. Sure, I am worried about expense too. I am
> admin to manage and babysit this ...
>
> read more »
Peglar, Robert
It's funny; on one hand, many cry out for efficiency and with the other hand, design solutions which are far less than optimal in order to justify the use of certain piece parts. Several decades ago, this was called "Rube Goldberg" design.
T10 DIF is not only a good idea, but a standard. It exists for a reason. It takes about 50 lines of code on both initiator and target to implement. For me, it makes sense. If it doesn't to you, so be it.
Rob
Jayarama Shenoy
On one hand GFS & it's various knock-offs contain appalling practices when you think in terms of efficiency (power, space, what have you). And then, they find that their data centers burn too much power. Duh, what did you expect?
I realize that I'm talking of a 5+ year old architecture and think (or hope) that saner minds are now prevailing.
Backblaze in particular, from what they've described (very likely there's stuff they didnt say) - has no choice but to take the same very brute force way out - except it is 2009 and not 1999.
> Subject: [ Cloud Computing ] Re: Build your own storage - blog from BackBlaze
> From: Robert...@xiotech.com
> To: cloud-c...@googlegroups.com
>
>