info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
any good books about the (art?, economics and) science of optimizing IO performance? ...
1 view
Skip to first unread message
Albretch Mueller
May 23, 2022, 5:00:05 AM5/23/22
to
there is quite a bit of partial and somewhat obviously misconstrued
("buy my great sh!t") information out there about how to combine RAM,
NVMe, SSD, SATA and RAID in order to optimize IO performance. You also
hear about ZFS licensing and performance issues in Linux.
I'd wish I could find a book explaining IO performance right from the
physics of it to the OS system level algorithms to optimize transfer
rates in a "nullius in verba", "and-here-is-how-you-test-it" kind of
way.
Any comprehensive readings regarding such matters you would share?
lbrtchx
("buy my great sh!t") information out there about how to combine RAM,
NVMe, SSD, SATA and RAID in order to optimize IO performance. You also
hear about ZFS licensing and performance issues in Linux.
I'd wish I could find a book explaining IO performance right from the
physics of it to the OS system level algorithms to optimize transfer
rates in a "nullius in verba", "and-here-is-how-you-test-it" kind of
way.
Any comprehensive readings regarding such matters you would share?
lbrtchx
Datakanja de Bruyn
May 23, 2022, 5:40:06 AM5/23/22
to
... somewhat surprised ...
methinks, if such a guide would be published, it is likely outdated.
Things are moving too fast.
I myself do use ZFS, NVMe, SATA and RAID on my SOHO machine, and the
complaints about licensing seem to be outdated by now. But there are
people busy with performance on the zfs-discuss mail list
<https://zfsonlinux.topicbox.com/groups/zfs-discuss>
Hope this helps.
--
Liebe ist ...
Datakanja
methinks, if such a guide would be published, it is likely outdated.
Things are moving too fast.
I myself do use ZFS, NVMe, SATA and RAID on my SOHO machine, and the
complaints about licensing seem to be outdated by now. But there are
people busy with performance on the zfs-discuss mail list
<https://zfsonlinux.topicbox.com/groups/zfs-discuss>
Hope this helps.
Liebe ist ...
Datakanja
John Conover
May 23, 2022, 6:10:05 AM5/23/22
to
gate per bit, to do the Boolean arithmetic is only 6 CMS transistors
per bit, and the through put is about 12 pico seconds, (12 trillionth
of a second,) using 12 nano-meter lithography, ie, the CPU execution
time is not the issue.
In the PC, (a Von Neuman architecture,) the buss is the issue, as has
been known since the early 1940's, (it was named the Von Neuman
bottleneck, by the early 1950's on the ENIAC.) The PC buss structure
is a remarkable accomplishment, but the PC is a terminal, (and many
theorist argue it is not a computer, any more than a pair of light
switches controlled by traveler wires, as in an EXOR gate, a half
adder in Boolean logic, is a computer, with a buss function via the
traveler wires-but that's another issue.)
So, in the PC, (the Intel 8008 was designed for cash register
applications, BTW,) the buss structure, (both memory and IO,) is the
speed, (and speed power product,) issue, which can be enhanced by
data/instruction caching, speculative execution, (particularly if the
caches are not dumped as in the Spectre security issues,) etc.
Good luck on testing things. The size of the caches can increase buss
data/instruction throughput dramatically-up to a point. Multiple
processes can, (because of the latencies, through possibly multiple
buss structures,) slow things to a crawl-the slowest one controls the
speed.
Your message kind of implied networked disk IO, (think about all the
serial caches in the chain.) The way blade super computers address the
problem is to write, NOT to the disk, but to the on disk cache memory,
(ie semiconductor memory physically on the disk drive-there is a
latency for the spinning media to keep up, so powering down MUST flush
the cache-good luck finding disk that will do that.)
To pursue further, you might want to look at the way Luster does it,
(Luster is the distributed file system under the parallelization used
in many super computers using Linux-mpich, etc., managing the
parallelization/concurrency issues.)
John
--
John Conover, con...@panix.com, http://www.johncon.com/
IL Ka
May 23, 2022, 7:30:06 AM5/23/22
to
I am not aware of such books, but to develop deep understanding one probably needs long path.
1. How memory/cache/hdd (ssd/nvme not covered) works: https://www.amazon.com/Memory-Systems-Cache-DRAM-Disk-ebook/dp/B00BXETR06
2. General understanding of PC architecture: https://www.mindshare.com/Books/Titles/ISA_System_Architecture_(3rd_Edition)
4. How Linux kernel works (VFS, Ext3, cache, etc): https://www.amazon.com/Understanding-Linux-Kernel-Third-Daniel/dp/0596005652
If you need short path, try any fio tutorial (``fio`` is **nix tool to measure IO performance).
Albretch Mueller
May 24, 2022, 7:10:06 PM5/24/22
to
On 5/23/22, John Conover <con...@panix.com> wrote:
> . . . the problem is to write, NOT to the disk, but to the on disk cache memory . . .
and I could imagine depending on the kinds of data loads one expects
there are best practices about how to optimally make RAM, SSDs and
RAID configurations of mechanically spinning disks play together.
That is what I am looking for, which I thought should be cultured in
these "big data" times
thank you,
lbrtchx
> . . . the problem is to write, NOT to the disk, but to the on disk cache memory . . .
and I could imagine depending on the kinds of data loads one expects
there are best practices about how to optimally make RAM, SSDs and
RAID configurations of mechanically spinning disks play together.
That is what I am looking for, which I thought should be cultured in
these "big data" times
thank you,
lbrtchx
0 new messages