Re: Not ECC RAM risk with Redis as main database

[Salvatore Sanfilippo]

Hello Didier,

I'm very addicted to memory testing and memory errors, simply because
with Redis I lost N nights trying to fix problems, from multiple
users, that never existed, but was instead broken RAM.

Now, there are three levels of safeness:

1) You have broken RAM.
2) You have sane non ECC RAM.
3) You have sane ECC RAM.

With 1 you are in BIG troubles, but this is easy to check. Use
memtest86, or, if you can't reboot the machine, use redis-server
--test-memory that worked reliably many times.

With 2 you are safe, but there is always the possibility of a bit
flipped by a cosmic ray or alike. It's a remote possibility, but it
happens.

With 3 you are safer.

So I think that if you don't have ECC RAM, test your memory for a few
hours before installing Redis. If the test is ok, you are ok.
If your data is not super important like money transactions or alike,
I would go for "2" if the price for ECC is noticeable bigger.
Otherwise if you have a mission critical thing here, go for ECC.

Now about the technology itself, Redis does not do any run-time
checksumming or alike, so if RAM is broken or a bit flips, nobody will
notice unless it happens on a pointer causing a likely crash.

Cheers,
Salvatore

On Mon, Nov 19, 2012 at 7:19 PM, didier rano <didie...@gmail.com> wrote:
> I need to choose ECC or not for my hosting. And, I am using Redis as main
> database.
>
> What is the risk to choose to not use ECC memory ? How to mitigate it with
> Redis itself ? Without using an other database to resynchronize Redis
> database ?
>
> Sorry for useless issue https://github.com/antirez/redis/issues/774 !
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/tqiN1Vx8ouQJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.



--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[didier rano]

Thanks Salvatore for your quick answer !

In my main database, I have a reference data (users profiles, and their configuration). These data have to be correct forever. But, it is a just small part of all my data. Then, I need Redis performance for it, but integrity is very important too

One solution is obvious: My reference data are stored in a no Redis database (NoRedis versus NoSQL ;)), and sometimes a job resynchronizes these data in Redis.

The second solution: Use Redis to keep integrity forever.

I think that I am not alone to use Redis as a main database. And, it could be useful to share ideas about the second solution.

[Salvatore Sanfilippo]

Hi Didier,

in our conversation perhaps one thing is not given per granted, that
is, on-disk data stores have the same problem with broken memory, it's
not something special with Redis. If ram is broken, broken data can be
stored in any kind of DB. So basically you can reason about error
correction in the same line as with other DBs :-)

Salvatore

> https://groups.google.com/d/msg/redis-db/-/-B8B3bCMedEJ.

[Jeremy Zawodny]

The InnoDB storage engine inside of MySQL/XtraDB/MariaDB has page checksums to prevent corrupted data (due to bad memory or disk) from going undetected.  I've seem them trigger myself on a few occasions.  Usually a few days later, that host would start experiencing lock-ups or would toss a disk out of the RAID array.

I also lobbied for a long time to get checksums build into the MySQL replication protocol to detect bit-flips on the wire, which I've seen at least twice in the last 10 years.

Bottom line: you can't REALLY trust the disks, RAM, or network without some extra work.

Jeremy

[Josiah Carlson]

On Mon, Nov 19, 2012 at 2:32 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:
> Bottom line: you can't REALLY trust the disks, RAM, or network without some
> extra work.

This times 1000.

I've had memtest86 succeed one week, fail the next. I've had memory
that works 99% of the time, unless it is on the 4th+ pass of memtest86
(which is why I typically run it for 2 days for new memory).

If you are looking to test your disks (and fix them on occasion), you
may want to look into Spinrite. It has discovered failing disks for
me, though usually I discard them upon discovery.

Also, on that list of things you can't trust is the processor. There
was an article that showed that desktop/server processors have
occasional random bit flips that are occur under load. Kind of scary,
really. Sadly, I have no potential solution to this issue.

- Josiah

[Salvatore Sanfilippo]

That's cool Jeremy, this is a great InnoDB feature indeed!

We can't do the same easily (at page level) but what about this?

https://github.com/antirez/redis/commit/eb2a5d6333739bf4c6f3ddd1aa3cb74c8089f5fd

Performance hit is non-measurable but this can provide a valuable tool
for users and developers.

Cheers,
Salvatore

[didier rano]

Checksum looks like a good idea to catch some data corruption. But, how to avoid it ?

Why not have two copies of a key, and check for each access, if one copy has not a good value, then correct it. I know that performance could be bad, but it could be a solution for some keys only in a Redis database: "SETINTEGRITY mykey' ?

[Jeremy Zawodny]

That seems like a useful feature as long as people realize that it increases the odds of finding memory errors, but will likely miss some, since it's only checking at allocation time.  But that's better than doing nothing at all.  And having it as an output field in INFO is great because we can monitor and alert on it.

Jeremy

[Salvatore Sanfilippo]

That's the idea Jeremy, but indeed I'm not 100% satisfied because this
is too a long term and random process... so I'm experimenting with
other possibilities.

For instance:

1) Do a fast memory test at startup. At least on Linux we could use
the proc filesystem to see how free memory there is around in order to
avoid allocating too much.
2) While running, test a bigger memory piece when given checkpoints
are reached. For instance if used_memory is > 0 (so at startup) we
fast-test 64 MB. When used_memory > 64 MB we allocate more 64 MB and
test, and so forth. This probably may result into a bigger percentage
of probabilities to discover real memory issues. At the same time
we'll use additional RSS memory that will game a bit the fragmentation
ratio reporting. Maybe instead of 64 MB it is possible to select a
smaller amount of memory so that this will be a non issue.
3) The solution I probably prefer: on crash, consult /proc/self/smap,
and test all the regions, with "rw" permission in a way that data is
retained after the test.

If "3" is technically possible, it is probably the best we can do...
otherwise the current system is perhaps the most practical as it has
no impact and no OS-specific code. Unfortunately I don't have a system
to test it in the real world, and unfortunately I can't find a Linux
kernel module that simulates a memory error, or an emulator like
Virtualbox that can do this... but probably I can easily simulate a
memory error accessing /proc/self/mem so this could be a good start
indeed.

Cheers,
Salvatore

[Salvatore Sanfilippo]

Solution "3" implemented here ->

https://github.com/antirez/redis/compare/memtest-on-crash

On Wed, Nov 21, 2012 at 11:16 AM, Salvatore Sanfilippo

[didier rano]

I read this post: http://news.cnet.com/8301-30685_3-10370026-264.html

"4,000 errors per year
That may not sound like a high fraction, but bear these factors in mind, too: each memory module experienced an average of nearly 4,000 correctible errors per year..."

Not so much errors, but it could corrupt reference data.

I think that those errors cannot be found with a memory check before that data are corrupted. Then, I cannot trust my hard disk and not my RAM ! 

SETINTEGRITY or something equivalent could be a solution for a small part of my keys !

[Felix Gallo]

You could implement a baby version of SETINTEGRITY today by running BITCOUNT on your keys and storing that value separately; that would provide a solution which would give you detection of any odd number of bit flips, although it wouldn't detect most even number bit flips or give you correction ability.  Since most bit flips will be odd that might be ok with you.

If you wanted double bit detection and single bit correction, I'd suggest you run a Hamming(8,4) encoder on your data  before it gets into redis and decode it on the way out.  Running it in lua or by hacking together BITOP commands would probably be nonperformant unless Salvatore includes bit32 in the redis lua distribution.

F.

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/Ij1LQ427kUUJ.

[M. Edward (Ed) Borasky]

Pardon me for getting all religious here, but IMHO detecting and
disabling "bad RAM" is a service that the *operating system* should
provide to both applications programmers like the Redis team and
equipment operators like Amazon, Rackspace, Google, Facebook, ... I
think it's great that the Redis team wants to provide its users with a
more reliable product in the face of imperfect RAM, but at the same
time I wonder whether something of this ilk is already in Linux (and
BSD) and, if not, why aren't the kernel developers doing it? It's been
a few years since I went digging into the Linux kernel, so if it's
there already, I wouldn't know. But I think it should be.

--
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?

[Salvatore Sanfilippo]

The patch aims at discarding bug reports about crashes if there is bad
RAM mounted in the system that generated the crash.

Everything else like making operations safer for people is outside the
scope of the patch implementing behavior "3".

Cheers,
Salvatore

[Felix Gallo]

Well, it's even more important than the kernel developers -- many computer users want to be defended against these cases even if it is rare that they affect running applications.  That is why, for a fee, DRAM manufacturers build in ECC RAM, which can not only detect error conditions, but correct the occasional cosmic ray bit flip...and so we're back full circle to the original question, which is 'is it best to use ECC RAM for database machines?'

Short summary of the thread: yes, but if you are super cheap and/or using virtualized hosts which are often run on ugly cheap commodity metal (e.g. Amazon), or if your data is not that important to you, then you can get part of the benefit of ECC RAM with software tricks.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

[M. Edward (Ed) Borasky]

On Wed, Nov 21, 2012 at 11:04 AM, Felix Gallo <felix...@gmail.com> wrote:
> Well, it's even more important than the kernel developers -- many computer
> users want to be defended against these cases even if it is rare that they
> affect running applications. That is why, for a fee, DRAM manufacturers
> build in ECC RAM, which can not only detect error conditions, but correct
> the occasional cosmic ray bit flip...and so we're back full circle to the
> original question, which is 'is it best to use ECC RAM for database
> machines?'
>
> Short summary of the thread: yes, but if you are super cheap and/or using
> virtualized hosts which are often run on ugly cheap commodity metal (e.g.
> Amazon), or if your data is not that important to you, then you can get part
> of the benefit of ECC RAM with software tricks.

I didn't answer the original question, but had I done so, I would have
said, 'Yes, by all means use ECC RAM (and hardware RAID, and other
high-availability options) if the engineering economics of the
situation call for it." ;-)

Meanwhile, back at the Linux kernel, it turns out that since 2.6.26
there has been an in-kernel memory test *available*. I haven't been
able to find coherent documentation on how to use it, though, or
whether the associated modules are configured by default on the
distros we regularly deploy to - Ubuntu LTS, RHEL/CentOS, Amazon's
default, etc. And I don't think running a kernel or application based
RAM test from a guest in a hypervisor makes much sense - again, that's
a service the *hypervisor* should be providing IMHO.