[signin] help planning DB changes

[Jared Hirsch]

[This is a slightly modified version of an email I sent to the private list
two days ago. Seanmonstar rightly pointed out that there was no good reason
to keep it private.]

Identity wants a more highly available DB option than our current single
write master MySQL setup. Has to work in AWS, has to fail gracefully during
region outages.

Do any experts on this list have time to help in the next few weeks?

We'll need help assessing our requirements and choosing a few potential
options. After that, Identity devs plan to prototype the best 2-3 options,
bang on them a bit, and choose a winner. (Help's welcome here as well, but
I don't really expect it.) Let me know if you're interested in offering
technical guidance, and we can get discussion started in a separate thread
and some etherpads. The more, the merrier.

Some salient facts:
* we're moving to AWS + want to build something that can handle region
failures gracefully
* we store a very small amount of data per user (order of KB)
* we have a very simple schema[1], would be easy to denormalize
* we can live with eventual consistency (it's just staging new users,
logging existing users into sites, adding/removing emails)
* we have 50k logins/day currently, aiming for 2.5 million/day (and
hoping for 20x that ;-)
* not sure on the mix of reads/writes
* as a whole, Persona is CPU-bound, not DB-bound
* diagrams of current (non-AWS, soon to change) network architecture:
[2][3]

[1] https://github.com/mozilla/browserid/blob/dev/lib/db/mysql.js#L10-L30
[2] http://lloyd.io/persona-architectural-changes
[3] https://intranet.mozilla.org/images/2/2c/Browserid_architecture.png
(from https://intranet.mozilla.org/Services/Ops/BrowserID#Architecture)

Cheers

Jared

[Chris Karlof]

I've been thinking about scaling a bit recently, and came across this talk on scaling Pinterest by their engineers:

http://www.infoq.com/presentations/Pinterest

Notes here:

http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html

My takeaway:

1. Keep everything really, really simple
2. Use only technologies that are mature or simple, preferably both
3. At scale, everything eventually fails, albeit in its own special way. When something fails, you want a large community (i.e., lots of docs, people to turn to, support companies, etc) and to be able to figure out what the hell is going on.
4. Clustering sucks. Too much magic and too hard to reason about, especially when it fails. Use sharding. It's dead simple.

Pinterest uses a "schema-less" data model built on MySQL, Redis, and Memcache. They use MySQL as a KV store, build indices into separate tables, denormalize the crap out of things, stand a wall of cache in front of everything, and reap the rewards of MySQL's reliability, predictability, maturity, and community and Redis's/Memcache's simplicity.

Here is a similar argument from 2009:

http://backchannel.org/blog/friendfeed-schemaless-mysql

I'd strongly encourage consideration of this approach, especially given how minimal our schema requirements are.

-chris

> _______________________________________________
> dev-identity mailing list
> dev-id...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-identity

[Jared Hirsch]

Thanks for the linkage and feedback, Chris. We'll definitely consider
elements of that approach.

I think it's pretty likely we'll move to a fully-denormalized backend; we
currently already have one, the JSON test db used by our unit tests, and
it's indeed really simple. Denormalizing frees us up to consider NoSQL
stores, or even SSD-backed file storage.

Personally, I'm hoping we can avoid adding a caching layer, as that
significantly increases the ops work in terms of monitoring, management,
and quirks/bugs to uncover--but it's an option we should consider. As we're
CPU-bound overall, I am hoping it's not necessary.

I'll send out a kickoff email in a bit, focused on beginning by carefully
defining our requirements. I'll ping the interested folks (including you)
and cc the public list.

Cheers,

Jared

[Chris Karlof]

On Apr 22, 2013, at 4:50 PM, Jared Hirsch <6a...@mozilla.com> wrote:

> Thanks for the linkage and feedback, Chris. We'll definitely consider elements of that approach.
>
> I think it's pretty likely we'll move to a fully-denormalized backend; we currently already have one, the JSON test db used by our unit tests, and it's indeed really simple. Denormalizing frees us up to consider NoSQL stores, or even SSD-backed file storage.
>
> Personally, I'm hoping we can avoid adding a caching layer, as that significantly increases the ops work in terms of monitoring, management, and quirks/bugs to uncover--but it's an option we should consider. As we're CPU-bound overall, I am hoping it's not necessary.

Main takeaway was to strip out all complexity, so if a caching layer isn't needed, then there's no reason to have it! However, denormalization can lead to an increase in queries, which is where the caching can help. However, if Persona is CPU bound, maybe the extra queries, if any, aren't too bad.

One main culprit of complexity in their experience were automagick cluster management algorithms (e.g., in Cassandra and Riak). They argue (for now) that the algorithms are too complicated to trust and the failure modes are bad.

-chris

[Lloyd Hilaiel]

On Apr 22, 2013, at 6:07 PM, Chris Karlof <cka...@mozilla.com> wrote:

> On Apr 22, 2013, at 4:50 PM, Jared Hirsch <6a...@mozilla.com> wrote:
>
>> Thanks for the linkage and feedback, Chris. We'll definitely consider elements of that approach.
>>
>> I think it's pretty likely we'll move to a fully-denormalized backend; we currently already have one, the JSON test db used by our unit tests, and it's indeed really simple. Denormalizing frees us up to consider NoSQL stores, or even SSD-backed file storage.
>>
>> Personally, I'm hoping we can avoid adding a caching layer, as that significantly increases the ops work in terms of monitoring, management, and quirks/bugs to uncover--but it's an option we should consider. As we're CPU-bound overall, I am hoping it's not necessary.
>
> Main takeaway was to strip out all complexity, so if a caching layer isn't needed, then there's no reason to have it! However, denormalization can lead to an increase in queries, which is where the caching can help. However, if Persona is CPU bound, maybe the extra queries, if any, aren't too bad.
>
> One main culprit of complexity in their experience were automagick cluster management algorithms (e.g., in Cassandra and Riak). They argue (for now) that the algorithms are too complicated to trust and the failure modes are bad.

I think that's important insight. I'm both enamored by the feature sets of these two options, and petrified by the notion of dealing with failure modes that we don't understand in production - the inherent risk that comes with deploying the new hotness. Downtime SUCKS.

(this coming from the guy who advocated we build persona on node.js. ironic, right?)

Our challenge remains the same: What solution can we implement that is resilient to DC loss. The answer could very well end up being stick to MySQL, and optimize master promotion so that in the worst case (DC failure where the write master lives), we incur < 10 minutes of failed signups…

I think jared's job (and now dcoates too?!) is to distill our solutions down to four or five options spread across the spectrum (optimal but risky, to pretty good but safe).

?

lloyd