1TB memcached

[MikeG.]

Hi,

I'm starting a project in which I would like to have the entire DB in
cache.
The reason is that my transactions are reading large amount of data
from the DB to generate
a deliverable blob.

I have no concurrency issues. I will always have around 5 users max
concurrently and most
of the time it will be single user. SO I look at the context as
virtually single user.

To avoid this massive DB reading (of large count of large chunks) I
would like to have it
permanently in memory.
When any modification of a record happened it is not being written to
the DB until the
big blob final product is delivered and the local system goes idle.
Only then modifications are written to the DB. Upon successful
completion of DB update the local system sends a
message to the recipient of the product to inform it that the DB is
now in sync with the product
at hand and it can be consumed.

Now, to hold 1TB in memory I need a cluster and not a small one. I
have decided to use
memory mapped files such that my RAM is virtual memory. Easy to get
large file system of
several TBs.

My question - is there any limit memcached has as far as cache size?

Also, memcached (the C implementation) has a 1MB record size limit. 1)
What's the reason
for that? 2) Can it be changed (with a hacked private version) 3) Does
Jmemcached has the
same limit?

Thanks,
-Michael

[Joseph Engo]

If you need larger than 1MB objects and you are only serving these objects to 5 users at a time.  It really sounds like memcache is the wrong tool for your project.  You might want to look into something like MongoDB which has a larger object limit of 4MB.  There are a number of key value stores that can handle even larger size objects.

Could you explain a little more on what type of binary data you are manipulating ?

[MikeG.]

First, thanks for taking the time.

Now, MongoDB is a cache engine? I think it's a DB?

I need a cache engine that can handle (some) objects bigger then 1MB.
I don't have binary data. All I have is ascii elements. My final
product is an aggregation
of meta data of many binary code pieces.

Just for context - my company has many products while each may have
several hardware
versions. Each version has it's own binary image.
If you have one of our products and you need to upgrade the software
you get the binary from on
place (I have nothing to do with it) and a build meta data from
another place. It's a scheme to have
only the legitimate users able to upgrade.

My application builds or assembles all the various products builds
meta information and download it
to the distribution cluster.

Currently my application takes forever because everything needs to be
read from the DB.
Caching it will do wonders. I'm sure about that. Profiling my
application I found that the total time
spent on DB reading is hugh.

Do you see any problem using memcached?

Thanks,
-Michael

[Dustin]

On Sep 20, 8:25 pm, "MikeG." <miki...@gmail.com> wrote:

> The reason is that my transactions are reading large amount of data
> from the DB to generate
> a deliverable blob.

All of them all the time? It may be possible to optimize your
source for whatever type of efficiency you're seeking (latency?
throughput?).

> Now, to hold 1TB in memory I need a cluster and not a small one.

It only takes 16 64GB nodes to give you 1 TB of RAM. Of course, not
all will be usable. It depends on how many objects you plan to have
loaded and other overhead.

> Also, memcached (the C implementation) has a 1MB record size limit. 1)
> What's the reason for that?

The limit is configurable, but defaults to 1MB. If you are caching
items larger than 1MB, latency is probably not your issue, so
memcached may not provide the most benefit. It'll do it, of course,
but as you get larger, it starts to be less beneficial than, say,
varnish.

> 2) Can it be changed (with a hacked private version)

-I

> 3) Does Jmemcached has the same limit?

You should try your design at a smaller scale and see what
components benefit you.

[Joseph Engo]

Yes, MongoDB is a database.

You mentioned blobs so I figured you were dealing with binary objects.

The problem with your use case is objects larger then 1MB.  Thats really not what Memcached is intended to be used for.  You could split the objects into smaller chunks.  But keep in mind, no object is guaranteed to be in the cache.  So if 1 chunk of your large object is missing you need to pull the entire object back from your datastore and split it back into chunks again.  Sure, its very rare that it would happen but it does.

What happens if you need to reboot your server for patches or you have a crash.  You will need to warm up the cache and that could take a while depending on how you assemble the objects.

It really sounds to be that you should do optimizations on your DB layer first before just throwing a cache at it. :)

[MikeG.]

The cycle happened at least once a day.
Several people need to do something to the data. (In sequence). Once
the last one completes
the assembly of the final deliverable blob is starting. Lots of data
is read from the DB and it takes
long time.

If I'll have this magic machine with 1TB RAM available and my entire
DB was in RAM - the assembly
of the deliverable would be 10 times faster.

I currently have hibernate in front of the DB and the bottom line is
big latency.
I'm considering hibernate-memcached to get my DB cached.

Thanks,
-Michael

[MikeG.]

Sorry I used blob to describe the final deliverable bundle not DB
blob.

As far as having partial objects in memory - this should never
happened based on my
intended design.
I would like to have the entire DB permanently in cache. Since I don't
have that much RAM on one machine
I plan to use virtual memory or memory mapped files. In this case my
only limit is as big as
my hard drive I'll get.

-Michael

[Les Mikesell]

On 9/21/10 1:26 AM, MikeG. wrote:
> Sorry I used blob to describe the final deliverable bundle not DB
> blob.
>
> As far as having partial objects in memory - this should never
> happened based on my
> intended design.
> I would like to have the entire DB permanently in cache. Since I don't
> have that much RAM on one machine
> I plan to use virtual memory or memory mapped files. In this case my
> only limit is as big as
> my hard drive I'll get.

Why do you expect the file activity of virtual memory to be any faster than a
traditional database designed to use files efficiently? I'd expect the
opposite. You should generally assume that the disk head is going to be halfway
across the disk from the data you want and add up the seek time it will take to
get there. On the other hand, using real memory across several machines is very
fast.

--
Les Mikesell
lesmi...@gmail.com

[MikeG.]

You are right Les.

I was afraid that such reply may come.....
If file i/o was working fast everybody would do that rather then
clustering
with lots of RAM.

Thanks,
-Michael

>     lesmikes...@gmail.com

[ligerdave]

MongoDB is actually "cached" db, meaning that, most of its records are
in memory.

I think there is also a memcached and DB hybrid which comes w/ a
persistent option. i think it's called memcachedDB, which runs a in-
memory db(like mongodb). this shares most of common api w/ memcached
so you dont have to change code very much

On Sep 21, 2:11 am, Joseph Engo <dev.toas...@gmail.com> wrote:
> Yes, MongoDB is a database.
>
> You mentioned blobs so I figured you were dealing with binary objects.
>
> The problem with your use case is objects larger then 1MB.  Thats really not
> what Memcached is intended to be used for.  You could split the objects into
> smaller chunks.  But keep in mind, no object is guaranteed to be in the
> cache.  So if 1 chunk of your large object is missing you need to pull the
> entire object back from your datastore and split it back into chunks again.
>  Sure, its very rare that it would happen but it does.
>
> What happens if you need to reboot your server for patches or you have a
> crash.  You will need to warm up the cache and that could take a while
> depending on how you assemble the objects.
>
> It really sounds to be that you should do optimizations on your DB layer
> first before just throwing a cache at it. :)
>

[MikeG.]

Thanks,

Sounds good. Unfortunately my situation is unique.
I'm trying to expedite a "meta production" application. This is an
application that
create content for a production application. The creation is slow
because there is
a lot of big records that need to be updated and fetched.

Looks like I can't avoid a cluster cache because I need 1TB and
growing DB.
This means at least 16 x 64GB machines. There is no justification to
have such cluster
for non-production engine.

I'm back to look at the architecture to get some more speed.

[Matt Ingenthron]

On 9/22/10 6:12 AM, ligerdave wrote:
> MongoDB is actually "cached" db, meaning that, most of its records are
> in memory.
>
> I think there is also a memcached and DB hybrid which comes w/ a
> persistent option. i think it's called memcachedDB, which runs a in-
> memory db(like mongodb). this shares most of common api w/ memcached
> so you dont have to change code very much

membase is compatible with memcached protocol, has a 20MByte default
object size limit, lets you define memory and disk usage across nodes in
different "buckets".

memcacheDB is challenging to deploy for a few reasons, one of which is
that the topology is fixed at deployment time.

- Matt

p.s.: full disclosure: I'm one of the membase guys

[MikeG.]

Thanks Matt!
This is great information.

BTY - I appreciate anybody who takes the time to read and contribute.
Sometimes just throwing in a name of a product is enough. We all know
to Google.
It is great help.
Promise to do the same.... :)

[Les Mikesell]

Does anyone know how these would compare to 'riak', a distributed
database that can do redundancy with some fault tolerance and knows how
to rebalance the storage across nodes when they are added or removed?
(Other than the different client interface...).

--
Les Mikesell
lesmi...@gmail.com

[Matt Ingenthron]

This is a very detailed question, but...

Without going too much into advocacy (I'd defer you to the membase
list/site), membase does have redundancy, fault tolerance and can
rebalance when nodes are added and removed. The interface to membase is
memcached protocol. It does so by making sure there is an authoritative
place for any given piece of data at any given point in time. That
doesn't mean data's not replicated or persisted, just that there are
rules about the state changes for a given piece of data based on vbucket
hashing and a shared configuration.

This was actually inspired by similar concepts that in memcached's
codebase up through the early 1.2.x, but not in use in anywhere that I'm
familiar with.

riak is more designed around eventually consistent and lots of tuning
W+R>N, meaning that it is designed more to always take writes and deal
with consistency for reads by doing multiple reads. This is different
than memcached in that memcached expects one and only one location for a
given piece of data with a given topology. If the topology changes
(node failures, additions), things like consistent hashing dictate a new
place, but there aren't multiple places to write to.

Any time you accept concurrent writes in more than one place, you have
to deal with conflict resolution. In some cases this means dealing with
it at the application level.

I don't know it well, but it's my understanding that MemcacheDB is
really just memcached with disk (BDB, IIRC) in place of memory on the
back end. This has been done a few different times and in a few
different ways. Topology changes are the killers here. Consistent
hashing can't really help you deal with changes in this kind of deployment.

- Matt

[ligerdave]

what about map-reduce? chop the job to much smaller sub-jobs and then
have someone(another server or PC) else to process.
i assume your app probably loads data from a DB, and wont load the 1TB
data and then process it entirely.

doing something kinda like pagination? fetch a chunk and have someone
else to do the calculation? then reduce, combine the results and map
again if necessary?

as for the 1mb limit, convert the object to byte(serialization) and
split it to pieces(can come up w/ a data structure)? still much
quicker than writing to disk.
accessing memory in general is 250,000 times quicker than doing the
same against the disk.

[MikeG.]

Great collection of ideas.

My situation is rather unique. The application is pre production -
means it prepares content
for a production platform. The problem is that it is slow. You would
say - so what, this is pre production.
The issue is that it's done at least once a day sometimes 2-3 time.

Every final build MUST go through QA. If they have to wait several
hours to get to QA something that
may need to be put to production quickly - this is no good. It delays
the production update by a day.
Our people work late but not 24 hours.....:D.

On the other hand, since it's pre production, to assemble a 16x64GB
cluster is un reasonable. Any other
solution with multi machines will also yield twisted faces. It's not
that we can't afford machines. We make
them. It just create a vision that you are using a Rolls Royce where a
bicycle should do the job. You know the
fellows upstairs.

Since I decided to drop the caching solution, I turned back to
architecture. I was so glad to realize that the
I have done so because I found the following:

The assembly of the deliverable start at one point when a timer wakes
up and find that there is work to do.
From this point the code, while processing the required, collect a
growing cluster of objects. At the end point
all the components are put in one object, serialize and shipped to the
appropriate server.
At this point all the object that were used are free to be GC'ed.
In this case why not object pooling? I have the starting point AND the
end point!.
The problem with object pooling is that one needs to run around the
objects and figure out when can they be
released back to the pool. Having one point in which ALL the objects
are freed to be GCed - they will be
put back to the various pools by hand. Simple!

With the above I save tons of "new ObjectXXX" AND save a lot of GC.
Can it get better then that?

On top I realize that I can consolidate DB tables AND take a wider DB
queries for code down the line.
A bigger ResultSet beats multiple DB access.

On top I realize that ton of code segments are synchronize very
grossly such that it makes the whole
application running effectively as single thread or sequentially.

My work is cut out for me but I'm sure I will gain tons of speed.

Thanks for all replays. Great ideas for future projects.

-Michael