Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Meta-cache Performance
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  3 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Ben Greenberg  
View profile  
 More options May 27 2011, 12:50 am
From: Ben Greenberg <schna...@gmail.com>
Date: Thu, 26 May 2011 21:50:54 -0700 (PDT)
Local: Fri, May 27 2011 12:50 am
Subject: Meta-cache Performance
Hi all,

I'm using rack-cache in front of an image generation server. My cached
files are large (images) so I currently write both caches to disk. I
read that I can increase performance by putting the meta cache onto
memcached. The problem is that I already have many gigabytes cached;
the meta cache is currently about 1GB and I expect it to grow quite a
bit. I have two questions:

1) If I switch the meta cache to memcached but keep the entity cache
in the same location, will rack-cache know not to regenerate the
entities? Will this keep my cache relatively warm?

2) Are there any good alternatives to memcached for large meta caches?
Any particular filesystems better than others? Would it help to have
it on a separate volume from the entity cache?

Thanks
Ben


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Ryan Tomayko  
View profile  
 More options May 27 2011, 1:30 am
From: Ryan Tomayko <r...@tomayko.com>
Date: Thu, 26 May 2011 22:30:45 -0700
Local: Fri, May 27 2011 1:30 am
Subject: Re: [rack-cache] Meta-cache Performance

On Thu, May 26, 2011 at 9:50 PM, Ben Greenberg <schna...@gmail.com> wrote:
> Hi all,

> I'm using rack-cache in front of an image generation server. My cached
> files are large (images) so I currently write both caches to disk. I
> read that I can increase performance by putting the meta cache onto
> memcached. The problem is that I already have many gigabytes cached;
> the meta cache is currently about 1GB and I expect it to grow quite a
> bit. I have two questions:

> 1) If I switch the meta cache to memcached but keep the entity cache
> in the same location, will rack-cache know not to regenerate the
> entities?

Yes. The metastore cache will need to fill in but the disk cache
should not be rewritten.

> Will this keep my cache relatively warm?

I think it would result in each of the cached URLs being generated by
the app in full once, even though the response won't need to be
written to disk. It's hard to say how much this will impact your app's
performance. If a relatively small percentage of URLs are being hit
very often, it should warm up very quickly. Otherwise, it could take
some time to rebuild the metastore cache.

> 2) Are there any good alternatives to memcached for large meta caches?
> Any particular filesystems better than others? Would it help to have
> it on a separate volume from the entity cache?

Depends on the type of disk, how much other activity is happening on
it, amount of RAM for disk cache, etc. If you're on linux than using a
tmpfs volume to back the metastore cache would probably be a nice win.
You'll lose it on restart and tmpfs uses the kernel's vm subsystem so
this will eat into RAM and eventually swap if RAM is low, but you
won't be moving disk heads loading things from cache which will
probably be a win.

Ryan


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Ben Greenberg  
View profile  
 More options May 28 2011, 2:54 pm
From: Ben Greenberg <mailing_li...@bengreenberg.net>
Date: Sat, 28 May 2011 11:54:47 -0700 (PDT)
Local: Sat, May 28 2011 2:54 pm
Subject: Re: Meta-cache Performance

On May 27, 1:30 am, Ryan Tomayko <r...@tomayko.com> wrote:

Thanks for the replies. This is on Amazon EC2 with the stores on an
EBS volume. So EBS itself is probably a bottleneck. I think that you
nailed it with the tmpfs suggestion; this will allow me to move my
meta-cache into RAM without having to regenerate everything.

Thanks
Ben


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions Older topic »