Caching and SimpleDB query speed

[Danyal]

This is my first time using SimpleDB for a project so I am not sure
what to expect. My data set is fairly simple and my queries are very
simple as well (column = value). My query times average 800ms which
seems really slow. Originally my queries took 1.5s but I reduced this
to ~800ms by keeping connections open per app so it doesn't keep
opening and closing connections.

I'm fairly certain similar queries with a MySQL setup would be <50ms.
Is there something I am doing wrong or is SimpleDB just that slow?

Could I speed queries by using a caching solution like memcache and
write/read data to the cache and have it asynchronously commit that
data to simpledb? If so, how would that actually work? By setting the
simple record cache = memcache what does that get you and what would I
need to implement myself?

Thanks,
Danyal

[Travis Reeder]

There are many factors that have an effect on performance:

- how complex is your query?

- are you running these from your local dev machine or EC2?

- how big is the domain?

That said, MySQL is generally much faster at queries so you have to weight the pros and cons of each to see what will work for you.

--
You received this message because you are subscribed to the Google Groups "SimpleRecord" group.
To post to this group, send email to simple...@googlegroups.com.
To unsubscribe from this group, send email to simple-recor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-record?hl=en.

[Gabriel Williams]

Queries going over the wire are always going to be slow compared to using MySQL.  I highly recommend using something like memcache.  It's easy to set up and if your app is read heavy like most are it's what you should be doing anyway.

You can find a lot of different examples of using memcache but the basic idea is that if you have a normal SimpleDB find like:  Product.find(1)

You'd put it into a block call to your cache like this:

Cache.get "Product:#{1}" do

  Product.find 1

end

The basic implementation should take you 15 minutes.  Just make sure you use cache keys that are as specific as possible and you'll be fine.

Gabriel

On Thu, Dec 23, 2010 at 1:14 AM, Danyal <danya...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "SimpleRecord" group.
To post to this group, send email to simple...@googlegroups.com.
To unsubscribe from this group, send email to simple-recor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-record?hl=en.

--
"I woke the same, as any other day, except a voice was in my head."

[Travis Reeder]

Danyal, I missed the last part of your question but Gabriel is right, you can definitely increase performance with caching.  SimpleRecord has it built in though like you said so you don't need to do a cache block like Gabriel suggested. And if you don't feel like setting up memcached yet, you can always start with something like https://github.com/appoxy/local_cache

cache = LocalCache.new

SimpleRecord::Base.cache_store = cache

[Gabriel Williams]

Whups.  I didn't realize SimpleRecord had caching built in.  I guess I'm so used to setting up memcached that I didn't even look into it. :)

Gabriel

[Danyal]

Thanks for the help. Local cache is really tempting but the problem is
that there would be an instance of the cache for each different ruby
worker. So if I have 6 workers, I would potentially get cache misses 6
times.

Since caching is built in, if I used memcached, would simple_record
write to the cache asynchronously and then the cache would update
simple db? Otherwise I am guessing writes would take just as long as
they take now.

Currently I am testing all this on my local machine so there
definitely is some latency between me and simple db, but upon pinging
aws servers I get a rtt of 90ms, which means simple db is still taking
700ms or so. I searched around for benchmarks and some people reported
their queries taking ~30ms or so on simple db.

On Dec 23, 12:16 pm, Gabriel Williams <ummo...@gmail.com> wrote:
> Whups.  I didn't realize SimpleRecord had caching built in.  I guess I'm so
> used to setting up memcached that I didn't even look into it. :)
>
> Gabriel
>
>
>
>
>
>
>
>
>
> On Thu, Dec 23, 2010 at 12:00 PM, Travis Reeder <tree...@gmail.com> wrote:
> > Danyal, I missed the last part of your question but Gabriel is right, you
> > can definitely increase performance with caching.  SimpleRecord has it built
> > in though like you said so you don't need to do a cache block like Gabriel
> > suggested. And if you don't feel like setting up memcached yet, you can
> > always start with something likehttps://github.com/appoxy/local_cache
>
> > <https://github.com/appoxy/local_cache>
> > cache = LocalCache.new
> > SimpleRecord::Base.cache_store = cache
>

> > On Thu, Dec 23, 2010 at 11:47 AM, Gabriel Williams <ummo...@gmail.com>wrote:
>
> >> Queries going over the wire are always going to be slow compared to using
> >> MySQL.  I highly recommend using something like memcache.  It's easy to set
> >> up and if your app is read heavy like most are it's what you should be doing
> >> anyway.
>
> >> You can find a lot of different examples of using memcache but the basic
> >> idea is that if you have a normal SimpleDB find like:  Product.find(1)
>
> >> You'd put it into a block call to your cache like this:
>
> >> Cache.get "Product:#{1}" do
> >>   Product.find 1
> >> end
>
> >> The basic implementation should take you 15 minutes.  Just make sure you
> >> use cache keys that are as specific as possible and you'll be fine.
>
> >> Gabriel
>

> >> On Thu, Dec 23, 2010 at 1:14 AM, Danyal <danyal.a...@gmail.com> wrote:
>
> >>> This is my first time using SimpleDB for a project so I am not sure
> >>> what to expect. My data set is fairly simple and my queries are very
> >>> simple as well (column = value). My query times average 800ms which
> >>> seems really slow. Originally my queries took 1.5s but I reduced this
> >>> to ~800ms by keeping connections open per app so it doesn't keep
> >>> opening and closing connections.
>
> >>> I'm fairly certain similar queries with a MySQL setup would be <50ms.
> >>> Is there something I am doing wrong or is SimpleDB just that slow?
>
> >>> Could I speed queries by using a caching solution like memcache and
> >>> write/read data to the cache and have it asynchronously commit that
> >>> data to simpledb? If so, how would that actually work? By setting the
> >>> simple record cache = memcache what does that get you and what would I
> >>> need to implement myself?
>
> >>> Thanks,
> >>> Danyal
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "SimpleRecord" group.
> >>> To post to this group, send email to simple...@googlegroups.com.
> >>> To unsubscribe from this group, send email to

> >>> simple-recor...@googlegroups.com<simple-record%2Bunsubscribe@goog legroups.com>
> >>> .

> >>> For more options, visit this group at
> >>>http://groups.google.com/group/simple-record?hl=en.
>
> >> --
> >> "I woke the same, as any other day, except a voice was in my head."
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "SimpleRecord" group.
> >> To post to this group, send email to simple...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> simple-recor...@googlegroups.com<simple-record%2Bunsubscribe@goog legroups.com>
> >> .

> >> For more options, visit this group at
> >>http://groups.google.com/group/simple-record?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "SimpleRecord" group.
> > To post to this group, send email to simple...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > simple-recor...@googlegroups.com<simple-record%2Bunsubscribe@goog legroups.com>
> > .

[Gabriel Williams]

None of the caches will be asynchronous by default but that wouldn't be difficult to implement. Memcached would be shared across workers so you wouldn't have to worry about cache misses. I assume you're using Heroku?

Gabriel

compassion

> To unsubscribe from this group, send email to simple-recor...@googlegroups.com.

[Travis Reeder]

And btw, if you want to do some asynchronous stuff, check out www.simpleworker.com . Makes it almost effortless.