Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
No locking the data down during the transaction?
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  15 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Felix E. Klee  
View profile  
 More options Oct 12 2012, 5:45 am
From: "Felix E. Klee" <felix.k...@inka.de>
Date: Fri, 12 Oct 2012 11:45:28 +0200
Local: Fri, Oct 12 2012 5:45 am
Subject: No locking the data down during the transaction?
Quote from the book [Learning Node][1]:

"There's no locking the data down during the transaction, either, as
you'd also expect with a relational database transaction. So any changes
to the Redis database during the query can impact the results."

Quote from the [documentation about transactions][2]:

"It can never happen that a request issued by another client is served
*in the middle* of the execution of a Redis transaction."

Who is right?

[1]: http://shop.oreilly.com/product/0636920024606.do
[2]: http://redis.io/topics/transactions


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Felix Gallo  
View profile  
 More options Oct 12 2012, 1:07 pm
From: Felix Gallo <felixga...@gmail.com>
Date: Fri, 12 Oct 2012 10:07:12 -0700
Local: Fri, Oct 12 2012 1:07 pm
Subject: Re: No locking the data down during the transaction?
The documentation.  And on a personal note, I wouldn't trust a
'learning node' book to get the number of letters in the alphabet
right, as a general practice.

F.

On Fri, Oct 12, 2012 at 2:45 AM, Felix E. Klee <felix.k...@inka.de> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
AcidZombie24  
View profile  
 More options Oct 12 2012, 1:08 pm
From: AcidZombie24 <acidzombi...@gmail.com>
Date: Fri, 12 Oct 2012 10:08:18 -0700 (PDT)
Local: Fri, Oct 12 2012 1:08 pm
Subject: Re: No locking the data down during the transaction?

The 2nd but the first isn't wrong. Redis is single threaded so it can only
do one at a time.
However the first means when you do WATCH varname other queries can happen
and varname may be changed. However when you get to the MULTI keyword
(where all you can do are writes) it will just simply not execute any of
them since a variable you watch has been changed. So technically they both
are right, there is no locking but it is single threaded. I rather like
that it forces me to do all reads then write all my variables at once.
There is something that feels very right about that.

Anyways see
http://redis.io/topics/transactions
http://redis.io/commands#transactions (especially watch, multi and exec)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Felix E. Klee  
View profile  
 More options Oct 12 2012, 2:59 pm
From: "Felix E. Klee" <felix.k...@inka.de>
Date: Fri, 12 Oct 2012 20:59:22 +0200
Local: Fri, Oct 12 2012 2:59 pm
Subject: Re: No locking the data down during the transaction?
On Fri, Oct 12, 2012 at 7:08 PM, AcidZombie24 <acidzombi...@gmail.com>
wrote:

> However the first means when you do WATCH varname other queries can
> happen and varname may be changed.

I'm not so sure that the author is referring to `WATCH`. In fact `WATCH`
is mentioned nowhere in the chapter. Furthermore, to me, "running during
the query" means: while `EXEC` is running. And while `EXEC` is running,
things are atomic (single thread).

Perhaps, I'll proposal an errata.

I've read this even before reading what I quoted from the book.

By the way, in the end of the article, it is strongly suggested to use
LUA scripting instead of `MULTI` and `WATCH`. So, I am playing with
that.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Josiah Carlson  
View profile  
 More options Oct 12 2012, 11:14 pm
From: Josiah Carlson <josiah.carl...@gmail.com>
Date: Fri, 12 Oct 2012 20:14:37 -0700
Local: Fri, Oct 12 2012 11:14 pm
Subject: Re: No locking the data down during the transaction?
Note that Node doesn't support WATCH/MULTI/EXEC transactions.
Also note that while there are many things that Lua does support,
there are some operations that Lua doesn't support that are possible
with WATCH/MULTI/EXEC transactions (or locks built from Lua scripting
or WATCH/MULTI/EXEC transactions).

I can list them out if you'd like.

Regards,
 - Josiah

On Fri, Oct 12, 2012 at 11:59 AM, Felix E. Klee <felix.k...@inka.de> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Felix E. Klee  
View profile  
 More options Oct 13 2012, 4:33 am
From: "Felix E. Klee" <felix.k...@inka.de>
Date: Sat, 13 Oct 2012 10:32:34 +0200
Local: Sat, Oct 13 2012 4:32 am
Subject: Re: No locking the data down during the transaction?
On Sat, Oct 13, 2012 at 5:14 AM, Josiah Carlson

<josiah.carl...@gmail.com> wrote:
> Note that Node doesn't support WATCH/MULTI/EXEC transactions.

What? As far as I understand the node_redis module, it just forwards
commands to Redis. From the [node_redis documentation][1]:

"`MULTI` commands are queued up until an `EXEC` is issued, and then all
commands are run atomically by Redis."

> Also note that while there are many things that Lua does support,
> there are some operations that Lua doesn't support that are possible
> with WATCH/MULTI/EXEC transactions [...]

> I can list them out if you'd like.

Just one, please. That could be enlightening.

By the way, I am using scripting for very simple and well constrained
tasks. Example:

 1. Get elements 1-7 from sorted list (`ZRANGE`).

 2. Get score for each element (`ZSCORE`).

 3. Return elements with score.

[1]: https://github.com/mranney/node_redis#clientmulticommands


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Felix E. Klee  
View profile  
 More options Oct 13 2012, 5:57 pm
From: "Felix E. Klee" <felix.k...@inka.de>
Date: Sat, 13 Oct 2012 23:56:38 +0200
Local: Sat, Oct 13 2012 5:56 pm
Subject: Re: No locking the data down during the transaction?
On Sat, Oct 13, 2012 at 10:32 AM, Felix E. Klee <felix.k...@inka.de>
wrote:

> By the way, I am using scripting for very simple and well constrained
> tasks. Example: [...]

Just found out that exactly this task should be doable in one step
anyhow, by using the `ZRANGE` option `WITHSCORES`...

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Josiah Carlson  
View profile  
 More options Oct 14 2012, 3:03 am
From: Josiah Carlson <josiah.carl...@gmail.com>
Date: Sun, 14 Oct 2012 00:02:55 -0700
Local: Sun, Oct 14 2012 3:02 am
Subject: Re: No locking the data down during the transaction?
On Sat, Oct 13, 2012 at 1:32 AM, Felix E. Klee <felix.k...@inka.de> wrote:

> On Sat, Oct 13, 2012 at 5:14 AM, Josiah Carlson
> <josiah.carl...@gmail.com> wrote:
>> Note that Node doesn't support WATCH/MULTI/EXEC transactions.

> What? As far as I understand the node_redis module, it just forwards
> commands to Redis. From the [node_redis documentation][1]:

> "`MULTI` commands are queued up until an `EXEC` is issued, and then all
> commands are run atomically by Redis."

I had a discussion here with a fellow who swore up and down that the
Node Redis client only used a single connection to Redis, so if you
wanted to do a WATCH/.../MULTI/.../EXEC, while you were waiting for
the GET result in a callback, someone else could hop in and run some
other commands on that same connection - which would mess up what you
were doing.

If Node lets you create multiple connections, then I withdraw my claim.

Also note that there is a fundamental difference between MULTI/EXEC.
MULTI/EXEC lets you do an atomic sequence of operations as if they
were 1. But WATCH/MULTI/EXEC lets you do this:
WATCH foo
x = GET foo
if foo < 3:
  UNWATCH
  return
MULTI
INCR foo
EXEC

You can also do that with Lua, but AFAIK, that sequence of operations
is not possible with the Node Redis libraries (unless you can use
multiple connections).

>> Also note that while there are many things that Lua does support,
>> there are some operations that Lua doesn't support that are possible
>> with WATCH/MULTI/EXEC transactions [...]

>> I can list them out if you'd like.

> Just one, please. That could be enlightening.

1. Fetch data from Redis
2. Use the fetched data to fetch other data from a *different* Redis,
local cache, a database, or some other resource
3. When the secondary data has been fetched, combine the two pieces of
data and write back to Redis - failing if some other process has
already done the same

You can think of this as the explicit cache-update scenario. It is
easy with WATCH/MULTI/EXEC or higher-level locks. Not possible with
Lua, as you can't make remote requests (by design). There are a couple
other things that I mention in my book about why something like
WATCH/MULTI/EXEC or locks built on W/M/E or Lua are still necessary
(but that's in chapter 11, which won't be out for another 3 weeks or
so).

> By the way, I am using scripting for very simple and well constrained
> tasks. Example:

I'm glad you figured out how to do this with a single command :)

 - Josiah


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Marc Gravell  
View profile  
 More options Oct 14 2012, 5:21 am
From: Marc Gravell <marc.grav...@gmail.com>
Date: Sun, 14 Oct 2012 10:20:50 +0100
Local: Sun, Oct 14 2012 5:20 am
Subject: Re: No locking the data down during the transaction?

Indeed, transactions on a multiplexed implementation is really hard for checked (WATCH) exceptions. BookSleeve is also multiplexed, and for ages I didn't expose WATCH because it breaks horribly on a multiplexer. So I did the following, which could perhaps be of use for the node implementation?

http://marcgravell.blogspot.com/2012/10/multiplexed-transactions-in-b...

This covers most of the common scenarios without having to drop to lua.

Of course, it would be even nicer if there was some kind of "ASSERT" command, but this'll do!

Marc

On 14 Oct 2012, at 08:02, Josiah Carlson <josiah.carl...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
AcidZombie24  
View profile  
 More options Oct 14 2012, 1:10 pm
From: AcidZombie24 <acidzombi...@gmail.com>
Date: Sun, 14 Oct 2012 10:10:56 -0700 (PDT)
Local: Sun, Oct 14 2012 1:10 pm
Subject: Re: No locking the data down during the transaction?

Funny that you wrote this now. I was just about to start a redis prototype
today and it requires transactions/watch. Now i wont have to ask how to do
watch.

I don't see an easy way to pull the sources unless i install mecurial.
Maybe i will install it. But i guess i'll write a test not using
transactions and just figure out the data structure of everything. Please
do implement it. I pretty much assume i'd create a new connection to use
watch and i dont know why that wouldnt be recommended (it is in memory and
should be fast anyways???)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Marc Gravell  
View profile  
 More options Oct 14 2012, 3:07 pm
From: Marc Gravell <marc.grav...@gmail.com>
Date: Sun, 14 Oct 2012 20:06:46 +0100
Local: Sun, Oct 14 2012 3:06 pm
Subject: Re: No locking the data down during the transaction?

If that is a direct reply to me (I know you use BookSleeve), then please read the blog for more information on why "watch" isn't directly exposed. However, the new build with constraint support (which is implemented via watch, but which is multiplexer friendly) is available on NuGet.

Please take a look at the AddConstraint approach I've cited in the blog post. If what you need doesn't fit into that, please let me know the scenario (here or directly) so that I can out some thoughts together. Unless there is an **absolute need** I don't plan on exposing "watch" directly - it really is that much of a deal-breaker in a shared-connection (multiplexed) scenario, and that is a fundamental part of BookSleeve's design philosophy.

If it wasn't a direct rely to me, then ignore me :)

Marc

On 14 Oct 2012, at 18:10, AcidZombie24 <acidzombi...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
AcidZombie24  
View profile  
 More options Oct 14 2012, 4:00 pm
From: AcidZombie24 <acidzombi...@gmail.com>
Date: Sun, 14 Oct 2012 13:00:36 -0700 (PDT)
Local: Sun, Oct 14 2012 4:00 pm
Subject: Re: No locking the data down during the transaction?

@Marc: I think ServiceStack.Redis should be used/recommended for
transactions requiring WATCH.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Josiah Carlson  
View profile  
 More options Oct 15 2012, 5:15 pm
From: Josiah Carlson <josiah.carl...@gmail.com>
Date: Mon, 15 Oct 2012 14:15:42 -0700
Local: Mon, Oct 15 2012 5:15 pm
Subject: Re: No locking the data down during the transaction?
You may want to ping the node-redis folks about it.

Many other languages just end up using connection pools. On the Python
side of things, any time a command is executed directly, a connection
is grabbed, used, and returned. With MULTI/EXEC transactions, the
connection is only grabbed when EXEC is called (at which point all of
the commands are passed), and with WATCH/MULTI/EXEC, connections are
grabbed at WATCH. This works *very* well in practice.

Regards,
 - Josiah


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Felix E. Klee  
View profile  
 More options Oct 17 2012, 4:45 am
From: "Felix E. Klee" <felix.k...@inka.de>
Date: Wed, 17 Oct 2012 10:44:33 +0200
Local: Wed, Oct 17 2012 4:44 am
Subject: Re: No locking the data down during the transaction?
On Sun, Oct 14, 2012 at 9:02 AM, Josiah Carlson

<josiah.carl...@gmail.com> wrote:
> MULTI/EXEC lets you do an atomic sequence of operations as if they
> were 1.

MULTI/EXEC was what I was thinking about when I wrote my initial email.

There was a misunderstanding on my part. When you wrote "Node doesn't
support WATCH/MULTI/EXEC transactions", I was thinking that you mean
`node_redis` neither supports `WATCH`, nor `MULTI`, nor `EXEC`. Of
course, you specifically mean the sequence WATCH/MULTI/EXEC, i.e.
*optimistic locking*.

I just thought about the assertion of that fellow you mentioned. Example
scenario:

 * Run 1:

     1. WATCH Redis key *x*.

     2. GET value of Redis key *x*, with callback *f*.

 * Run 2: Callback *f(value)*:

     1. Calculate with *value*.

     2. Prepare transaction with MULTI.

     3. EXEC transaction.

Between runs 1 and 2, there could be another run making use of WATCH,
MULTI, or EXEC, and that will lead to unexpected results. Thanks for
pointing me to this issue!

Fortunately, I don't need to use WATCH/MULTI/EXEC at the moment. LUA
scripting is just great and really easy to use.

>>> there are some operations that Lua doesn't support that are possible
>>> with WATCH/MULTI/EXEC transactions [...]

>>> I can list them out if you'd like.

>> Just one, please. That could be enlightening.

> [...]

Thanks for the example!

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Felix E. Klee  
View profile  
 More options Oct 17 2012, 5:01 am
From: "Felix E. Klee" <felix.k...@inka.de>
Date: Wed, 17 Oct 2012 11:01:12 +0200
Local: Wed, Oct 17 2012 5:01 am
Subject: Re: No locking the data down during the transaction?
On Fri, Oct 12, 2012 at 8:59 PM, Felix E. Klee <felix.k...@inka.de>
wrote:

> Perhaps, I'll propos[e] an errata.

Done:

<http://oreilly.com/catalog/errataunconfirmed.csp?isbn=0636920024606>


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »