Handling of Pipeline Errors - Redis DB

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Handling of Pipeline Errors
Received-SPF: pass (google.com: domain of sed...@gmail.com designates 209.85.160.49 as permitted sender) client-ip=209.85.160.49;
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\))
Subject: Re: Handling of Pipeline Errors
From: Andy McCurdy <sed...@gmail.com>
In-Reply-To: <CA+XzkVezksqjyoubvEOYuPtdzx+MLhhM_Pc+hm=COXnaV5s...@mail.gmail.com>
Date: Fri, 5 Oct 2012 10:02:32 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B16D068F-E596-46FB-B847-58FFFBC9E...@gmail.com>
References: <C16477BE-B941-46C1-9B1A-26EAC9D17...@gmail.com> <CA+XzkVezksqjyoubvEOYuPtdzx+MLhhM_Pc+hm=COXnaV5s...@mail.gmail.com>
To: redis-db@googlegroups.com

Great, thanks. My only concern about raising errors at EXEC time is that =
you lose the results of any successful command that completed in the =
pipeline. This probably isn't a terrible thing, and raising does seem =
more appropriate to let people know that there is in fact a bug in their =
code.

-andy


On Oct 5, 2012, at 12:29 AM, Salvatore Sanfilippo <anti...@gmail.com> =
wrote:

> Hi Andy,
>=20
> thanks for raising this issue, I think it is an interesting topic to
> discuss and one where guidelines are a good idea as it's not a matter
> of API or language, but about semantics.
>=20
> I would always raise an exception on parse-time errors, because
> clearly there is no point into executing a transaction that will have
> missing commands because they are misspelled or alike. So unless the
> user traps the exception in some language-specific way, the error will
> stop the execution of the program and the transaction will not be
> executed at all.
>=20
> Unfortunately once you add pipelining into the mix things are not as
> simple as they should be, because:
>=20
> 1) If you send MULTI / commands / EXEC in the same pipeline you can't
> prevent the EXEC even if you detected errors.
> 2) If you split the pipeline sending "MULTI / commands" before, read
> the reply, and than EXEC, you can handle it correctly, but this means
> to pay two times the round trip time, that's not cool.
>=20
> So since parse-time errors should only happen on *syntax errors*, I =
think that:
>=20
> 1) If the library does not use pipelining for MULTI/EXEC the
> parse-time error should raised ASAP, if reply is not "QUEUED".
> 2) If the library uses pipelining it's ok to raise the error after
> sending the EXEC, but the error raised should be the first parse-time
> error encountered if any.
>=20
> After all, even in an ACID SQL system, if there is a syntax error in
> the query, no body can save the user from a disaster, but this should
> only happen in development environments.
>=20
> About exec-time errors, I think this category of errors should also
> raise an exception that should stop the execution of the program by
> default, because in this stage errors can happen only for severe
> conditions like wrong type (problem in the application logic), or out
> of memory in Redis if it's configured with maxmemory, and other
> similar things that are not expected usually.
>=20
> However I think it's a good idea if, optionally, libraries have an API
> that allows to just call EXEC without raising errors but just
> returning the array of replies, that may include errors elements
> inside, with a way to test for errors. Like:
>=20
> result =3D redis.exec(trap_errors =3D true)
> if result[3].is_error() ...
>=20
> in this way the default behaviour is the safest, but the library still
> delivers all the power to handle everything by hand if needed (think
> about writing a shell that returns an array of replies including
> errors for MULTI/EXEC blocks).
>=20
> Cheers,
> Salvatore
>=20
> On Thu, Oct 4, 2012 at 8:55 PM, Andy McCurdy <sed...@gmail.com> wrote:
>> Hey everyone,
>>=20
>> A redis-py user recently encountered an error with the way errors are =
handled in the pipeline implementation. Fixing the error is trivial, but =
I'm curious if there's a standard way client libs should be handling =
errors in pipelines.
>>=20
>> In pipelines that are wrapped in MULTI/EXEC, there's two times that =
errors can happen -- parse time and exec time:
>>=20
>>  * Parse time errors include any error that happens when the command =
is sent to the server to be QUEUED.
>>  * Exec time errors are only found when the command is actually =
executed at EXEC time (such as performing a list operation on a key =
holding a string value)
>>=20
>> Currently redis-py handles exec time errors by placing the raised =
exception instance into the list of returned values from the EXEC =
command. For instance:
>>=20
>>>>> redis.set('foo', '1')
>>>>> pipe =3D redis.pipeline()
>>>>> pipe.lpush('foo', 'some-value')
>>>>> pipe.set('bar', '1')
>>>>> pipe.execute()
>>    [redis.exceptions.ResponseError('Operation against a key holding =
the wrong kind of value'), True]
>>    # but the SET to 'bar' still got executed
>>>>> redis.get('bar')
>>    '1'
>>=20
>> My question is: should parse time errors be handled in the same way? =
We have the opportunity to detect any parse time error before EXEC is =
run, potentially DISCARDing all the statements and raise the error =
immediately. Alternatively parse time errors could be treated identical =
to exec time errors and exceptions could be inserted into EXEC's return =
value just like above.
>>=20
>> Thoughts?
>>=20
>> -andy
>>=20
>> --
>> You received this message because you are subscribed to the Google =
Groups "Redis DB" group.
>> To post to this group, send email to redis-db@googlegroups.com.
>> To unsubscribe from this group, send email to =
redis-db+unsubscribe@googlegroups.com.
>> For more options, visit this group at =
http://groups.google.com/group/redis-db?hl=3Den.
>>=20
>=20
>=20
>=20
> --=20
> Salvatore 'antirez' Sanfilippo
> open source developer - VMware
> http://invece.org
>=20
> Beauty is more important in computing than anywhere else in technology
> because software is so complicated. Beauty is the ultimate defence
> against complexity.
>       =97 David Gelernter
>=20
> --=20
> You received this message because you are subscribed to the Google =
Groups "Redis DB" group.
> To post to this group, send email to redis-db@googlegroups.com.
> To unsubscribe from this group, send email to =
redis-db+unsubscribe@googlegroups.com.
> For more options, visit this group at =
http://groups.google.com/group/redis-db?hl=3Den.
>=20
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
