[ANN] queue: streamlining error handling and piping through a queue of go functions

[meta keule]

Hi gophers,

here is a new library I have written to streamline error handling when calling
a bunch of functions that might return errors.

Code:
https://github.com/go-on/queue  (verbose Syntax)

are more compact syntax provides

github.com/go-on/queue/q

Motivation:

In go, sometimes you need to run a bunch of functions that return errors and/or results. You might end up writing stuff like this

err = fn1(...)

if err != nil {
   // handle error somehow
}

err = fn2(...)

if err != nil {
   // handle error somehow
}

...

a lot of times. This is especially annoying if you want to handle all errors the same way (e.g. return the first error).

queue provides a way to call functions in a queue while collecting the errors via a predefined or custom error handler. The predefined handler returns on the first error and custom error handlers might be used to catch/handle some/all kinds of errors while keeping the queue running.

Example:

    // create a new queue with the default error handler
    err := queue.New().
        // get the name from the map
        Add(get, "Name", m).
        // set the name in the struct
        Add(p.SetName, queue.PIPE).
        // get the age from the map
        Add(get, "Age", m).
        // convert the age to int
        Add(strconv.Atoi, queue.PIPE).
        // set the age in the struct
        Add(p.SetAge, queue.PIPE).
        // inspect the struct
        Add(fmt.Printf, "SUCCESS %#v\n\n", p).
        // run the whole queue
        Run()

or with the more compact github.com/go-on/queue/q

err := q.Q(get, "Age", m)(strconv.Atoi, q.V)(p.SetAge, q.V).Run()

Comments are welcome.

Regards,
benny / metakeule

[Dustin Sallings]

meta keule <marcre...@googlemail.com>
writes:

> Example:
>
> // create a new queue with the default error handler
> err := queue.New().
> // get the name from the map
> Add(get, "Name", m).
> // set the name in the struct
> Add(p.SetName, queue.PIPE).
> // get the age from the map
> Add(get, "Age", m).
> // convert the age to int
> Add(strconv.Atoi, queue.PIPE).
> // set the age in the struct
> Add(p.SetAge, queue.PIPE).
> // inspect the struct
> Add(fmt.Printf, "SUCCESS %#v\n\n", p).
> // run the whole queue
> Run()

I don't think this is clearer to the next maintainer than the more go
idiomatic means of doing things:


if err := p.SetName(m["Name"]); err != nil {
return err
}
age, err := strconv.Atoi(m["Age"])
if err != nil {
return err
}
p.SetAge(age)


It looks like a direct port of a node.js "flow control" library where
performing sequential tasks is more of a challenge than it is in go.
This model will make things slower and less obvious to developers, I
fear.

You're also seeming to loop things together that don't strictly need
to be. Did you intentionally wish a failure to print success to be a
reason for your mutations chain to be considered a failure?

> err := q.Q(get, "Age", m)(strconv.Atoi, q.V)(p.SetAge, q.V).Run()

age, err := strconv.Atoi(m["Age"])
if err != nil {
return err
}
p.SetAge(age)

While lacking the elegance of being on a single line, I'm confident
more people wishing to contribute to your project will understand what's
happening there.

--
dustin

[meta keule]

Am Mittwoch, 29. Januar 2014 20:46:03 UTC+1 schrieb Dustin:
 

      It looks like a direct port of a node.js "flow control" library where
    performing sequential tasks is more of a challenge than it is in go.
    This model will make things slower and less obvious to developers, I
    fear.

It makes things a bit slower, because it uses reflect. As far as I know, the
"flow control" libraries in nodejs are made to help with callback soup not
primary for error handling. Also the last time I checked, javascript did not
have interfaces. I think the combination of sequential code execution and
the error interface and well defined error types could allow the use of very clear
and elegant universal error handlers that would be used throughout a project
while being refined without the thousand places knowing it.

While its a kind of magic, it is not that much, because:
1. it is obvious where the error is handled
2. it is obvious which function is called after which
3. any shared variables that are not piped through are defined outside
the chain.
 
In fact the only "magic" thing is the PIPE parameter that is explained in one sentence and
set explicitely.

      You're also seeming to loop things together that don't strictly need
    to be.  Did you intentionally wish a failure to print success to be a
    reason for your mutations chain to be considered a failure?





    > err := q.Q(get, "Age", m)(strconv.Atoi, q.V)(p.SetAge, q.V).Run()

            age, err := strconv.Atoi(m["Age"])
            if err != nil {
                    return err
            }
            p.SetAge(age)

      While lacking the elegance of being on a single line, I'm confident
    more people wishing to contribute to your project will understand what's
    happening there.

    --
    dustin

Am Mittwoch, 29. Januar 2014 20:46:03 UTC+1 schrieb Dustin:

      You're also seeming to loop things together that don't strictly need
    to be.  Did you intentionally wish a failure to print success to be a
    reason for your mutations chain to be considered a failure?

Well printing a success should either be inside one of the called functions (that should not return the
printing failure as a failure) or it should be done at the end of the chain, because most of the time a
chain is considered as failed if a single part fails. And printing a success is not part of such a chain.

Anyway you could prevent some  errors from stopping the chain, if you use a custom error handler that returns nil for the kind of errors that should be ignored and the error otherwise (see the documentation of OnError and the ErrHandler interface or the definition of STOP and IGNORE).

> err := q.Q(get, "Age", m)(strconv.Atoi, q.V)(p.SetAge, q.V).Run()

        age, err := strconv.Atoi(m["Age"])
        if err != nil {
                return err
        }
        p.SetAge(age)
 

      While lacking the elegance of being on a single line, I'm confident
    more people wishing to contribute to your project will understand what's
    happening there.

Once you know the rules, it is easy to read:

1. The first parameter is the function and the other parameters are passed to that function.
2. The special parameter queue.PIPE (or q.V in the short syntax) is replaced by the return values of the previous function minus errors.

But if you  have  an idea how to improve the transparency of what is going on by an alternative syntax I am open to suggestions.

The library is meant to be used in larger projects as standard (making use of project wide error handlers). When the syntax is nearly everywhere , it should not be hard to remember how it works.

Also it could be used for prototyping. If I develop a new library I often find myself in the situation, where
I don't know yet, how to handle the different errors. It is more like a "there is an error, it should be returned". While improving the library when the dust is setteled down, it becomes more and more clear
where which error should be returned or handled. Refactoring then becomes a lot of work, moving
the error handling around.

Also some times the following problem arises. A function A calls function B that might return an error.
But function A is called from somewhere else and has not enough context to determine, if the returned
error should stop function A from further proceeding or not. But the calling environment has.
Also function A might be called from different environments. With the queue library function A could
simply work like this:

func A(eh queue.ErrHandler) (s Something){
     queue.New().OnError(eh).Add(B, ...).Add(C,...).Run()
}

Now function A does not have to care, if there error is a show stopper or not. It does not know it anyway. Also the type signature and return parameters of A are not cluttered with errors it knows
nothing about. In contexts where it is clear, that B can't return an error, A may be called with the error handler nil.

 

[Dustin Sallings]

meta keule <marcre...@googlemail.com>
writes:

> While its a kind of magic, it is not that much, because:
> 1. it is obvious where the error is handled
> 2. it is obvious which function is called after which
> 3. any shared variables that are not piped through are defined outside
> the chain.

Do you have an example of, say, how one might make an HTTP request?

1. Create request.
2. Set some headers.
3. Have client execute it.
4. On err, do error thing.
5. On success, verify HTTP status, consider it an error if not 200.
5a. If 304, reuse existing content
6. Parse JSON response.
7. Handle JSON errors.
8. Close request if we got past 4.

--
dustin