Pyramid 1.7.3 Using HTTPFound exception for JSON response when resource not found in backend database.

[dcs3spp]

Hi,

Wonder if anyone can help/advise with this query. 

I am using the built in HTTPException hierarchy to raise a JSON exception with the use of content_type and body parameters, e.g. 

raise HTTPBadRequest (content_type='application/json', body=json.dumps(myErrorDict)). 

This works fine rendering JSON for exception types other than HTTPNotFound and HTTPForbidden. 

The documentation at http://docs.pylonsproject.org/projects/pyramid/en/latest/api/httpexceptions.html#module-pyramid.httpexceptions 

states that when these types of exception are RAISED the appropriate default view (not found or forbidden) will be called. This is indeed the case when testing and always appear to be rendered as HTML, despite setting the content_type and body parameters as JSON. 

I have noticed that if I RETURN HttpNotFound exception as a response then JSON is rendered, e.g. 

return HTTPNotFound (content_type='application/json', body=json.dumps(myErrorDict))

In my application I raise a HTTPNotFound exception whenever a resource id for a GET request cannot be found in the backend database. Currently this directs me to the HTML landing view for not found errors.

In this case is it acceptable to RETURN HTTPNotFound so that I can achieve a JSON response? Or, should I be using a different exception type when a resource cannot be located in the database?  This way I am not confusing HTTPNotFound exception with the scenario where the server did not find anything matching the Request-URI? What are other developers opinions on this?

Cheers

Simon

[Michael Merickel]

The default exceptions raised when the framework determines notfound and forbidden errors are reliant on the request's Accept headers to determine which response to see. If you're seeing html it's because the request favored it over application/json. You can override these exception views and then return whatever response you'd like via the notfound_view_config and forbidden_view_config decorators [1].

As an aside, you could be doing `json_body=myErrorDict` instead of doing the json.dumps yourself if you wanted.

[1] http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/hooks.html#changing-the-not-found-view

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/b3d1eb75-5e46-409d-95a5-dc10061b9009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dcs3spp]

Cheers Michael. Thanks for responding :) 

Can't seem to locate where request is favouring HTML over application/json. I have tested using curl request to set the accept and content-type headers as follows:

curl -i -H "Accept: application/json" -H "Content-Type: application/json" http://localhosthost:6543/assignments/101

Have also tried passing in headers dictionary with Accept=application/json and Content-Type=application/json values to the HTTPNotFound exception class but still renders HTML. The only way I can seem to get JSON response is if I RETURN a HTTPNotFound exception or use the notfound and forbidden_view decorators as suggested. Will probably go down the route of using the decorators to render JSON for unmatched URLs AND resource not found urls.

Cheers. Thanks again for your help :)

Simon

 

   

 

On Thursday, 15 December 2016 16:18:29 UTC, Michael Merickel wrote:

The default exceptions raised when the framework determines notfound and forbidden errors are reliant on the request's Accept headers to determine which response to see. If you're seeing html it's because the request favored it over application/json. You can override these exception views and then return whatever response you'd like via the notfound_view_config and forbidden_view_config decorators [1].

As an aside, you could be doing `json_body=myErrorDict` instead of doing the json.dumps yourself if you wanted.

[1] http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/hooks.html#changing-the-not-found-view

On Thu, Dec 15, 2016 at 9:47 AM, 'dcs3spp' via pylons-discuss <pylons-...@googlegroups.com> wrote:

Hi,

Wonder if anyone can help/advise with this query. 

I am using the built in HTTPException hierarchy to raise a JSON exception with the use of content_type and body parameters, e.g. 

raise HTTPBadRequest (content_type='application/json', body=json.dumps(myErrorDict)). 

This works fine rendering JSON for exception types other than HTTPNotFound and HTTPForbidden. 

The documentation at http://docs.pylonsproject.org/projects/pyramid/en/latest/api/httpexceptions.html#module-pyramid.httpexceptions 

states that when these types of exception are RAISED the appropriate default view (not found or forbidden) will be called. This is indeed the case when testing and always appear to be rendered as HTML, despite setting the content_type and body parameters as JSON. 

I have noticed that if I RETURN HttpNotFound exception as a response then JSON is rendered, e.g. 

return HTTPNotFound (content_type='application/json', body=json.dumps(myErrorDict))

In my application I raise a HTTPNotFound exception whenever a resource id for a GET request cannot be found in the backend database. Currently this directs me to the HTML landing view for not found errors.

In this case is it acceptable to RETURN HTTPNotFound so that I can achieve a JSON response? Or, should I be using a different exception type when a resource cannot be located in the database?  This way I am not confusing HTTPNotFound exception with the scenario where the server did not find anything matching the Request-URI? What are other developers opinions on this?

Cheers

Simon

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

[Mike Orr]

On Thu, Dec 15, 2016 at 7:47 AM, 'dcs3spp' via pylons-discuss
<pylons-...@googlegroups.com> wrote:
> In this case is it acceptable to RETURN HTTPNotFound so that I can achieve a
> JSON response?

Yes.

> Or, should I be using a different exception type when a
> resource cannot be located in the database? This way I am not confusing
> HTTPNotFound exception with the scenario where the server did not find
> anything matching the Request-URI? What are other developers opinions on
> this?

It's your decision whether the user should see 404 Not Found or
another status. The HTTP definition of 404 is flexible and can mean
anything from "invalid URL" to "record not in database" to "I don't
want to tell you why". Some people use it for security (over
HTTPForbidden) to avoid revealing that that URL would ever give a 200
response to anybody.

So if you want the user to see a JSON HTTPNotFound, just return it.
Raising it tells Pyramid to catch it and create another response via
the notfound view mechanism. That's unnecessary overhead and puts the
response details out of the original view's control. If Pyramid's
default notfound view doesn't return JSON when you want it to, then
you should configure a custom notfound view that does.

If you want to troubleshoot the default notfound view, you can copy it
from the Pyramid source, configure it as your notfound view, and
insert debugging statements to see what the Accept: header is at that
point and whether there's a bug in the function.

--
Mike Orr <slugg...@gmail.com>

[Simon Pears]

Cheers Mike thanks for the suggestion :) Interesting point regarding performance overhead of raising the exception. Hmmmm. Now swaying towards RETURN HTTPNotFound for missing database record and using a custom notfound view to return JSON for missing request-URI.

Thanks for all suggestions regarding debugging, etc. Very helpful insight :)

Cheers

Simon 

--
You received this message because you are subscribed to a topic in the Google Groups "pylons-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylons-discuss/Rm9S34IM9fY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DuoJk-N%3DbA%2B2A-HTCMEu8JhyMSw%2BOgv56bcCgUV6%3DK%2B5Nw%40mail.gmail.com.

[Michael Merickel]

On Thu, Dec 15, 2016 at 12:06 PM, 'dcs3spp' via pylons-discuss <pylons-...@googlegroups.com> wrote:

Cheers Michael. Thanks for responding :) 

Can't seem to locate where request is favouring HTML over application/json. I have tested using curl request to set the accept and content-type headers as follows:

curl -i -H "Accept: application/json" -H "Content-Type: application/json" http://localhosthost:6543/assignments/101

Have also tried passing in headers dictionary with Accept=application/json and Content-Type=application/json values to the HTTPNotFound exception class but still renders HTML. The only way I can seem to get JSON response is if I RETURN a HTTPNotFound exception or use the notfound and forbidden_view decorators as suggested. Will probably go down the route of using the decorators to render JSON for unmatched URLs AND resource not found urls.

The exception responses will auto-select the appropriate response by default, however this is only the case when you don't override the body [1]. In your case your custom exceptions are fully specifying that they are json. However like I said the default exceptions raised by pyramid for notfound/forbidden are caught and have their own default body, and based on [1] will return json or html or plain text. If you want this to always be json, just like your example Response(content_type=..., body=json.dumps(...)), then you should override the notfound/forbidden views in your app. If you don't believe what I'm saying about this (or not seeing it in your own app) then you will need to submit a reproducible example to talk about it further.

For example, the most basic pyramid app that always shows a notfound view:

from pyramid.config import Configurator

from wsgiref.simple_server import make_server

config = Configurator()

app = config.make_wsgi_app()

server = make_server('127.0.0.1', 8080, app)

server.serve_forever()

curl -D - -H 'Accept: application/json' http://localhost:8080

HTTP/1.0 404 Not Found

Date: Thu, 15 Dec 2016 19:35:04 GMT

Server: WSGIServer/0.2 CPython/3.5.2

Content-Type: application/json

Content-Length: 105

{"message": "The resource could not be found.\n\n\n/\n\n", "code": "404 Not Found", "title": "Not Found"}

curl -D - -H 'Accept: text/html' http://localhost:8080

HTTP/1.0 404 Not Found

Date: Thu, 15 Dec 2016 19:36:32 GMT

Server: WSGIServer/0.2 CPython/3.5.2

Content-Type: text/html; charset=UTF-8

Content-Length: 153

<html>

 <head>

  <title>404 Not Found</title>

 </head>

 <body>

  <h1>404 Not Found</h1>

  The resource could not be found.<br/><br/>

/

 </body>

</html>

[1] https://github.com/Pylons/pyramid/blob/1.7-branch/pyramid/httpexceptions.py#L245

[Mike Orr]

At first I always raised HTTPException's to make them more obvious un
the Python code. But after having problems with HTTPNotFound and
HTTPForbidden for similar reasons as yours (although not regarding
JSON), I switched to returning them. The only time I raise
HTTPException views now is if I'm in a utility function or method, to
bust through several layers of the call stack without having to put an
'if' in every caller to
recognize this special result.

I found the workflow of what happens when you return or raise an
HTTPException hard to understand, and gradually came to the
explanation I described. Which is why I described it here. If there's
any mistake in the description, Michael can correct it since he
maintains the Pyramid code and knows it inside and out.

>> pylons-discus...@googlegroups.com.
>> To post to this group, send email to pylons-...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DuoJk-N%3DbA%2B2A-HTCMEu8JhyMSw%2BOgv56bcCgUV6%3DK%2B5Nw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discus...@googlegroups.com.

> To post to this group, send email to pylons-...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pylons-discuss/CAEJBWeiZ5kQMBKC%2BCgQ%3D7Nfg3BjZMvxr%3Dvw-T-ws-K8%2BvhNfBg%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
Mike Orr <slugg...@gmail.com>

[Jonathan Vanasco]

Since you explicitly want to return a JSON error (and not adapt a response for the path) I can offer another strategy (which i use).

This is from memory so the code may need some tweaking.  (working from home today and don't have the source)

1. Define a custom class for JSON exceptions:

    class ViewException_JSON(Exception):

        def __init__(self, message, pyramid_exc):

            self.message = message

            self.pyramid_exc = pyramid_exc

2. Define a custom view exception:

    @view_config(context=ViewException_JSON, renderer='json')

    def exception_view_json(exc, request):

        rval = {'status': error,

                'message': exc.message,  # this can be a python dict!

                }

        request.response.status_code = exc.pyramid_exc.code

        # handle the `location` stuff

3. In your code, return/raise a wrapped exception. Both should work.

       raise ViewException_JSON(HTTPFound(location=))

       return ViewException_JSON(HTTPFound(location=))

Another option is to use a custom HttpFound class, and handle that with an view_config `context` too.  I've settled on wrapping the Pyramid exceptions though, because I've found it a little bit easier in terms of maintenance (documentation, understanding, etc)

[dcs3spp]

Cheers, thanks all for the valuable suggestions. Much appreciated :) Hmm, plenty of options and lots to think about. Interesting to hear what others are doing regards this scenario :) Ok will soak up all the info and have a play around with suggestions. Thanks again :)  

[Mike Orr]

Why does this work if you return the wrapped exception? Since it's a
Response, shouldn't Pyramid just return it to the user?

> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discus...@googlegroups.com.
> To post to this group, send email to pylons-...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pylons-discuss/9060c7ee-312a-47f3-b04d-13b6039909d6%40googlegroups.com.