Django broken pipe error

[dr.rom...@gmail.com]

Hi,

I'm facing strange Django broken pipe error (Python 2.7 on Ubuntu) that apparently is a not fixed Django bug. Does anybody now how to fix it? I've been searching a lot and didn't find any solution.

This error happens very irregularly by Post request in Django. Sometimes it works sometimes not:

[06/Dec/2016 13:33:57] "POST /export_report/ HTTP/1.1" 500 59
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 593, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/home/ait/.virtualenvs/env1/local/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 129, in __init__
super(WSGIRequestHandler, self).__init__(*args, **kwargs)
File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 279, in close
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 38224)

It is also described in: https://www.reddit.com/r/joinmarket/comments/4atqrm/is_this_exception_normal_exception_happened/

On https://bugs.python.org/issue14574 is stated that this error should already be fixed but apparently not.

Best regards,

Roman

[justin walters]

Hi Roman,

Is this happening on the dev server or a production server? Did you use the
command ./manage.py runserver to start the server?

Can you please provide the code for the view class/function that is
throwing the error? I believe that this issue
may be caused by a call to the database not being structured correctly.

[dr.rom...@gmail.com]

Thank you Justin,

I'm on the dev server and should present results in this way.

Yes, I use manage.py runserver --insecure to start the server (from PyCharm).

My views.py call:

@detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
def export_report(request):
body_unicode = request.body.decode('utf-8')
body_str = body_unicode.encode('ascii','ignore')
attr_list = body_str.split('&')
attr_dict = {}
if (len(attr_list) > 0):
for attr in attr_list:
...
key = key_value_pair[0]
attr_dict[key] = key_value_pair[1]
trend = trends.calculate_trend(
attr_dict['search_phrase']
, attr_dict['time_from']
, attr_dict['time_to']
, attr_dict['time_scale']
)
attr_dict['trend'] = trend
res = str(json.dumps(attr_dict))
return HttpResponse(res, content_type="text/plain; charset=utf-8")

and trend calculation in trends.py with database calls:

def calculate_trend(query_phrase, time_from, time_to, time_scale):
# check in database if trend already exists
try:
db_trend = Trend.objects.get(pk=query_phrase)
if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
and db_trend.granularity == time_scale:
logger.info("trend already exists.")
existing_trend_dict = ast.literal_eval(db_trend.content)
return existing_trend_dict
except Trend.DoesNotExist:
logger.info("It is a new trend search.")
trend_dict = {}
start_time = pd.Timestamp(value[0])
end_time = pd.Timestamp(value[-1])
freq = ... get frequency using pandas lib
trend_dict[key] = freq
json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
trend = Trend(
phrase=query_phrase,
content=json_trend_content,
from_time=time_from,
to_time=time_to,
granularity=time_scale,
)
if trend is not None:
try:
db_trend = Trend.objects.get(pk=query_phrase)
db_trend.delete()
logger.info("delete old trend: %s. " % trend)
except Trend.DoesNotExist:
logger.info("create trend: %s. " % trend)
trend.save()
return trend_dict

Thank you in advance!

Roman

[justin walters]

> --
> https://mail.python.org/mailman/listinfo/python-list
>


It looks like you can probably get rid of the try/except block at the end
of the calculate_trend
method as any existing Trend object will have already been caught in the
first try/except block.

>From what I'm reading here:
http://stackoverflow.com/questions/11866792/how-to-prevent-errno-32-broken-pipe
,
this issue can be caused by the client closing the connection before
sendall() finishes writing. Can you estimate
how long it takes for a request to this endpoint takes to resolve? If it's
a long time(maybe due to the pandas call?),
your browser/client may be timing out. It could be because it takes a while
for the Db to find an existing Trend object
as well.

I can't give you any advice on how to fix it exactly, but I can tell you
what the problem is: The client is closing the
connection before socket.sendall() has finished writing to the socket. My
guess is that the calculate_trend() method
takes a long time to complete and the client is timing out.

[roma]

Thanks Justin,

I believe, the whole database story has no influence on the broken pipe error. I've commented out the whole block and leave only return line:

return HttpResponse(res, content_type="text/plain; charset=utf-8")

The error is still present. And I have no influence on that.

I call python from js client:

var newTrendReport = new App.TrendReport();
newTrendReport.set('search_phrase', search_phrase);
newTrendReport.set('time_from', time_from);
newTrendReport.set('time_to', time_to);
newTrendReport.set('time_scale', time_scale);
newTrendReport.set('category', category);
newTrendReport.startExport(

function(response){
console.log("Successfully calculated trend report.");
App.trendPage = new App.TrendPageView();
App.trendPage.render(response);
},
);

go throw:

App.TrendReport = Backbone.Model.extend({
urlRoot: "/api/trend_reports/",
defaults: {
search_phrase: "",
time_from: "",
time_to: "",
time_scale: "",
category: ""
},

startExportSuffix: "/export_report/",

startExport: function( successCallback, errorCallback ) {
console.log("start trend calculation");
var that = this;
var ajaxUrl = this.startExportSuffix;
var options = {
method: "POST",
data: this.attributes,
contentType: "application/json;charset=UTF-8",
dataType: "json",

error: errorCallback,
success: successCallback
};
console.log("start trend export sync");
App.ajax(ajaxUrl, options);
}

});

and come in export_report method.

My urls.py:

url(r'^export_report', ensure_csrf_cookie(views.export_report), name="export_report"),

[justin walters]

> --
> https://mail.python.org/mailman/listinfo/python-list
>

I'm not super familiar with the way backbone does http requests, but
something seems off about the startExport function.
It seems to me that you are sending a post request to "/export_report/"
which is an endpoint that I'm guessing is nested
in an include from "/api/trend_reports/". However, it looks like the error
you're getting above says you aren't sending
the request to "https://root.com/api/trend_reports/export_report/".
Instead, you are sending the request to "https://root.com/export_report/" .
Though, I'm also not sure that's the case because that would normally throw
a 404.

I also noticed that you set content type to 'application/json' in your js,
but the view function returns a 'text/plain' content type. Is
There a reason for this?

The data key in your js http function is set to the attributes variable
which, as far as I can tell, does not exist.

There's a lot going on here, but I think you can probably narrow it down to
something in your backbone code or the way
backbone handles http requests as the error you are getting is caused by
the client prematurely closing the socket.
It's possible backbone will stop reading the response since it isn't the
same content-type as the request.

[dr.rom...@gmail.com]

Thanks a lot Justin,

The problem was solved when I employed standard Framework methods for creation of new database object:

in JS:
var trendModel = new App.TrendModel();
trendModel.set("phrase", search_phrase);
trendModel.set("from_time", time_from);
trendModel.set(...
trendModel.save(...

in PY:
def create(self, request):
...

I've also created extra template and additional View. PageView for some unclear reason didn't support creation of new object - this event just disappeared.

Regards,

Roman

[justin walters]

On Mon, Jan 2, 2017 at 6:14 AM, <dr.rom...@gmail.com> wrote:

>
> Thanks a lot Justin,
>
> The problem was solved when I employed standard Framework methods for
> creation of new database object:
>
> in JS:
> var trendModel = new App.TrendModel();
> trendModel.set("phrase", search_phrase);
> trendModel.set("from_time", time_from);
> trendModel.set(...
> trendModel.save(...
>
> in PY:
> def create(self, request):
> ...
>
> I've also created extra template and additional View. PageView for some
> unclear reason didn't support creation of new object - this event just
> disappeared.
>
> Regards,
>
> Roman

> --
> https://mail.python.org/mailman/listinfo/python-list
>

Glad to hear that you solved the problem!