Gunicor error

[Wilfried Anuzet]

I installed taiga at home and at my work and there's an error I encounter on both install.
I follow the production setup tutorial to install both instance. Eveything seems fine but when I want to add a userstory right when I clic on the create button nothing seems to happen (exept the following error appear in the gunicorn error log) and I'm still on the userstory creation page. If I close that page and refresh the kanban page I see that my user story was created.

==> gunicorn.stderr.log <==
Trying import local.py settings...
Raven is not configured (logging is disabled). Please see the documentation for more information.
ERROR:2016-03-25 12:01:52,078: Internal Server Error: /api/v1/userstories
Traceback (most recent call last):
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/taiga/taiga-back/taiga/base/api/viewsets.py", line 105, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/taiga/taiga-back/taiga/base/api/views.py", line 444, in dispatch
    response = self.handle_exception(exc)
  File "/home/taiga/taiga-back/taiga/base/api/views.py", line 441, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.4/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/django/db/transaction.py", line 271, in __exit__
    connection.set_autocommit(True)
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/transaction_hooks/mixin.py", line 58, in set_autocommit
    self.run_and_clear_commit_hooks()
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/transaction_hooks/mixin.py", line 38, in run_and_clear_commit_hooks
    func()
  File "/home/taiga/taiga-back/taiga/events/signal_handlers.py", line 45, in <lambda>
    emit_event = lambda: events.emit_event_for_model(instance, sessionid=sesionid, type=type)
  File "/home/taiga/taiga-back/taiga/events/events.py", line 78, in emit_event_for_model
    data=data)
  File "/home/taiga/taiga-back/taiga/events/events.py", line 44, in emit_event
    backend = backends.get_events_backend()
  File "/home/taiga/taiga-back/taiga/events/backends/base.py", line 59, in get_events_backend
    return cls(**options)
TypeError: object() takes no parameters

I also have an error when I tried to delete an userstory. A red ribbon appears and say: "Oops something happen" (the following error appear in the gunicorn log) but when I clic on the cancel button I see that my userstory was deleted:

ERROR:2016-03-25 12:16:26,842: Internal Server Error: /api/v1/userstories/2
Traceback (most recent call last):
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/taiga/taiga-back/taiga/base/api/viewsets.py", line 105, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/taiga/taiga-back/taiga/base/api/views.py", line 444, in dispatch
    response = self.handle_exception(exc)
  File "/home/taiga/taiga-back/taiga/base/api/views.py", line 441, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.4/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/django/db/transaction.py", line 271, in __exit__
    connection.set_autocommit(True)
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/transaction_hooks/mixin.py", line 58, in set_autocommit
    self.run_and_clear_commit_hooks()
  File "/home/taiga/.virtualenvs/taiga/lib/python3.4/site-packages/transaction_hooks/mixin.py", line 38, in run_and_clear_commit_hooks
    func()
  File "/home/taiga/taiga-back/taiga/events/signal_handlers.py", line 59, in <lambda>
    emit_event = lambda: events.emit_event_for_model(instance, sessionid=sesionid, type="delete")
  File "/home/taiga/taiga-back/taiga/events/events.py", line 78, in emit_event_for_model
    data=data)
  File "/home/taiga/taiga-back/taiga/events/events.py", line 44, in emit_event
    backend = backends.get_events_backend()
  File "/home/taiga/taiga-back/taiga/events/backends/base.py", line 59, in get_events_backend
    return cls(**options)
TypeError: object() takes no parameters

Did you have any clues ? I see that the log said that "Raven is not configured" but don't see anything on the taiga doc about Raven...


[Wilfried Anuzet]

i just enabled the mail in the local.py file (before it was still commented as the tutorial don't uncomment it) and the gunicorn error disapear. But I stil have an issue with nginx that time:

2016/03/25 13:40:39 [error] 18818#0: *177 upstream prematurely closed connection while reading response header from upstream, client: 10.98.24.53, server: _, request: "POST /api/v1/userstories HTTP/1.1", upstream: "http://127.0.0.1:8001/api/v1/userstories", host: "taiga.stluc.ucl.ac.be", referrer: "http://taiga.stluc.ucl.ac.be/project/aw1538-linux-ad-authentication/kanban"

It seems that the port is opened and listening:

(taiga)taiga@taiga:~/logs$ sudo netstat -tunlp| grep 8001
tcp        0      0 127.0.0.1:8001          0.0.0.0:*               LISTEN      18783/python3.4

...

[Wilfried Anuzet]

It seems that nginx use the enterprise proxy to resolve localhost :/

(taiga)taiga@taiga:~/logs$ curl http://localhost:8001/api/v1/
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <meta http-equiv="Content-Type" CONTENT="text/html; charset=utf-8"> <title>ERREUR: L'URL demandée n'a pas pu être trouvé</title> <style type="text/css"><!--   /*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
...

but when i use the noproxy parameter I have acces to the api:

(taiga)taiga@taiga:~/logs$ curl -v --noproxy localhost, http://localhost:8001/api/v1/
* Hostname was NOT found in DNS cache
*   Trying ::1...
* connect to ::1 port 8001 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8001 (#0)
> GET /api/v1/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8001
> Accept: */*
>
< HTTP/1.1 200 OK
* Server gunicorn/19.3.0 is not blacklisted
< Server: gunicorn/19.3.0
< Date: Fri, 25 Mar 2016 12:53:17 GMT
< Connection: close
...

Is there's a way to say to nginx to resolve the localhost without questionning the proxy ?

[Wilfried Anuzet]

I just installed mailutils and came back with the first unicorn error.
It seems that the post still receive a 500 but I can't see something in the nginx error log. just the gunicor error:

==> gunicorn.stderr.log <==
ERROR:2016-03-25 13:52:49,003: Internal Server Error: /api/v1/userstories

Damned. Is someone have the same issue than me ?

[bene...@gmail.com]

I am also having similar problem with 500 error on user stories. 

I have setup local.py and it the emails send for changing user emails and basically everything else however, It seems that when I try to do do anything around an actual project it is sending user stories email to exa...@example.com. I can confirm this as I am using Sendgrid and it's the fact that it's rejecting the email that the system fails. 

I find that the post still updates the database. The weird thing is exa...@example.com is no where in my /admin/ and I am completely lost trying to fix this bug. 

[Luke Ashe-Browne]

I've had the same issue, setup new server, any modification to any card in a project results in 500 error, the server log is very hard to parse, and there's no clear indication of what is causing the error. 

I was wondering if it's the email component failing to send email, i saw some discussions around the google results that touched on this. I dont know how to completely deactivate the email configuration, i tried setting the backend value to '' empty string, that seemed to kill my django proecess completely. 

[Luke Ashe-Browne]

I just modified my local.py to include

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

this seems to have no effect and I still get TypeError: object() takes no parameters

[Alejandro Alonso]

Hello Luke,

I've had the same issue, setup new server, any modification to any card in a project results in 500 error, the server log is very hard to parse, and there's no clear indication of what is causing the error. 

Could you paste here the piece of log generated just when the 500 error is launched?.

Regards,

--

Alejandro Alonso Fernández   CIO & Co-founder alejandr...@kaleidos.net  | Tel: (+34) 627263808 | (+34) 913562995 www.kaleidos.net/FC8EAC/

[Luke Ashe-Browne]

posting here now. 

ERROR:2016-04-20 16:54:17,897: Internal Server Error: /api/v1/userstories/9

Traceback (most recent call last):

File "/home/luke/taiga-back/taiga/lib/python3.4/site-packages/django/core/handlers/base.py", line 149, in get_response

response = self.process_exception_by_middleware(e, request)

File "/home/luke/taiga-back/taiga/lib/python3.4/site-packages/django/core/handlers/base.py", line 147, in get_response

response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/luke/taiga-back/taiga/base/api/viewsets.py", line 105, in view

return self.dispatch(request, *args, **kwargs)

File "/home/luke/taiga-back/taiga/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view

return view_func(*args, **kwargs)

File "/home/luke/taiga-back/taiga/base/api/views.py", line 445, in dispatch

response = self.handle_exception(exc)

File "/home/luke/taiga-back/taiga/base/api/views.py", line 442, in dispatch

response = handler(request, *args, **kwargs)

File "/home/luke/taiga-back/taiga/base/api/mixins.py", line 199, in partial_update

return self.update(request, *args, **kwargs)

File "/home/luke/taiga-back/taiga/projects/userstories/api.py", line 113, in update

return super().update(request, *args, **kwargs)

File "/usr/lib/python3.4/contextlib.py", line 30, in inner

return func(*args, **kwds)

File "/home/luke/taiga-back/taiga/lib/python3.4/site-packages/django/db/transaction.py", line 267, in __exit__

connection.set_autocommit(True)

File "/home/luke/taiga-back/taiga/lib/python3.4/site-packages/django/db/backends/base/base.py", line 397, in set_autocommit

self.run_and_clear_commit_hooks()

File "/home/luke/taiga-back/taiga/lib/python3.4/site-packages/django/db/backends/base/base.py", line 623, in run_and_clear_commit_hooks

func()

File "/home/luke/taiga-back/taiga/events/signal_handlers.py", line 45, in <lambda>

emit_event = lambda: events.emit_event_for_model(instance, sessionid=sesionid, type=type)

File "/home/luke/taiga-back/taiga/events/events.py", line 78, in emit_event_for_model

data=data)

File "/home/luke/taiga-back/taiga/events/events.py", line 44, in emit_event

backend = backends.get_events_backend()

File "/home/luke/taiga-back/taiga/events/backends/base.py", line 59, in get_events_backend

return cls(**options)

TypeError: object() takes no parameters

[Alejandro Alonso]

Hello Luke,

It seems your instance is failing when trying to use the asynchronous notifications backend, check your settings/local.py, the attribute EVENTS_PUSH_BACKEND_OPTIONS should be pointing correctly to the backend. I assume you are trying to use rabbitmq as that backend, check if the user/password/path specified there is correct. 

To check if Taiga is failing just in that part comment that line so it tries to work in synchronous mode.

Regards,

--
Please help us keep the Taiga.io Community open and inclusive, follow our Code of Conduct:
https://github.com/taigaio/code-of-conduct/blob/master/CODE_OF_CONDUCT.md
---
You received this message because you are subscribed to the Google Groups "taigaio" group.
To unsubscribe from this group and stop receiving emails from it, send an email to taigaio+u...@googlegroups.com.
To post to this group, send email to tai...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/taigaio/8a5ba70e-4f46-482e-986e-21de86be3f15%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luke Ashe-Browne]

Thanks for that, seems to be the answer, I guess the line: 

backend = backends.get_events_backend()
File "/home/luke/taiga-back/taiga/events/backends/base.py", line 59, in get_events_backend

is the evidence that points to it, without familiarity i would never find this on my own. Should you not add exception handling with descriptive text to these failure cases? Would make outsider debugging much more straight forward.

Looking at the code, https://github.com/taigaio/taiga-back/blob/master/taiga/events/backends/base.py#L59 maybe wrap the cls(**options) call in a try catch block and reraise the exception after adding some error message text.

eg.

def get_events_backend(path:str=None, options:dict=None):
  .....  
  cls = load_class(path)
  try:
    backend = cls(**options)
  except TypeError as err:
    err.message=err.message+" - Check push backend configuration, [or something more informative here....] "
    raise
  return backend

[Alejandro Alonso]

I think you should be receiving some more logs complaining about the rabbitmq connection in your logs folder, did you check it?. Also if you try to launch "python manage.py runserver" just to test if django can boot properly should point errors in that connection.

Regards,

To view this discussion on the web, visit https://groups.google.com/d/msgid/taigaio/a209b5fa-bed0-4403-ad12-e42f433edc52%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luke Ashe-Browne]

Put the settings back in place and tried the runserver test.. 

~/taiga-back$ python manage.py runserver

Trying import local.py settings...

Trying import local.py settings...

Performing system checks...

System check identified no issues (0 silenced).

April 22, 2016 - 10:09:10

Django version 1.9.2, using settings 'settings'

Starting development server at http://127.0.0.1:8000/

Quit the server with CONTROL-C

no error's evident.

In terms of the local log folder? Where is that? i dont seem to have one. My logs are going to the syslog, i'm actually using systemd to launch my whole setup, journalctl shows me most of my logs which is where i got the previous stack trace. 

as a matter of interest for you guys, if you thing circus can be optionally replaced, here are my systemd unit files if it helps at all. also, is logging better off done with a setting in the unit file? Or in settings/local.py somewhere? 

taiga-django.service

[Unit]

Description=Gunicorn instance for Taiga kanban project management django application

After=network.target

[Service]

Type=simple

User=luke

Group=luke

WorkingDirectory=/home/luke/taiga-back

StandardOutput=syslog

StandardError=syslog

Environment="VIRTUALENVWRAPPER_SCRIPT=/usr/share/virtualenvwrapper/virtualenvwrapper.sh"

ExecStart=/bin/bash -c 'source /home/luke/taiga-back/taiga/bin/activate ;  /home/luke/taiga-back/taiga/bin/gunicorn -w 3 -t 60 --pythonpath=. -b 127.0.0.1:8001 taiga.wsgi'

TimeoutSec=20

[Install]

WantedBy=multi-user.target

taiga-events.service

[Unit]

Description=taiga events npm instance for Taiga kanban project management django application

After=network.target

[Service]

User=luke

Group=luke

WorkingDirectory=/home/luke/taiga-events

StandardOutput=syslog

StandardError=syslog

ExecStart=/usr/local/bin/coffee index.coffee

[Install]

WantedBy=multi-user.target

taiga-celery.service

[Unit]

Description=celery instance for Taiga kanban project management django application

After=network.target

[Service]

Type=forking

User=luke

Group=luke

WorkingDirectory=/home/luke/taiga-back

StandardOutput=syslog

StandardError=syslog

Environment="VIRTUALENVWRAPPER_SCRIPT=/usr/share/virtualenvwrapper/virtualenvwrapper.sh"

ExecStart=/bin/bash -c 'source /home/luke/taiga-back/taiga/bin/activate ;  /home/luke/taiga-back/taiga/bin/celery multi start worker -A taiga -c 4 --logfile=/var/log/celery/taiga-celery.log --pidfile=/var/run/celery/%n.pid'

ExecStop=/bin/bash -c 'source /home/luke/taiga-back/taiga/bin/activate ; /home/luke/taiga-back/taiga/bin/celery multi stopwait worker --pidfile=/var/run/celery/%n.pid'

ExecReload=/bin/bash -c 'source /home/luke/taiga-back/taiga/bin/activate ; /home/luke/taiga-back/taiga/bin/celery multi restart worker -A taiga -c 4 --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/taiga-celery.log --loglevel=INFO'

[Install]

WantedBy=multi-user.target

There is certaily some refinement of those needed, eg, i've not done execstops or reloads for django or npm, not sure i'm tracking pid's as you should with systemd yet. and not sure i need the VIRTUALENVWRAPPER_SCRIPT env variable at all.. I'll have to toggle it to test if it's needed or not. 

Anyway, right now the PUSH backend isn't working , have to diagnose that. 

[Alejandro Alonso]

You are right, there is no evidence, I thought something we had included something in the Taiga code 

If taiga-celery can't connect to rabbitmq you should see logs like the following:

[2016-04-22 13:00:00,017: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds...
[2016-04-22 13:00:02,026: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 4.00 seconds...
[2016-04-22 13:00:06,036: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 6.00 seconds...

I'm using log files for them so I know exactly where is the place to look:

stderr_stream.filename = /tmp/celery.stderr.log

With syslog you should find those logs too

If you find any concrete update you would like to do on the doc it would be great if you could send a pull requests with it: https://github.com/taigaio/taiga-doc

Regards!,

[Luke Ashe-Browne]

I have been banging my head against the wall on this, it works fine if i comment out the EVENTS_PUSH_BACKEND_OPTIONS.

But i've verified on the cli with:

curl -i -u taiga:password locahost:15672/api/whoami   and i get Not a management user(which is correct, would get Login failed if password is wrong)

so all is well. the systemctl list_user_permissions taiga   shows:

Listing permissions for user "taiga" ...

taiga   .*      .*      .*

With the EVENTS_PUSH_BACKEND_OPTIONS enabled,  attached to this post is the html output of the entire django debug output returned by the api request to update a ticket.

I really would like to see this problem solved, but i dont see any output anywhere else. 

i've tailed /var/log/celery/taiga-celery, i've journalctl -f -u taiga-django/taiga-celery/taiga-events    as per the unti files i posted above and only django is outputting anything.

there's no sign of an amqp failure in any of the logs, 

Is there any other way i can debug this? 

[Alejandro Alonso]

Hello Luke,

What's the value for your EVENTS_PUSH_BACKEND_OPTIONS setting?

I think this thread can be related to your problem: https://groups.google.com/d/msg/taigaio/maGHbisgYII/OVBzTAtxBQAJ

Regards, 

--

Please help us keep the Taiga.io Community open and inclusive, follow our Code of Conduct:
https://github.com/taigaio/code-of-conduct/blob/master/CODE_OF_CONDUCT.md
---
You received this message because you are subscribed to the Google Groups "taigaio" group.
To unsubscribe from this group and stop receiving emails from it, send an email to taigaio+u...@googlegroups.com.
To post to this group, send email to tai...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/taigaio/9e3cfc0f-f029-458f-8cfd-83b8e95b7487%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[g.g...@k-tech.it]

Hi all, not sure if this still of interest.

got same problem after upgrading to the latest stable release.

solved putting in my local settings ( $INST_DIR/taiga-back/settings/local.py) a consistent set of entries for EVENTS_PUSH_*  

EVENTS_PUSH_BACKEND = "taiga.events.backends.rabbitmq.EventsPushBackend"

EVENTS_PUSH_BACKEND_OPTIONS = {"url": "amqp://taiga:taiga_rmq@taiga:5672/taiga"}

the error

return cls(**options)

TypeError: object() takes no parameters

comes when using EVENTS_PUSH_BACKEND_OPTIONS valid for a the postgres backend class instead of rabbitmq. 

I noticed these entries in the common.py settings file:

common.py:EVENTS_PUSH_BACKEND = "taiga.events.backends.postgresql.EventsPushBackend"

common.py:# EVENTS_PUSH_BACKEND = "taiga.events.backends.rabbitmq.EventsPushBackend"

and the default is the one based on postgres, not rabbitmq.

after restart 

$circusctrl restart taiga

 

the URL  /api/v1/userstories/<id> worked fine.

HTH

Ciao

GG