can't terminate task /getting server error (500)

[Sergii Luchkovskyi]

Hi,

I installed 1.0.0b2 version (Native Linux installation) on Ubuntu server 22.04 and was able to send data directly from modality and retrieve from PACS.

But when I tried to query seems too many studies, server runs out of memory and freeze.

Had to force reboot server. And terminated all tasks, except one.

Now there is one running task, which I can't terminate.

When i am trying to terminate it, I am getting server error (500).

All new tasks stuck because of the last one, so cant import data anymore.

Is there way to fix this?

Thanks,

Sergii

[Ed McDonagh]

Hi Sergii

Did you reboot the server, or restart the services? If you rebooted the services, did you restart redis?

Kind regards

Ed

--
You received this message because you are subscribed to the Google Groups "OpenREM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrem+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/91743ffc-e5ef-449f-8a47-6c69655db2b8n%40googlegroups.com.

[Sergii Luchkovskyi]

Hi Ed,

Thank for answer.

I just rebooted the server, and checked status of nginx, gunicorne, and orthanc, they all live and running.

Best regards

Sergii

пʼятниця, 3 листопада 2023 р. о 14:53:13 UTC+2 Ed McDonagh пише:

[Ed McDonagh]

Do you get any logs in nginx? If you change the debug mode to true do you get any clues?

https://docs.openrem.org/en/latest/trouble500.html#non-docker-installs

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/9d9dd8cd-7517-4788-af49-79d8301aa455n%40googlegroups.com.

[Sergii Luchkovskyi]

Hi Ed,

Nginx error logs has only this:

2023/11/03 07:53:14 [error] 1127#1127: *13430 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.1.1.158, server: openrem-server, request: "GET /openrem/dicom/querysummary HTTP/1.1", upstream: "http://unix:/tmp/openrem-server.socket/openrem/dicom/querysummary", host: "172.16.2.203", referrer: "http://172.16.2.203/openrem/dicom/querydetails/18"
2023/11/03 07:58:30 [error] 1127#1127: *13430 upstream timed out (110: Unknown error) while reading response header from upstream, client: 10.1.1.158, server: openrem-server, request: "GET /favicon.ico HTTP/1.1", upstream: "http://unix:/tmp/openrem-server.socket/favicon.ico", host: "172.16.2.203", referrer: "http://172.16.2.203/openrem/dicom/querysummary"

When change to debug mode, i got following results, when trying to terminate task:

Environment:
Request Method: GET
Request URL: http://172.16.2.203/openrem/tasks/tasks/abort_task/3142f993-a908-4369-80a9-707e795749cc/

Django Version: 3.2.22
Python Version: 3.10.12
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin',
 'remapp',
 'django_filters',
 'django.contrib.humanize',
 'solo',
 'crispy_forms',
 'django_js_reverse',
 'huey.contrib.djhuey')
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback (most recent call last):
  File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/dose/veopenrem3/lib/python3.10/site-packages/openrem/remapp/views_admin.py", line 2308, in task_abort
    task = get_object_or_404(BackgroundTask, uuid=task_id)
  File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/shortcuts.py", line 76, in get_object_or_404
    return queryset.get(*args, **kwargs)
  File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/models/query.py", line 439, in get
    raise self.model.MultipleObjectsReturned(

Exception Type: MultipleObjectsReturned at /openrem/tasks/tasks/abort_task/3142f993-a908-4369-80a9-707e795749cc/
Exception Value: get() returned more than one BackgroundTask -- it returned 2!

Sergii

пʼятниця, 3 листопада 2023 р. о 14:37:32 UTC Ed McDonagh пише:

[Ed McDonagh]

Hi Sergii. That gives me what I need.

Somehow there are more than one entries in the BackgroundTask table with the same uuid value.

I have created issue 1009 to fix the issue for the future, but in the meantime you might like to:

Activate your virtual environment:

. /var/dose/veopenrem3/bin/activate

Move to the openrem python package folder:

cd /var/dose/veopenrem3/lib/python3.10/site-packages/openrem/

Start a Django Shell:

python manage.py shell

Find and delete the task(s) that are causing the issue:

from remapp.models import BackgroundTask
tasks = BackgroundTask.objects.filter(uuid__exact="3142f993-a908-4369-80a9-707e795749cc")
tasks.count()
# Expecting an answer of 2
tasks.delete()

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/61175ada-928b-4e07-8060-e6b07788d8ean%40googlegroups.com.

[Sergii Luchkovskyi]

Dear Ed,

Thank you for the solution and overall for developing and maintaining this awesome project.

Best regards,

Sergii

ср, 8 лист. 2023 р. о 00:32 Ed McDonagh <e...@openrem.org> пише:

[CS Tima]

Hi Ed,

thanks for looking into this. I am ancountering the same issue running 1.0.0b2 in a DOCKER environment on Ubuntu. 

How could I delete the tasks there? I haven`t been able to translate your workaround for the native installation to the docker type installation according to the installation guide.

Excuse if this question is very basic. 

[CS Tima]

The solution was pretty easy:

open terminal where you performed docker-compose up -d

open shell: docker-compose exec openrem python manage.py shell

use this commands from Ed, replacing with your task ID:

from remapp.models import BackgroundTask
tasks = BackgroundTask.objects.filter(uuid__exact="0b0d70af-fb57-40b7-a270-7dfe49f63168")

tasks.count()
# Expecting an answer of 2
tasks.delete()

[Ed McDonagh]

Thanks for following up - I started to compose a reply this morning but then I was overtaken by events!

Kind regards

Ed

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/cf4539ed-2288-4db0-920e-67a061049cdcn%40googlegroups.com.