year out of range

[t.c....@gmail.com]

After migration to a new Ubuntu server (openrem version still 0.10.0, but newer pip packages) I started getting a "year out of range" error during one of my xlsx exports. On the old machine the export is ok.

Assuming this has something to do with timestamps in the postgres database I compared the remapp_generalstudymoduleattr tables between the two servers and found some entries on the new server (with same id) having study_workload_chart_time of the form 1899-12-31-* while on the old server these were all 1900-01-01-*. Could this be the issue? What's the purpose of this column? Can I edit/wipe these values? Any suggestions on how to fix this problem?

thanks! 

Tim de Wit

[2023-05-23 16:09:30,782: ERROR/ForkPoolWorker-2] Task remapp.exports.rf_export.rfxlsx[80671a66-f622-4b2b-82db-230c3cf247aa] raised unexpected: ValueError('year is out of range',)
Traceback (most recent call last):
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "remapp/exports/rf_export.py", line 296, in rfxlsx
    book, sheetlist = generate_sheets(e, book, protocolheaders, modality=u"RF", pid=pid, name=name, patid=patid)
  File "remapp/exports/export_common.py", line 164, in generate_sheets
    for s in events:
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/django/db/models/query.py", line 162, in __iter__
    self._fetch_all()
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/django/db/models/query.py", line 254, in iterator
    for row in compiler.results_iter(results):
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 797, in results_iter
    for rows in results:
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1152, in cursor_iter
    sentinel):
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1151, in <lambda>
    for rows in iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
  File "/var/dose/veopenrem/local/lib/python2.7/site-packages/django/db/utils.py", line 105, in inner
    return func(*args, **kwargs)
ValueError: year is out of range

[dpla...@gmail.com]

Dear Tim,

The  study_workload_chart_time database fileld is used when calculating workload charts in the workload_chart_data method of the chart_functions.py file. The date part of the field is not used, and is set to 1/1/1900 by the extractor routines. The time part of the field corresponds to the study time.

I don't understand why they have become 1899-12-31-*. I'm not sure how to update them all to be 1900-01-01-*, but doing so should make the ValueError go away.

Thanks,

David

[dpla...@gmail.com]

I should also say that the study_workload_chart_time is no longer needed by OpenREM 1.0

[t.c....@gmail.com]

Thanks!
In my test-environment I managed to change these entries to 1900-01-01-* but that didn't change anything..
Next I created an OpenREM 1.0 installation (native linux install) and migrated my production database to it. Same problem, slightly different error message:

Traceback (most recent call last): File "/var/dose/veopenrem3/lib/python3.10/site-packages/openrem/remapp/tools/background.py", line 148, in run_as_task func(*args, **kwargs) File "/var/dose/veopenrem3/lib/python3.10/site-packages/openrem/remapp/exports/rf_export.py", line 397, in rfxlsx book, sheetlist = generate_sheets( File "/var/dose/veopenrem3/lib/python3.10/site-packages/openrem/remapp/exports/export_common.py", line 243, in generate_sheets for s in events: File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/models/query.py", line 280, in __iter__ self._fetch_all() File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/models/query.py", line 51, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1208, in execute_sql return list(result) File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1646, in cursor_iter for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel): File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1646, in for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel): File "/var/dose/veopenrem3/lib/python3.10/site-packages/django/db/utils.py", line 97, in inner return func(*args, **kwargs) ValueError: year -1 is out of range

Op woensdag 24 mei 2023 om 12:11:24 UTC+2 schreef dpla...@gmail.com:

[Ed McDonagh]

Does it happen no matter what the exported studies? Does it happen just for RF?

--
You received this message because you are subscribed to the Google Groups "OpenREM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrem+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/f90f06f1-2c6a-4655-929d-b0263d341f05n%40googlegroups.com.

[t.c....@gmail.com]

So far it seems to be happening only for one specific RF system... several other RF systems didn't give any problems. 

Op woensdag 24 mei 2023 om 12:46:30 UTC+2 schreef Ed McDonagh:

[dpla...@gmail.com]

Hi Tim,

I wonder if there are missing or null study_date entries in the remapp_generalstudymoduleattr database table for the modality that is causing the export issue?

Kind regards,

David

[Ed McDonagh]

If you can isolate the issue Tim, we can add a try except to allow it to work or to add a friendlier error message.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/9bbb9f10-757a-4939-b17b-4f5918a95b99n%40googlegroups.com.

[t.c....@gmail.com]

I managed to isolate the problem, see attached screenshot... the second entry seems to be the culprit (without milliseconds; probably due to some internal conversion somewhere from 33.000 to 33 during the insert). Below db entries are identical between the old and new server (apart from study_workload_chart_time which I think isn't the problem, considering the problem is still present with OpenREM 1.0) so I suspect the issue is related to the postgres version or package versions. Here are a few differences:

Ubuntu: old=Ubuntu 16.04.6 new=18.04.6 (OpenREM 1.0 test-server: 22.04)

postgres:  old=9.5  new=10

Django: old=1.8.14 new=1.8.19
psycopg2: old=2.6.1 new=2.8.6 (called psycopg2-binary on latter)

After editing the second row in the database (appending .001 everywhere) the error is still not resolved...
As for the  study_workload_chart_time:

old: 1900-01-01 12:07:33+00:19:32

new:  1900-01-01 11:48:01.001+00

Op woensdag 24 mei 2023 om 16:40:23 UTC+2 schreef Ed McDonagh: