TM, DA converters?

[Matthew Brett]

Hi,

I am right in thinking there are no utility functions I missed in
pydicom to convert TM and DA strings to times, dates, as in:

0008, 0023) Content Date DA: '20091020'
(0008, 0030) Study Time TM: '172423.578000'

going to a date, time representation in Python?

Cheers,

Matthew

[Darcy Mason]

Yes, that's right, there is nothing built-in currently in pydicom. I think I didn't bother originally because I didn't think there would be much need for working with them. Do you have a particular case where it would be helpful?

 

[Matthew Brett]

Hi,

More an idea for a case - but don't worry - just checking - I will
write them and submit a patch if I end up using them,

Cheers,

Matthew

[Romain Valabregue]

Date and time are very useful ! At least for my use.
We facture our exam based on scan time. (So Serie Time is need to compute exam duration, but the series duration is not always easy to get)
For Qaulity Control over a large number of dicom, date is needed to detect periods of scanner perturbation ...

cheers

Romain

 

[Dimitri]

DA and TM are currently converted to string as can be seen in values.py:

converters = {

    [...]

    'DA': convert_string,

    'TM': convert_string,

    [...]

}

I could contribute specific conversion functions convert_DA, convert_TM and convert_DT. Should they convert DA, TM and DT into Python datetime.date and datetime.time objects or a mere time.struct_time sequence? What would you suggest?

[Darcy Mason]

On Monday, March 17, 2014 6:48:02 AM UTC-4, Dimitri wrote:

...

I could contribute specific conversion functions convert_DA, convert_TM and convert_DT. Should they convert DA, TM and DT into Python datetime.date and datetime.time objects or a mere time.struct_time sequence? What would you suggest?

I'm open to suggestions on this one.  My preference, by the way, would be that this be an optional conversion -- perhaps the dicom.config settings could be used to specify datetime conversion, with the default to None, no conversion, as it is now.  In which case, there could be an option for datetime, and another option for time.struct_time.

My reason for it to be optional is due to speed -- although in most cases there will not be many dates, we've heard many cases where people are looking for the fastest possible reading, to sift through a large number of files for specific information.

[Dimitri]

I think it's best to stick to datetime because DA and TM cannot be expressed as time.struct_time anyway:

• DA → datetime.date
  
• DT → datetime.datetime
  
• TM → datetime.time
  

I'm not certain how this should be handled. Create new classes in valuerep.py that could be converted to datetime classes using a specific conversion function? Create new functions in values.py (for example convert_DT_datetime) that would be called instead of convert_string depending on dicom.config settings?

[Dimitri]

I have added a ticket to contribute some code that converts DA, DT and TM tags into objects that inherit respectively from:

• DA → datetime.date
  
• DT → datetime.datetime
  
• TM → datetime.time
  

The code is not fully tested yet, it's there as a proof of concept, feedback would be appreciated:

https://code.google.com/p/pydicom/issues/detail?id=142

[Dimitri]

The code posted above breaks compatibility.

The value of DA, DT and TM data elements used to be a string and that's what existing client code expects:

>>> import dicom

>>> dataset = dicom.read_file('file.dcm')

>>> type(dataset.AcquisitionDate)

<type 'str'>

>>> 

The above code converts DA data elements to dicom.valuerep.DA which inherits datetime.date and cannot be used as string:

>>> import dicom

>>> dataset = dicom.read_file('file.dcm')

>>> type(dataset.AcquisitionDate)

dicom.valuerep.DA

>>> len(dataset.AcquisitionDate)

 Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

TypeError: object of type 'DA' has no len()

>>>

I wish pydicom had returned DA, DT and TM as datetime objects from the start. It's too late now, compatibility cannot be broken, can it? here are alternatives I have thought of:

1. DA, DT and TM data elements could be converted to a class inheriting both str and a datetime object. However it is not possible to inherit from both these classes because multiple bases have instance lay-out conflict.
   
2. DA, DT and TM data elements could be converted to classes inheriting str only. Such classes would provide additional date(), datetime() and time() methods respectively, that would return corresponding datetime objects.
   
3. DA, DT and TM data elements could be converted to a mere string as is the case today. External Functions could be provided to convert such strings to datetime objects.
   

What would you recommend?

[Darcy Mason]

Hi Dimitri,

I looked through your code and generally speaking, I like it.  I see that the conversion of DT in particular requires some care, so having this code worked out and added to pydicom could be a benefit for users.  The question, as you posted, is how to fold it together, which parts are automatic and which are user-selectable.

On Sunday, April 20, 2014 7:46:32 AM UTC-4, Dimitri wrote:

The code posted above breaks compatibility.

...

I wish pydicom had returned DA, DT and TM as datetime objects from the start. It's too late now, compatibility cannot be broken, can it?

There is one possibility for breaking compatibility.  The current plan is to prepare a pydicom release with the latest bug-fixes, and start new with a python3-only version of pydicom, which will include some backwards incompatibilities.  However, I'm not convinced it is still needed in this case. While some people may want all dates and times converted, I'm not sure the costs in time and dependencies, etc. is enough benefit. Pydicom's design philosophy was to be as light and dependency-free as possible, parsing files for users and then leaving the details of manipulating dicom objects to the users. We've also learned in the automatic Decimal conversion added in 0.9.7 and then later retracted, that any extra classes can add a lot of overhead.

My preferred path would still be to have it as a configurable option.  For example:

import dicom

dicom.config.dates = 'datetime'

ds = dicom.read_file(...)

...

Would allow you to work with automatically converted dates, using the code which you posted (with an extra if block to check the config before the conversion). It would also allow future change to better date/datetime libraries, if such were to emerge.

I have another thought as well ... if filewriter code was updated to nicely accept date types in addition to str, then one could do something like:

ds.StudyDate = convert_DA_string(ds.StudyDate)

wherever needed. This could be extended to convert all dates/datetimes using the Dataset walk() function too.  This solution is not as elegant, but it has the advantage of being explicit. The config option is a single line somewhere, and would not be so obvious to someone trying to follow the code's logic.

 

here are alternatives I have thought of:

1. DA, DT and TM data elements could be converted to a class inheriting both str and a datetime object. However it is not possible to inherit from both these classes because multiple bases have instance lay-out conflict.
   
2. DA, DT and TM data elements could be converted to classes inheriting str only. Such classes would provide additional date(), datetime() and time() methods respectively, that would return corresponding datetime objects.
   
3. DA, DT and TM data elements could be converted to a mere string as is the case today. External Functions could be provided to convert such strings to datetime objects.
   

What would you recommend?

So I guess my recommendation is something like your option 3, but with the ability to have it automatic with the use of the dicom.config setting. Option 2 could be interesting, but my experience in timing tests is that built-in classes such as str become much slower when subclassed, and are less explicit for someone following the code.

As always, I'm interested in others' thoughts as well...

Darcy

[Eli Stevens (Gmail)]

On Sun, Apr 20, 2014 at 4:46 AM, Dimitri <dimitri.pa...@gmail.com> wrote:
> The code posted above breaks compatibility.

...

> I wish pydicom had returned DA, DT and TM as datetime objects from the
> start. It's too late now, compatibility cannot be broken, can it?

Compatibility can certainly be broken; it has in the past. We're
still running 0.9.6 because something in 0.9.8 didn't work for us out
of the box, and we haven't felt the need to upgrade (sadly, I don't
recall what it actually was that was giving us trouble).

I'm not sure if the project uses Semantic Versioning[1] or not, but
this could be handled by announcing that the project will be using SV,
cutting a 1.0.0 with the desired behavior, and clearly stating what
the compatibility breaking change was in the release notes (see points
4 and 5 on the page linked below).

I would recommend that if if a change like this is going to go into
the project, that the pytz[2] project be looked into to make sure that
the problems that it solves aren't going to be issues for this
project. I'm not an expert, but the engineer who was the expert at my
last position swore by pytz for fixing whatever the deficiency was in
the stdlib datetime implementation.

Cheers,
Eli

[1] - http://semver.org/
[2] - http://pytz.sourceforge.net/

[Matthew Brett]

Hi,

On Mon, Apr 21, 2014 at 7:58 AM, Darcy Mason <darcy...@gmail.com> wrote:
> Hi Dimitri,
>
> I looked through your code and generally speaking, I like it. I see that
> the conversion of DT in particular requires some care, so having this code
> worked out and added to pydicom could be a benefit for users. The question,
> as you posted, is how to fold it together, which parts are automatic and
> which are user-selectable.
>
> On Sunday, April 20, 2014 7:46:32 AM UTC-4, Dimitri wrote:
>>
>> The code posted above breaks compatibility.
>> ...
>> I wish pydicom had returned DA, DT and TM as datetime objects from the
>> start. It's too late now, compatibility cannot be broken, can it?
>
>
>
> There is one possibility for breaking compatibility. The current plan is to
> prepare a pydicom release with the latest bug-fixes, and start new with a
> python3-only version of pydicom, which will include some backwards
> incompatibilities.

On Python 2 / 3 - the strong current consensus seems to be that python
2.7 will be around for a long time - say until 2020, and that the best
way to support this is having a common codebase for Python 2 and
Python 3:

http://ondrejcertik.blogspot.com/2013/08/how-to-support-both-python-2-and-3.html

The common codebase means - using a compatibility routine such as
six.py, and running code directly in Python 2 or Python 3 without the
use of 2to3.

Projects I know of currently doing that are:

numpy
scipy
matplotlib
scikit-learn
scikit-image
sympy
nibabel (my package)

I highly recommend it ...

Cheers,

Matthew

[Dimitri]

Hi,

Le lundi 21 avril 2014 18:24:09 UTC+2, Eli Stevens a écrit :

...

I would recommend that if if a change like this is going to go into
the project, that the pytz[2] project be looked into to make sure that
the problems that it solves aren't going to be issues for this
project.  I'm not an expert, but the engineer who was the expert at my
last position swore by pytz for fixing whatever the deficiency was in
the stdlib datetime implementation.

The pytz module is about timezones but there are no timezones in the DICOM standard. The approach of the DICOM standard is simple:

• either time is "naive", without any indication whether it is local time or UTC time,
  
• or time is given with a fixed offset to UTC.
  

In either cases, no timezones are involved (in the sense of complex timezones with DST changing over the years). The standalone datetime module can be used in the first case. In the second case dateutil.tz.tzoffset can be used to represent a fixed offset to UTC. The pytz module is not required and does not help in any way within pydicom.

The pytz module may help in client code with specific knowledge of a specific scanner:

• Are DICOM files stamped using local time?
  
• If so, which is the timezone of the scanner?
  
• What happens when clocks change due to DST? Does the scanner change timestamps accordingly? In that case an MRI sequence may start at 02:05 and finish at 1:10 the same day!
  

This specific knowledge is outside the scope of the DICOM format and hence of the pydicom module.

Dimitri

[Darcy Mason]

I just wanted to say thanks to everyone for their comments.

I'll certainly consider the six module route for "pydicom 1.0" and python 3, even if there are also backwards incompatible changes with the major version number change.  It doesn't look like there are that many differences to deal with, and six.py is a single file that can be included with the distribution.  So long as it doesn't overly complicate the code (which it looks like it shouldn't) then it is quite reasonable to go that route.

Darcy