Recursion during anonymization/modify

153 views
Skip to first unread message

John Roberts

unread,
Jun 24, 2021, 5:18:36 PM6/24/21
to Orthanc Users
Sebastien,

You mentioned that UI elements are handled recursively now after 1.9.3


Could you elaborate?  Is that only for calls to anonymize and only UI elements?

Are there other recursive aspects?  I couldn't find it mentioned in the documentation.

For example, I'm finding private tags buried by Siemens in their new enhanced DICOM format.  The private tags aren't at the top level, so I'm trying to figure out how to remove them recursively.  The modify command doesn't appear to catch them, I'm guessing because it doesn't recurse or I'm not employing it correctly.

I'm just now testing out 1.9.4 using the Osimis docker image.

Thanks,
John.

Sébastien Jodogne

unread,
Jun 25, 2021, 1:21:35 AM6/25/21
to Orthanc Users
Hello,

The "Full support of hierarchical relationships for UID tags during anonymization" feature corresponds to the rule "U" ("replace with a non-zero length UID that is internally consistent within a set of Instances") in Table E.1-1 "Application Level Confidentiality Profile Attributes" of the DICOM standard:

As can be seen in the source code of Orthanc (cf. method "DicomModification::RelationshipsVisitor::VisitString()"), the only kind of tags that are replaced correspond to tags whose VR is UI:

That being said, since Orthanc 1.9.4, you can also remove subsequences that are manually specified (and not just top-level tags as in Orthanc <= 1.9.3). The fields "Keep", "Remove" and "Replace" have indeed been extended to allow the same wildcards as the "dcmodify" command-line tool:

You can find examples in the integration tests "test_modify_subsequences":

Regards,
Sébastien-

John Roberts

unread,
Jun 25, 2021, 10:45:23 AM6/25/21
to Orthanc Users
Sebastien,

Thanks for the clarification.  I'll look into the use of subsequences to remove the nested proprietary tags.

John.

Salim Kanoun

unread,
Jun 27, 2021, 10:48:30 AM6/27/21
to Orthanc Users
Dear Sebastien, Dear All,

This question of sequence is terribly complex. 

In the dicom standard part 15 we have : 
"The Attributes listed in Table E.1-1 for each profile are contained in Standard IODs, or may be contained in Standard Extended IODs. An implementation claiming conformance to an Application Level Confidentiality Profile as a de-identifier shall protect or retain all instances of the Attributes listed in Table E.1-1, whether contained in the main dataset or embedded in an Item of a Sequence of Items. "

Which would mean that the same deidentification transformation should be applied to all PHI tags wathever in the main or in sequences.
This if we apply this rule, John's problem should diseapear as the in sequence private tag will follow the same de identification rule (removed or not as choosing in Orthanc payload api).
Is there a way to apply this rule with current Orthanc version ?

Then we have a warning 
"Some profiles do not allow selective protection of parts of a Sequence of Items. If an Attribute to be protected is contained in a Sequence of Items, the complete Sequence of Items may need to be protected."
So if I understood correctly if a sequence contains something that shouldn't be de identified we should not edit the whole sequence. This would mean that the we would need to escape the sequence from edition if one of the nested tag are in the Keep array. Do we have something like this (I think no). However I think the Keep property can fit for this, as if you know that a specific sequence should be protected you can specify it in the Keep options.

All of this nested tags are a nightmare and can be prone to dangerous side effects. The consequences of all these choices are pretty hard to calculate.

Sebastien, do you think it would be dangerous to apply the deidentification rule to all tags in the PHI list for first level or nested in sequence ?
Maybe the best is to keep the anon like this, and add an option for private nested tags, at least for these ones the consequences should be minor.

Best regards,

Salim 

John Roberts

unread,
Jun 28, 2021, 7:10:34 PM6/28/21
to Orthanc Users
Sebastien,

With regard to the examples in test_modify_sequences, I see the addresses are specified using the human readable terms.  For example,

'Remove' : [ 'ReferencedPerformedProcedureStepSequence', 'PerformedProtocolCodeSequence[0].CodeValue',  
                     'SharedFunctionalGroupsSequence[*].ReferencedImageSequence[*].ReferencedSOPInstanceUID',
                     'SharedFunctionalGroupsSequence[*].ReferencedImageSequence[1].ReferencedSOPClassUID',
                     'SharedFunctionalGroupsSequence[2].ReferencedImageSequence', # Inexistent tag ]

Is it possible to use the GGGG-EEEE type strings?  For example (not the same tags as above),

'Remove' : ['0021-1201',
                    '5200-9230[0].0021-1201'
                    '5200-9230[0].0018-9226[0].0021-11fe]

I have used the GGGG-EEEE scheme for top level tags before, but I wasn't sure about these indexed subsequence type entries.

John.
On Thursday, June 24, 2021 at 11:21:35 PM UTC-6 s.jo...@gmail.com wrote:

Sébastien Jodogne

unread,
Jun 29, 2021, 12:47:59 AM6/29/21
to John Roberts, Orthanc Users
Yes, you can use GGGG-EEEE. For instance, the following call works:

$ curl http://localhost:8042/instances/19816330-cb02e1cf-df3a8fe8-bf510623-ccefe9f5/modify -d '{"Remove":["0008-0080","0018,6011[0].0018-6028"]}'

Sébastien-


--
You received this message because you are subscribed to the Google Groups "Orthanc Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/bbabde36-fb06-4621-941b-0061b5fb31d3n%40googlegroups.com.


--

Sébastien Jodogne

unread,
Jul 6, 2021, 2:16:31 AM7/6/21
to Orthanc Users
Dear Salim,

I have just committed an important changeset that recursively applies anonymization to nested tags:

As you noticed, should an user want to protect a specific sequence, the "Keep" option can be used.

I would love to receive a quick feedback so that I can release Orthanc 1.9.5 this Thursday.

Regards,
Sébastien-

Salim Kanoun

unread,
Jul 6, 2021, 5:32:12 AM7/6/21
to Orthanc Users
Dear Sebastien,

I think the anonymization algorithm is really good. I think you apply the anonymization properly as specified by NEMA, it's hard to imagine all impacts in all sequences of all sorts of DICOM but we are following the rules (and I doubt that so many anonymization software take care of all these aspects). 

It will be too short to make extensive testing but the algorithm sounds really cool. When 1.9.5 will be release I will make several testing of PET/CT and RTSTRUCT. 

Thank you !

Best regards,

Salim

John Roberts

unread,
Jul 6, 2021, 10:10:26 AM7/6/21
to Orthanc Users
I can also give the recursion a test, assuming the Osimis docker images have the same release schedule.  I've built my docker framework around those.

I was testing 1.9.4 last week, but didn't realize that the recursion wasn't implemented just yet.  I've had my own recursion code in Lua to handle this issue, at least for UID.

John.

Sébastien Jodogne

unread,
Jul 6, 2021, 10:38:49 AM7/6/21
to Orthanc Users
Hello John,

You can test using the "jodogne/orthanc:latest" Docker image, that contains the latest nightly build.

Make sure to run "docker pull jodogne/orthanc:latest" in order to make sure to use the latest version.

Sébastien-

Sébastien Jodogne

unread,
Jul 8, 2021, 3:39:30 AM7/8/21
to Orthanc Users
Thanks for your feedback, Salim: I'm now launching the 1.9.5 release process.

Sébastien-


On Tuesday, July 6, 2021 at 11:32:12 AM UTC+2 salim....@gmail.com wrote:
Dear Sebastien,

John Roberts

unread,
Jul 8, 2021, 4:45:46 PM7/8/21
to Orthanc Users
I did not get to test this until today and saw that the osimis/orthanc image was also updated to 1.9.5.

I tried anonymizing a study and find some UID buried in sequences that reflect the original UID.

Could you clarify what recursion and consistency with the UID mean in this context?  We might be talking about different things and I haven't been clear.

I wrote my original Lua scripts to recurse the DICOM tags looking for any UID strings building a graph of their relationships.  For example, in MR, later series often reference earlier localizer series.  A given study may contain a reference UID completely separate from the current study.  These referencing UID are often buried in sequences, hence the need to recurse.

Following anonymization, I would reconstruct the same graph for the anonymized images.  Then, with the two graphs in hand, I would recurse the anonymized data and modify the UID to reflect the original relationships between series and studies.  In cases where references were made to a UID not in the original study, I simply replaced it with a new Orthanc generated UID at the appropriate study/series/instance level.

So I ended up with a recursively consistent anonymized set of UID that reflect the original relationships between the original images.

Is that the sort of thing the new recursive/consisent anonymization of UID is doing?  Or do you mean something else?

John.

Sébastien Jodogne

unread,
Jul 9, 2021, 3:32:52 AM7/9/21
to Orthanc Users
Hello,

I cannot answer you without a concrete example of "some UID buried in sequences that reflect the original UID."

In theory, Orthanc should recursively replace all the UIDs that must be anonymized according to the DICOM standard, even in nested sequences. Maybe you are considering a UID that is not included in the basic profile, cf. "Table E.1-1. Application Level Confidentiality Profile Attributes":

Please specify the UID you refer to, together with a minimal working example (sample DICOM file + cURL command-line):

Sébastien-

John Roberts

unread,
Jul 9, 2021, 11:45:09 AM7/9/21
to Orthanc Users
Ah, you're probably right.  I should be interpreting this within the context of the DICOM standard.  In my case, I'm likely dealing with nested UID not specifically requiring anonymization under the standard.

I'll take a look again with that in mind.  I'm sure I'll see the distinction in the results.  If I find a UID that should have been anonymized, I'll repost here.

As background, this is not an issue with Orthanc, but we have a particular privacy challenge at our institution where study date is considered to be protected health information (PHI), but manufacturers tend to generate UID incorporating the local clock time - effectively capturing the study date in all the UID.  Hence, I've had to find and replace all UID with non-clock based UID in order to eliminate the PHI.

John.

Salim Kanoun

unread,
Jul 19, 2021, 5:19:56 PM7/19/21
to Orthanc Users
Dear Sebastien,

I think I picked up a bug with Orthanc 1.9.5,

In my anonymization rules I have a special "Keep" for some Philips private tags : 7053,1000 and 7053,1009. For philips tags those tags even in the private group are in fact essential for PET/CT quantification.

So in Orthanc I have these tags in my Keep anonymization rule.
For now these tags were kept but with the new release they are not. 

I there a bug making the keep private tag overriding the keep rule ? The Keep rule should be prioritary compare to the private tag rule as some private tags are sometimes important. 

Best regards,

Salim 

Salim Kanoun

unread,
Jul 19, 2021, 6:02:10 PM7/19/21
to Orthanc Users
Dear Sebastien,

In fact the issue might not be new, I failed to find a previous version where the Keep parameter escape private tags for desctruction (but I think it was doing it initially).

Is it possible to get this priority that so manually defined "Keep" tag are applied even for private tags ?

Best regards,

Salim  

Salim Kanoun

unread,
Jul 19, 2021, 6:48:45 PM7/19/21
to Orthanc Users
Sorry forget everything, I'm wrong in my code before sending image to Orthanc,

Sorry !

Reply all
Reply to author
Forward
0 new messages