Why have complex Lua scripts in Orthanc?

[John Roberts]

Hi all,

   Alain and Sebastien have both mentioned in various threads here regarding Lua that the Lua interpreter was never intended to be used for complex scripting.  I completely respect that assertion and understand that while developing this great system, Orthanc, complex Lua support likely falls on the low end of their priorities.  Further, complex Lua support is really tangential to their aims and after all, ultimately provides hooks into the same API we could call from external programs.

   All that said, I thought I would explain why I find complex Lua scripts so useful and why I have focused a lot of my work with Orthanc there.  I wanted to see what others thought about complex Lua scripting and maybe whether anyone has insights or encouragement for moving to either plugin development or external (ex. python) based interactions with Orthanc.

   Probably the main reason I have developed complex Lua scripts has been that they leverage the "server" aspect of Orthanc. 

   My background is in medical imaging research.  I've done lots of programming over my career in many languages on multiple platforms, but have for the most part focused on data analysis with data management subservient to analysis.  I tend to avoid developing sophisticated GUIs and/or service-like applications because they are mostly outside my experience and the effort required overwhelms any gains I might make in the data analysis.

    I think of GUIs and services as involving a lot of asynchronous event handling, something that requires a different mindset than the more linear flow of data analysis.

   Because the embedded Lua hooks into the Orthanc server by way of the "stable" age parameters and incominghttprequest filters, I find it more attractive to use Lua directly rather than the implement an event driven service elsewhere that would interact with the Orthanc.

   Ignoring the plugin approach, if I were to use the restful API from an external application, wouldn't I need to essentially develop an event driven service?  One that would periodically check Orthanc for changes and then act accordingly?

   This has been the attraction of using the embedded Lua interpreter - not having to design my own event driven service to accomplish the same thing.  It's the same reason I've been quite happy with the stock Explorer web interface - not having to write a GUI/server app for my users (fellow medical image researchers).

   I admit that some day I may need to switch to an external app using the API or develop a plugin, but I thought I'd offer some insight into why this particular medical researcher went the direction of complicated Lua scripts.  In a way, the Orthanc developers have made it almost too easy for a non GUI/service coder like myself.

   I'd be interested in anyone else's thoughts. 

Cheers,
John.

[David Platten]

I'm a hospital-based medical physicist who works with x-ray imaging equipment. I've recently discovered Orthanc, and am using it as a DICOM store for a patient dose management system (OpenREM, http://www.openrem.org, http://docs.openrem.org/en/latest/). I'm using a Lua script with Orthanc in order to automatically populate OpenREM with the incoming DICOM data, and also to reject certain data. I've recently written a document to help other OpenREM users set up Orthanc for the same purpose - this includes an example Orthanc Lua script (http://docs.openrem.org/en/latest/netdicom-orthanc-config.html). I'm not sure whether my Lua script falls into the `complex` category though.

Regards,

David

[John Roberts]

Thanks, David for the input.

I'm not sure either what qualifies as "complex" here, though I suspect my scripts fall on the complex end of things.

In my case, we use Orthanc to completely anonymize our images while maintaining longitudinal visits by the same subject.  This presents a number of challenges that I have to address with Lua scripts and how I run Orthanc:

• Run Orthanc in conjunction with a PostGreSQL back end database. 
  
• Employ Lua SQL modules to map original subjects/studies to anonymized subjects/studies with calls to the PostGreSQL over the network.
• Add javascript driven web pages to let me tweak the anonymization parameters
  
• Generate some additional HTML pages to serve the "lookup" table between original and anonymized subjects/studies
  
• Most of our vendors create UID with the exam date embedded.  Our institution considers exam date to be protected health information (that must be replaced with anonymized substitutes).  DICOM employs UID among other things to connect separate series with each other - for example, where later series refer to the coordinate space established by an earlier localizer.  To address this, I wrote a rather sophisticated set of functions in Lua to recursively search the DICOM tags before/after anonymization and reconnect all series appropriately following anonymization with new anonymized UID.
• Anonymize all time stamps
• Recursively find all time and date stamps
• Generate a one time random time shift (one per subject) and apply to all dates/times
• Send email updates when anonymization is triggered and completed
  

All of that requires a number of modules not bundled with the embedded Lua interpreter as well as a complex series of calls to modify and anonymize with variations on what is to be Replaced, Removed, or Kept.

John.

[Thibault Nélis]

On Tue, 2018-06-12 at 11:54 -0700, John Roberts wrote:
> Ignoring the plugin approach, if I were to use the restful API
> from an external application, wouldn't I need to essentially develop
> an event driven service? One that would periodically check Orthanc
> for changes and then act accordingly?

Yes, precisely. The "changes" resource ("/changes") is intended to
provide facilities for most use-cases.

http://book.orthanc-server.com/users/rest.html#tracking-changes

The rest of the HTTP API will often be useful of course, as will the
Orthanc SDK for creating new plugins for even more use-cases (e.g.
intercept various requests to call-out to other libraries, programs or
even services via web-hooks). Finally, Orthanc can also be used as a
framework to create other programs using its components in more
flexible ways.

> I admit that some day I may need to switch to an external app
> using the API or develop a plugin, but I thought I'd offer some
> insight into why this particular medical researcher went the
> direction of complicated Lua scripts. In a way, the Orthanc
> developers have made it almost too easy for a non GUI/service coder
> like myself.
>
> I'd be interested in anyone else's thoughts.

"Complex" in most discussions on this topic usually refers to many
things. Consider the following: maintainability, language and
ecosystem, efficient testing facilities, available APIs. Sometimes it
also refers to the load incurred on the Orthanc server; consider how
long scripts will hold up threads and their impact at scale (thread
pool starvation, increased memory pressure, etc).

If none of these considerations (nor any considerations I haven't
thought of but which others will surely do) apply to your scenarios,
I'm sure your use of Lua scripts is perfectly fine.

In particular, I don't think "program length" has much impact on its
own. It often correlates with maintainability issues of course (and
likely other relevant considerations), but not always.

HTH,
--
Thibault Nélis <t...@osimis.io>
Osimis

[John Roberts]

Hi Thibault,

Your points are well taken.  I've periodically bumped up against problems of my own making in terms of maintainability and scaling.  It doesn't help that my scripts have grown somewhat organically as new requirements arise - as opposed to an application based on systematic design, coding and production.

I do think I am in a niche user's group, that of medical imaging researchers, especially those needing to work with anonymized images because of regulatory requirements.  I might have a combined store of 10s to 100s of thousands of images, but they are acquired and processed relatively slowly and so don't present large computational demands on the Orthanc.  My Lua script approach would likely rapidly fall apart in a more clinical setting.

One point that I did not make earlier is to note the tradeoff of time invested in open source software.  On tight research budgets, open source is an attractive way to go, but you always have to consider things like how long the particular software is likely to be supported, what sort of documentation exists, what the payoff is for invesment in it, etc.  You don't want to invest too much if the project is likely to go away.

From my perspective as a researcher, I can justify the time to write a Lua script to "get the job done" but probably not to write a full external application using the API.  It's a credit to your project that you have such an extraordinarily stable system and the kind of extensive documentation that enables a tangential user like myself to derive so much benefit with relatively little effort.

Since we shared our experience with Orthanc with our fellow researchers, I have been inundated with requests to set up additional Orthancs for them to use to anonymize data for their own clinical research.  Given the regulatory framework in the States, there is a pent up demand for anonymization tools that the vendor PACS stations and other solutions apparently don't satisfy. 

Most of our clinical systems, for example, cannot manage anonymization in a longitudinal context - at least without time consuming manual intervention.  Researchers made do with various other tools (ex. Osirix), but again found these methods time consuming.  Systems like xNAT require a lot of investment and dedicated support that can't be justified under these small cohort studies that the clinicians run.

John.

[Bryan Dearlove]

For me, I have used Orthanc and Mirth Connect together to better manage flow. For example using Orthanc to initiate events (Study Stable) to communicate to Mirth Connect, where I use Mirth Connect for most of the "complex" actions, such as prior fetching C-moves, requesting study demographics ti populate systems , etc. So far this method has worked well for me as within Mirth Connect you can control any aspect of Orthanc via the REST API, with built in error handling, etc. I can provide more info if interested. 

[Sébastien Jodogne]

Dear all,

Thanks you all for this very interesting discussion!

I have decided not to interfere with this thread until now, in order to let the Orthanc users express their insight.

As far as I'm concerned, I really understand the interest of Lua scripting because, as expressed by John, it allows to automate DICOM tasks by bundling all intelligence into one single software. Historically, avoiding to launch a separate, external script to drive Orthanc was the reason I introduced a Lua routing engine in Orthanc 0.8.0.

At this point, one must carefully consider two facts:

1. The "deprecated" part of the Lua engine is the "SendToPeer()", "SendToModality()", "Modify()" and "Delete()" primitives (http://book.orthanc-server.com/users/lua.html#important-remarks-about-auto-routing). From my point of view, it is perfectly fine to run Lua scripts if it meets your requirements, and if you avoid these deprecated primitives. You have the guarantee that Lua will never be removed from the core of Orthanc.
2. Until Orthanc 1.4.1, deadlocks could arise if you were using complex Lua scripts involving modification/anonymization/upload interactions through the REST API (https://bitbucket.org/sjodogne/orthanc/issues/25/deadlock-with-lua-scripts). Such deadlocks should now be fixed, but we are still looking for validation by the Orthanc community. This explains my own prudence regarding complex Lua scripting.

That being said, the question is: What improvements do you expect wrt. the built-in Lua engine?

As far as I'm concerned, given the fact that Lua has full access to the REST API of Orthanc, I don't see much features to add. I would like to ask you to create kind of a "wishlist", that could be stored in the dedicated "OrthancContributed" repository:

https://github.com/jodogne/OrthancContributed/tree/master/Documentation

Such a "wishlist" document could serve as a basis to either rework the Lua engine, or to create a separate plugin that would provide a "version 2" Lua scripting engine. The nice thing is that the Orthanc plugin SDK allows third-party developers such as you to create a new environment that better fills their workflow, without having to discuss the development with the core Orthanc team:

http://book.orthanc-server.com/developers/creating-plugins.html

I also remember that, even though I personally like Lua, the Orthanc project has very sparse resources on its own, which obliges us to focus on higher-priority tasks. If you find Orthanc useful, you are all warmly invited to contribute by creating such a "Lua v2 plugin".

Kind Regards,

Sébastien-