Docker instance: transition from Trusty to Xenial base Ubuntu image

[John Roberts]

I've been working with an older Docker version of the Osimis-orthanc build that was built on top of the Trusty Ubuntu Docker image.

I've developed a fairly sophisticated Lua server side script and in the process have installed additional Lua models vai the Debian apt-get system.

Unfortunately, I'm running into problems with the latest Osimis-Orthanc (I'm looking at osimis/orthanc:18.1.4) that's built on top of the Xenial Ubuntu image.

I can apt-get my additional lua modules (lua-sql-postgres and lua-socket) and they appear to install without problems.

However, at run time, when Orthanc attempts to load the modules (ex. socket.smtp), it is searching in the wrong path and generates a lot of errors:

    no field package.preload['socket.smtp']
    no file './socket/smtp.lua'
    no file '/usr/local/share/lua/5.1/socket/smtp.lua'
    no file '/usr/local/share/lua/5.1/socket/smtp/init.lua'
    no file '/usr/local/lib/lua/5.1/socket/smtp.lua'
    no file '/usr/local/lib/lua/5.1/socket/smtp/init.lua'
    no file './socket/smtp.so'
    no file '/usr/local/lib/lua/5.1/socket/smtp.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'
    no file './socket.so'
    no file '/usr/local/lib/lua/5.1/socket.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'

Now, according to both the Trusty and Xenial package descriptions, those files are not installed in those locations.  I can confirm that on working Orthancs built on Trusty Docker images, the lua files are all installed in the locations indicated here.

I'm left wondering whether something has changed perhaps in the environmental search paths.  The Orthanc Lua engine cannot apparently see the additional lua modules I've installed under Xenial, but it had no problem seeing them under the Trusty system.

Any ideas?  I can't tell if this is a Xenial configuration issue or whether I'm misunderstanding something about running Orthanc on Xenial.

Thanks,
John.

[Alain Mazy]

Hi John,

Indeed I could reproduce your problem.

Here's what I've done so far (for the record):

In the container, before starting Orthanc:

export LUA_PATH="./?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua"

export LUA_CPATH="./?.so;/usr/local/lib/lua/5.1/?.so;/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;/usr/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so"

After this, Orthanc finds the socket module but, unfortunately, refuses to load it:

E0219 16:44:36.176965 LuaContext.cpp:580] Error while executing Lua script: error loading module 'socket.core' from file '/usr/lib/x86_64-linux-gnu/lua/5.1/socket/core.so': dynamic libraries not enabled; check your Lua installation

I then realized that, in previous orthanc images, lua was linked dynamically while it is now built statically.  This probably explains the difference.

I'll try to patch the docker images and keep you informed.

Best regards,

Alain.

--

Alain Mazy / CTO 
a...@osimis.io / +32 494 31 67 27

	OSIMIS S.A.  Rue bois Saint-Jean 15/1BE-4102 Seraing  www.osimis.io

[John Roberts]

Hi Alain,

Thanks for checking that.  Playing with the LUA path was going to be my next step.

Regarding the static linking, I began to suspect that was the case.  In another thread, I knew we said the Windows version of Orthanc was statically linked and that had me wondering whether the same cmake setup was used on linux platforms.

I'll wait to test moving my setup to Xenial when you have a chance to patch the build.

Thanks,
John.

[Alain Mazy]

Hi John,

Sébastien has patched Orthanc such that the static builds can now call lua external modules and this has been included in the osimis/orthanc:orthanc-mainline images.  LUA_PATH and LUA_CPATH are configured correctly in this image.

Could you give it a try ?

[John Roberts]

Let me try it out. 

I haven't checked the description of those images.  Do they include the plugins?  I think at some point, you switched to having all the plugins in the docker image and leaving it up to the user to copy which ones they wanted into the working plugin directory path. 

I'm still getting used to the new setup since I worked with Orthanc 1.2 for so long.

John.

[Alain Mazy]

yes, they contain all plugins and you can enable and configure the plugins through environment variables.  Full doc is available here: https://osimis.atlassian.net/wiki/spaces/OKB/pages/26738689/How+to+use+osimis+orthanc+Docker+images

[John Roberts]

Thanks.  Lua appears to be working and my use of socket is working as well.

Now that my Lua script is getting farther than before, I've currently run into an unrelated problem and am addressing it.  It appears that the native Dicom dictionary no longer includes the human readable tag, OtherPatientIDs, so I'm either going to modify the dictionary or switch to using the numeric tag descriptor for that field.

I0220 17:25:18.134807 FromDcmtkBridge.cpp:1077] Unknown DICOM tag: "OtherPatientIDs"
E0220 17:25:18.134878 LuaScripting.cpp:151] Lua: Unknown DICOM tag

Either the dictionary is missing that tag, or the embedded dcmtk routines aren't pointing to the tag dictionary.  OtherPatientIDs is the first tag in my Remove list.  I wouldn't be surprised if the dcmtk libraries are missing the pointer to the dicom dictionary in order to translate the human readable tags into numeric addresses.

John.

[John Roberts]

As a followup, if I use the web interface to drill down to an individual DICOM instance, the DICOM tag OtherPatientIDs is displayed with the human readable text.

So, at least for the display of existing tags, the dcmtk back end is working.  For some reason, it is rejecting my human readable tags during the anonymize/modify step.

John.

[Alain Mazy]

Actually, when upgrading DCMTK, some tag names have changed.  It seems that OtherPatientIDs is now considered as obsolete by the DICOM standards and is therefore now named "Retired_OtherPatiendIDs" in the dico.

[John Roberts]

I'm a little confused that Orthanc displays "OtherPatientIDs" as the nickname in the browser, but that behind the scenes my Lua script generates an Unknown DICOM Tag error when using the same nickname.

In the Orthanc Explorer, I can see:

0010,1000 (OtherPatientIDs)

along with all the other tags embedded in a particular DICOM file.

If DCMTK has deprecated "OtherPatientIDs", why doesn't Orthanc's browser also generate an error?  Or at least display the tag as "Retired"?

I'm going to have to add this nickname into my local dictionary because I make extensive use of it in my anonymization Lua script. 

Do you have a link to documentation describing how to add to the local dcmtk dictionary used by Orthanc?

Thanks,
John.

[John Roberts]

Is this a DICOM 2008 vs 2017b problem?  I did upgrade from Orthanc 1.2 to 1.3 and see that 2017b was introduced in Orthanc 1.3.

I tried setting the DicomVersion in my calls to anonymize/modify to 2008, but I still encounter the error:

Unknown DICOM tag: "OtherPatientIDs"

I also confirmed that the API call to instances/####/simplified-tags also returns "OtherPatientIDs" in the same way that the Orthanc Explorer does.  I suspect the Orthanc Explorer is calling the simplified-tags API anyway.

I'm going to run a test where I get rid of these nicknames that used to work in my old Orthanc 1.2 Lua setup.

John.

[John Roberts]

I have switched now to using numeric DICOM tags in the anonymize/modify steps and this seems to solve the "Unknown DICOM Tag" problem.

However, this seems to point to an asymmetry between the anonymize/modify API POST calls and the GET methods for retrieving meta data (ex. instances/####/simplified-tags).

The latter GET methods still seem to return human readable DICOM tags that are no longer recognized by the POST API calls.

[Sébastien Jodogne]

Hello,

When Orthanc receives a DICOM file, it pre-computes a JSON summary of its DICOM tags, and caches this JSON file as an attachment to the DICOM instance (accessible at the "/instances/{...}/attachments/dicom-as-json/" URI):

http://book.orthanc-server.com/faq/features.html#metadata-attachments

When Orthanc Explorer displays some DICOM instance, it accesses the cached JSON file, in order to avoid parsing again the source DICOM instance. In your case, you have updated the DICOM dictionary that is used by Orthanc, which implies that the cached JSON file does not perfectly match the new dictionary.

Since Orthanc 1.2.0, you can force the re-generation of the cached JSON file by DELETE-ing it, for instance:

# curl -X DELETE http://localhost:8042/instances/301896f2-1416807b-3e05dcce-ff4ce9bb-a6138832/attachments/dicom-as-json

The next time you open this particular instance with Orthanc Explorer, you will see messages in the Orthanc logs (in verbose mode) stating that the Orthanc server has reconstructed the JSON summary, which will match the new content of the dictionary:

I0222 08:56:00.923070 FilesystemStorage.cpp:155] Reading attachment "2309c47b-1cbd-4601-89b5-1be1ad80382c" of "DICOM" content type

I0222 08:56:00.923394 ServerContext.cpp:401] Reconstructing the missing DICOM-as-JSON summary for instance: 301896f2-1416807b-3e05dcce-ff4ce9bb-a6138832

I0222 08:56:00.929117 ServerContext.cpp:540] Adding attachment dicom-as-json to resource 301896f2-1416807b-3e05dcce-ff4ce9bb-a6138832

I0222 08:56:00.929425 FilesystemStorage.cpp:118] Creating attachment "3c830b66-8a00-42f0-aa3a-5e37b4a8b5a4" of "JSON summary of DICOM" type (size: 1MB)

HTH,

Sébastien-

[John Roberts]

Hi Sebastien,

Of course!  I should have thought of that - the storing of old meta data in the database prior to my updating Orthanc.

For the time being, I have switched entirely to using numeric DICOM tags, with a human readable form in comments in my Lua code.

Thanks for the explanation,
John.

[John Roberts]

Sebastien,

What would trigger a similar update of the meta data at the series and study level?  Is that only possible by forcing all the instances to regenerate their dicom-tags attachments?

Thanks,
John.

[Sébastien Jodogne]

Hello,

The following REST API call will force the re-generation of the "DICOM-as-JSON" summary for all the instances stored inside Orthanc:

# curl -X POST http://localhost:8042/tools/invalidate-tags -d ''

Sébastien-

[John Roberts]

Returning to an old topic, I recently rebuilt my Docker image based on the latest osimis/orthanc:orthanc-mainline image and ran into this same LUA dynamic library linking problem again. 

I suspect things were set up statically in the latest image.  I can see that the module is properly located, but the embedded Lua cannot dynamically link to it.

In addition to the socket module, I've added the luasql-postgres module and it's this one that is failing to load now.  I have not checked whether the socket module will load, though I suspect that it would not due to similary dynamic link problems.

John.

[Sébastien Jodogne]

Hello,

The CMake option "-DENABLE_LUA_MODULES=ON" was introduced in Orthanc 1.3.2, in order to allow the (statically-linked) Lua engine of Orthanc to be used in conjunction with system-wide Lua modules.

Here is a minimalist command-line session to call Lua modules installed on Ubuntu 16.04, from a statically-linked version of Orthanc compiled with "-DENABLE_LUA_MODULES=ON":

$ cat << EOF > Sample.json

{

"LuaScripts" : [ "Sample.lua" ]

}

EOF

$ cat << EOF > Sample.lua

socket = require ("socket")

EOF

$ LUA_PATH=/usr/share/lua/5.1/?.lua LUA_CPATH=/usr/lib/x86_64-linux-gnu/lua/5.1/?.so ./Orthanc Sample.json

Your problem stems from the fact that the Docker images for Orthanc 1.4.x are now using the official LSB binaries (Linux Standard Base), that were not compiled with this CMake option.

The "jodogne/orthanc" and "jodogne/orthanc-plugins" Docker images have just been updated with LSB binaries compiled with "-DENABLE_LUA_MODULES=ON". Similar Osimis images should be automatically generated soon.

Regards,

Sébastien-

[John Roberts]

Thanks, Sebastien, for the quick response. 

I'll work with my current version until those newer versions of the Osimis images are posted with the compile option turned on.

Thanks,
John.

[John Roberts]

Hi Sebastien,

Is there documentation somewhere as to which docker images have the "-DENABLE_LUA_MODULES=ON" compilation flag?

I am still having trouble with the latest osimis/orthanc docker images, apparently because this CMAKE option was not turned on for the latest images.

Thanks,
John.

[Sébastien Jodogne]

Hello,

Yes, the Docker images do have the "-DENABLE_LUA_MODULES=ON" compilation flag.

You can find more information in the two following threads:

https://groups.google.com/d/msg/orthanc-users/C9HfrMat3FQ/H9yeYlqxBwAJ

https://groups.google.com/d/msg/orthanc-users/Y1HAGTnx9Cs/MhdYmWIiGAAJ

Sébastien-

[John Roberts]

Thanks, Sebastien. 

I should clarify my request.  I realize you've added the compilation flag to your Docker images, but I was wondering about the Osimis images. 

From our last exchange last year, I thought that Alain was going to add the compilation flag to the Osimis images.  Since I've had trouble with the most recent Osimis image, I wondered where I could look to see whether the flag was turned on.

I thought that maybe sometimes the compilation flag is on and sometimes it's not - and that would be documented somewhere in the documentation for the Osimis docker images.

I'll try your sample Lua snippet to test the Osimis image.

John.

[Sébastien Jodogne]

Contrarily to last year, both "jodogne/orthanc" and "osimis/orthanc" images are now based upon the same Linux Standard Base (LSB) binaries. So, they should have the same behavior wrt. Lua modules.

Sébastien-

[John Roberts]

Thanks for the clarification. 

I'm going to have to dig deeper into my lua script to determine why it's failing to load a system installed module.

John.

[Alain Mazy]

FYI, here's the Docker file of the osimis/orthanc docker image: https://bitbucket.org/osimis/orthanc-builder/src/master/docker/orthanc/Dockerfile

--

Alain Mazy

a...@osimis.io / +32 494 31 67 27

	OSIMIS S.A.  Quai Banning 6BE-4000 Liège  www.osimis.io

[John Roberts]

     I thought I would document some of what I've been attempting to do.  I don't expect an answer at this point, unless anyone has helpful hints.  I've got a pretty sophisticated set of Lua scripts running on Orthanc.  So I realize I'm pushing the capability.  Ultimately, I'm going to rewrite the functionality to use the RestAPI outside of Orthanc, rather than the native Lua.  But for the moment, I'm stuck with my complicated Lua Scripts.

     I'm probably running into the similar lua version problems as discussed in the thread on the lua socket module.  In my case, I'm working with the luasql-postgres to make calls to the same backend postgres server I'm using for Orthanc's postgres index.  In the Osimis images of the last year, my calls to connect to the remote database simply fail with very little error reporting - even with Orthanc run in verbose mode.

   

     I can install lua5.3 on the host system (docker container) and produce the same errors loading the luasql-postgres module.  So, this really does seem to be a lua5.3 vs luasql-postgres 5.1/2 issue than an Orthanc lua problem.  The trick is figuring out how to get the host lua to work with the 5.3 lua compiled into Orthanc.

     As near as I can tell, that luasql-postgres module is supported by the ubuntu distributions (<=16) and available for apt-get install up to Lua 5.2, but not 5.3.  At some point, I guess the internal Orthanc Lua evolved to 5.3 and then stopped working well with the older Lua 5.1 extensions.  I think the base Docker image I've been using that still works dates back to late 2017.

     I tried installing the luasql-postgres module with luarocks.  Luarocks sort of worked and compiled luasql-postgres, but not quite, with the module crashing during actual Lua execution with complaints about missing symbols in the dynamic link library.  I suspect there might be some version conflict between the host (ubuntu 16 docker based Osimis Orthanc) and luarock's installed version of luasql-postgres, but I'm not sure.

     Compiling luasql-postgres completely from scratch isn't all that appealing, since I'm working from within docker containers.  I'm not so keen on installing the compilation infrastructure necessary to pull this off, only to then have to strip it out to reduce the docker container size back to a reasonable size.

     Following Sebastien's suggestions in that socket thread, I took the Osimis Dockerfile and attempted to modify it to build on top of ubuntu 18.04 since bionic has direct (apt-get) support for the luasq-postgres plugin for lua 5.3.  Everything seemed to build ok, except that when I attempt to start the Orthanc container I get an error during boot

"SQL error: ERROR:  syntax error at or near "ASSERT"
LINE 32:       ASSERT studyKey IS NULL;

This seems to arise from a postgres sql file used when Orthanc selects the postgres backend for its indexing (which is what I do).  I can only guess that moving to Ubuntu bionic 18.04 was too much for the Osimis build combined with my use of Postgres.  Granted, the Osimis build has grown a lot more sophisticated and it's possible I might not be turning off some plugin that's causing a problem here.  The above postgres error comes right after Orthanc reporting "Installing the CreateInstance extension".

Again, I'm not expecting a solution here.  I think it's probably a lot to expect all the new things (the OS, luasql and Osimis) to play nice together.

John.

[Sébastien Jodogne]

Hello,

I recognize this error message:

"SQL error: ERROR:  syntax error at or near "ASSERT"

LINE 32:       ASSERT studyKey IS NULL;

It happens when using an older version of PostgreSQL server (I have seen this on Ubuntu 14.04). It has been fixed by the following changeset that is pending in the mainline of the PostgreSQL plugin:

https://bitbucket.org/sjodogne/orthanc-databases/commits/b559af8fe6e0b393d646f4ed9105326b44482044

So, I think that everything is fine in your Lua setup (luasql-postgres is loaded fine). Give a try upgrading your PostgreSQL server.

HTH,

Sébastien-

[John Roberts]

I updated my Postgres (9.4) Docker image.  Looking at the Dockerfile on dockerhub, Postgres 9.4 is now based on debian/slim instead of Ubuntu.

I still see the error when I launch Orthanc, but I'm assuming I need to wait for the changes to make their way into mainline in order for the Osimis Dockerfile build (that I modified to point to Ubuntu 18) to work?  I'm working with the Dockerfile Alain provided a link to earlier in the thread.

I could also look into what's required to migrate my actual database from 9.4 to some later Postgres version if that would work.

Thanks for your suggestion,

John.

[Alain Mazy]

Hi John, 

since you're building the Docker image yourself, you can download the latest mainline PG SQL server binary from http://orthanc.osimis.io/lsb/plugin-postgresql/changesets/b559af8fe6e0/libOrthancPostgreSQLIndex.so

[John Roberts]

Woo!  Progress!  

At least the Orthanc comes up now and seems to be using some of the Lua system libraries I installed for my lua script. 

I also pulled the postgre storage plugin from the same link location.

[John Roberts]

Returning to an old topic, I've been working on upgrading my Orthancs to the latest osimis/orthanc and running into the problem of loading externally installed Lua modules again.

I'm working with 1.9.6 at the moment.  Was the ENABLE_LUA_MODULES=ON compiler option defined?

Is there a way to determine what compiler flags were used to compile the Orthanc in these docker images?  I was poking around the source code repository as well as the docker images repository and couldn't find anything, though I'm not quite sure where to look.

Thanks,

John.

[Alain Mazy]

Hi John,

The cmake command that is used to compile Orthanc in the osimis/orthanc docker images is available here: https://bitbucket.org/osimis/orthanc-builder/src/fbef8e045b6003ec06a281fb7d3f8ad3a0ae2af1/docker/orthanc/Dockerfile#lines-8

This build option is set to OFF by default: https://hg.orthanc-server.com/orthanc/file/tip/OrthancFramework/Resources/CMake/OrthancFrameworkParameters.cmake#l56

Honestly, I'm not very familiar with that part of the code.  I read that this option is only meaningful if using the STATIC build of the lua engine.  Note that the docker image now contains a DYNAMICALLY linked version of Orthanc but I don't know what it means wrt lua engine.  Could you check on your side if the system lua modules can be loaded with this version of Orthanc ?

Best regards,

Alain

--
You received this message because you are subscribed to the Google Groups "Orthanc Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/4c0fc0bc-6878-4631-981e-0969a1cde442n%40googlegroups.com.

[John Roberts]

Ugh.  I just wrote a long reply detailing my efforts, hit send and it seems to have gone nowhere.

Thanks for your response.

I looked at the CMAKE file and I think my solution lies somewhere between setting appropriate environmental LUA_CPATH/LUA_PATH variables and needing to compile Orthanc with modules enabled.

If I explicitly add a Lua command line interpreter to the container and then shell into the container, run Lua (outside of Orthanc API), and attempt to load the modules I installed (ex. lua-sql-postgres), everything works.  If I make the same call via the RestAPI, it fails - as though I've either got the wrong CPATH/PATH variables set for Lua to find the installed modules OR external modules are currently turned off by compiler flag in Orthanc.

I'm attempting to figure out which of those is the case or maybe it's something else.

It would be nice to confirm that Orthanc Lua is running Lua 5.3.  That seems to be the case, but it's not explicitly set by the compilation flags that you sent me.

Thanks,
John.

[John Roberts]

Duh!  (Smacks forehead!)  My bad.

I found my problem.  In transitioning from the old Osimis/Orthanc I was using, I was moving from the old environmental variable settings (ex. PG_USER for the postgres user) to the new naming convention for environmental variables (ex. ORTHANC__POSTGRESQL__USERNAME).

I had forgotten to update my Lua scripts to reflect the change in naming convention.  Thus, it was failing due to using the wrong or absent credentials rather than failing to load the postgres Lua module.

So, to recap, things seem to be working with external Lua modules.  Additionally, I found I no longer have to play with the LUA_CPATH or LUA_PATH variables to get things to work.  I only need to modify the original osimis/orthanc image to apt-get install the modules I want and rely on the default LUA path.

Thanks again for pointing me to the cmake repository.

John.