Temporary file collection ?

[Salim Kanoun]

Hi there,

Small question, I just figured that Orthanc Temporary file created with /tools/archive API (synchroneous, simply by using Orthanc Explorer) is not removed once the http transfer is ended.

The temporary files are correctly collected at orthanc shutdown.

But for servers that remains open months this would land to a hard disk leak, as each archive generation will remain a temporary file and progressively fill the hard disk. 

I have some servers running for a while meanwhile I often consume this Archive API. So I guess there is a file collection mechanism somewhere.

What is Orthanc politics regarding temporary files ? 

is there a defined interval of time to collect temporary file ? Or is the OS who take the responsability to collect these temporary file to free hard disk space ?

Or maybe only collection occurs only at exit and I have been lucky so far to not run out memory ?

Why the Archive temporary zip is not collected at the end of HTTP transfers ? When asking the same ressources in another call this temporary file does not serve as cache, a new one is generated. They cannot be collected immediatly after the end of the http connexion ?

Best regards,

Salim 

[Sébastien Jodogne]

Hello Salim,

Orthanc does not delete the ZIP archive after its first access, as it might be accessed by other subsequent accesses to the REST API.

Orthanc deletes older temporary files as new archives are created. The configuration file of Orthanc contains explanations about how Orthanc deals with this recycling policy. Check out the explanations associated with the options "TemporaryDirectory", "JobsHistorySize" and "MediaArchiveSize":

https://hg.orthanc-server.com/orthanc/file/default/OrthancServer/Resources/Configuration.json

HTH,

Sébastien-

[Salim Kanoun]

Dear Sébastien,

Yes I Seen this but this only refer to asyncronous call if the archive API, indeed is this case you can have multiple Access to the api to get the asyncroneously generated zip.

But what about syncronous call ? In this case each trigger of the archive API generate a new zip (even when calling several Time the same ressources ID array) and the zip is accessible only once (at the single syncronous call)

The MediaArchiveSize does not apply to syncronous calls, so what is the policy as regard of this syncronous zip ?

Best regards,

Salim

[Sébastien Jodogne]

In the case of ZIP archives, "synchronous mode" simply consists in triggering a job in "asynchronous mode", in waiting for this job to end, then in sending the ZIP in the REST answer.

In other words, "synchronous mode" is just a simpler API on the top of "asynchronous" (but that might result in HTTP timeouts for large archives). So, the synchronous generation of archive is also affected by "MediaArchiveSize".

[Salim Kanoun]

Dear Sebastien,

In this case I think you have a bug in Orthanc

I have a MediaArchiveSize of 1 in my settings.

For asynchronous call, no problem I see that the previous generated temporary zip is removed at the new async /archive generation (as my limit is 1, this is consistant).

But for Syncronous calls the temporary file are not removed and stays even when over the MediaArchiveSize limit. 

See in attachement that I generated 7 zip file that are not collected using synchronous generation of archive while my MediaArchiveSize = 1

I'm pretty sure there are a protection somewhere, In both OrthancTools and GaelO I do a lot of calls on this synchronous archive API, I would have run out disk space for sure if no safety system was collecting these files, maybe the operating system is saving us ?

Best regards,

Salim

[Jacopo De Luca]

Hello Salim,

I had same problem, but instead of trusting Orthanc json tag description, I tested behaviour of MediaArchiveSize and JobsHistorySize.
I noted that is JobsHistorySize which manages zips on tmp folder, so try to set it.

Then my question for everyone is: what is MediaArchiveSize role in async mode?

In https://hg.orthanc-server.com/orthanc/file/default/OrthancServer/Resources/Configuration.json description is:
// Maximum number of ZIP/media archives that are maintained by

// Orthanc, as a response to the asynchronous creation of archives.

// The least recently used archives get deleted as new archives are

// generated. This option was introduced in Orthanc 1.5.0, and has

// no effect on the synchronous generation of archives.

"MediaArchiveSize" : 1

Thank you so much

Jacopo

[Jacopo De Luca]

Hello to everyone

Differently from what I said, tmp zip files collection does not respond as expected.

I use:
- docker osimis/orthanc:20.9.5

- fs index

- asynchronous mode for CMove

- JobHistorySize: 5

- MediaArchiveSize: 5

I have several test servers with same configuration and I'm noting that
- in some of them, tmp zip collection is limited to 5

- in others, tmp zip collection is not limited

I don't know why. I think that when Orthanc falls in error (e.g. OutOfMemory, I've to fix this) and docker restarts the service (more precisely the container), some options on json like zip limitation does not work properly.

Could someone help me?

Thank you

Jacopo

[Sébastien Jodogne]

Hello Jacopo and Salim,

I have just committed a change that seems to be related to your question:

https://hg.orthanc-server.com/orthanc/rev/b02dc8303cf6

Summarizing, in Orthanc <= 1.9.2, if creating ZIP archives in *synchronous* mode, the temporary files containing the archives were only removed once their job was deleted from Orthanc. As a consequence, the number of temporary files could grow up to the value of "JobHistorySize", whereas the user would expect the temporary files to be removed as soon as the synchronous call is over.

Also note that the value of "MediaArchiveSize" only applies to archives created in *asynchronous* mode.

Please could you confirm that this change answers your issue?

Thanks in advance,

Sébastien-

PS: The explanation of "synchronous" vs. "asynchronous" mode is part of the Orthanc Book:

https://book.orthanc-server.com/users/advanced-rest.html#synchronous-vs-asynchronous-calls

[Salim Kanoun]

Dear Sebastien,

Yes sounds good to me !

Thanks !

Salim

[Jacopo De Luca]

Thank you so much Sebastien!

Sounds good to me too!

Jacopo