# PyMedPhys Activity Report -- The new PyMedPhys GUI

517 views

### PyMedPhys

Apr 29, 2020, 10:25:44 PM4/29/20
to PyMedPhys
PyMedPhys GUI
A tool to compare Linac deliveries directly to the TPS plan

Within our clinics at RCCC, NBCC, and SASH we have developed a new tool for comparing the Linac delivery directly to the plan recorded within Monaco. We have a server set up at RCCC, and we use SSH tunnels, dns aliases and nginx to make it so that at all three sites if http://physics-server/delivery is typed into a browser the user is greeted by a MU Density comparison tool. Some demo data has been created, and this GUI we use internally can be run on your own machine by running the following within a command prompt:

pip install pymedphys==0.25.1
pymedphys gui

Of note, in order to roll out this GUI within our centre we have implemented automated GUI testing with Cypress (https://www.cypress.io/). This means that for every code change undergone within the PyMedPhys repo a machine spins up and runs the GUI to make sure that key tasks within the GUI have not been broken by those code changes. For example, for the 0.25.1 release, you can watch a video online of those automated tests running by going to https://dashboard.cypress.io/projects/tgt8f6/runs/57/specs and then pressing the video play button as in the screenshot below:

If you would be interested in using this GUI in your centre please reach out.

Some notes on local configuration

For those who are interested, to stop using the demo data, and instead use data from your own clinic, you will need to create a config.toml file within the directory ~/.pymedphys. To see the config.toml file used in our own centre see:

Of note, if you would like to have your configuration file in either a shared directory, or under git version control you can specify a "redirect field" within your root config file. For example, the file found at ~/.pymedphys/config.toml on our server looks like the following:

redirect = 'C:\Users\PExIT\git\pymedphys\examples\site-specific\cancer-care-associates\config.toml'

Under the hood, pymedphys gui uses streamlit (https://www.streamlit.io/). A brilliant tool for GUI creation aimed at science programmers. If you would like to customise how pymedphys gui runs on your machine you can do this through streamlit's configuration file. For example, on our server we have the following file at ~/.streamlit/config.toml:

[server]
port = 38894
baseUrlPath = "/delivery/"

[browser]
serverPort = 80
gatherUsageStats = false

New Releases

Since the last activity report the following releases have occurred:

### LipTeck Chew

May 26, 2020, 9:47:03 PM5/26/20
to PyMedPhys
Hi Simon,

I have installed pymedphys and ImageMagick both in 64 bits Windows 7 and 64 bits Ubuntu 20.04 LTS Desktop versions running good in demo mode. I am now trying to setup in the actual clinical environment. Our department is using full Elekta Solution: Mosaiq v2.64, Monaco v5.11.03 with 2 Agility Head Linacs: 1 VersaHD and 1 Infinity.

1. I am confused with iComLogFiles directory and the iComLogFiles\live directory described

May I know what the differences are?

What are the recommendation for the location of logfiles, live and grouped patient directory: local or NAS?

2. In the example of listeners at https://nbviewer.jupyter.org/url/simonbiggs.net/icom.ipynb, both batch scripts consolidate the logfiles from two linacs' NSS to D:\iComLogFilesThen, in the patient archiving (grouping) it does something using D:\iComLogFiles\live and Z:\iCom\live to produce something at D:\iComLogFiles\patients

May I know what is “Z:\iCom\live” ?

Is Z:\ a local drive or network drive?

Can a single computer listen to both NSS simultaneously or we need to deploy one listener per computer for each NSS?

3. In the process of grouping patients at https://nbviewer.jupyter.org/url/simonbiggs.net/icom.ipynb, there is a line: “This is done by setting up a .bat file to run on machine boot”.

Could you elaborate further?

Is the machine here referring to the linacs or physics-server/pc?

Do you mean patient archiving (grouping) is done daily in the morning?

3.       Regarding the command line tool at https://docs.pymedphys.com/ref/cli/logfile.html: pymedphys logfile orchestration [-h] [-m MOSAIQ_SQL] [-l LINAC_DETAILS] data_directory (orchestration.bat)

May I know whether this data_directory referring to ~.\pymedphys\data or D:\iComLogFiles\patients or the TRF directory?

Which data directory should we configure?

Do we need to map network drive to each separate NSS with the rtduser?

4.       How do we enable reading all clinics or designated clinics appear under the \\FocalData\Installation directory described at https://github.com/pymedphys/pymedphys/blob/master/examples/site-specific/cancer-care-associates/config.toml? We have several folders for different groups of doctors.

5.       Do we need the D:\iComLogFiles\patients for the gui to read trf ? What are the configurations required just to make the trf portion work? Would the trf files from two NSS override each other?

Thank you for your patience :)

warm regards,

LipTeck

### Simon Biggs

May 27, 2020, 9:29:37 PM5/27/20
to LipTeck Chew, PyMedPhys

Hi LipTeck,

I’m keen to send you some nice responses to this. Currently quite under the pump, might have to respond to you tonight when I get home from work.

Cheers,

Simon

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### Simon Biggs

May 29, 2020, 3:25:48 AM5/29/20
to LipTeck Chew, PyMedPhys

Hi LipTeck,

As a first point, the iCOM command line tool recently underwent a small revamp. See https://github.com/pymedphys/pymedphys/releases/tag/v0.28.0 for more details. The GUI itself has undergone a change too, see https://github.com/pymedphys/pymedphys/releases/tag/v0.29.0.

As such, I would recommend updating to pymedphys==0.29.1 before implementing the feedback below.

Also, given this is going down the path of implementing the iCOM listener please be aware of the following bug that appears to plague all iCOM listener tools (commercial tools included) https://github.com/pymedphys/pymedphys/issues/849.

Cheers,

Simon

Hi Simon,

I have installed pymedphys and ImageMagick both in 64 bits Windows 7 and 64 bits Ubuntu 20.04 LTS Desktop versions running good in demo mode. I am now trying to setup in the actual clinical environment. Our department is using full Elekta Solution: Mosaiq v2.64, Monaco v5.11.03 with 2 Agility Head Linacs: 1 VersaHD and 1 Infinity.

1. I am confused with iComLogFiles directory and the iComLogFiles\live directory described

May I know what the differences are?

Both the live and the patients directories within the iCOM directory are for storing iCOM data. The live directory is a simple .txt dump of the most recent iCOM stream records. These files a regularly overwritten. The patients directory is where iCOM stream data is grouped and indexed by patient.

What are the recommendation for the location of logfiles, live and grouped patient directory: local or NAS?

I would have the iCOM files be stored on the same machine that is running the iCOM listener, and then have these files shared via the network so that they are accessible by the machine that is running pymedphys gui.

2. In the example of listeners at https://nbviewer.jupyter.org/url/simonbiggs.net/icom.ipynb, both batch scripts consolidate the logfiles from two linacs' NSS to D:\iComLogFilesThen, in the patient archiving (grouping) it does something using D:\iComLogFiles\live and Z:\iCom\live to produce something at D:\iComLogFiles\patients

May I know what is “Z:\iCom\live” ?

Is Z:\ a local drive or network drive?

Z:\ is a network drive shared via intranet from another site. We run this tool for 4 Linacs at 3 sites.

Can a single computer listen to both NSS simultaneously or we need to deploy one listener per computer for each NSS?

A single computer can listen to both NSS’. We have three computers doing it since there is one computer at each site and I don’t want the internet connection to cause the dropout issues discussed above.

3. In the process of grouping patients at https://nbviewer.jupyter.org/url/simonbiggs.net/icom.ipynb, there is a line: “This is done by setting up a .bat file to run on machine boot”.

Could you elaborate further?

Is the machine here referring to the linacs or physics-server/pc?

Do you mean patient archiving (grouping) is done daily in the morning?

Actually, if you install pymedphys 0.29.1, this archiving tool is no longer needed (and the CLI interface has been removed). It is now undergone automatically by the listener process already. No need to manage any of that part yourself any longer.

3.       Regarding the command line tool at https://docs.pymedphys.com/ref/cli/logfile.html: pymedphys logfile orchestration [-h] [-m MOSAIQ_SQL] [-l LINAC_DETAILS] data_directory (orchestration.bat)

May I know whether this data_directory referring to ~.\pymedphys\data or D:\iComLogFiles\patients or the TRF directory?

That data directory is actually where ever you would like it to be... I personally put it on the D drive of a server’s local machine where that D drive points to a rather large spinning disk storage. The TRF files end up taking quite a lot of space.

Which data directory should we configure?

In your config.toml point to the same directory as written within the logfile orchestration tool.

Do we need to map network drive to each separate NSS with the rtduser?

No, don’t map the NSS network drive, but do save the rtduser password within the Windows credential manager:

4.       How do we enable reading all clinics or designated clinics appear under the \\FocalData\Installation directory described at https://github.com/pymedphys/pymedphys/blob/master/examples/site-specific/cancer-care-associates/config.toml? We have several folders for different groups of doctors.

Unfortunately this is not currently directly supported at the moment, I might be able to make this possible. I do wonder if it might “just work” if you treat each clinic as its own “site” within the config file, as in create a new site for each clinic, and have all the configuration be the same except change the clinic path. It would be nice to directly support this use case though.

5.       Do we need the D:\iComLogFiles\patients for the gui to read trf ? What are the configurations required just to make the trf portion work?

Nope, you don’t need that patient directory if you’re going to be just using TRF. However, now that patient indexing is included within the listener maybe this is not as much of an issue?

Would the trf files from two NSS override each other?

They shouldn’t. The TRF files are organised by IP address. Let me know if you have any issues here though.

Thank you for your patience :)

My pleasure :) Things are quite hectic at the moment, thanks for baring with response delays :). Feel free to let me know if you have any more issues.

warm regards,

LipTeck

### LipTeck Chew

May 30, 2020, 6:35:25 AM5/30/20
to PyMedPhys
Hi Simon,

I am excited to see the development is getting faster and I have upgraded pymedphys to v0.29.1 successfully.

I have used the nssm tool to create the iCom listeners for two NSS according to https://nbviewer.jupyter.org/url/simonbiggs.net/icom.ipynb without patient_archiving since it is now built-in. I have screenshot the tasks manager on both listeners. Hope this is a good indication of correct implementation. (iCom listening services.png)
Would these listeners automatically start when the linacs are turned on?

I have set the credential ok but there was time zone error when running the orchestration script. Please see get TRF error screenshots. FYI, I am using "Asia/Singapore" for this field in the cvs files and config.toml

I have managed to setup multiple clinics by duplicating the [[site]], [site.monaco], [site.mosaiq], [site.linac] , using site names as clinic names and clinic folders in FocalData\Installation directory.

Hope to get the TRF tool and iCom going :)

regards,
LipTeck

iCom listening services.png
get TRF error pg1.png
get TRF error pg2.png

### Simon Biggs

May 30, 2020, 7:23:39 AM5/30/20
to LipTeck Chew, PyMedPhys
Great to see the progress :). What does your config_mosaiq_sql.csv (https://docs.pymedphys.com/ref/cli/logfile.html#config-mosaiq-sql-csv) look like?

After running the iCOM listener are the directories "live" and "patient" appearing within the directory you are pointing to?

Regarding your question about the services starting, no you'll need to initially press "start", but once you've done that it should automatically be running whenever the Linac reboots.

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### LipTeck CHEW

May 30, 2020, 12:58:39 PM5/30/20
to Simon Biggs, PyMedPhys
Hi Simon

I have attached my config_mosaiq_sql.csv and the getTRF_error.txt  We are looking at the possibility of getting the new TRF files nightly to a NAS. Suppose we have a cronjob to keep adding the fresh TRF files to D:\pymedphys\TRF, will the GUI able to recognise the files or we still need to run the orchestration CLI?

I have manually created the live and patients directories within D:\pymedphys\iComLogFiles\live and D:\pymedphys\patients earlier on. Nevertheless, I do have the listeners_log.txt for both NSS in the live directory after clicking on the "start" for both listeners. Please see attached screenshot. Yes, I think it should work when the linacs are turned on.

regards,
LipTeck
getTRF_error.txt
config_mosaiq_sql.csv
iCom listening services started.png

### Simon Biggs

May 30, 2020, 5:44:04 PM5/30/20
to LipTeck CHEW, PyMedPhys
Hi LipTeck,

When you run the orchestration CLI command there is a file created called something like "to_be_indexed". Any trf file placed in there will be picked up by the orchestration CLI and indexed as per its standard processes. Also, if ever a duplicate TRF file is found it is discarded instead of causing duplicate data entries. So, if you are able to have TRF files nightly, you could have that process running, placing the files within "to_be_indexed" then, once that was finished you can then run the orchestration CLI and it will index them for you, as well as pull files from any diagnostic backups in case there is an issue with your nightly pull.

In order for the GUI to recognise them, they do need to be indexed according to the orchestration CLI. But as long as you run both that should work just fine.

For the timezone issue, I might need to get more details off of you. Just to cover our bases before diving in too deep could you tell me what version of pandas you are using?

Cheers,
Simon

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### LipTeck CHEW

May 30, 2020, 7:47:20 PM5/30/20
to Simon Biggs, PyMedPhys
Hi Simon

Thanks for a detail explanation on the Treatment Record File (TRF) orchestration CLI.

Running “conda list pandas” shows that it’s version 1.0.3 Build pypi_0 channel pypi

Regards
Lip Teck

### LipTeck CHEW

May 31, 2020, 10:58:14 PM5/31/20
to Simon Biggs, PyMedPhys
Hi Simon

The iCom listeners are not working with error message attached.

regards,
LipTeck
iCom listeners start or resume error.PNG

### Simon Biggs

May 31, 2020, 11:42:48 PM5/31/20
to LipTeck CHEW, PyMedPhys
Hi LipTeck,

Instead of running the command as a service could you run:


pymedphys --debug icom listen ip.address.goes.here D:/path/to/file/saving/location


And then let me know what happens

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### LipTeck CHEW

Jun 1, 2020, 2:29:59 AM6/1/20
to Simon Biggs, PyMedPhys
Hi Simon

Please see the attached text file.

regards,
LipTeck
icom_listen_debug.txt

### LipTeck CHEW

Jun 1, 2020, 2:39:56 AM6/1/20
to Simon Biggs, PyMedPhys
Hi Simon

We have just upgraded NSS from v3 to v4 on 10-May-2020. Hope it is not NSS version dependent.

regards,
LipTeck

### Simon Biggs

Jun 1, 2020, 4:49:37 AM6/1/20
to LipTeck CHEW, PyMedPhys
That debug file is perfect :). All good, we've got integrity 4 also.

Could you run:

pip install pymedphys==0.30.0dev4

and let me know if that issue has been fixed :).

Cheers,
Simon

On Mon, 1 Jun 2020 at 16:39, LipTeck CHEW <liptec...@gmail.com> wrote:
Hi Simon

We have just upgraded NSS from v3 to v4 on 10-May-2020. Hope it is not NSS version dependent.

regards,
LipTeck

On Mon, 1 Jun 2020 at 14:29, LipTeck CHEW <liptec...@gmail.com> wrote:
Hi Simon

Please see the attached text file.

regards,
LipTeck

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### LipTeck CHEW

Jun 1, 2020, 6:08:21 AM6/1/20
to Simon Biggs, PyMedPhys
Hi Simon

I have managed to install but:
1. the linacs were shutdown so the services are not able to run. I'll check that again tomorrow
2. the gui was not able to launch

regards,
LipTeck
icom_listen_debug2.txt

### Simon Biggs

Jun 1, 2020, 5:52:53 PM6/1/20
to LipTeck CHEW, Simon Biggs, PyMedPhys

Hi LipTeck,

So that bug never sees the light of day again would I be able to get a bit of data from you? Should it be fixed, would you be able to run a warm up beam or service mode beam (one that doesn’t have any patient data in it) and then, within the patients directory find the corresponding .xz file (under that particular dummy patient name). And I would be keen for you to upload this file for use within PyMedPhys’ automated testing suite. This will make sure any iCOM features that are particular to that particular file will always be verified as being able to be decoded with each code submission to PyMedPhys.

In preparation for uploading I’d like you to take a scan of the contents (so that you are happy with what you are uploading). To read the file, you will need to first extract it.

If you run “ipython” in the directory where the .xz file exists then you can run the following code within ipython:

In [1]: import lzma

In [2]: import pathlib

In [3]: icom_paths = pathlib.Path('.').glob('*.xz')

In [4]: for icom_path in icom_paths:
...:     with lzma.open(icom_path, "r") as f:
...:
...:
with open(icom_path.with_suffix(".txt"), "wb") as f:
...:         f.write(contents)

This will then convert all of the xz files to .txt files. You can then open the .txt files with something like VSCode (https://code.visualstudio.com/). When you open the text file with VSCode you will need to press “Do you want to open it anyway?”:

There will be lots of “boxes” and weird characters, but all patient identification information will appear in sections that look like the following:

Once you’re confident that the xz file doesn’t have any patient information within it would you then be able to use the following link to upload the .xz file to the Zenodo repository:

If you have issues with Zenodo, feel free to email that .xz file on this mailing list and I will upload it.

Thanks LipTeck,

Cheers,

Simon

Sent: Monday, 1 June 2020 8:08 PM
To: Simon Biggs
Cc: PyMedPhys
Subject: Re: PyMedPhys Activity Report -- The new PyMedPhys GUI

Hi Simon

--

You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### LipTeck CHEW

Jun 1, 2020, 11:26:40 PM6/1/20
to Simon Biggs, Simon Biggs, PyMedPhys
Hi Simon

We have rebooted the physicspc this morning.
Updates on the icom listen --debug
- the listeners are running but not generating the xz files
- this is confirmed with the _log.txt are updating the time stamp
- the gui is now able to launch. I suspect the physicspc could be running on low resources

- We have a new cronjob running every minute to get the new TRF files from both NSS to live folders in our NAS:
\\Fph-nas01\nas1\VHD\TRF\Live
\\Fph-nas01\nas1\INF\TRF\Live

I just need a robocopy script at the physicspc to bring them incrementally to a single directory:
D:\pymedphys\TRF\to_be_indexed

If you have a new development version, we can install and test during the lunch time today SG time 1230-1330.

regards,
LipTeck

icom_listen_debug3.txt

### Simon Biggs

Jun 2, 2020, 12:33:25 AM6/2/20
to LipTeck CHEW, Simon Biggs, PyMedPhys
Hi LipTeck,

That's pretty cool with the TRF files. It'll be nice to see it all up and running. Once all this Covid stuff is done maybe I should come visit :). I do have an uncle in Singapore...

Glad to hear the GUI is now launching, that was confusing me as to what the issue was there. I have produced a new development version, the only thing changed that affects your icom setup is that I have made it produce more debugging information to help me see what's going on within the logs you're providing.

Would you be able to run the following and then provide the logs for running the listener in debug mode:

pip install pymedphys==0.30.0dev5

Cheers :),
Simon

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### LipTeck CHEW

Jun 2, 2020, 1:07:10 AM6/2/20
to Simon Biggs, Simon Biggs, PyMedPhys
Hi Simon

Yes, sure!

Yes, now we can see there are some text files generated in
D:\pymedphys\iComlogfiles\live\172.31.255.2

Let me know when we can run the orchestration CLI. I can manually copy over some trf over to
D:\pymedphys\TRF
and test this CLI.

Cheers,
LipTeck

icom_listen_debug4.txt

### Simon Biggs

Jun 2, 2020, 1:15:48 AM6/2/20
to LipTeck CHEW, Simon Biggs, PyMedPhys
What issues were you having with the orchestration CLI again?

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### Simon Biggs

Jun 2, 2020, 1:17:33 AM6/2/20
to LipTeck CHEW, Simon Biggs, PyMedPhys
Also, I'd be quite keen to be able to include one of those .txt files within the PyMedPhys testing suite if you're in a position to share it :).

Cheers :),
Simon

### LipTeck CHEW

Jun 2, 2020, 8:22:06 AM6/2/20
to Simon Biggs, PyMedPhys
Hi Simon

I have tried resuming the iCom Listening Services manually during the day but they have continued to stay in the "pause" mode. Please see the windows popup message.

regards,
LipTeck
iCom services1.png

### LipTeck CHEW

Jun 2, 2020, 8:26:43 AM6/2/20
to Simon Biggs, PyMedPhys
Hi Simon

Is there any solution to the timezone error? Singapore is at the GMT +8. Are there alternatives to replace "Asia/Singapore" to enable the CLI to run first?
Thanks :)

regards,
LipTeck

---------- Forwarded message ---------
From: LipTeck CHEW
Date: Tue, 2 Jun 2020 at 14:36
Subject: Re: PyMedPhys Activity Report -- The new PyMedPhys GUI
To: Simon Biggs <m...@simonbiggs.net>

Hi Simon

Please see attached orchestration error text file.

regards,
LipTeck

orchestration_error1.txt

### Simon Biggs

Jun 2, 2020, 2:59:48 PM6/2/20
to LipTeck CHEW, Simon Biggs, PyMedPhys
When you created the service did you get it to create logs of its output? What does the iCOM logs say?

--
You received this message because you are subscribed to the Google Groups "PyMedPhys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pymedphys+...@googlegroups.com.

### Simon Biggs

Jun 2, 2020, 6:01:50 PM6/2/20
to LipTeck CHEW, PyMedPhys

Hi LipTeck,

I can reproduce your error with the file you provided by running the following:

It appears the issue is due to the spaces within the csv file.

I have reattached the config file you provided just with those spaces removed. Let me know if that fixes your issues.

-->

I’ll need to update the docs (https://docs.pymedphys.com/ref/cli/logfile.html#config-mosaiq-sql-csv). It seems my best intentions to make the displayed csv files be human r