2021 Evaluation Code

257 views
Skip to first unread message

Hamed Taheri Gorji

unread,
Jan 19, 2021, 8:12:50 PM1/19/21
to physionet-challenges
Hello all,

I've tried to test the python code for Cardiology Challenge 2021. I've downloaded the code from PhysioNet/CinC Challenge 2021 evaluation code.
First of all, it seems the code needs some modification because "evaluate_12ECG_score" is not defined in the code.
However, after modification, when I run the code, it gives me the following result, which is not what I expected.

(['270492004', '164889003', '164890007', '426627000', '713427006', '713426002', '445118002', '39732003', '164909002', '251146004', '698252002', '10370003', '284470004', '427172004', '164947007', '111975006', '164917005', '47665007', '427393009', '426177001', '426783006', '427084000', '164934002', '59931005'], 1.0, 1.0, array([ 1.,  1., nan, nan,  1., nan, nan, nan,  1., nan, nan, nan,  1.,
       nan, nan, nan, nan, nan, nan, nan,  1., nan, nan, nan]), array([ 1.,  1., nan, nan,  1., nan, nan, nan,  1., nan, nan, nan,  1.,
       nan, nan, nan, nan, nan, nan, nan,  1., nan, nan, nan]), 1.0, 1.0, array([ 1.,  1., nan, nan,  1., nan, nan, nan,  1., nan, nan, nan,  1.,
       nan, nan, nan, nan, nan, nan, nan,  1., nan, nan, nan]), 1.0)

Moreover, when I try to add the 'scores.csv'  to the function, I get the following error:

TypeError: evaluate_12ECG_score() takes 2 positional arguments but 3 were given

Since it is written on the website that we cannot change the main codes, I would be so thankful if you could let me know how to solve this problem?

Thanks,
Hamed

________________________________________________________
Le informazioni contenute in questo messaggio di posta elettronica sono strettamente riservate e indirizzate esclusivamente al destinatario. Si prega di non leggere, fare copia, inoltrare a terzi o conservare tale messaggio se non si è il legittimo destinatario dello stesso. Qualora tale messaggio sia stato ricevuto per errore, si prega di restituirlo al mittente e di cancellarlo permanentemente dal proprio computer.
The information contained in this e mail message is strictly confidential and intended for the use of the addressee only.  If you are not the intended recipient, please do not read, copy, forward or store it on your computer. If you have received the message in error, please forward it back to the sender and delete it permanently from your computer system.


physionet-challenges

unread,
Jan 20, 2021, 12:14:42 AM1/20/21
to physionet-challenges

Dear Hamed,

Thanks for your message and pointing out the errors.

We fixed the typos in the evaluation code. Could you please download it again from https://github.com/physionetchallenges/evaluation-2021 and try it once more?

The evaluation scripts accept two, three or four inputs. Please make sure that you use the correct commands for MATLAB or python code using the instructions. 

The output in your post are from the classifier code, not the evaluation code. Could you please provide more information about whether you ran the unaltered example classifier from https://github.com/physionetchallenges/python-classifier-2021 or your own code? If you used the unaltered classifier, then could you please identify the training database and example recording that produced the NaNs in the output of your model so that we can reproduce this error. Also, were you running the code in Docker when you produced this output? We can’t guarantee that every version of every package will work the same way, which is one of the main reasons that we use containers for the Challenge. 

Please don’t hesitate to contact us if you have further questions.

Kind regards, 

Nadi

(on behalf of the PhysioNet Challenge team)

Hamed Taheri Gorji

unread,
Jan 21, 2021, 1:22:11 AM1/21/21
to physionet-challenges
Dear Nadi,

Thank you so much for your response.
I didn't write my own code yet, and I just wanted to test everything before starting to code.
I am using pycharm, and I downloaded your classifier python codes. To test the codes, first of all, I trained and tested the model using one of the datasets (CPSC2018 training set) using the following code:
training_code('Training_WFDB','Save_Model')
test_model('Save_Model','Training_WFDB', 'Output')
It works well for trainig and testing but when I want to evaluate the model using  evaluate_model('Training_WFDB', 'Output') it gives me the result I already sent you and when
I use the three or four input as follows: evaluate_model('Training_WFDB', 'Output','scores.csv', 'class_scores.csv') it give me the following error.
TypeError: evaluate_model() takes 2 positional arguments but 4 were given.
I would be so thankful if you colud let me know how can solve this problem and run your code in pycharm.

Thanks,
Hamed



--
You received this message because you are subscribed to the Google Groups "physionet-challenges" group.
To unsubscribe from this group and stop receiving emails from it, send an email to physionet-challe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/physionet-challenges/fc078596-1bf2-4b52-93d5-ce0dd06acccan%40googlegroups.com.

physionet-challenges

unread,
Jan 21, 2021, 1:25:01 AM1/21/21
to physionet-challenges
Dear Hamed,
 
Thanks for your message and information.

You are welcome to use PyCharm for development, but it is difficult to troubleshoot bugs that you encounter in your local computing environment from a different computing environment. Docker and other containerization platforms help solve this problem by providing consistent and reproducible computing environments, and we require Docker for Python entries in the Challenge so that we can more reliably run your code (and vice versa). Could you please try to run the example code in Docker to see if it fixes your error? Please see the following link for instructions and a list of commands for running your code in Docker:
 
I hope this helps, and we look forward to receiving your entries when we start accepting submissions soon.
 
Kind regards,
 
Nadi
(On behalf of the Challenge team)

Hamed Taheri Gorji

unread,
Apr 20, 2021, 8:29:42 PM4/20/21
to physionet-challenges
Dear PhysioNet Challenge Team,

Could you please let me know whether saving models in .sav format is mandatory or not?
Can we save our model using JSON format?

Best,
Hamed


________________________________________________________
Le informazioni contenute in questo messaggio di posta elettronica sono strettamente riservate e indirizzate esclusivamente al destinatario. Si prega di non leggere, fare copia, inoltrare a terzi o conservare tale messaggio se non si è il legittimo destinatario dello stesso. Qualora tale messaggio sia stato ricevuto per errore, si prega di restituirlo al mittente e di cancellarlo permanentemente dal proprio computer.
The information contained in this e mail message is strictly confidential and intended for the use of the addressee only.  If you are not the intended recipient, please do not read, copy, forward or store it on your computer. If you have received the message in error, please forward it back to the sender and delete it permanently from your computer system.



Fai crescere i nostri giovani ricercatori,
dona il 5 per mille alla Sapienza
codice fiscale 80209930587

physionet-challenges

unread,
Apr 20, 2021, 8:32:10 PM4/20/21
to physionet-challenges
Dear Hamed,

Using .sav format for saving models is not mandatory and you can use any format that you wish to save your models to be used for test.

Best,

Nadi
(On behalf of the Challenge team)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email chal...@physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually. 
Reply all
Reply to author
Forward
0 new messages