evaluate_code.py newer numpy version

[Kelvin Khang Dinh Nguyen]

Hi, 

First off, thank you for all of your hard work on running this challenge! I noticed in evaluate_code.py, the np.argsort function uses the `stable` keyword, which was introduced in numpy 2.0. I am currently using numpy 1.26, since this version has been around for a very long time, which doesn't have the `stable` keyword. 

In essence, will the evaluate_code.py file use our environment in the docker container (which in my case has np 1.26), or will you use your own environment (therefore preventing any issues like this)? Thank you!

[PhysioNet Challenge]

Dear Kelvin,

We will use our own environment to run the evaluation code (and our own copy of the evaluation code), so there's no need to worry about conflicts with versions of libraries that you are using for your code.

Also, you can replace "stable = True" with "kind = 'stable'" in the argsort command (currently line 272 of the helper_code.py script) to fix this error for NumPy 1.x.

​​Best,
Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

[Lucas Erlacher]

Hi,

Thank you very much for the suggestion!
We were also encountering issues with NumPy 2.0 (some dependencies require <2.0, so we can't upgrade), and changing stable = True to kind = 'stable' resolved everything for us locally.

However, we’re wondering whether this change will persist when we submit our code. Specifically, if the challenge organizers overwrite helper_code.py with its original version during submission processing, the fix our code relies on would be reverted, which would make it crash again.

Could you please confirm whether any changes we make to helper_code.py will be retained in the version executed on your system?

Additionally, in your previous message, you mentioned that the evaluation will run in your own environment. Since this evaluation presumably involves loading and invoking our model, we’re wondering how you will be able to install libraries that our model code imports that might not be part of your environment?

Thank you again for your support!

Best regards,
Lucas Erlacher

[PhysioNet Challenge]

Dear Lucas,

We use the helper_code script to run the entries consistently, so we do overwrite any changes that teams make to this file. However, you can define and use new functions outside of the helper_code script if, for example, you do not want to use one of the provided functions for some part of your entry.

Yes, we use different environments for training (train_model), inference (run_model), and evaluation (evaluate_model). If your entry specifies NumPy 1.6.2, then we will run your training and inference code using NumPy 1.6.2. Our evaluation code specifies NumPy 2.0, so we will evaluate the models using NumPy 2.0. We only run your code for training and inference and we only run our code for evaluation, so there should be no problems here. This part of the example code README illustrates how this works:
https://github.com/physionetchallenges/python-example-2025?tab=readme-ov-file#how-do-i-run-these-scripts

In general, we try to use language features that have been supported for several years so that users with different environments can run the code as-is, so this is usually not a problem.

​​Best,
Matt
(On behalf of the Challenge team.)

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email info at physionetchallenge.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address. This email is maintained by a group. Please do not email us individually.

[Lucas Erlacher]

Thank you very much for your answer, that clears a lot of things up!