Potential issue with reproducibility

[Ryan Friedman]

I ran Selene twice, with the exact same config file, and got slightly different results. The overall shape of the loss function and my performance metrics vs epoch are the same, but the exact values are slightly different from one run to the next, which makes me suspect the parameters of the model are varying too.

I've included random seeds in the config file (attached), so I'm not sure how this is happening. Am I forgetting to include a seed somewhere, or could this an issue with Selene itself?

Thanks for any help!

[chen.ka...@gmail.com]

Hi Ryan,

Thanks for your message - upon looking into this more, I do think Selene needs a few more parameters to be set in order to make model training as deterministic as possible.

You might want to insert the following lines into your code as a temporary workaround:

```

random.seed(<seed>)
np.random.seed(<seed>)

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

torch.use_deterministic_algorithms(True)

```

(1) If you are using a local installation of selene (e.g. cloned the Github repo), you can add to this section directly: https://github.com/FunctionLab/selene/blob/master/selene_sdk/utils/config_utils.py#L336

(2) Otherwise, if you are using the conda installation, try using the Selene CLI script https://github.com/FunctionLab/selene/blob/0.4.8/selene_cli.py to run your training and add the lines right before `configs` variable is set.

Please see this page for more information: https://pytorch.org/docs/stable/notes/randomness.html

They mentions that in some versions of CUDA, specific operations or modules in the model architecture might have non-deterministic behavior and you might have to incorporate a few more workarounds. 

Please let me know if this helps, and feel free to submit a pull request to the repository if you go with option (1) 

Thank you!

Kathy 

[Ryan Friedman]

Thanks for looking into this Kathy! The lines you provided solved the problem, although I didn't use `torch.use_deterministic_algorithms` because it is not implemented in versions prior to 1.9.0 and Selene is only compatible with versions through 1.4.0.

I did option 2, but I will go ahead and add it into the source code and submit a PR.

Best,

Ryan