Python 3 transition
21 views
Skip to first unread message
Steve Ludtke
May 16, 2019, 8:31:11 AM5/16/19
to eman2-de...@googlegroups.com, Tunay Durmaz
Ok, I think we have established the plan for our shift to Python 3. Since Python 2 becomes deprecated on Jan 1 next year, we clearly should target having this done by the fall (next release). While we will do the initial development in a branch, my hope is that this will last a reasonably short period of time. Once the majority of end-user programs work properly, the plan is to shift this branch back to master, then sort out the rest of the problems which arise before the next release.
- Tunay will establish one official Python 3 branch, and let everyone know which one it is
- Tunay will create a source installation page with instructions for setting up a build environment (with appropriate Anaconda dependencies) for the Python 3 branch and post here when it's ready.
- If we encounter trivial bugs in the system, like print -> print() or other simple syntactical changes, we will just fix them, regardless of whose code they are in. This will be limited to changes with no risk of changed behavior.
- We are not taking responsibility for fixing all such things, just saying that if we happen to encounter them, we will fix them.
- If anyone encounters problems where the fix could potentially change behavior, like issues with "/" vs "//", please suggest a change to the author/maintainer of the code.
- We will be experimenting with some more significant changes to the system designed to improve numpy interoperability. Once we have a recommendation, we will discuss with all of the stakeholders before proceeding with any major change.
We would like to set an initial target of mid-October for the Python 3 release. This will inevitably slip, but we would definitely like it to be out before the beginning of 2020.
--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
Academic Director, CryoEM Core (cryoem.bcm.edu)
Co-Director CIBR Center (www.bcm.edu/research/cibr)
- Tunay will establish one official Python 3 branch, and let everyone know which one it is
- Tunay will create a source installation page with instructions for setting up a build environment (with appropriate Anaconda dependencies) for the Python 3 branch and post here when it's ready.
- If we encounter trivial bugs in the system, like print -> print() or other simple syntactical changes, we will just fix them, regardless of whose code they are in. This will be limited to changes with no risk of changed behavior.
- We are not taking responsibility for fixing all such things, just saying that if we happen to encounter them, we will fix them.
- If anyone encounters problems where the fix could potentially change behavior, like issues with "/" vs "//", please suggest a change to the author/maintainer of the code.
- We will be experimenting with some more significant changes to the system designed to improve numpy interoperability. Once we have a recommendation, we will discuss with all of the stakeholders before proceeding with any major change.
We would like to set an initial target of mid-October for the Python 3 release. This will inevitably slip, but we would definitely like it to be out before the beginning of 2020.
--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
Academic Director, CryoEM Core (cryoem.bcm.edu)
Co-Director CIBR Center (www.bcm.edu/research/cibr)
shadow walker
Jul 2, 2019, 9:09:23 PM7/2/19
to eman2-de...@googlegroups.com
I have just pushed the last of few pull requests related to py3 transition.
All the changes here work with py2. So, hopefully, we will have minimal py3 specific changes left. All the pull requests in the list depend on previous ones, so we will have to deal with them in order. First two are cmake related and only the first one has a thing or two that may need quick reviews and discussions. The second PR is cmake upgrade which will be useful for the boost upgrades.
The current boost version we use is 1.63. Unfortunately, switching to py3 was causing segfaults with on mac. The suggested solution was to switch to the new Boost.Numpy introduced in boost 1.63 and 1.64. The old array API that we currently have was removed in boost 1.65. The third PR in the list deals with switching to the new boost.numpy API. While switching, I removed some code that seemed not be used or necessary and the current tests we have still pass. So, that PR will need some testing. Once we get there, I will work on setting up eman-deps to install the correct versions and post some instructions how setup build environments.
The last PRs are builds with boost versions 1.66-1.69. I have a similar set of builds for py3 for which I have not pushed any PRs yet. But, it looks like boost 1.66 is the only version that works without any additional environment setups. 1.68 and 1.69 require gcc5 or higher, but are OK otherwise. Rest of the boost versions are broken.
For specific technical discussions, I propose to use the corresponding pull requests on GitHub and use this thread for general discussions. Otherwise, it may get very messy very quickly here.
I want to keep the git branches linear, because with merge commits it becomes practically impossible to look at changes. If you submit changes, you don't need to do anything different than what you do normally, but, I will be rebasing, changing some commits which is not the recommended way to work on shared branches. So, you will need to be careful if you have local changes when you are pulling from a remote branch. Instead of running "git pull", you will have to run "git pull --rebase" or change your IDE default for pull strategy from merge to rebase.
--
You received this message because you are subscribed to the Google Groups "EMAN2 and SPARX Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2-develope...@googlegroups.com.
To post to this group, send email to eman2-de...@googlegroups.com.
Visit this group at https://groups.google.com/group/eman2-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2-developers/BD44A1EB-DC81-4F64-8AA3-90D0002E3415%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
shadow_walker
shadow walker
Jul 3, 2019, 12:30:39 AM7/3/19
to eman2-de...@googlegroups.com
Instructions to setup environments with CMake 3.14 and Boost 1.66 are here, https://blake.bcm.edu/emanwiki/EMAN2/Python3.
shadow_walker
shadow walker
Nov 2, 2019, 3:42:17 PM11/2/19
to eman2-de...@googlegroups.com
The branch we need to merge next is py2-boost-164. The relevant pull request is https://github.com/cryoem/eman2/pull/408. Since there is some code changes related to handling numpy arrays, it would be good to test the branch and review the changes if possible. If there are no objections, I will merge the branch next week. Since these changes break Windows builds, I am not sure if we will merge it into master or another branch. The command to install the relevant environment for testing is here, https://blake.bcm.edu/emanwiki/EMAN2/Python3#CA-e722e80c4c1f55578c824a2866294150a7c4f3c7_1. Note, also, that the instructions for source installation have been updated for master, https://blake.bcm.edu/emanwiki/EMAN2/COMPILE_EMAN2_ANACONDA.
After that branch is merged, we will upgrade boost to 1.69 and switch to python 3.
shadow_walker
Steve Ludtke
Nov 2, 2019, 11:13:28 PM11/2/19
to eman2-de...@googlegroups.com
I think it's acceptable for the nightly builds for Windows to be broken briefly during the Python3 upgrading process. Once we make the switch, make fixing it a priority, but we still have the 2.31 binaries people can use.
--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2-developers/CAMkj2E%3DQkksNvydd6Mf7xUG-tgmG7gC2UDyiW-LBx35rbMDZDA%40mail.gmail.com.
shadow walker
Nov 3, 2019, 5:23:12 PM11/3/19
to eman2-de...@googlegroups.com
Branches py2-boost-166 and py2-boost-all are ready. Since we will be upgrading boost anyways, maybe, there isn't any point testing the intermediate boost versions. The conda-install commands for the branches are below.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2-developers/22511B9D-DBF1-4CD8-A301-651C3317D84D%40gmail.com.
--
shadow_walker
Steve Ludtke
Nov 4, 2019, 2:43:58 PM11/4/19
to eman2-de...@googlegroups.com
I can't think of any reason to test intermediate versions. Let's take the plunge.
--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
Academic Director, CryoEM Core (cryoem.bcm.edu)
Co-Director CIBR Center (www.bcm.edu/research/cibr)
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
Academic Director, CryoEM Core (cryoem.bcm.edu)
Co-Director CIBR Center (www.bcm.edu/research/cibr)
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2-developers/CAMkj2EnK8fXSFKB9Ou-wDQoyhsYbpLd%3DnruPfMmxWVQ3Egg1yA%40mail.gmail.com.
shadow walker
Nov 4, 2019, 2:53:11 PM11/4/19
to eman2-de...@googlegroups.com
To clarify, py2-boost-all has boost at 1.69. The command to install the environment is
conda install eman-deps=17.1 cmake=3.14 boost=1.69 -c cryoem -c defaults -c conda-forgeTo view this discussion on the web visit https://groups.google.com/d/msgid/eman2-developers/3E118AC9-CA97-4F1A-8106-78F7C7827475%40gmail.com.
--
shadow_walker
Steve Ludtke
Nov 4, 2019, 9:39:21 PM11/4/19
to eman2-de...@googlegroups.com
CMake Warning at libpyEM/CMakeLists.txt:2 (add_library):
Cannot generate a safe runtime search path for target pyPointArray2 because
files in some directories may conflict with libraries in implicit
directories:
runtime library [libboost_python27.so.1.69.0] in /usr/lib may be hidden by files in:
/home/stevel/EMAN2/miniconda3/envs/test/lib
Some of these libraries may not be found correctly.
Call Stack (most recent call first):
libpyEM/CMakeLists.txt:31 (ADD_PYSTE_LIB)
CMake Warning at libpyEM/CMakeLists.txt:2 (add_library):
Cannot generate a safe runtime search path for target pyTransform2 because
files in some directories may conflict with libraries in implicit
directories:
runtime library [libboost_python27.so.1.69.0] in /usr/lib may be hidden by files in:
/home/stevel/EMAN2/miniconda3/envs/test/lib
Some of these libraries may not be found correctly.
Call Stack (most recent call first):
libpyEM/CMakeLists.txt:30 (ADD_PYSTE_LIB)
CMake Warning at libpyEM/CMakeLists.txt:2 (add_library):
Cannot generate a safe runtime search path for target pyFundamentals2
because files in some directories may conflict with libraries in implicit
directories:
runtime library [libboost_python27.so.1.69.0] in /usr/lib may be hidden by files in:
/home/stevel/EMAN2/miniconda3/envs/test/lib
Some of these libraries may not be found correctly.
Call Stack (most recent call first):
libpyEM/CMakeLists.txt:34 (ADD_PYSTE_LIB)
...
--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Steven Ludtke, Ph.D. <slu...@bcm.edu> Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
Dept. of Biochemistry and Molecular Biology (www.bcm.edu/biochem)
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2-developers/CAMkj2E%3D2uwoxABFCAMHXpUkn3kJgpi9xn69vUgsBddaUcaUzrg%40mail.gmail.com.
shadow walker
Nov 4, 2019, 11:15:00 PM11/4/19
to eman2-de...@googlegroups.com
I've seen this warning for a while. If it doesn't cause any problems, I'll leave it alone for now.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2-developers/0F9B7BD7-94AB-412A-A3ED-DCE5E5C8D3AB%40gmail.com.
--
shadow_walker
shadow walker
Nov 14, 2019, 11:01:11 AM11/14/19
to eman2-de...@googlegroups.com
The branch with boost upgrade to 1.66 will be merged into master as soon as the running tests finish. Due to compiler ABI compatibility issues on Linux, I think we won't upgrade boost any further to avoid any further delays in switching to python 3. You can update your environment with the following command once the changes are in master.
conda install eman-deps=16.1 cmake=3.14 boost=1.66 -c cryoem -c defaults -c conda-forge
shadow_walker
Reply all
Reply to author
Forward
0 new messages