Devsim compiling on Windows (Newbie help)

[Christopher Buurma]

Hi all,

I was looking into Devsim to see if it would work for us. I've spent a lot of time reading the docs and trying to figure out how to compile this without asking, but I'm running out of ideas. Could really use some help, and I'm not experienced with Visual Studio. My ultimate goal is to compile devsim so that I can 'import ds' using any of my python environments (setting the path appropriate each time)

I downloaded the deploytest50 from here (https://github.com/devsim/devsim/releases), and I cloned the git-hub repo from here: https://github.com/devsim/devsim

The python3.exe from the /bin/ folder works fine, but that puts me in a python environment that doesn't have the packages I need. It also doesn't work with PyCharm or IDLE. So I wanted to compile/install it so that I can run 'import ds' in different python environments on my machine. 

I do have Anaconda Python installed, but not as admin. This means that C:\Anaconda does not exist, and my 'include' directories are buried inside of \%APPDATA%\. 

I ran the CMAKE script, and it succeeded generating the visual studio files (I set the python3 flag). Once inside of visual studio I first recieved a bug that '01 cannot be imported' and traced it down to a space instead of a '_' character in the version name XML tag. So after fixing that it fails to import python.h and some other directories. Inside VC++ I added the build path for my anaconda environment of choice, e.g. 'C\users\chris\appdata\local\continuum\anaconda3\envs\py36\include', and some of those errors went away for some packages, but not all. Perhaps something in the build order? or my search path? I'm at a loss as what to check next. 

Here's my output when in visual studio running 'ALL_BUILD'

1>------ Build started: Project: Data, Configuration: Debug Win32 ------

2>------ Build started: Project: math, Configuration: Debug Win32 ------

3>------ Build started: Project: meshing, Configuration: Debug Win32 ------

4>------ Build started: Project: python3api_api, Configuration: Debug Win32 ------

5>------ Build started: Project: python3api_interpreter, Configuration: Debug Win32 ------

6>------ Build started: Project: pythonapi_interpreter, Configuration: Debug Win32 ------

7>------ Build started: Project: utility, Configuration: Debug Win32 ------

8>------ Build started: Project: modelcomp, Configuration: Debug Win32 ------

3>Building Custom Rule D:/Users/cbuurma/Code/Python/devsim/src/meshing/CMakeLists.txt

3>CMake does not need to re-run because D:/Users/cbuurma/Code/Python/devsim/src/meshing/CMakeFiles/generate.stamp is up-to-date.

3>Generating DevsimParser.cc, DevsimParser.hh

3>The system cannot find the path specified.

3>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" exited with code 3.

3>Done building project "meshing.vcxproj" -- FAILED.

1>MaterialDB.cc

2>SuperLUPreconditioner.cc

4>ObjectHolder.cc

5>GetArgs.cc

4>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\ObjectHolder.cc(19): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

5>d:\users\cbuurma\code\python\devsim\src\pythonapi\GetArgs.hh(20): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

4>OutputStream.cc

4>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\OutputStream.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

4>Interpreter.cc

6>GetArgs.cc

4>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\Interpreter.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

4>Generating Code...

5>PythonAppInit.cc

6>d:\users\cbuurma\code\python\devsim\src\pythonapi\GetArgs.hh(20): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

7>base64.cc

6>PythonAppInit.cc

5>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\PythonAppInit.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

5>PythonCommands.cc

4>Done building project "python3api_api.vcxproj" -- FAILED.

8>Building Custom Rule D:/Users/cbuurma/Code/Python/devsim/src/adiff/CMakeLists.txt

8>CMake does not need to re-run because D:/Users/cbuurma/Code/Python/devsim/src/adiff/CMakeFiles/generate.stamp is up-to-date.

8>Generating mcParser.cc, mcParser.hh

5>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\PythonCommands.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

5>CommandHandler.cc

5>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\CommandHandler.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

5>Generating Code...

6>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\PythonAppInit.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

6>PythonCommands.cc

6>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\PythonCommands.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

6>CommandHandler.cc

8>The system cannot find the path specified.

8>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" exited with code 3.

8>Done building project "modelcomp.vcxproj" -- FAILED.

5>Done building project "python3api_interpreter.vcxproj" -- FAILED.

6>D:\Users\cbuurma\Code\Python\devsim\src\pythonapi\CommandHandler.cc(18): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory

6>Generating Code...

2>d:\users\cbuurma\code\python\devsim\src\math\SuperLUData.hh(25): fatal error C1083: Cannot open include file: 'supermatrix.h': No such file or directory

1>D:\Users\cbuurma\Code\Python\devsim\src\Data\MaterialDB.cc(21): fatal error C1083: Cannot open include file: 'sqlite3.h': No such file or directory

6>Done building project "pythonapi_interpreter.vcxproj" -- FAILED.

2>SuperLUData.cc

1>Done building project "Data.vcxproj" -- FAILED.

7>D:\Users\cbuurma\Code\Python\devsim\src\utility\base64.cc(19): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory

7>Done building project "utility.vcxproj" -- FAILED.

2>d:\users\cbuurma\code\python\devsim\src\math\SuperLUData.hh(25): fatal error C1083: Cannot open include file: 'supermatrix.h': No such file or directory

2>SuperLUDataZ.cc

2>d:\users\cbuurma\code\python\devsim\src\math\SuperLUData.hh(25): fatal error C1083: Cannot open include file: 'supermatrix.h': No such file or directory

2>Generating Code...

2>Done building project "math.vcxproj" -- FAILED.

9>------ Build started: Project: devsim_py, Configuration: Release Win32 ------

10>------ Build started: Project: devsim_py3, Configuration: Release Win32 ------

9>LINK : fatal error LNK1181: cannot open input file '..\commands\Release\commands.lib'

10>LINK : fatal error LNK1181: cannot open input file '..\commands\Release\commands.lib'

10>Done building project "devsim_py3.vcxproj" -- FAILED.

9>Done building project "devsim_py.vcxproj" -- FAILED.

11>------ Build started: Project: ALL_BUILD, Configuration: Release Win32 ------

11>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppBuild.targets(398,5): error MSB3191: Unable to create directory "D:\Users\cbuurma\Code\Python\devsim\Release\". Cannot create "D:\Users\cbuurma\Code\Python\devsim\Release" because a file or directory with the same name already exists.

11>Done building project "ALL_BUILD.vcxproj" -- FAILED.

========== Build: 0 succeeded, 11 failed, 14 up-to-date, 0 skipped ==========

Any help would be greatly appreciated! 

[Juan Sanchez]

Hi,

Thanks for your interest in the software.  The windows build is rather
tricky, and not well documented.

DEVSIM has 3 interpreters:
Tcl 8.6
Python 2.7
Python 3.6

Almost all of the python scripts run in python 2, but could be easily
converted using 2to3 for python 3.  Many of the scripts in the testing
directory are written in Tcl.

The deploytest51 was just released.  It builds the Tcl interpreter
from the one available in Anaconda, as opposed to the one from
ActiveState.

Below are instructions for 2 options.  Using a binary release with
option 1 is recommended.  Compiling from source with option 2 requires
a lot more work.

Please let us know if you need any help, or have additional questions.

Regards,

Juan

Option 1

To run devsim, you can just start the Anaconda Prompt with a python 2
or python 3 environment.  By activating the appropriate environment,
you set the paths required for devsim to run.

You then run devsim (python 2) or devsim_py3 (python 3) on the scripts
in the examples or testing directory.  The "mkl" package is needed in
both environments for the blas libraries.  The "tk" package is
required if you would like to run one of the Tcl examples using
devsim_tcl.

For some of the examples, you need to set PYTHONPATH to the base
directory of the devsim installation.


Option 2

Below is a detailed discussion of compiling devsim yourself.  If you
are happy with the binary releases, I suggest following the
instructions above.

The windows release version of devsim is an automated build using the
appveyor ci at https://appveyor.com.  These scripts can be readily
adapted for a software developer environment.

I recommend following the build steps in appveyor.yml.  The paths will
certainly need to be updated from those in the appveyor build
environment.

The most important files are:
appveyor.yml
cmake/appveyor.cmake
scripts/setup_appveyor.sh

For the symdiff submodule:
external/symdiff/appveyor.cmake
external/symdiff/scripts/setup_appveyor.sh

The SYMDIFF_CONFIG and DEVSIM_CONFIG variables in both of the setup
scripts set the base name for the ".cmake" script that gets included.

The ".cmake" files are where you set the paths appropriate to your installation.

The PYTHON, TCL, and MKL variables will point into your Anaconda (or
Miniconda) installation.

I recommend getting the 64 bit build windows dependencies from here:
https://github.com/devsim/devsim_win32_deps/releases/download/VisualStudio2017v3/devsim_win32_deps.zip

This provides the precompiled static libraries for:
SuperLU
zlib
sqlite
CGNS

If a 32-bit build is required, it should be straighforward to download
the devsim_win32_deps repo and adapt and run the build script.

Note that you need to install cygwin to run the bash scripts, bison, flex.

The dist directory contains the package_win32.sh script for creating a
binary distribution from the build.  For the Python 2.7 version, it
also copies the manifest from python.exe to devsim.exe.  Without this
manifest, devsim will not run.

> --
> You received this message because you are subscribed to the Google Groups
> "DEVSIM TCAD" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to devsim+un...@googlegroups.com.
> Visit this group at https://groups.google.com/group/devsim.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/devsim/1d85d0ce-9f9c-4976-ba62-cab8e43a1703%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Christopher Buurma]

Thanks for the quick response. I'll give it a try. 

If I can get a handle on this, I'd like to put together a pip-install branch perhaps using wheels. Then windows and heavy python users could get more exposure. I've already run 2to3 on the exmaple files and it resulted in minimal changes. The biggest changes was how imports work. 

Regarding the pre-compiled binary. That certainly does work, however it is very different than my preferred workflow. I need to open a terminal, point my pythonpath to the directory (or permanent link it), then execute the program pointing it at a python file to test each time I make a change. I generally use an IDE like PyCharm and Jupyter notebooks, code inspection for function signatures and docs, and I use a lot of different python environments with different packages. Being unable to create an environment with devsim in it messes with that workflow. Not a show-stopper but certainly discouraging.

Thanks again, I'll keep at it.

[Juan Sanchez]

Hi Christopher,

Thanks for your efforts. Since the dynamic library dependencies of
the devsim binaries are located in the Anaconda environments, you can
just copy the executables into the appropriate bin directory of your
environments. For example:
cp devsim_py3.exe /path/to/anaconda/envs/python3
cp devsim.exe /path/to/anaconda/envs/python2
cp devsim_tcl.exe /path/to/anaconda/envs/python2

You can copy the python_packages directory from the root installation
directory to the directory of your project, set the PYTHONPATH, or
create some kind of file link. I suspect that you may want to modify
these files anyway. Another strategy would be to modify the scripts
to append the directory containing python_packages using
sys.path.append.

Anaconda typically uses hard file links when installing the same files
across different environments, at least in Linux and Mac OS X.

If you want to have the devsim ds module imported from the python
executable, it should be possible to recompile it as dll. There would
just be some C++ initialization code that would need to called by the
module initialization function.

It looks like this work was started by someone in the past:
https://github.com/weatherhead99/devsim/network
https://github.com/weatherhead99/symdiff/network

I do not know how the IDE works when running batch-oriented scripts,
and when the underlying program maintains state.

My work flow is command line oriented from linux and os x. I run
scripts interactively, only when using the pdb debugger.

Regards,

Juan

> https://groups.google.com/d/msgid/devsim/53899cde-a7a7-4751-8566-05a1d6e87fbe%40googlegroups.com.

[Christopher Buurma]

Juan,

Thanks very much. Indeed creating a conda environment, navigating to it and replacing the python.exe with the devsim_p3.exe did work, and let me load Jupyter and Pycharm without troubles. Setting the python_packages within my PYTHONPATH solved dependencies as well and i was able to run some of the tests.

I'll keep looking at the .dll compilation, and the pip / conda install work. But this way I can at least start running some sims to see if the tool with work for us.

Thanks again,

~Chris

[Juan E. Sanchez]

Hi Chris,

This is great news.

I am not familiar with python packaging. I am open to having DEVSIM
treated as a module, instead of as a program with an embedded python
interpreter. When development was started, CMAKE was not as capable of
creating a large DLL spanning multiple directories.

The testing/CMakeLists.txt file has information about how some of the
tests and examples should be ordered. Test regression results are
available here:
https://github.com/devsim/devsim_tests_win64

The windows test results have not been updated in a while, and rely on
exact text matching. They are CPU dependent, but they should be
reproducible when run on the same machine.

A BJT example is here:
https://github.com/devsim/devsim_bjt_example

Please let me know if you have any questions about the software.

Regards,

Juan