Blocking Issues with libclang.so versioning

193 views
Skip to first unread message

Lewis John Mcgibbney

unread,
May 13, 2015, 9:26:06 PM5/13/15
to xdr...@googlegroups.com
Hi Folks,
I am really struggling to use xdress in my current project. I've been coming back to this for around three weeks now with no luck and it is a serious blocker for me.
Essentially, I have a xdressrc.py file which looks like the following

package = 'runtime'       # top-level python package name
packagedir = 'runtime'    # location of the python package

# wrappers for non-standard types (uints, complex)
extra_types = 'xdress_extra_types'

# List of C++ standard library container template types
# to instantiate and wrap with Cython. See the type
# system documentation for more details.  Note that
# vectors are wrapped as numpy arrays of the appropriate
# type.  If the type has no corresponding primitive C++
# type, then a new numpy dtype is created to handle it.
# For example, this allows the wrapping of vector< vector<int> >
# as an np.array(..., dtype=xd_vector_int).
stlcontainers = [
    ('vector', 'str'),
    ('vector', 'int32'),
    ('vector', 'complex'),
    ('vector', 'float32'),
    ('vector', 'float64'),
    ('vector', ('vector', 'float64')),
    ('set', 'int'),
    ('set', 'str'),
    ('set', 'uint'),
    ('set', 'char'),
    ('map', 'str', 'str'),
    ('map', 'str', 'int'),
    ('map', 'int', 'str'),
    ('map', 'str', 'uint'),
    ('map', 'uint', 'str'),
    ('map', 'uint', 'uint'),
    ('map', 'str', 'float'),
    ('map', 'int', 'int'),
    ('map', 'int', 'bool'),
    ('map', 'int', 'char'),
    ('map', 'int', 'float'),
    ('map', 'uint', 'float'),
    ('map', 'int', 'complex'),
    ('map', 'int', ('set', 'int')),
    ('map', 'int', ('set', 'str')),
    ('map', 'int', ('set', 'uint')),
    ('map', 'int', ('set', 'char')),
    ('map', 'int', ('vector', 'str')),
    ('map', 'int', ('vector', 'int')),
    ('map', 'int', ('vector', 'uint')),
    ('map', 'int', ('vector', 'char')),
    ('map', 'int', ('vector', 'bool')),
    ('map', 'int', ('vector', 'float')),
    ('map', 'int', ('vector', ('vector', 'float64'))),
    ('map', 'int', ('map', 'int', 'bool')),
    ('map', 'int', ('map', 'int', 'char')),
    ('map', 'int', ('map', 'int', 'float')),
    ('map', 'int', ('map', 'int', ('vector', 'bool'))),
    ('map', 'int', ('map', 'int', ('vector', 'char'))),
    ('map', 'int', ('map', 'int', ('vector', 'float'))),
    ]

functions = [('setup_mapping', 'src/espa_geoloc.*'),
             ('get_geoloc_info', 'src/espa_geoloc.*'),
             ('main', 'src/scag.*'),
             ('input_file_io', 'src/scag.*'),
             ('output_file_io', 'src/scag.*'),
             ('model_types_io', 'src/scag.*'),
             ('next_field', 'src/scag.*'),
             ('next_blank', 'src/scag.*'),
             ('spectral_library_io', 'src/scag.*'),
             ('initialize_models', 'src/scag.*'),
             ('next_comma', 'src/scag.*'),
             ('mixture', 'src/scag.*'),
             ('scag_process_thread', 'src/scag.*'),
             ('fraction', 'src/scag.*'),
             ('fit_constraints', 'src/scag.*'),
             ('shade_normalization', 'src/scag.*'),
             ('signal_handler', 'src/scag.*')
             ]

I am using the following dependencies

lmcgibbn@LMC-032857 /usr/local/jpl/snow/trunk/tmscag/trunk(master) $ xdress --version
XDress: 0.5-dev
pycparser (optional): 2.13
lxml (optional): 3.4.2
NumPy (optional): 1.6.2
Cython (optional): 0.21.1

I am trying to use xdress as follows

lmcgibbn@LMC-032857 /usr/local/jpl/snow/trunk/tmscag/trunk(master) $ xdress -p clang -I/usr/include/libxml2 --debug
autoall: discovering API names
autoall: no API names to discover!
dtypes: registering numpy dtypes
stlwrap: registering C++ standard library types
autodescribe: scraping C/C++ APIs from source
autodescribe: describing setup_mapping
Traceback (most recent call last):
  File "/usr/local/bin/xdress", line 5, in <module>
    pkg_resources.run_script('xdress==0.5-dev', 'xdress')
  File "/Library/Python/2.7/site-packages/setuptools-4.0.1-py2.7.egg/pkg_resources.py", line 534, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Library/Python/2.7/site-packages/setuptools-4.0.1-py2.7.egg/pkg_resources.py", line 1434, in run_script
    execfile(script_filename, namespace, namespace)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/EGG-INFO/scripts/xdress", line 6, in <module>
    main()
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/main.py", line 225, in main
    plugins.execute()
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/plugins.py", line 374, in execute
    self.exit(e)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/plugins.py", line 372, in execute
    plugin.execute(rc)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/autodescribe.py", line 2303, in execute
    self.compute_functions(rc)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/autodescribe.py", line 2454, in compute_functions
    desc = self.compute_desc(fnc, 'func', rc)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/autodescribe.py", line 2389, in compute_desc
    clang_includes=rc.clang_includes)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/autodescribe.py", line 2248, in describe
    language=language, clang_includes=clang_includes)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/autodescribe.py", line 1163, in clang_describe
    clang_includes=clang_includes)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/astparsers.py", line 119, in memoizer
    value = f(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/astparsers.py", line 246, in clang_parse
    index = cindex.Index.create()
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/clang/cindex.py", line 2204, in create
    return Index(conf.lib.clang_createIndex(excludeDecls, 0))
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/clang/cindex.py", line 162, in __get__
    value = self.wrapped(instance)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/clang/cindex.py", line 3537, in lib
    register_functions(lib, not Config.compatibility_check)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/clang/cindex.py", line 3484, in register_functions
    register_function(lib, item, ignore_errors)
  File "/Library/Python/2.7/site-packages/xdress-0.5_dev-py2.7.egg/xdress/clang/cindex.py", line 3464, in register_function
    raise LibclangError(msg)
xdress.clang.cindex.LibclangError: dlsym(0x7fb9a947c640, clang_isTemplateArg): symbol not found. Please ensure that your python bindings are compatible with your libclang.so version.

My clang version is as follows

lmcgibbn@LMC-032857 /usr/local/jpl/snow/trunk/tmscag/trunk(master) $ clang --version
clang version 3.6.0 (tags/RELEASE_360/final)
Target: x86_64-apple-darwin13.4.0
Thread model: posix

I have been installing llvm and clang from source over and over again, configuring and building the codebases from SVN, failing, uninstalling them, reinstalling them, failing... lather rise repeat and I acknowledge I now need some help to drive this to a solution!

There are various threads on this mailing list which mention the clang version which comes packaged with xdress however I am not having any luck getting past the above issue.

I t would be greatly appreciated if anyone on this list, who has possibly encountered the same or similar issues is able to help me navigate this issue. It is literally driving me mental :(
Thanks in advance.
Lewis

Lewis John Mcgibbney

unread,
May 13, 2015, 10:31:33 PM5/13/15
to xdr...@googlegroups.com
Hi Folks,
OK some updated here, I checked out the following cwrap fork and ran a test to see if I would get any results.

lmcgibbn@LMC-032857 /usr/local/cwrap_libclang(master) $ python runtest.py /usr/local/jpl/snow/trunk/tmscag/trunk/src/meta_stack.h
['runtest.py', '/usr/local/jpl/snow/trunk/tmscag/trunk/src/meta_stack.h']
------------------------

Parsing /usr/local/jpl/snow/trunk/tmscag/trunk/src/meta_stack.h

Traceback (most recent call last):
  File "runtest.py", line 24, in <module>
    config_clang.generate()
  File "/usr/local/cwrap_libclang/cwrap/config.py", line 31, in generate
    cw_asts = frontend.generate_asts(self)
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/__init__.py", line 44, in generate_asts
    ast_items = gen_c_ast(path, include_dirs, language)
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/__init__.py", line 17, in gen_c_ast
    c_ast = clang_parser.parse(header_path, include_dirs, language)
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/clang_parser.py", line 891, in parse
    parser.parse(cfile, include_dirs, language)
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/clang_parser.py", line 109, in parse
    index = clang.cindex.Index.create()
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/clang/cindex.py", line 2204, in create
    return Index(conf.lib.clang_createIndex(excludeDecls, 0))
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/clang/cindex.py", line 162, in __get__
    value = self.wrapped(instance)
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/clang/cindex.py", line 3537, in lib
    register_functions(lib, not Config.compatibility_check)
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/clang/cindex.py", line 3484, in register_functions
    register_function(lib, item, ignore_errors)
  File "/usr/local/cwrap_libclang/cwrap/frontends/clang/clang/cindex.py", line 3464, in register_function
    raise LibclangError(msg)
cwrap.frontends.clang.clang.cindex.LibclangError: dlsym(0x7f81c9e00620, clang_isTemplateArg): symbol not found. Please ensure that your python bindings are compatible with your libclang.so version.

This is certainly a problem with my version of libclang. Can anyone suggest a suitable version to use?
Thanks again in advance for any help. Once I get this sorted out I am hoping my headache will go away :)
Lewis

Anthony Scopatz

unread,
May 13, 2015, 10:36:05 PM5/13/15
to Lewis John Mcgibbney, xdr...@googlegroups.com
Hi Lewis,

I think that this is an issue with clang v3.6, which xdress has not been updated for. (Clang is notorious for breaking APIs on minor version numbers).  I believe that v3.5 works, and I am almost certain that v3.4 does too. I also believe that GCC-XML for gcc 4.9+ are also broken, too.

Unfortunately, I don't have the time personally to update these anymore.  However, I will still review and merge pull requests if anyone wants to dive in and do development.

Be Well
Anthony

--
You received this message because you are subscribed to the Google Groups "xdress" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/xdress/b84c2b0c-bd02-49c5-b7a8-9e7e795a30cb%40googlegroups.com.

Matthew McCormick

unread,
May 13, 2015, 11:47:08 PM5/13/15
to Anthony Scopatz, Lewis John Mcgibbney, xdress, Brad King
Hi Lewis, Anthony,

Another option may be CastXML [1]. This is a Clang-based GCCXML
replacement that works with Clang, GCC, and MSVC. It builds against
Clang 3.6.0. I did most of the dirty work of putting it into xdress
here [2]. What's left is the fun of making the tests pass. Alas, I
don't have enough time to allocate to this fun stuff, so feel free to
run with it ;-).

Happy hacking,
Matt

[1] https://github.com/CastXML/CastXML

[2] https://github.com/thewtex/xdress/tree/castxml
> https://groups.google.com/d/msgid/xdress/CAPk-6T7eSqiryKomYO7_iaNsa-RC0Ytkr1fECYKUbv4zKPZQUQ%40mail.gmail.com.

Lewis John Mcgibbney

unread,
May 14, 2015, 1:10:09 PM5/14/15
to xdr...@googlegroups.com, lewis.m...@gmail.com, sco...@gmail.com, brad...@kitware.com
Thanks both for your replies.
I think I'll patch the website with some versioning such that people don't run into the problems with (particularly) clang.
@Matthew I honestly don't know if I can dedicate time to getting the tests to pass... I've got a couple of pressing issues right now. :(
Thank you for the references.

Anthony Scopatz

unread,
May 14, 2015, 3:41:43 PM5/14/15
to Lewis John Mcgibbney, xdr...@googlegroups.com, brad...@kitware.com
Hi Lewis, Thanks!

Hi Matt, I noticed that CastXML from the SciPy talk list.  It looks super promising.  It could definitely solve the problems that we have been having with xdress.  Could you open up a PR for your CastXML support, please?  If this is going to be a robust solution, I'd be interested in putting some time/effort into it, or trying to find someone who could.

Be Well
Anthony

Lewis John Mcgibbney

unread,
May 14, 2015, 8:25:34 PM5/14/15
to xdr...@googlegroups.com, lewis.m...@gmail.com
I've now tried clang v3.6, 3.5 and 3.4 and none are working for me with the same message as before
xdress.clang.cindex.LibclangError: dlsym(0x7fc311e9f1c0, clang_isTemplateArg): symbol not found. Please ensure that your python bindings are compatible with your libclang.so version.


lmcgibbn@LMC-032857 /usr/local/jpl/snow/trunk/tmscag/trunk(master) $ clang --version
clang version 3.5.0 (tags/RELEASE_350/final)

Target: x86_64-apple-darwin13.4.0
Thread model: posix

Lewis John Mcgibbney

unread,
May 14, 2015, 8:27:53 PM5/14/15
to xdr...@googlegroups.com
I've set up a symlink as follows

/usr/local/lib/libclang.dylib -> /usr/local/clang+llvm-3.5.0-macosx-apple-darwin/lib/libclang.dylib

Lewis John Mcgibbney

unread,
May 14, 2015, 9:01:48 PM5/14/15
to xdr...@googlegroups.com
I've now tried clang version 3.4.0, 3.4.1, 3.4.2, 3.5.0, 3.5.2 and 3.6.0
None of the above work on MacOSX.
I'm going to log in to another box and try using xdress there. I literally cannot get the software to run from source!

Anthony Scopatz

unread,
May 14, 2015, 9:17:58 PM5/14/15
to Lewis John Mcgibbney, xdr...@googlegroups.com
Uggh I am sorry this is so frustrating!

--
You received this message because you are subscribed to the Google Groups "xdress" group.

Lewis John Mcgibbney

unread,
May 14, 2015, 9:20:18 PM5/14/15
to xdr...@googlegroups.com
I've also tried clang versions 3.0, 3.1, 3.2 and 3.3.

This must be something to do with either my OS or with my Python version.

Mac OSX v10.9.5
Python 2.7.5

Is Python 3.X a prerequisite for running clang and subsequently xdress? Does anyone know?
Thanks
Lewis

Anthony Scopatz

unread,
May 14, 2015, 9:22:52 PM5/14/15
to Lewis John Mcgibbney, xdr...@googlegroups.com
Hi Lewis,

Python 3 is not a prereq.

Be Well
Anthony

--
You received this message because you are subscribed to the Google Groups "xdress" group.

Lewis John Mcgibbney

unread,
May 14, 2015, 9:42:12 PM5/14/15
to xdr...@googlegroups.com, lewis.m...@gmail.com
I'm going to shift to CentOS and see if I can get it running there.
Is anyone else able to get dress and clang running nicely on Mac OSX?

Matthew McCormick

unread,
May 15, 2015, 10:51:22 AM5/15/15
to Anthony Scopatz, Lewis John Mcgibbney, xdress, Brad King

Anthony Scopatz

unread,
May 15, 2015, 11:00:23 AM5/15/15
to Matthew McCormick, Lewis John Mcgibbney, xdress, Brad King
Thanks Matt!

Anthony Scopatz

unread,
May 15, 2015, 11:11:16 AM5/15/15
to Matthew McCormick, Lewis John Mcgibbney, xdress, Brad King
Also Lewis, I would recommend that you try going the CastXML route, since this has a hope of working.

Lewis John Mcgibbney

unread,
May 15, 2015, 1:08:25 PM5/15/15
to Anthony Scopatz, Matthew McCormick, xdress, Brad King
Thank. You gents 
--
Lewis

Lewis John Mcgibbney

unread,
May 26, 2015, 7:58:59 PM5/26/15
to xdr...@googlegroups.com, lewis.m...@gmail.com, brad...@kitware.com, matthew.m...@gmail.com
Hi Folks.
I just wanted to close this thread out.
I ended up running this on an Ubuntu 14.04 VM with xdress 0.5dev (master).
Based on the following packages

$ apt-get install -y gccxml cython python-numpy python-pycparser python-nose cmake python-lxml llvm-dev libedit-dev libclang-3.5-dev libxml2-dev

Everything seemed to work OK first time!
It is annoying that MacOSX integration is a painpoint. I really don't have the time to get it working 100% and even ended up finding bugs in clang code for MacOSX which I logged and provided a fix for!

Anthony Scopatz

unread,
May 27, 2015, 4:03:37 AM5/27/15
to Lewis John Mcgibbney, xdr...@googlegroups.com, brad...@kitware.com, matthew.m...@gmail.com
Reply all
Reply to author
Forward
0 new messages