[reportlab-users] Symbol not found in flat namespace when running on macOS 12.0.1

[Markus Gerards]

Hello everybody,

I have issues running my Django application that has reportlab as an indirect dependency from easy_thumbnails.
The error is:

Invalid template library specified. ImportError raised when trying to load 'backend.templatetags.get_thumbnail': dlopen(/Users/markusgerards/.pyenv/versions/myapp/lib/python3.9/site-packages/reportlab/graphics/_renderPM.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_FT_Done_Face'

My environment is:

macOS 12.0.1 with XCode 13.1 installed
Python 3.9.7 within a virtual env (via Homebrew)
Reportlab 3.6.2 (as a dependency from easy_thumbnails 2.8)
Freetype 2.11.0 (via Homebrew)

I took a look into the problematic .so-file. Here is what otool says:

/Users/markusgerards/.pyenv/versions/myapp/lib/python3.9/site-packages/reportlab/graphics/_renderPM.cpython-39-darwin.so:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)

I suspect freetype should be listed there as a load command, am I right? Is it possible, that this fails due to some stricter checks for symbols in recent XCode versions? When searching through the interwebs I stumbled upon this issue: https://github.com/openzfsonosx/openzfs/pull/75

Could it be, that reportlab has the same problem?

As I am no lib-genius I suspect that this has to be fixed by reportlab, right?

Thanks!
Markus

[Robin Becker]

Hi Markus,

I don't have a proper mac and have been building out these darwin dylibs using a github action. I suspect you are right
and that this is a version compatibility issue. However, I have no real idea, what version of OS/xcode is the issue here.

I looked at the 3.9 log here and at least for some architectures the tests seem to have run without this error see eg

https://github.com/MrBitBucket/reportlab-ci/runs/3923521082?check_suite_focus=true

however I think we are running system pythons and I see the following

CIBW_TEST_SKIP: "*-manylinux_{aarch64,ppc64le,s390x} *-macosx_arm64 *-macosx_universal2:arm64"

So perhaps the arm64 build has this problem. I believe I added the test skips because of problems with the imaging
library, but it could also be a problem with the freetype lib. However, looking at the build logs I see this

########## SUMMARY INFO #########
##### setup-python-3.9.7-macosx-11.0-arm64: ================================================
##### setup-python-3.9.7-macosx-11.0-arm64: Attempting build of _rl_accel
##### setup-python-3.9.7-macosx-11.0-arm64: extensions from 'src/rl_addons/rl_accel'
##### setup-python-3.9.7-macosx-11.0-arm64: ================================================
##### setup-python-3.9.7-macosx-11.0-arm64: ===================================================
##### setup-python-3.9.7-macosx-11.0-arm64: Attempting build of _renderPM
##### setup-python-3.9.7-macosx-11.0-arm64: extensions from 'src/rl_addons/renderPM'
##### setup-python-3.9.7-macosx-11.0-arm64: ===================================================
##### setup-python-3.9.7-macosx-11.0-arm64: will use package libart 2.3.21
##### setup-python-3.9.7-macosx-11.0-arm64: # installing with freetype version 21
##### setup-python-3.9.7-macosx-11.0-arm64: ################################################
##### setup-python-3.9.7-macosx-11.0-arm64: Downloading standard T1 font curves
##### setup-python-3.9.7-macosx-11.0-arm64: Finished download of standard T1 font curves
##### setup-python-3.9.7-macosx-11.0-arm64: Finished creation of _glyphlist.py

so at least for that build it claims to have built with freetype version 21 and the compile command was

> gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -g -arch arm64 -DRENDERPM_FT -DLIBART_COMPILATION -DLIBART_VERSION=2.3.21 -Isrc/rl_addons/renderPM -Isrc/rl_addons/renderPM/libart_lgpl -Isrc/rl_addons/renderPM/gt1 -I/usr/local/include/freetype2 -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/rl_addons/renderPM/_renderPM.c -o build/temp.macosx-11.0-arm64-3.9/src/rl_addons/renderPM/_renderPM.o

the load command contains this

> -L/usr/local/lib -L/usr/lib -L/Library/Frameworks/Python.framework/Versions/3.9/lib -lfreetype -o build/lib.macosx-11.0-arm64-3.9/reportlab/graphics/_renderPM.cpython-39-darwin.so

so theoretically that dylib ought to contain the right stuff. However, it's for 11.0 not the 12.0 you have.

As you are using homebrew what happens when you just do brew reportlab or do they punt to downloading from pypi?

You can well blame us for lack of support, but I suspect this is just a problem with Apple constantly degrading old
support to drag people into upgrading. I need to build a new version and will check if cibuildwheel now supports a more
modern set of macosx builds.
--
Robin Becker

> _______________________________________________
> reportlab-users mailing list
> reportl...@lists2.reportlab.com
> https://pairlist2.pair.net/mailman/listinfo/reportlab-users
>

--
Robin Becker
_______________________________________________
reportlab-users mailing list
reportl...@lists2.reportlab.com
https://pairlist2.pair.net/mailman/listinfo/reportlab-users

[Markus Gerards]

Hi Robin,

thanks for your response! I’m not blaming you for anything. The
fruit-company is doing there stuff and breaking things while moving
seems to be in their DNA so….

In the meantime I digged around Stackoverflow and I think I found a
solution though I’m still no expert and have no in-depth-knowledge of
pip, but I managed to build a fresh instance of reportlab with this
command:

pip install reportlab --force-reinstall --no-cache-dir
--global-option=build_ext

I think this forces pip to run setup.py of reportlab for a fresh build.
Am I right? After running this command, everything works as intended…
as far as I tested this for now.

Markus

[Robin Becker]

On 15/11/2021 13:12, Markus Gerards wrote:

> Hello everybody,
>
> I have issues running my Django application that has reportlab as an indirect dependency from easy_thumbnails.
> The error is:

........
for what it's worth I think these are the environments supported by github actions

https://github.com/actions/virtual-environments

cibuildwheel does support some others by using different docker environments.

[Robin Becker]

On 15/11/2021 20:47, Markus Gerards wrote:
> Hi Robin,
>
> thanks for your response! I’m not blaming you for anything. The fruit-company is doing there stuff and breaking things
> while moving seems to be in their DNA so….
>
> In the meantime I digged around Stackoverflow and I think I found a solution though I’m still no expert and have no
> in-depth-knowledge of pip, but I managed to build a fresh instance of reportlab with this command:
>
> pip install reportlab --force-reinstall --no-cache-dir --global-option=build_ext
>
> I think this forces pip to run setup.py of reportlab for a fresh build. Am I right? After running this command,
> everything works as intended… as far as I tested this for now.

this makes sense, but I suppose you don't have the build log as pip probably hides the build_ext command output. What
does your otool say about the created package ie is there a _renderPM.so somewhere in the venv lib. That was the thing
that used freetype. So the renderPM package can use ttf fonts.
>
> Markus
>
,,,,

[Markus Gerards]

Hi Robin,

> this makes sense, but I suppose you don't have the build log as pip
> probably hides the build_ext command output. What does your otool say
> about the created package ie is there a _renderPM.so somewhere in the
> venv lib. That was the thing that used freetype. So the renderPM
> package can use ttf fonts.

hmmm… it still does not reference anything of freetype…. But it
still works, so I really don’t know, what was the problem before. I
only saw other libraries that where you can read out the reference of
the shared library and my assumption was that the same is true with this
one…
But I think running pip with the additional command is a feasible
workaround for the moment.
I guess things are getting more transparent when macOS 12 hits the build
pipelines.

Thanks!
Markus

[Robin Becker]

On 16/11/2021 14:46, Markus Gerards wrote:
> Hi Robin,
>
>> this makes sense, but I suppose you don't have the build log as pip probably hides the build_ext command output. What
>> does your otool say about the created package ie is there a _renderPM.so somewhere in the venv lib. That was the thing
>> that used freetype. So the renderPM package can use ttf fonts.
>
> hmmm… it still does not reference anything of freetype…. But it still works, so I really don’t know, what was the
> problem before. I only saw other libraries that where you can read out the reference of the shared library and my
> assumption was that the same is true with this one…

inaa linux build I see this sort of thing with ldd

> $ ldd build/lib.linux-x86_64-3.10/reportlab/graphics/_renderPM.cpython-310-x86_64-linux-gnu.so
> linux-vdso.so.1 (0x00007ffdcb356000)
> libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f5f0753a000)
> libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f5f07519000)
> libc.so.6 => /usr/lib/libc.so.6 (0x00007f5f0734d000)
> libz.so.1 => /usr/lib/libz.so.1 (0x00007f5f07333000)
> libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f5f07320000)
> libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f5f072e9000)
> libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007f5f0720e000)
> libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x00007f5f07200000)
> /usr/lib64/ld-linux-x86-64.so.2 (0x00007f5f07651000)
> libm.so.6 => /usr/lib/libm.so.6 (0x00007f5f070bc000)
> libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007f5f07097000)
> libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f5f06f61000)
> libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x00007f5f06f3e000)
> libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007f5f06ec5000)

did you try the brew install reportlab route? It's nice to know that if we can always fall back on simple. I think brew
requires xcode availability for compiling extensions so perhaps your pip command is just as easy.

> But I think running pip with the additional command is a feasible workaround for the moment.
> I guess things are getting more transparent when macOS 12 hits the build pipelines.
>
> Thanks!
> Markus

...........
--
Robin Becker

[Markus Gerards]

Hi Robin,

> did you try the brew install reportlab route? It's nice to know that
> if we can always fall back on simple. I think brew requires xcode
> availability for compiling extensions so perhaps your pip command is
> just as easy.

not yet as the pip command worked and I’m not yet keen enough to
change that running system ;)

Markus

[Robin Becker]

On 17/11/2021 14:46, Markus Gerards wrote:
> Hi Robin,
>
>> did you try the brew install reportlab route? It's nice to know that if we can always fall back on simple. I think
>> brew requires xcode availability for compiling extensions so perhaps your pip command is just as easy.
>
> not yet as the pip command worked and I’m not yet keen enough to change that running system ;)
>
> Markus

no worries about that. My boss tried a couple of things and brew install reportlab did not work for him. Not broken,
don't fix is probably the right approach

thanks for the detective work re pip install command.
--
Robin Becker