Resolving lazy_load/importlib/scipy issues resulting from stream/core.load

[Graham Coleman]

Greetings librosa community,

I was wondering if someone could suggest anything for a blocking bug that popped up unexpectedly in some of my scripts. From my perspective, librosa.stream and librosa.core.load appear to kick off bugs resulting in loading scipy.

When I use the code path that uses librosa.stream, I get the following:

(cbFilter) C:\Users\Graham\Desktop\projects\cbFilter>python -m cbFilter.test_basic
Traceback (most recent call last):
  File "C:\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\test_basic.py", line 50, in <module>
    test_basic()
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\test_basic.py", line 34, in test_basic
    srcDescList.append( Modesc(f) )
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\modesc.py", line 151, in __init__
    stream = librosa.stream(path,
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\lazy_loader\__init__.py", line 77, in __getattr__
    attr = getattr(submod, name)
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\lazy_loader\__init__.py", line 76, in __getattr__
    submod = importlib.import_module(submod_path)
  File "C:\python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\librosa\core\audio.py", line 13, in <module>
    import scipy.signal
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\scipy\signal\__init__.py", line 312, in <module>
    from ._upfirdn import upfirdn
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\scipy\signal\_upfirdn.py", line 36, in <module>
    from ._upfirdn_apply import _output_len, _apply, mode_enum
  File "_upfirdn_apply.pyx", line 1, in init scipy.signal._upfirdn_apply
TypeError: numpy.ndarray is not a type object

When I use the branch that uses librosa.load.core, I get the following:

(cbFilter) C:\Users\Graham\Desktop\projects\cbFilter>python -m cbFilter.test_basic
Traceback (most recent call last):
  File "C:\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\test_basic.py", line 49, in <module>
    test_basic()
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\test_basic.py", line 33, in test_basic
    srcDescList.append( Modesc(f, doStream=False) )
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\modesc.py", line 164, in __init__
    y,sr = librosa.core.load( path, sr=None, mono=False )
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\lazy_loader\__init__.py", line 76, in __getattr__
    submod = importlib.import_module(submod_path)
  File "C:\python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\librosa\core\audio.py", line 13, in <module>
    import scipy.signal
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\scipy\signal\__init__.py", line 312, in <module>
    from ._upfirdn import upfirdn
  File "C:\Users\Graham\Envs\cbFilter\lib\site-packages\scipy\signal\_upfirdn.py", line 36, in <module>
    from ._upfirdn_apply import _output_len, _apply, mode_enum
  File "_upfirdn_apply.pyx", line 1, in init scipy.signal._upfirdn_apply
TypeError: numpy.ndarray is not a type object

This is being run on Windows 10 with the following configuration:

(cbFilter) C:\Users\Graham\Desktop\projects\cbFilter>python --version
Python 3.9.5

(cbFilter) C:\Users\Graham\Desktop\projects\cbFilter>pip list
Package             Version      Editable project location
------------------- ------------ -----------------------------------------
annoy               1.17.1
appdirs             1.4.4
asttokens           2.2.1
audioread           3.0.0
backcall            0.2.0
cbFilter            0.1          c:\users\graham\desktop\projects\cbfilter
certifi             2022.12.7
cffi                1.15.1
charset-normalizer  3.1.0
colorama            0.4.6
contourpy           1.0.7
cycler              0.11.0
decorator           5.1.1
executing           1.2.0
fonttools           4.39.3
h5py                3.8.0
idna                3.4
importlib-resources 5.12.0
ipython             8.12.0
jedi                0.18.2
joblib              1.2.0
kiwisolver          1.4.4
lazy_loader         0.2
librosa             0.10.0.post2
llvmlite            0.39.1
matplotlib          3.7.1
matplotlib-inline   0.1.6
msgpack             1.0.5
numba               0.56.4
numpy               1.23.5
packaging           23.0
pandas              2.0.0
parso               0.8.3
pickleshare         0.7.5
Pillow              9.5.0
pip                 23.0.1
plotly              5.14.1
pooch               1.6.0
prompt-toolkit      3.0.38
pure-eval           0.2.2
pyarrow             11.0.0
pycparser           2.21
Pygments            2.15.0
pyparsing           3.0.9
python-dateutil     2.8.2
pytz                2023.3
requests            2.28.2
resampy             0.4.2
scikit-learn        1.2.2
scipy               1.10.1
seaborn             0.12.2
setuptools          67.6.1
six                 1.16.0
soundfile           0.12.1
soxr                0.3.5
stack-data          0.6.2
tenacity            8.2.2
threadpoolctl       3.1.0
traitlets           5.9.0
typing_extensions   4.5.0
tzdata              2023.3
urllib3             1.26.15
wcwidth             0.2.6
wheel               0.38.4
zipp                3.15.0

Any hints to fixing this would be appreciated!

best regards

Graham Coleman

[Brian McFee]

This is impossible to debug without seeing your actual code, but my first guess is an installation error with scipy.

For reference, my development machine has the same versions of scipy and lazy loader as you (but running linux).  Our windows CI (windows server 2022, 10.0.20348) test environment has the following packages:

# packages in environment at C:\Miniconda\envs\test:

#

# Name Version Build Channel

aom 3.5.0 h63175ca_0 conda-forge

appdirs 1.4.4 pyh9f0ad1d_0 conda-forge

attrs 22.2.0 pyh71513ae_0 conda-forge

audioread 3.0.0 py310h5588dad_1 conda-forge

brotli 1.0.9 hcfcfb64_8 conda-forge

brotli-bin 1.0.9 hcfcfb64_8 conda-forge

brotlipy 0.7.0 py310h8d17308_1005 conda-forge

bzip2 1.0.8 h8ffe710_4 conda-forge

ca-certificates 2022.12.7 h5b45459_0 conda-forge

certifi 2022.12.7 pyhd8ed1ab_0 conda-forge

cffi 1.15.1 py310h628cb3f_3 conda-forge

charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge

colorama 0.4.6 pyhd8ed1ab_0 conda-forge

contourpy 1.0.7 py310h232114e_0 conda-forge

coverage 7.2.2 py310h8d17308_0 conda-forge

cryptography 40.0.1 py310h6e82f81_0 conda-forge

cycler 0.11.0 pyhd8ed1ab_0 conda-forge

decorator 5.1.1 pyhd8ed1ab_0 conda-forge

exceptiongroup 1.1.1 pyhd8ed1ab_0 conda-forge

expat 2.5.0 h63175ca_1 conda-forge

ffmpeg 5.1.2 gpl_h5b1d025_106 conda-forge

font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge

font-ttf-inconsolata 3.000 h77eed37_0 conda-forge

font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge

font-ttf-ubuntu 0.83 hab24e00_0 conda-forge

fontconfig 2.14.2 hbde0cde_0 conda-forge

fonts-conda-ecosystem 1 0 conda-forge

fonts-conda-forge 1 0 conda-forge

fonttools 4.39.3 py310h8d17308_0 conda-forge

freetype 2.12.1 h546665d_1 conda-forge

gettext 0.21.1 h5728263_0 conda-forge

glib 2.74.1 h12be248_1 conda-forge

glib-tools 2.74.1 h12be248_1 conda-forge

gst-plugins-base 1.22.0 h001b923_2 conda-forge

gstreamer 1.22.0 h6b5321d_2 conda-forge

icu 72.1 h63175ca_0 conda-forge

idna 3.4 pyhd8ed1ab_0 conda-forge

importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge

iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge

intel-openmp 2023.0.0 h57928b3_25922 conda-forge

jinja2 3.1.2 pyhd8ed1ab_1 conda-forge

joblib 1.2.0 pyhd8ed1ab_0 conda-forge

kiwisolver 1.4.4 py310h232114e_1 conda-forge

krb5 1.20.1 heb0366b_0 conda-forge

lame 3.100 hcfcfb64_1003 conda-forge

lazy_loader 0.2 pyhd8ed1ab_0 conda-forge

lcms2 2.15 h3e3b177_1 conda-forge

lerc 4.0.0 h63175ca_0 conda-forge

libblas 3.9.0 16_win64_mkl conda-forge

libbrotlicommon 1.0.9 hcfcfb64_8 conda-forge

libbrotlidec 1.0.9 hcfcfb64_8 conda-forge

libbrotlienc 1.0.9 hcfcfb64_8 conda-forge

libcblas 3.9.0 16_win64_mkl conda-forge

libclang 16.0.0 default_h8b4101f_1 conda-forge

libclang13 16.0.0 default_h45d3cf4_1 conda-forge

libdeflate 1.18 hcfcfb64_0 conda-forge

libexpat 2.5.0 h63175ca_1 conda-forge

libffi 3.4.2 h8ffe710_5 conda-forge

libflac 1.4.2 h63175ca_0 conda-forge

libglib 2.74.1 he8f3873_1 conda-forge

libhwloc 2.9.0 h51c2c0f_0 conda-forge

libiconv 1.17 h8ffe710_0 conda-forge

libjpeg-turbo 2.1.5.1 hcfcfb64_0 conda-forge

liblapack 3.9.0 16_win64_mkl conda-forge

libogg 1.3.4 h8ffe710_1 conda-forge

libopus 1.3.1 h8ffe710_1 conda-forge

libpng 1.6.39 h19919ed_0 conda-forge

libsndfile 1.2.0 h2628c91_0 conda-forge

libsqlite 3.40.0 hcfcfb64_0 conda-forge

libtiff 4.5.0 h6c8260b_6 conda-forge

libvorbis 1.3.7 h0e60522_0 conda-forge

libwebp-base 1.3.0 hcfcfb64_0 conda-forge

libxcb 1.13 hcd874cb_1004 conda-forge

libxml2 2.10.3 hc3477c8_6 conda-forge

libzlib 1.2.13 hcfcfb64_4 conda-forge

llvmlite 0.39.1 py310hb84602e_1 conda-forge

m2w64-gcc-libgfortran 5.3.0 6 conda-forge

m2w64-gcc-libs 5.3.0 7 conda-forge

m2w64-gcc-libs-core 5.3.0 7 conda-forge

m2w64-gmp 6.1.0 2 conda-forge

m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge

markupsafe 2.1.2 py310h8d17308_0 conda-forge

matplotlib 3.7.1 py310h5588dad_0 conda-forge

matplotlib-base 3.7.1 py310h51140c5_0 conda-forge

mkl 2022.1.0 h6a75c08_874 conda-forge

mpg123 1.31.3 h63175ca_0 conda-forge

msgpack-python 1.0.5 py310h232114e_0 conda-forge

msys2-conda-epoch 20160418 1 conda-forge

munkres 1.1.4 pyh9f0ad1d_0 conda-forge

numba 0.56.4 py310h19bcfe9_0 conda-forge

numpy 1.23.5 py310h4a8f9c9_0 conda-forge

openh264 2.3.1 h63175ca_2 conda-forge

openjpeg 2.5.0 ha2aaf27_2 conda-forge

openssl 3.1.0 hcfcfb64_0 conda-forge

packaging 23.0 pyhd8ed1ab_0 conda-forge

pcre2 10.40 h17e33f8_0 conda-forge

pillow 9.4.0 py310h3dcae36_2 conda-forge

pip 23.0.1 pyhd8ed1ab_0 conda-forge

pluggy 1.0.0 pyhd8ed1ab_5 conda-forge

ply 3.11 py_1 conda-forge

pooch 1.6.0 pyhd8ed1ab_0 conda-forge

pthread-stubs 0.4 hcd874cb_1001 conda-forge

pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge

pycparser 2.21 pyhd8ed1ab_0 conda-forge

pyopenssl 23.1.1 pyhd8ed1ab_0 conda-forge

pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge

pyqt 5.15.7 py310h1fd54f2_3 conda-forge

pyqt5-sip 12.11.0 py310h00ffb61_3 conda-forge

pysocks 1.7.1 pyh0701188_6 conda-forge

pysoundfile 0.12.1 pyhd8ed1ab_0 conda-forge

pytest 7.2.2 pyhd8ed1ab_0 conda-forge

pytest-cov 4.0.0 pyhd8ed1ab_0 conda-forge

pytest-mpl 0.16.1 pyhd8ed1ab_0 conda-forge

python 3.10.10 h4de0772_0_cpython conda-forge

python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge

python_abi 3.10 3_cp310 conda-forge

qt-main 5.15.8 h7f2b912_9 conda-forge

requests 2.28.2 pyhd8ed1ab_1 conda-forge

resampy 0.4.2 pyhd8ed1ab_0 conda-forge

samplerate 0.1.0 pypi_0 pypi

scikit-learn 1.2.2 py310hd266714_1 conda-forge

scipy 1.10.1 py310h578b7cb_0 conda-forge

setuptools 67.6.1 pyhd8ed1ab_0 conda-forge

sip 6.7.7 py310h00ffb61_0 conda-forge

six 1.16.0 pyh6c4a22f_0 conda-forge

soxr 0.1.3 hcfcfb64_3 conda-forge

soxr-python 0.3.3 py310h9b08ddd_0 conda-forge

svt-av1 1.4.1 h63175ca_0 conda-forge

tbb 2021.8.0 h91493d7_0 conda-forge

threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge

tk 8.6.12 h8ffe710_0 conda-forge

toml 0.10.2 pyhd8ed1ab_0 conda-forge

tomli 2.0.1 pyhd8ed1ab_0 conda-forge

tornado 6.2 py310h8d17308_1 conda-forge

typing_extensions 4.5.0 pyha770c72_0 conda-forge

tzdata 2023c h71feb2d_0 conda-forge

ucrt 10.0.22621.0 h57928b3_0 conda-forge

unicodedata2 15.0.0 py310h8d17308_0 conda-forge

urllib3 1.26.15 pyhd8ed1ab_0 conda-forge

vc 14.3 hb6edc58_10 conda-forge

vs2015_runtime 14.34.31931 h4c5c07a_10 conda-forge

wheel 0.40.0 pyhd8ed1ab_0 conda-forge

win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge

x264 1!164.3095 h8ffe710_2 conda-forge

x265 3.5 h2d74725_3 conda-forge

xorg-libxau 1.0.9 hcd874cb_0 conda-forge

xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge

xz 5.2.6 h8d14728_0 conda-forge

zipp 3.15.0 pyhd8ed1ab_0 conda-forge

zstd 1.5.2 h12be248_6 conda-forge

[Graham Coleman]

Thank you for the feedback, Brian.

After I couldn't reproduce the issue with basic streaming operations, I got the idea of disabling lazy loading with the EAGER_IMPORT variable from lazy_loader. This makes the error a bit more legible.

This is a branch using librosa.core.load that calls librosa.stft:

(cbFilter2) C:\Users\Graham\Desktop\projects\cbFilter>python -m cbFilter.test_basic
<class 'numpy.ndarray'> (701070,)

Traceback (most recent call last):

  File "C:\python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\python310\lib\runpy.py", line 86, in _run_code

    exec(code, run_globals)
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\test_basic.py", line 50, in <module>
    test_basic()

  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\test_basic.py", line 33, in test_basic
    srcDescList.append( Modesc(f, doStream=False) )

  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\modesc.py", line 184, in __init__
    cgram_c, mgram_c, powspec_c = mosex.extract_stft_desc(
  File "C:\Users\Graham\Desktop\projects\cbFilter\cbFilter\mosex.py", line 110, in extract_stft_desc
    stft = librosa.stft(data,n_fft=winsize,hop_length=hopsize,center=center)
  File "C:\Users\Graham\Envs\cbFilter2\lib\site-packages\librosa\core\spectrum.py", line 229, in stft
    util.valid_audio(y, mono=False)
  File "C:\Users\Graham\Envs\cbFilter2\lib\site-packages\librosa\util\utils.py", line 296, in valid_audio
    if not isinstance(y, np.ndarray):
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

I checked the dimension and type of the input to stft.

Here, I don't understand the type error. Here is the context from librosa\util\utils.py:

    if not isinstance(y, np.ndarray):

        raise ParameterError("Audio data must be of type numpy.ndarray")

But the type error complains about np.ndarray not being a type. However, when I run the following in ipython:

In [1]: import numpy as np

In [2]: isinstance(np.zeros(3), np.ndarray)
Out[2]: True

In [3]: isinstance([0,2], np.ndarray)
Out[3]: False

So I don't see how the "import numpy as np" within utils.py, the np.ndarray is not accepted by isinstance as a type.

utils.valid_audio, however, works as expected. Obviously, I could be giving some invalid value somewhere, but I haven't found it yet.

The error message is a bit befuddling, but I will keep trying. I keep trying to build a minimal example, but my script that just loads the audio and calls stft with the same parameters returns.

Graham

--
You received this message because you are subscribed to a topic in the Google Groups "librosa" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/librosa/MnM1UbSUtT8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to librosa+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/librosa/6638c083-3b48-46e4-8bf6-538f19798bc7n%40googlegroups.com.

[Graham Coleman]

Finally, I arrived at a conclusion. And so, I will share the silly cause of my woe.

I added a print statement to utils.py, just before the isinstance error:

#gkc: what are np and np.ndarray?

    print( np, np.ndarray )

<module 'numpy' from 'C:\\Users\\Graham\\Envs\\cbFilter2\\lib\\site-packages\\numpy\\__init__.py'> None

So, numpy is what it should be, but np.ndarray has somehow been set to None. Here, I get a flash of recognition and search my code for assignments to ndarray.

@dataclass

class MeasureConfig:

    """Data for a single configuration in a measure plot."""

    configName: string

    samples: np.ndarray

    weights: np.ndarray = None

    doKDE: bool = True

Surely that type annotation can't be assigning np.ndarray? Then, I found the offending typo:

@dataclass

class ConfigResult:

    configName: string

    logMeas: MeasureSet

    logMeasSim: MeasureSet

    methodTime: float = None

    didResamp: np.ndarray = None

    wtHistory: np.ndarray = None

    diffFilt: np.ndarray = None

    diffSim: np.ndarray = None

    xHistory: np.ndarray = None

    xSim = np.ndarray = None

In conclusion, my user code reset np.ndarray and broke everything. After fixing this, my scripts run again. Let this be a lesson!

best regards

Graham