Segmentation Faults debugging

20 views

Skip to first unread message

Molly Foley

unread,

Oct 15, 2024, 3:54:07 PM10/15/24

to modwsgi

I'm struggling to get multiple conda environments working and I'm hoping to get some insight here.

I have installed python with the miniforge3 distribution (conda, basically). In the base environment, I have installed mod_wsgi 5.0.0 which appears to compile again python 3.12. The base python environment has python 3.12.6 installed (I left it as default because messing with the base environment is usually a no-no). Additionally, I have created two conda virtual environments, one for each virtual host I have in Apache. Each virtual environment has python 3.12.7 installed.

sys.prefix is /opt/miniforge3

As root user, echo $PATH results in:

/root/.nvm/versions/node/v20.5.0/bin:/root/.local/bin:/root/bin:/opt/miniforge3/bin/:/sbin:/bin:/usr/sbin:/usr/bin:/var/cfengine/bin

LoadModule config looks like this:

# NOTE: mod_wsgi_python3 can not coexist in the same apache process as

# mod_wsgi (python2). Only load if mod_wsgi is not already loaded.

LoadModule wsgi_module /opt/miniforge3/lib/python3.12/site-packages/mod_wsgi/server/mod_wsgi-py312.cpython-312-x86_64-linux-gnu.so

</IfModule>

In Apache, I have set the apache LogLevel to info. I have also set the virtual hosts' specific logs to LogLevel to info.

Here is the relevant config for the virtual hosts:

ServerName myServerName.com

## Vhost docroot

DocumentRoot "/var/www/vhosts/hydrosource_chap"

## Logging

ErrorLog "/var/log/httpd/chap-error.log"

LogLevel info

ServerSignature Off

CustomLog "/var/log/httpd/chap-access.log" combined

## WSGI configuration

WSGIApplicationGroup %{GLOBAL}

WSGIDaemonProcess hydrosource_chap display-name=%{GROUP} home=/var/www/vhosts/hydrosource_chap python-home=/opt/miniforge3/envs/chap threads=1 user=hydro

WSGIProcessGroup hydrosource_chap

WSGIScriptAlias / "/var/www/vhosts/hydrosource_chap/wsgi.py"

## Anything start with dot

Require all denied

</DirectoryMatch>

Require all denied

</LocationMatch>

Require all granted

</Location>

</VirtualHost>

ServerName mySecondName.com

## Vhost docroot

DocumentRoot "/var/www/vhosts/hydrosource-2"

## Logging

ErrorLog "/var/log/httpd/dataexplorer-error.log"

LogLevel info

ServerSignature Off

CustomLog "/var/log/httpd/dataexplorer-access.log" combined

## WSGI configuration

WSGIApplicationGroup %{GLOBAL}

WSGIDaemonProcess dataexplorer display-name=%{GROUP} home=/var/www/vhosts/hydrosource-2 python-home=/opt/miniforge3/envs/dataexplorer threads=1 user=hydro

WSGIProcessGroup dataexplorer

WSGIScriptAlias / "/var/www/vhosts/hydrosource-2/wsgi.py"

## Anything start with dot

Require all denied

</DirectoryMatch>

Require all denied

</LocationMatch>

Require all granted

</Location>

</VirtualHost>

When I systemctl restart httpd and try to load the dataexplorer application, for instance, I get segmentation fault errors in the apache error_log and issues with oversized headers in the virtual host specific log.

Apache Log:

[Tue Oct 15 15:10:34.242996 2024] [wsgi:info] [pid 568871:tid 568871] mod_wsgi (pid=568871): Starting process 'hydrosource_chap' with uid=36143, gid=48 and threads=1.

[Tue Oct 15 15:10:34.243772 2024] [wsgi:info] [pid 568872:tid 568872] mod_wsgi (pid=568872): Starting process 'dataexplorer' with uid=36143, gid=48 and threads=1.

[Tue Oct 15 15:10:34.246679 2024] [mpm_event:notice] [pid 568866:tid 568866] AH00489: Apache/2.4.57 (Red Hat Enterprise Linux) OpenSSL/3.0.7 mod_wsgi/5.0.0 Python/3.12 configured -- resuming normal operations

[Tue Oct 15 15:10:34.246702 2024] [mpm_event:info] [pid 568866:tid 568866] AH00490: Server built: Aug 5 2024 00:00:00

[Tue Oct 15 15:10:34.246719 2024] [core:notice] [pid 568866:tid 568866] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

[Tue Oct 15 15:10:34.248317 2024] [wsgi:info] [pid 568871:tid 568871] mod_wsgi (pid=568871): Python home /opt/miniforge3/envs/chap.

[Tue Oct 15 15:10:34.248321 2024] [wsgi:info] [pid 568872:tid 568872] mod_wsgi (pid=568872): Python home /opt/miniforge3/envs/dataexplorer.

[Tue Oct 15 15:10:34.248377 2024] [wsgi:info] [pid 568871:tid 568871] mod_wsgi (pid=568871): Initializing Python.

[Tue Oct 15 15:10:34.248387 2024] [wsgi:info] [pid 568872:tid 568872] mod_wsgi (pid=568872): Initializing Python.

[Tue Oct 15 15:10:34.249092 2024] [http2:info] [pid 568874:tid 568874] h2_workers: created with min=25 max=37 idle_ms=600000

[Tue Oct 15 15:10:34.250137 2024] [wsgi:info] [pid 568874:tid 568874] mod_wsgi (pid=568874): Initializing Python.

[Tue Oct 15 15:10:34.256663 2024] [http2:info] [pid 568875:tid 568875] h2_workers: created with min=25 max=37 idle_ms=600000

[Tue Oct 15 15:10:34.257760 2024] [wsgi:info] [pid 568875:tid 568875] mod_wsgi (pid=568875): Initializing Python.

[Tue Oct 15 15:10:34.258652 2024] [http2:info] [pid 568873:tid 568873] h2_workers: created with min=25 max=37 idle_ms=600000

[Tue Oct 15 15:10:34.259732 2024] [wsgi:info] [pid 568873:tid 568873] mod_wsgi (pid=568873): Initializing Python.

[Tue Oct 15 15:10:34.262413 2024] [wsgi:info] [pid 568874:tid 568874] mod_wsgi (pid=568874): Attach interpreter ''.

[Tue Oct 15 15:10:34.271322 2024] [wsgi:info] [pid 568873:tid 568873] mod_wsgi (pid=568873): Attach interpreter ''.

[Tue Oct 15 15:10:34.277995 2024] [wsgi:info] [pid 568875:tid 568875] mod_wsgi (pid=568875): Attach interpreter ''.

[Tue Oct 15 15:10:34.279542 2024] [wsgi:info] [pid 568874:tid 568874] mod_wsgi (pid=568874): Imported 'mod_wsgi'.

[Tue Oct 15 15:10:34.288265 2024] [wsgi:info] [pid 568873:tid 568873] mod_wsgi (pid=568873): Imported 'mod_wsgi'.

[Tue Oct 15 15:10:34.296894 2024] [wsgi:info] [pid 568875:tid 568875] mod_wsgi (pid=568875): Imported 'mod_wsgi'.

[Tue Oct 15 15:10:49.266694 2024] [http2:info] [pid 569108:tid 569108] h2_workers: created with min=25 max=37 idle_ms=600000

[Tue Oct 15 15:10:49.267916 2024] [wsgi:info] [pid 569108:tid 569108] mod_wsgi (pid=569108): Initializing Python.

[Tue Oct 15 15:10:49.281851 2024] [wsgi:info] [pid 569108:tid 569108] mod_wsgi (pid=569108): Attach interpreter ''.

[Tue Oct 15 15:10:49.298865 2024] [wsgi:info] [pid 569108:tid 569108] mod_wsgi (pid=569108): Imported 'mod_wsgi'.

[Tue Oct 15 15:10:50.264356 2024] [core:notice] [pid 568866:tid 568866] AH00051: child pid 568872 exit signal Segmentation fault (11), possible coredump in /etc/httpd

[Tue Oct 15 15:10:50.264458 2024] [wsgi:info] [pid 568866:tid 568866] mod_wsgi (pid=568872): Process 'dataexplorer' has died, deregister and restart it.

[Tue Oct 15 15:10:50.264466 2024] [wsgi:info] [pid 568866:tid 568866] mod_wsgi (pid=568872): Process 'dataexplorer' terminated by signal 11

[Tue Oct 15 15:10:50.264473 2024] [wsgi:info] [pid 568866:tid 568866] mod_wsgi (pid=568872): Process 'dataexplorer' has been deregistered and will no longer be monitored.

[Tue Oct 15 15:10:50.265514 2024] [wsgi:info] [pid 569173:tid 569173] mod_wsgi (pid=569173): Starting process 'dataexplorer' with uid=36143, gid=48 and threads=1.

[Tue Oct 15 15:10:50.268450 2024] [wsgi:info] [pid 569173:tid 569173] mod_wsgi (pid=569173): Python home /opt/miniforge3/envs/dataexplorer.

[Tue Oct 15 15:10:50.268513 2024] [wsgi:info] [pid 569173:tid 569173] mod_wsgi (pid=569173): Initializing Python.

[Tue Oct 15 15:10:51.265981 2024] [core:notice] [pid 568866:tid 568866] AH00051: child pid 569173 exit signal Segmentation fault (11), possible coredump in /etc/httpd

[Tue Oct 15 15:10:51.266037 2024] [wsgi:info] [pid 568866:tid 568866] mod_wsgi (pid=569173): Process 'dataexplorer' has died, deregister and restart it.

[Tue Oct 15 15:10:51.266044 2024] [wsgi:info] [pid 568866:tid 568866] mod_wsgi (pid=569173): Process 'dataexplorer' terminated by signal 11

[Tue Oct 15 15:10:51.266050 2024] [wsgi:info] [pid 568866:tid 568866] mod_wsgi (pid=569173): Process 'dataexplorer' has been deregistered and will no longer be monitored.

[Tue Oct 15 15:10:51.267079 2024] [wsgi:info] [pid 569188:tid 569188] mod_wsgi (pid=569188): Starting process 'dataexplorer' with uid=36143, gid=48 and threads=1.

[Tue Oct 15 15:10:51.269694 2024] [wsgi:info] [pid 569188:tid 569188] mod_wsgi (pid=569188): Python home /opt/miniforge3/envs/dataexplorer.

[Tue Oct 15 15:10:51.269761 2024] [wsgi:info] [pid 569188:tid 569188] mod_wsgi (pid=569188): Initializing Python.

dataexplorer log:

[Tue Oct 15 15:10:34.267712 2024] [wsgi:info] [pid 568872:tid 568872] mod_wsgi (pid=568872): Attach interpreter ''.

[Tue Oct 15 15:10:48.949190 2024] [wsgi:info] [pid 568872:tid 568981] [remote 10.159.64.61:59510] mod_wsgi (pid=568872, process='dataexplorer', application=''): Loading Python script file '/var/www/vhosts/hydrosource-2/wsgi.py'.

[Tue Oct 15 15:10:49.636199 2024] [wsgi:error] [pid 568875:tid 569031] [client 10.159.64.61:59510] Truncated or oversized response headers received from daemon process 'dataexplorer': /var/www/vhosts/hydrosource-2/wsgi.py

[Tue Oct 15 15:10:50.282619 2024] [wsgi:info] [pid 569173:tid 569173] mod_wsgi (pid=569173): Attach interpreter ''.

[Tue Oct 15 15:10:50.293954 2024] [wsgi:info] [pid 569173:tid 569176] [remote 10.159.64.61:59511] mod_wsgi (pid=569173, process='dataexplorer', application=''): Loading Python script file '/var/www/vhosts/hydrosource-2/wsgi.py'.

[Tue Oct 15 15:10:50.956465 2024] [wsgi:error] [pid 568874:tid 568980] [client 10.159.64.61:59511] Truncated or oversized response headers received from daemon process 'dataexplorer': /var/www/vhosts/hydrosource-2/wsgi.py, referer: https://dataexplorer.hydroclimate-dev.ornl.gov/

[Tue Oct 15 15:10:51.281032 2024] [wsgi:info] [pid 569188:tid 569188] mod_wsgi (pid=569188): Attach interpreter ''.

Given the segmentation faults, I do a core dump and then use gdb to look at the problem. To be honest, I don't really know what I'm looking at here. If I do "bt", it prints out a lot more stuff, but I can't make heads or tails of it. This is the output without backtracing with "bt".

GNU gdb (GDB) Red Hat Enterprise Linux 10.2-13.el9

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<https://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/sbin/httpd...

Reading symbols from .gnu_debugdata for /usr/sbin/httpd...

(No debugging symbols found in .gnu_debugdata for /usr/sbin/httpd)

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing

[New LWP 568981]

[New LWP 568872]

[New LWP 568979]

[New LWP 568977]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `(wsgi:dataexplo -DFOREGROUND'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 PyObject_GetAttr (v=0x7fa2e8a44130, name=0x7fa2eae33c20 <_PyRuntime+45280>) at /usr/local/src/conda/python-3.12.6/Include/object.h:968

968 /usr/local/src/conda/python-3.12.6/Include/object.h: No such file or directory.

[Current thread is 1 (Thread 0x7fa2e9425640 (LWP 568981))]

Missing separate debuginfos, use: dnf debuginfo-install httpd-core-2.4.57-11.el9_4.1.x86_64

It seems like it's routing to an incorrect python install somehow. /usr/local/src/conda is not a path that exists on this machine, so the error makes sense, but I don't know what it's trying to do on this line.

I have attempted to set header-buffer-size on my WSGIProcessDaemon directives but haven't had any luck with that. If I remove the offending packages (things that have to do with SQL like flask-sqlalchemy, psycopg2, geoalchemy2, etc) then the apps will run without crashing, but will obviously give me errors about being unable to import the packages I removed since they're needed in my app. I'm really not sure how to move forward on this. Is there something obviously wrong in my configuration? Do I need to install mod_wsgi in each virtual environment instead of the base environment (and if so, then how would LoadModule work in Apache?).

Lastly, here's my wsgi.py script file (each app has the same one). I am using blueprints in my Flask app, so I import the create_app function to generate the application.

import sys
from api import create_app
from os import getcwd

sys.path.insert(0, getcwd() + "/api")
sys.path.append(getcwd() + "/api/models")

application = create_app()

Any insight on how to debug this would be very helpful. I have reviewed the debugging technique docs, but I really feel lost with the gnu debugger. 

Thanks for your time!

Graham Dumpleton

unread,

Oct 15, 2024, 4:22:40 PM10/15/24

to mod...@googlegroups.com

Do you have PHP enabled for the same Apache?

Generally I discourage use of Anaconda Python distributions as they do things which break embedding for Python.

One of the problems is that Anaconda Python ships its own versions of some libraries which are usually installed system wide. The problem is that PHP loads the system wide variants and when the Anaconda versions are incompatible, the process trying to use the Anaconda version will crash.

So if you have PHP enabled, are you able to disable it so it isn't loaded by Apache?

Not that it will help, but since you are using mod_wsgi daemon mode, also add outside of VirtualHost definitions (good spot is after the LoadModule), the directive:

WSGIRestrictEmbedded On

That will mean Python isn't initialised in Apache child worker processes where it isn't needed.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/b6af7e32-21b7-4f04-9e5b-ab1e8cfaa7dan%40googlegroups.com.

Molly Foley

unread,

Oct 16, 2024, 3:37:41 PM10/16/24

to modwsgi

Hi Graham, thanks for the reply. We do not have any PHP modules enabled on our Apache server, so I don't think that's it.

That's interesting to hear about the Anaconda distribution and I can see how that would be true for the "official" distribution, but is that also true for community driven distributions like miniforge3? They only use the conda-forge channel for installing packages in all environments, including the base environment, unless you specifically request it install from some other channel (we've never needed to). If miniforge3 doesn't work, I might try micromamba instead and see if that makes a difference somehow. We don't like using venv because many scientific packages are an absolute pain to install using pip, such as gdal.

Graham Dumpleton

unread,

Oct 16, 2024, 4:16:56 PM10/16/24

to mod...@googlegroups.com

Can you try and get a full backtrace for all threads using "thread apply all bt" as described in:

* https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb

Really need the context with all stack frames so can narrow down which package the problem occurs in.

Graham

On 17 Oct 2024, at 6:37 AM, Molly Foley <molly....@gmail.com> wrote:

Hi Graham, thanks for the reply. We do not have any PHP modules enabled on our Apache server, so I don't think that's it.

That's interesting to hear about the Anaconda distribution and I can see how that would be true for the "official" distribution, but is that also true for community driven distributions like miniforge3? They only use the conda-forge channel for installing packages in all environments, including the base environment, unless you specifically request it install from some other channel (we've never needed to). If miniforge3 doesn't work, I might try micromambainstead and see if that makes a difference somehow. We don't like using venv because many scientific packages are an absolute pain to install using pip, such as gdal.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a96269ef-2e4a-4c4c-9a1f-877cf9e65e04n%40googlegroups.com.

Molly Foley

unread,

Oct 16, 2024, 4:46:21 PM10/16/24

to modwsgi

Sure thing. This is what I'm doing to get the output, hopefully I'm doing it right:

1. Get a PID of the httpd process of the site I'm trying to run

2. gdb /usr/bin/httpd <PID>

3. I navigate to the site to try and get the 500 error, but instead it just hangs like it's loading, I guess because of the debugger attachment I have going?

4. While it's hanging I run thread apply all bt

This is what I get:

Thread 4 (Thread 0x7fae28225640 (LWP 577783) "httpd"):

#0 0x00007fae2ad0e21e in epoll_wait () from target:/lib64/libc.so.6

#1 0x00007fae2af56310 in impl_pollset_poll.lto_priv () from target:/lib64/libapr-1.so.0

#2 0x00007fae29e27322 in wsgi_daemon_worker (thread=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:9019

#3 wsgi_daemon_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:9188

#4 0x00007fae2ac89c02 in start_thread () from target:/lib64/libc.so.6

#5 0x00007fae2ad0ec40 in clone3 () from target:/lib64/libc.so.6

Thread 3 (Thread 0x7fae28a26640 (LWP 577782) "httpd"):

#0 0x00007fae2ad0422d in select () from target:/lib64/libc.so.6

#1 0x00007fae2af58849 in apr_sleep () from target:/lib64/libapr-1.so.0

#2 0x00007fae29e179d3 in wsgi_deadlock_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:9228

#3 0x00007fae2ac89c02 in start_thread () from target:/lib64/libc.so.6

#4 0x00007fae2ad0ec40 in clone3 () from target:/lib64/libc.so.6

Thread 2 (Thread 0x7fae29227640 (LWP 577781) "httpd"):

#0 0x00007fae2ad0422d in select () from target:/lib64/libc.so.6

#1 0x00007fae2af58849 in apr_sleep () from target:/lib64/libapr-1.so.0

#2 0x00007fae29e155cd in wsgi_monitor_thread (thd=<optimized out>, data=0x5579f103aad8) at src/server/mod_wsgi.c:9510

#3 0x00007fae2ac89c02 in start_thread () from target:/lib64/libc.so.6

#4 0x00007fae2ad0ec40 in clone3 () from target:/lib64/libc.so.6

Thread 1 (Thread 0x7fae2ae82540 (LWP 577780) "httpd"):

#0 0x00007fae2ad019ff in poll () from target:/lib64/libc.so.6

#1 0x00007fae2af50de1 in apr_poll () from target:/lib64/libapr-1.so.0

#2 0x00007fae29e2a821 in wsgi_daemon_main (daemon=0x5579f103aad8, p=0x5579f0e41c28) at src/server/mod_wsgi.c:9811

#3 wsgi_start_process (p=0x5579f0e41c28, daemon=daemon@entry=0x5579f103aad8) at src/server/mod_wsgi.c:10542

#4 0x00007fae29e2cc8c in wsgi_manage_process (reason=<optimized out>, data=0x5579f103aad8, status=<optimized out>) at src/server/mod_wsgi.c:8275

#5 0x00007fae2af4e928 in apr_proc_other_child_alert () from target:/lib64/libapr-1.so.0

#6 0x00007fae2a84d86d in event_run () from target:/etc/httpd/modules/mod_mpm_event.so

#7 0x00005579ef3686d8 in ap_run_mpm ()

#8 0x00005579ef35664f in main ()

I don't know if this helps at all, but if I get a core dump of the process after I receive the 500 error and then run coredumpctl debug <pid> it'll throw it into gdb where I then can run bt. This is the output I get from that:

(gdb) bt

#0 PyObject_GetAttr (v=0x7fae27844130, name=0x7fae29c33c20 <_PyRuntime+45280>) at /usr/local/src/conda/python-3.12.6/Include/object.h:968

#1 0x00007fae2980dc4b in _PyObject_GetMethod (obj=0x7fae27844130, name=0x7fae29c33c20 <_PyRuntime+45280>, method=0x7fae2821ae60) at /usr/local/src/conda/python-3.12.6/Objects/object.c:1307

#2 0x00007fae2980d8e6 in PyObject_VectorcallMethod (name=0x7fae29c33c20 <_PyRuntime+45280>, args=args@entry=0x7fae2821af90, nargsf=nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0)

at /usr/local/src/conda/python-3.12.6/Objects/call.c:870

#3 0x00007fae277c830e in PyObject_CallMethodOneArg (arg=0x7fae26aa1740, name=<optimized out>, self=<optimized out>) at /usr/local/src/conda/python-3.12.7/Include/cpython/abstract.h:103

#4 datetime_fromtimestamp (cls=<optimized out>, args=<optimized out>, kw=<optimized out>) at /usr/local/src/conda/python-3.12.7/Modules/_datetimemodule.c:5181

#5 0x00007fae29828092 in cfunction_call (func=0x7fae2662ac50, args=0x7fae26633d40, kwargs=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:537

#6 0x00007fae29808f83 in _PyObject_MakeTpCall (tstate=0x7fae20004cd0, callable=0x7fae2662ac50, args=0x7fae2a633558, nargs=<optimized out>, keywords=0x0)

at /usr/local/src/conda/python-3.12.6/Objects/call.c:240

#7 0x00007fae297119b5 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a6334f8, throwflag=<optimized out>) at Python/bytecodes.c:2715

#8 0x00007fae298c2f79 in builtin___build_class__ (self=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:200

#9 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae2a633418, nargsf=<optimized out>, kwnames=0x0)

at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#10 0x00007fae2981d920 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775813, args=0x7fae2a633418, callable=0x7fae294082c0, tstate=0x7fae20004cd0)

at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#11 PyObject_Vectorcall (callable=0x7fae294082c0, args=0x7fae2a633418, nargsf=9223372036854775813, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:325

#12 0x00007fae297119b5 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a6333c0, throwflag=<optimized out>) at Python/bytecodes.c:2715

#13 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae2067c970, globals=<optimized out>, locals=0x7fae26594f40) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#14 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26594f40, globals=0x7fae26594f40, source=0x7fae2067c970)

at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#15 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#16 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae265bced8, nargsf=<optimized out>, kwnames=0x0)

at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#17 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae2656a100, args=0x7fae265bcec0, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a633338, throwflag=<optimized out>) at Python/bytecodes.c:3263

#19 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821b850, callable=0x7fae29410040, tstate=0x7fae20004cd0)

at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#20 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821b8e0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#21 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#22 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae266ecc70, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#23 PyImport_ImportModuleLevelObject (name=0x7fae26da9770, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae265a1fc0, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#24 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae265a1fc0, name=0x7fae26da9770, frame=0x7fae2a633018, tstate=<optimized out>)

at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#25 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a633018, throwflag=<optimized out>) at Python/bytecodes.c:2144

#26 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae2066a320, globals=<optimized out>, locals=0x7fae2657bc80) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#27 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae2657bc80, globals=0x7fae2657bc80, source=0x7fae2066a320)

at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#28 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#29 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae265bc9d8, nargsf=<optimized out>, kwnames=0x0)

at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#30 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26594440, args=0x7fae265bc9c0, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a748f30, throwflag=<optimized out>) at Python/bytecodes.c:3263

#32 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821bf80, callable=0x7fae29410040, tstate=0x7fae20004cd0)

at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#33 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821c010) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#34 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#35 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae266cb1e0, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#36 PyImport_ImportModuleLevelObject (name=0x7fae26da9730, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae267046d0, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#37 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae267046d0, name=0x7fae26da9730, frame=0x7fae2a748c28, tstate=<optimized out>)

at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a748c28, throwflag=<optimized out>) at Python/bytecodes.c:2144

#39 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae206b74e0, globals=<optimized out>, locals=0x7fae266c2980) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#40 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae266c2980, globals=0x7fae266c2980, source=0x7fae206b74e0)

at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#41 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#42 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae266c3518, nargsf=<optimized out>, kwnames=0x0)

at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

--Type <RET> for more, q to quit, c to continue without paging--c

#43 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae266c0c40, args=0x7fae266c3500, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#44 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a748ba0, throwflag=<optimized out>) at Python/bytecodes.c:3263

#45 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821c6b0, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#46 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821c740) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#47 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#48 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae2679dc70, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#49 PyImport_ImportModuleLevelObject (name=0x7fae26da4690, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae2669f460, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#50 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae2669f460, name=0x7fae26da4690, frame=0x7fae2a7488b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#51 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7488b0, throwflag=<optimized out>) at Python/bytecodes.c:2144

#52 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae206017c0, globals=<optimized out>, locals=0x7fae266a9480) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#53 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae266a9480, globals=0x7fae266a9480, source=0x7fae206017c0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#54 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#55 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae266aab58, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#56 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae266a9500, args=0x7fae266aab40, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#57 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a748828, throwflag=<optimized out>) at Python/bytecodes.c:3263

#58 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821cde0, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#59 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821ce70) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#60 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#61 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae2669b6b0, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#62 PyImport_ImportModuleLevelObject (name=0x7fae2669b6b0, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#63 0x00007fae298646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7fae2669b6b0, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#64 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#65 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae2669ce68, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#66 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae266a8040, args=0x7fae2669ce50, callable=0x7fae29408310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#67 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a748528, throwflag=<optimized out>) at Python/bytecodes.c:3263

#68 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7fae2821d2e0, callable=0x7fae29410180, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#69 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410180, vargs=0x7fae2821d370) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#70 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#71 0x00007fae298550af in PyImport_ImportModuleLevelObject (name=0x7fae29c2eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26aa17e0, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#72 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae26aa17e0, name=0x7fae29c2eb70 <_PyRuntime+24624>, frame=0x7fae2a7483b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#73 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7483b0, throwflag=<optimized out>) at Python/bytecodes.c:2144

#74 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae205e5b00, globals=<optimized out>, locals=0x7fae2667fa80) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#75 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae2667fa80, globals=0x7fae2667fa80, source=0x7fae205e5b00) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#76 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#77 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26699718, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#78 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae266995c0, args=0x7fae26699700, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#79 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a748328, throwflag=<optimized out>) at Python/bytecodes.c:3263

#80 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821da10, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#81 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821daa0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#82 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#83 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26699430, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#84 PyImport_ImportModuleLevelObject (name=0x7fae26699430, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#85 0x00007fae298646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7fae26699430, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#86 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#87 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26671108, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#88 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae2667fa40, args=0x7fae266710f0, callable=0x7fae29408310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#89 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a748028, throwflag=<optimized out>) at Python/bytecodes.c:3263

#90 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7fae2821df10, callable=0x7fae29410180, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#91 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410180, vargs=0x7fae2821dfa0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#92 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#93 0x00007fae298550af in PyImport_ImportModuleLevelObject (name=0x7fae29c2eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae2687f130, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#94 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae2687f130, name=0x7fae29c2eb70 <_PyRuntime+24624>, frame=0x7fae2a747ed8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#95 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a747ed8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#96 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae205f0ec0, globals=<optimized out>, locals=0x7fae26883bc0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#97 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26883bc0, globals=0x7fae26883bc0, source=0x7fae205f0ec0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#98 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#99 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26698b98, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#100 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26872c80, args=0x7fae26698b80, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#101 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a747e50, throwflag=<optimized out>) at Python/bytecodes.c:3263

#102 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821e640, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#103 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821e6d0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#104 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#105 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26871e30, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#106 PyImport_ImportModuleLevelObject (name=0x7fae26871e30, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#107 0x00007fae298646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7fae26871e30, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#108 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#109 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26817e38, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#110 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae2679fe40, args=0x7fae26817e20, callable=0x7fae29408310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#111 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a747b50, throwflag=<optimized out>) at Python/bytecodes.c:3263

#112 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7fae2821eb40, callable=0x7fae29410180, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#113 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410180, vargs=0x7fae2821ebd0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#114 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#115 0x00007fae298550af in PyImport_ImportModuleLevelObject (name=0x7fae29c2eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26b7f880, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#116 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae26b7f880, name=0x7fae29c2eb70 <_PyRuntime+24624>, frame=0x7fae2a747950, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#117 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a747950, throwflag=<optimized out>) at Python/bytecodes.c:2144

#118 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae205da130, globals=<optimized out>, locals=0x7fae267ac900) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#119 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae267ac900, globals=0x7fae267ac900, source=0x7fae205da130) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#120 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#121 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae2684dad8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#122 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26b981c0, args=0x7fae2684dac0, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#123 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7478c8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#124 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821f270, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#125 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821f300) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#126 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#127 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26b980f0, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#128 PyImport_ImportModuleLevelObject (name=0x7fae27278ef0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26b110f0, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#129 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae26b110f0, name=0x7fae27278ef0, frame=0x7fae2a7475e8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#130 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7475e8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#131 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae205641a0, globals=<optimized out>, locals=0x7fae26b53840) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#132 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26b53840, globals=0x7fae26b53840, source=0x7fae205641a0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#133 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#134 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26b19898, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#135 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26b2fb80, args=0x7fae26b19880, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#136 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a747560, throwflag=<optimized out>) at Python/bytecodes.c:3263

#137 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821f9a0, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#138 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821fa30) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#139 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#140 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26b1a4b0, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#141 PyImport_ImportModuleLevelObject (name=0x7fae26b1a4b0, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#142 0x00007fae298646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7fae26b1a4b0, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#143 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#144 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26b10b68, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#145 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26b19ec0, args=0x7fae26b10b50, callable=0x7fae29408310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#146 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a747260, throwflag=<optimized out>) at Python/bytecodes.c:3263

#147 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae2821fea0, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#148 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae2821ff30) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#149 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#150 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26b19cf0, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#151 PyImport_ImportModuleLevelObject (name=0x7fae26dd8ab0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26b10250, level=2) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#152 0x00007fae29716e98 in import_name (level=0x7fae29c29848 <_PyRuntime+3336>, fromlist=0x7fae26b10250, name=0x7fae26dd8ab0, frame=0x7fae2a747060, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#153 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a747060, throwflag=<optimized out>) at Python/bytecodes.c:2144

#154 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae205482a0, globals=<optimized out>, locals=0x7fae26ae1f80) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#155 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26ae1f80, globals=0x7fae26ae1f80, source=0x7fae205482a0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#156 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#157 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26b0b598, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#158 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26ae2080, args=0x7fae26b0b580, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#159 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a746fd8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#160 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae282205d0, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#161 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae28220660) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#162 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#163 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26adadd0, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#164 PyImport_ImportModuleLevelObject (name=0x7fae26e777b0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26da5f90, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#165 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae26da5f90, name=0x7fae26e777b0, frame=0x7fae2a746cd8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#166 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a746cd8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#167 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae20541e00, globals=<optimized out>, locals=0x7fae26ae2000) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#168 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26ae2000, globals=0x7fae26ae2000, source=0x7fae20541e00) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#169 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#170 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26af0558, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#171 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26ae20c0, args=0x7fae26af0540, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#172 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a746c50, throwflag=<optimized out>) at Python/bytecodes.c:3263

#173 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae28220d00, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#174 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae28220d90) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#175 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#176 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26ae3570, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#177 PyImport_ImportModuleLevelObject (name=0x7fae29c31fd8 <_PyRuntime+38040>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26da5600, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#178 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae26da5600, name=0x7fae29c31fd8 <_PyRuntime+38040>, frame=0x7fae2a746960, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#179 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a746960, throwflag=<optimized out>) at Python/bytecodes.c:2144

#180 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae205055c0, globals=<optimized out>, locals=0x7fae26da8d40) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#181 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26da8d40, globals=0x7fae26da8d40, source=0x7fae205055c0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#182 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#183 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26ae0ed8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#184 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26e2e880, args=0x7fae26ae0ec0, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#185 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7468d8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#186 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae28221430, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#187 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae282214c0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#188 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#189 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26dad890, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#190 PyImport_ImportModuleLevelObject (name=0x7fae26dad890, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#191 0x00007fae298646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7fae26dad890, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#192 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#193 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26da53d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#194 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26ae1940, args=0x7fae26da53c0, callable=0x7fae29408310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#195 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7465d8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#196 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7fae28221930, callable=0x7fae29410180, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#197 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410180, vargs=0x7fae282219c0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#198 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#199 0x00007fae298550af in PyImport_ImportModuleLevelObject (name=0x7fae29c2eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26da4760, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#200 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae26da4760, name=0x7fae29c2eb70 <_PyRuntime+24624>, frame=0x7fae2a7464b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#201 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7464b0, throwflag=<optimized out>) at Python/bytecodes.c:2144

#202 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae204f9460, globals=<optimized out>, locals=0x7fae26d9c940) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#203 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26d9c940, globals=0x7fae26d9c940, source=0x7fae204f9460) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#204 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#205 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26ee5398, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#206 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26d9ed80, args=0x7fae26ee5380, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#207 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a746428, throwflag=<optimized out>) at Python/bytecodes.c:3263

#208 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae28222060, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#209 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae282220f0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#210 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#211 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26d9ee30, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#212 PyImport_ImportModuleLevelObject (name=0x7fae26d73ab0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26d98a30, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#213 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae26d98a30, name=0x7fae26d73ab0, frame=0x7fae2a746148, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#214 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a746148, throwflag=<optimized out>) at Python/bytecodes.c:2144

#215 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae2043c1b0, globals=<optimized out>, locals=0x7fae26d9c880) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#216 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26d9c880, globals=0x7fae26d9c880, source=0x7fae2043c1b0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#217 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#218 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26ee5298, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#219 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26d823c0, args=0x7fae26ee5280, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#220 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7460c0, throwflag=<optimized out>) at Python/bytecodes.c:3263

#221 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae28222790, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#222 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae28222820) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#223 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#224 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26d82b30, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#225 PyImport_ImportModuleLevelObject (name=0x7fae26d82b30, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae29c201c0 <_Py_NoneStruct>, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#226 0x00007fae29716e98 in import_name (level=0x7fae29c29808 <_PyRuntime+3272>, fromlist=0x7fae29c201c0 <_Py_NoneStruct>, name=0x7fae26d82b30, frame=0x7fae2a745db8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#227 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745db8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#228 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae202b0be0, globals=<optimized out>, locals=0x7fae26d81d80) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#229 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26d81d80, globals=0x7fae26d81d80, source=0x7fae202b0be0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#230 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#231 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26ee5158, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#232 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26d81fc0, args=0x7fae26ee5140, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#233 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745d30, throwflag=<optimized out>) at Python/bytecodes.c:3263

#234 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae28222ec0, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#235 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae28222f50) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#236 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#237 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26d7dd90, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#238 PyImport_ImportModuleLevelObject (name=0x7fae29b605a0 <const_str_extension>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae272762f0, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#239 0x00007fae29716e98 in import_name (level=0x7fae29c29828 <_PyRuntime+3304>, fromlist=0x7fae272762f0, name=0x7fae29b605a0 <const_str_extension>, frame=0x7fae2a745a60, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#240 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745a60, throwflag=<optimized out>) at Python/bytecodes.c:2144

#241 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae26d51d30, globals=<optimized out>, locals=0x7fae26d81500) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#242 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26d81500, globals=0x7fae26d81500, source=0x7fae26d51d30) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#243 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#244 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26ee4958, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#245 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae26d81040, args=0x7fae26ee4940, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#246 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a7459d8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#247 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae282235f0, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#248 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae28223680) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#249 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#250 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae26d81530, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#251 PyImport_ImportModuleLevelObject (name=0x7fae26d81530, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae26d72f20, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#252 0x00007fae29716e98 in import_name (level=0x7fae29c29808 <_PyRuntime+3272>, fromlist=0x7fae26d72f20, name=0x7fae26d81530, frame=0x7fae2a745708, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#253 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745708, throwflag=<optimized out>) at Python/bytecodes.c:2144

#254 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae26f1bc90, globals=<optimized out>, locals=0x7fae26d5fa80) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#255 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae26d5fa80, globals=0x7fae26d5fa80, source=0x7fae26f1bc90) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#256 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#257 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae26ee4a98, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#258 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae2799f340, args=0x7fae26ee4a80, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#259 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745680, throwflag=<optimized out>) at Python/bytecodes.c:3263

#260 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae28223d20, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#261 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae28223db0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#262 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#263 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae2799e770, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#264 PyImport_ImportModuleLevelObject (name=0x7fae2799e770, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae292cf040, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#265 0x00007fae29716e98 in import_name (level=0x7fae29c29808 <_PyRuntime+3272>, fromlist=0x7fae292cf040, name=0x7fae2799e770, frame=0x7fae2a745398, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#266 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745398, throwflag=<optimized out>) at Python/bytecodes.c:2144

#267 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae27990030, globals=<optimized out>, locals=0x7fae292e5a00) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#268 0x00007fae298dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fae292e5a00, globals=0x7fae292e5a00, source=0x7fae27990030) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#269 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#270 0x00007fae2981da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7fae2799c958, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#271 0x00007fae297125d1 in PyCFunction_Call (kwargs=0x7fae292e7f80, args=0x7fae2799c940, callable=0x7fae29408770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#272 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745310, throwflag=<optimized out>) at Python/bytecodes.c:3263

#273 0x00007fae29827b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fae28224450, callable=0x7fae29410040, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#274 object_vacall (tstate=0x7fae20004cd0, base=<optimized out>, callable=0x7fae29410040, vargs=0x7fae282244e0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#275 0x00007fae29855bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#276 0x00007fae29854e45 in import_find_and_load (abs_name=0x7fae292cec10, tstate=0x7fae20004cd0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#277 PyImport_ImportModuleLevelObject (name=0x7fae292cec10, globals=<optimized out>, locals=<optimized out>, fromlist=0x7fae29268610, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#278 0x00007fae29716e98 in import_name (level=0x7fae29c29808 <_PyRuntime+3272>, fromlist=0x7fae29268610, name=0x7fae292cec10, frame=0x7fae2a745020, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#279 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fae2a745020, throwflag=<optimized out>) at Python/bytecodes.c:2144

#280 0x00007fae298c185f in PyEval_EvalCode (co=0x7fae20005b20, globals=<optimized out>, locals=0x7fae292db600) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#281 0x00007fae298c1715 in exec_code_in_module (tstate=0x7fae20004cd0, name=0x7fae29292730, module_dict=<optimized out>, code_object=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/import.c:1712

#282 0x00007fae297e119c in PyImport_ExecCodeModuleObject (name=0x7fae29292730, co=0x7fae20005b20, pathname=<optimized out>, cpathname=0x0) at /usr/local/src/conda/python-3.12.6/Python/import.c:1755

#283 0x00007fae297e124c in PyImport_ExecCodeModuleWithPathnames (name=<optimized out>, co=0x7fae20005b20, pathname=<optimized out>, cpathname=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/import.c:1672

#284 0x00007fae29e20511 in wsgi_load_source (pool=0x7fae20000b88, r=r@entry=0x7fae20000c00, name=name@entry=0x7fae20002468 "_mod_wsgi_4cfdb6942418cf3781bf1fcff69d2c41", exists=exists@entry=0, filename=filename@entry=0x7fae2000353d "/var/www/vhosts/hydrosource-2/wsgi.py", process_group=<optimized out>, application_group=0x7fae200035ea "", ignore_system_exit=0) at src/server/mod_wsgi.c:3745

#285 0x00007fae29e229bd in wsgi_execute_script (r=r@entry=0x7fae20000c00) at src/server/mod_wsgi.c:4174

#286 0x00007fae29e28afc in wsgi_hook_daemon_handler (c=<optimized out>) at src/server/mod_wsgi.c:13318

#287 wsgi_process_socket (daemon=<optimized out>, bucket_alloc=<optimized out>, sock=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:8770

#288 wsgi_daemon_worker (thread=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:9098

#289 wsgi_daemon_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:9188

#290 0x00007fae2ac89c02 in start_thread () from /lib64/libc.so.6

#291 0x00007fae2ad0ec40 in clone3 () from /lib64/libc.so.6

Graham Dumpleton

unread,

Oct 16, 2024, 5:55:53 PM10/16/24

to mod...@googlegroups.com

You seemed to have missed it with way I said, did you remember to resume the debugger, or did you generate stack trace as soon as you attached.

Anyway, the second full trace is really weird if that is where it crashed, as it was inside of date time code.

#0 PyObject_GetAttr (v=0x7fae27844130, name=0x7fae29c33c20 <_PyRuntime+45280>) at /usr/local/src/conda/python-3.12.6/Include/object.h:968

#1 0x00007fae2980dc4b in _PyObject_GetMethod (obj=0x7fae27844130, name=0x7fae29c33c20 <_PyRuntime+45280>, method=0x7fae2821ae60) at /usr/local/src/conda/python-3.12.6/Objects/object.c:1307

#2 0x00007fae2980d8e6 in PyObject_VectorcallMethod (name=0x7fae29c33c20 <_PyRuntime+45280>, args=args@entry=0x7fae2821af90, nargsf=nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0)

at /usr/local/src/conda/python-3.12.6/Objects/call.c:870

#3 0x00007fae277c830e in PyObject_CallMethodOneArg (arg=0x7fae26aa1740, name=<optimized out>, self=<optimized out>) at /usr/local/src/conda/python-3.12.7/Include/cpython/abstract.h:103

#4 datetime_fromtimestamp (cls=<optimized out>, args=<optimized out>, kw=<optimized out>) at /usr/local/src/conda/python-3.12.7/Modules/_datetimemodule.c:5181

Specifically was trying to work out timezone stuff.

if (self != NULL && tzinfo != Py_None) {

/* Convert UTC to tzinfo's zone. */

self = _PyObject_CallMethodId(tzinfo, &PyId_fromutc, "N", self);

}

That doesn't make sense if that is the case.

I don't know what miniforge distro is and how it differs from Anaconda Python variants or PSF variants.

Was mod_wsgi also built with the same miniforge Python version as the virtual environments?

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a4de3578-6d60-4374-ab3f-18a0122262ecn%40googlegroups.com.

Molly Foley

unread,

Oct 16, 2024, 9:00:42 PM10/16/24

to modwsgi

My bad, you're right. I missed the "cont" - sorry, this is the first time I've ever worked with gdb! Anyway, here's the output, but I do see the something about datetime under Thread #4, line #4 again.

(gdb) cont

Continuing.

Missing separate debuginfo for target:/lib64/librt.so.1

Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/1a/11e03063e9160803ab6a87cece2ad25346f20f.debug

Thread 4 "httpd" received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0x7ff5e8c25640 (LWP 1145405)]

PyObject_GetAttr (v=0x7ff5e8244130, name=0x7ff5ea633c20 <_PyRuntime+45280>) at /usr/local/src/conda/python-3.12.6/Include/object.h:968

968 /usr/local/src/conda/python-3.12.6/Include/object.h: No such file or directory.

(gdb) thread apply all bt

Thread 4 (Thread 0x7ff5e8c25640 (LWP 1145405) "httpd"):

#0 PyObject_GetAttr (v=0x7ff5e8244130, name=0x7ff5ea633c20 <_PyRuntime+45280>) at /usr/local/src/conda/python-3.12.6/Include/object.h:968

#1 0x00007ff5ea20dc4b in _PyObject_GetMethod (obj=0x7ff5e8244130, name=0x7ff5ea633c20 <_PyRuntime+45280>, method=0x7ff5e8c1ae60) at /usr/local/src/conda/python-3.12.6/Objects/object.c:1307

#2 0x00007ff5ea20d8e6 in PyObject_VectorcallMethod (name=0x7ff5ea633c20 <_PyRuntime+45280>, args=args@entry=0x7ff5e8c1af90, nargsf=nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:870

#3 0x00007ff5e81c830e in PyObject_CallMethodOneArg (arg=0x7ff5e3799740, name=<optimized out>, self=<optimized out>) at /usr/local/src/conda/python-3.12.7/Include/cpython/abstract.h:103

#4 datetime_fromtimestamp (cls=<optimized out>, args=<optimized out>, kw=<optimized out>) at /usr/local/src/conda/python-3.12.7/Modules/_datetimemodule.c:5181

#5 0x00007ff5ea228092 in cfunction_call (func=0x7ff5e3012cf0, args=0x7ff5e301fa40, kwargs=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:537

#6 0x00007ff5ea208f83 in _PyObject_MakeTpCall (tstate=0x7ff5dc008cf0, callable=0x7ff5e3012cf0, args=0x7ff5ea70a558, nargs=<optimized out>, keywords=0x0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:240

#7 0x00007ff5ea1119b5 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5ea70a4f8, throwflag=<optimized out>) at Python/bytecodes.c:2715

#8 0x00007ff5ea2c2f79 in builtin___build_class__ (self=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:200

#9 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5ea70a418, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#10 0x00007ff5ea21d920 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775813, args=0x7ff5ea70a418, callable=0x7ff5e9e082c0, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#11 PyObject_Vectorcall (callable=0x7ff5e9e082c0, args=0x7ff5ea70a418, nargsf=9223372036854775813, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:325

#12 0x00007ff5ea1119b5 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5ea70a3c0, throwflag=<optimized out>) at Python/bytecodes.c:2715

#13 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc680040, globals=<optimized out>, locals=0x7ff5e2f802c0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#14 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e2f802c0, globals=0x7ff5e2f802c0, source=0x7ff5dc680040) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#15 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#16 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e2fa9ad8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#17 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e2fa8ac0, args=0x7ff5e2fa9ac0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5ea70a338, throwflag=<optimized out>) at Python/bytecodes.c:3263

#19 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1b850, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#20 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1b8e0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#21 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#22 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3c11630, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#23 PyImport_ImportModuleLevelObject (name=0x7ff5e37954f0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e2f8e200, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#24 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e2f8e200, name=0x7ff5e37954f0, frame=0x7ff5ea70a018, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#25 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5ea70a018, throwflag=<optimized out>) at Python/bytecodes.c:2144

#26 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc66d9f0, globals=<optimized out>, locals=0x7ff5e30c4240) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#27 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e30c4240, globals=0x7ff5e30c4240, source=0x7ff5dc66d9f0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#28 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#29 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e2f838d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#30 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e2f7fe00, args=0x7ff5e2f838c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13cf30, throwflag=<optimized out>) at Python/bytecodes.c:3263

#32 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1bf80, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#33 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1c010) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#34 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#35 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e30b72d0, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#36 PyImport_ImportModuleLevelObject (name=0x7ff5e37954b0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e30eca90, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#37 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e30eca90, name=0x7ff5e37954b0, frame=0x7ff5eb13cc28, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13cc28, throwflag=<optimized out>) at Python/bytecodes.c:2144

#39 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc6baa30, globals=<optimized out>, locals=0x7ff5e30ae6c0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#40 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e30ae6c0, globals=0x7ff5e30ae6c0, source=0x7ff5dc6baa30) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#41 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

--Type <RET> for more, q to quit, c to continue without paging--c

#42 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e30af298, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#43 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e30ac980, args=0x7ff5e30af280, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#44 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13cba0, throwflag=<optimized out>) at Python/bytecodes.c:3263

#45 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1c6b0, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#46 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1c740) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#47 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#48 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3056df0, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#49 PyImport_ImportModuleLevelObject (name=0x7ff5e3790990, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e308b820, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#50 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e308b820, name=0x7ff5e3790990, frame=0x7ff5eb13c8b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#51 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13c8b0, throwflag=<optimized out>) at Python/bytecodes.c:2144

#52 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc604810, globals=<optimized out>, locals=0x7ff5e3095200) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#53 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e3095200, globals=0x7ff5e3095200, source=0x7ff5dc604810) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#54 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#55 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e30968d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#56 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e3095280, args=0x7ff5e30968c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#57 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13c828, throwflag=<optimized out>) at Python/bytecodes.c:3263

#58 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1cde0, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#59 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1ce70) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#60 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#61 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3087d70, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#62 PyImport_ImportModuleLevelObject (name=0x7ff5e3087d70, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#63 0x00007ff5ea2646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7ff5e3087d70, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#64 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#65 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e3089228, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#66 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e3087440, args=0x7ff5e3089210, callable=0x7ff5e9e08310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#67 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13c528, throwflag=<optimized out>) at Python/bytecodes.c:3263

#68 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7ff5e8c1d2e0, callable=0x7ff5e9e10180, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#69 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10180, vargs=0x7ff5e8c1d370) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#70 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#71 0x00007ff5ea2550af in PyImport_ImportModuleLevelObject (name=0x7ff5ea62eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e37e1cc0, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#72 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e37e1cc0, name=0x7ff5ea62eb70 <_PyRuntime+24624>, frame=0x7ff5eb13c3b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#73 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13c3b0, throwflag=<optimized out>) at Python/bytecodes.c:2144

#74 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc5e9620, globals=<optimized out>, locals=0x7ff5e306b800) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#75 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e306b800, globals=0x7ff5e306b800, source=0x7ff5dc5e9620) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#76 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#77 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e3085498, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#78 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e3085300, args=0x7ff5e3085480, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#79 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13c328, throwflag=<optimized out>) at Python/bytecodes.c:3263

#80 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1da10, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#81 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1daa0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#82 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#83 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3085170, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#84 PyImport_ImportModuleLevelObject (name=0x7ff5e3085170, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#85 0x00007ff5ea2646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7ff5e3085170, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#86 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#87 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e305d3d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#88 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e306b7c0, args=0x7ff5e305d3c0, callable=0x7ff5e9e08310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#89 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13c028, throwflag=<optimized out>) at Python/bytecodes.c:3263

#90 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7ff5e8c1df10, callable=0x7ff5e9e10180, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#91 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10180, vargs=0x7ff5e8c1dfa0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#92 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#93 0x00007ff5ea2550af in PyImport_ImportModuleLevelObject (name=0x7ff5ea62eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e326b400, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#94 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e326b400, name=0x7ff5ea62eb70 <_PyRuntime+24624>, frame=0x7ff5eb13bed8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#95 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13bed8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#96 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc5f4770, globals=<optimized out>, locals=0x7ff5e3054ac0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#97 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e3054ac0, globals=0x7ff5e3054ac0, source=0x7ff5dc5f4770) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#98 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#99 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e30848d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#100 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e326f640, args=0x7ff5e30848c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#101 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13be50, throwflag=<optimized out>) at Python/bytecodes.c:3263

#102 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1e640, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#103 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1e6d0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#104 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#105 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3246c70, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#106 PyImport_ImportModuleLevelObject (name=0x7ff5e3246c70, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#107 0x00007ff5ea2646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7ff5e3246c70, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#108 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#109 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e323c148, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#110 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e323a840, args=0x7ff5e323c130, callable=0x7ff5e9e08310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#111 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13bb50, throwflag=<optimized out>) at Python/bytecodes.c:3263

#112 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7ff5e8c1eb40, callable=0x7ff5e9e10180, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#113 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10180, vargs=0x7ff5e8c1ebd0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#114 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#115 0x00007ff5ea2550af in PyImport_ImportModuleLevelObject (name=0x7ff5ea62eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e3567b80, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#116 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e3567b80, name=0x7ff5ea62eb70 <_PyRuntime+24624>, frame=0x7ff5eb13b950, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#117 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13b950, throwflag=<optimized out>) at Python/bytecodes.c:2144

#118 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc5ddf60, globals=<optimized out>, locals=0x7ff5e353df80) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#119 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e353df80, globals=0x7ff5e353df80, source=0x7ff5dc5ddf60) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#120 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#121 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e32398d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#122 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e357be00, args=0x7ff5e32398c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#123 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13b8c8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#124 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1f270, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#125 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1f300) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#126 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#127 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3586230, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#128 PyImport_ImportModuleLevelObject (name=0x7ff5e3c70f70, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e34fd4b0, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#129 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e34fd4b0, name=0x7ff5e3c70f70, frame=0x7ff5eb13b5e8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#130 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13b5e8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#131 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc567ab0, globals=<optimized out>, locals=0x7ff5e353f2c0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#132 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e353f2c0, globals=0x7ff5e353f2c0, source=0x7ff5dc567ab0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#133 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#134 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e35053d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#135 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e351b780, args=0x7ff5e35053c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#136 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13b560, throwflag=<optimized out>) at Python/bytecodes.c:3263

#137 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1f9a0, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#138 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1fa30) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#139 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#140 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3504670, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#141 PyImport_ImportModuleLevelObject (name=0x7ff5e3504670, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#142 0x00007ff5ea2646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7ff5e3504670, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#143 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#144 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e34fcf28, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#145 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e3505a00, args=0x7ff5e34fcf10, callable=0x7ff5e9e08310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#146 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13b260, throwflag=<optimized out>) at Python/bytecodes.c:3263

#147 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c1fea0, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#148 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c1ff30) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#149 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#150 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3505970, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#151 PyImport_ImportModuleLevelObject (name=0x7ff5e37c4870, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e34fc580, level=2) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#152 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629848 <_PyRuntime+3336>, fromlist=0x7ff5e34fc580, name=0x7ff5e37c4870, frame=0x7ff5eb13b060, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#153 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13b060, throwflag=<optimized out>) at Python/bytecodes.c:2144

#154 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc54bd20, globals=<optimized out>, locals=0x7ff5e34cdbc0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#155 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e34cdbc0, globals=0x7ff5e34cdbc0, source=0x7ff5dc54bd20) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#156 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#157 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e34f70d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#158 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e34cdcc0, args=0x7ff5e34f70c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#159 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13afd8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#160 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c205d0, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#161 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c20660) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#162 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#163 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e34c6f10, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#164 PyImport_ImportModuleLevelObject (name=0x7ff5e37fbb30, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e3792290, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#165 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e3792290, name=0x7ff5e37fbb30, frame=0x7ff5eb13acd8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#166 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13acd8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#167 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc545880, globals=<optimized out>, locals=0x7ff5e34cdc40) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#168 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e34cdc40, globals=0x7ff5e34cdc40, source=0x7ff5dc545880) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#169 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#170 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e34dc098, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#171 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e34cdd00, args=0x7ff5e34dc080, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#172 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13ac50, throwflag=<optimized out>) at Python/bytecodes.c:3263

#173 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c20d00, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#174 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c20d90) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#175 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#176 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e34cf0b0, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#177 PyImport_ImportModuleLevelObject (name=0x7ff5ea631fd8 <_PyRuntime+38040>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e3791900, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#178 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e3791900, name=0x7ff5ea631fd8 <_PyRuntime+38040>, frame=0x7ff5eb13a960, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#179 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13a960, throwflag=<optimized out>) at Python/bytecodes.c:2144

#180 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc508b30, globals=<optimized out>, locals=0x7ff5e3794ac0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#181 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e3794ac0, globals=0x7ff5e3794ac0, source=0x7ff5dc508b30) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#182 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#183 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e34ccb18, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#184 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e381aa40, args=0x7ff5e34ccb00, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#185 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13a8d8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#186 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c21430, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#187 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c214c0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#188 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#189 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e379d890, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#190 PyImport_ImportModuleLevelObject (name=0x7ff5e379d890, globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#191 0x00007ff5ea2646e6 in builtin___import___impl (level=<optimized out>, fromlist=<optimized out>, locals=<optimized out>, globals=0x0, name=0x7ff5e379d890, module=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:275

#192 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:107

#193 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e37916d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#194 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e34cd6c0, args=0x7ff5e37916c0, callable=0x7ff5e9e08310) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#195 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13a5d8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#196 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7ff5e8c21930, callable=0x7ff5e9e10180, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#197 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10180, vargs=0x7ff5e8c219c0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#198 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#199 0x00007ff5ea2550af in PyImport_ImportModuleLevelObject (name=0x7ff5ea62eb70 <_PyRuntime+24624>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e3790a60, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2931

#200 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e3790a60, name=0x7ff5ea62eb70 <_PyRuntime+24624>, frame=0x7ff5eb13a4b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#201 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13a4b0, throwflag=<optimized out>) at Python/bytecodes.c:2144

#202 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc4fcaa0, globals=<optimized out>, locals=0x7ff5e37886c0) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#203 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e37886c0, globals=0x7ff5e37886c0, source=0x7ff5dc4fcaa0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#204 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#205 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e38d5418, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#206 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e378ab00, args=0x7ff5e38d5400, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#207 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13a428, throwflag=<optimized out>) at Python/bytecodes.c:3263

#208 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c22060, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#209 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c220f0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#210 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#211 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e378abb0, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#212 PyImport_ImportModuleLevelObject (name=0x7ff5e375fd80, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e3784d30, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#213 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e3784d30, name=0x7ff5e375fd80, frame=0x7ff5eb13a148, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#214 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13a148, throwflag=<optimized out>) at Python/bytecodes.c:2144

#215 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc43f3e0, globals=<optimized out>, locals=0x7ff5e3788600) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#216 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e3788600, globals=0x7ff5e3788600, source=0x7ff5dc43f3e0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#217 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#218 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e38d5318, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#219 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e3776140, args=0x7ff5e38d5300, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#220 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb13a0c0, throwflag=<optimized out>) at Python/bytecodes.c:3263

#221 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c22790, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#222 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c22820) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#223 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#224 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e37768b0, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#225 PyImport_ImportModuleLevelObject (name=0x7ff5e37768b0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5ea6201c0 <_Py_NoneStruct>, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#226 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629808 <_PyRuntime+3272>, fromlist=0x7ff5ea6201c0 <_Py_NoneStruct>, name=0x7ff5e37768b0, frame=0x7ff5eb139db8, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#227 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139db8, throwflag=<optimized out>) at Python/bytecodes.c:2144

#228 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc2b44f0, globals=<optimized out>, locals=0x7ff5e3775b00) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#229 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e3775b00, globals=0x7ff5e3775b00, source=0x7ff5dc2b44f0) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#230 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#231 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e38d51d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#232 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e3775d40, args=0x7ff5e38d51c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#233 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139d30, throwflag=<optimized out>) at Python/bytecodes.c:3263

#234 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c22ec0, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#235 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c22f50) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#236 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#237 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e3769d90, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#238 PyImport_ImportModuleLevelObject (name=0x7ff5ea5605a0 <const_str_extension>, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e3c6e350, level=1) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#239 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629828 <_PyRuntime+3304>, fromlist=0x7ff5e3c6e350, name=0x7ff5ea5605a0 <const_str_extension>, frame=0x7ff5eb139a60, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#240 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139a60, throwflag=<optimized out>) at Python/bytecodes.c:2144

#241 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5e373dd30, globals=<optimized out>, locals=0x7ff5e3775300) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#242 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e3775300, globals=0x7ff5e3775300, source=0x7ff5e373dd30) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#243 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#244 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e38d49d8, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#245 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e3774e00, args=0x7ff5e38d49c0, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#246 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb1399d8, throwflag=<optimized out>) at Python/bytecodes.c:3263

#247 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c235f0, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#248 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c23680) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#249 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#250 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e37755b0, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#251 PyImport_ImportModuleLevelObject (name=0x7ff5e37755b0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e375f1c0, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#252 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629808 <_PyRuntime+3272>, fromlist=0x7ff5e375f1c0, name=0x7ff5e37755b0, frame=0x7ff5eb139708, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#253 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139708, throwflag=<optimized out>) at Python/bytecodes.c:2144

#254 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5e390bc90, globals=<optimized out>, locals=0x7ff5e374b840) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#255 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e374b840, globals=0x7ff5e374b840, source=0x7ff5e390bc90) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#256 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#257 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e38d4b18, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#258 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e839f340, args=0x7ff5e38d4b00, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#259 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139680, throwflag=<optimized out>) at Python/bytecodes.c:3263

#260 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c23d20, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#261 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c23db0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#262 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#263 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e839e770, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#264 PyImport_ImportModuleLevelObject (name=0x7ff5e839e770, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e9ccf040, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#265 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629808 <_PyRuntime+3272>, fromlist=0x7ff5e9ccf040, name=0x7ff5e839e770, frame=0x7ff5eb139398, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#266 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139398, throwflag=<optimized out>) at Python/bytecodes.c:2144

#267 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5e8390030, globals=<optimized out>, locals=0x7ff5e9ce5a00) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#268 0x00007ff5ea2dd9b0 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ff5e9ce5a00, globals=0x7ff5e9ce5a00, source=0x7ff5e8390030) at /usr/local/src/conda/python-3.12.6/Python/bltinmodule.c:1096

#269 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/clinic/bltinmodule.c.h:586

#270 0x00007ff5ea21da66 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<optimized out>, args=0x7ff5e839c958, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.6/Objects/methodobject.c:438

#271 0x00007ff5ea1125d1 in PyCFunction_Call (kwargs=0x7ff5e9ce7f80, args=0x7ff5e839c940, callable=0x7ff5e9e08770) at /usr/local/src/conda/python-3.12.6/Objects/call.c:387

#272 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139310, throwflag=<optimized out>) at Python/bytecodes.c:3263

#273 0x00007ff5ea227b76 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ff5e8c24450, callable=0x7ff5e9e10040, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Include/internal/pycore_call.h:92

#274 object_vacall (tstate=0x7ff5dc008cf0, base=<optimized out>, callable=0x7ff5e9e10040, vargs=0x7ff5e8c244e0) at /usr/local/src/conda/python-3.12.6/Objects/call.c:850

#275 0x00007ff5ea255bb1 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at /usr/local/src/conda/python-3.12.6/Objects/call.c:911

#276 0x00007ff5ea254e45 in import_find_and_load (abs_name=0x7ff5e9ccec10, tstate=0x7ff5dc008cf0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2779

#277 PyImport_ImportModuleLevelObject (name=0x7ff5e9ccec10, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ff5e9c68610, level=0) at /usr/local/src/conda/python-3.12.6/Python/import.c:2862

#278 0x00007ff5ea116e98 in import_name (level=0x7ff5ea629808 <_PyRuntime+3272>, fromlist=0x7ff5e9c68610, name=0x7ff5e9ccec10, frame=0x7ff5eb139020, tstate=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:2482

#279 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ff5eb139020, throwflag=<optimized out>) at Python/bytecodes.c:2144

#280 0x00007ff5ea2c185f in PyEval_EvalCode (co=0x7ff5dc009b40, globals=<optimized out>, locals=0x7ff5e9cdb600) at /usr/local/src/conda/python-3.12.6/Python/ceval.c:578

#281 0x00007ff5ea2c1715 in exec_code_in_module (tstate=0x7ff5dc008cf0, name=0x7ff5e9c92730, module_dict=<optimized out>, code_object=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/import.c:1712

#282 0x00007ff5ea1e119c in PyImport_ExecCodeModuleObject (name=0x7ff5e9c92730, co=0x7ff5dc009b40, pathname=<optimized out>, cpathname=0x0) at /usr/local/src/conda/python-3.12.6/Python/import.c:1755

#283 0x00007ff5ea1e124c in PyImport_ExecCodeModuleWithPathnames (name=<optimized out>, co=0x7ff5dc009b40, pathname=<optimized out>, cpathname=<optimized out>) at /usr/local/src/conda/python-3.12.6/Python/import.c:1672

#284 0x00007ff5ea817511 in wsgi_load_source (pool=0x7ff5dc002b98, r=r@entry=0x7ff5dc002c10, name=name@entry=0x7ff5dc007658 "_mod_wsgi_4cfdb6942418cf3781bf1fcff69d2c41", exists=exists@entry=0, filename=filename@entry=0x7ff5dc005588 "/var/www/vhosts/hydrosource-2/wsgi.py", process_group=<optimized out>, application_group=0x7ff5dc00562a "", ignore_system_exit=0) at src/server/mod_wsgi.c:3745

#285 0x00007ff5ea8199bd in wsgi_execute_script (r=r@entry=0x7ff5dc002c10) at src/server/mod_wsgi.c:4174

#286 0x00007ff5ea81fafc in wsgi_hook_daemon_handler (c=<optimized out>) at src/server/mod_wsgi.c:13318

#287 wsgi_process_socket (daemon=<optimized out>, bucket_alloc=<optimized out>, sock=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:8770

#288 wsgi_daemon_worker (thread=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:9098

#289 wsgi_daemon_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:9188

#290 0x00007ff5eb689c02 in start_thread () from target:/lib64/libc.so.6

#291 0x00007ff5eb70ec40 in clone3 () from target:/lib64/libc.so.6

Thread 3 (Thread 0x7ff5e9426640 (LWP 1145404) "httpd"):

#0 0x00007ff5eb68679a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6

#1 0x00007ff5eb6892a4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6

#2 0x00007ff5ea1ef7b4 in PyCOND_TIMEDWAIT (us=<optimized out>, mut=<optimized out>, cond=0x7ff5ea63ba08 <_PyRuntime+77512>) at /usr/local/src/conda/python-3.12.6/Python/condvar.h:73

#3 take_gil (tstate=0x7ff5e40013d0) at /usr/local/src/conda/python-3.12.6/Python/ceval_gil.c:376

#4 0x00007ff5ea210862 in PyEval_RestoreThread (tstate=0x7ff5e40013d0) at /usr/local/src/conda/python-3.12.6/Python/ceval_gil.c:708

#5 0x00007ff5ea2f5c13 in PyGILState_Ensure () at /usr/local/src/conda/python-3.12.6/Python/pystate.c:2236

#6 0x00007ff5ea80e9eb in wsgi_deadlock_thread (thd=<optimized out>, data=<optimized out>) at src/server/mod_wsgi.c:9233

#7 0x00007ff5eb689c02 in start_thread () from target:/lib64/libc.so.6

#8 0x00007ff5eb70ec40 in clone3 () from target:/lib64/libc.so.6

Thread 2 (Thread 0x7ff5e9c27640 (LWP 1145403) "httpd"):

#0 0x00007ff5eb70422d in select () from target:/lib64/libc.so.6

#1 0x00007ff5eb94f849 in apr_sleep () from target:/lib64/libapr-1.so.0

#2 0x00007ff5ea80c5cd in wsgi_monitor_thread (thd=<optimized out>, data=0x558fe9244ce0) at src/server/mod_wsgi.c:9510

#3 0x00007ff5eb689c02 in start_thread () from target:/lib64/libc.so.6

#4 0x00007ff5eb70ec40 in clone3 () from target:/lib64/libc.so.6

Thread 1 (Thread 0x7ff5eb879540 (LWP 1145305) "httpd"):

#0 0x00007ff5eb7019ff in poll () from target:/lib64/libc.so.6

#1 0x00007ff5eb947de1 in apr_poll () from target:/lib64/libapr-1.so.0

#2 0x00007ff5ea821821 in wsgi_daemon_main (daemon=0x558fe9244ce0, p=0x558fe9043c28) at src/server/mod_wsgi.c:9811

#3 wsgi_start_process (p=p@entry=0x558fe9043c28, daemon=0x558fe9244ce0) at src/server/mod_wsgi.c:10542

#4 0x00007ff5ea822d95 in wsgi_start_daemons (p=0x558fe9043c28) at src/server/mod_wsgi.c:10780

#5 0x0000558fe77d1ced in ap_run_pre_mpm ()

#6 0x00007ff5eb241695 in event_run () from target:/etc/httpd/modules/mod_mpm_event.so

#7 0x0000558fe77c76d8 in ap_run_mpm ()

#8 0x0000558fe77b564f in main ()

Regardless, your last question helped me figure this out. In my original post I noted that the base python installation had python 3.12.6 installed by default upon installing miniforge3 and the base python installation is also where I installed mod_wsgi (mamba install mod_wsgi). Then, when I made the virtual environments and installed python in those, I just let it install the latest version (mamba install python) which resulted in python 3.12.7 being installed. I thought this wouldn't matter to mod_wsgi since I thought mod_wsgi just needed any python version used to be 3.12.x, but it turns out I misunderstood and the python used in the virtual environments must match exactly the python version wherever mod_wsgi is installed. Once I downgraded the 3.12.7 versions in the virtual environments to 3.12.6 to match the base environment where mod_wsgi was installed, everything started working. Thanks Graham!

Reply all

Reply to author

Forward

0 new messages