Segmentation fault while using requests module with lemoncheesecake
23 views
Skip to first unread message
Akshay Maldhure
Mar 11, 2020, 11:02:51 PM3/11/20
to lemoncheesecake
I'm getting segmentation fault quite frequently while running some code. Here's the nearly full trace of the issue.
Thread 0x00007f6c37fff700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c54e79700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c5567a700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c55e7b700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c566bc700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c56ebd700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c576be700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c57f7f700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c6ced6700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/multiprocessing/pool.py", line 463 in _handle_results
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c6d6d7700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/multiprocessing/pool.py", line 415 in _handle_tasks
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f6c6ded8700 (most recent call first):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 406 in _handle_workers
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Current thread 0x00007f6c6e6d9700 (most recent call first):
File "/usr/lib/python3.6/json/decoder.py", line 355 in raw_decode
File "/usr/lib/python3.6/json/decoder.py", line 339 in decode
File "/usr/lib/python3.6/json/__init__.py", line 354 in loads
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/requests/models.py", line 897 in json
File "/var/lib/jenkins/workspace/my_project/api_helpers/some_consumer_module.py", line 13 in get_some_status
The code in highlighted line 13 does .json()["access_some_value"] on the response object of the requests module.
What I'm unable to understand is whether this issue is arising from the lemoncheesecake library or from the requests library or from some other module.
Nicolas Delon
Mar 12, 2020, 6:10:05 PM3/12/20
to lemoncheesecake
Hello,
I guess it's not coming from lemoncheesecake specifically.
Did you try to isolate the issue ?
Does it happen on only one machine (your CI host) ?
Does it happen when you run test with multiple theads ? One thread ?
Does it happen on a particular test ?
Regards.
Akshay Maldhure
Mar 12, 2020, 10:39:35 PM3/12/20
to lemoncheesecake
Hi Nicolas,
I forgot to put the full stack trace. Here it is.
Thread 0x00007ffae9ffb700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffaea7fc700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffaeaffd700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffaeb7fe700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffaebfff700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb08f39700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb0973a700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb09ffb700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb0a7fc700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/multiprocessing/pool.py", line 463 in _handle_results
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb0affd700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/usr/lib/python3.6/multiprocessing/pool.py", line 415 in _handle_tasks
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb0b7fe700 (most recent call first):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 406 in _handle_workers
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb0bfff700 (most recent call first):
File "/usr/lib/python3.6/socket.py", line 745 in getaddrinfo
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61 in create_connection
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/urllib3/connection.py", line 157 in _new_conn
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/urllib3/connection.py", line 184 in connect
File "/usr/lib/python3.6/http/client.py", line 974 in send
File "/usr/lib/python3.6/http/client.py", line 1036 in _send_output
File "/usr/lib/python3.6/http/client.py", line 1249 in endheaders
File "/usr/lib/python3.6/http/client.py", line 1300 in _send_request
File "/usr/lib/python3.6/http/client.py", line 1254 in request
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387 in _make_request
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672 in urlopen
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/requests/adapters.py", line 449 in send
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/requests/sessions.py", line 646 in send
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/requests/sessions.py", line 533 in request
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/requests/api.py", line 60 in request
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/requests/api.py", line 75 in get
... test code here references here
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/task.py", line 67 in run_task
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/task.py", line 94 in handle_task
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119 in worker
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Current thread 0x00007ffb10f8f700 (most recent call first):
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/report.py", line 34 in format_time_as_iso8601
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 24 in _serialize_time
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 52 in _serialize_steps
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 96 in _serialize_result
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 103 in _serialize_test_result
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 112 in _serialize_suite_result
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 139 in serialize_report_into_json
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 148 in save_report_into_file
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backends/json_.py", line 300 in save_report
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backend.py", line 51 in _save
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/reporting/backend.py", line 56 in _handle_event
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/events.py", line 44 in handle
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/events.py", line 98 in handle_event
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/events.py", line 125 in _handler_loop
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007ffb16bed700 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 295 in wait
File "/usr/lib/python3.6/queue.py", line 164 in get
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/task.py", line 149 in run_tasks
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/runner.py", line 562 in run_session
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/runner.py", line 594 in run_suites
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/cli/commands/run.py", line 156 in run_suites_from_project
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/cli/commands/run.py", line 235 in run_cmd
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/lemoncheesecake/cli/main.py", line 53 in main
File "/var/lib/jenkins/.local/bin/lcc", line 11 in <module>
Do you think we should first ensure that lemoncheesecake is not the potential cause here since it is the entry point to the failure? Would it be a straightforward process to conclude the same?
Nicolas Delon
Mar 13, 2020, 5:40:02 PM3/13/20
to lemoncheesecake
lemoncheesecake is a library like another, and I would not make it more "suspect" than another one.
Did you see my "check list" in my previous message ?
Akshay Maldhure
Mar 20, 2020, 2:54:45 AM3/20/20
to lemoncheesecake
Yes, I saw that, and we have decided to debug the issue on the current machine itself. Since I do not see the issue locally on my machine, we believe it needs more investigation on the Jenkins CI machine where this is seen.
Akshay Maldhure
Mar 23, 2020, 2:37:03 AM3/23/20
to lemoncheesecake
Apparently, it seems pip was pointing to the incorrect Python version (Python 3.5) on the target machine. After fixing the same, this issue is not seen anymore.
pip 20.0.2 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)
Reply all
Reply to author
Forward
0 new messages