bug: Deadlock starting capture

Stijn

unread,

Apr 15, 2026, 8:57:18 AM (7 days ago) Apr 15

to Pyomo Forum

Dear Pyomo,

We occassionally get the following bug:

Internal Pyomo implementation error:
'Deadlock starting capture_output'
Please report this to the Pyomo Developers.

And a bit more rare the following:

'Deadlock closing capture_output'

We create Azure C# functions running Python that use Pyomo to build the models that are sent to the Gurobi compute server. We use gurobi_direct from Pyomo. (as follows: model_solver = pyo.SolverFactory("gurobi_direct", solver_io="python", manage_env=True, options=options) )

When the bugs occur it is when 2 or more of these async functions are started in a short time period, but that is no guarantee of getting the bug.
(The Gurobi compute server cluster manager show these calls stuck in INIT status)

Can you help us? The bug message seems to suggest we will not be able to fix this ourselves

We are using Python version 3.13, Pyomo 6.10.0, azure-functions 1.24.0, python-dotenv 1.2.1.

Kind regards, Stijn

Siirola, John

unread,

Apr 15, 2026, 9:04:20 AM (7 days ago) Apr 15

to pyomo...@googlegroups.com

Hi Stijn,

Please see https://github.com/Pyomo/pyomo/issues/3908. We could certainly use your help diagnosing / reproducing / testing solutions.

Best,

John

From: pyomo...@googlegroups.com <pyomo...@googlegroups.com> On Behalf Of Stijn
Sent: Wednesday, April 15, 2026 6:12 AM
To: Pyomo Forum <pyomo...@googlegroups.com>
Subject: [EXTERNAL] bug: Deadlock starting capture_output

You don't often get email from stijn...@gmail.com. Learn why this is important

Dear Pyomo,

We occassionally get the following bug:

Internal Pyomo implementation error:
'Deadlock starting capture_output'
Please report this to the Pyomo Developers.

And a bit more rare the following:

'Deadlock closing capture_output'

We create Azure C# functions running Python that use Pyomo to build the models that are sent to the Gurobi compute server. We use gurobi_direct from Pyomo. (as follows: model_solver = pyo.SolverFactory("gurobi_direct", solver_io="python", manage_env=True, options=options) )

When the bugs occur it is when 2 or more of these async functions are started in a short time period, but that is no guarantee of getting the bug.
(The Gurobi compute server cluster manager show these calls stuck in INIT status)

Can you help us? The bug message seems to suggest we will not be able to fix this ourselves

We are using Python version 3.13, Pyomo 6.10.0, azure-functions 1.24.0, python-dotenv 1.2.1.

Kind regards, Stijn

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pyomo-forum/bb588622-01b0-4731-81b5-b226b5a2eac0n%40googlegroups.com.

Stijn

unread,

Apr 15, 2026, 9:25:43 AM (7 days ago) Apr 15

to Pyomo Forum

Thanks John, with regards to your comment there. We see the ~100 seconds pop up in a different way. In our Gurobi Compute Server manager, we can see all runs. Whenever we encounter this deadlock bug, those runs get stuck in INIT status. The first one times out after 30 minutes, but all the other ones time out after ~100 seconds.

What I forgot to mention is that after a first deadlock issue, every single new attempt within 30 minutes will also give the deadlock bug. See the screenshot: (the normal optimization status would be e.g. OPTIMAL or INF_OR_UNBD)

Stijn

unread,

Apr 16, 2026, 4:30:09 AM (7 days ago) Apr 16

to Pyomo Forum

Please find the stacktrace below.

It seems to originate with self._solver_model.setParam, which has a bit of a interesting comment above it.

# We would like to just set OutputFlag to turn off the console

# log, but that prevents users from using the Gurobi LogFile

# option (see #3589 / #3716). BUT just setting LogToConsole

# triggers Gurobi to write a message to the console. We will

# capture that message here:

with capture_output(capture_fd=True):

self._solver_model.setParam('LogToConsole', int(bool(self._tee)))

Stacktrace local

File \"C:\\Users\\..\\..\\second-instance\\..\\Solver\\source\\..\\ModelExecution\\model_runner.py\", line 96, in _run_model_solver
results = model_solver.solve(
self.pyomo_model,
tee=True if (local and verbose) else False,
symbolic_solver_labels=True,
)
File \"C:\\Users\\..\\..\\second-instance\\..\\Solver\\.venv\\Lib\\site-packages\\pyomo\\solvers\\plugins\\solvers\\direct_solver.py\", line 140, in solve
_status = self._apply_solver()
File \"C:\\Users\\..\\..\\second-instance\\..\\Solver\\.venv\\Lib\\site-packages\\pyomo\\solvers\\plugins\\solvers\\gurobi_direct.py\", line 244, in _apply_solver
with capture_output(capture_fd=True):
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File \"C:\\Users\\..\\..\\second-instance\\..\\Solver\\.venv\\Lib\\site-packages\\pyomo\\common\\tee.py\", line 361, in __enter__
raise DeveloperError(\"Deadlock starting capture_output\")
pyomo.common.errors.DeveloperError: Internal Pyomo implementation error:

'Deadlock starting capture_output'
Please report this to the Pyomo Developers.

Stacktrace in cloud environment

{"statusCode":"InternalServerError","errorMessage":"ERROR:
Traceback (most recent call last):
File \u0022C:\\home\\site\\wwwroot\\Solver\\run_optimizer.py\u0022, line 18, in run_optimizer
response_json = xyz.run_model(local=False, json_dict=optimizer_input, verbose=False)
File \u0022C:\\home\\site\\wwwroot\\Solver\\source\\..\\application\\...py\u0022, line 64, in run_model
raise e
File \u0022C:\\home\\site\\wwwroot\\Solver\\source\\..\\application\\...py\u0022, line 41, in run_model
solved_model = xyz.model_controller.solve_model(
model=model,
...\u003C2 lines\u003E...
timeout=timeout,
)
File \u0022C:\\home\\site\\wwwroot\\Solver\\source\\..\\Controllers\\model_controller.py\u0022, line 45, in solve_model
solved_model = ModelRunner(model, data_model).run_model(
local=local,
timeout=timeout,
verbose=self.verbose,
)
File \u0022C:\\home\\site\\wwwroot\\Solver\\source\\..\\ModelExecution\\model_runner.py\u0022, line 74, in run_model
raise e
File \u0022C:\\home\\site\\wwwroot\\Solver\\source\\..\\ModelExecution\\model_runner.py\u0022, line 66, in run_model
self._run_model_solver(
~~~~~~~~~~~~~~~~~~~~~~^
solver=solver,
^^^^^^^^^^^^^^
...\u003C2 lines\u003E...
verbose=verbose,
^^^^^^^^^^^^^^^^
)
^
File \u0022C:\\home\\site\\wwwroot\\Solver\\source\\..\\ModelExecution\\model_runner.py\u0022, line 108, in _run_model_solver
raise e
File \u0022C:\\home\\site\\wwwroot\\Solver\\source\\..\\ModelExecution\\model_runner.py\u0022, line 96, in _run_model_solver
results = model_solver.solve(
self.pyomo_model,
tee=True if (local and verbose) else False,
symbolic_solver_labels=True,
)
File \u0022C:\\home\\site\\wwwroot\\Solver\\.venv\\Lib\\site-packages\\pyomo\\solvers\\plugins\\solvers\\direct_solver.py\u0022, line 140, in solve
_status = self._apply_solver()
File \u0022C:\\home\\site\\wwwroot\\Solver\\.venv\\Lib\\site-packages\\pyomo\\solvers\\plugins\\solvers\\gurobi_direct.py\u0022, line 244, in _apply_solver
with capture_output(capture_fd=True):
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File \u0022C:\\home\\site\\wwwroot\\Solver\\.venv\\Lib\\site-packages\\pyomo\\common\\tee.py\u0022, line 361, in __enter__
raise DeveloperError(\u0022Deadlock starting capture_output\u0022)
pyomo.common.errors.DeveloperError: Internal Pyomo implementation error:
\u0027Deadlock starting capture_output\u0027

Please report this to the Pyomo Developers.
"}

Stijn

unread,

Apr 20, 2026, 4:28:31 AM (3 days ago) Apr 20

to Pyomo Forum

We've seem to have fixed the issue by creating a gurobi_direct_override.py, that removes this with capture_output.
Essentially we revert this commit: https://github.com/Pyomo/pyomo/commit/6b8ac9d5e13c06121e68b1e14aa80d5a009f1278

That line wasn't safe in our async/parallel setup.

Reply all

Reply to author

Forward