I have found an issue where the IsolatedProcessHost will end up in a state where it is trying to read from the IPC connection but never gets an answer.
I can replicate the issue by starting a lot of processes at the same time (I am just starting like 10 Gallio echo clients running around 1000 tests). With that said, the problem can also occur with less processes running, but the likelihood is increased with the number of processes starting and running.
The problem is occurring when the IsolatedProcessHost is pinging the hostservice in the method WaitUntilReady. The ping is expecting an exception to be thrown if the processes is not connected to the pipe, which is mostly what is happening. But in some cases the IsolatedProcessHost will end up trying to read from the pipe but never get an answer, which will end up in a timeout.
As far as I have noticed from by debugging the state is occurring when a ping is sent and the process is starting listening at the same time. Here is a piece from one of my logs:
<logEntry severity="debug" message="Ping: 3:45:06 PM" />
<logEntry severity="debug" message="Ping failed: 3:45:06 PM" />
<logEntry severity="debug" message="Ping: 3:45:06 PM" />
<logEntry severity="debug" message="Listening for connections on IPC port: 'IsolatedProcessHost.ca1395abc958a834'" />
<logEntry severity="debug" message="Ping timed out: 3:45:12 PM" />
I have made a patch that will solve the problem, at least for the tests I ran. I had a hard time getting the Project build, and I have not succeeded in getting the tests to run, so I am not sure if I have broken anything els.