strptime issue in multi-threaded application
Joe Holloway
We've noticed on a couple occasions that calls into time.strptime have
failed with this exception:
ImportError: Failed to import _strptime because the import lockis
[sic] held by another thread.
I poked around the source code enough to realize that this is
apparently due to time.strptime using PyImport_ImportModuleNoBlock
which potentially raises an ImportError rather than waiting for the
"import lock" to be released [1]. This appears to have been
introduced as a workaround for other thread safety concerns [2].
Does this indicate that strptime and any other library function that
uses the non-blocking import call in this fashion are not thread safe?
Is there an idiomatic way of dealing with this error in
multi-threaded applications?
Like I mentioned, it's only happened on a couple occasions because the
right conditions have to be in place, but something doesn't seem right
about it. I thought I'd ask on the mailing list before going so far
as to open a ticket, but feel free to direct me there if that's the
appropriate place for this.
Thanks,
Joe
[1] http://www.python.org/doc/2.6/c-api/import.html#PyImport_ImportModuleNoBlock
[2] http://svn.python.org/view?view=rev&revision=59678
MRAB
A simple workaround might be to sleep a short time and then retry.
Scott David Daniels
> Joe Holloway wrote:
>> We recently uplifted our web application to run on Python 2.6.2.
>> We've noticed on a couple occasions that calls into time.strptime have
>> failed with this exception:
>
> A simple workaround might be to sleep a short time and then retry.
Can you just import and use time.strptime once before you start the
threads?
--Scott David Daniels
Scott....@Acm.Org
Christian Heimes
> We recently uplifted our web application to run on Python 2.6.2.
> We've noticed on a couple occasions that calls into time.strptime have
> failed with this exception:
>
> [sic] held by another thread.
The error message is my fault. The cause of the mistake is obvious:
PyErr_Format(PyExc_ImportError,
"Failed to import %.200s because the import lock"
"is held by another thread.",
name);
> I poked around the source code enough to realize that this is
> apparently due to time.strptime using PyImport_ImportModuleNoBlock
> which potentially raises an ImportError rather than waiting for the
> "import lock" to be released [1]. This appears to have been
> introduced as a workaround for other thread safety concerns [2].
Without PyImport_ImportModuleNoBlock() your application would block forever.
> Does this indicate that strptime and any other library function that
> uses the non-blocking import call in this fashion are not thread safe?
> Is there an idiomatic way of dealing with this error in
> multi-threaded applications?
It's not a matter of thread safety per se. I've added the function to
work around a dead lock situation. Python allows you to import a module
in a subthread but the entire import system is protected by a lock.
> Like I mentioned, it's only happened on a couple occasions because the
> right conditions have to be in place, but something doesn't seem right
> about it. I thought I'd ask on the mailing list before going so far
> as to open a ticket, but feel free to direct me there if that's the
> appropriate place for this.
I have an idea what might happen in your application. Is an import
triggering the start of a thread? You can get around the issue by
decoupling imports from thread startups. Your application should import
all modules before it starts its threaded components.
For now you can decrease the severity of your issue by placing "import
_strptime" next to "import time" somewhere in your code. Once it a
module is loaded PyImport_ImportModuleNoBlock() will not fail to import
it a second time.
Christian