documentation?
Gary Robinson
Also, the Status page says: "In the current revision (58353-2), shareddict is a mutable dict shared between threads. It may be modified at any time, with the only constraint being that the contents must themselves be shareable." I'm not clear on what it means to be shareable. Is that based on inheriting from Monitor?
It would be helpful if there was some kind of tutorial page. (With the caveat that I know the project is far from complete at this point.)
I look forward to playing with it when it's ready -- I'm not sure if it's now at a place where it's pretty stable if you stick with the current features.
--
Gary Robinson
CTO
Emergent Music, LLC
personal email: gar...@mac.com
work email: grob...@emergentmusic.com
Company: http://www.emergentmusic.com
Blog: http://www.garyrobinson.net
Adam Olsen
>
> Hi, I'm looking at safethread again. It seems like there must be something I'm missing. For instance at http://code.google.com/p/python-safethread/wiki/Finalization you say "http://code.google.com/p/python-safethread/wiki/Finalization" you way "Since this happens in another thread, MyFile instances must be shareable, thus the use of Monitor and monitormethod," but I'm not seeing any documentation on Monitor and monitormethod. So I'm finding it very hard to get a sense of how I would use safethread in real code.
>
> Also, the Status page says: "In the current revision (58353-2), shareddict is a mutable dict shared between threads. It may be modified at any time, with the only constraint being that the contents must themselves be shareable." I'm not clear on what it means to be shareable. Is that based on inheriting from Monitor?
>
> It would be helpful if there was some kind of tutorial page. (With the caveat that I know the project is far from complete at this point.)
>
> I look forward to playing with it when it's ready -- I'm not sure if it's now at a place where it's pretty stable if you stick with the current features.
I've gone and added 3 more pages: Shareability, Monitors, and
shareddict. Hopefully this'll be enough to clear up any confusion.
Either way, thank you for your feedback!
--
Adam Olsen, aka Rhamphoryncus
Gary Robinson
I'd probably be trying it first on OS X... any reason that would be an issue? Or do you advise another OS?
--
Gary Robinson
CTO
Emergent Music, LLC
personal email: gar...@mac.com
work email: grob...@emergentmusic.com
Company: http://www.emergentmusic.com
Blog: http://www.garyrobinson.net
Adam Olsen
>
> I took a look at the new pages. They seem helpful. I'm not sure whether I'll have more questions when I try to use it -- I'd be trying it today if I didn't have a lot of other stuff I need to get done.
>
> I'd probably be trying it first on OS X... any reason that would be an issue? Or do you advise another OS?
I've only tested it on linux. There's probably various little
breakages for most other platforms, except the pythread wrappers which
are going to be really broken if you're not using pthreads. A quick
google suggests OS X uses cthreads, so.. breakage.
Ultimately it needs to be fixed, but it doesn't seem worthwhile right now.
Adam Olsen
I spoke too soon. It seems OS X also provides a pthreads API, it's
even the preferred API. Whether python uses it by default is another
matter..
Gary Robinson
Another issue. An application I'll be building has a couple of GB of data. The data is constant after it's loaded. But, it consists of complex objects -- instances of a class I'll be defining. The instances are accessed through a dictionary. I don't want to run separate processes because I don't want to have to duplicate all that data -- it would just require too much memory. Hence my interest in safethread.
My understanding is that using a shareddict for the dictionary doesn't solve the problem in itself. The official solution appears to be to make the mapped items inherit from Monitor -- is that right? But I'm concerned about the overhead associated with Monitor's automatic blocking and unblocking the threads all the time when, say, 7 threads on 7 CPU's are working with my objects.
Maybe the blocking and unblocking operations are fast enough relative to the work done with that data, that it wouldn't be an issue and threads would only rarely be blocked. That's very possible.
But out of curiosity, I'm wondering what would happen if I just made shareddict map to the original objects themselves without involving Mutable. Since none of the threads would be modifying the data while the worker threads were using it, I don't see any reason why corruption or difficulty would happen.
I know that you are working to create a foolproof system that protects people from corruption, but if someone really knows that the data isn't going to be modified, is it possible to use a shareddict as I outline above?
I tend to assume that the answer is no because the GIL would come into play for multiple threads accessing the objects -- or have you disabled it completely? Or are there other problems?
--
Gary Robinson
CTO
Emergent Music, LLC
personal email: gar...@mac.com
work email: grob...@emergentmusic.com
Company: http://www.emergentmusic.com
Blog: http://www.garyrobinson.net
Adam Olsen
>
> OK, good. Though I have Linux boxes available too so if there are any problems I'll try it there.
>
> Another issue. An application I'll be building has a couple of GB of data. The data is constant after it's loaded. But, it consists of complex objects -- instances of a class I'll be defining. The instances are accessed through a dictionary. I don't want to run separate processes because I don't want to have to duplicate all that data -- it would just require too much memory. Hence my interest in safethread.
>
> My understanding is that using a shareddict for the dictionary doesn't solve the problem in itself. The official solution appears to be to make the mapped items inherit from Monitor -- is that right? But I'm concerned about the overhead associated with Monitor's automatic blocking and unblocking the threads all the time when, say, 7 threads on 7 CPU's are working with my objects.
>
> Maybe the blocking and unblocking operations are fast enough relative to the work done with that data, that it wouldn't be an issue and threads would only rarely be blocked. That's very possible.
>
> But out of curiosity, I'm wondering what would happen if I just made shareddict map to the original objects themselves without involving Mutable. Since none of the threads would be modifying the data while the worker threads were using it, I don't see any reason why corruption or difficulty would happen.
>
> I know that you are working to create a foolproof system that protects people from corruption, but if someone really knows that the data isn't going to be modified, is it possible to use a shareddict as I outline above?
>
> I tend to assume that the answer is no because the GIL would come into play for multiple threads accessing the objects -- or have you disabled it completely? Or are there other problems?
The official solution is "not yet implemented". Monitor is suitable
for many things, but you're right that yours isn't one of them.
However, there's no underlying reason why something suitable couldn't
be created - it just hasn't been designed and implemented yet.
It sounds like what you want is immutable instances. The design
questions for them revolve around how to construct them in the first
place and if they allow future modifications or make it outright
impossible.
Probably the simplest solution is to give them a shareddict for their
__dict__. That ends up with an honour system for not modifying them
later, which I'm not sure I like.
Adam Olsen
Oh, and I should say that I don't want to be "foolproof". It's more
about tradeoffs here. I want a strong *tendency* towards correctness,
and robustness when it's not correct (such as detecting deadlocks),
but I don't want to stop people from writing useful programs.
Also, the GIL is completely gone. It's impossible to share an object
between threads unless it's already thread-safe (such as the
shareddict) or without protecting it inside a Monitor (from which it
can't escape).